Monitor and Troubleshoot KalamDB

Start with connection health, then inspect the query, job, or subscription that is failing. Use the same server URL and identity as the affected application.

Check KalamDB connectivity

BASH

1kalam doctor2kalam --url http://127.0.0.1:2900 -c 'SELECT 1;'3curl --fail http://127.0.0.1:2900/health

doctor checks CLI configuration and connectivity. SQL verifies an authenticated request. The health endpoint is localhost-only: run it on the server host; a remote health rejection does not mean the database is down. For container deployments, check the published port and the container logs too.

Inspect background work

Run as a DBA or system account:

SQL

1SELECT job_id, job_type, status, message, exception_trace2FROM system.jobs3ORDER BY created_at DESC4LIMIT 20;5 6SELECT * FROM system.live;7SELECT * FROM system.cluster;8SELECT * FROM system.cluster_groups;

Jobs expose states such as queued, running, retrying, completed, and failed. A returned job ID means work was accepted; wait for completion before relying on a backup or flush. system.live is node-local, so inspect the node serving the WebSocket when diagnosing an active subscription.

Use Admin UI Logs & Monitoring to inspect activity visually and Jobs for background work. Configure log destinations and levels in Logging.

Match the symptom to a check

Symptom	Check	Next action
Connection refused	Exact host, port, process, container port mapping	Start or expose the intended listener; distinguish HTTP from PGWire
Authentication rejected	Login method, token expiry, OIDC issuer and audience	Follow authentication setup
SELECT returns no rows	Effective user and table type; SHARED table policies	Test with the application identity and inspect RLS
Live view is stale	WebSocket errors, query restrictions, `system.live` on the serving node	Follow reconnection and recovery
Query is slow	Query plan, scanned rows, cold files, result size	Use EXPLAIN ANALYZE and compare like-for-like runs
Backup or flush has not finished	`system.jobs.status`, message and exception trace	Resolve the recorded failure before restarting or copying output
Writes fail during node loss	Group leadership, reachable peers, quorum	Inspect cluster operations
CLI says ready but the app stopped	App process and CLI/app logs	Verify actual app output; database readiness alone is insufficient

Diagnose a slow query

SQL

1EXPLAIN ANALYZE2SELECT * FROM app.todos WHERE completed = FALSE;

Run this against your actual table. ANALYZE executes the query. Compare returned rows, latency, hot rows scanned, and cold files scanned before and after a change. Do not infer realtime delivery capacity from a single SQL query’s latency.

Collect a useful incident report

Record server and CLI versions, deployment topology, timestamps, sanitized SQL, the full error code/message, relevant job or subscription IDs, and the smallest reproduction. Remove tokens and passwords from logs before sharing them. Keep raw benchmark reports when investigating throughput changes.

Next step

Understand query plans →

Reference: System views.