# KalamDB: full documentation corpus

> Single-file Markdown corpus of current KalamDB and KoldStore documentation for
> LLM context windows, AI search engines, and developer agents. Prefer this file
> when you need dense technical detail in one request. Prefer
> [https://kalamdb.org/llms.txt](https://kalamdb.org/llms.txt) for a short index.

- Site: https://kalamdb.org
- Index: https://kalamdb.org/llms.txt
- Sitemap: https://kalamdb.org/sitemap.xml
- Generated: 2026-07-31
- Scope: current docs under `content/` (archived version folders omitted)
- License: Apache 2.0 for KalamDB / KoldStore product docs on this site

## How to use this file

- Read the index at `/llms.txt` first for routing.
- Use this corpus for install steps, SQL APIs, architecture, and SDK reference.
- Cite canonical page URLs included under each section header.
- KoldStore (PostgreSQL tiered storage) lives under `/docs/pg-koldstore`.
- pg_kalam (FDW to KalamDB) lives under `/docs/pg-kalam` — different product.


================================================================================
# Documentation Home
URL: https://kalamdb.org/docs
Source: content/index.mdx

> Browse KalamDB documentation by product area: server, pg_kalam, pg Koldstore, TypeScript SDK, Rust SDK, Dart SDK, and use cases.
# KalamDB Documentation

KalamDB docs are organized by the part of the stack you are working with. Start with the server docs for running KalamDB and SQL, then jump into an SDK, a PostgreSQL extension, or implementation patterns.

## Choose A Docs Area

- [Server](https://kalamdb.org/docs/server): run KalamDB, configure auth, use SQL, operate storage, and deploy production services.
- [CLI](https://kalamdb.org/docs/cli): install `kalam`, scaffold projects with `kalam init`, run `kalam dev`, manage migrations, and use the SQL client.
- [pg_kalam PostgreSQL Extension](https://kalamdb.org/docs/pg-kalam): connect PostgreSQL to KalamDB through the pg_kalam extension.
- [pg Koldstore](https://kalamdb.org/docs/pg-koldstore): tier hot PostgreSQL rows and cold Parquet on one table — install, manage, flush, merge scan, and `changes_since`.
- [TypeScript SDK](https://kalamdb.org/docs/ts-sdk): build browser apps, services, workers, and agents with the TypeScript packages.
- [Rust SDK](https://kalamdb.org/docs/rust-sdk): native async client for services and workers — install [`kalam-client`](https://crates.io/crates/kalam-client) from crates.io.
- [Dart / Flutter SDK](https://kalamdb.org/docs/dart-sdk): connect Flutter and Dart apps to KalamDB with realtime subscriptions.
- [Use Cases](https://kalamdb.org/docs/use-cases): runnable TypeScript example apps from the `examples/` folder.

## Start Here

1. [Run KalamDB locally in minutes](https://kalamdb.org/docs/server/getting-started)
2. [Create tables and query data with SQL](https://kalamdb.org/docs/server/sql-reference)
3. [Connect a TypeScript app or service](https://kalamdb.org/docs/ts-sdk/setup)
4. [Install the Rust SDK from crates.io](https://kalamdb.org/docs/rust-sdk/setup) and try the [Quickstart example](https://kalamdb.org/docs/rust-sdk/examples/quickstart)
5. [Use pg_kalam for PostgreSQL-facing workflows](https://kalamdb.org/docs/pg-kalam/getting-started)
6. [Try pg Koldstore for tiered hot/cold tables](https://kalamdb.org/docs/pg-koldstore/getting-started)
7. [Review production security settings](https://kalamdb.org/docs/server/security)

================================================================================
# Benchmarks
URL: https://kalamdb.org/docs/pg-koldstore/benchmarks
Source: content/pg-koldstore/benchmarks.mdx

> Published KoldStore storage and VACUUM FULL wins on a 10M-row sample, plus contextual DML and PK lookup trade-offs.
# Benchmarks

KoldStore is a **storage lifecycle** tool — not a universal query accelerator.
The primary published wins are footprint and whole-table maintenance after
flush. DML and cold-path lookups are contextual single samples, not release
guarantees.

Source sample: 10M wide rows, `hot_row_limit = 100000`, local PostgreSQL 16.13,
2026-07-31, single wiped pgrx instance per side (draft publication methodology
in the [koldstore repo](https://github.com/kalamdb/koldstore/tree/main/docs/benchmarks)).

## Storage and maintenance wins

| Result | Before → after flush | Trade-off |
|--------|----------------------|-----------|
| Total footprint (hot + cold) | 5.85 GiB → 671 MiB | **89%** smaller |
| Hot in PostgreSQL (heap + `__cl`) | 5.85 GiB → 72 MiB | **99%** smaller |
| Cold Parquet | — → ~599 MiB | Outside the database |
| Indexes (hot + `__cl`) | 415 MiB → 11.5 MiB | **97%** smaller |
| `VACUUM (FULL, ANALYZE)` | 227.02 s → 3.21 s | **~71×** faster |

The VACUUM timing is specifically `VACUUM (FULL, ANALYZE)` — a whole-table
rewrite. Routine autovacuum was disabled for the benchmark and is not covered
by that number.

## DML and query path (contextual)

Foreground DML and PK lookups are **not** guaranteed to improve.

| Operation | PostgreSQL only | KoldStore (WAL) | Notes |
|-----------|-----------------|-----------------|-------|
| INSERT | 47,202 ops/s | 101,325 ops/s | Noise / order — not a product win |
| UPDATE | 63,453 ops/s | 55,350 ops/s | Single sample; ~13% lower |
| DELETE | 36,905 ops/s | 140,874 ops/s | Do not claim faster from one sample |
| Hot-only PK lookup | 3,990 ops/s | 3,818 ops/s | ≈ same |
| Hot+cold PK lookup | 4,204 ops/s | 1,055 ops/s | ~75% slower vs full-heap baseline |
| Cold-only PK lookup | 4,182 ops/s | 653 ops/s | ~84% slower vs full-heap baseline |

Cold-path lookups compare Parquet merge against a full-heap PostgreSQL
baseline. Treat them as indicative.

## How to read the numbers

- Lead with storage + VACUUM when explaining the product
- Do not market single-sample INSERT/DELETE “speedups”
- Expect cold PK latency until cold lookup work lands on the roadmap
- Results vary by schema, hardware, storage backend, and workload

Product page summary: [/koldstore](https://kalamdb.org/koldstore).

================================================================================
# Change Feed
URL: https://kalamdb.org/docs/pg-koldstore/change-feed
Source: content/pg-koldstore/change-feed.mdx

> Use koldstore.changes_since for exclusive seq catch-up across hot and cold, including last_rows newest-N rewind.
# Change Feed

`koldstore.changes_since` is the packaged SQL change API for managed tables. It
merges the hot change-log mirror and cold Parquet into one **latest-state**
stream ordered by `seq`.

This is catch-up for clients (and aligns with KalamDB subscribe semantics for
`from` / newest-N). It is **not** a full temporal audit log of every historical
event.

## Signature

```sql
koldstore.changes_since(
  table_name regclass,
  since_seq  bigint  DEFAULT 0,      -- exclusive resume (seq > since_seq)
  limit_rows integer DEFAULT 1000,   -- forward page size
  last_rows  integer DEFAULT NULL    -- newest-N rewind when since_seq = 0
)
RETURNS TABLE (
  seq        bigint,
  op         smallint,   -- 1=INSERT, 2=UPDATE, 3=DELETE
  pk         jsonb,
  deleted    boolean,
  row_image  jsonb,
  source     text        -- 'hot' | 'cold'
)
```

## Modes

| Mode | Arguments | Behavior |
|------|-----------|----------|
| Resume | `since_seq > 0` | Rows with `seq > since_seq`, ascending, capped by `limit_rows`. **`last_rows` is ignored.** |
| Rewind | `since_seq = 0` and `last_rows` set | Newest N retained changes, delivered **oldest → newest** |
| From start | `since_seq = 0`, no `last_rows` | Start of retained history |

Precedence rule: a positive `since_seq` always wins over `last_rows`.

## Examples

Fence first when you need a strong boundary against the heap:

```sql
SELECT koldstore.wait_for_async_mirror();
```

### Page from the beginning

```sql
SELECT seq, op, pk, deleted, source
FROM koldstore.changes_since(
  table_name => 'app.messages'::regclass,
  since_seq  => 0,
  limit_rows => 100
);
```

### Resume after the last consumed seq

```sql
SELECT seq, op, pk, deleted, source
FROM koldstore.changes_since(
  table_name => 'app.messages'::regclass,
  since_seq  => 184467440737095516,  -- last seq you already applied
  limit_rows => 500
);
```

Advance your cursor from the **highest** `seq` you successfully processed.

### Newest N (subscribe-style rewind)

```sql
SELECT seq, op, pk, deleted, source
FROM koldstore.changes_since(
  table_name => 'app.messages'::regclass,
  since_seq  => 0,
  limit_rows => 1000,
  last_rows  => 50
);
```

`last_rows` must be `<= limit_rows`.

### After flush

The same cursor continues to return flushed rows from cold (`source = 'cold'`).
Do not treat “mirror empty” as “caught up” if cold still holds retained history.

## Semantics and caveats

- **Exclusive** cursor: `seq > since_seq`, never inclusive.
- Authoritative `seq` values come only from the serialized WAL applier.
- Latest-state per PK — updates replace prior images; deletes appear as tombstones until pruned by flush retention rules.
- `since_seq = 0` means “start of retained history,” not “I fell behind retention.”
- A **positive** `since_seq` older than the retained cold floor raises a **retention-gap error** — do not treat an empty page as success in that case.
- Hot + cold merge; `source` tells you which tier contributed the row image.

## Direct mirror SQL (advanced)

You can inspect the hot mirror for debugging:

```sql
SELECT id, seq, op
FROM koldstore.messages__cl
WHERE seq > $1
ORDER BY seq
LIMIT 100;
```

Prefer `changes_since` in application code so cold history stays visible after
flush and retention-gap errors stay consistent.

## Next

- [SQL API Reference](https://kalamdb.org/docs/pg-koldstore/sql-api)
- [Under the Hood](https://kalamdb.org/docs/pg-koldstore/concepts)

================================================================================
# Under the Hood
URL: https://kalamdb.org/docs/pg-koldstore/concepts
Source: content/pg-koldstore/concepts.mdx

> How KoldStore captures committed WAL into a latest-state mirror, flushes to Parquet, and merges hot and cold with KoldMergeScan.
# Under the Hood

KoldStore keeps PostgreSQL authoritative for the hot tier and stores flushed
history as open Parquet plus catalog metadata. This page explains the moving
parts so API behavior makes sense.

## End-to-end path

```text
heap commit
  → logical slot (PK-only pgoutput)
  → database worker
  → koldstore.<table>__cl  (latest-state mirror)
  → flush (policy or force)
  → Parquet segments + manifest
  → SELECT via KoldMergeScan (newest visible wins)
```

There is **no** trigger/strict capture path today. Foreground DML writes the
heap only; the serialized WAL applier allocates authoritative `seq` values.

## Hot tier

- Ordinary heap table with a primary key
- Application columns only (clean schema — no KoldStore columns on the user table)
- Native indexes cover **hot rows only** after flush
- Transactions, locking, MVCC, RLS, and permissions stay PostgreSQL-native

## Change-log mirror (`koldstore.<table>__cl`)

One row per primary key (latest state), not an append-only event log:

| Column | Meaning |
|--------|---------|
| PK columns | Same as the heap |
| `seq` | Snowflake effect id — ordering / flush cutoffs / change cursor |
| `op` | `1` = INSERT, `2` = UPDATE, `3` = DELETE |
| `commit_lsn` | Diagnostics only — not the public resume cursor |

Tombstones (`op = 3`) remain until flush prunes them. `seq` is **not** a commit
LSN and is not a commit-order guarantee beyond “allocated by the serialized
applier.”

## WAL capture consistency

| Mode | Behavior |
|------|----------|
| Normal | Worker polls (~100 ms) and applies available WAL |
| Strong fence | `koldstore.wait_for_async_mirror()` drains to a boundary |
| Flush | Calls the same fence automatically |

Lag is expected. Call the fence before comparing heap state to mirror/`seq`
cursors when you need a strong boundary.

Other capture rules:

- Primary-key `UPDATE`s are rejected (BEFORE UPDATE OF pk guard)
- `TRUNCATE` on async managed tables is rejected
- Publication `koldstore_async_mirror` + one deterministic logical slot per database are auto-managed

## Cold tier

- Parquet segments under a registered storage backend
- Manifest metadata in PostgreSQL catalogs (`koldstore.manifest`, `koldstore.cold_segments`, …)
- Object storage is **not** WAL-protected — back up PostgreSQL **and** the cold prefix together

Default path templates:

- Shared: `{namespace}/{tableName}/`
- User-scoped (intended): `{namespace}/{tableName}/{scopeId}/`

Scoped cold folders and full per-`user_id` physical layout are still hardening;
session `koldstore.user_id` is required for user-typed tables today.

## Flush policy (intuition)

Example: `hot_row_limit = 10000`, `min_flush_rows = 1000`, `max_rows_per_file = 1000`

| Mirror rows | Result |
|-------------|--------|
| 10,505 | No flush (505 excess &lt; min) |
| 11,000 | Flush 1,000 → 1 file |
| 11,250 | Flush 1,000, keep 10,250 hot |

Age policy uses `koldstore_move_after` (for example `'90 days'`) measured from
mirror `seq`. It is mutually exclusive with `hot_row_limit`.
`koldstore_move_when` is reserved and fails closed.

## KoldMergeScan

```text
KoldMergeScan
├── native hot child (Index / Bitmap / Seq Scan)
├── KoldParquetScan
└── MirrorOverlay (inserts / updates / tombstones)
```

Contracts that matter for operators:

- Cold-capable predicates → `KoldMergeScan` with hot child + planned cold access
- Hot-only / cold-proven-empty PK lookups may keep a **native** plan with **no** wrapper (intentional performance path)
- `koldstore.enable_merge_scan = off` → **ERROR** at execution (never silent hot-only)
- Unavailable cold storage → query **fails** (no partial hot-only results)

## Catalog touchpoints

Useful relations/functions while debugging:

- `koldstore.schemas` — managed table options
- `koldstore.jobs` — migrate / flush progress
- `koldstore.describe_table(...)` — hot/mirror/cold counts, sizes, pending jobs
- `koldstore.async_mirror_status()` — slot lag, retained bytes, health

## Next

- [Manage & Flush](https://kalamdb.org/docs/pg-koldstore/manage-and-flush) — how to drive policy and jobs
- [Change Feed](https://kalamdb.org/docs/pg-koldstore/change-feed) — exclusive `seq` catch-up
- [SQL API Reference](https://kalamdb.org/docs/pg-koldstore/sql-api) — full function list

================================================================================
# Getting Started
URL: https://kalamdb.org/docs/pg-koldstore/getting-started
Source: content/pg-koldstore/getting-started.mdx

> Register storage, manage a heap table, fence the async mirror, flush to Parquet, query one table, and page the change feed.
# Getting Started

This walkthrough creates a managed table on local filesystem storage, flushes
excess rows to Parquet, and shows both merge-scan reads and the change feed.

Prerequisites: [Installation](https://kalamdb.org/docs/pg-koldstore/installation) completed —
`shared_preload_libraries` includes `koldstore`, `wal_level=logical`, and
`CREATE EXTENSION koldstore` succeeded.

## 1. Register storage

```sql
SELECT koldstore.register_storage(
  name         => 'local-dev',
  storage_type => 'filesystem',
  base_path    => '/tmp/koldstore-demo',
  credentials  => '{}'::jsonb,
  config       => '{}'::jsonb
);
```

`storage_type` may also be `s3`, `gcs`, or `azure`. For MinIO-style S3:

```sql
SELECT koldstore.register_storage(
  name         => 'minio',
  storage_type => 's3',
  base_path    => 'koldstore-bucket',
  credentials  => '{"access_key_id":"minioadmin","secret_access_key":"minioadmin"}'::jsonb,
  config       => '{"endpoint":"http://127.0.0.1:9000","region":"us-east-1","path_style":true}'::jsonb
);
```

## 2. Create a normal heap table

Managed tables need a **primary key**. Supported types today include `boolean`,
integers, `real`, `double precision`, `text`, `varchar`, `uuid`, `jsonb`, and
`timestamptz`.

```sql
CREATE SCHEMA IF NOT EXISTS app;

CREATE TABLE app.messages (
  id bigint PRIMARY KEY,
  body text NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now()
);

INSERT INTO app.messages (id, body)
SELECT gs, 'row ' || gs
FROM generate_series(1, 1012) AS gs;
```

## 3. Manage the table

Two equivalent styles:

**A. `manage_table` (structured options)**

```sql
SELECT koldstore.manage_table(
  table_name         => 'app.messages',
  storage            => 'local-dev',
  hot_row_limit      => 1000,
  min_flush_rows     => 1,
  max_rows_per_file  => 500,
  migration_order_by => 'id'
);
```

**B. `ALTER TABLE` options**

```sql
ALTER TABLE app.messages SET (
  koldstore_enabled = true,
  koldstore_storage = 'local-dev',
  koldstore_hot_row_limit = 1000,
  koldstore_min_flush_rows = 1,
  koldstore_max_rows_per_file = 500
);
```

Do **not** turn management off with `koldstore_enabled = false` — use
`koldstore.unmanage_table(...)` instead. Storage cannot be changed after manage.

With `hot_row_limit` set, `manage_table` rejects non-PK `UNIQUE` constraints and
foreign keys (those stay hot-only after flush). See
[Status & Limits](https://kalamdb.org/docs/pg-koldstore/limitations).

## 4. Fence the async mirror

Foreground DML writes the heap only. A database worker applies committed WAL
into the latest-state mirror (`koldstore.messages__cl`) with short lag. Before
work that must observe every commit visible at the start of the call:

```sql
SELECT koldstore.wait_for_async_mirror();
```

`flush_table` runs this fence automatically.

## 5. Page the change feed

```sql
SELECT seq, op, pk, deleted, source
FROM koldstore.changes_since(
  table_name => 'app.messages'::regclass,
  since_seq  => 0,
  limit_rows => 100
);
```

Resume with an exclusive cursor (`seq > since_seq`). Or rewind to the newest N
rows (delivered oldest → newest) when `since_seq = 0`:

```sql
SELECT seq, op, pk, deleted, source
FROM koldstore.changes_since(
  table_name => 'app.messages'::regclass,
  since_seq  => 0,
  limit_rows => 1000,
  last_rows  => 50
);
```

Full semantics: [Change Feed](https://kalamdb.org/docs/pg-koldstore/change-feed).

## 6. Flush excess rows to cold storage

```sql
SELECT koldstore.flush_table(table_name => 'app.messages'::regclass);
```

With `hot_row_limit = 1000` and more than 1000 mirror keys, the oldest excess
rows (by mirror `seq`) move to Parquet and leave the hot heap. Inspect status:

```sql
SELECT jsonb_pretty(
  koldstore.describe_table(table_name => 'app.messages'::regclass)
);
```

## 7. Query one table

```sql
SELECT count(*) FROM app.messages;  -- hot + cold via KoldMergeScan
SELECT * FROM app.messages WHERE id = 1;
```

After flush, cold-capable plans show `KoldMergeScan`. Hot-only PK lookups that
cannot hit cold may keep a native Index/Seq/Bitmap Scan with no wrapper — that
path is intentional and performance-critical.

## What just happened

1. PostgreSQL still owns transactions, MVCC, permissions, and the hot heap
2. WAL capture wrote latest-state rows into `koldstore.messages__cl`
3. Flush published Parquet segments and pruned flushed hot/mirror rows
4. `SELECT` merges hot + cold; `changes_since` pages the same latest-state stream

Next: [Under the Hood](https://kalamdb.org/docs/pg-koldstore/concepts) for the architecture, or
[Manage & Flush](https://kalamdb.org/docs/pg-koldstore/manage-and-flush) for policy details.

================================================================================
# pg Koldstore
URL: https://kalamdb.org/docs/pg-koldstore
Source: content/pg-koldstore/index.mdx

> Keep hot rows in PostgreSQL, flush history to open Parquet, and query one table with KoldStore — install, manage, flush, merge scan, and changes_since.
# pg Koldstore

**KoldStore** is an open-source PostgreSQL extension for forever-growing app
tables. It keeps the active working set in the PostgreSQL heap, moves older rows
to compressed Parquet you control, and still lets you query **one table** with
normal SQL.

This is different from [pg_kalam](https://kalamdb.org/docs/pg-kalam): pg_kalam is a foreign-data
wrapper that talks to a KalamDB server. KoldStore runs **inside** your PostgreSQL
instance and tiers storage for local heap tables.

## What you get

| Tier | Where it lives | Optimized for |
|------|----------------|---------------|
| Hot | PostgreSQL heap + native indexes | Active data, low-latency reads, transactional writes |
| Cold | Open Parquet (filesystem, S3/MinIO, GCS, Azure) | History, cost, retention |

Placement is **policy-driven** (`hot_row_limit` or age via `move_after`), not
access-frequency measurement. Reads go through `KoldMergeScan` so the newest
visible row wins across both tiers. Catch-up clients use
`koldstore.changes_since` over an exclusive `seq` cursor.

## Good fit today

- Messages and chat history
- AI memory / conversation transcripts
- Audit logs and append-heavy events
- Notifications, activity feeds, IoT telemetry

## Not a good fit yet

- Payment ledgers that need global uniqueness across hot and cold
- Workloads that need cold point lookups as fast as a B-tree
- Schema evolution, compaction, and backup/restore that are still hardening

## Read this section in order

1. [Installation](https://kalamdb.org/docs/pg-koldstore/installation) — preload, Docker, from source
2. [Getting Started](https://kalamdb.org/docs/pg-koldstore/getting-started) — first managed table
3. [Under the Hood](https://kalamdb.org/docs/pg-koldstore/concepts) — WAL capture, mirror, merge scan
4. [Manage & Flush](https://kalamdb.org/docs/pg-koldstore/manage-and-flush) — policies and jobs
5. [Change Feed](https://kalamdb.org/docs/pg-koldstore/change-feed) — `changes_since` / `last_rows`
6. [SQL API Reference](https://kalamdb.org/docs/pg-koldstore/sql-api) — every shipped function and GUC
7. [Operations](https://kalamdb.org/docs/pg-koldstore/operations) — auto-flush, mirror health
8. [Benchmarks](https://kalamdb.org/docs/pg-koldstore/benchmarks) — storage and VACUUM wins
9. [Status & Limits](https://kalamdb.org/docs/pg-koldstore/limitations) — honest beta boundaries

## Source and packaging

- Repository: [github.com/kalamdb/koldstore](https://github.com/kalamdb/koldstore)
- License: Apache 2.0
- PostgreSQL: **15–18** (published Docker image targets PG 16)
- Product page: [/koldstore](https://kalamdb.org/koldstore)

================================================================================
# Installation
URL: https://kalamdb.org/docs/pg-koldstore/installation
Source: content/pg-koldstore/installation.mdx

> Install KoldStore with Docker or from source, set shared_preload_libraries, enable wal_level=logical, and verify preload_status.
# Installation

KoldStore must load at **postmaster start**. Without
`shared_preload_libraries = 'koldstore'`, planner hooks never register and
managed `SELECT`s can silently miss cold rows after flush.

## Fastest path: Docker

Published images preload the extension and create it on first init:

```bash
docker pull jamals86/pg-koldstore:latest
docker run --rm \
  -e POSTGRES_PASSWORD=postgres \
  -p 5432:5432 \
  jamals86/pg-koldstore:latest
```

Connect:

```bash
psql postgres://postgres:postgres@127.0.0.1:5432/koldstoredb
```

| Detail | Value |
|--------|-------|
| Default database | `koldstoredb` |
| PostgreSQL major | 16 |
| Tags | `latest`, `<version>-pg16` (for example `0.1.5-beta.0-pg16`) |
| Architectures | amd64, arm64 |

Confirm:

```sql
SHOW shared_preload_libraries;     -- must include koldstore
CREATE EXTENSION IF NOT EXISTS koldstore;
SELECT koldstore.preload_status(); -- loaded_via_shared_preload = true
```

`pg_cron` may be packaged in the image but is **not** preloaded. Enable it
yourself if you want cron-based flush scheduling.

## Shared preload (required on every install)

Install order:

1. Install extension package files
2. Set `shared_preload_libraries = 'koldstore'` (merge with any existing list)
3. **Restart** PostgreSQL (`reload` is not enough)
4. `CREATE EXTENSION koldstore`

Ubuntu / Debian example:

```bash
echo "shared_preload_libraries = 'koldstore'" | \
  sudo tee /etc/postgresql/16/main/conf.d/koldstore.conf
sudo systemctl restart postgresql@16-main
```

Rules:

- `session_preload_libraries` is **not** sufficient
- Removing preload after `manage_table` is **unsupported**
- Check with `SELECT koldstore.preload_status();`

## Logical WAL (required for managed tables)

Committed-WAL capture needs:

```conf
wal_level = logical
```

Restart after changing `wal_level`. Do **not** create the publication or logical
slot yourself: `CREATE EXTENSION` ensures the empty publication
`koldstore_async_mirror`, and the first `manage_table` creates the database’s
deterministic slot. Provisioning is idempotent.

## Recommended production GUC baseline

Prefer `ALTER SYSTEM` / `ALTER DATABASE` for settings the background worker
reads (session `SET` does not always reach the worker):

| Setting | Baseline |
|---------|----------|
| `shared_preload_libraries` | include `koldstore` |
| `wal_level` | `logical` |
| `koldstore.async_mirror_max_retained_bytes` | `1073741824` (1 GiB health alert) |
| `koldstore.flush_check_interval_seconds` | `30` |
| `koldstore.async_apply_poll_interval_ms` | `100` |

## From source (developers)

Build and install with pgrx against a local PostgreSQL (typical loop for PG 16):

```bash
cargo pgrx install -p pg_koldstore \
  --no-default-features \
  --features "pg16 s3" \
  --pg-config "$(cargo pgrx info pg-config 16)"
```

Then set preload, restart, and `CREATE EXTENSION`. See the repository
[quickstart](https://github.com/kalamdb/koldstore/blob/main/docs/quickstart.md)
and [Docker](https://github.com/kalamdb/koldstore/tree/main/docker) folder for
compose + MinIO layouts.

## Upgrades (beta)

There are **no** packaged `ALTER EXTENSION … UPDATE` edges yet while the product
is in development. Local installs reinstall or resync when the bootstrap SQL
changes. Treat in-place upgrade and `pg_upgrade` as future ops work — see
[Status & Limits](https://kalamdb.org/docs/pg-koldstore/limitations).

## Next step

[Getting Started](https://kalamdb.org/docs/pg-koldstore/getting-started) — register storage, manage
a table, flush, and read with `changes_since`.

================================================================================
# Status & Limits
URL: https://kalamdb.org/docs/pg-koldstore/limitations
Source: content/pg-koldstore/limitations.mdx

> Honest KoldStore beta boundaries: preload correctness, hot-only UNIQUE/FK, cold storage failure modes, and unfinished recovery work.
# Status & Limits

KoldStore is in **early development** and is **not production-ready**. Manage,
flush, auto-flush, hot/cold query, and `changes_since` work. Recovery,
backup/restore, compaction, schema evolution, PK changes, and export/import are
still hardening.

## Shared preload is mandatory

`shared_preload_libraries` must include `koldstore`. Without it, backends can
silently run ordinary heap scans and return **hot-only** rows after flush.

- Install → set preload → **restart** → `CREATE EXTENSION`
- `session_preload_libraries` is not enough
- Removing preload after `manage_table` is unsupported

Check: `SELECT koldstore.preload_status();`

## UNIQUE and foreign keys

PostgreSQL `UNIQUE` and FK constraints on managed tables are enforced on the
**hot heap only**. After flush, cold values live in Parquet without a global
hot+cold constraint layer.

| Constraint | Hot | Cold | Normal DML checks cold? |
|------------|-----|------|-------------------------|
| Primary key | Yes | Logical winner via merge | Cold presence via segment index + Parquet |
| Non-PK `UNIQUE` | Yes | No | No |
| Foreign keys | Yes | No | No |

When `hot_row_limit` is set, `manage_table` **rejects** tables that already have
non-PK `UNIQUE` constraints or foreign keys.

## Indexes and extensions

PostgreSQL indexes (including pgvector HNSW/IVFFlat and similar) cover **hot
rows only**. Flushed rows lose heap index entries. There is no automatic Parquet
index translation yet. The `vector` type is not in the v0.1 supported type
matrix.

## Cold storage availability

If cold storage is unavailable, managed queries that need cold segments
**fail**. KoldStore does not silently return hot-only results.
`koldstore.enable_merge_scan = off` also errors rather than allowing an
incorrect heap-only read.

## Capture limits

- WAL-only capture (no trigger mode)
- Primary-key updates rejected
- `TRUNCATE` on async managed tables rejected
- Bounded mirror lag unless you call `wait_for_async_mirror()`
- Scoped physical cold folders still incomplete vs registration templates

## Change feed limits

- Latest-state catch-up, not full event history
- Positive cursors older than retained cold floor → retention-gap **error**
- `last_rows` ignored when `since_seq > 0`
- `last_rows` must be `<= limit_rows`

## Unmanage, not disable

Use `koldstore.unmanage_table(...)`. Do not set `koldstore_enabled = false`.
Storage binding is immutable after manage.

## Not shipped

- Packaged extension upgrade edges (`ALTER EXTENSION … UPDATE`)
- Cold DML helpers and cold-only `UPDATE`/`DELETE` that mutate Parquet in place
- `backup_manifest` / `validate_cold_storage` / polished export-import
- Compaction and schema evolution tooling

## When to wait

Prefer waiting if you need global uniqueness across tiers, cold lookups as fast
as B-trees, or turnkey PITR that spans heap + object storage out of the box.

## When to try

Local or staging workloads that grow forever (messages, audit, AI memory) where
shrinking the hot heap and indexes is the main goal, and you can tolerate beta
ops boundaries.

================================================================================
# Manage & Flush
URL: https://kalamdb.org/docs/pg-koldstore/manage-and-flush
Source: content/pg-koldstore/manage-and-flush.mdx

> Enable KoldStore on a heap table, choose hot_row_limit or move_after, run flush_table, and inspect jobs with describe_table.
# Manage & Flush

Managing a table attaches KoldStore policy and WAL mirror capture. Flushing
moves eligible latest-state rows from the hot heap into Parquet and prunes them
from PostgreSQL.

## Enable management

Prefer named arguments. `hot_row_limit` is required in the `manage_table` call
— pass `NULL` for hot-only tables that should not flush.

```sql
SELECT koldstore.manage_table(
  table_name        => 'chat.messages',
  storage           => 's3_archive',
  hot_row_limit     => 10000,
  min_flush_rows    => 1000,
  max_rows_per_file => 1000,
  auto_flush        => true,
  migration_order_by => 'created_at'
);
```

| Parameter | Default | Meaning |
|-----------|---------|---------|
| `table_name` | required | Table to manage (`regclass`) |
| `storage` | required | Registered storage backend name |
| `hot_row_limit` | required (`NULL` allowed) | Max mirror rows to keep hot |
| `min_flush_rows` | `1000` | Minimum excess before a policy flush moves data |
| `max_rows_per_file` | `1000` | Max rows per Parquet segment (floor via `koldstore.min_max_rows_per_file`) |
| `table_type` | `'shared'` | `shared` or `user` (+ `scope_column`) |
| `compression` | `NULL` | Optional Parquet codec (`snappy`, `zstd`, `uncompressed`) |
| `target_file_size_mb` | `NULL` | Stored for future size-aware flush |
| `auto_flush` | `true` | Let the DB worker evaluate and run needed flushes |

### ALTER TABLE style

```sql
ALTER TABLE chat.messages SET (
  koldstore_enabled = true,
  koldstore_storage = 's3_archive',
  koldstore_hot_row_limit = 10000,
  koldstore_min_flush_rows = 1000,
  koldstore_max_rows_per_file = 1000
);
```

Age-based eviction (mutually exclusive with `hot_row_limit`):

```sql
ALTER TABLE chat.messages SET (
  koldstore_enabled = true,
  koldstore_storage = 's3_archive',
  koldstore_move_after = '90 days',
  koldstore_min_flush_rows = 1000,
  koldstore_max_rows_per_file = 10000
);
```

Age is measured from the latest captured mirror mutation encoded in `seq`.

## Flush

### Policy flush

```sql
SELECT koldstore.flush_table(table_name => 'chat.messages'::regclass);
```

Keeps the newest rows hot by mirror `seq` and flushes oldest eligible excess
first, honoring `hot_row_limit` / `min_flush_rows` / `max_rows_per_file`.

### Force flush

```sql
SELECT koldstore.flush_table(
  table_name => 'chat.messages'::regclass,
  force      => true
);
```

Flushes **all** pending mirror rows. Use carefully on large tables.

### Enqueue without running

```sql
SELECT koldstore.enqueue_flush_job(
  table_name => 'chat.messages'::regclass,
  force      => false
);
```

Returns `1` if a job was inserted, `0` if an active flush job already exists.
`auto_flush` does **not** gate manual flush / enqueue.

## Inspect progress

```sql
SELECT jsonb_pretty(
  koldstore.describe_table(table_name => 'chat.messages'::regclass)
);

SELECT koldstore.list_jobs(
  statuses   => '["running","pending"]'::jsonb,
  job_types  => '["flush"]'::jsonb,
  table_name => 'chat.messages'::regclass
);
```

Useful `describe_table` fields: `hot_rows`, `mirror_rows`, `cold_row_count`,
`cold_segment_count`, heap/index sizes, `manifest_state`, `pending_jobs`,
`last_error`.

## Cancel

```sql
SELECT koldstore.cancel_job(job_id => '…'::uuid);
SELECT koldstore.cancel_table_jobs(table_name => 'chat.messages'::regclass);
```

Pending jobs cancel immediately when unlocked. Running jobs stop cooperatively
at the next wave boundary. If cancel lands after cold publish already committed,
the job may finish `completed` with
`payload.cancel_requested_after_publish = true` (data is not unpublished).

## Auto-flush toggle

```sql
SELECT koldstore.set_table_auto_flush(
  table_name => 'chat.messages'::regclass,
  enabled    => false
);
```

Manual `flush_table` still works. See [Operations](https://kalamdb.org/docs/pg-koldstore/operations)
for the built-in worker tick and pg_cron fallback.

## Unmanage

```sql
SELECT koldstore.unmanage_table(
  table_name => 'chat.messages'::regclass,
  rehydrate  => true,
  drop_cold  => false
);
```

Use this instead of `koldstore_enabled = false`. Optional `rehydrate` /
`drop_cold` control whether cold rows return to the heap and whether cold
objects are deleted.

## Next

- [Change Feed](https://kalamdb.org/docs/pg-koldstore/change-feed)
- [SQL API Reference](https://kalamdb.org/docs/pg-koldstore/sql-api)

================================================================================
# Operations
URL: https://kalamdb.org/docs/pg-koldstore/operations
Source: content/pg-koldstore/operations.mdx

> Run KoldStore auto-flush, monitor async mirror health, schedule pg_cron fallbacks, and understand DROP TABLE cold cleanup.
# Operations

Day-2 concerns for a managed database: auto-flush, mirror health, optional
cron, and drop/cleanup behavior.

## Built-in auto-flush

The same database worker that applies WAL also evaluates `auto_flush` tables on
each `koldstore.flush_check_interval_seconds` tick (default `30`):

1. Apply available WAL
2. Evaluate tables with `auto_flush = true`
3. Run at most one needed `flush_table`

Failed auto-flush skips that table for about **60 seconds**. Opt out per table:

```sql
SELECT koldstore.set_table_auto_flush(
  table_name => 'app.messages'::regclass,
  enabled => false
);
```

Or pass `auto_flush => false` to `manage_table`. Manual flush still works.

Prefer `ALTER DATABASE` / `ALTER SYSTEM` for worker-facing GUCs — session `SET`
does not always reach the background worker.

## pg_cron fallback

Useful when you want wall-clock schedules or `auto_flush = false`:

```sql
SELECT cron.schedule(
  'koldstore-flush-messages',
  '*/5 * * * *',
  $$SELECT koldstore.flush_table(table_name => 'app.messages'::regclass)$$
);
```

A policy flush with nothing eligible returns safely (`rows_flushed = 0`).

## Mirror health

```sql
SELECT koldstore.async_mirror_status();
SELECT koldstore.async_mirror_slot_name();
```

Alert on:

- `healthy` / retention flags in `async_mirror_status()`
- Retained WAL bytes vs `koldstore.async_mirror_max_retained_bytes` (default 1 GiB)
- Stale `updated_at` / apply rates

Crossing the retained-bytes threshold marks retention unhealthy but **never
stops** the applier from draining WAL. Separately configure PostgreSQL disk
guards and `max_slot_wal_keep_size` (an invalidated slot may require rebuild).

Strong application fence:

```sql
SELECT koldstore.wait_for_async_mirror();
```

## Disable capture infrastructure

Only after every managed table is unmanaged:

```sql
SELECT koldstore.disable_async_mirror();
```

Idempotent. A later `manage_table` recreates compatible infrastructure.

## DROP TABLE

Dropping a managed table cancels jobs, waits for locks, deactivates catalogs,
deletes cold objects under the table prefix (when deletion is enabled for that
path), drops `__cl`, and records a `drop_table_cleanup` job.

## Backup (honest beta)

- Back up PostgreSQL **and** cold object prefixes together
- Cold storage is not WAL-protected
- `recover_segments(..., dry_run => true|false)` exists for orphan discovery
- Packaged `backup_manifest` / `validate_cold_storage` are **not** shipped yet
- `pg_dump` / `COPY TO` see the logical merged view; physical Parquet is outside plain dumps

See [Status & Limits](https://kalamdb.org/docs/pg-koldstore/limitations).

## Suggested monitoring checklist

1. `SHOW shared_preload_libraries` still includes `koldstore`
2. `koldstore.preload_status()` → `loaded_via_shared_preload = true`
3. `async_mirror_status()` healthy; retained bytes under threshold
4. `describe_table` for hot/cold growth and `last_error`
5. Flush job backlog (`list_jobs` pending/running)
6. Disk for PostgreSQL data directory **and** cold backend

## Next

- [Benchmarks](https://kalamdb.org/docs/pg-koldstore/benchmarks)
- [Status & Limits](https://kalamdb.org/docs/pg-koldstore/limitations)

================================================================================
# SQL API Reference
URL: https://kalamdb.org/docs/pg-koldstore/sql-api
Source: content/pg-koldstore/sql-api.mdx

> Complete KoldStore SQL surface: session helpers, GUCs, storage, manage/flush, async mirror, changes_since, jobs, and describe_table.
# SQL API Reference

Signatures match the installed extension SQL today. Prefer **named arguments**
so optional parameters stay readable. Positional calls remain valid.

```sql
-- Preferred
SELECT koldstore.register_storage(
  name => 'local-dev',
  storage_type => 'filesystem',
  base_path => '/tmp/koldstore-demo',
  credentials => '{}'::jsonb,
  config => '{}'::jsonb
);
```

## Session helpers

| Function | Returns | Meaning |
|----------|---------|---------|
| `snowflake_id()` | `bigint` | Monotonic Snowflake-like id |
| `koldstore_version()` | `text` | Extension version |
| `koldstore_user_id()` | `text` | Active `koldstore.user_id`, or `NULL` |
| `koldstore.preload_status()` | `jsonb` | Preload / merge-scan status |

## Public GUCs

| GUC | Default | Meaning |
|-----|---------|---------|
| `koldstore.user_id` | empty | Scope id for user-typed managed tables |
| `koldstore.cold_reads` | `auto` | `auto` / `on` / `off` (`off` errors when cold is required) |
| `koldstore.enable_merge_scan` | `on` | `off` → ERROR at exec (never silent hot-only) |
| `koldstore.max_open_parquet_readers` | `32` | Per-backend open reader cap (`1..=1024`) |
| `koldstore.max_merge_seen_keys` | `1000000` | Per-scan PK identity cap; `0` disables |
| `koldstore.log_level` | `info` | `error` / `warn` / `info` / `debug` / `trace` |
| `koldstore.min_max_rows_per_file` | `1000` | Floor for `max_rows_per_file` |
| `koldstore.flush_check_interval_seconds` | `30` | Auto-flush worker tick (`1..=86400`) |
| `koldstore.async_apply_poll_interval_ms` | `100` | Apply loop latch poll (`50..=5000`) |
| `koldstore.async_apply_max_rows_per_tick` | `0` | Max source changes per tick (`0` = unlimited) |
| `koldstore.async_apply_max_ms_per_tick` | `0` | Max ms per tick (`0` = unlimited) |
| `koldstore.flush_prelock_max_passes` | `3` | Flush phase-5.5 pre-lock apply passes |
| `koldstore.flush_prelock_max_ms` | `5000` | Flush phase-5.5 wall budget |
| `koldstore.async_mirror_max_retained_bytes` | `1 GiB` | Health threshold only — never blocks apply |

Internal GUCs (`internal_system_write`, `internal_flush_cleanup`) are reserved
for extension maintenance paths and cannot be set by application roles.

Configure durable settings with `ALTER SYSTEM` / `ALTER DATABASE` / `ALTER ROLE`
and follow PostgreSQL reload rules. Session `SET` does not always reach the
background worker.

## Function catalog

| Function | Returns |
|----------|---------|
| `koldstore.register_storage(...)` | `uuid` |
| `koldstore.alter_storage_credentials(...)` | `void` |
| `koldstore.alter_storage_location(...)` | `uuid` |
| `koldstore.manage_table(...)` | `uuid` (migration job id) |
| `koldstore.set_table_auto_flush(...)` | `boolean` |
| `koldstore.unmanage_table(...)` | `bigint` |
| `koldstore.wait_for_async_mirror()` | `bigint` |
| `koldstore.async_mirror_status()` | `jsonb` |
| `koldstore.async_mirror_slot_name()` | `text` |
| `koldstore.disable_async_mirror()` | `boolean` |
| `koldstore.enqueue_flush_job(...)` | `bigint` (`1` / `0`) |
| `koldstore.flush_table(...)` | `uuid` |
| `koldstore.list_jobs(...)` | `jsonb` array |
| `koldstore.cancel_job(...)` / `cancel_table_jobs(...)` | cancel status |
| `koldstore.describe_table(...)` | `jsonb` |
| `koldstore.recover_segments(...)` | `bigint` |
| `koldstore.changes_since(...)` | set of change rows |

## Storage

```sql
SELECT koldstore.register_storage(
  name         => 'local-dev',
  storage_type => 'filesystem',  -- filesystem | s3 | gcs | azure
  base_path    => '/tmp/koldstore-demo',
  credentials  => '{}'::jsonb,
  config       => '{}'::jsonb
  -- optional: regular_path_tmpl, scoped_path_tmpl
);

SELECT koldstore.alter_storage_credentials(
  name => 'local-dev',
  credentials => '{"access_key_id":"...","secret_access_key":"..."}'::jsonb
);

SELECT koldstore.alter_storage_location(
  name => 'local-dev',
  base_path => '/var/lib/koldstore',
  config => '{}'::jsonb
);
```

Default path templates: `{namespace}/{tableName}/` and
`{namespace}/{tableName}/{scopeId}/`.

## Manage / unmanage

See [Manage & Flush](https://kalamdb.org/docs/pg-koldstore/manage-and-flush) for parameters and
policy examples. Quick reference:

```sql
SELECT koldstore.manage_table(
  table_name => 'chat.messages',
  storage => 's3_archive',
  hot_row_limit => 10000
);

SELECT koldstore.set_table_auto_flush('chat.messages'::regclass, false);

SELECT koldstore.unmanage_table(
  table_name => 'chat.messages'::regclass,
  rehydrate => true,
  drop_cold => false
);
```

## Async mirror

```sql
SELECT koldstore.wait_for_async_mirror();  -- strong fence; 0 ≠ disabled
SELECT koldstore.async_mirror_status();
SELECT koldstore.async_mirror_slot_name();
-- After unmanaging every table:
SELECT koldstore.disable_async_mirror();
```

## Flush and jobs

```sql
SELECT koldstore.enqueue_flush_job(table_name => 'chat.messages'::regclass);
SELECT koldstore.flush_table(table_name => 'chat.messages'::regclass, force => false);
SELECT koldstore.list_jobs();
SELECT koldstore.describe_table(table_name => 'chat.messages'::regclass);
SELECT koldstore.recover_segments(table_name => 'chat.messages'::regclass, dry_run => true);
```

## Change feed

```sql
SELECT * FROM koldstore.changes_since(
  table_name => 'chat.messages'::regclass,
  since_seq => 0,
  limit_rows => 1000,
  last_rows => NULL
);
```

Full semantics: [Change Feed](https://kalamdb.org/docs/pg-koldstore/change-feed).

## Not shipped yet (do not call)

Documented as roadmap / ops intent only:

- Cold DML helpers (`hydrate_pk`, cold `update_row` / `delete_row`)
- `backup_manifest`, `validate_cold_storage`, packaged export/import
- Packaged `ALTER EXTENSION … UPDATE` edges

Standard SQL `UPDATE`/`DELETE` that only matches cold rows affects **zero**
hot rows in the current MVP — use unmanage/rehydrate paths when you need cold
rows back in the heap.

## Next

- [Operations](https://kalamdb.org/docs/pg-koldstore/operations)
- [Status & Limits](https://kalamdb.org/docs/pg-koldstore/limitations)

================================================================================
# Architecture
URL: https://kalamdb.org/docs/pg-kalam/architecture
Source: content/pg-kalam/architecture.mdx

> Understand how pg_kalam uses FDW callbacks, a ProcessUtility DDL hook, gRPC sessions, and Arrow batches to bridge PostgreSQL to KalamDB.
# PostgreSQL Extension Architecture

The current PostgreSQL extension is remote-only. PostgreSQL hosts the extension entrypoints, the FDW callbacks, and the local relation surface, while a running KalamDB server performs the actual execution over gRPC.

## Main Components

1. A PostgreSQL backend process: owns SQL parsing, planning, and the visible PostgreSQL-side relation surface.
2. The `pg_kalam` extension: registers `_PG_init`, the FDW callbacks, the DDL `ProcessUtility` hook, and the helper SQL functions.
3. A foreign server definition: stores remote connection settings such as `host`, `port`, timeout, TLS material, and optional auth metadata.
4. A per-backend remote state cache: holds a reusable tonic channel, Tokio runtime, and a logical remote session id.
5. KalamDB's shared RPC listener: exposes the `PgService` surface for scans, mutations, transactions, and direct SQL execution.

## Transport and Session Model

`pg_kalam` talks to KalamDB through the shared RPC listener, not through the HTTP SQL API.

Current server options:

- `host`
- `port`
- `timeout`
- `auth_header`
- `ca_cert`
- `client_cert`
- `client_key`

On first use in a PostgreSQL backend and for a specific remote-server config, the extension:

1. parses the foreign-server options,
2. creates a current-thread Tokio runtime,
3. opens a tonic channel,
4. computes a logical session id in the form `pg-<backend-pid>-<config-hash>`,
5. sends `OpenSession`, and
6. caches that state for reuse across later statements in the same backend process.

`on_proc_exit` then attempts to send `CloseSession` for every cached remote state owned by that PostgreSQL backend.

Two important consequences follow from the current code:

- The transport is reused per PostgreSQL backend and per remote config instead of reconnecting for every statement.
- PostgreSQL current schema or `search_path` is not currently pushed into the remote session automatically, so direct `kalam_exec(...)` statements should be schema-qualified.

## DDL Path

The extension registers a `ProcessUtility` hook from `_PG_init()`. That hook is what powers the current DDL surface.

The hook intercepts:

- `CREATE TABLE ... USING kalamdb`
- `CREATE FOREIGN TABLE`
- `ALTER FOREIGN TABLE`
- `DROP FOREIGN TABLE`
- top-level explicit transaction statements so the extension can coordinate remote transaction lifecycle

There are two DDL entry points:

1. `CREATE TABLE ... USING kalamdb`
	This is the shorthand authoring path. It is currently bound to the default foreign server name `kalam_server`.

2. `CREATE FOREIGN TABLE ... SERVER <name>`
	This is the explicit path for non-default server names and manual mirroring.

In both paths, the current relation identity is what matters:

- the PostgreSQL schema becomes the remote KalamDB namespace,
- the PostgreSQL relation name becomes the remote KalamDB table name.

The runtime scan and modify paths do not treat `OPTIONS (namespace, table)` as the canonical mapping source.

### What the DDL hook does

For `CREATE TABLE ... USING kalamdb`, the extension:

1. validates that `_seq`, `_userid`, and `_deleted` were not declared explicitly,
2. strips the PostgreSQL `USING kalamdb` access-method marker before forwarding SQL to KalamDB,
3. normalizes `SERIAL`, `BIGSERIAL`, `SMALLSERIAL`, and identity columns into explicit integer columns with `DEFAULT SNOWFLAKE_ID()`,
4. forwards the user-column DDL and `WITH (...)` table options to KalamDB,
5. creates a local foreign table, and
6. injects `_seq BIGINT` on all tables plus `_userid TEXT` on user tables.

The local object becomes a foreign table after interception, which is why follow-up DDL should use `ALTER FOREIGN TABLE` and `DROP FOREIGN TABLE`.

## Read Path

For `SELECT` statements, the FDW scan flow is:

1. PostgreSQL plans the extension-managed relation scan.
2. `pg_kalam` resolves the relation identity from the local relation metadata and parses the foreign-server options.
3. The extension reads the current `kalam.user_id` value, if present.
4. Any pending buffered writes for that table are flushed first so reads see local writes.
5. The FDW builds a `ScanRequest` with physical-column projection and simple equality filters that are safe to push down.
6. A remote scan request is sent to KalamDB over `PgService`.
7. The returned Arrow IPC batches are converted into PostgreSQL tuples.

Current pushdown behavior is intentionally partial:

- projection pushdown is used for physical columns,
- simple `column = constant` or `column = $param` filters can be pushed,
- PostgreSQL still reapplies all quals locally,
- `ORDER BY` and `LIMIT` are not pushed down today.

`_userid` is synthesized locally from the current session user id. `_seq` comes back from the remote Arrow schema.

## Write Path

For `INSERT`, `UPDATE`, and `DELETE` statements:

1. PostgreSQL routes the operation through FDW modify callbacks.
2. `pg_kalam` resolves the target table and tenant context.
3. The extension converts PostgreSQL datums into DataFusion `ScalarValue` payloads.
4. Single-row inserts are buffered for better batching behavior inside explicit transactions.
5. Multi-row inserts go through the batch insert path directly.
6. Updates and deletes flush pending inserts first, then send remote mutation requests.

Single-row inserts are buffered so PostgreSQL transactions can batch work more efficiently. Pending writes are flushed before scans and before transaction completion so read-your-writes behavior stays predictable inside the PostgreSQL session.

Current user-id behavior:

- `_userid` is only treated as an explicit tenant override for `INSERT`.
- `UPDATE` and `DELETE` use the session `kalam.user_id` scope.
- In a multi-row insert, all rows must use the same `_userid` value.

Current row-identity behavior:

- `UPDATE` and `DELETE` use the first non-system column as the row-identity heuristic.
- In practice, keep the primary key as the first user column in the table definition.

## Explicit Transaction Flow

When a PostgreSQL session opens an explicit transaction with `BEGIN` or `START TRANSACTION`, the extension lazily opens a remote KalamDB transaction on the first FDW operation in that block.

The extension uses PostgreSQL xact callbacks to coordinate remote `BeginTransaction`, `CommitTransaction`, and `RollbackTransaction` calls.

That transaction state is observable through two server views:

- `system.sessions` for the live pg bridge session and its current remote transaction metadata.
- `system.transactions` for the active explicit transaction itself alongside other active transaction origins.

This makes PostgreSQL transaction behavior visible without materializing extra persisted bookkeeping on the server.

Current transaction limits come from the backend coordinator:

- DDL is rejected inside explicit transactions.
- Stream-table writes are rejected inside explicit transactions.
- The documented surface is top-level `BEGIN`, `COMMIT`, and `ROLLBACK` blocks.

## Direct Statement Passthrough

The extension also exposes a direct SQL bridge:

```sql
SELECT kalam_exec('SELECT * FROM system.tables');
```

Use `kalam_exec(...)` when you want to send a statement directly to KalamDB without wrapping it in PostgreSQL relation syntax first.

Current behavior:

- it uses the default foreign server `kalam_server`,
- it reuses the same per-backend remote session cache as the FDW path,
- it returns a JSON array string for query results,
- it returns a JSON string status message for DDL and DML,
- it is revoked from `PUBLIC`.

## Tenant Context

`kalam.user_id` is a PostgreSQL session setting registered by the extension.

```sql
SET kalam.user_id = 'user-alice';
SELECT kalam_user_id();
```

The session value is attached to remote requests and is the main way user-scoped reads, updates, and deletes are isolated.

`kalam.user_id` is registered as a PostgreSQL `SUSET` GUC, so your PostgreSQL role needs permission to set `SUSET` parameters.

## Related

- [Getting Started](https://kalamdb.org/docs/pg-kalam/getting-started)
- [SQL Syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)
- [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions)
- [Status & Limits](https://kalamdb.org/docs/pg-kalam/limitations)

================================================================================
# Getting Started
URL: https://kalamdb.org/docs/pg-kalam/getting-started
Source: content/pg-kalam/getting-started.mdx

> Set up pg_kalam by preloading the extension, exposing KalamDB's RPC port, creating the default foreign server, and using the current authoring paths correctly.
# PostgreSQL Extension Getting Started

This page covers both the release-download path and the current usage path after the `pg_kalam` binaries are installed.

If you still need to build or package the extension itself, use the repository guide: [pg/README.md](https://github.com/kalamdb/KalamDB/tree/main/pg).

If you want PostgreSQL with the extension already installed, use the published Docker image: [jamals86/pg-kalam](https://hub.docker.com/r/jamals86/pg-kalam).

## Download a Release Package

KalamDB publishes a prebuilt PostgreSQL extension package in each GitHub release:

- Releases page: [github.com/kalamdb/KalamDB/releases](https://github.com/kalamdb/KalamDB/releases)
- Current asset pattern: 
- Check the matching `SHA256SUMS` asset before installing in production

The install example below automatically uses the latest published GitHub release tag.

Important constraints:

- The published release package currently targets Linux x86_64.
- The asset name includes the PostgreSQL feature it was built for, such as `pg16`.
- The binary must match the PostgreSQL major version of the server where you install it.
- If you need another platform or PostgreSQL major, build from source with `pgrx`.

Example install from a release asset:

## Run with Docker

For local or CI usage, you can run PostgreSQL with `pg_kalam` preinstalled directly from Docker Hub:

```bash
docker run --name pg-kalam \
  -e POSTGRES_USER=kalamdb \
  -e POSTGRES_PASSWORD=kalamdb123 \
  -e POSTGRES_DB=kalamdb \
  -p 5433:5432 \
  -d jamals86/pg-kalam:latest-pg16
```

Then connect with `psql` and install the extension in the database if your init SQL has not already done it:

```sql
CREATE EXTENSION IF NOT EXISTS pg_kalam;
```

## Prerequisites

- PostgreSQL with `pg_kalam` installed into that exact PostgreSQL instance.
- A running KalamDB server reachable from PostgreSQL.
- A stable KalamDB RPC endpoint host and port. The default repository examples use `2910`.
- An auth header if your KalamDB server requires one.

## 1. Preload and Create the Extension

`pg_kalam` installs SQL objects with `CREATE EXTENSION`, but the DDL mirroring hook is registered from `_PG_init()`. For the shorthand `CREATE TABLE ... USING kalamdb` path, add the extension to PostgreSQL's preload list and restart PostgreSQL first:

```conf
shared_preload_libraries = 'pg_kalam'
```

Then connect as a superuser and install the extension into the target database:

```sql
CREATE EXTENSION IF NOT EXISTS pg_kalam;
```

Verify that PostgreSQL sees the extension functions:

```sql
SELECT kalam_version(), kalam_compiled_mode();
```

`kalam_compiled_mode()` should currently return `remote`.

## 2. Give KalamDB a Stable RPC Address

Point PostgreSQL at KalamDB's shared gRPC/RPC listener, not at the HTTP API port.

For local source runs, configure a fixed RPC address instead of relying on a random ephemeral port:

```toml
[cluster]
rpc_addr = "127.0.0.1:2910"
api_addr = "127.0.0.1:2900"
```

If PostgreSQL runs in Docker on macOS while KalamDB runs on the host machine, use `host.docker.internal` instead of `127.0.0.1` in the foreign-server definition.

## 3. Register the Default KalamDB Foreign Server

Create the default foreign server that the shorthand authoring path expects:

```sql
CREATE SERVER IF NOT EXISTS kalam_server
  FOREIGN DATA WRAPPER pg_kalam
  OPTIONS (
    host '127.0.0.1',
    port '2910',
    auth_header 'Bearer <your-kalamdb-token>'
  );
```

Supported server options today:

- `host` and `port` are required.
- `auth_header`, `timeout`, `ca_cert`, `client_cert`, and `client_key` are optional.
- If you set `client_cert` or `client_key`, you must set both.

## 4. Choose the Right Authoring Path

There are two current SQL entry points:

| Goal | Use |
|---|---|
| Create a KalamDB table from PostgreSQL on the default server | `CREATE TABLE ... USING kalamdb` |
| Use a non-default foreign server or mirror a relation explicitly | `CREATE FOREIGN TABLE ... SERVER <name>` |

The shorthand `USING kalamdb` path is convenient, but it is hard-wired to the default server name `kalam_server`.

## 5. Create Your First Shared Table

The primary authoring syntax is `CREATE TABLE ... USING kalamdb`:

```sql
CREATE SCHEMA IF NOT EXISTS app;

CREATE TABLE app.shared_items (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  title TEXT NOT NULL,
  value INTEGER,
  created_at TIMESTAMP DEFAULT NOW()
) USING kalamdb WITH (
  type = 'shared',
  flush_policy = 'rows:1000,interval:60'
);
```

That statement is intercepted by the DDL hook, creates the remote namespace and table in KalamDB, then creates a local foreign table and injects the PostgreSQL-visible system columns that the extension manages locally.

Any KalamDB table shape that the server accepts can be authored this way from PostgreSQL. The same option names you would use in KalamDB are forwarded through the PostgreSQL `WITH (...)` clause.

Important rules for new tables:

- Do not declare `_seq`, `_userid`, or `_deleted` in the `CREATE TABLE ... USING kalamdb` column list. `pg_kalam` rejects that DDL because those columns are managed by the extension.
- `_seq BIGINT` is added to every PostgreSQL table surface that `pg_kalam` creates.
- `_userid TEXT` is added only to user tables.
- `_deleted` is reserved, but it is not auto-injected into the current PostgreSQL table surface.
- Keep the primary key as the first user column. The current `UPDATE` and `DELETE` FDW path uses the first non-system column as its row-identity heuristic.
- The local object is a foreign table after interception, so use `ALTER FOREIGN TABLE` and `DROP FOREIGN TABLE` for follow-up DDL.

## 6. Query and Mutate Through PostgreSQL

```sql
INSERT INTO app.shared_items (id, title, value)
VALUES (SNOWFLAKE_ID(), 'Alpha', 10), (SNOWFLAKE_ID(), 'Beta', 20);

SELECT id, title, value
FROM app.shared_items
ORDER BY id;

UPDATE app.shared_items
SET value = 25
WHERE id = 2;

DELETE FROM app.shared_items
WHERE id = 1;
```

Autocommit statements flush at statement end. Top-level explicit transactions are covered later in this section.

## 7. Use Session Identity for User Tables

User tables are tenant-scoped. Set `kalam.user_id` in each PostgreSQL session before querying or mutating them:

```sql
SET kalam.user_id = 'user-alice';

CREATE TABLE app.profiles (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  name TEXT,
  age INTEGER
) USING kalamdb WITH (
  type = 'user'
);

INSERT INTO app.profiles (id, name, age)
VALUES (SNOWFLAKE_ID(), 'Alice', 30);

INSERT INTO app.profiles (id, name, age, _userid)
VALUES (SNOWFLAKE_ID(), 'Bob', 41, 'user-bob');

SELECT id, name, age, _userid, _seq
FROM app.profiles
ORDER BY _seq;
```

The tenant scope comes from the PostgreSQL session value unless you explicitly provide `_userid` in an `INSERT`.

`kalam.user_id` is registered as a PostgreSQL `SUSET` parameter. In practice, the role running these statements must be allowed by PostgreSQL to change `SUSET` settings.

Current `_userid` rules:

- On user tables, omitting `_userid` uses the current `kalam.user_id` session value.
- On user tables, an explicit `_userid` in `INSERT` overrides the session value for that inserted row batch.
- In a multi-row `INSERT`, every row must use the same `_userid` value.
- `UPDATE` and `DELETE` still use the session's `kalam.user_id` scope. Changing row ownership with `UPDATE ... SET _userid = ...` is not part of the supported surface.

## 8. Create a Stream Table

Stream tables use the same authoring path, but they must supply `ttl_seconds`:

```sql
CREATE TABLE app.events (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  event_type TEXT NOT NULL,
  payload JSONB,
  created_at TIMESTAMP DEFAULT NOW()
) USING kalamdb WITH (
  type = 'stream',
  ttl_seconds = '3600'
);
```

Normal FDW access uses the stream table type, but explicit PostgreSQL transactions reject stream-table writes.

## 9. Reuse KalamDB Table Options with `WITH (...)`

The `WITH (...)` clause is the main place to express KalamDB table options from PostgreSQL:

```sql
CREATE TABLE app.orders (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  customer_id TEXT NOT NULL,
  status TEXT DEFAULT 'pending',
  created_at TIMESTAMP DEFAULT NOW()
) USING kalamdb WITH (
  type = 'shared',
  flush_policy = 'rows:1000,interval:60'
);
```

Use the same option names and values that KalamDB expects. The extension forwards them to the server when it constructs the real KalamDB `CREATE TABLE` statement.

## 10. Work with `FILE` Columns

`FILE` is the PostgreSQL extension's main KalamDB-specific type shortcut.

```sql
CREATE TABLE app.assets (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  attachment FILE
) USING kalamdb WITH (
  type = 'shared'
);

SELECT attachment->>'name', attachment->>'mime'
FROM app.assets;
```

Current behavior:

- The remote KalamDB column stays `FILE`.
- The local PostgreSQL column is rewritten to `JSONB`.
- PostgreSQL sees a serialized `FileRef` payload with fields such as `id`, `sub`, `name`, `size`, `mime`, and `sha256`.
- Binary upload itself usually happens through KalamLink or another KalamDB file-aware client. PostgreSQL sees the resulting `FileRef` JSON representation.

## 11. Send Direct SQL to KalamDB

When you want to run a KalamDB statement directly, use `kalam_exec(...)`:

```sql
SELECT kalam_exec('ALTER TABLE app.orders ADD COLUMN priority INTEGER DEFAULT 0');

SELECT kalam_exec('DROP TABLE app.orders');

SELECT kalam_exec('SELECT * FROM system.tables');
```

`kalam_exec(...)` forwards the statement to the KalamDB server as-is.

- For `SELECT` statements, it returns a JSON array string.
- For DDL and DML statements, it returns a JSON string status message.
- It currently uses the default foreign server `kalam_server`.
- It is revoked from `PUBLIC`, so use a superuser or grant `EXECUTE` explicitly.
- It does not currently synchronize PostgreSQL `search_path` or current schema to the remote session automatically, so schema-qualify direct SQL.

## 12. Use a Non-Default Server Name

If you need a different foreign server name, create it and use the explicit `CREATE FOREIGN TABLE` path:

```sql
CREATE SERVER kalam_eu
  FOREIGN DATA WRAPPER pg_kalam
  OPTIONS (
    host '10.0.0.25',
    port '2910',
    auth_header 'Bearer <token>'
  );

CREATE FOREIGN TABLE app.audit_log (
  id BIGINT,
  action TEXT,
  payload JSONB
) SERVER kalam_eu
OPTIONS (
  table_type 'shared'
);
```

Current relation-mapping rule: the local PostgreSQL schema and relation name are the remote namespace and table. `OPTIONS (namespace, table)` are not the runtime source of truth for scans and writes.

If you mirror a remote `FILE` column manually, declare the local PostgreSQL column as `JSONB`.

## 13. Maintain the Server Definition

Update server options with standard PostgreSQL syntax:

```sql
ALTER SERVER kalam_server OPTIONS (SET host 'kalamdb.internal', SET port '2910');
ALTER SERVER kalam_server OPTIONS (ADD auth_header 'Bearer <replacement-token>');
```

Remove the bridge when you are done:

```sql
DROP SERVER kalam_server CASCADE;
```

## Next Steps

- [Architecture](https://kalamdb.org/docs/pg-kalam/architecture)
- [SQL Syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)
- [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions)
- [Status & Limits](https://kalamdb.org/docs/pg-kalam/limitations)

================================================================================
# PostgreSQL Extension
URL: https://kalamdb.org/docs/pg-kalam
Source: content/pg-kalam/index.mdx

> Use the remote-only pg_kalam PostgreSQL extension to expose KalamDB tables in PostgreSQL and understand the current FDW, DDL, and type-conversion surface.
# PostgreSQL Extension

`pg_kalam` is a remote-only PostgreSQL foreign data wrapper extension. PostgreSQL hosts the FDW callbacks, DDL hook, and local relation surface, while KalamDB executes scans, writes, and mirrored DDL over gRPC on the shared RPC listener.

Prebuilt release assets are published on the [KalamDB GitHub Releases page](https://github.com/kalamdb/KalamDB/releases) for the current Linux x86_64 PostgreSQL extension target. Other platforms or PostgreSQL majors can still be built from source with `pgrx`.

KalamDB also publishes a ready-to-run Docker image with PostgreSQL and `pg_kalam` preinstalled at [jamals86/pg-kalam on Docker Hub](https://hub.docker.com/r/jamals86/pg-kalam).

This chapter documents the current supported remote-mode surface. The codebase no longer contains an embedded-mode PostgreSQL runtime.

Keep these rules in mind from the start:

- Point `CREATE SERVER ... OPTIONS (host, port)` at KalamDB's RPC listener, not the HTTP API port.
- Add `shared_preload_libraries = 'pg_kalam'` and restart PostgreSQL if you want `CREATE TABLE ... USING kalamdb` and the DDL mirroring hook to work.
- `CREATE TABLE ... USING kalamdb` and `kalam_exec(...)` currently assume a default foreign server named `kalam_server`.
- Use `CREATE FOREIGN TABLE ... SERVER <name>` when you need a non-default server or you want to mirror a relation explicitly.

## What You Can Do Today

- Install the extension into PostgreSQL 13 through 18, with `pg16` as the default build target.
- Create remote server definitions with host, port, optional auth header, timeout, and TLS material.
- Define KalamDB tables from PostgreSQL with `CREATE TABLE ... USING kalamdb`.
- Expose relations explicitly with `CREATE FOREIGN TABLE ... SERVER ...` when you need a non-default server name.
- Run `SELECT`, `INSERT`, `UPDATE`, and `DELETE` through FDW callbacks.
- Use top-level `BEGIN`, `COMMIT`, and `ROLLBACK` blocks for explicit transactions on shared and user tables.
- Observe bridge sessions and explicit transactions with `system.sessions` and `system.transactions`.
- Use `kalam_exec(...)` for direct remote SQL when you want to bypass the relation wrapper.
- Work with `FILE` columns as `JSONB` `FileRef` payloads on the PostgreSQL side.

## Current Capability Map

| Area | Status | Notes |
|---|---|---|
| Remote bridge | Available | `kalam_compiled_mode()` returns `remote` |
| `CREATE TABLE ... USING kalamdb` | Available with limits | Requires `shared_preload_libraries = 'pg_kalam'` and the default server `kalam_server` |
| `CREATE FOREIGN TABLE ... SERVER ...` | Available | Use this path for non-default server names and explicit mirroring |
| Shared and user table DML | Available | `SELECT`, `INSERT`, `UPDATE`, and `DELETE` run through FDW callbacks |
| Stream tables | Available with limits | Stream-table DDL works; explicit transactions reject stream-table writes |
| Explicit transactions | Available with limits | Top-level `BEGIN`, `COMMIT`, and `ROLLBACK`; no DDL inside explicit transactions |
| Pushdown | Partial | Projection plus simple equality filters; PostgreSQL still reapplies quals locally |
| Direct statement passthrough | Available with limits | `kalam_exec(...)` uses the default server and is revoked from `PUBLIC` |
| Helper SQL functions | Available | `kalam_version`, `kalam_compiled_mode`, `kalam_user_id`, `kalam_user_id_guc_name`, `snowflake_id`, `kalam_exec` |
| `IMPORT FOREIGN SCHEMA` | Not available | Remote mode currently returns an unsupported error |
| Data type conversions | Mixed | See [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions) for the exact current mapping and runtime caveats |

## Read This Section In Order

1. [Getting Started](https://kalamdb.org/docs/pg-kalam/getting-started)
2. [Architecture](https://kalamdb.org/docs/pg-kalam/architecture)
3. [SQL Syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)
4. [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions)
5. [Status & Limits](https://kalamdb.org/docs/pg-kalam/limitations)

## Recommended Workflow

Use the PostgreSQL extension when you want PostgreSQL tooling, SQL clients, or migrations to operate against KalamDB tables without switching your operators to the HTTP API or an SDK.

In most cases the flow is:

1. Download the matching `pg_kalam` release asset for your PostgreSQL major, or build it from source.
2. Install the extension into the exact PostgreSQL instance you will use and add it to `shared_preload_libraries`.
3. Give KalamDB a stable RPC address and point the default `kalam_server` foreign server at that endpoint.
4. Define tables with `CREATE TABLE ... USING kalamdb`, or use `CREATE FOREIGN TABLE ... SERVER ...` for non-default servers.
5. Set `kalam.user_id` in sessions that work with user-scoped tables.
6. Use top-level `BEGIN ... COMMIT` when you need explicit transaction batching on shared or user tables.
7. Use `kalam_exec(...)` only when you need direct remote SQL and can satisfy its privilege model.

## Related

- [SQL Reference](https://kalamdb.org/docs/server/sql-reference)
- [Data Types Reference](https://kalamdb.org/docs/server/architecture/datatypes)
- [Docker Deployment](https://kalamdb.org/docs/server/getting-started/docker)
- [API Reference](https://kalamdb.org/docs/server/api)
- [KalamDB PostgreSQL extension source guide](https://github.com/kalamdb/KalamDB/tree/main/pg)

================================================================================
# Status and Limits
URL: https://kalamdb.org/docs/pg-kalam/limitations
Source: content/pg-kalam/limitations.mdx

> Review the current remote-only scope, hard limits, and troubleshooting guidance for pg_kalam so the PostgreSQL extension is used within its actual code-backed surface.
# PostgreSQL Extension Status and Limits

The PostgreSQL extension is usable, but it is still an active development surface. Plan for incremental feature growth rather than assuming complete PostgreSQL FDW parity.

## Supported Today

- Remote-mode PostgreSQL bridge to a running KalamDB server
- Shared and user table reads and writes through FDW callbacks
- Stream-table authoring and normal FDW access outside explicit transactions
- Explicit `BEGIN`, `COMMIT`, and `ROLLBACK` transaction blocks for Kalam-backed tables
- `CREATE TABLE ... USING kalamdb` as the public PostgreSQL-side authoring syntax
- Auto-injected system columns: `_seq` on all tables and `_userid` on user tables
- Forwarding KalamDB table options from PostgreSQL `WITH (...)`
- Direct pass-through execution with `kalam_exec(...)`
- Session-scoped tenant routing with `kalam.user_id`
- Extension helper functions including `kalam_version()` and `SNOWFLAKE_ID()`

## Hard Limits and Current Constraints

| Area | Current behavior |
|---|---|
| Compile mode | `kalam_compiled_mode()` currently reports `remote`; this chapter documents that mode only |
| DDL hook | `CREATE TABLE ... USING kalamdb` requires `shared_preload_libraries = 'pg_kalam'` and a PostgreSQL restart |
| Default server binding | `CREATE TABLE ... USING kalamdb` and `kalam_exec(...)` currently assume the default foreign server name `kalam_server` |
| Schema import | `IMPORT FOREIGN SCHEMA` is not available in remote mode |
| Runtime mapping source | Relation schema and relation name are the runtime namespace and table; `OPTIONS (namespace, table)` are not the canonical scan/write mapping |
| Filter pushdown | Only simple equality filters are pushed down; PostgreSQL still reapplies quals locally |
| Order and limit pushdown | `ORDER BY` and `LIMIT` are not pushed down today |
| User scoping | User-table operations should set `kalam.user_id`; `_userid` is only an explicit `INSERT` override |
| `kalam.user_id` privileges | `kalam.user_id` is registered as a PostgreSQL `SUSET` parameter |
| Row identity for `UPDATE`/`DELETE` | The current FDW path uses the first non-system column as its row-identity heuristic |
| Explicit transactions | The documented surface is top-level `BEGIN`, `COMMIT`, and `ROLLBACK` blocks |
| DDL in transactions | DDL is rejected inside explicit transactions |
| Stream tables in transactions | Stream-table writes are rejected inside explicit transactions |
| Direct SQL helper | `kalam_exec(text)` is revoked from `PUBLIC` and does not automatically mirror PostgreSQL `search_path` or current schema |
| Type conversions | The logical mapping is broader than the extension's dedicated PostgreSQL runtime branches; see [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions) |
| Reserved columns | `_seq`, `_userid`, and `_deleted` are reserved names; only `_seq` and `_userid` are auto-injected into the current PostgreSQL relation surface |

Additional practical limits:

- `_seq` is read-only metadata projected from KalamDB. Values supplied in `INSERT` or `UPDATE` are not forwarded.
- Multi-row user-table `INSERT` statements must use one `_userid` value consistently.
- `UPDATE` and `DELETE` on user tables use the session `kalam.user_id` scope. Reassigning ownership through `_userid` updates is not supported.
- The server and table validators are permissive. Some bad option combinations fail on first use rather than at `CREATE SERVER` or `CREATE FOREIGN TABLE` time.

## Prototype Syntax vs Supported Syntax

This docs section focuses on the current remote-mode flow.

If you see older prototype scripts that mention experimental helpers such as schema opt-in functions, treat those as development artifacts unless they are covered in this chapter. The supported documentation surface today is:

- `CREATE EXTENSION`
- `CREATE SERVER` / `ALTER SERVER` / `DROP SERVER`
- `CREATE TABLE ... USING kalamdb`
- `BEGIN` / `COMMIT` / `ROLLBACK`
- `kalam_exec(...)`
- `SELECT`, `INSERT`, `UPDATE`, `DELETE`
- extension helper functions and `kalam.user_id`

## Troubleshooting

| Symptom | Likely cause | What to check |
|---|---|---|
| `extension "pg_kalam" is not available` | Extension files are not installed into the active PostgreSQL instance | Reinstall the extension into that PostgreSQL 16 installation |
| `could not access file "$libdir/pg_kalam"` | PostgreSQL cannot find the shared library | Verify the extension was installed into the correct PostgreSQL library directory |
| `CREATE TABLE ... USING kalamdb` does not mirror DDL | The preload hook is not active | Check `SHOW shared_preload_libraries` and confirm it includes `pg_kalam`, then restart PostgreSQL |
| Remote connection failure | The foreign-server host or port is wrong, the HTTP API port was used, or KalamDB is down | Verify the foreign server options and point them at the KalamDB RPC endpoint |
| Auth failures | `auth_header` is missing or stale | Update the foreign server definition with the right bearer token or shared secret |
| User-table reads or writes fail | `kalam.user_id` was not set, or the role cannot set it | Set `kalam.user_id` in the PostgreSQL session and verify the PostgreSQL role can change `SUSET` parameters |
| `permission denied for function kalam_exec` | `kalam_exec(text)` is revoked from `PUBLIC` | Use a superuser or `GRANT EXECUTE ON FUNCTION kalam_exec(text)` to the desired role |
| `UPDATE` or `DELETE` targets the wrong row | The table's primary key is not the first user column | Recreate the table with the primary key first or use a carefully controlled direct SQL path |
| `IMPORT FOREIGN SCHEMA` fails | Feature gap in current remote mode | Use `CREATE TABLE ... USING kalamdb` or `kalam_exec(...)` instead |

## Related

- [Getting Started](https://kalamdb.org/docs/pg-kalam/getting-started)
- [SQL Syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)
- [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions)
- [KalamDB PostgreSQL extension source guide](https://github.com/kalamdb/KalamDB/tree/main/pg)

================================================================================
# SQL Syntax
URL: https://kalamdb.org/docs/pg-kalam/sql-syntax
Source: content/pg-kalam/sql-syntax.mdx

> Reference the current pg_kalam SQL surface, including CREATE TABLE ... USING kalamdb, explicit foreign tables, system columns, direct SQL passthrough, and transaction constraints.
# PostgreSQL Extension SQL Syntax

This page documents the current SQL surface for the `pg_kalam` PostgreSQL extension.

The key implementation detail is that the extension exposes two different table-definition paths:

1. `CREATE TABLE ... USING kalamdb`
  The shorthand authoring path backed by the DDL hook. This currently assumes the default server name `kalam_server`.

2. `CREATE FOREIGN TABLE ... SERVER <name>`
  The explicit PostgreSQL FDW path. Use this when you need a non-default server name or want to mirror a relation directly.

## Extension Metadata

Install and verify the extension:

```sql
CREATE EXTENSION IF NOT EXISTS pg_kalam;

SELECT kalam_version(), kalam_compiled_mode();
```

Current helper functions:

```sql
SELECT kalam_version();
SELECT kalam_compiled_mode();
SELECT kalam_user_id();
SELECT kalam_user_id_guc_name();
SELECT SNOWFLAKE_ID();
```

`kalam_compiled_mode()` should currently return `remote`.

## Foreign Server Syntax

Create the PostgreSQL foreign server that points at KalamDB:

```sql
CREATE SERVER IF NOT EXISTS kalam_server
  FOREIGN DATA WRAPPER pg_kalam
  OPTIONS (
    host '127.0.0.1',
    port '2910',
    auth_header 'Bearer <token>'
  );
```

Update it with standard PostgreSQL FDW server syntax:

```sql
ALTER SERVER kalam_server OPTIONS (SET host 'kalamdb.internal', SET port '2910');
ALTER SERVER kalam_server OPTIONS (ADD auth_header 'Bearer <replacement-token>');
ALTER SERVER kalam_server OPTIONS (SET timeout '5000');
ALTER SERVER kalam_server OPTIONS (DROP auth_header);
```

Drop it when you no longer need the bridge:

```sql
DROP SERVER kalam_server CASCADE;
```

### Supported Server Options

| Option | Required | Purpose |
|---|---|---|
| `host` | Yes | KalamDB host name or IP |
| `port` | Yes | KalamDB gRPC port |
| `auth_header` | No | Authorization header sent to KalamDB |
| `timeout` | No | Request timeout in milliseconds |
| `ca_cert` | No | PEM CA bundle for TLS |
| `client_cert` | No | PEM client certificate for mTLS |
| `client_key` | No | PEM client private key for mTLS |

If you set either `client_cert` or `client_key`, you must set both.

Point `host` and `port` at KalamDB's shared RPC listener, not the HTTP API port.

## `CREATE TABLE ... USING kalamdb`

The public table-definition syntax is `CREATE TABLE ... USING kalamdb`:

```sql
CREATE TABLE app.orders (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  customer_id TEXT NOT NULL,
  status TEXT DEFAULT 'pending',
  created_at TIMESTAMP DEFAULT NOW()
) USING kalamdb WITH (
  type = 'shared',
  flush_policy = 'rows:1000,interval:60'
);
```

This does three things:

1. resolves the PostgreSQL schema and table name into the mirrored KalamDB namespace and table,
2. creates the target namespace and KalamDB table if needed,
3. creates the local foreign table surface and injects the PostgreSQL-visible system columns that `pg_kalam` manages locally.

### Shared table example

```sql
CREATE TABLE app.shared_items (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  title TEXT NOT NULL,
  value INTEGER,
  created_at TIMESTAMP DEFAULT NOW()
) USING kalamdb WITH (
  type = 'shared',
  flush_policy = 'rows:1000,interval:60'
);
```

### User table example

```sql
SET kalam.user_id = 'user-alice';

CREATE TABLE app.profiles (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  name TEXT,
  age INTEGER
) USING kalamdb WITH (
  type = 'user'
);
```

### Stream table example

```sql
CREATE TABLE app.events (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  event_type TEXT NOT NULL,
  payload TEXT,
  created_at TIMESTAMP DEFAULT NOW()
) USING kalamdb WITH (
  type = 'stream',
  ttl_seconds = '3600'
);
```

### Notes for `USING kalamdb`

- `USING kalamdb` selects the extension-managed PostgreSQL authoring path.
- `WITH (type = 'shared' | 'user' | 'stream')` selects the KalamDB table type.
- `PRIMARY KEY` and `DEFAULT SNOWFLAKE_ID()` are preserved in KalamDB.
- `SERIAL`, `BIGSERIAL`, `SMALLSERIAL`, and `GENERATED ... AS IDENTITY` are normalized into integer columns with `DEFAULT SNOWFLAKE_ID()` before the KalamDB DDL is sent.
- Additional `WITH (...)` options are forwarded to the mirrored KalamDB `CREATE TABLE` statement.
- This shorthand path currently depends on the DDL preload hook and the default server name `kalam_server`.
- The resulting PostgreSQL object is a foreign table, so follow-up DDL should use `ALTER FOREIGN TABLE` and `DROP FOREIGN TABLE`.

## `CREATE FOREIGN TABLE ... SERVER ...`

Use explicit foreign tables when you need a non-default server or want to mirror a remote relation directly:

```sql
CREATE SERVER kalam_eu
  FOREIGN DATA WRAPPER pg_kalam
  OPTIONS (
    host '10.0.0.25',
    port '2910',
    auth_header 'Bearer <token>'
  );

CREATE FOREIGN TABLE app.audit_log (
  id BIGINT,
  action TEXT,
  payload JSONB
) SERVER kalam_eu
OPTIONS (
  table_type 'shared'
);
```

Current mapping rules:

- The local PostgreSQL schema is the remote namespace.
- The local PostgreSQL relation name is the remote table name.
- `OPTIONS (namespace, table)` are not the runtime source of truth for the current scan and modify paths.
- If a remote column is `FILE`, declare the local PostgreSQL column as `JSONB`.

## JSON and JSONB

`pg_kalam` maps PostgreSQL `JSON` and `JSONB` authoring types to KalamDB `JSON`.
On the `CREATE TABLE ... USING kalamdb` path, the local PostgreSQL column keeps the type you declared, so use `JSONB` when you want PostgreSQL's richer local JSON operator surface.
`FILE` is the special case: it is always mirrored locally as `JSONB`.

```sql
CREATE TABLE app.events (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  payload JSONB NOT NULL
) USING kalamdb WITH (
  type = 'shared'
);

SELECT payload->>'kind' AS kind
FROM app.events
WHERE payload ? 'kind';
```

That syntax compatibility is the contract; PostgreSQL may still evaluate the JSONB expression locally on the mirrored value unless you explicitly route the statement through `kalam_exec(...)`.

When you do send SQL through `kalam_exec(...)`, KalamDB itself supports the same common PostgreSQL-style JSON operators:

```sql
SELECT kalam_exec($$
  SELECT
    payload->>'kind' AS kind,
    payload->'metadata' AS metadata
  FROM app.events
  WHERE payload ? 'kind'
$$);
```

KalamDB-side JSON helper functions are also available, including `json_get`, `json_as_text`, `json_contains`, `json_length`, `json_object_keys`, and typed extractors such as `json_get_int`, `json_get_float`, and `json_get_bool`.

Current scope:

- KalamDB directly plans `->`, `->>`, and `?` today.
- Do not assume full PostgreSQL `jsonb` operator parity for operators such as `#>`, `#>>`, or `@>` yet.

## `FILE`

`FILE` is the main PostgreSQL extension-specific datatype shortcut.

```sql
CREATE TABLE app.assets (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  attachment FILE
) USING kalamdb WITH (
  type = 'shared'
);

SELECT attachment->>'name', attachment->>'mime'
FROM app.assets;
```

Current `FILE` behavior:

- The remote KalamDB schema keeps the `FILE` logical type.
- The local PostgreSQL surface rewrites that column to `JSONB`.
- PostgreSQL reads and writes the serialized `FileRef` JSON representation.
- Binary upload and file lifecycle usually happen through KalamLink or another file-aware KalamDB client, not through a raw PostgreSQL `BYTEA` upload path.

## System Columns

`pg_kalam` manages system columns itself. Do not declare them explicitly in `CREATE TABLE ... USING kalamdb`.

| Column | Where it appears | Rules |
|---|---|---|
| `_seq` | All `pg_kalam` PostgreSQL tables | Auto-injected as `BIGINT`, populated from KalamDB read results, read-only from PostgreSQL, and not forwarded when supplied in `INSERT` or `UPDATE` |
| `_userid` | User tables only | Auto-injected as `TEXT`, exposed on reads, and usable as an explicit tenant override only for `INSERT` |

Current rules:

- `CREATE TABLE ... USING kalamdb (..., _seq BIGINT, ...)` is rejected. The same applies to `_userid` and `_deleted`.
- `_seq` is a read-only metadata column. Query it, sort by it, or return it to clients, but do not treat it as an application-managed field.
- For user tables, omitting `_userid` on `INSERT` uses the current `kalam.user_id` session value.
- For user tables, including `_userid` on `INSERT` overrides the session value for that inserted row batch.
- In a multi-row `INSERT`, all rows must use the same `_userid` value.
- `UPDATE` and `DELETE` continue to use the session's `kalam.user_id` scope. Reassigning ownership with `UPDATE ... SET _userid = ...` is not in the supported surface.
- `_deleted` is reserved but is not auto-injected into the current PostgreSQL relation surface.

Example:

```sql
SET kalam.user_id = 'user-alice';

INSERT INTO app.profiles (id, name, age)
VALUES (SNOWFLAKE_ID(), 'Alice', 30);

INSERT INTO app.profiles (id, name, age, _userid)
VALUES (SNOWFLAKE_ID(), 'Bob', 41, 'user-bob');

SELECT id, name, _userid, _seq
FROM app.profiles
ORDER BY _seq;
```

## Forwarded Table Options

Any supported KalamDB table option can be authored from PostgreSQL with the table `WITH (...)` clause.

That means PostgreSQL can act as the entry point for KalamDB-native table definitions as long as the resulting table options are valid for KalamDB once the extension forwards them.

### Common forwarded `WITH (...)` values

| Option | Purpose |
|---|---|
| `type` | Table type: `shared`, `user`, or `stream` |
| `flush_policy` | Hot-path flush behavior |
| `storage_id` | Target storage backend |
| `access_level` | Shared-table access policy |
| `ttl_seconds` | Stream retention TTL |
| `deleted_retention_hours` | Soft-delete retention window |

## Direct Statement Execution

To pass any statement directly to KalamDB, use `kalam_exec(...)`:

```sql
SELECT kalam_exec('ALTER TABLE app.orders ADD COLUMN priority INTEGER DEFAULT 0');

SELECT kalam_exec('DROP TABLE app.orders');

SELECT kalam_exec('SELECT * FROM system.tables');
```

`kalam_exec(...)` sends the SQL string to the KalamDB server as-is.

- For `SELECT` statements, it returns a JSON array string.
- For DDL and DML statements, it returns a JSON string message.
- It uses the default foreign server `kalam_server`.
- It is revoked from `PUBLIC`.
- It does not currently mirror PostgreSQL `search_path` or current schema automatically, so schema-qualify the statement.

If you were expecting a schema-qualified helper such as `kalam.exec(...)`, the currently exposed PostgreSQL function name is `kalam_exec(...)`.

## Query and Mutation Syntax

### Select

```sql
SELECT id, title, value
FROM app.shared_items
WHERE value >= 10
ORDER BY id;
```

### Insert

```sql
INSERT INTO app.shared_items (id, title, value)
VALUES (SNOWFLAKE_ID(), 'Alpha', 10), (SNOWFLAKE_ID(), 'Beta', 20);
```

`_seq` is visible for reads but is not a writable application column.

For PostgreSQL-style upsert and `RETURNING`, send the statement through `kalam_exec(...)` so
KalamDB executes it on the server:

```sql
SELECT kalam_exec($$
  INSERT INTO app.shared_items (id, title, value)
  VALUES (1, 'Beta', 20)
  ON CONFLICT (id) DO UPDATE SET title = EXCLUDED.title, value = EXCLUDED.value
  RETURNING id, title, value
$$);
```

See [/docs/server/sql-reference/dml](https://kalamdb.org/docs/server/sql-reference/dml) for upsert and `RETURNING`
rules and limits.

### Update

```sql
UPDATE app.shared_items
SET value = 25
WHERE id = 2;
```

### Delete

```sql
DELETE FROM app.shared_items
WHERE id = 1;
```

### User-scoped DML

```sql
SET kalam.user_id = 'user-alice';

INSERT INTO app.profiles (id, name, age)
VALUES ('p1', 'Alice', 30);

INSERT INTO app.profiles (id, name, age, _userid)
VALUES ('p2', 'Bob', 41, 'user-bob');

SELECT id, name, age, _userid, _seq
FROM app.profiles;

UPDATE app.profiles
SET age = 31
WHERE id = 'p1';

DELETE FROM app.profiles
WHERE id = 'p1';

RESET kalam.user_id;
```

In practice, `kalam.user_id` is a PostgreSQL `SUSET` setting, so the executing role must be allowed to change `SUSET` parameters.

## Explicit Transaction Syntax

The PostgreSQL extension supports explicit transaction control for Kalam-backed tables:

```sql
BEGIN;
START TRANSACTION;

COMMIT;
COMMIT WORK;

ROLLBACK;
ROLLBACK WORK;
```

### Commit example

```sql
BEGIN;

INSERT INTO app.shared_items (id, title, value)
VALUES (SNOWFLAKE_ID(), 'Alpha', 10);

UPDATE app.shared_items
SET value = 25
WHERE title = 'Alpha';

COMMIT;
```

### Read-your-writes and rollback example

```sql
BEGIN;

INSERT INTO app.shared_items (id, title, value)
VALUES (SNOWFLAKE_ID(), 'Draft', 99);

SELECT id, title, value
FROM app.shared_items
WHERE title = 'Draft';

ROLLBACK;
```

Within the same PostgreSQL transaction, reads see the staged writes from that transaction. On `COMMIT`, the staged writes are made durable in KalamDB. On `ROLLBACK`, they are discarded.

### Transaction observability

Use direct KalamDB SQL or `kalam_exec(...)` when you want to inspect the server-side session and transaction views from PostgreSQL.

Example via `kalam_exec(...)` for active pg bridge sessions:

```sql
SELECT kalam_exec(
  'SELECT session_id, transaction_id, transaction_state, transaction_has_writes
   FROM system.sessions
   WHERE transaction_id IS NOT NULL
   ORDER BY last_seen_at DESC, session_id'
);
```

Example via `kalam_exec(...)` for active explicit transactions across pg and native KalamDB SQL:

```sql
SELECT kalam_exec(
  'SELECT transaction_id, owner_id, origin, state, write_count
   FROM system.transactions
   ORDER BY origin, transaction_id'
);
```

### Current limits inside explicit transactions

- Shared and user table DML are supported.
- Stream-table writes are rejected inside explicit transactions.
- DDL inside explicit transactions is not supported.
- Savepoints and PostgreSQL subtransactions are not yet supported.

## Row Identity for `UPDATE` and `DELETE`

The current FDW modify path uses the first non-system column as its row-identity heuristic.

Practical guidance:

- Keep the primary key as the first user column in the table definition.
- If you are mirroring an existing remote table and expect heavy `UPDATE` or `DELETE` use, validate the behavior against your exact schema before depending on it in production.

## Not Yet Supported

`IMPORT FOREIGN SCHEMA` is not yet supported in the current remote mode:

```sql
IMPORT FOREIGN SCHEMA app FROM SERVER kalam_server INTO app;
```

For now, use `CREATE TABLE ... USING kalamdb` for PostgreSQL-side authoring or `kalam_exec(...)` for direct server execution.

## Related

- [Getting Started](https://kalamdb.org/docs/pg-kalam/getting-started)
- [Architecture](https://kalamdb.org/docs/pg-kalam/architecture)
- [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions)
- [Status & Limits](https://kalamdb.org/docs/pg-kalam/limitations)

================================================================================
# Data Type Conversions
URL: https://kalamdb.org/docs/pg-kalam/type-conversions
Source: content/pg-kalam/type-conversions.mdx

> See how pg_kalam maps PostgreSQL types to KalamDB logical types, Arrow payloads, and back into PostgreSQL, including FILE and current runtime caveats.
# PostgreSQL Extension Data Type Conversions

This page documents the current conversion behavior in `pg_kalam`.

There are three distinct layers to keep straight:

1. PostgreSQL-side DDL authoring and logical schema mapping.
2. PostgreSQL datum conversion on writes.
3. Arrow batch materialization on reads.

The important constraint is that the canonical KalamDB logical type system is broader than the extension's dedicated PostgreSQL runtime branches. The logical mapping exists for more types than the extension can currently materialize natively on the PostgreSQL side.

If this page and the code diverge, the code wins.

## 1. PostgreSQL Authoring to KalamDB Logical Types

These are the common PostgreSQL-facing type names that map into KalamDB's logical schema model today.

| PostgreSQL-side authoring type | KalamDB logical type | Arrow or storage representation | Notes |
|---|---|---|---|
| `BOOLEAN`, `BOOL` | `Boolean` | `Boolean` | Dedicated native read and write branches exist |
| `SMALLINT`, `INT2` | `SmallInt` | `Int16` | Dedicated native read and write branches exist |
| `INTEGER`, `INT`, `INT4`, `MEDIUMINT` | `Int` | `Int32` | Dedicated native read and write branches exist |
| `BIGINT`, `INT8` | `BigInt` | `Int64` | Dedicated native read and write branches exist |
| `REAL`, `FLOAT`, `FLOAT4` | `Float` | `Float32` | Dedicated native read and write branches exist |
| `DOUBLE PRECISION`, `DOUBLE`, `FLOAT8`, `FLOAT64` | `Double` | `Float64` | Dedicated native read and write branches exist |
| `TEXT`, `VARCHAR`, `CHAR`, `STRING`, `NVARCHAR` | `Text` | `Utf8` | `TEXT` and `VARCHAR` have dedicated write fast paths; `CHAR` currently uses the generic text-output fallback on writes |
| `BYTEA`, `BYTES`, `BINARY`, `VARBINARY`, `BLOB` | `Bytes` | `Binary` | Read path has a dedicated `BYTEA` materializer; write path currently falls back to PostgreSQL text output |
| `JSON`, `JSONB` | `Json` | `Utf8` containing JSON text | JSON and JSONB are first-class in the extension on both read and write |
| `DATE` | `Date` | `Date32` | Read path has a dedicated `DATE` materializer; write path currently falls back to PostgreSQL text output |
| `TIME` | `Time` | `Time64(Microsecond)` | Logical mapping exists, but read and write still rely on generic conversion fallback |
| `TIMESTAMP` | `Timestamp` | `Timestamp(Microsecond, None)` | Read path has a dedicated timestamp materializer; write path currently falls back to PostgreSQL text output |
| `NUMERIC(p,s)`, `DECIMAL(p,s)` | `Decimal { precision, scale }` | `Decimal128(p, s)` | Logical mapping exists; PostgreSQL numeric materialization is not yet a dedicated branch |
| `UUID` | `Uuid` | `FixedSizeBinary(16)` | Logical mapping exists; PostgreSQL UUID materialization is not yet a dedicated branch |
| `FILE` | `File` | `Utf8` containing serialized `FileRef` JSON | The remote KalamDB column is `FILE`; the local PostgreSQL column is rewritten to `JSONB` on the `USING kalamdb` path |

Two special cases need separate callouts:

- `KalamDataType::DateTime` is the timezone-aware logical type and stores as `Timestamp(Microsecond, Some("UTC"))`. Its reverse PostgreSQL type name is `TIMESTAMPTZ`, but PostgreSQL-side timezone-aware authoring spellings should be validated against your exact build.
- `KalamDataType::Embedding(n)` is the logical embedding type. Its reverse PostgreSQL type name is `VECTOR(n)`, but the KalamDB logical DDL spelling is still `EMBEDDING(n)`, so PostgreSQL-side authoring should be validated carefully instead of assuming full shorthand parity.

## 2. KalamDB Logical Types Back to PostgreSQL

When the extension needs to generate or mirror PostgreSQL column type names from a KalamDB logical schema, it uses the following canonical mapping.

| KalamDB logical type | Generated or mirrored PostgreSQL type name |
|---|---|
| `Boolean` | `BOOLEAN` |
| `SmallInt` | `SMALLINT` |
| `Int` | `INTEGER` |
| `BigInt` | `BIGINT` |
| `Float` | `REAL` |
| `Double` | `DOUBLE PRECISION` |
| `Text` | `TEXT` |
| `Bytes` | `BYTEA` |
| `Date` | `DATE` |
| `Time` | `TIME` |
| `Timestamp` | `TIMESTAMP` |
| `DateTime` | `TIMESTAMPTZ` |
| `Uuid` | `UUID` |
| `Json` | `JSONB` |
| `File` | `JSONB` |
| `Decimal { precision, scale }` | `NUMERIC(p, s)` |
| `Embedding(n)` | `VECTOR(n)` |

Current local-surface rules modify that canonical reverse mapping in a few places:

- On `CREATE TABLE ... USING kalamdb`, most local PostgreSQL columns keep the type you declared.
- `JSON` stays `JSON` locally if you declared `JSON`.
- `JSONB` stays `JSONB` locally if you declared `JSONB`.
- `FILE` is the special case: it is always rewritten locally to `JSONB`.

If you mirror a remote `FILE` column manually with `CREATE FOREIGN TABLE`, declare the local PostgreSQL column as `JSONB`.

## 3. DDL Normalizations Before Remote Create

The DDL hook performs a small set of PostgreSQL-specific normalizations before sending the user-column definition to KalamDB.

| PostgreSQL authoring form | Remote KalamDB DDL result |
|---|---|
| `SERIAL` | Integer column plus `DEFAULT SNOWFLAKE_ID()` |
| `BIGSERIAL` | Bigint column plus `DEFAULT SNOWFLAKE_ID()` |
| `SMALLSERIAL` | Smallint column plus `DEFAULT SNOWFLAKE_ID()` |
| `GENERATED ... AS IDENTITY` | Explicit integer type plus `DEFAULT SNOWFLAKE_ID()` |

This normalization happens before the remote `CREATE TABLE` is sent.

## 4. Runtime Read and Write Behavior Today

The table above describes the logical schema contract. The next table describes the actual extension implementation status at runtime.

| Type family | PostgreSQL -> KalamDB write path | KalamDB -> PostgreSQL read path | Current status |
|---|---|---|---|
| `BOOLEAN`, `SMALLINT`, `INTEGER`, `BIGINT`, `REAL`, `DOUBLE PRECISION` | Dedicated native `Datum -> ScalarValue` branches | Dedicated native Arrow array branches | Best-covered native round-trip surface |
| `TEXT`, `VARCHAR` | Dedicated text fast path | Dedicated `Utf8` and `LargeUtf8` branches | Best-covered string surface |
| `CHAR`, `CHARACTER` | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | Reads through the normal `Utf8` text path once the backend returns text | Logical mapping exists, but writes are not a dedicated branch |
| `JSON`, `JSONB` | Dedicated JSON and JSONB normalization to JSON text | Dedicated PostgreSQL type-input path for JSON and JSONB | Best-covered document surface |
| `FILE` | PostgreSQL sees the `FileRef` JSON representation, usually as `JSONB`; binary upload itself typically happens through KalamLink or another file-aware client | Dedicated JSONB materialization from UTF-8 `FileRef` JSON | Stable local representation, special-case local `JSONB` surface |
| `BYTEA` | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | Dedicated Arrow `Binary -> bytea` branch | Read path is native; write path relies on generic coercion |
| `DATE` | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | Dedicated Arrow `Date32 -> DATE` branch | Read path is native; write path relies on generic coercion |
| `TIMESTAMP` and generated `TIMESTAMPTZ` surface | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | Dedicated Arrow timestamp branches reuse PostgreSQL's internal timestamp representation | Read path is native; write path relies on generic coercion |
| `TIME` | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | Generic fallback string rendering | Not yet a dedicated PostgreSQL round-trip branch |
| `UUID` | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | No dedicated UUID materialization branch in `arrow_to_pg` | Logical mapping exists; validate against your build before treating UUID as a stable PG round-trip surface |
| `NUMERIC(p,s)` | Generic PostgreSQL type-output fallback to `ScalarValue::Utf8` | No dedicated numeric materialization branch in `arrow_to_pg` | Logical mapping exists; native PostgreSQL numeric round-trip is not implemented yet |
| `VECTOR(n)` or `Embedding(n)` | PostgreSQL-side authoring depends on local type availability and current DDL spelling | No dedicated vector materialization branch in `arrow_to_pg` | Reverse mapping exists, but PostgreSQL vector round-trip is not yet a native branch |

## 5. Under the Hood

### Write path

For `INSERT`, `UPDATE`, and `DELETE`, the extension does the following:

1. PostgreSQL exposes a `TupleTableSlot` to the FDW modify callbacks.
2. `slot_to_row()` walks the local columns, skipping `_seq` and `_deleted`.
3. `_userid` is captured separately and only used as an explicit tenant override for `INSERT`.
4. `pg_to_kalam::datum_to_scalar()` converts each remaining datum into a DataFusion `ScalarValue`.
5. Supported primitive OIDs use dedicated branches.
6. Everything else currently falls back to PostgreSQL's type output function and becomes `ScalarValue::Utf8`.
7. The client serializes mutations as JSON row payloads in the gRPC request.

That is why the write path is broader than the dedicated fast-path branches but still depends on server-side coercion for several PostgreSQL type families.

### Read path

For `SELECT`, the extension does the following:

1. Builds a `ScanRequest` with relation identity, table type, tenant context, and simple equality filters.
2. Sends the request over gRPC.
3. Receives Arrow IPC batches from KalamDB.
4. `arrow_to_pg::arrow_value_to_datum()` materializes supported Arrow types directly into PostgreSQL datums.
5. JSON and JSONB use PostgreSQL type-input functions so PostgreSQL sees real document datums, not plain text.
6. Unsupported Arrow types fall back to `ScalarValue::to_string()`.

That final fallback is why logical schema support is currently broader than the stable PostgreSQL materialization surface.

## 6. What Is Safest Today

If you need the most predictable PostgreSQL behavior today, prefer these surfaces first:

- `BOOLEAN`, `SMALLINT`, `INTEGER`, `BIGINT`, `REAL`, `DOUBLE PRECISION`
- `TEXT` and `VARCHAR`
- `JSON`, `JSONB`, and `FILE` as local `JSONB`
- `DATE`, `TIMESTAMP`, and the timestamp-like generated `TIMESTAMPTZ` read path

Use extra care and validate against your target build for:

- `TIME`
- `UUID`
- `NUMERIC(p,s)`
- `VECTOR(n)` or other embedding-heavy PostgreSQL surfaces

The strongest current automated coverage in the extension repo is around JSON, JSONB, and `FILE` behavior. The rest of the mapping surface is primarily code-backed rather than broad end-to-end type-matrix coverage.

## Related

- [SQL Syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)
- [Status & Limits](https://kalamdb.org/docs/pg-kalam/limitations)
- [KalamDB Data Types Reference](https://kalamdb.org/docs/server/architecture/datatypes)

================================================================================
# Create a Table in Admin UI
URL: https://kalamdb.org/docs/server/admin-ui/create-table
Source: content/server/admin-ui/create-table.mdx

> Create a KalamDB table from the Admin UI table editor, define fields, choose storage options, and verify the schema in SQL Studio.
# Create a Table in the KalamDB Admin UI

Create a table in the KalamDB Admin UI with the visual table editor, where you define the namespace, table type, table name, storage options, flush policy, compression, and fields before reviewing the generated schema change.

## Open The Table Editor

Open SQL Studio, choose the target namespace, then select `New table` from the explorer toolbar.

The editor starts with:

- namespace context
- table type selector
- table name
- storage ID and optional user-storage toggle
- flush policy and compression controls
- column rows with primary key, type, nullable, unique, and default controls
- `Discard` and `Review & Create` actions

## Define Fields

For a starter `agent_events` table:

1. Set `Table name` to `agent_events`.
2. Keep the default `id` primary key unless your schema needs a different key.
3. Use `Add column` for each additional field.
4. Set field names such as `agent_id` and `event_type`.
5. Choose each field type from the type selector.
6. Adjust nullable, unique, and default values before review.

Use `Review & Create` to inspect the schema change, then confirm creation. The action is disabled until required fields are valid.

## Verify The Table

After creation, the table appears in the namespace explorer. Open SQL Studio and run:

```sql
SELECT id, agent_id, event_type
FROM default.agent_events
LIMIT 20;
```

If you create the table in another namespace, replace `default` with that namespace.

## Related SQL Reference

- [Tables](https://kalamdb.org/docs/server/sql-reference/tables)
- [Data Manipulation](https://kalamdb.org/docs/server/sql-reference/dml)
- [Query Data](https://kalamdb.org/docs/server/sql-reference/query-data)

================================================================================
# Admin UI Dashboard
URL: https://kalamdb.org/docs/server/admin-ui/dashboard
Source: content/server/admin-ui/dashboard.mdx

> Read the KalamDB Admin UI dashboard to inspect server health, activity summaries, active connections, and operational pressure.
# KalamDB Admin UI Dashboard

The KalamDB Admin UI dashboard is the first operational page for checking server health, recent activity, workload pressure, storage usage, and cluster status.

## What The Dashboard Shows

Use the dashboard before deeper inspection. It includes:

- session greeting and quick access to the statistics SQL view
- time-range selector and manual refresh
- summary cards for uptime, tables, namespaces, connections, subscriptions, pub/sub, total queries, query rate, and average latency
- charts for SQL query activity, connections and subscriptions, pub/sub consumers, resource usage, and manifest/parquet rates
- slow query table with timestamp, SQL statement, duration, priority, and pagination
- storage usage card with backend selector, used/free capacity, path, usage percentage, and health
- cluster node health with node count, leader status, active/joining state, memory, CPU, uptime, and replication lag

## When To Use It

Start here when you need to decide where to drill in next: SQL Studio for query behavior, Live Queries for subscription pressure, Streaming for topics and consumers, Logs & Analytics for slow queries and jobs, or Settings for runtime configuration.

================================================================================
# Admin UI
URL: https://kalamdb.org/docs/server/admin-ui
Source: content/server/admin-ui/index.mdx

> Use the KalamDB Admin UI to inspect server state, run SQL, manage users, monitor live queries, and operate realtime workloads.
# KalamDB Admin UI

The KalamDB Admin UI is the browser workspace for authentication-aware server operations, SQL Studio work, schema editing, user management, realtime monitoring, logging, and runtime configuration review.

## Admin UI Navigation

Open the Admin UI at:

```text
http://localhost:2900/ui
```

The sidebar groups the primary operator pages:

- [Login](https://kalamdb.org/docs/server/admin-ui/login) for local and OIDC browser authentication
- [Dashboard](https://kalamdb.org/docs/server/admin-ui/dashboard) for health cards, metric charts, slow queries, storage usage, and cluster node status
- [SQL Studio](https://kalamdb.org/docs/server/admin-ui/sql-studio) for namespace browsing, table inspection, saved query tabs, SQL execution, and visual schema editing
- [Create a Table](https://kalamdb.org/docs/server/admin-ui/create-table) for the visual table editor workflow
- [Streaming](https://kalamdb.org/docs/server/admin-ui/streaming) for topic inventory, topic inspection, consumer offsets, and topic-to-SQL navigation
- [Users](https://kalamdb.org/docs/server/admin-ui/users) for users, roles, pending invites, pagination, edits, and account lifecycle actions
- [Live Queries](https://kalamdb.org/docs/server/admin-ui/live-queries) for active WebSocket subscriptions, filters, change counts, and kill actions
- [Logging](https://kalamdb.org/docs/server/admin-ui/logging) for audit events, server logs, background jobs, and slow query analytics
- [Settings](https://kalamdb.org/docs/server/admin-ui/settings) for current-user context and grouped server, storage, cluster, backup, and security settings

## Shared Layout

Every authenticated page keeps the same shell:

- top bar with connection action, global search, WebSocket status, notifications, and current-user menu
- icon sidebar for Dashboard, SQL Studio, Streaming, Users, Live Queries, Logs, and Settings
- authenticated session context from the same login and token flow used by the HTTP and WebSocket APIs

## Authentication Context

The Admin UI discovers login modes from `/v1/api/auth/login-options` and uses the same local and OIDC authentication surface described in [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication).

For OIDC deployments, register the browser redirect URI documented in [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc).

================================================================================
# Admin UI Jobs
URL: https://kalamdb.org/docs/server/admin-ui/jobs
Source: content/server/admin-ui/jobs.mdx

> Inspect KalamDB background jobs in Logs & Analytics, including status, job type, namespace, duration, node, and job details.
# KalamDB Admin UI Jobs

Jobs are shown inside the Admin UI [Logs & Analytics](https://kalamdb.org/docs/server/admin-ui/logging) workspace. The Jobs section lists background work and maintenance activity that can explain server load, storage behavior, and delayed operations.

## What Jobs Shows

The Jobs section includes:

- filters panel
- page-size selector and pagination
- refresh action
- status column
- job type column such as cleanup or stream eviction
- namespace/table context when the job applies to a data object
- created timestamp, duration, and node
- detail action for opening a specific job record

Pair this section with Server logs when a job needs event-level detail.

================================================================================
# Admin UI Live Queries
URL: https://kalamdb.org/docs/server/admin-ui/live-queries
Source: content/server/admin-ui/live-queries.mdx

> Inspect active KalamDB live queries in the Admin UI, including subscriptions, realtime workload behavior, and query activity.
# KalamDB Admin UI Live Queries

The Live Queries page shows active realtime subscriptions and WebSocket query activity that affect server load and user-facing updates.

## What Live Queries Shows

The page includes:

- active query count and total query count
- auto-refresh state
- filters for user ID, namespace, table, and status
- table columns for subscription ID, user, namespace, table, status, duration, change count, SQL query, serving node, and actions
- `Kill` action for terminating a problematic subscription

## When To Use It

Use this page to identify unexpectedly broad realtime queries, confirm client cleanup during disconnects, compare live query behavior with stream activity and logs, or stop a runaway subscription.

For the architecture model, see [Live Query Architecture](https://kalamdb.org/docs/server/architecture/live-query).

================================================================================
# Admin UI Logging
URL: https://kalamdb.org/docs/server/admin-ui/logging
Source: content/server/admin-ui/logging.mdx

> Use the KalamDB Admin UI Logging page to inspect server, job, authentication, and operational events during troubleshooting.
# KalamDB Admin UI Logging

The Logging page is the Logs & Analytics workspace for event-level troubleshooting across audit events, server logs, background jobs, and slow SQL.

## What Logs & Analytics Shows

The page is split into four sections:

- `Audit`: security and administrative activity
- `Server`: query-backed view of `system.server_logs` with log frequency graph, pause/refresh controls, text filter, case-sensitivity toggle, level filter, row limit selector, timestamp, level, target, and message
- `Jobs`: background job table with status, job type, namespace/table, created time, duration, node, filters, pagination, refresh, and detail action
- `Slow Queries`: slow query analytics for statements that exceed the configured threshold

## When To Use It

Use logs to confirm login/setup behavior, trace failed SQL or maintenance operations, correlate jobs with streaming and live-query events, and collect evidence before changing production configuration.

For log configuration, see [Logging Configuration](https://kalamdb.org/docs/server/configurations/logging).

================================================================================
# Admin UI Login
URL: https://kalamdb.org/docs/server/admin-ui/login
Source: content/server/admin-ui/login.mdx

> Sign in to the KalamDB Admin UI with local credentials or OpenID Connect, and verify the login methods exposed by the server.
# KalamDB Admin UI Login

The KalamDB Admin UI login page uses the server authentication configuration to show local password login, OpenID Connect login, setup routing, or a combination of those options.

## What Login Shows

The browser reads available methods from:

```bash
curl http://127.0.0.1:2900/v1/api/auth/login-options
```

Depending on server configuration, the page can show:

- local username/password fields
- provider button such as `Continue with Dex`
- setup link for an unconfigured node
- explanatory note for local cluster scripts and root credentials
- invalid-credentials feedback after failed local login

If local auth is enabled, sign in with the DBA account created during bootstrap. If OIDC is enabled, use the configured provider button and complete the provider flow.

## Related Authentication Pages

- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)

================================================================================
# Admin UI Settings
URL: https://kalamdb.org/docs/server/admin-ui/settings
Source: content/server/admin-ui/settings.mdx

> Review KalamDB Admin UI settings to inspect exposed runtime configuration, authentication state, and operational defaults.
# KalamDB Admin UI Settings

The Settings page shows current-user context, runtime configuration, and operational defaults exposed by the server to authenticated administrators.

## What Settings Shows

The Settings page includes a left-side settings section menu:

- `All Settings`
- `General`
- `Cluster`
- `Storages`
- `Backup`
- `Security`

The main view starts with current admin session details: username, role, email, and user ID. It then groups configuration values such as server host/port/workers, public origin, storage paths and RocksDB profiles, cluster ID, DataFusion limits, query limits, logging format/path, flush defaults, manifest cache, stream TTLs, topic retention, WebSocket timeouts, and rate limits.

## When To Use It

Use this page to confirm the server origin and Admin UI paths match deployment expectations, verify authentication and security posture, check storage and backup configuration, and review exposed runtime defaults before production rollout.

For the full config matrix, see [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced).

================================================================================
# Admin UI SQL Studio
URL: https://kalamdb.org/docs/server/admin-ui/sql-studio
Source: content/server/admin-ui/sql-studio.mdx

> Use KalamDB SQL Studio to browse namespaces and tables, run SQL statements, inspect results, and validate schema changes.
# KalamDB Admin UI SQL Studio

SQL Studio is the Admin UI workspace for exploring namespaces, writing SQL, checking table metadata, editing schemas, and validating query results.

## What SQL Studio Contains

SQL Studio has two main work areas:

- `Explorer`: namespace picker, namespace create/drop actions, table filter, favorites, table list, table expansion, and table drop actions
- `Editor`: table editor and query editor modes, saved/draft query tabs, Monaco SQL editor, results/log tabs, live toggle, save/run controls, execute options, and query actions

When a table is selected, the right pane can show metadata and editable schema fields. For new tables, the same area becomes the visual table editor described in [Create a Table](https://kalamdb.org/docs/server/admin-ui/create-table).

## Run SQL

Start with a system query:

```sql
SELECT * FROM system.namespaces LIMIT 100;
```

Then inspect the result grid and execution details.

## Table And Query Workflows

Use this page to:

- inspect namespaces and table columns without leaving the browser
- save and reopen common query tabs
- run SQL and review results or execution logs
- turn live mode on for realtime result updates where supported
- switch from table inspection into the visual table editor

Use [Create a Table](https://kalamdb.org/docs/server/admin-ui/create-table) for the first schema workflow, then return to SQL Studio to query and verify inserted rows.

================================================================================
# Admin UI Streaming
URL: https://kalamdb.org/docs/server/admin-ui/streaming
Source: content/server/admin-ui/streaming.mdx

> Monitor KalamDB streaming topics from the Admin UI, inspect topic activity, and connect realtime stream behavior to SQL topics.
# KalamDB Admin UI Streaming

The Streaming page shows topic-oriented realtime activity so operators can inspect topic inventory, consumer state, and stream behavior from the Admin UI.

## What Streaming Shows

The Streaming area has separate views for topics and consumers:

- `Topics`: searchable topic inventory with topic name, partitions, routes, retention window, last update time, and actions
- `Consumers`: consumer and offset monitoring for active consumer groups and claims
- topic actions to open the relevant SQL view or inspect a topic in place
- refresh controls for pulling current topic and consumer state

## When To Use It

Use this page to confirm expected topics are visible, check retention and routing metadata, inspect producer/consumer behavior, and correlate stream activity with Jobs and Logs.

For SQL topic setup, see [SQL Topics](https://kalamdb.org/docs/server/topic-pubsub/sql-topics).

================================================================================
# Admin UI Users
URL: https://kalamdb.org/docs/server/admin-ui/users
Source: content/server/admin-ui/users.mdx

> Use the KalamDB Admin UI Users page to inspect local users, roles, OIDC identities, and account state during auth operations.
# KalamDB Admin UI Users

The Users page is the Admin UI surface for reviewing KalamDB identities, roles, pending invites, and authentication state.

## What Users Shows

The Users page includes:

- search box for filtering users
- page-size selector and previous/next pagination controls
- refresh action
- `Create User` action
- `Pending Invites` table with invite ID, email, role, expiration, creation time, reinvite action, and delete action
- `Users` table with name, user ID, role, email, creation time, edit action, and delete action
- protected handling for system/root accounts where destructive actions may be disabled

## When To Use It

Use this page after bootstrap, local-user changes, or OIDC rollout to confirm the DBA account exists, roles match expectations, invites are still valid, and service/system accounts are not accidentally removed.

For SQL user commands, see [Users SQL Reference](https://kalamdb.org/docs/server/sql-reference/users).

================================================================================
# HTTP API Reference
URL: https://kalamdb.org/docs/server/api/http-reference
Source: content/server/api/http-reference.mdx

> Complete HTTP API reference for KalamDB. Explore route maps, auth rules, request/response payloads, error codes, and endpoint contracts.
# HTTP API Reference

Base URL: `http://<host>:2900`  
Version prefix: `/v1`

## Route Map

### Health and status

- `GET /health` (localhost-only)
- `GET /v1/api/healthcheck` (localhost-only)
- `GET /v1/api/cluster/health` (localhost-only)

### SQL and files

- `POST /v1/api/sql`
- `GET /v1/files/{namespace}/{table_name}/{subfolder}/{file_id}`

### WebSocket

- `GET /v1/ws`

### Auth

- `POST /v1/api/auth/login`
- `POST /v1/api/auth/refresh`
- `POST /v1/api/auth/logout`
- `GET /v1/api/auth/me`
- `POST /v1/api/auth/setup`
- `GET /v1/api/auth/status`

### Topic HTTP API

- `POST /v1/api/topics/consume`
- `POST /v1/api/topics/ack`

## Authentication Rules

### Bearer token required

```http
Authorization: Bearer <JWT_TOKEN>
```

- `POST /v1/api/sql`
- `GET /v1/files/...`
- `POST /v1/api/topics/consume`
- `POST /v1/api/topics/ack`

Basic auth is rejected on these endpoints.

### Cookie or bearer accepted

- `POST /v1/api/auth/refresh`
- `GET /v1/api/auth/me`

### Public endpoints

- `POST /v1/api/auth/login`
- `POST /v1/api/auth/logout`
- `POST /v1/api/auth/setup` (localhost-only unless `auth.allow_remote_setup = true`)
- `GET /v1/api/auth/status` (localhost-only unless remote setup enabled)
- `GET /health` and `GET /v1/api/healthcheck` (localhost-only)
- `GET /v1/api/cluster/health` (localhost-only)

## SQL Endpoint

### `POST /v1/api/sql`

Headers:

- `Authorization: Bearer <JWT_TOKEN>`
- `Content-Type: application/json` or `multipart/form-data`

### JSON body

```json
{
  "sql": "SELECT * FROM app.messages WHERE id = $1",
  "params": [123],
  "namespace_id": "app"
}
```

`namespace_id` applies only to that request. Interactive clients such as the
CLI can store the namespace locally after a successful `USE namespace` and send
it again on later requests.

### Multipart body for `FILE(...)`

Parts:

- `sql`
- `params` (optional JSON array string)
- `namespace_id` (optional request-scoped default namespace)
- file parts named `file:<placeholder>`

SQL:

```sql
INSERT INTO app.docs (id, attachment) VALUES ('d1', FILE("contract"));
```

Multipart file key must be `file:contract`.

### Success shape

`SELECT` statements and `INSERT ... ON CONFLICT ... DO UPDATE ... RETURNING` statements
return query rows in `results[]`:

```json
{
  "status": "success",
  "results": [
    {
      "schema": [{"name":"id","data_type":"BigInt","index":0}],
      "rows": [[1]],
      "row_count": 1,
      "as_user": "alice"
    }
  ],
  "took": 12.3
}
```

Plain `INSERT`, `UPDATE`, and `DELETE` statements without `RETURNING` return an affected-row
count instead of data rows. Upsert without `RETURNING` follows the insert result shape.

Example upsert that returns rows:

```sql
INSERT INTO app.items (id, name)
VALUES (1, 'beta')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name
RETURNING id, name;
```

### Error shape

```json
{
  "status": "error",
  "results": [],
  "took": 1.2,
  "error": {
    "code": "INVALID_SQL",
    "message": "...",
    "details": null
  }
}
```

### Common SQL error codes

- `INVALID_SQL`
- `PERMISSION_DENIED`
- `TABLE_NOT_FOUND`
- `RATE_LIMIT_EXCEEDED`
- `NOT_LEADER`
- `CLUSTER_UNAVAILABLE`
- `FILE_TOO_LARGE`
- `TOO_MANY_FILES`
- `INVALID_MIME_TYPE`

## File Download

### `GET /v1/files/{namespace}/{table_name}/{subfolder}/{file_id}`

- bearer token required
- optional `user_id` query parameter for user-table scope resolution
- `SYSTEM` and `STREAM` tables are rejected for file paths

Response:

- `200` with binary content
- `400/403/404` for validation, permission, or not-found failures

## Auth Endpoints

### `POST /v1/api/auth/login`

```json
{
  "user": "alice",
  "password": "Secret123!"
}
```

Returns user profile + access/refresh tokens and sets auth cookie.

### `POST /v1/api/auth/refresh`

Accepts bearer token or cookie. Returns rotated token pair.

### `POST /v1/api/auth/logout`

Clears auth cookie.

### `GET /v1/api/auth/me`

Returns current authenticated user info.

### `POST /v1/api/auth/setup`

Initial bootstrap only, when root has no password yet.

```json
{
  "user": "admin",
  "password": "AdminPass123!",
  "root_password": "RootPass123!",
  "email": "admin@example.com"
}
```

Sets root password and creates the DBA user. Does not auto-login.

### `GET /v1/api/auth/status`

Returns setup status:

```json
{
  "needs_setup": true,
  "message": "Server requires initial setup..."
}
```

## Topic Endpoints

Both require bearer auth and role in `{service, dba, system}`.

### `POST /v1/api/topics/consume`

```json
{
  "topic_id": "orders_topic",
  "group_id": "worker_group",
  "start": "Latest",
  "limit": 100,
  "partition_id": 0,
  "timeout_seconds": 10
}
```

Accepted `start` formats:

- `"Latest"`
- `"Earliest"`
- `{ "Offset": 123 }`

### `POST /v1/api/topics/ack`

```json
{
  "topic_id": "orders_topic",
  "group_id": "worker_group",
  "partition_id": 0,
  "upto_offset": 10
}
```

## Health Endpoints

`GET /health` and `GET /v1/api/healthcheck` return the same payload and are localhost-only:

```json
{
  "status": "healthy",
  "version": "...",
  "api_version": "v1",
  "build_date": "..."
}
```

For live protocol details, continue to [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol).

================================================================================
# API Reference
URL: https://kalamdb.org/docs/server/api
Source: content/server/api/index.mdx

> Complete HTTP and WebSocket API reference for KalamDB. Learn how to integrate backend services, build custom clients, and use realtime streaming endpoints.
# API Reference

This root chapter contains transport-level contracts for backend integrations, services, and custom clients.

- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference): Routes, auth rules, request/response payloads, and error codes.
- [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol): Auth handshake, subscription lifecycle, message schema, limits, and protocol errors.

For SQL command syntax, use [SQL Reference](https://kalamdb.org/docs/server/sql-reference).
For first-time server setup/login endpoint usage, see [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication).

================================================================================
# WebSocket Protocol
URL: https://kalamdb.org/docs/server/api/websocket-protocol
Source: content/server/api/websocket-protocol.mdx

> KalamDB WebSocket protocol reference. Learn about connection handshakes, auth flows, message contracts, rate limits, and realtime stream behavior.
# WebSocket Protocol

Endpoint: `ws://<host>:2900/v1/ws`  
Transport: RFC 6455  
Message format: JSON text frames (server may send gzip-compressed binary frames for larger payloads)

## Connection And Authentication

### Upgrade

Client opens `GET /v1/ws`.

Server can reject pre-upgrade for:

- shutdown mode
- origin policy failure
- strict origin check with missing `Origin` header

### Origin policy

Origin allow-list resolution:

1. `security.allowed_ws_origins` if non-empty
2. otherwise `security.cors.allowed_origins`

### Post-connect auth (required)

```json
{
  "type": "authenticate",
  "method": "jwt",
  "token": "<JWT_TOKEN>"
}
```

Success:

```json
{
  "type": "auth_success",
  "user_id": "u_...",
  "role": "Dba"
}
```

Failure:

```json
{
  "type": "auth_error",
  "message": "Invalid username or password"
}
```

## Client Messages

- `authenticate`
- `subscribe`
- `next_batch`
- `unsubscribe`

### `subscribe`

```json
{
  "type": "subscribe",
  "subscription": {
    "id": "orders_live",
    "sql": "SELECT * FROM app.orders WHERE status = 'open'",
    "options": {
      "batch_size": 200,
      "last_rows": 100,
      "from": 12345
    }
  }
}
```

### `next_batch`

```json
{
  "type": "next_batch",
  "subscription_id": "orders_live",
  "last_seq_id": 12345
}
```

### `unsubscribe`

```json
{
  "type": "unsubscribe",
  "subscription_id": "orders_live"
}
```

## Server Messages

- `auth_success` / `auth_error`
- `subscription_ack`
- `initial_data_batch`
- `change`
- `error`

### `subscription_ack`

```json
{
  "type": "subscription_ack",
  "subscription_id": "orders_live",
  "total_rows": 0,
  "batch_control": {
    "batch_num": 0,
    "has_more": true,
    "status": "loading",
    "last_seq_id": 12345,
    "snapshot_end_seq": 13000
  },
  "schema": [
    {"name":"id","data_type":"BigInt","index":0}
  ]
}
```

### `initial_data_batch`

```json
{
  "type": "initial_data_batch",
  "subscription_id": "orders_live",
  "rows": [{"id":1,"status":"open","_seq":12346}],
  "batch_control": {
    "batch_num": 0,
    "has_more": true,
    "status": "loading",
    "last_seq_id": 12346,
    "snapshot_end_seq": 13000
  }
}
```

### `change`

```json
{
  "type": "change",
  "subscription_id": "orders_live",
  "change_type": "insert",
  "rows": [{"id":2,"status":"open","_seq":13001}]
}
```

`change_type` values:

- `insert`
- `update`
- `delete`

### `error`

```json
{
  "type": "error",
  "subscription_id": "orders_live",
  "code": "INVALID_SQL",
  "message": "..."
}
```

## Limits

- `rate_limit.max_subscriptions_per_user` (default `10`)
- hard cap per connection: `100`
- `security.max_ws_message_size` (default `1MB`)
- `rate_limit.max_messages_per_sec` (default `50`)
- auth timeout: `websocket.auth_timeout_secs` (default `3`)

## Error Codes

Common WebSocket protocol/error codes include:

- `AUTH_REQUIRED`
- `INVALID_SUBSCRIPTION_ID`
- `SUBSCRIPTION_LIMIT_EXCEEDED`
- `INVALID_SQL`
- `RATE_LIMIT_EXCEEDED`
- `MESSAGE_TOO_LARGE`
- `UNSUPPORTED_DATA`
- `PROTOCOL`

For endpoint auth matrix and SQL payload contracts, see [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference).

================================================================================
# Clustering & HA
URL: https://kalamdb.org/docs/server/architecture/clustering
Source: content/server/architecture/clustering.mdx

> Learn how KalamDB clustering works with Raft consensus for high availability, leader election, log replication, and automatic failover.
# Clustering & High Availability

KalamDB uses a **Multi-Raft** architecture to scale reads, writes, and subscriptions across nodes while keeping data strongly replicated per shard.

Cluster routing depends on table type: `USER` and `STREAM` data route by effective `user_id`, while
`SHARED` data currently routes through the shared data group. For table-type semantics, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## How the Cluster Is Structured

KalamDB does not run a single Raft group for everything. It runs multiple groups:

- `Meta`: cluster/system metadata
- `DataUserShard(N)`: user-data shards
- `DataSharedShard(N)`: shared-data shards

Each group has its own leader and followers. This means leadership is distributed across the cluster, not centralized to a single node for all workloads.

Sharding is handled by a router that hashes user/table identifiers to shard groups, so users are naturally distributed across shards. This is how KalamDB scales tenant load and isolates shard-level hotspots.

A single node participates in multiple Raft groups simultaneously. One node is not limited to one group; instead, each node can be a leader for some groups and a follower for others at the same time.

## Multi-Raft Cluster Diagram

```mermaid
flowchart TB
  subgraph Cluster["KalamDB Cluster"]

    %% Nodes
    N1["Node 1"]
    N2["Node 2"]
    N3["Node 3"]

    %% Meta Group
    subgraph Meta["Meta Raft Group"]
      M1["Leader (Node 1)"]
      M2["Follower (Node 2)"]
      M3["Follower (Node 3)"]
    end

    %% User Shard Example
    subgraph UserShard7["DataUserShard(7)"]
      U1["Leader (Node 2)"]
      U2["Follower (Node 1)"]
      U3["Follower (Node 3)"]
    end

    %% Another User Shard
    subgraph UserShard12["DataUserShard(12)"]
      U4["Leader (Node 3)"]
      U5["Follower (Node 1)"]
      U6["Follower (Node 2)"]
    end

    %% Shared Shard
    subgraph SharedShard["DataSharedShard(0)"]
      S1["Leader (Node 1)"]
      S2["Follower (Node 2)"]
      S3["Follower (Node 3)"]
    end
  end

  %% Routing
  Client["Client Request"]
  Router["Shard Router (hash user_id / table)"]
  Client --> Router
  Router --> UserShard7
  Router --> UserShard12
  Router --> SharedShard

  %% Raft replication arrows
  U1 --> U2
  U1 --> U3
  U4 --> U5
  U4 --> U6
  S1 --> S2
  S1 --> S3
  M1 --> M2
  M1 --> M3
```

### How to read this diagram

- The cluster has multiple Raft groups, not one.
- Each group such as `Meta`, `DataUserShard(N)`, or `DataSharedShard(N)` runs its own Raft consensus.
- Each group has one leader that handles writes and followers that replicate data and can serve follower-side access patterns.
- A single node participates in many groups at once, which is why Node 1, Node 2, and Node 3 appear repeatedly across the diagram.

Example:

- `DataUserShard(7)` has its leader on Node 2.
- `DataUserShard(12)` has its leader on Node 3.

So:

- load is spread across nodes
- there is no single leader bottleneck
- multiple writes can happen in parallel across different groups

### Why this matters

Instead of one cluster having one leader and turning that leader into a bottleneck, one cluster has many leaders, one per active Raft group.

That is the core scaling property of Multi-Raft in KalamDB.

### Tiny mental model

```text
user-1  -> shard 7  -> leader on Node 2
user-2  -> shard 12 -> leader on Node 3
user-3  -> shard 7  -> leader on Node 2
```

So:

- the same user stays on the same shard for consistent writes
- different users can land on different leaders for parallel scaling

## High Concurrent Connection Model

KalamDB accepts WebSocket clients on any node and manages them in a lock-free connection registry:

- `ConnectionsManager` uses `DashMap` indexes for fast concurrent connection/subscription access.
- Connection lifecycle, authentication timeout, heartbeat timeout, and subscription routing are all centralized there.
- The default concurrent connection guard is high (`10,000`), with configurable limits.

This allows high fan-in of realtime clients while keeping routing overhead low.

## Leader/Follower Access Pattern

Clients can connect to either leader or follower nodes.

- **Reads/subscription bootstrap** can be served on the node the client is connected to.
- **Writes** are always leader-authoritative per Raft group.
- If a request lands on a follower for a write path, KalamDB forwards it to the current leader.
- If leadership changed mid-request, typed `NotLeader` handling retries through the known leader.

This gives you flexible client routing without sacrificing consistency for writes.

## Notification Forwarding Across Nodes

Live subscribers do not need to be connected to the leader node for their shard.

When data changes:

1. The shard leader processes the change notification.
2. The leader broadcasts forwarded notifications to follower nodes over cluster RPC.
3. Each follower dispatches to subscriptions connected locally on that follower.

This is the key reason you can attach users to any node and still receive correct realtime updates for their shard data.

```mermaid
flowchart LR
  A["Client connected to Follower"] --> B["Follower stores local WS subscription"]
  C["Write hits shard leader"] --> D["Leader commits via Raft"]
  D --> E["Leader emits change notification"]
  E --> F["Leader broadcasts notify_followers RPC"]
  F --> G["Follower receives forwarded event"]
  G --> H["Follower pushes update to local client"]
```

## Sharding + HA in Practice

KalamDB shards users across multiple Raft data groups. Each shard group has its own:

- leader election
- replication log
- follower set
- failover boundary

Operationally, this gives you:

- better parallelism (many active leaders across groups)
- shard-level fault isolation
- horizontal scaling by increasing cluster nodes and shard count
- high availability from follower replicas per shard

When a node fails, Raft elects new leaders for affected groups and clients can reconnect to any healthy node.

## Supported Cluster Commands

Cluster operations are surfaced through CLI meta-commands. See
[Interactive Commands — cluster](https://kalamdb.org/docs/cli/interactive-commands)
for the operator command list.

For raw inspection without CLI formatting, query `system.cluster` and `system.cluster_groups` directly. Old `CLUSTER LIST` / `STATUS` / `LS` SQL forms are replaced by the CLI and system views.

`CLUSTER LEAVE` remains unsupported. `CLUSTER TRANSFER LEADER` is still routed by the backend, but some builds can report it as unsupported at runtime depending on the OpenRaft version in use.

## How user data is divided across groups

KalamDB uses Multi-Raft rather than one Raft group for the whole cluster.

- One group handles metadata.
- Multiple user data groups handle user and stream data.
- The target user group is chosen by hashing `user_id` into one of `cluster.user_shards` groups.

That means all writes for a specific user are coordinated by the same user-data group leader at a given time. The data is still replicated to that group's followers, but the active write path for one user's workload is not scattered across many leaders. This improves locality and reduces cross-group coordination, which is one of the reasons the cluster can stay fast as concurrency grows.

## Follower insert example

Suppose `user-42` is mapped to `DataUserShard(7)`, and the client sends an insert to node 2:

```bash
kalam --url http://node-2:2900 --command "INSERT INTO app.messages (id, body) VALUES (101, 'hello')"
```

If node 2 is a follower for `DataUserShard(7)`, KalamDB:

1. prepares and classifies the SQL once,
2. resolves the target group from the table type and effective `user_id`,
3. forwards the original SQL, params, auth header, and request id to the leader of `DataUserShard(7)`,
4. lets that leader append and replicate the write through Raft,
5. returns the committed result back through the follower to the client.

So clients can write through followers, but the actual commit still happens on the correct group leader.

## Shared tables today

Shared tables are currently not truly sharded. Today they route to a single shared data group.

More flexible shared-table sharding is a work in progress. The planned direction is partition-by-key so each shared table can define how rows are partitioned and which group should own them.

## Configuration Knobs

In cluster mode, tune shard topology from cluster config:

- `cluster.user_shards`
- `cluster.shared_shards`

These values determine how many independent Raft data groups exist for user and shared workloads.
To view all the configurations for the Cluster setup: 
- [/docs/server/configurations/advanced#cluster](https://kalamdb.org/docs/server/configurations/advanced#cluster)

## Deploying a 3-Node Cluster

The quickest way to start a cluster is with Docker:

```bash
KALAMDB_JWT_SECRET="$(openssl rand -base64 32)" \
curl -sSL https://raw.githubusercontent.com/kalamdb/KalamDB/main/docker/run/cluster/docker-compose.yml | docker-compose -f - up -d
```

## Backup & Restore

```sql
-- Backup a namespace
BACKUP DATABASE myapp TO '/backups/myapp-2026-02-18';

-- Restore from backup
RESTORE DATABASE myapp FROM '/backups/myapp-2026-02-18';

-- View backup history
SHOW BACKUPS FOR DATABASE myapp;
```

## Related Docs

- [/docs/server/sql-reference/cluster](https://kalamdb.org/docs/server/sql-reference/cluster)
- [Interactive Commands — cluster](https://kalamdb.org/docs/cli/interactive-commands)
- [/docs/server/configurations/advanced#cluster](https://kalamdb.org/docs/server/configurations/advanced#cluster)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)

================================================================================
# Data Types
URL: https://kalamdb.org/docs/server/architecture/datatypes
Source: content/server/architecture/datatypes.mdx

> Practical guide to choosing and using KalamDB data types in SQL. Covers BIGINT, TEXT, BOOLEAN, TIMESTAMP, FILE, EMBEDDING, and more with examples.
# Data Types

This page is a practical guide for **using data types in SQL** when building with KalamDB.

Data types are independent from table types, but some types have table-type-specific behavior. `FILE`
columns are supported on `USER` and `SHARED` tables, and `EMBEDDING(n)` columns become most useful
with vector search. See [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
for the table boundary.

## Type Selection Flow

```mermaid
flowchart TD
  A["Data you need to store"] --> B{"Structured document?"}
  B -- "Yes" --> C["JSON"]
  B -- "No" --> D{"Time-based value?"}
  D -- "Timestamped event" --> E["TIMESTAMP"]
  D -- "Calendar date only" --> F["DATE"]
  D -- "Not time" --> G{"Exact arithmetic needed?"}
  G -- "Yes (money/rates)" --> H["DECIMAL(p,s)"]
  G -- "No" --> I{"Whole number?"}
  I -- "Small/medium" --> J["INT"]
  I -- "Large / IDs" --> K["BIGINT"]
  I -- "No" --> L{"Boolean?"}
  L -- "Yes" --> M["BOOLEAN"]
  L -- "No" --> N["TEXT"]
```

## Quick Type Selection

Use this table when deciding a column type:

| You store... | Use | Why |
|---|---|---|
| IDs from `SNOWFLAKE_ID()` | `BIGINT` | 64-bit integer ID generation |
| Small whole numbers (status, counters) | `INT` | Compact and fast |
| Large counters | `BIGINT` | Higher range |
| User-facing text | `TEXT` | Flexible string storage |
| True/false values | `BOOLEAN` | Clear semantics |
| Decimal money-like values | `DECIMAL(p,s)` | Exact precision |
| Timestamps | `TIMESTAMP` | Time-based queries and ordering |
| Dates only | `DATE` | Calendar values without time |
| Structured document payload | `JSON` | Nested object/array data |
| Vector embeddings | `EMBEDDING(n)` | Semantic retrieval and similarity search |

## Common Types in Practice

### Integer Types

```sql
CREATE TABLE app.orders (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  user_id BIGINT NOT NULL,
  item_count INT NOT NULL DEFAULT 1
) WITH (TYPE = 'USER');
```

- Prefer `INT` for bounded counts.
- Prefer `BIGINT` for IDs and values that can grow large.

### Text and Boolean

```sql
CREATE TABLE app.users (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  email TEXT NOT NULL,
  display_name TEXT,
  is_active BOOLEAN NOT NULL DEFAULT TRUE
) WITH (TYPE = 'USER');
```

- Use `TEXT` for variable-length strings.
- Use `BOOLEAN` instead of `0/1` integers for feature flags and state.

### Decimal Values

```sql
CREATE TABLE billing.invoice_items (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  sku TEXT NOT NULL,
  amount DECIMAL(12,2) NOT NULL,
  tax_rate DECIMAL(5,4) NOT NULL DEFAULT 0.0000
) WITH (TYPE = 'SHARED');
```

- Use `DECIMAL` for exact arithmetic (pricing, balances, rates).
- Pick precision/scale deliberately, e.g. `DECIMAL(12,2)`.

### Time Types

```sql
CREATE TABLE chat.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  user_id BIGINT NOT NULL,
  content TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
) WITH (TYPE = 'USER');

SELECT *
FROM chat.messages
WHERE created_at >= NOW() - INTERVAL '7 days'
ORDER BY created_at DESC;
```

- Use `TIMESTAMP` for event records and ordering.
- Use `DATE` when time-of-day is not needed.

### JSON Payloads

```sql
CREATE TABLE app.events (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  event_name TEXT NOT NULL,
  payload JSON,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
) WITH (
  TYPE = 'STREAM',
  TTL_SECONDS = 300
);
```

- Use `JSON` for dynamic payloads and schema-flexible metadata.
- Keep frequently filtered fields as typed top-level columns.

### Embeddings for Vector Search

```sql
CREATE TABLE rag.documents_vectors (
  id BIGINT PRIMARY KEY,
  doc_embedding EMBEDDING(384)
) WITH (TYPE = 'USER');

ALTER TABLE rag.documents_vectors
  CREATE INDEX doc_embedding USING COSINE;

SELECT id
FROM rag.documents_vectors
ORDER BY COSINE_DISTANCE(doc_embedding, '[0.12,0.03,0.98]')
LIMIT 5;
```

- Use `EMBEDDING(n)` for fixed-size vector columns.
- Pick `n` to match the embedding model you generate in your app or worker.
- Index the embedding column before using it for nearest-neighbor retrieval.

## Type Conversion

Use explicit casts when moving between compatible types.

```sql
SELECT CAST('42' AS INT) AS count_value;
SELECT CAST(amount AS DECIMAL(12,2)) AS normalized_amount
FROM billing.invoice_items;
```

Prefer explicit conversion over relying on implicit casting in critical queries.

## Nullability and Defaults

Use `NOT NULL` for required fields and `DEFAULT` for stable inserts.

```sql
CREATE TABLE app.sessions (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  user_id BIGINT NOT NULL,
  started_at TIMESTAMP NOT NULL DEFAULT NOW(),
  ended_at TIMESTAMP,
  is_valid BOOLEAN NOT NULL DEFAULT TRUE
) WITH (TYPE = 'USER');
```

## Recommended Patterns

- IDs: `BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID()`
- Created timestamps: `TIMESTAMP NOT NULL DEFAULT NOW()`
- Money/rates: `DECIMAL` (not floating point)
- Optional user text: nullable `TEXT`
- Real-time transient events: `STREAM` + `TTL_SECONDS`

## Related Docs

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search)
- [/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype)
- [/docs/server/sql-reference/tables](https://kalamdb.org/docs/server/sql-reference/tables)
- [/docs/server/sql-reference/file-datatype](https://kalamdb.org/docs/server/sql-reference/file-datatype)

================================================================================
# File - Upload & DataType
URL: https://kalamdb.org/docs/server/architecture/file-upload-datatype
Source: content/server/architecture/file-upload-datatype.mdx

> Understand how FILE upload and read flows map to USER and SHARED tenant storage paths in KalamDB. Covers multipart uploads, FileRef metadata, physical object-store layout, and file retrieval.
# File - Upload & DataType

KalamDB separates file metadata from file bytes:

- SQL rows store a **`FileRef` JSON payload** in `FILE` columns (id, size, sha256, mime, subfolder, original name).
- **Binary content is written as a real object** in the configured storage backend — a file on the local filesystem or an object in S3, GCS, or Azure Blob Storage.

This is **not** a BLOB-in-the-row model like PostgreSQL `BYTEA`, MySQL `BLOB`, or SQLite `BLOB`. KalamDB never stores file bytes inside RocksDB row values or inside Parquet column pages. Queries return metadata; downloads stream bytes from the storage path the `FileRef` points at.

This allows database queries and file transfer to scale independently while preserving table-level security.

File storage follows table-type scope. For the full `USER`/`SHARED`/`STREAM` capability matrix, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## Upload and Read Flow

1. Client sends SQL using `FILE("placeholder")`.
2. Client attaches multipart file part as `file:<placeholder>`.
3. KalamDB validates auth and table type (`USER` / `SHARED` only).
4. Binary is staged briefly, then **finalized as a standalone object** under the table's storage path (see [Physical storage layout](#physical-storage-layout)).
5. The hot-tier row stores a `FileRef` JSON payload used for future reads/downloads.

For endpoint details, see [/docs/server/api/http-reference](https://kalamdb.org/docs/server/api/http-reference).

## Physical storage layout

FILE uploads land in the **`base_directory`** of the table's assigned storage (from `system.storages`, default **`local`**). For a typical local dev server that directory is under `storage.data_path`:

```text
{data_path}/storage/          # default base_directory for the local storage
├── {namespace}/{tableName}/           # SHARED table root
│   └── f0001/1234567890123456789-contract.pdf
└── {namespace}/{tableName}/{userId}/  # USER table root (per-user scope)
    └── f0001/1234567890123456789-profile.md
```

With the default server templates (`shared_tables_template`, `user_tables_template`) and `data_path = "./data"`:

| Table type | Template | Example relative path under `storage/` |
|------------|----------|--------------------------------------|
| `SHARED` | `{namespace}/{tableName}` | `app/documents/f0001/1234567890123456789-contract.pdf` |
| `USER` | `{namespace}/{tableName}/{userId}` | `okf_sync/context_files/9876543210987654321/f0001/1234567890123456789-profile.md` |

Each upload gets:

1. A **snowflake file id** (used in the on-disk filename).
2. A **subfolder** such as `f0001` (rotates as folders fill; configurable via `max_files_per_folder`).
3. A **stored filename** `{id}-{sanitized-original-name}.{ext}` built from the `FileRef`.

The full object key / filesystem path is:

```text
{base_directory}/{resolved_template}/{subfolder}/{stored_filename}
```

Remote storages use the same relative layout inside the bucket or container prefix configured in `base_directory` (for example `s3://my-bucket/kalam/`).

### What lives in SQL vs what lives on disk

| Layer | Holds | Example |
|-------|-------|---------|
| RocksDB hot row | `FileRef` JSON in the `FILE` column | `{"id":"123…","sub":"f0001","name":"profile.md","size":142,"mime":"text/markdown","sha256":"a1b2…"}` |
| Parquet cold segment (after flush) | Same `FileRef` JSON in the column — **not** the bytes | Metadata only |
| Storage object store | **Actual file bytes** | `…/okf_sync/context_files/{userId}/f0001/123…-profile.md` |

Clients download through `GET /v1/files/{namespace}/{table}/{sub}/{stored_filename}` (or SDK helpers built on that path). The handler reads the object from storage; it does not decode bytes from a SQL BLOB column.

### End-to-end finalize path (backend)

From `kalamdb-filestore` + SQL file handlers:

1. Multipart upload bytes are written to a **temporary staging directory** per request.
2. `finalize_file` generates a `FileRef`, computes sha256/mime, allocates a subfolder, and **`put`s the bytes** to the table's storage backend via the object-store API.
3. The staging directory is removed; only the permanent object and the row's `FileRef` remain.
4. On row delete or FILE replacement, the old object can be garbage-collected separately from MVCC row versions.

See the [Live OKF Context Sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync#file-bytes-on-disk-not-in-sql) use case for a concrete walkthrough with `kalam dev` paths you can inspect on disk.

## USER vs SHARED Pathing Model

KalamDB routes file bytes according to table type:

- **USER tables**: file objects are stored under the authenticated user's tenant scope.
- **SHARED tables**: file objects are stored under shared table scope.

This matches KalamDB's tenant-aware architecture:

- user-private data stays isolated in user-tenant paths
- shared assets are accessible through shared-table policy

`SYSTEM` and `STREAM` table paths are not valid for file APIs.

### Backend-Verified Path Resolution

From backend implementation (`kalamdb-filestore` + API handlers):

- For `USER` tables, file operations are resolved with `user_id` context.
- For `SHARED` tables, file operations are resolved without `user_id` context.
- Download handler enforces this behavior and rejects `STREAM`/`SYSTEM` file access.

Default storage templates are:

- `shared_tables_template = "{namespace}/{tableName}"`
- `user_tables_template = "{namespace}/{tableName}/{userId}"`

So for a table `chat.attachments`:

- shared object example: `chat/attachments/f0001/1234567890123456789-photo.png`
- user object example: `chat/attachments/<userId>/f0001/1234567890123456789-photo.png`

Templates are configurable per storage with `CREATE STORAGE` / `ALTER STORAGE`, but the user-scoped resolution model remains the same. Changing templates affects **new** path resolution; existing objects keep their stored paths via the `FileRef` recorded in each row.

## Architecture Diagram

```mermaid
flowchart LR
  Client[Client] -->|upload request| Api[SQL and File API]
  Api -->|validate auth and table type| Router{Table Type Router}

  Router -->|USER table with user_id| UserPath[User Tenant Folder]
  Router -->|SHARED table without user_id| SharedPath[Shared Table Folder]

  UserPath --> Ref[Store file reference in row]
  SharedPath --> Ref
  UserPath --> ObjStore[(Object store / storage folder)]
  SharedPath --> ObjStore
  Ref --> Download[Read via file endpoint]
  ObjStore --> Download
  Download --> Client
```

## Why This Design Helps

- Keeps tenant boundaries explicit for uploads and reads.
- Avoids mixing user-private files with shared table assets.
- Lets applications query metadata with SQL while streaming bytes through file endpoints.
- **Object-store friendly:** bytes behave like normal files or S3 keys, so backup, CDN offload, and lifecycle policies apply to file content without rewriting SQL dumps.
- **Smaller hot/cold SQL tiers:** Parquet segments and RocksDB hold references and queryable fields, not multi-megabyte blobs per row.

## Related Docs

- [/docs/server/sql-reference/file-datatype](https://kalamdb.org/docs/server/sql-reference/file-datatype)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage)
- [/docs/use-cases/live-okf-context-sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync) — sync worker that uploads Markdown through `FILE` columns; inspect `kalam/server/data/storage/` after `kalam dev`

================================================================================
# Architecture Overview
URL: https://kalamdb.org/docs/server/architecture
Source: content/server/architecture/index.mdx

> Explore KalamDB's architecture: query execution, multi-tier storage (RocksDB/Parquet), realtime delivery, and cluster consensus.
# Architecture Overview

KalamDB combines SQL execution, realtime delivery, and multi-tier storage in one runtime.

It also supports vector search with `EMBEDDING(n)` columns, cosine indexes, and similarity ranking in SQL for semantic retrieval workflows.

Use this page as the map. The deeper architecture pages explain one boundary at a time:
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types),
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers),
[/docs/server/architecture/stream-storage](https://kalamdb.org/docs/server/architecture/stream-storage),
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests),
[/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query),
[/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering),
[/docs/server/architecture/datatypes](https://kalamdb.org/docs/server/architecture/datatypes),
[/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search), and
[/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype).

## Major Runtime Layers

- `kalamdb-api`: HTTP + WebSocket surface (`/v1/api/*`, `/v1/ws`)
- `kalamdb-core`: orchestration, SQL execution flow, authorization checks, job dispatch
- `kalamdb-sql`: SQL parser/classifier/extensions (`SUBSCRIBE`, `TOPIC`, `STORAGE`, `EXECUTE AS '<user_id>'`, etc.)
- `kalamdb-store` + RocksDB: hot write/read path
- `kalamdb-filestore` + Parquet: cold segments and manifests
- `kalamdb-raft`: cluster consensus and replication

## Runtime Flow

```mermaid
flowchart TD
  A["Client (HTTP / WebSocket)"] --> B["kalamdb-api"]
  B --> C["kalamdb-core orchestration"]
  C --> D["kalamdb-sql parser/classifier"]
  D --> E["DataFusion execution"]
  E --> F["Hot Tier (RocksDB)"]
  F --> G["Flush Jobs"]
  G --> H["Cold Tier (Parquet + manifest)"]
  C --> I["Realtime notifications"]
  C --> J["kalamdb-raft (cluster mode)"]
```

## Architecture Reading Map

```mermaid
flowchart TD
  A["Start: Architecture Overview"] --> B["Table Types"]
  B --> C["USER / SHARED / STREAM scope"]
  C --> D["Storage Tiers"]
  D --> E["Manifests"]
  C --> F["Stream Storage"]
  C --> G["Live Query"]
  G --> H["Clustering & HA"]
  C --> I["FILE Upload & DataType"]
  C --> J["Data Types"]
  J --> K["Vector Search"]
```

Read [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types) first when you
are deciding data ownership or security. Read
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers) and
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests) when you are tuning
flush, compaction, or storage templates. Read
[/docs/server/architecture/stream-storage](https://kalamdb.org/docs/server/architecture/stream-storage) when you are
tuning `STREAM` table retention, transient UI state, or TTL cleanup behavior. Read
[/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query) and
[/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering) when you are scaling
realtime connections or multi-node deployments.

## Data Directory Layout

Default paths are rooted at `storage.data_path` (usually `./data`):

```text
data/
├── rocksdb/      # hot tier
├── storage/      # cold parquet tier
├── snapshots/    # raft snapshots
└── streams/      # stream table logs
```

Table-specific cold paths are generated from templates:

- shared tables: `storage.shared_tables_template`
- user tables: `storage.user_tables_template`

## Query Path (Simplified)

1. API receives SQL (`POST /v1/api/sql`) with bearer auth.
2. SQL is parsed/classified (standard + KalamDB extensions).
3. Authorization and table-type rules are applied.
4. Query executes against hot tier, cold tier, or both.
5. Response returns normalized JSON schema/rows.
6. Optional change notifications flow to subscriptions/topics.

For detailed flush behavior and cold-tier movement, see
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers).
For the file-backed `STREAM` table path, see
[/docs/server/architecture/stream-storage](https://kalamdb.org/docs/server/architecture/stream-storage).

## Technology Stack

| Component | Technology | Notes |
|-----------|-----------|-------|
| Language | Rust 1.92+ | concurrency + memory safety |
| Query engine | Apache DataFusion 54.0 | SQL planning/execution (DuckDB dialect for lambdas and JSON operators) |
| Columnar format | Apache Arrow 58.x | in-memory batches |
| Cold storage | Apache Parquet 58.x | compressed columnar files |
| Hot storage | RocksDB 0.24 | write-heavy low-latency path |
| API runtime | Actix Web 4.12 | HTTP + WebSocket |
| Auth | bcrypt + JWT | password + token flows |

## Continue Reading

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/stream-storage](https://kalamdb.org/docs/server/architecture/stream-storage)
- [/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)
- [/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering)
- [/docs/server/architecture/datatypes](https://kalamdb.org/docs/server/architecture/datatypes)
- [/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search)
- [/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype)

================================================================================
# Live Query
URL: https://kalamdb.org/docs/server/architecture/live-query
Source: content/server/architecture/live-query.mdx

> How KalamDB achieves high-performance real-time subscriptions and offline-first sync using Sequence IDs and tenant-aware routing.
# Live Query Architecture

KalamDB is designed from the ground up to support real-time, reactive applications. Unlike traditional databases that require complex polling mechanisms or external message brokers (like Kafka or Redis) to stream changes, KalamDB natively supports **Live Queries** via WebSockets.

This architecture allows clients to subscribe to changes on `USER`, `SHARED`, and `STREAM` tables and receive instant push notifications whenever data is inserted, updated, or deleted.

Live queries inherit the table-type access and scope rules from
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types). `USER` and `STREAM`
subscriptions are user-scoped; `SHARED` subscriptions use the shared table's `ACCESS_LEVEL` policy.

## How It Works

When a client subscribes to a query (e.g., `SELECT * FROM chat.messages WHERE conversation_id = 123`), KalamDB does not just execute the query once. It registers a **Live Query Subscription** in the cluster.

You can subscribe in two common ways:

- **Whole-row subscription**: `SELECT * FROM chat.changes WHERE ...`
- **Projection subscription**: `SELECT username, status FROM chat.changes WHERE ...`

Projection subscriptions are useful when the client needs only a few columns, which reduces payload size and improves UI update performance.

### Tenant-Aware Routing for Performance

One of the main reasons KalamDB achieves better performance than other databases for real-time workloads is its **tenant-aware storage architecture**.

In traditional databases, a live query often requires the system to scan or index the entire table or maintain complex global replication logs to find relevant changes. In KalamDB, `USER` and `STREAM` tables are physically sharded by `user_id`.

When a live query is registered for a specific user, the system doesn't need to scan the whole table. It directly monitors the isolated storage partition for that specific user tenant. This drastically reduces I/O overhead and allows the system to scale horizontally to millions of concurrent connections.

In clustered mode, the client may be connected to a follower node while writes are accepted by the leader. KalamDB forwards those changes through Raft replication so the follower applies them locally, evaluates matching subscriptions, and pushes updates to the connected client.

```mermaid
flowchart LR
    Client[Client App] -->|subscribe| Follower[Follower Node]
    Follower -->|register subscription| Leader[Leader Node]

    subgraph Cluster
        Leader -->|replicate subscription metadata| Follower
        Write[Write Operation] --> Leader
        Leader --> Router{Tenant Router}
        Router --> UserStore[User Tenant Storage]
        Router --> SharedStore[Shared Storage]
        UserStore --> Matcher[Query Matcher]
        SharedStore --> Matcher
        Matcher --> Leader
        Leader -->|replicate data change| Follower
    end

    Follower -->|local notify + push update| Client
```

## Offline-First and Sequence IDs

KalamDB's live queries are built to support **offline-first** applications seamlessly.

When subscribing to a table, you don't just get future events. You can also request historical data to synchronize your local state before the live stream begins. This is powered by **Sequence IDs** (`SeqId`), which provide a strict, monotonically increasing order of operations.

When initiating a subscription, you can specify:
- **`since_seq`**: Pull all changes that happened after a specific Sequence ID.
- **`fetch_last`**: Load only the latest *N* number of rows.

### Example: Chat Application

Imagine a user opening an old conversation in a chat app. Loading the entire history of the conversation would be slow and consume unnecessary bandwidth.

Instead, the client can subscribe to the conversation and request only the **latest 10 messages**:

1. The client sends a subscription request with `fetch_last: 10`.
2. KalamDB immediately returns an `initial_data_batch` containing those 10 messages.
3. The subscription remains open.
4. As new messages are sent in that conversation, KalamDB pushes them to the client in real-time.

If the user goes offline and reconnects later, the client can simply resubscribe using the `since_seq` of the last message it received. KalamDB will instantly push any messages missed during the offline period, followed by a seamless transition back to the live stream.

## Cluster-Wide Visibility

Active live query subscriptions are node-local runtime state. Each node exposes
its current in-memory subscriptions through `system.live`; that view is not
persisted to RocksDB and it is not replicated through Raft.

Clients can stay connected to follower nodes while writes are processed by the leader. The change stream is still delivered to followers, so followers can fan out updates to their connected clients. This helps increase total concurrent connection capacity because client sessions are distributed across nodes.

## Shared Table Subscriptions at High Scale

KalamDB has a dedicated shared-table streaming notification path optimized for high subscriber fan-out.

For `SHARED` tables, the runtime keeps a dedicated table-level subscription index so it can resolve shared subscribers quickly without user-partition lookup.

When a shared-table change arrives, KalamDB uses a streaming fan-out pipeline:

1. **Pre-compute row JSON once** and share it across workers.
2. **Snapshot subscriber handles** for the target shared table.
3. **Process subscribers in parallel chunks** (chunked dispatch) while still applying per-subscriber filter, projection, flow control, and channel delivery semantics.

Why this enables high subscriber counts:

- avoids repeating full row conversion for every subscriber
- uses shared payload memory for lower CPU and allocation overhead
- parallelizes fan-out while preserving subscription-specific behavior
- keeps backpressure behavior intact with flow control and non-blocking send paths

This is a core part of KalamDB's realtime power for `SHARED` tables, where a single change may need to reach very large subscriber groups with low latency.

## Table-Type Delivery Summary

| Table type | Subscription scope | Access policy used | Delivery shape |
|---|---|---|---|
| `USER` | Effective `user_id` partition | User-table read gate | Tenant-local matching and fan-out |
| `SHARED` | Whole shared table scope | Shared `ACCESS_LEVEL` read gate | Shared-table index plus chunked fan-out |
| `STREAM` | Effective `user_id` stream partition | User/stream read gate | TTL-oriented transient updates |
| `SYSTEM` | Not supported for app live query | DBA/System-only metadata access | No normal live-query stream |

```mermaid
sequenceDiagram
    participant C as Client
    participant F as Follower Node
    participant L as Leader Node
    participant S as Tenant Storage

    C->>F: WebSocket live subscribe
    F->>L: Register subscription metadata
    L-->>F: Subscription replicated

    L->>S: Apply write (insert/update/delete)
    S-->>L: Change with SeqId
    L-->>F: Replicate change event
    F-->>C: Push live update
```

## Related Docs

- [/docs/server/sql-reference/subscriptions](https://kalamdb.org/docs/server/sql-reference/subscriptions)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering)
- [/docs/ts-sdk/subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)

================================================================================
# Manifests
URL: https://kalamdb.org/docs/server/architecture/manifests
Source: content/server/architecture/manifests.mdx

> How KalamDB tracks cold Parquet segments across memory, RocksDB, and storage manifest.json files, including tail compaction semantics.
# Manifests

KalamDB uses a manifest as the authoritative index for cold Parquet segments. Every manifest is
owned by the manifest service, which decides where the current copy lives, when it is dirty, and
when storage `manifest.json` is rewritten.

Manifests exist behind the hot/cold flow described in
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers). They apply to
`USER` and `SHARED` cold-storage scopes; see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types) for the table-type
boundary.

## Manifest Tiers

| Table type | Hot manifest path | Cold manifest path |
|------------|-------------------|--------------------|
| `SHARED` | process memory -> `system.manifest` in RocksDB | `manifest.json` in table storage |
| `USER` | `system.manifest` in RocksDB | `manifest.json` in the user storage path |

This split is intentional:

- `SHARED` manifests stay in process memory because they are low-cardinality and widely reused.
- `USER` manifests stay out of process memory so millions of user-scoped manifests do not expand server RSS.
- Both table types still persist their durable hot copy in RocksDB before storage writes.

The canonical read path is `ManifestService::get_or_load()`:

1. check the fastest hot tier for the scope
2. fall back to RocksDB manifest copy
3. fall back to storage `manifest.json`
4. hydrate faster tiers above the answering layer

## Control Flow

```mermaid
flowchart TD
  A["SQL / flush / metadata change"] --> B["ManifestService"]

  subgraph Shared["SHARED table"]
    B --> S1["In-memory manifest cache"]
    S1 <--> S2["RocksDB system.manifest row"]
    S2 -->|flush or explicit persist| S3["storage manifest.json"]
    S3 -->|cache miss reload| S2
    S2 -->|shared hot reload| S1
  end

  subgraph User["USER table"]
    B --> U1["RocksDB system.manifest row"]
    U1 -->|flush or explicit persist| U2["user storage manifest.json"]
    U2 -->|cache miss reload| U1
  end
```

## What Happens On Normal Writes

For normal DML and flush-preparation updates, KalamDB does not rewrite `manifest.json` on every change.

1. The write path updates hot data in RocksDB.
2. The manifest service updates manifest metadata in the hot tier.
3. The manifest is marked `pending_write`.
4. Cold storage is updated later by the flush path.

That means manifest bookkeeping stays cheap on the write path while still keeping the latest authoritative manifest state available for reads, flush planning, and primary-key checks.

The hot manifest entry also tracks a sync state:

- `in_sync`
- `pending_write`
- `syncing`
- `stale`
- `error`

During a flush, the scope becomes `syncing` while the Parquet temp file is being written.

## When `manifest.json` Is Actually Written

`manifest.json` is persisted when KalamDB commits cold-storage state, not on every mutation.

- Automatic or manual flushes write Parquet, update manifest segment metadata, and then persist `manifest.json`.
- Reloads after cache misses can hydrate the hot tier from the stored manifest.
- Vector-index DDL metadata is a deliberate exception: it is persisted immediately as a control-plane metadata write so vector state is not lost when there is no row flush to trigger storage persistence.

Flush commits a new segment through one canonical path: append the segment metadata, write
`manifest.json`, then refresh RocksDB and shared-scope memory cache.

## Flush-Time Segment Append

For a normal cold append, KalamDB writes `batch-N.parquet.tmp`, renames it to `batch-N.parquet`,
then appends a new segment entry that includes:

- `min_seq` and `max_seq`
- `row_count`
- `size_bytes`
- `schema_version`
- `status`
- `column_stats` for indexed columns

`last_sequence_number` tracks the latest `batch-N` slot used for future flush naming.

## How Manifest Compaction Works

Post-flush small-segment compaction does not rebuild the whole manifest. It compactly rewrites only
the trailing small-file suffix.

The compactor:

1. selects the trailing run of small, committed, same-schema segments
2. rewrites the newest MVCC winners into `compact-<uuid>.parquet`
3. reacquires the manifest scope lock
4. verifies the selected inputs are still the current trailing suffix
5. truncates that suffix and appends the compacted replacement segment, or appends nothing if the
  suffix was fully pruned
6. persists the updated `manifest.json`

If another flush changed the manifest tail while compaction was running, the swap is skipped and
the compacted file is deleted.

Compaction filenames do not use the `batch-N` numbering scheme, so they do not consume a new batch
sequence slot.

## What A Real `manifest.json` Looks Like

The example below is taken from a real user-table manifest written by the KalamDB backend. The values vary by table and flush, but the field names and nesting match the on-disk JSON shape.

```json
{
  "table_id": "flush_manifest_ns_mpczl3q7_s3k_0.user_flush_test_mpczl3q7_s3k_0",
  "user_id": "admin",
  "version": 2,
  "created_at": 1779216537052,
  "updated_at": 1779216537063,
  "segments": [
    {
      "min_seq": 315199164996489218,
      "max_seq": 315199165030043648,
      "row_count": 20,
      "size_bytes": 2711,
      "created_at": 1779216537063,
      "id": "batch-0.parquet",
      "path": "batch-0.parquet",
      "column_stats": {
        "1": {
          "min": { "Int64": "315199164994551810" },
          "max": { "Int64": "315199165028106240" },
          "null_count": 0
        }
      },
      "schema_version": 1,
      "status": "committed"
    }
  ],
  "last_sequence_number": 0,
  "files": null,
  "vector_indexes": {}
}
```

What these fields mean in practice:

- `segments` is the durable list of cold Parquet batches for that exact table scope.
- `column_stats` is keyed by stable column ID, not by column name, so renames do not invalidate old segment metadata.
- `last_sequence_number` tracks the last `batch-N.parquet` slot used for segment file naming.
- `files` stays `null` until the table enables FILE-column subfolder tracking.
- `vector_indexes` stays empty until the table has vector index metadata to persist.

## Why Primary-Key Checks Use The Manifest Service

Primary-key existence and cold-segment pruning go through the manifest service instead of reopening `manifest.json` on every check.

That gives KalamDB one source of truth for:

- the freshest shared-table manifest already in memory
- the freshest user-table manifest already in RocksDB
- fallback reload from storage only when the hot tier does not have the manifest yet

This avoids repeated storage reads and keeps pruning decisions aligned with the same manifest state the flush path updates.

It also means manifest-aware reads see the same suffix-replacement result after compaction that the
flush and compaction jobs committed.

## Shared vs User Storage Layout

A `SHARED` table writes one manifest per table path:

```text
storage/<namespace>/<table>/manifest.json
```

A `USER` table writes one manifest per user-scoped table path:

```text
storage/<namespace>/<table>/<user_id>/manifest.json
```

Each manifest tracks the Parquet segments for that exact scope only.

## Related Docs

- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/sql-reference/storage](https://kalamdb.org/docs/server/sql-reference/storage)
- [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage)

================================================================================
# Storage Tiers
URL: https://kalamdb.org/docs/server/architecture/storage-tiers
Source: content/server/architecture/storage-tiers.mdx

> Understand KalamDB's hot RocksDB tier, cold Parquet tier, manifest state, and the current post-flush tail-compaction path.
# Storage Tiers

KalamDB uses a dual-tier storage architecture that keeps the write path hot in RocksDB while moving
queryable historical state into immutable Parquet segments.

This page describes how `USER` and `SHARED` tables move from hot to cold storage. For the table-type
rules that decide whether a table can flush at all, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## Hot Tier (RocksDB)

The hot tier handles incoming writes in RocksDB-backed table stores.

**Characteristics:**
- ⚡ Sub-millisecond write latency
- Optimized for point lookups, recent writes, and write-heavy workloads
- Holds the newest live rows and the newest delete tombstones
- Feeds the flush scheduler through manifest scopes marked `pending_write`

## Cold Tier (Parquet)

Flushed data is written to Apache Parquet files for efficient scans, pruning, and long-term
storage.

**Characteristics:**
- 📊 Columnar format for efficient analytics
- High compression ratios
- Each segment tracked in `manifest.json` and a hot manifest copy
- Supports multiple storage backends (local, S3, Azure, GCS)

### Parquet Compression

Table `COMPRESSION` is a cold-tier Parquet setting for `USER` and `SHARED` tables. KalamDB
currently supports three values:

| Value | Meaning | Typical use |
|-------|---------|-------------|
| `none` | Writes uncompressed Parquet pages. | Debugging, CPU-constrained flushes, or data already compressed at the value level |
| `snappy` | Fast Parquet compression and decompression with low CPU overhead. This is the default. | General workloads and latency-sensitive flushes |
| `zstd` | Zstandard Parquet compression at level 1 for better storage density with modest extra CPU. | Larger historical datasets or storage-cost-sensitive tables |

The selected codec is applied when `USER` and `SHARED` table scopes write `batch-N.parquet.tmp` and
when tail compaction writes `compact-<uuid>.parquet.tmp`. After the atomic rename, readers discover
the codec from the Parquet footer, so no manifest-side decompression setting is needed. This setting
does not affect RocksDB hot storage, WebSocket gzip, backups, or stream table log files. `STREAM`
tables do not accept table Parquet compression because current stream storage does not flush to
Parquet cold segments.

## End-to-End Tier Flow

```mermaid
flowchart LR
  A["INSERT / UPDATE / DELETE"] --> B["Hot write in RocksDB"]
  B --> C["Manifest marked as pending_write"]
  C --> D["STORAGE FLUSH job"]
  D --> E["Resolve latest hot versions per scope"]
  E --> F["Write batch-N.parquet.tmp"]
  F --> G["Atomic rename to batch-N.parquet"]
  G --> H["Persist manifest.json + hot manifest copy"]
  H --> I["Delete flushed hot rows"]
  I --> J["Optional post-flush small-segment compaction"]
  J --> K["Reads use hot + cold paths"]
```

## Flush Policy

Tables are configured with a flush policy that determines when data moves from hot to cold tier:

```sql
CREATE TABLE app.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  content TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (
  TYPE = 'USER',
  FLUSH_POLICY = 'rows:1000,interval:60'
);
```

| Policy | Description |
|--------|-------------|
| `rows:N` | Flush after N rows accumulated |
| `interval:N` | Flush every N seconds |
| `rows:N,interval:N` | Flush on whichever threshold is hit first |

## How Flushing Works (Current Engine Path)

The flush flow is job-driven and crash-safe:

1. DML writes land in RocksDB first.
2. The manifest scope is marked `pending_write`.
3. `STORAGE FLUSH TABLE` or `STORAGE FLUSH ALL` creates background flush jobs in `system.jobs`.
4. The flush executor resolves the target scope and scans hot rows in bounded batches.
5. The latest `_seq` wins per primary key. Latest tombstones stay hot instead of being written to
  Parquet.
6. `FlushScopeWriter` writes `batch-N.parquet.tmp`.
7. The temp file is atomically renamed to `batch-N.parquet`.
8. Manifest metadata for the new segment is persisted through the manifest service.
9. Flushed hot rows are deleted from RocksDB in bounded cleanup batches.

Notes:

- `STREAM` and `SYSTEM` tables are not part of this flush-to-Parquet path.
- `STORAGE FLUSH` is asynchronous; monitor progress via `system.jobs`.
- Shared scopes flush once per table scope. User scopes flush once per `(table, user)` scope.

## Flush State Machine

```mermaid
stateDiagram-v2
  [*] --> InSync
  InSync --> PendingWrite: DML write
  PendingWrite --> Syncing: flush starts
  Syncing --> InSync: parquet + manifest committed
  Syncing --> Error: flush failure
  Error --> PendingWrite: retry / next flush
```

## Post-Flush Small-Segment Compaction

KalamDB now has an optional post-flush tail-compaction path under `[flush.compaction]`:

```toml
[flush.compaction]
enabled = false
min_eligible_segments = 5
max_segments_per_run = 8
user_max_segment_rows = 10000
shared_max_segment_rows = 25000
```

This path is separate from the flush critical section. After a successful flush, the leader may
enqueue a `segment_compact` job for the scopes that actually wrote Parquet files.

The compactor:

1. walks the manifest from newest to oldest
2. selects only the trailing run of small, committed, same-schema segments
3. stops when it hits an already-large segment, a non-readable segment, or a schema-version
   boundary
4. rewrites only the latest MVCC winners into `compact-<uuid>.parquet`
5. swaps the manifest tail only if the selected suffix is still unchanged

If a newer flush changed the manifest tail while compaction was running, the compacted output is
discarded and the existing manifest remains authoritative.

Only `USER` and `SHARED` tables participate in this mechanism.

## Manual Flush & Compaction

```sql
-- Flush a specific table
STORAGE FLUSH TABLE myapp.messages;

-- Flush all tables in a namespace
STORAGE FLUSH ALL IN myapp;

-- Compact cold storage
STORAGE COMPACT TABLE myapp.messages;

-- Check storage health
STORAGE CHECK local EXTENDED;
```

## S3 User-Data Example

Create an S3 storage and bind a `USER` table to it:

For MinIO-compatible local S3 setup, see
[/docs/server/integrations/minio](https://kalamdb.org/docs/server/integrations/minio).

```sql
CREATE STORAGE s3_prod
  TYPE 's3'
  BUCKET 'my-kalamdb-prod'
  REGION 'us-east-1'
  USER_TABLES_TEMPLATE 'users/{namespace}/{tableName}/{userId}'
  SHARED_TABLES_TEMPLATE 'shared/{namespace}/{tableName}';

CREATE TABLE chat.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  conversation_id BIGINT NOT NULL,
  content TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (
  TYPE = 'USER',
  STORAGE_ID = 's3_prod',
  FLUSH_POLICY = 'rows:1000,interval:60'
);
```

For user `u_42`, cold-tier objects are stored under the user template:

```text
s3://my-kalamdb-prod/users/chat/messages/u_42/manifest.json
s3://my-kalamdb-prod/users/chat/messages/u_42/batch-0.parquet
s3://my-kalamdb-prod/users/chat/messages/u_42/batch-1.parquet
```

Illustrative manifest excerpt:

```json
{
  "segments": [
    {
      "id": "batch-1.parquet",
      "path": "batch-1.parquet",
      "row_count": 1000,
      "size_bytes": 184320,
      "schema_version": 1,
      "status": "committed"
    }
  ],
  "last_sequence_number": 1
}
```

## Per-User Storage Isolation

```text
data/storage/
├── <namespace>/<tableName>/manifest.json                   # shared table path template
├── <namespace>/<tableName>/batch-<index>.parquet
└── <namespace>/<tableName>/<userId>/manifest.json          # user table path template
```

Path layout is controlled by:

- `storage.shared_tables_template` (default: `{namespace}/{tableName}`)
- `storage.user_tables_template` (default: `{namespace}/{tableName}/{userId}`)

Each user's cold-tier data can still live in a separate directory. This enables:
- **Trivial data export** — just copy the user's directory
- **Instant deletion** — remove the directory for GDPR compliance
- **Independent scaling** — no cross-user interference

For the manifest-layer details behind this flow, see
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests).

## Related Docs

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests)
- [/docs/server/sql-reference/storage](https://kalamdb.org/docs/server/sql-reference/storage)
- [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage)
- [/docs/server/integrations/minio](https://kalamdb.org/docs/server/integrations/minio)

================================================================================
# Stream Storage
URL: https://kalamdb.org/docs/server/architecture/stream-storage
Source: content/server/architecture/stream-storage.mdx

> Learn how KalamDB stores STREAM table rows in TTL-sized window files, how eviction deletes expired directories, and why stream reads stay user-scoped.
# Stream Storage Architecture

KalamDB stream storage is the runtime path behind `STREAM` tables. It is optimized for short-lived,
user-scoped rows such as presence, typing indicators, cursor positions, and transient events.

Unlike `USER` and `SHARED` tables, stream rows do not flush into Parquet or participate in the
manifest-driven cold tier. The engine keeps stream data in append-only log files sized for fast TTL
eviction.

For the SQL creation model and table-type rules, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## Stream Storage File Layout

With the default data directory, each stream table gets a table-scoped base path under:

```text
data/streams/<namespace>/<table>
```

Inside that base path, rows are written into TTL-sized window directories:

```text
data/streams/<namespace>/<table>/w<window_start_ms>-<duration_ms>/<shard>/<user_id>.log
```

Example:

```text
data/streams/chat/typing_events/
├── w1748343000000-60000/
│   ├── shard-0/alice.log
│   └── shard-2/bob.log
└── w1748343060000-60000/
    └── shard-0/alice.log
```

The layout is intentionally simple:

- one immutable time window directory per retention bucket
- one shard subdirectory to distribute active writers
- one `<user_id>.log` file per user inside that shard/window

KalamDB uses `<user_id>.log` directly instead of `<user_id>/records.log` because the validated
`user_id` alphabet is already filename-safe, so the extra directory level would add metadata churn
without adding useful structure.

## Why the Window Name Encodes Time

Each window directory name encodes two values:

- `window_start_ms`
- `duration_ms`

That lets cleanup decide whether a whole window is expired without opening any log file.

If `window_end <= ttl_cutoff`, the entire directory can be removed in one operation. This avoids a
row scan during normal retention and keeps cleanup proportional to the number of expired windows,
not the number of rows.

## Bucket Sizing From `TTL_SECONDS`

The bucket size is derived from the table's `TTL_SECONDS` value:

| `TTL_SECONDS` | Window size |
|---|---|
| `<= 15 minutes` | minute windows |
| `<= 1 day` | hour windows |
| `<= 1 week` | day windows |
| `<= 30 days` | week windows |
| `> 30 days` | month windows |

Short-lived streams use minute windows so eviction stays close to the actual retention period.
Longer-lived streams use coarser windows to avoid creating too many tiny files.

## Write Path

When a row is inserted or deleted in a `STREAM` table:

1. the effective `user_id` selects the per-user stream scope
2. the row timestamp selects the current time window
3. the user is routed into a shard directory
4. the record is appended to `<user_id>.log` as a length-prefixed flexbuffers frame

The file-backed stream store keeps a bounded writer cache:

- 64 KiB buffered writes per open segment
- up to 256 cached segment writers per store
- cold writers flushed and closed when the cache exceeds the cap

The stream log does not call `fsync` on every append. Stream tables are intended for ephemeral
state, so the design favors low write latency and relies on the OS page cache for writeback.

## Stream Storage Cleanup

TTL eviction is window-based, not row-by-row.

The stream eviction job computes a cutoff timestamp and then:

1. scans window directory names under the table's stream base path
2. identifies windows whose computed `window_end` is older than the cutoff
3. closes any cached writers under those directories
4. removes the expired window directories

```mermaid
flowchart LR
  A["STREAM write"] --> B["Pick TTL-sized window"]
  B --> C["Route by shard + user_id"]
  C --> D["Append to <user_id>.log"]

  E["Eviction job"] --> F["Read window directory names"]
  F --> G{"window_end <= cutoff?"}
  G -- "yes" --> H["Close cached writers"]
  H --> I["remove_dir_all(window)"]
  G -- "no" --> J["Keep active window"]
```

This is why the time-window layout matters: cleanup is mostly directory unlink work instead of file
parsing or per-row filtering.

The pre-validation path also benefits from this layout. `has_logs_before()` can stop as soon as it
finds one expired window directory, and the job executor runs that filesystem work on Tokio's
blocking pool so async workers are not pinned by directory traversal or removal.

## Read Path

Reads remain user-scoped.

To serve a range or latest query, the stream store:

1. computes the shard for the target `user_id`
2. lists only that user's candidate window files
3. filters windows by overlap with the requested time range
4. reads only those candidate log files
5. applies delete records so latest state wins for each stream row ID

This keeps reads narrow even when many users share the same stream table.

## What Stream Storage Is Not

Stream storage is deliberately separate from the Parquet cold tier used by `USER` and `SHARED`
tables.

- no Parquet flush path
- no manifest-driven segment lifecycle
- no long-term analytic storage guarantees
- no file-column support

If you need durable historical records or analytics, use a `USER` or `SHARED` table and the cold
storage flow documented in [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers).

If you need short-lived realtime state that should expire automatically, `STREAM` storage is the
intended path.

## Related Docs

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)
- [/docs/server/topic-pubsub](https://kalamdb.org/docs/server/topic-pubsub)
- [/docs/server/sql-reference/tables](https://kalamdb.org/docs/server/sql-reference/tables)

================================================================================
# Table Types
URL: https://kalamdb.org/docs/server/architecture/table-types
Source: content/server/architecture/table-types.mdx

> When to use USER, SHARED, STREAM, and SYSTEM table types in KalamDB. Covers isolation behavior, access rules, and backend execution semantics.
# Table Types

KalamDB uses four runtime table types. The table type decides data scope, write path, storage
layout, live-query routing, and which role checks run before the server exposes rows.

- `USER`
- `SHARED`
- `STREAM`
- `SYSTEM` (internal)

## Table-Type Routing Model

```mermaid
flowchart LR
  A["CREATE TABLE / runtime lookup"] --> B{"Table type"}
  B -- "TYPE = 'USER'" --> C["USER scope"]
  B -- "TYPE = 'SHARED'" --> D["Shared table scope"]
  B -- "TYPE = 'STREAM'" --> E["Per-user stream scope"]
  B -- "Engine-created" --> F["SYSTEM scope"]

  C --> C1["Effective user_id selects tenant partition"]
  C1 --> C2["Hot RocksDB + optional Parquet flush"]

  D --> D1["ACCESS_LEVEL policy controls read/write roles"]
  D1 --> D2["One shared table scope + optional Parquet flush"]

  E --> E1["Effective user_id selects stream partition"]
  E1 --> E2["TTL eviction, no Parquet flush path"]

  F --> F1["Metadata and operational views"]
  F1 --> F2["DBA/System-only access"]
```

## Quick Selection

| Choose | When the data is | Typical examples |
|---|---|---|
| `USER` | Private to the authenticated or effective user | chat messages, agent memory, preferences, tenant records |
| `SHARED` | Global within the namespace and governed by table policy | feature flags, catalogs, plans, public reference data |
| `STREAM` | Short-lived and user-scoped | typing indicators, presence, cursor positions, transient events |
| `SYSTEM` | Engine metadata or runtime state | users, schemas, jobs, cluster views, transactions |

Application SQL can create `USER`, `SHARED`, and `STREAM` tables. `SYSTEM` tables are created and
evolved by the server.

For SQL syntax, see [/docs/server/sql-reference/tables](https://kalamdb.org/docs/server/sql-reference/tables).

## SQL Creation Patterns

Recommended unified syntax:

```sql
CREATE TABLE app.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  content TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (TYPE = 'USER');
```

Also supported for compatibility:

- `CREATE USER TABLE ...`
- `CREATE SHARED TABLE ...`
- `CREATE STREAM TABLE ...`

If neither a typed prefix nor `WITH (TYPE = ...)` is supplied, the parser currently defaults to a
`SHARED` table. Prefer setting `TYPE` explicitly in new application SQL so the data boundary is
obvious in reviews.

## USER Tables

Per-user logical isolation with user-aware routing and permissions.

```sql
CREATE TABLE chat.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  conversation_id BIGINT NOT NULL,
  content TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (
  TYPE = 'USER',
  FLUSH_POLICY = 'rows:1000,interval:60',
  STORAGE_ID = 'local'
);
```

Use for: conversations, preferences, private records, tenant data.

### USER Table Behavior

- Rows are scoped by the authenticated or effective `user_id` from the session.
- Reads do not implicitly cross user boundaries, even for `system`, `dba`, or `service` roles.
- Cross-user work must use an authorized impersonation flow such as `EXECUTE AS USER`.
- `USER` tables require a primary key.
- `STORAGE_ID`, `USE_USER_STORAGE`, and `FLUSH_POLICY` apply to this table type.
- Hot rows can flush into cold Parquet segments. The flush scope is `(table, user)`.
- `FILE` columns and file APIs are supported.
- Live queries are evaluated in the user-scoped partition.

### USER Role Access

| Operation | Anonymous | User | Service | DBA | System |
|---|---:|---:|---:|---:|---:|
| Read own/effective `USER` scope | No | Yes | Yes | Yes | Yes |
| Insert/update/delete own/effective `USER` scope | No | Yes | Yes | Yes | Yes |
| Create `USER` table | No | No | Yes | Yes | Yes |
| Alter `USER` table | No | No | Yes | Yes | Yes |
| Drop `USER` table | No | No | No | Yes | Yes |

## SHARED Tables

Global shared scope for all users.

```sql
CREATE TABLE app.config (
  key TEXT PRIMARY KEY,
  value TEXT NOT NULL,
  updated_at TIMESTAMP DEFAULT NOW()
) WITH (
  TYPE = 'SHARED',
  ACCESS_LEVEL = 'PRIVATE'
);
```

Use for: reference data, global settings, shared catalogs.

### SHARED Table Behavior

- Rows are not partitioned by `user_id`.
- Access is controlled by the table's `ACCESS_LEVEL` option.
- If `ACCESS_LEVEL` is omitted, the server stores `PRIVATE`.
- `SHARED` tables require a primary key.
- `STORAGE_ID`, `FLUSH_POLICY`, and `ACCESS_LEVEL` apply to this table type.
- `USE_USER_STORAGE` is rejected because shared data has no user-specific storage scope.
- Hot rows can flush into cold Parquet segments. The flush scope is the shared table scope.
- `FILE` columns and file APIs are supported.
- Live queries use a shared-table subscription path optimized for large fan-out.

### ACCESS_LEVEL Semantics

The SQL parser accepts `PUBLIC`, `PRIVATE`, `RESTRICTED`, and `DBA` for `ACCESS_LEVEL` on shared
tables. Values are case-insensitive. `PRIVATE` and `RESTRICTED` currently enforce the same runtime
role matrix; keep `RESTRICTED` for data you want to label as elevated service/admin-only access.

```mermaid
flowchart TD
  A["Request touches SHARED table"] --> B["Load table definition"]
  B --> C["Read ACCESS_LEVEL from table options"]
  C --> D{"Operation"}
  D -- "SELECT / live initial read" --> E["can_access_shared_table"]
  D -- "INSERT / UPDATE / DELETE" --> F["can_write_shared_table"]
  E --> G{"Role allowed?"}
  F --> G
  G -- "yes" --> H["Continue execution"]
  G -- "no" --> I["Access denied"]
```

| `ACCESS_LEVEL` | Read: Anonymous | Read: User | Read: Service | Read: DBA | Read: System | Write: User | Write: Service | Write: DBA | Write: System |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| `PUBLIC` | Yes | Yes | Yes | Yes | Yes | No | Yes | Yes | Yes |
| `PRIVATE` | No | No | Yes | Yes | Yes | No | Yes | Yes | Yes |
| `RESTRICTED` | No | No | Yes | Yes | Yes | No | Yes | Yes | Yes |
| `DBA` | No | No | No | Yes | Yes | No | No | Yes | Yes |

Notes:

- `PUBLIC` means readable by every session role, including anonymous session context, but regular
  `user` role sessions still cannot write.
- `PRIVATE` is the default and is service/admin read-write.
- `DBA` is the strictest shared-table option and excludes `service` roles.
- DDL still has its own gate: `service`, `dba`, and `system` can create or alter application
  tables; regular `user` sessions cannot create, alter, or drop tables.

### Changing Shared Access

Two ALTER forms are supported:

```sql
ALTER TABLE app.config SET TBLPROPERTIES (ACCESS_LEVEL = 'PUBLIC');
ALTER TABLE app.config SET ACCESS LEVEL public;
```

The `SET TBLPROPERTIES` form accepts `PUBLIC`, `PRIVATE`, `RESTRICTED`, and `DBA`. The shorthand
`SET ACCESS LEVEL ...` form currently accepts `PUBLIC`, `PRIVATE`, and `RESTRICTED`.

## STREAM Tables

Ephemeral rows with TTL-based eviction. Like `USER` tables, `STREAM` tables are scoped per user tenant and are **not** shared across all users.

```sql
CREATE TABLE chat.typing_events (
  event_id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  user_id TEXT NOT NULL,
  event_type TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (
  TYPE = 'STREAM',
  TTL_SECONDS = 30
);
```

Use for: typing indicators, presence, cursor events, transient signals.

### STREAM Table Behavior

- Rows are scoped by the authenticated or effective `user_id`.
- `TTL_SECONDS` is required in SQL creation.
- Stream rows are evicted by the stream lifecycle rather than flushed into Parquet cold storage.
- `STREAM` tables use the same read/write permission gate as `USER` tables.
- `FILE` columns and file APIs are not supported.
- Live queries are supported and are commonly used for transient UI state.

## Realtime Subscriptions

Today, KalamDB supports realtime subscriptions to all three primary application table types: `USER`, `SHARED`, and `STREAM`. This allows applications to listen for live changes (inserts, updates, deletes) as they happen, enabling reactive UIs and realtime workflows.

Subscriptions can be full-row (`SELECT *`) or projected (`SELECT username, status FROM chat.changes ...`) depending on the payload shape your client needs.

For the delivery path, shared-table fan-out, and reconnect semantics, see
[/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query).

## SYSTEM Tables

Internal control-plane and metadata tables (`system.*`).

- not intended for normal app DDL lifecycle
- role-restricted access for sensitive operations
- file upload/download flows reject `SYSTEM` tables
- direct DML against `system.*` tables is blocked for normal application workflows
- computed `system.*` views expose runtime metadata such as sessions, transactions, cluster state,
  and jobs

`SYSTEM` table access is restricted to `dba` and `system` roles. `service`, `user`, and anonymous
roles cannot read system tables through the normal table-access gate.

For inspectable system views, see [/docs/server/sql-reference/system-views](https://kalamdb.org/docs/server/sql-reference/system-views).

## Table Options By Type

| Option | `USER` | `SHARED` | `STREAM` | `SYSTEM` | Server behavior |
|---|---:|---:|---:|---:|---|
| `TYPE` | Yes | Yes | Yes | No app DDL | Selects table type when not using `CREATE USER/SHARED/STREAM TABLE`. |
| `STORAGE_ID` | Yes | Yes | Parsed but not used | Internal | Defaults to `local`; must reference an existing storage backend. |
| `USE_USER_STORAGE` | Yes | No | No | No | Valid only on `USER`; rejected on `SHARED`/`STREAM`. |
| `FLUSH_POLICY` | Yes | Yes | Parsed but no flush path | Internal | Triggers hot-to-cold movement for `USER` and `SHARED`. |
| `DELETED_RETENTION_HOURS` | Parsed | Parsed | Parsed | Internal | Parsed into the DDL statement; runtime deletion retention is handled by server lifecycle code. |
| `TTL_SECONDS` | No | No | Required | No | Valid only on `STREAM`; omitted TTL is rejected. |
| `ACCESS_LEVEL` | No | Yes | No | No | Valid only on `SHARED`; omitted value defaults to `PRIVATE`. |

For storage templates and `STORAGE_ID` examples, see
[/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage).

## DDL Role Matrix

| DDL operation | Anonymous | User | Service | DBA | System |
|---|---:|---:|---:|---:|---:|
| Create `USER` / `SHARED` / `STREAM` table | No | No | Yes | Yes | Yes |
| Create `SYSTEM` table | No | No | No | No | No via app SQL |
| Alter application table | No | No | Yes | Yes | Yes |
| Drop application table | No | No | No | Yes | Yes |
| Create view | No | No | No | Yes | Yes |

## Capability Matrix

| Capability | USER | SHARED | STREAM | SYSTEM |
|---|---|---|---|---|
| App DDL creation | Yes | Yes | Yes | Internal only |
| Per-user isolation | Yes | No | Yes | Internal |
| `ACCESS_LEVEL` option | No | Yes | No | No |
| `TTL_SECONDS` option | No | No | Required | No |
| `USE_USER_STORAGE` option | Yes | No | No | No |
| Cold-tier Parquet lifecycle | Yes | Yes | No | Internal-specific |
| File APIs supported | Yes | Yes | No | No |
| Live Query | Yes | Yes | Yes | No |
| Cluster multi-group Raft support | Yes | Yes | Yes | Internal |

### Capability Definitions

- **App DDL creation**: Whether application SQL can create/manage this table type directly.
- **Per-user isolation**: Whether data is automatically isolated by authenticated user/tenant.
- **`ACCESS_LEVEL` option**: Whether table-level access policy can be configured through `ACCESS_LEVEL`.
- **`TTL_SECONDS` option**: Whether the table supports TTL-based automatic row eviction.
- **`USE_USER_STORAGE` option**: Whether table storage can be explicitly pinned to user-scoped storage paths.
- **Cold-tier Parquet lifecycle**: Whether hot data can flush/compact into cold-tier Parquet segments.
- **File APIs supported**: Whether `/v1/files/...` upload/download flows are supported for rows in this table type.
- **Live Query**: Whether realtime subscriptions can stream change events from this table type.
- **Cluster multi-group Raft support**: Whether metadata/data operations for this table type participate in clustered Raft replication groups.

## Related Docs

- [/docs/server/sql-reference/tables](https://kalamdb.org/docs/server/sql-reference/tables)
- [/docs/server/sql-reference/dml](https://kalamdb.org/docs/server/sql-reference/dml)
- [/docs/server/sql-reference/impersonation](https://kalamdb.org/docs/server/sql-reference/impersonation)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)
- [/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype)
- [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage)

================================================================================
# Vector Search
URL: https://kalamdb.org/docs/server/architecture/vector-search
Source: content/server/architecture/vector-search.mdx

> Learn how to store embeddings, build cosine indexes, and run vector search in KalamDB with EMBEDDING columns and COSINE_DISTANCE queries.
# Vector Search

KalamDB supports vector search with native `EMBEDDING(n)` columns, cosine indexes, and SQL ranking via `COSINE_DISTANCE(...)`.

Use this when you need semantic retrieval for agent memory, RAG document recall, or similarity search over user-scoped data.

Vector search is table-type aware because embeddings live in normal KalamDB tables. Use
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types) to choose `USER` for
private semantic memory or `SHARED` for global catalogs.

## What Vector Search Looks Like

The current, tested flow in KalamDB is:

1. Create one or more `EMBEDDING(n)` columns.
2. Build a vector index on each embedding column with `ALTER TABLE ... CREATE INDEX ... USING COSINE`.
3. Query nearest rows with `ORDER BY COSINE_DISTANCE(...) LIMIT k`.

## Create a Vector Table

```sql
CREATE NAMESPACE IF NOT EXISTS rag;

CREATE TABLE rag.documents_vectors (
  id BIGINT PRIMARY KEY,
  doc_embedding EMBEDDING(3),
  attachment_a_embedding EMBEDDING(3),
  attachment_b_embedding EMBEDDING(3)
) WITH (TYPE = 'USER');
```

Why this shape works well:

- Keep document metadata and files in your main table.
- Keep embeddings in a parallel table keyed by the same `id`.
- Use `TYPE = 'USER'` when each signed-in user should search only their own vectors.

## Build Cosine Indexes

```sql
ALTER TABLE rag.documents_vectors
  CREATE INDEX doc_embedding USING COSINE;

ALTER TABLE rag.documents_vectors
  CREATE INDEX attachment_a_embedding USING COSINE;

ALTER TABLE rag.documents_vectors
  CREATE INDEX attachment_b_embedding USING COSINE;
```

KalamDB currently documents and tests cosine similarity indexes. If you create multiple embedding columns, index each column you plan to search independently.

## Insert Embeddings

Embeddings are inserted as JSON-like numeric arrays in SQL strings:

```sql
INSERT INTO rag.documents_vectors (
  id,
  doc_embedding,
  attachment_a_embedding,
  attachment_b_embedding
) VALUES (
  1,
  '[1.0,0.0,0.0]',
  '[0.95,0.05,0.0]',
  '[0.90,0.10,0.0]'
);
```

Make sure the vector length matches the `EMBEDDING(n)` dimension declared in the schema.

## Run a Similarity Query

```sql
SELECT id
FROM rag.documents_vectors
ORDER BY COSINE_DISTANCE(doc_embedding, '[1.0,0.0,0.0]')
LIMIT 3;
```

Smaller `COSINE_DISTANCE(...)` values are more similar, so the nearest matches appear first.

Pass embeddings as a JSON array literal in the second argument (for example `'[1.0,0.0,0.0]'`).
That is the supported KalamDB pattern for `EMBEDDING(n)` columns. When both arguments are
array/list values, KalamDB can also use DataFusion 54's native `cosine_distance` implementation.

You can search any indexed embedding column:

```sql
SELECT id
FROM rag.documents_vectors
ORDER BY COSINE_DISTANCE(attachment_b_embedding, '[0.0,1.0,0.0]')
LIMIT 2;
```

## Join Search Results Back to Documents

In most apps, vector search is only the retrieval step. After ranking the embedding rows, join back to your main document table:

```sql
SELECT d.id, d.title, d.body
FROM rag.documents AS d
JOIN rag.documents_vectors AS v ON v.id = d.id
ORDER BY COSINE_DISTANCE(v.doc_embedding, '[1.0,0.0,0.0]')
LIMIT 5;
```

That pattern lets you keep rich rows, FILE attachments, and embeddings separate while still querying them together.

## RAG Pattern With Files

One tested KalamDB scenario stores:

- a USER-scoped documents table with two `FILE` columns
- a USER-scoped vectors table keyed by the same `id`
- vector indexes persisted across flushes
- similarity queries that continue working across hot and cold storage

This is a good fit for:

- private agent memory per user
- semantic search over uploaded files
- document recall before LLM summarization or answer generation

## Practical Notes

- Use `TYPE = 'USER'` for tenant-safe semantic search.
- Keep embedding generation in your application or worker pipeline; KalamDB stores and indexes the vectors.
- If you update or delete rows, the vector search results follow the table state.
- Vector search continues to work when some rows are still hot and others have already flushed to cold storage.

## Related Docs

- [/docs/server/architecture/datatypes](https://kalamdb.org/docs/server/architecture/datatypes)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/sql-reference/tables](https://kalamdb.org/docs/server/sql-reference/tables)
- [/docs/server/sql-reference/functions](https://kalamdb.org/docs/server/sql-reference/functions)
- [/docs/ts-sdk/examples](https://kalamdb.org/docs/ts-sdk/examples)

================================================================================
# Auto-Provisioning Users
URL: https://kalamdb.org/docs/server/auth/auto-provisioning
Source: content/server/auth/auto-provisioning.mdx

> Understand when regular OIDC users are rowless, when explicit persisted users are required, and how OIDC subjects become KalamDB user IDs.
# Auto-Provisioning Users

KalamDB's current OIDC model has two distinct paths:

- regular external users can authenticate directly from validated token claims
- elevated or explicit external users are stored locally with `issuer` and `subject`

That means auto-provisioning is not just "create a row on first login" anymore. For the common `user` role path, KalamDB can stay stateless after checking whether a local row already exists for the OIDC subject.

## Enable Auto-Provisioning

```toml
[auth]
jwt_trusted_issuers = "kalamdb,https://idp.example.com/realms/kalamdb"

[auth.oidc]
enabled = true
issuer = "https://idp.example.com/realms/kalamdb"
client_id = "kalamdb"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
```

Environment variable form:

```bash

```

If `auth.oidc.auto_provision = false`, authentication succeeds only when a matching local OIDC user already exists.

## What Happens For Regular Users

When these conditions are both true:

- `auth.oidc.auto_provision = true`
- `auth.oidc.default_role = "user"`

KalamDB uses the OIDC `sub` claim directly as the KalamDB `user_id`.

Regular users resolve from validated token claims when no persisted `system.users` row exists for that subject. KalamDB checks the row first so explicit persisted OIDC users, deleted users, and same-ID local password users keep their local policy.

## What Still Gets Persisted

Persisted local state is still important for:

- elevated users such as `service`, `dba`, or `system`
- soft-deleted external users that must remain blocked
- deployments with `auto_provision = false`
- deployments that intentionally use a default role other than `user`

Persisted OIDC users store only this identity binding:

```json
{
  "issuer": "https://idp.example.com/realms/kalamdb",
  "subject": "provider-subject"
}
```

For persisted OIDC users, `subject` must match `system.users.user_id`.

## Pre-Create Or Elevate An OIDC User

```sql
CREATE USER 'provider-subject'
  WITH OIDC '{"issuer":"https://idp.example.com/realms/kalamdb","subject":"provider-subject"}'
  ROLE dba
  EMAIL 'alice@example.com';
```

Use explicit provisioning when you need:

- pre-approved access only
- an elevated role before first login
- no end-user auto-provisioning in production

## What Happens When The User Does Not Exist

If `auth.oidc.auto_provision = false`, the first request for an unknown external identity fails even when the provider token itself is valid.

That is expected. Token validation and user mapping are separate steps.

## Identity Stability Rules

KalamDB uses the token subject as the user ID, not the email address and not a provider code.

That means:

- changing email at the provider does not change the KalamDB user ID
- a different issuer or provider may emit a different subject for the same person
- deleting and recreating a provider-side account can create a new subject and therefore a new KalamDB user ID

## Recommended Rollout Pattern

1. Enable trusted issuers.
2. Turn on `auth.oidc.auto_provision = true` for normal end users.
3. Keep elevated roles explicit.
4. Check the created identity with `GET /v1/api/auth/me` or `SELECT CURRENT_USER();`.
5. Promote only the accounts that need broader access.

## Related

- [Authentication Overview](https://kalamdb.org/docs/server/auth)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Single OIDC Provider Model](https://kalamdb.org/docs/server/auth/providers)
- [User Management](https://kalamdb.org/docs/server/sql-reference/users)

================================================================================
# Authentication Providers
URL: https://kalamdb.org/docs/server/auth
Source: content/server/auth/index.mdx

> Authentication model overview for KalamDB. Learn how local auth and the single external OIDC provider fit together, how tokens are routed, and how OIDC subjects become user IDs.
# Authentication Overview

KalamDB has one authentication surface with two modes:

- local username/password login through `[auth.local]`
- one external OpenID Connect provider through `[auth.oidc]`

The same API and SDK surfaces work for both:

- `POST /v1/api/auth/login`
- `POST /v1/api/auth/refresh`
- `GET /v1/api/auth/me`
- `GET /v1/api/auth/login-options`
- bearer auth for `/v1/api/sql`, files, topics, and WebSocket traffic

## Authentication Methods

| Method | Header | Description |
|---|---|---|
| Basic Auth | `Authorization: Basic <base64(user:pass)>` | Username + password login, usually exchanged for JWT |
| Bearer (Internal JWT) | `Authorization: Bearer <token>` | KalamDB-issued `HS256` token |
| Bearer (External OIDC) | `Authorization: Bearer <id-token>` | Token from the configured trusted OIDC issuer |

All bearer tokens go through the same validation pipeline. KalamDB routes internal and external tokens automatically from the JWT algorithm and issuer.

## Bearer Token Routing

```
Client                   Identity Provider            KalamDB
  │                            │                          │
  │── sign in ────────────────►│                          │
  │◄── ID token (RS256) ────────│                          │
  │                            │                          │
  │── Authorization: Bearer <id-token> ─────────────────►│
  │                            │  1. Check iss is trusted │
  │                            │  2. OIDC discovery       │
  │                            │  3. Fetch + cache JWKS   │
  │                            │  4. Verify RS256 sig     │
  │                            │  5. Resolve/provision user│
  │◄── SQL response ────────────────────────────────────── │
```

KalamDB inspects the token before verification:
1. **Algorithm** — `HS256` → internal shared-secret validation; `RS256/ES256/…` → external OIDC flow.
2. **Issuer (`iss`)** — must appear in `jwt_trusted_issuers`; untrusted issuers are rejected before any network I/O.
3. **JWKS** — keys are fetched on first use and cached per-issuer; key rotation is handled automatically.
4. **Audience (`aud`)** — validated against `auth.oidc.audience` or `auth.oidc.client_id` when configured.

## Single External OIDC Provider

KalamDB no longer uses a built-in provider enum or a provider matrix. The active external identity model is issuer-based:

```toml
[auth]
jwt_trusted_issuers = "kalamdb,https://idp.example.com/realms/kalamdb"

[auth.oidc]
enabled = true
display_name = "Company SSO"
issuer = "https://idp.example.com/realms/kalamdb"
client_id = "kalamdb"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
```

Examples of issuers that fit this model:

- Dex
- Keycloak
- Okta
- Auth0
- Entra ID
- Google
- Firebase

The important point is that one KalamDB server process supports one configured external OIDC issuer.

## Identity Mapping

External users use the OIDC `sub` claim directly as the KalamDB `user_id`. There is no provider code, username prefix, or generated hash ID in the active model.

The same subject value appears in SQL identity, `system.users.user_id`, and PG extension `EXECUTE AS USER '<user_id>'` workflows. The subject must be a valid KalamDB user ID: ASCII letters, digits, `_`, or `-`, up to 128 characters.

When KalamDB persists a local override or explicit OIDC user, it stores only this provider identity data:

```json
{
  "issuer": "https://idp.example.com/realms/kalamdb",
  "subject": "provider-subject"
}
```

## Auto-Provision vs Explicit Users

When `auth.oidc.auto_provision = true` and `auth.oidc.default_role = "user"`, regular external users authenticate as role `user` without a per-user row on the hot path.

KalamDB checks for an existing `system.users` row with `user_id = sub` first. That keeps explicit persisted OIDC users, deleted users, and same-ID local password users from being bypassed.

Elevated users still need explicit persisted overrides. Use this when a user must be `service`, `dba`, or `system`:

```sql
CREATE USER 'provider-subject'
  WITH OIDC '{"issuer":"https://idp.example.com/realms/kalamdb","subject":"provider-subject"}'
  ROLE dba
  EMAIL 'alice@example.com';
```

## Browser And Device Login Surfaces

The Admin UI and CLI both discover auth capabilities from `GET /v1/api/auth/login-options`.

Current user-facing OIDC flows:

- Admin UI browser login through `/v1/api/auth/oidc/exchange-code`
- CLI browser login with `kalam login --oidc`
- CLI direct device login with `kalam login --oidc --no-browser`
- CLI brokered device login with `kalam login --oidc --no-browser --brokered`

## Role Management

Roles remain the same regardless of auth source:

- `user`: normal app or tenant user
- `service`: workers, automation, agents
- `dba`: database administration
- `system`: highest-privilege internal role

## Supported Algorithms

| Algorithm | Supported |
|---|---|
| HS256 | ✅ Internal tokens only |
| RS256 | ✅ |
| RS384 | ✅ |
| RS512 | ✅ |
| PS256 | ✅ |
| PS384 | ✅ |
| PS512 | ✅ |
| ES256 | ✅ |
| ES384 | ✅ |
| ES512 | ❌ Not supported |

## Related

- [Single OIDC Provider Model](https://kalamdb.org/docs/server/auth/providers)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)
- [Auto-Provisioning Users](https://kalamdb.org/docs/server/auth/auto-provisioning)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Dex](https://kalamdb.org/docs/server/integrations/dex)

================================================================================
# Single OIDC Provider Model
URL: https://kalamdb.org/docs/server/auth/providers
Source: content/server/auth/providers.mdx

> Understand KalamDB's current OIDC identity model: one configured issuer, direct subject user IDs, and no provider-family enum.
# Single OIDC Provider Model

KalamDB's active external auth model is issuer-based, not provider-family-based.

That means:

- one KalamDB server process supports one configured external OIDC provider
- the provider is identified by `auth.oidc.issuer` and `auth.oidc.client_id`
- there is no built-in provider enum, provider prefix table, or provider-specific username format in the active model

Dex, Keycloak, Okta, Auth0, Entra ID, Google, Firebase, and similar services all fit this same shape as long as they expose standards-compliant OIDC discovery.

## The Current `server.toml` Shape

```toml
[auth]
jwt_trusted_issuers = "kalamdb,https://idp.example.com/realms/kalamdb"

[auth.oidc]
enabled = true
display_name = "Company SSO"
issuer = "https://idp.example.com/realms/kalamdb"
client_id = "kalamdb"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
```

## No Provider Matrix

The current docs and backend surface should not talk about:

- `[oauth]`
- `[oauth.providers.*]`
- provider name lists such as `keycloak`, `firebase`, `auth0`, and so on
- provider-code username formats such as `oidc:kcl:<subject>`

Those were part of an older design. The active model is a single configured OIDC issuer.

## How External Users Are Identified

External users use the provider `sub` claim directly as the KalamDB `user_id`. The same value is used by SQL identity, `system.users.user_id`, and PG extension `EXECUTE AS USER '<user_id>'`.

Subjects must be valid KalamDB user IDs: ASCII letters, digits, `_`, or `-`, up to 128 characters. If the provider emits unsafe subject characters, normalize that at the IdP layer before using it with KalamDB.

Persisted OIDC users store the issuer and subject, and the `subject` must match the row's `user_id`:

```json
{
	"issuer": "https://idp.example.com/realms/kalamdb",
	"subject": "provider-subject"
}
```

## Auto-Provision And Explicit Users

Regular users can authenticate without a per-user row when:

- `auth.oidc.auto_provision = true`
- `auth.oidc.default_role = "user"`

Elevated roles should be explicit. Example:

```sql
CREATE USER 'provider-subject'
  WITH OIDC '{"issuer":"https://idp.example.com/realms/kalamdb","subject":"provider-subject"}'
  ROLE dba
  EMAIL 'alice@example.com';
```

## Related

- [Authentication Overview](https://kalamdb.org/docs/server/auth)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)
- [Auto-Provisioning Users](https://kalamdb.org/docs/server/auth/auto-provisioning)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Dex](https://kalamdb.org/docs/server/integrations/dex)

================================================================================
# Token Auth
URL: https://kalamdb.org/docs/server/auth/token-auth
Source: content/server/auth/token-auth.mdx

> Set up token authentication in KalamDB. Covers local login, OIDC exchange flows, direct external bearer tokens, refresh, and SDK token-provider patterns.
# Token Auth

Use token auth when your app should send `Authorization: Bearer <token>` on every request instead of raw user/password credentials.

KalamDB supports two token sources:

- KalamDB-issued access and refresh tokens returned by local login or OIDC exchange endpoints
- direct external OIDC ID tokens from the configured trusted issuer

## Internal Token Flow

The built-in login flow is:

1. Send user and password to `POST /v1/api/auth/login`.
2. Receive `access_token` and `refresh_token`.
3. Send the access token on API and WebSocket requests.
4. Refresh it through `POST /v1/api/auth/refresh` when needed.

### Login

```bash
curl -X POST http://127.0.0.1:2900/v1/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!"}'
```

Typical response fields:

- `access_token`
- `refresh_token`
- `expires_at` or token-expiry metadata
- authenticated user profile fields

### Use the access token

```bash
TOKEN="<access_token>"

curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"sql":"SELECT CURRENT_USER();"}'
```

## Refresh Tokens

```bash
REFRESH="<refresh_token>"

curl -X POST http://127.0.0.1:2900/v1/api/auth/refresh \
  -H "Authorization: Bearer $REFRESH"
```

Use the returned access token for subsequent requests.

## Checking Auth State

There are two practical checks:

### Check the current authenticated user

```bash
curl http://127.0.0.1:2900/v1/api/auth/me \
  -H "Authorization: Bearer $TOKEN"
```

This confirms whether the token is accepted and shows the current user record.

### Check from SQL

```bash
curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"sql":"SELECT CURRENT_USER();"}'
```

This is the fastest way to confirm the runtime identity that SQL sees.

## OIDC Browser And Device Exchange Flows

The Admin UI and CLI normally exchange provider login results for KalamDB-issued tokens.

Endpoints:

- `POST /v1/api/auth/oidc/exchange-code` for browser Authorization Code + PKCE
- `POST /v1/api/auth/oidc/exchange-token` for direct device login when the CLI obtains a provider ID token
- `POST /v1/api/auth/oidc/device/start` and `POST /v1/api/auth/oidc/device/poll` for brokered device flow

CLI entry points:

```bash
# Browser PKCE login
kalam login --instance local --url http://127.0.0.1:2900 --oidc

# Direct device login
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser

# Brokered device login
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser --brokered
```

All three flows return or save KalamDB access and refresh tokens. They do not persist provider credentials.

## Direct External Bearer Tokens

If you already have a valid token from the configured trusted OIDC issuer, you can also skip the exchange endpoints and send that provider token directly as a bearer token.

KalamDB will:

1. read `alg` and `iss`
2. reject untrusted issuers
3. perform OIDC discovery for the issuer
4. fetch and cache the issuer JWKS
5. validate the token signature and claims
6. resolve or create the matching local user

OIDC setup details live in:

- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Single OIDC Provider Model](https://kalamdb.org/docs/server/auth/providers)
- [Dex](https://kalamdb.org/docs/server/integrations/dex)

## SDK Pattern

For long-lived apps, return a fresh token from the SDK auth provider callback instead of hard-coding a token once.

### TypeScript

```ts

const authProvider: AuthProvider = async () => {
  const token = await myApp.getOrRefreshJwt();
  return Auth.jwt(token);
};

const client = createClient({
  url: 'https://db.example.com',
  authProvider,
});
```

### Dart

```dart
final client = await KalamClient.connect(
  url: 'https://db.example.com',
  authProvider: () async {
    final token = await myApp.getOrRefreshJwt();
    return Auth.jwt(token);
  },
);
```

## Configuration Notes

Internal token auth still depends on a strong shared secret:

```toml
[auth]
jwt_secret = "replace-with-strong-random-secret-32-plus-chars"
cookie_secure = true
```

OIDC also depends on the configured single provider block:

```toml
[auth.oidc]
enabled = true
issuer = "https://idp.example.com/realms/kalamdb"
client_id = "kalamdb"
scopes = ["openid", "email", "profile"]
```

Use `KALAMDB_JWT_SECRET` and the `KALAMDB_AUTH_OIDC_*` env vars in production instead of committing secrets to `server.toml`.

## Related

- [Authentication Overview](https://kalamdb.org/docs/server/auth)
- [Auto-Provisioning Users](https://kalamdb.org/docs/server/auth/auto-provisioning)
- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)
- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)

================================================================================
# Benchmarking
URL: https://kalamdb.org/docs/server/benchmarking
Source: content/server/benchmarking/index.mdx

> Static KalamDB benchmark report links synced from benchv2/results, with one HTML benchmark report listed for each tracked version.
# Benchmarking

This page publishes the newest benchmark HTML report for each KalamDB version currently synced from the `benchv2/results` folder in the main repository.

The June 17 [`v0.5.3-rc.1` pre-release](https://kalamdb.org/docs/server/release-v0-5-3-rc-1) is the newest synced run: **38/38 benchmarks passed** with **213,667.71 ms measured benchmark time**, an **8.0% improvement** over `0.5.2-rc.1`. Open the [exact HTML benchmark artifact](https://kalamdb.org/benchmark/results/bench-2026-06-17-131609-0-5-3-rc-1.html) or read the [release blog post](https://kalamdb.org/blog/kalamdb-v0-5-3-rc-1-pre-release).

## What The Table Shows

- `Version` is read from the synced benchmark metadata when available.
- `Run at` is the benchmark timestamp associated with the synced report.
- `Passed/Total` comes from the synced benchmark summary.
- `Duration` is the displayed benchmark runtime for that report, preferring measured time when newer reports provide it.
- `Delta` compares each row's displayed duration against the next older listed run, with an arrow and color showing whether it got faster or slower.
- `HTML report` opens the static benchmark report directly.

================================================================================
# Advanced Configuration
URL: https://kalamdb.org/docs/server/configurations/advanced
Source: content/server/configurations/advanced.mdx

> User-friendly guide to every KalamDB configuration option, with plain-English explanations, practical tuning advice, and production-ready examples.
# Advanced Configuration

This page is the canonical public reference for every supported `server.toml` key and every documented `KALAMDB_*` runtime override.

Use it when you need exact key names, actual defaults, or a clear answer to whether a setting is configured through TOML, an environment variable, or both.

## Before You Start

- If you are new to setup, start with
  [/docs/server/getting-started/configuration](https://kalamdb.org/docs/server/getting-started/configuration).
- If you want the fully commented sample file, download [server.example.toml (full commented sample)](https://raw.githubusercontent.com/kalamdb/KalamDB/refs/heads/main/backend/server.example.toml).
- The tables below show the real defaults from the config crate. The checked-in sample config may intentionally use different values for local development, CI, or demo environments.

Focused guides:

- [/docs/server/configurations/logging](https://kalamdb.org/docs/server/configurations/logging)
- [/docs/server/configurations/otel](https://kalamdb.org/docs/server/configurations/otel)
- [/docs/server/configurations/oidc](https://kalamdb.org/docs/server/configurations/oidc)
- [/docs/server/integrations/minio](https://kalamdb.org/docs/server/integrations/minio)
- [/docs/server/integrations/keycloak](https://kalamdb.org/docs/server/integrations/keycloak)
- [/docs/server/integrations/jaeger](https://kalamdb.org/docs/server/integrations/jaeger)

## Loading Order

1. Load `server.toml`.
2. Apply config-loader environment overrides from `KALAMDB_*` variables.
3. Apply runtime-only environment behavior that lives outside the config crate.
4. Normalize paths and validate the final configuration.

Runtime-only env vars are currently:

- `KALAMDB_ROOT_PASSWORD` for root bootstrap password selection during startup.
- `KALAMDB_TOKIO_WORKER_THREADS` for overriding Tokio runtime worker threads.

## Environment Override Reference

Only the variables below are supported today. If a setting is not listed here, configure it in `server.toml`.

Boolean parsing notes:

- Most positive boolean env vars treat `true`, `1`, and `yes` as enabled.
- `KALAMDB_COOKIE_SECURE` is inverted for convenience: `false`, `0`, and `no` disable it; any other non-empty value enables it.

List parsing notes:

- `KALAMDB_SECURITY_CORS_ALLOWED_ORIGINS` and `KALAMDB_SECURITY_TRUSTED_PROXY_RANGES` accept comma-separated values.
- `KALAMDB_CLUSTER_PEERS` accepts `node_id@rpc_addr@api_addr[@rpc_server_name]` entries separated by semicolons.

### Server, Logging, and Storage

| Environment variable | Maps to | Notes |
|---|---|---|
| `KALAMDB_SERVER_HOST` | `server.host` | Bind host/interface. |
| `KALAMDB_SERVER_PORT` | `server.port` | HTTP listen port. |
| `KALAMDB_SERVER_PUBLIC_ORIGIN` | `server.public_origin` | Empty string preserves browser-origin fallback behavior. |
| `KALAMDB_SERVER_WORKERS` | `server.workers` | Actix worker threads. |
| `KALAMDB_LOG_LEVEL` | `logging.level` | `error`, `warn`, `info`, `debug`, `trace`. |
| `KALAMDB_LOG_FORMAT` | `logging.format` | `compact`, `pretty`, or `json`. |
| `KALAMDB_LOGS_DIR` | `logging.logs_path` | Directory for `server.log`, `server.jsonl`, and `slow.log`. |
| `KALAMDB_LOG_TO_CONSOLE` | `logging.log_to_console` | Also emit logs to stdout/stderr. |
| `KALAMDB_SLOW_QUERY_THRESHOLD_MS` | `logging.slow_query_threshold_ms` | Slow query threshold in milliseconds. |
| `KALAMDB_OTLP_ENABLED` | `logging.otlp.enabled` | Enable OTLP trace export. |
| `KALAMDB_OTLP_ENDPOINT` | `logging.otlp.endpoint` | Collector endpoint. |
| `KALAMDB_OTLP_PROTOCOL` | `logging.otlp.protocol` | `grpc` or `http`. |
| `KALAMDB_OTLP_SERVICE_NAME` | `logging.otlp.service_name` | Service name shown by the collector/backend. |
| `KALAMDB_OTLP_TIMEOUT_MS` | `logging.otlp.timeout_ms` | OTLP export timeout. |
| `KALAMDB_DATA_DIR` | `storage.data_path` | Base data directory. |

### Topics, Auth, Security, and WebSockets

| Environment variable | Maps to | Notes |
|---|---|---|
| `KALAMDB_TOPIC_VISIBILITY_TIMEOUT_SECS` | `topics.visibility_timeout_secs` | Canonical topic visibility timeout override. |
| `KALAMDB_VISIBILITY_TIMEOUT_SECS` | `topics.visibility_timeout_secs` | Legacy compatibility alias. |
| `KALAMDB_TOPIC_DEFAULT_RETENTION_SECONDS` | `topics.default_retention_seconds` | Default topic time retention. |
| `KALAMDB_TOPIC_DEFAULT_RETENTION_MAX_BYTES` | `topics.default_retention_max_bytes` | Default topic byte retention. |
| `KALAMDB_TOPIC_RETENTION_CHECK_INTERVAL_SECONDS` | `topics.retention_check_interval_seconds` | Scheduler interval. |
| `KALAMDB_TOPIC_RETENTION_BATCH_SIZE` | `topics.retention_batch_size` | Delete batch size. |
| `KALAMDB_JWT_SECRET` | `auth.jwt_secret` | Internal JWT signing/verification secret. |
| `KALAMDB_JWT_TRUSTED_ISSUERS` | `auth.jwt_trusted_issuers` | Comma-separated issuer list. |
| `KALAMDB_JWT_EXPIRY_HOURS` | `auth.jwt_expiry_hours` | Internal token TTL. |
| `KALAMDB_COOKIE_SECURE` | `auth.cookie_secure` | HTTPS-only auth cookies. |
| `KALAMDB_ALLOW_REMOTE_SETUP` | `auth.allow_remote_setup` | Enables first-time setup from non-localhost clients. |
| `KALAMDB_PG_AUTH_TOKEN` | `auth.pg_auth_token` | Pre-shared token for `pg_kalam` gRPC auth. |
| `KALAMDB_AUTH_OIDC_AUTO_PROVISION` | `auth.oidc.auto_provision` | OIDC auto-provisioning override. |
| `KALAMDB_SECURITY_CORS_ALLOWED_ORIGINS` | `security.cors.allowed_origins` | Comma-separated list or `*`. |
| `KALAMDB_SECURITY_TRUSTED_PROXY_RANGES` | `security.trusted_proxy_ranges` | Canonical proxy trust override. |
| `KALAMDB_TRUSTED_PROXY_RANGES` | `security.trusted_proxy_ranges` | Legacy env alias kept for compatibility. |
| `KALAMDB_RATE_LIMIT_AUTH_REQUESTS_PER_IP_PER_SEC` | `rate_limit.max_auth_requests_per_ip_per_sec` | Brute-force protection rate limit. |
| `KALAMDB_WEBSOCKET_CLIENT_TIMEOUT_SECS` | `websocket.client_timeout_secs` | Client heartbeat timeout. |
| `KALAMDB_WEBSOCKET_AUTH_TIMEOUT_SECS` | `websocket.auth_timeout_secs` | Time allowed before auth message arrives. |
| `KALAMDB_WEBSOCKET_HEARTBEAT_INTERVAL_SECS` | `websocket.heartbeat_interval_secs` | Heartbeat scan interval. |

### Cluster and RPC TLS

| Environment variable | Maps to | Notes |
|---|---|---|
| `KALAMDB_CLUSTER_ID` | `cluster.cluster_id` | Creates the cluster section if needed. |
| `KALAMDB_NODE_ID` | `cluster.node_id` | Canonical node ID override. |
| `KALAMDB_CLUSTER_NODE_ID` | `cluster.node_id` | Legacy alias for `KALAMDB_NODE_ID`. |
| `KALAMDB_CLUSTER_RPC_ADDR` | `cluster.rpc_addr` | Advertised Raft RPC address. |
| `KALAMDB_CLUSTER_API_ADDR` | `cluster.api_addr` | Advertised API address. |
| `KALAMDB_CLUSTER_PEERS` | `cluster.peers` | `node_id@rpc_addr@api_addr[@rpc_server_name];...` |
| `KALAMDB_RPC_TLS_ENABLED` | `rpc_tls.enabled` | Enable TLS/mTLS on the gRPC listener. |
| `KALAMDB_RPC_TLS_CA_CERT` | `rpc_tls.ca_cert` | File path or inline PEM. |
| `KALAMDB_RPC_TLS_SERVER_CERT` | `rpc_tls.server_cert` | File path or inline PEM. |
| `KALAMDB_RPC_TLS_SERVER_KEY` | `rpc_tls.server_key` | File path or inline PEM. |
| `KALAMDB_RPC_TLS_REQUIRE_CLIENT_CERT` | `rpc_tls.require_client_cert` | Require client certs signed by `ca_cert`. |

### Runtime-Only Environment Variables

| Environment variable | Effect | Notes |
|---|---|---|
| `KALAMDB_ROOT_PASSWORD` | Sets the initial root password during startup bootstrap. | This does not come from `apply_env_overrides`; it is read in server lifecycle startup. It takes precedence over `auth.root_password`. |
| `KALAMDB_TOKIO_WORKER_THREADS` | Overrides Tokio runtime worker thread count. | This is read in the server binary and takes precedence over `performance.tokio_worker_threads`. |

## Full `server.toml` Reference

### Top-Level Keys

| Key | Default | Description |
|---|---|---|
| `transaction_timeout_secs` | `300` | Maximum lifetime of an open transaction before the server aborts it. |
| `max_transaction_buffer_bytes` | `104857600` (100 MB) | Maximum in-memory transaction overlay size before new writes are rejected. |

### `[server]`

| Key | Default | Description |
|---|---|---|
| `host` | `"127.0.0.1"` | Network interface to bind. Use `0.0.0.0` only with explicit CORS/origin policy. |
| `port` | `2900` | HTTP port for REST, SQL, auth, and WebSocket entrypoints. |
| `public_origin` | unset | Public browser-facing origin used by the Admin UI for `/v1/api` and `/v1/ws`. |
| `workers` | `0` | Actix worker threads. `0` means automatic sizing. |
| `api_version` | `"v1"` | Version prefix for API routes. |
| `enable_http2` | `true` | Enables HTTP/1.1 + cleartext HTTP/2 negotiation. |
| `ui_path` | unset | Path to a built Admin UI bundle to serve from the server. |

### `[storage]`

| Key | Default | Description |
|---|---|---|
| `data_path` | `"./data"` | Base path for RocksDB, Parquet storage, snapshots, streams, exports, and temp files. |
| `shared_tables_template` | `"{namespace}/{tableName}"` | Folder template for shared tables. |
| `user_tables_template` | `"{namespace}/{tableName}/{userId}"` | Folder template for user-scoped tables. |

#### `[storage.remote_timeouts]`

| Key | Default | Description |
|---|---|---|
| `request_timeout_secs` | `60` | Request timeout for remote storage backends. |
| `connect_timeout_secs` | `10` | Connection timeout for remote storage backends. |

#### `[storage.rocksdb]`

| Key | Default | Description |
|---|---|---|
| `block_cache_size` | `2097152` (2 MB) | Shared RocksDB read block cache across all column families. |
| `max_background_jobs` | `4` | Background compaction and flush worker count. |
| `max_open_files` | `512` | Maximum open files kept by RocksDB. `-1` means unlimited. |
| `sync_writes` | `false` | Fsync WAL on each write for stronger durability and lower throughput. |
| `disable_wal` | `false` | Disables WAL for speed at the cost of crash recovery guarantees. |
| `compact_on_startup` | `false` | Compact column families during startup to reduce SST spread. Disabled by default to keep startup fast. |

#### `[storage.rocksdb.cf_profiles.system_meta]`

| Key | Default | Description |
|---|---|---|
| `write_buffer_size` | `32768` | Write buffer size for system metadata CFs. |
| `max_write_buffers` | `2` | Max RocksDB memtables for system metadata CFs. |

#### `[storage.rocksdb.cf_profiles.system_index]`

| Key | Default | Description |
|---|---|---|
| `write_buffer_size` | `32768` | Write buffer size for system index CFs. |
| `max_write_buffers` | `2` | Max RocksDB memtables for system index CFs. |

#### `[storage.rocksdb.cf_profiles.hot_data]`

| Key | Default | Description |
|---|---|---|
| `write_buffer_size` | `131072` | Write buffer size for hot data CFs. |
| `max_write_buffers` | `2` | Max RocksDB memtables for hot data CFs. |

#### `[storage.rocksdb.cf_profiles.hot_index]`

| Key | Default | Description |
|---|---|---|
| `write_buffer_size` | `65536` | Write buffer size for hot index CFs. |
| `max_write_buffers` | `2` | Max RocksDB memtables for hot index CFs. |

#### `[storage.rocksdb.cf_profiles.raft]`

| Key | Default | Description |
|---|---|---|
| `write_buffer_size` | `262144` | Write buffer size for the Raft CF. |
| `max_write_buffers` | `2` | Max RocksDB memtables for the Raft CF. |

### `[datafusion]`

| Key | Default | Description |
|---|---|---|
| `memory_limit` | `33554432` (32 MB) | Query execution memory budget. |
| `query_parallelism` | `2` | Query execution worker concurrency. |
| `max_partitions` | `4` | Maximum partitions used in planning and scans. |
| `batch_size` | `1024` | Record batch size for execution operators. |

### `[flush]`

| Key | Default | Description |
|---|---|---|
| `default_row_limit` | `10000` | Row threshold that triggers flush when no per-table override exists. |
| `default_time_interval` | `300` | Time threshold in seconds for automatic flush. |
| `flush_batch_size` | `10000` | Rows loaded into memory per flush batch. |
| `check_interval_seconds` | `60` | Scheduler scan interval for pending writes. `0` disables the scheduler. |

#### `[flush.compaction]`

Optional post-flush Parquet compaction for trailing small segments. This is configured in
`server.toml`; there are no dedicated `KALAMDB_*` overrides for these keys today.

For the architecture behind these settings, see
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers) and
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests).

| Key | Default | Description |
|---|---|---|
| `enabled` | `false` | Enable leader-only post-flush compaction for eligible `USER` and `SHARED` table scopes. |
| `min_eligible_segments` | `5` | Minimum number of newest trailing small segments required before a compaction job is considered. |
| `max_segments_per_run` | `8` | Maximum number of newest trailing segments rewritten by one compaction job. |
| `user_max_segment_rows` | `10000` | User-table segments below this row count are considered small. |
| `shared_max_segment_rows` | `25000` | Shared-table segments below this row count are considered small. |

Compaction rewrites only the manifest tail: newest committed segments on the same schema version
and below the configured row target. If a newer flush changes that tail while compaction is writing,
the swap is skipped and the compacted output is discarded.

### `[retention]`

| Key | Default | Description |
|---|---|---|
| `enable_dba_stats` | `true` | Enables background `dba.stats` collection. |
| `dba_stats_retention_days` | `7` | Number of days to retain `dba.stats` samples. `0` disables automatic cleanup. |

### `[stream]`

| Key | Default | Description |
|---|---|---|
| `default_ttl_seconds` | `10` | Default lifetime of stream rows/events. |
| `default_max_buffer` | `10000` | Default in-memory stream buffer limit. |
| `eviction_interval_seconds` | `60` | Background eviction interval for expired stream data. |

### `[manifest_cache]`

Manifest cache behavior is explained in
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests).

| Key | Default | Description |
|---|---|---|
| `eviction_interval_seconds` | `300` | Interval between manifest cache eviction passes. |
| `max_entries` | `500` | Maximum hot manifest entries kept in memory. |
| `eviction_ttl_days` | `7` | Age after which unused manifest entries are evicted. |

### `[limits]`

| Key | Default | Description |
|---|---|---|
| `max_message_size` | `1048576` (1 MB) | Maximum accepted request payload/message size. |
| `max_query_limit` | `1000` | Hard maximum `LIMIT` the server will accept. |
| `default_query_limit` | `50` | Implicit `LIMIT` when the query omits one. |

### `[logging]`

For end-to-end examples and deployment guidance, use
[/docs/server/configurations/logging](https://kalamdb.org/docs/server/configurations/logging).

| Key | Default | Description |
|---|---|---|
| `level` | `"info"` | Global log verbosity. |
| `logs_path` | `"./logs"` | Directory for server log files. |
| `log_to_console` | `true` | Also emit logs to stdout/stderr. |
| `format` | `"compact"` | `compact`, `pretty`, or `json`. |
| `slow_query_threshold_ms` | `1200` | Slow query threshold in milliseconds. |

`[logging.targets]` is an optional dynamic table of per-target log levels, for example:

```toml
[logging.targets]
datafusion = "warn"
arrow = "warn"
parquet = "warn"
```

#### `[logging.otlp]`

| Key | Default | Description |
|---|---|---|
| `enabled` | `false` | Enables OpenTelemetry trace export. |
| `endpoint` | `"http://127.0.0.1:4317"` | Collector endpoint. |
| `protocol` | `"grpc"` | OTLP transport: `grpc` or `http`. |
| `service_name` | `"kalamdb-server"` | Service name attached to exported spans. |
| `timeout_ms` | `3000` | OTLP export timeout in milliseconds. |

### `[performance]`

| Key | Default | Description |
|---|---|---|
| `request_timeout` | `30` | Maximum full request execution time in seconds. |
| `keepalive_timeout` | `75` | Idle HTTP keep-alive timeout in seconds. |
| `max_connections` | `25000` | Maximum simultaneous client connections. |
| `backlog` | `4096` | Kernel listen backlog for pending TCP connections. |
| `tokio_worker_threads` | `0` | Tokio runtime worker threads. `0` means auto-size, capped in the binary. |
| `worker_max_blocking_threads` | `32` | Per-worker cap for blocking tasks such as RocksDB I/O. |
| `client_request_timeout` | `5` | Time allowed for a client to finish sending request headers. |
| `client_disconnect_timeout` | `2` | Graceful client disconnect timeout. |
| `max_header_size` | `16384` (16 KB) | Maximum HTTP header size. |

### `[rate_limit]`

| Key | Default | Description |
|---|---|---|
| `max_queries_per_sec` | `100` | Per-user SQL query rate limit. |
| `max_messages_per_sec` | `50` | Per-WebSocket incoming message rate limit. |
| `max_subscriptions_per_user` | `10` | Maximum live subscriptions per user. |
| `max_auth_requests_per_ip_per_sec` | `20` | Rate limit for auth/setup endpoints per IP. |
| `max_connections_per_ip` | `100` | Maximum concurrent connections per IP. |
| `max_requests_per_ip_per_sec` | `200` | Pre-auth request flood protection per IP. |
| `request_body_limit_bytes` | `10485760` (10 MB) | Request body cap used by connection protection. |
| `ban_duration_seconds` | `300` | Temporary IP ban duration after repeated abuse. |
| `enable_connection_protection` | `true` | Master switch for connection abuse protection. |
| `cache_max_entries` | `1000` | Maximum cached rate-limit entries. |
| `cache_ttl_seconds` | `600` | Idle TTL for cached rate-limit entries. |

### `[security]`

| Key | Default | Description |
|---|---|---|
| `trusted_proxy_ranges` | `[]` | Proxy IPs or CIDR ranges allowed to supply forwarded client IP headers. |
| `max_ws_message_size` | `1048576` (1 MB) | Maximum accepted WebSocket message size. |
| `strict_ws_origin_check` | `false` | Reject WebSocket connections that lack a valid `Origin` header. |
| `max_request_body_size` | `10485760` (10 MB) | Global HTTP request body cap. |

#### `[security.cors]`

| Key | Default | Description |
|---|---|---|
| `allowed_origins` | `[]` | Allowed browser origins. Empty behaves like wildcard at runtime, but non-localhost startup validation requires explicit origins. |
| `allowed_methods` | `["GET", "POST", "PUT", "DELETE", "PATCH", "OPTIONS"]` | Allowed CORS methods. |
| `allowed_headers` | `["Authorization", "Content-Type", "Accept", "Origin", "X-Requested-With"]` | Allowed browser request headers. |
| `expose_headers` | `[]` | Response headers exposed to browser JavaScript. |
| `allow_credentials` | `false` | Allow cookies or authorization headers on CORS requests. |
| `max_age` | `3600` | Preflight cache duration in seconds. |
| `allow_private_network` | `false` | Enables private network preflight support where browsers implement it. |

### `[websocket]`

| Key | Default | Description |
|---|---|---|
| `client_timeout_secs` | `10` | Maximum time without client heartbeat activity before disconnect. |
| `auth_timeout_secs` | `3` | Time allowed after connect before auth must arrive. |
| `heartbeat_interval_secs` | `5` | Interval between heartbeat scans/pings. |

### `[auth]` or `[authentication]`

Both section names are accepted. The config crate stores them as `auth` internally.

| Key | Default | Description |
|---|---|---|
| `root_password` | unset | Config-file root bootstrap password. Runtime env `KALAMDB_ROOT_PASSWORD` takes precedence. |
| `jwt_secret` | `"CHANGE_ME_IN_PRODUCTION"` | Secret for signing and validating internal HS256 tokens. |
| `jwt_trusted_issuers` | `""` | Comma-separated trusted external issuer list. |
| `jwt_expiry_hours` | `24` | Lifetime of internal JWTs in hours. |
| `cookie_secure` | `true` | Restricts auth cookies to HTTPS. |
| `allow_remote_setup` | `false` | Allow first-time setup from non-localhost clients. |
| `pg_auth_token` | unset | Pre-shared token for `pg_kalam` gRPC authentication. |

Compatibility note: older configs may still contain a pre-rename password-policy key; the current supported key is `auth.local.enforce_password_complexity`.

#### `[auth.local]`

| Key | Default | Description |
|---|---|---|
| `enabled` | `true` | Enables local username/password login and password setup. |
| `bcrypt_cost` | `12` | Password hash work factor. Valid range is determined by bcrypt. |
| `min_password_length` | `8` | Minimum accepted password length. |
| `max_password_length` | `1024` | Maximum locally accepted password length. |
| `enforce_password_complexity` | `false` | Require uppercase, lowercase, digit, and special characters in passwords. |

#### `[auth.oidc]`

KalamDB supports one external OpenID Connect provider per server process.

| Key | Default | Description |
|---|---|---|
| `enabled` | `false` | Enables the configured OIDC provider. |
| `display_name` | unset | Human-facing provider label for UI and CLI login. |
| `issuer` | unset | OIDC issuer URL. Must be included in `auth.jwt_trusted_issuers`. |
| `client_id` | unset | Client ID used for browser/device flows and default audience validation. |
| `client_secret` | unset | Client secret for confidential clients or brokered provider exchanges. |
| `scopes` | `["openid", "email", "profile"]` | OIDC scopes. Must include `openid`. |
| `device_authorization_endpoint` | unset | Optional device-flow endpoint override when discovery omits it. |
| `broker_device_flow_enabled` | `false` | Enables KalamDB-brokered OIDC device flow. |
| `auto_provision` | `false` | Allows absent regular OIDC subjects when default role is `user`; creates rows for elevated default roles. |
| `default_role` | `"user"` | Role used for auto-provisioned OIDC users. |
| `audience` | unset | Explicit audience override; defaults to `client_id`. |

### `[user_management]`

| Key | Default | Description |
|---|---|---|
| `deletion_grace_period_days` | `30` | Grace period before soft-deleted users are purged. |
| `cleanup_job_schedule` | `"0 2 * * *"` | Cron schedule for user cleanup. |

### `[files]`

| Key | Default | Description |
|---|---|---|
| `max_size_bytes` | `26214400` (25 MB) | Maximum size of a single uploaded file. |
| `max_files_per_request` | `20` | Maximum uploaded files accepted in one request. |
| `max_files_per_folder` | `5000` | Folder rotation threshold for durable file storage. |
| `staging_path` | `"./data/tmp"` | Temporary upload staging directory. |
| `allowed_mime_types` | `[]` | Allowed MIME types. Empty means allow all. |

### `[shutdown.flush]`

| Key | Default | Description |
|---|---|---|
| `timeout` | `300` | Maximum wait time in seconds for flush jobs during graceful shutdown. |

### `[jobs]`

| Key | Default | Description |
|---|---|---|
| `max_concurrent` | `10` | Maximum concurrently running background jobs. |
| `max_retries` | `3` | Retry attempts before a job is marked failed. |
| `retry_backoff_ms` | `100` | Initial retry backoff in milliseconds. |
| `wal_cleanup_interval_seconds` | `300` | Interval for memtable flushes that reclaim stale WAL files. `0` disables it. |

### `[execution]`

| Key | Default | Description |
|---|---|---|
| `handler_timeout_seconds` | `30` | Maximum execution handler duration in seconds. |
| `max_parameters` | `50` | Maximum parameter count per statement. |
| `max_parameter_size_bytes` | `524288` (512 KB) | Maximum size of a single parameter value. |
| `sql_plan_cache_max_entries` | `200` | Maximum cached SQL logical plans. |
| `sql_plan_cache_ttl_seconds` | `900` | Idle TTL for cached SQL logical plans. |

### `[topics]`

| Key | Default | Description |
|---|---|---|
| `visibility_timeout_secs` | `60` | Hide unacknowledged consumer claims for this many seconds before redelivery. |
| `default_retention_seconds` | `604800` (7 days) | Default topic time retention. |
| `default_retention_max_bytes` | `1073741824` (1 GiB) | Default topic byte retention per partition. |
| `retention_check_interval_seconds` | `3600` | Topic retention scheduler interval in seconds. `0` disables scheduling. |
| `retention_batch_size` | `10000` | Maximum deletes per partition in a single retention pass. |

### `[rpc_tls]`

This secures the shared gRPC listener used by Raft replication, cluster RPC, and the PostgreSQL extension.

| Key | Default | Description |
|---|---|---|
| `enabled` | `false` | Enable TLS or mTLS on the gRPC listener. |
| `ca_cert` | unset | CA certificate used to validate incoming client certificates. File path or inline PEM. |
| `server_cert` | unset | Server certificate. File path or inline PEM. |
| `server_key` | unset | Private key for `server_cert`. File path or inline PEM. |
| `require_client_cert` | `true` | Require clients to present a CA-signed certificate. |

### `[cluster]`

Omit this section entirely for standalone mode.

For the Multi-Raft model behind these options, see
[/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering).

| Key | Default | Description |
|---|---|---|
| `cluster_id` | `"cluster"` | Logical cluster identifier shared by all nodes. |
| `node_id` | required | Unique node ID for this server. Must be greater than `0`. |
| `rpc_addr` | `"127.0.0.1:2910"` | Advertised Raft RPC address. Use a reachable address, not a wildcard, in real clusters. |
| `api_addr` | `"0.0.0.0:2900"` | Advertised API address for this node. |
| `user_shards` | `8` | Number of user-data shards. |
| `shared_shards` | `1` | Number of shared-data shards. |
| `heartbeat_interval_ms` | `250` | Raft leader heartbeat interval. |
| `election_timeout_ms` | `(500, 1000)` | Min/max election timeout range. Min must exceed heartbeat. |
| `snapshot_policy` | `"LogsSinceLast(1000)"` | Snapshot policy. Use `LogsSinceLast(N)` or `Never`. |
| `max_snapshots_to_keep` | `3` | Snapshot retention count. `0` keeps all snapshots. |
| `replication_timeout_ms` | `5000` | Timeout for learner catch-up and replication progress. |
| `reconnect_interval_ms` | `3000` | Delay between reconnect attempts to unreachable peers. |
| `peer_wait_max_retries` | unset | Optional cap on peer-startup readiness retries. |
| `peer_wait_initial_delay_ms` | unset | Optional initial delay between peer readiness checks. |
| `peer_wait_max_delay_ms` | unset | Optional backoff cap for peer readiness checks. |

#### `[[cluster.peers]]`

| Key | Default | Description |
|---|---|---|
| `node_id` | required | Peer node ID. |
| `rpc_addr` | required | Peer RPC address. |
| `api_addr` | required | Peer API address. |
| `rpc_server_name` | unset | Optional TLS server-name override for SNI/hostname verification. |

**Total Raft groups** = 3 fixed groups (system metadata, user metadata, jobs) + `user_shards` + `shared_shards`.

Validation highlights:

- `node_id` must be greater than `0`
- `election_timeout_ms.0` must be greater than `heartbeat_interval_ms`
- `election_timeout_ms.1` must be greater than `election_timeout_ms.0`
- `user_shards` and `shared_shards` must be greater than `0`
- non-localhost multi-node clusters require `rpc_tls.enabled = true`

================================================================================
# Configurations
URL: https://kalamdb.org/docs/server/configurations
Source: content/server/configurations/index.mdx

> Complete configuration guide for KalamDB deployments. Learn how to tune your server, configure logging, configure OIDC, and set up OpenTelemetry.
# Configurations

This section centralizes runtime configuration for KalamDB.

If you are just getting started, begin with [Getting Started: Configuration](https://kalamdb.org/docs/server/getting-started/configuration).

For full production-grade tuning and every configuration section exposed by the config crate, continue to [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced).

For focused guides, continue to:

- [Logging & Traceability](https://kalamdb.org/docs/server/configurations/logging)
- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)

Integration-specific guides are in [Integrations](https://kalamdb.org/docs/server/integrations):

- [Dex](https://kalamdb.org/docs/server/integrations/dex)
- [MinIO (S3-Compatible)](https://kalamdb.org/docs/server/integrations/minio)
- [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger)

## Download Full Sample Config

If you want a complete config file with inline comments for every section, download:

- [server.example.toml (full commented sample)](https://raw.githubusercontent.com/kalamdb/KalamDB/refs/heads/main/backend/server.example.toml)

Quick download command:

```bash
curl -L https://raw.githubusercontent.com/kalamdb/KalamDB/refs/heads/main/backend/server.example.toml -o server.sample.toml
```

================================================================================
# Logging & Traceability
URL: https://kalamdb.org/docs/server/configurations/logging
Source: content/server/configurations/logging.mdx

> Configure KalamDB logging, JSON logs, OTLP tracing, Docker log output, and Dokploy-compatible runtime logging from server.toml or environment variables.
# Logging & Traceability

This page is the canonical guide for KalamDB runtime logging, JSON log output, Docker log collection, and distributed tracing.

Use it when you want to:

- switch the backend to JSON logs
- configure logging from `server.toml`
- run with logging overrides from environment variables
- ship logs through Docker or Dokploy
- enable OTLP tracing for Jaeger or another collector

## What KalamDB Supports Today

KalamDB currently supports two practical runtime log output modes:

- `compact`: human-readable text logs written to `server.log`
- `json`: JSON Lines logs written to `server.jsonl`

When `log_to_console = true`, the same events are also emitted to container stdout or your local terminal.

For traceability beyond logs, KalamDB also supports OTLP trace export under `[logging.otlp]`.

## JSON Logging Syntax

KalamDB JSON logs are emitted as JSON Lines.

- Each log event is one JSON object
- Each object is written on its own line
- The log file is `server.jsonl`
- This format is the one read by `system.server_logs`

Example real log line:

```json
{"timestamp":"2026-05-13T04:39:33.428337Z","level":"INFO","message":"KalamDB Server v0.5.0-beta.1 | Build: 2026-05-13 03:42:16 UTC","log.target":"kalamdb_server","log.module_path":"kalamdb_server","log.file":"backend/src/main.rs","log.line":340,"target":"kalamdb_server","threadName":"main"}
```

Common fields you should expect:

| Field | Meaning |
|---|---|
| `timestamp` | UTC event timestamp in RFC 3339 format |
| `level` | Log severity such as `INFO`, `WARN`, `ERROR`, `DEBUG`, `TRACE` |
| `message` | Event message text |
| `target` | Rust tracing target for the event |
| `threadName` | Thread name that emitted the event |
| `log.file` | Source file when available |
| `log.line` | Source line when available |
| `log.module_path` | Rust module path when available |

Depending on the event and span context, extra tracing metadata can also appear.

## Important Limitation: No Custom JSON Schema Yet

KalamDB does **not** currently support custom JSON log templates.

That means:

- you can choose `compact` or `json`
- you can choose the log level and destination
- you can export OTLP traces
- you **cannot** rename JSON keys or define a custom JSON logging pattern yet

If you need a different schema for downstream systems, transform the JSON lines in your log pipeline after KalamDB emits them.

## Configure Logging In `server.toml`

Use this when you want structured logs on disk and on stdout:

```toml
[logging]
level = "info"
format = "json"
logs_path = "./logs"
log_to_console = true
slow_query_threshold_ms = 1200

[logging.targets]
datafusion = "warn"
arrow = "warn"
parquet = "warn"
```

With that config:

- file logs go to `./logs/server.jsonl`
- console logs also use the same event stream
- `system.server_logs` can read the JSON log file

If you use `format = "compact"`, the server writes `server.log` instead.

## Configure Logging With Environment Variables

Environment variables override `server.toml` at startup.

### JSON Logging From Env

```bash
KALAMDB_LOG_LEVEL=info \
KALAMDB_LOG_FORMAT=json \
KALAMDB_LOGS_DIR=./logs \
KALAMDB_LOG_TO_CONSOLE=true \
cargo run --manifest-path backend/Cargo.toml --bin kalamdb-server
```

### Logging Environment Variables

| Environment variable | Maps to | Notes |
|---|---|---|
| `KALAMDB_LOG_LEVEL` | `logging.level` | `error`, `warn`, `info`, `debug`, `trace` |
| `KALAMDB_LOG_FORMAT` | `logging.format` | Use `json` for JSON Lines, `compact` for text |
| `KALAMDB_LOGS_DIR` | `logging.logs_path` | Directory for `server.log` or `server.jsonl` |
| `KALAMDB_LOG_TO_CONSOLE` | `logging.log_to_console` | `true`, `1`, `yes` enable console emission |
| `KALAMDB_SLOW_QUERY_THRESHOLD_MS` | `logging.slow_query_threshold_ms` | Slow query threshold in milliseconds |
| `KALAMDB_OTLP_ENABLED` | `logging.otlp.enabled` | Enables trace export |
| `KALAMDB_OTLP_ENDPOINT` | `logging.otlp.endpoint` | OTLP collector endpoint |
| `KALAMDB_OTLP_PROTOCOL` | `logging.otlp.protocol` | `grpc` or `http` |
| `KALAMDB_OTLP_SERVICE_NAME` | `logging.otlp.service_name` | Service name shown in tracing backends |
| `KALAMDB_OTLP_TIMEOUT_MS` | `logging.otlp.timeout_ms` | OTLP export timeout |

Current limitation:

- per-target overrides under `[logging.targets]` are currently configured in `server.toml`, not through dedicated environment variables

## Docker Logging

For Docker deployments, the simplest pattern is:

1. emit JSON logs to stdout
2. keep `log_to_console = true`
3. optionally persist the same logs under `/data/logs`

The single-node Docker config shipped with KalamDB already does this:

```toml
[logging]
level = "info"
logs_path = "/data/logs"
log_to_console = true
format = "json"
```

### `docker run` Example

```bash
docker run --rm \
  -p 2900:2900 \
  -v "$PWD/data:/data" \
  -e KALAMDB_LOG_LEVEL=info \
  -e KALAMDB_LOG_FORMAT=json \
  -e KALAMDB_LOG_TO_CONSOLE=true \
  -e KALAMDB_LOGS_DIR=/data/logs \
  jamals86/kalamdb:latest
```

### Docker Compose Example

```yaml
services:
  kalamdb:
    image: jamals86/kalamdb:latest
    ports:
      - "2900:2900"
    volumes:
      - ./data:/data
    environment:
      KALAMDB_LOG_LEVEL: info
      KALAMDB_LOG_FORMAT: json
      KALAMDB_LOG_TO_CONSOLE: "true"
      KALAMDB_LOGS_DIR: /data/logs
```

## Dokploy Compatibility

Dokploy runtime logs are compatible with KalamDB JSON logs **as raw log lines**.

What this means in practice:

- if KalamDB writes JSON logs to stdout, Dokploy can display them
- each KalamDB log event appears as one JSON line
- this works best when `KALAMDB_LOG_FORMAT=json` and `KALAMDB_LOG_TO_CONSOLE=true`

What you should **not** assume today:

- Dokploy does not appear to have a documented KalamDB-specific schema for these fields
- I did not find evidence that Dokploy automatically maps fields like `threadName`, `target`, or `log.line` into structured columns for KalamDB logs

The safe assumption is:

- Dokploy will show the JSON lines correctly
- any deeper structured parsing depends on Dokploy-side features or downstream tooling, not on a KalamDB-specific format contract

Dokploy’s log UI is built around Docker log streaming, so stdout compatibility matters more than matching a custom syntax.

## OTLP Tracing

For distributed tracing, configure `[logging.otlp]` in addition to normal logs.

```toml
[logging.otlp]
enabled = true
endpoint = "http://127.0.0.1:4317"
protocol = "grpc"
service_name = "kalamdb-server"
timeout_ms = 3000
```

Environment equivalent:

```bash

```

Use OTLP when you want:

- trace spans in Jaeger, Tempo, or an OpenTelemetry Collector
- request correlation across services
- timing and span visibility beyond plain log lines

Use JSON logging when you want:

- structured application logs
- Docker or Dokploy log shipping
- queryable `system.server_logs` entries

Most production setups use both.

## Query Logs From SQL

`system.server_logs` reads the JSON log file generated by the backend.

Example:

```sql
SELECT timestamp, level, target, message
FROM system.server_logs
ORDER BY timestamp DESC
LIMIT 50;
```

This view only works reliably when the backend log format is `json`.

## Recommended Production Setup

For most production deployments:

1. set `format = "json"`
2. keep `log_to_console = true`
3. write logs under a persistent directory such as `/data/logs`
4. enable OTLP if you need request tracing in Jaeger or Tempo
5. use Dokploy or Docker for raw log streaming, and a collector for deeper analytics if needed

## Related

- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)
- [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced#logging)
- [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger)

================================================================================
# OIDC & Issuer Trust
URL: https://kalamdb.org/docs/server/configurations/oidc
Source: content/server/configurations/oidc.mdx

> Configure KalamDB's single external OpenID Connect provider, issuer trust, device flow options, and the exact `server.toml` / env vars used by the Admin UI and CLI.
# OIDC & Issuer Trust

KalamDB supports one external OpenID Connect provider per server process, configured under `[auth.oidc]`.

Use this page when you want the Admin UI, the CLI, or direct bearer-token clients to authenticate through a standards-compliant OIDC issuer such as Dex, Keycloak, Okta, Auth0, Entra ID, Google, or Firebase.

There is no active `[oauth]` or `[oauth.providers.*]` matrix anymore. The current model is:

- `[auth.local]` for local username/password login
- `[auth.oidc]` for one external OIDC provider
- `auth.jwt_trusted_issuers` for the issuer allow-list used by bearer-token validation

## Token Routing Model

KalamDB inspects every bearer token **before** signature verification to decide how to validate it:

| Token `alg` | Token `iss` | Validation Path |
|---|---|---|
| HS256 | `kalamdb` | Internal: verified using `auth.jwt_secret` |
| RS256, RS384, RS512, PS256, PS384, PS512, ES256, ES384 | Any trusted issuer | External: verified via OIDC discovery + provider JWKS |

> **ES512 is not supported.**

This means external tokens from supported OIDC providers are validated natively through the bearer-token pipeline.

## `server.toml` Settings

```toml
[auth]
jwt_secret = "replace-with-strong-random-secret-32-plus-chars"
jwt_trusted_issuers = "kalamdb,https://issuer.example.com"
jwt_expiry_hours = 24
allow_remote_setup = false
cookie_secure = true

[auth.local]
enabled = true

[auth.oidc]
enabled = true
display_name = "Company SSO"
issuer = "https://issuer.example.com"
client_id = "kalamdb"
client_secret = "optional-confidential-client-secret"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
broker_device_flow_enabled = true
# Optional override when discovery metadata omits device flow.
device_authorization_endpoint = "https://issuer.example.com/oauth2/device_authorization"
# Optional explicit audience override. Defaults to client_id when unset.
# audience = "kalamdb-api"
```

Notes:

- `[auth]` is the canonical section name. `[authentication]` still deserializes as a compatibility alias for local files, but new docs and examples use `[auth]`.
- `auth.oidc.issuer` and `auth.oidc.client_id` are required when `auth.oidc.enabled = true`.
- `auth.oidc.scopes` must include `openid`.
- `auth.oidc.audience` is optional and defaults to `client_id` when omitted.
- `jwt_trusted_issuers` is a comma-separated string, not an array.
- Keep `kalamdb` in `jwt_trusted_issuers` so local KalamDB-issued tokens continue to verify.
- Legacy `[oauth]` and `[oauth.providers.*]` sections are no longer supported.

## Environment Variable Overrides

```bash

```

Boolean env vars accept `"true"`, `"1"`, or `"yes"` (case-insensitive).

Environment variables override `server.toml` values.

## OIDC Configuration Matrix

| `server.toml` key | Environment variable | Required when OIDC enabled | Notes |
|---|---|---|---|
| `auth.jwt_secret` | `KALAMDB_JWT_SECRET` | Yes | Used for KalamDB-issued HS256 access and refresh tokens |
| `auth.jwt_trusted_issuers` | `KALAMDB_JWT_TRUSTED_ISSUERS` | Yes | Include `kalamdb` plus the configured OIDC issuer |
| `auth.local.enabled` | `KALAMDB_AUTH_LOCAL_ENABLED` | No | Disable local username/password auth after OIDC migration if desired |
| `auth.local.bcrypt_cost` | — | No | Password hash work factor for local users |
| `auth.local.min_password_length` | — | No | Minimum accepted local password length |
| `auth.local.max_password_length` | — | No | Maximum accepted local password length |
| `auth.local.enforce_password_complexity` | — | No | Require uppercase, lowercase, digit, and special characters |
| `auth.oidc.enabled` | `KALAMDB_AUTH_OIDC_ENABLED` | Yes | Enables the single external OIDC provider |
| `auth.oidc.display_name` | `KALAMDB_AUTH_OIDC_DISPLAY_NAME` | No | Human-facing label in UI/CLI, for example `Dex` or `Company SSO` |
| `auth.oidc.issuer` | `KALAMDB_AUTH_OIDC_ISSUER` | Yes | Must start with `http://` or `https://` |
| `auth.oidc.client_id` | `KALAMDB_AUTH_OIDC_CLIENT_ID` | Yes | Used for browser and device flows and default audience validation |
| `auth.oidc.client_secret` | `KALAMDB_AUTH_OIDC_CLIENT_SECRET` | No | Needed for confidential clients or some brokered device flows |
| `auth.oidc.scopes` | `KALAMDB_AUTH_OIDC_SCOPES` | No | CSV env value; must include `openid` |
| `auth.oidc.device_authorization_endpoint` | `KALAMDB_AUTH_OIDC_DEVICE_AUTHORIZATION_ENDPOINT` | No | Optional override when discovery omits device flow |
| `auth.oidc.broker_device_flow_enabled` | `KALAMDB_AUTH_OIDC_BROKER_DEVICE_FLOW_ENABLED` | No | Enables KalamDB-brokered device flow endpoints |
| `auth.oidc.auto_provision` | `KALAMDB_AUTH_OIDC_AUTO_PROVISION` | No | Allows absent regular OIDC subjects when default role is `user`; creates rows for elevated default roles |
| `auth.oidc.default_role` | `KALAMDB_AUTH_OIDC_DEFAULT_ROLE` | No | Default role for auto-provisioned users |
| `auth.oidc.audience` | `KALAMDB_AUTH_OIDC_AUDIENCE` | No | Explicit audience override; defaults to `client_id` |

## Browser And Device Endpoints

Clients discover the current auth surface from:

```text
GET /v1/api/auth/login-options
```

KalamDB exposes these OIDC-facing endpoints:

| Endpoint | Used by | Purpose |
|---|---|---|
| `POST /v1/api/auth/oidc/exchange-code` | Admin UI and CLI browser login | Exchange authorization code and PKCE verifier for KalamDB tokens |
| `POST /v1/api/auth/oidc/exchange-token` | CLI direct device login | Exchange provider ID token for KalamDB tokens |
| `POST /v1/api/auth/oidc/device/start` | CLI brokered device login | Start KalamDB-brokered device flow |
| `POST /v1/api/auth/oidc/device/poll` | CLI brokered device login | Poll for brokered device completion |

Redirect URIs to register at the provider:

- Admin UI: `https://YOUR_KALAMDB_ORIGIN/ui/oauth/callback`
- CLI browser login: `http://127.0.0.1:8787/callback`

## OIDC Discovery Flow

When KalamDB encounters an external token (non-HS256):

1. Extracts `alg` and `kid` from the JWT header (unverified).
2. Extracts `iss` from the JWT payload (unverified).
3. Confirms `iss` is in `jwt_trusted_issuers`.
4. Fetches `{issuer_url}/.well-known/openid-configuration`.
5. Extracts `jwks_uri` from the discovery response.
6. Fetches the JWKS endpoint and caches keys by `kid`.
7. Matches the token `kid` to a cached signing key.
8. Validates signature, issuer, expiry, and audience (if configured).

## JWKS Caching Behavior

- Keys are cached per-issuer in memory, indexed by `kid`.
- **Cache-miss refresh**: if a token's `kid` is not found, KalamDB automatically re-fetches the JWKS endpoint.
- **No TTL-based eviction**: cache refreshes only on miss — keys remain cached until an unknown `kid` triggers a refresh.
- Keys without a `kid` field are silently discarded from the cache.
- Key rotation at the provider is handled automatically (new `kid` → cache miss → refresh).

## JWT Claims Structure

KalamDB reads these claims from external tokens:

| Claim | Type | Required | Notes |
|---|---|---|---|
| `sub` | string | Yes | KalamDB `user_id`; must be a valid `UserId` |
| `iss` | string | Yes | Must match a trusted issuer |
| `exp` | number | Yes | Expiry (Unix timestamp, seconds) |
| `iat` | number | Yes | Issued-at (Unix timestamp, seconds) |
| `username` / `preferred_username` | string | No | Informational only; not used as the KalamDB `user_id` |
| `email` | string | No | User email |
| `role` | string | No | KalamDB role: `user`, `service`, `dba`, `system` |
| `token_type` | string | No | `"access"` or `"refresh"` |

## Deployment Checklist

1. Confirm the exact issuer URL. Trailing slashes matter.
2. Register the Admin UI and CLI redirect URIs at the provider.
3. Set `auth.oidc.issuer`, `auth.oidc.client_id`, and `auth.jwt_trusted_issuers`.
4. Keep `auth.local.enabled = true` during bootstrap and migration.
5. Test `GET /v1/api/auth/login-options`.
6. Test Admin UI browser login and at least one CLI OIDC mode.
7. Disable local auth later only after OIDC access is verified.

## Troubleshooting

| Symptom | Likely Cause | Fix |
|---|---|---|
| `401 Unauthorized` on external token | Issuer not in `jwt_trusted_issuers` | Add exact issuer URL |
| `auth.oidc.issuer is required` on startup | OIDC enabled without issuer | Set `auth.oidc.issuer` |
| `auth.oidc.client_id is required` on startup | OIDC enabled without client ID | Set `auth.oidc.client_id` |
| `auth.oidc.scopes must include the 'openid' scope` | Missing required scope | Add `openid` to `auth.oidc.scopes` |
| `MissingKid` error | Token missing `kid` header | Provider must include `kid` in JWT header |
| `KeyNotFound` after JWKS refresh | Key rotated and `kid` no longer present | Check provider JWKS, ensure key is published |
| `DiscoveryFailed` | Cannot reach `/.well-known/openid-configuration` | Check network, DNS, and issuer URL format |
| `JwtValidationFailed: ExpiredSignature` | Token expired | Get fresh token from provider |
| User not found after auth | `auth.oidc.auto_provision` is `false` or elevated roles are explicit | Enable auto-provision or pre-create the user with `CREATE USER '<sub>' WITH OIDC ...` |

## Related

- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)
- [Single OIDC Provider Model](https://kalamdb.org/docs/server/auth/providers)
- [Dex](https://kalamdb.org/docs/server/integrations/dex)
- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)

================================================================================
# OpenTelemetry (OTEL)
URL: https://kalamdb.org/docs/server/configurations/otel
Source: content/server/configurations/otel.mdx

> Configure KalamDB distributed tracing with OpenTelemetry. Set up OTLP exporters, Jaeger integration, environment overrides, and server.toml settings.
# OpenTelemetry (OTEL)

This page shows how to enable distributed tracing for KalamDB with OTLP-compatible backends such as Jaeger.

## Quick Start (Jaeger)

Download a ready-to-run compose file:

- [Download docker-compose.yml (Jaeger)](https://kalamdb.org/downloads/otel/docker-compose.yml)

Run it from the folder where you downloaded it:

```bash
cd /path/to/downloaded/file
docker compose up -d
```

Jaeger UI: `http://127.0.0.1:16686`

This example mirrors the Jaeger section in `KalamDB/docker/utils/docker-compose.yml`.

## Configure `server.toml`

KalamDB OTEL settings live under `[logging.otlp]`:

```toml
[logging.otlp]
enabled = true
endpoint = "http://127.0.0.1:4317"
protocol = "grpc"
service_name = "kalamdb-server"
timeout_ms = 3000
```

## Environment Variable Overrides

Environment variables override `server.toml` values at startup.

```bash

```

`KALAMDB_OTLP_ENABLED` accepts `"true"`, `"1"`, or `"yes"` (case-insensitive).

## Full OTEL Configuration Matrix

| `server.toml` key | Environment variable | Default | Notes |
|---|---|---|---|
| `logging.otlp.enabled` | `KALAMDB_OTLP_ENABLED` | `false` | Enables OTLP trace exporting. |
| `logging.otlp.endpoint` | `KALAMDB_OTLP_ENDPOINT` | `http://127.0.0.1:4317` | Collector endpoint. |
| `logging.otlp.protocol` | `KALAMDB_OTLP_PROTOCOL` | `grpc` | Supported values: `grpc`, `http`. |
| `logging.otlp.service_name` | `KALAMDB_OTLP_SERVICE_NAME` | `kalamdb-server` | Service name shown in tracing backend. |
| `logging.otlp.timeout_ms` | `KALAMDB_OTLP_TIMEOUT_MS` | `3000` | Export timeout in milliseconds. |

Protocol behavior:

- `grpc` expects endpoint like `http://127.0.0.1:4317` (Jaeger port 4317)
- `http` expects collector HTTP endpoint like `http://127.0.0.1:4318`; the exporter appends `/v1/traces` automatically

## Validation Checklist

1. Start Jaeger/collector
2. Start KalamDB with OTEL enabled
3. Execute API requests or SQL calls
4. Open Jaeger UI and confirm service name appears

## Configuration Precedence

KalamDB applies settings in this order:

1. `server.toml`
2. `KALAMDB_*` environment overrides

If both are set, environment variables win.

## Supported Backends

Any OTLP-compatible collector works with KalamDB, including:

- [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger) (built-in compose example)
- Grafana Tempo
- Honeycomb
- Datadog (OTLP ingest)
- OpenTelemetry Collector (forwarding to any backend)

## Related

- [Logging & Traceability](https://kalamdb.org/docs/server/configurations/logging)
- [Jaeger Integration](https://kalamdb.org/docs/server/integrations/jaeger)
- [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced#loggingotlp)

================================================================================
# Admin UI Guide
URL: https://kalamdb.org/docs/server/getting-started/admin-ui
Source: content/server/getting-started/admin-ui.mdx

> Operate KalamDB from the built-in Admin UI. Learn how to access setup, login, use SQL Studio, monitor operations, and manage your database visually.
# KalamDB Admin UI Guide

The Admin UI is served by the KalamDB server at `/ui` and is the fastest way to run SQL, inspect metadata, and monitor operations.

## Access The UI

Open:

```text
http://localhost:2900/ui/sql
```

If server setup is not complete, finish bootstrap first from [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication).

## Sign In

Use a real DBA/system account you created during setup. There is no universal hardcoded production credential.

After login, the UI stores auth state using secure cookie/token flow from `/v1/api/auth/*` endpoints.

## SQL Studio

SQL Studio is the primary workspace:

- namespace/table explorer
- multi-tab SQL editor
- results and execution details
- schema context and recent query history

Starter query:

```sql
SELECT * FROM system.namespaces LIMIT 100;
```

## Main Navigation Areas

- [Dashboard](https://kalamdb.org/docs/server/admin-ui/dashboard): health and high-level runtime indicators
- [SQL Studio](https://kalamdb.org/docs/server/admin-ui/sql-studio): query execution and schema exploration
- [Create a Table](https://kalamdb.org/docs/server/admin-ui/create-table): first schema workflow from SQL Studio
- [Streaming](https://kalamdb.org/docs/server/admin-ui/streaming): topic monitoring and stream operations
- [Users](https://kalamdb.org/docs/server/admin-ui/users): user/role administration
- [Jobs](https://kalamdb.org/docs/server/admin-ui/jobs): background job tracking (flush/retention/cleanup)
- [Live Queries](https://kalamdb.org/docs/server/admin-ui/live-queries): active subscription inspection
- [Logging](https://kalamdb.org/docs/server/admin-ui/logging): server/job/audit log views
- [Settings](https://kalamdb.org/docs/server/admin-ui/settings): runtime configuration values exposed by the backend

For page-by-page screenshots and operational workflows, continue to [Admin UI](https://kalamdb.org/docs/server/admin-ui).

## Practical Operator Workflow

1. Check node health and queue pressure in `Dashboard`.
2. Validate schema and data paths in `SQL Studio`.
3. Inspect `Jobs` for flush and maintenance behavior.
4. Review `Live Queries` and `Streaming` for realtime workload behavior.
5. Verify production guardrails in `Settings` before rollout.

================================================================================
# AI Agent Coding Guidelines
URL: https://kalamdb.org/docs/server/getting-started/ai-agent-guidelines
Source: content/server/getting-started/ai-agent-guidelines.mdx

> Best practices for AI-generated KalamDB code: use subscriptions for realtime updates, avoid interval polling, and follow consumer ACK patterns.
# AI Agent Coding Guidelines

When AI agents generate code for KalamDB, use these rules as defaults.

Install the official KalamDB skill so supported coding agents can load KalamDB commands, SQL syntax, SDK examples, and operations guidance:

```bash
npx skills add kalamdb/kalamdb-skills
```

## Prefer subscriptions for realtime state

For live UI or state sync, use WebSocket subscriptions:

- `client.live(sql, callback, options)` for materialized row state
- `client.liveTable(table, callback, options)` when table-name sugar is enough
- `client.liveEvents(sql, callback, options)` only when you need raw protocol events

Use subscriptions as the primary realtime pattern instead of interval polling.

Keep live SQL in the strict supported shape: `SELECT ... FROM ... WHERE ...`.
Do not put `ORDER BY` or `LIMIT` inside live subscription SQL; sort or cap rows after they arrive.

## Do not use interval polling for realtime updates

Avoid timer-based loops such as:

- `setInterval(() => fetch(...), 1000)`
- periodic `SELECT` queries to detect row changes
- fixed sleep loops to simulate streaming

These patterns add unnecessary load and increase staleness compared to server-push subscriptions.

## Use topics and consumers for background workers

For worker pipelines and asynchronous processing, use topic consumers with ACK semantics:

- `createConsumerClient({...}).consumer({...}).run(handler)`
- `runConsumer({...})`

This is preferred over polling tables on a schedule.

## Reliability defaults for AI-generated code

- Keep writes idempotent.
- Track and restore progress with sequence IDs / offsets when reconnecting.
- Handle subscription `error` events explicitly.
- Use bounded retries and `onFailed` persistence in agent/consumer flows.

## Recommended references

- [Realtime Subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers)
- [Consumer Runtime](https://kalamdb.org/docs/ts-sdk/agents)
- [SQL Subscriptions](https://kalamdb.org/docs/server/sql-reference/subscriptions)
- [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics)

================================================================================
# Authentication & Bootstrap
URL: https://kalamdb.org/docs/server/getting-started/authentication
Source: content/server/getting-started/authentication.mdx

> Bootstrap KalamDB, keep or disable local password login, and configure the single OpenID Connect provider used by the Admin UI and CLI.
# Authentication & Bootstrap

KalamDB now has one authentication surface with two modes:

- local username/password login controlled by `[auth.local]`
- one external OpenID Connect provider controlled by `[auth.oidc]`

There is no active provider matrix anymore. Dex, Keycloak, Okta, Auth0, Entra ID, Google, Firebase, and similar services all fit the same single-provider OIDC slot as long as they expose standards-compliant discovery metadata.

Use this page for the practical rollout:

1. bootstrap the server locally
2. create the initial DBA and root credentials
3. configure `[auth.oidc]`
4. test Admin UI and CLI login
5. optionally disable local password login after migration

For the full config matrix, see [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc). For a local test IdP, use [Dex](https://kalamdb.org/docs/server/integrations/dex).

## 1. Bootstrap The Server

Check whether the server still needs first-time setup:

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
```

When the response contains `"needs_setup": true`, create the initial DBA and root password:

```bash
curl -X POST http://127.0.0.1:2900/v1/api/auth/setup \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{
  "user": "admin",
  "password": "AdminPass123!",
  "root_password": "RootPass123!",
  "email": "admin@example.com"
}
JSON
```

Important behavior:

- `POST /v1/api/auth/setup` is localhost-only unless `auth.allow_remote_setup = true`
- setup creates the initial local admin credentials; it does not return tokens
- if you plan to disable local auth later, do the bootstrap first while `[auth.local].enabled = true`

## 2. Configure Local + OIDC Auth In `server.toml`

Start with a combined rollout so you keep a local admin escape hatch while validating OIDC:

```toml
[auth]
jwt_secret = "replace-with-a-strong-random-secret-at-least-32-chars"
jwt_trusted_issuers = "kalamdb,https://idp.example.com/realms/kalamdb"
jwt_expiry_hours = 24
allow_remote_setup = false
cookie_secure = true

[auth.local]
enabled = true

[auth.oidc]
enabled = true
display_name = "Company SSO"
issuer = "https://idp.example.com/realms/kalamdb"
client_id = "kalamdb"
client_secret = "optional-confidential-client-secret"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
broker_device_flow_enabled = true
# Optional when discovery does not advertise device flow.
device_authorization_endpoint = "https://idp.example.com/realms/kalamdb/protocol/openid-connect/auth/device"
```

Rules that matter:

- KalamDB supports one external OIDC provider per server process
- `auth.oidc.issuer` and `auth.oidc.client_id` are required when `auth.oidc.enabled = true`
- `auth.oidc.scopes` must include `openid`
- `auth.jwt_trusted_issuers` should include both `kalamdb` and the external issuer

Equivalent environment overrides:

```bash

```

Boolean env vars accept `true`, `1`, or `yes` (case-insensitive).

## 3. Discover Available Login Methods

The Admin UI and CLI discover auth capabilities from the same endpoint:

```bash
curl http://127.0.0.1:2900/v1/api/auth/login-options
```

Typical response:

```json
{
  "local": { "enabled": true },
  "oidc": {
    "enabled": true,
    "display_name": "Company SSO",
    "issuer": "https://idp.example.com/realms/kalamdb",
    "client_id": "kalamdb",
    "authorization_endpoint": "https://idp.example.com/.../auth",
    "token_endpoint": "https://idp.example.com/.../token",
    "device_authorization_endpoint": "https://idp.example.com/.../device",
    "scopes": ["openid", "email", "profile"],
    "device_flow": {
      "direct_supported": true,
      "broker_supported": true,
      "broker_start_endpoint": "/v1/api/auth/oidc/device/start",
      "broker_poll_endpoint": "/v1/api/auth/oidc/device/poll"
    }
  }
}
```

## 4. Admin UI Browser Login

For browser login, configure the IdP redirect URI to:

```text
https://YOUR_KALAMDB_ORIGIN/ui/oauth/callback
```

The Admin UI uses Authorization Code + PKCE:

1. browser fetches `/v1/api/auth/login-options`
2. browser redirects to the configured OIDC authorization endpoint
3. the callback page posts the code and PKCE verifier to `/v1/api/auth/oidc/exchange-code`
4. KalamDB verifies the provider response and returns KalamDB access and refresh tokens

## 5. CLI Login

### Local username/password login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --user admin --password
```

### Browser OIDC login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc
```

For browser OIDC, register this redirect URI at the provider:

```text
http://127.0.0.1:8787/callback
```

### Device login without opening a browser callback

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser
```

This uses direct provider device flow when the provider exposes a device authorization endpoint.

### Brokered device login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser --brokered
```

Use brokered mode when the CLI host cannot reach the provider directly but can reach KalamDB.

Operational notes:

- successful interactive `kalam login` continues directly into the SQL shell
- non-interactive `kalam login` saves credentials and exits
- `--no-save` skips writing credentials to disk

## 6. Use KalamDB Tokens For API Calls

After local login or OIDC exchange, use the returned KalamDB access token on API calls:

```bash
curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer <access_token>" \
  -H 'Content-Type: application/json' \
  -d '{"sql":"SELECT CURRENT_USER();"}'
```

Use `/v1/api/auth/me` to confirm the resolved identity:

```bash
curl http://127.0.0.1:2900/v1/api/auth/me \
  -H "Authorization: Bearer <access_token>"
```

## 7. Auto-Provisioning And Explicit OIDC Users

With `auth.oidc.auto_provision = true` and `auth.oidc.default_role = "user"`, regular OIDC users authenticate as their OIDC `sub` without a persisted per-user row when no local row exists.

When you need an elevated role before first login, or when `auto_provision = false`, pre-create the OIDC user explicitly with issuer and subject:

```sql
CREATE USER 'provider-subject'
  WITH OIDC '{"issuer":"https://idp.example.com/realms/kalamdb","subject":"provider-subject"}'
  ROLE service
  EMAIL 'alice@example.com';
```

For persisted OIDC users, the `CREATE USER` id must match the OIDC `subject`.

## 8. Disable Local Password Login After Migration

Once OIDC is proven and you still have a safe DBA path, you can disable local password login:

```toml
[auth.local]
enabled = false
```

This disables local username/password login server-side. The Admin UI and CLI also learn that state through `/v1/api/auth/login-options` and should hide local login controls.

## Related

- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Kalam CLI](https://kalamdb.org/docs/cli)
- [Dex](https://kalamdb.org/docs/server/integrations/dex)
- [Single OIDC Provider Model](https://kalamdb.org/docs/server/auth/providers)

The bridge issuer (e.g. `kalamdb-keycloak-bridge`) must appear in `jwt_trusted_issuers`.

## 8. Production Hardening

Before exposing KalamDB publicly:

1. Set strong `auth.jwt_secret` (or `KALAMDB_JWT_SECRET`).
2. Keep `auth.allow_remote_setup = false`.
3. Set `auth.cookie_secure = true` behind HTTPS.
4. Configure `auth.jwt_trusted_issuers` when using external IdPs.
5. Tune `rate_limit.max_auth_requests_per_ip_per_sec`.

For full endpoint auth behavior, see [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference).

================================================================================
# GitHub Binaries
URL: https://kalamdb.org/docs/server/getting-started/binaries
Source: content/server/getting-started/binaries.mdx

> Download prebuilt kalamdb-server and kalam CLI binaries from GitHub Releases. Supports Linux, macOS Apple Silicon, and Windows platforms.
# GitHub Binaries

If you do not want Docker or a local Rust toolchain, use release artifacts from GitHub:

- [KalamDB Releases](https://github.com/kalamdb/KalamDB/releases)

Each release publishes `kalamdb-server` and `kalamcli` archives, a `SHA256SUMS` file, and `versions.json`. Always verify checksums before running downloaded binaries.

## What to download

From the selected release, download **both** the server and CLI archive for your platform, plus `SHA256SUMS`:

| Platform | Server archive | CLI archive |
|----------|----------------|-------------|
| Linux x86_64 | `kalamdb-server--linux-x86_64.tar.gz` | `kalamcli--linux-x86_64.tar.gz` |
| Linux ARM64 | `kalamdb-server--linux-aarch64.tar.gz` | `kalamcli--linux-aarch64.tar.gz` |
| macOS Apple Silicon | `kalamdb-server--macos-aarch64.tar.gz` | `kalamcli--macos-aarch64.tar.gz` |
| Windows x86_64 | `kalamdb-server--windows-x86_64.zip` | `kalamcli--windows-x86_64.zip` |

The file names below automatically use the latest GitHub release tag. macOS Intel (`x86_64`) builds are not published — use Docker, build from source, or run under Rosetta with a Linux binary.

## Important: binary names and config

Release archives extract to **versioned filenames** such as `kalamdb-server-0.5.3-rc.1-linux-x86_64`, not plain `kalamdb-server`. The server also requires a `server.toml` config path on startup — it will not run with zero arguments.

## Start the server

<ReleaseCodeBlock
	code={`# After downloading and extracting both archives into kalamdb-bin/:
cd kalamdb-bin
ln -sf kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64 kalamdb-server
ln -sf kalamcli-{{RELEASE_VERSION}}-linux-x86_64 kalam

cat > server.toml <<'EOF'
[server]
host = "127.0.0.1"
port = 2900

[storage]
data_path = "./data"

[limits]

[logging]
logs_path = "./logs"
log_to_console = true

[performance]

[auth]
jwt_secret = "local-binary-test-secret-32chars-min"
root_password = "kalamdb123"
EOF

./kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64 server.toml`}
/>

Confirm it is up:

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
```

Point the CLI at the same URL:

```bash
./kalamcli-<version>-linux-x86_64 --url http://127.0.0.1:2900 --user root --password
```

Optional: set `KALAMDB_SERVER_BIN` when using `kalam dev` in a project:

```bash

kalam dev
```

## Linux example (x86_64)

<ReleaseCodeBlock
	code={`sha256sum -c SHA256SUMS --ignore-missing

mkdir -p kalamdb-bin && cd kalamdb-bin
tar -xzf ../kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64.tar.gz
tar -xzf ../kalamcli-{{RELEASE_VERSION}}-linux-x86_64.tar.gz
ln -sf kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64 kalamdb-server
ln -sf kalamcli-{{RELEASE_VERSION}}-linux-x86_64 kalam

cat > server.toml <<'EOF'
[server]
host = "127.0.0.1"
port = 2900

[storage]
data_path = "./data"

[limits]

[logging]
logs_path = "./logs"
log_to_console = true

[performance]

[auth]
jwt_secret = "local-binary-test-secret-32chars-min"
root_password = "kalamdb123"
EOF

./kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64 server.toml`}
/>

In another terminal:

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
./kalam --help
```

Install onto your PATH (optional):

```bash
install -m 755 kalamdb-server kalam ~/.local/bin/
```

## macOS example (Apple Silicon)

<ReleaseCodeBlock
	code={`shasum -a 256 -c SHA256SUMS

mkdir -p kalamdb-bin && cd kalamdb-bin
tar -xzf ../kalamdb-server-{{RELEASE_VERSION}}-macos-aarch64.tar.gz
tar -xzf ../kalamcli-{{RELEASE_VERSION}}-macos-aarch64.tar.gz
ln -sf kalamdb-server-{{RELEASE_VERSION}}-macos-aarch64 kalamdb-server
ln -sf kalamcli-{{RELEASE_VERSION}}-macos-aarch64 kalam

cat > server.toml <<'EOF'
[server]
host = "127.0.0.1"
port = 2900

[storage]
data_path = "./data"

[limits]

[logging]
logs_path = "./logs"
log_to_console = true

[performance]

[auth]
jwt_secret = "local-binary-test-secret-32chars-min"
root_password = "kalamdb123"
EOF

./kalamdb-server-{{RELEASE_VERSION}}-macos-aarch64 server.toml
./kalam --help`}
/>

## Windows example (x86_64)

Verify and use the CLI:

```powershell
curl http://127.0.0.1:2900/v1/api/auth/status
.\kalamcli-<version>-windows-x86_64.exe --help
```

## Next steps

- [Quick Start](https://kalamdb.org/docs/server/getting-started)
- [Kalam CLI](https://kalamdb.org/docs/cli)
- [Configuration](https://kalamdb.org/docs/server/getting-started/configuration)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Firebase](https://kalamdb.org/docs/server/integrations/firebase)
- [Keycloak](https://kalamdb.org/docs/server/integrations/keycloak)
- [MinIO (S3-Compatible)](https://kalamdb.org/docs/server/integrations/minio)
- [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger)
- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)

================================================================================
# Configuration
URL: https://kalamdb.org/docs/server/getting-started/configuration
Source: content/server/getting-started/configuration.mdx

> Minimal configuration for running KalamDB safely in development and production. Covers server.toml settings and environment variable overrides.
# Configuration

KalamDB loads runtime settings from `server.toml` and then applies `KALAMDB_*` environment overrides.

For complete section-by-section tuning, use [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced).

Focused guides:

- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Dex](https://kalamdb.org/docs/server/integrations/dex)
- [MinIO (S3-Compatible)](https://kalamdb.org/docs/server/integrations/minio)
- [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger)

## Download Full Sample Config

- [server.example.toml (full commented sample)](https://raw.githubusercontent.com/kalamdb/KalamDB/refs/heads/main/backend/server.example.toml)

```bash
curl -L https://raw.githubusercontent.com/kalamdb/KalamDB/refs/heads/main/backend/server.example.toml -o server.example.toml
```

## Run With Config File

```bash
kalamdb-server --config /path/to/server.toml
```

## Minimal Production-Oriented Example

```toml
[server]
host = "0.0.0.0"
port = 2900
api_version = "v1"
enable_http2 = true

[storage]
data_path = "./data"

[auth]
jwt_secret = "replace-with-strong-random-secret-32-plus-chars"
cookie_secure = true
allow_remote_setup = false

[auth.local]
enabled = true

[rate_limit]
enable_connection_protection = true
max_requests_per_ip_per_sec = 200
max_auth_requests_per_ip_per_sec = 20

[topics]
visibility_timeout_secs = 60

[security]
max_request_body_size = 10485760
max_ws_message_size = 1048576

[security.cors]
allowed_origins = ["https://app.example.com"]
allow_credentials = true
```

Notes:

- `data_path` is the canonical storage root key (not `data_dir`).
- `auth` can also be written as `[authentication]` (alias supported by config loader).
- table flush behavior is set per table via SQL `WITH (FLUSH_POLICY = '...')`, not a top-level `storage.flush_policy` key.
- `topics.visibility_timeout_secs` controls when unacknowledged topic consumer claims become available for redelivery.

## Add OIDC To `server.toml`

KalamDB supports one external OIDC provider at a time.

```toml
[auth]
jwt_trusted_issuers = "kalamdb,https://idp.example.com/realms/kalamdb"

[auth.local]
enabled = true

[auth.oidc]
enabled = true
display_name = "Company SSO"
issuer = "https://idp.example.com/realms/kalamdb"
client_id = "kalamdb"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
broker_device_flow_enabled = true
```

Keep local auth enabled during bootstrap and rollout, then disable it later with `[auth.local].enabled = false` if you want an OIDC-only deployment.

## High-Value Environment Overrides

These are commonly used in Docker and CI:

- `KALAMDB_SERVER_HOST`, `KALAMDB_SERVER_PORT`, `KALAMDB_SERVER_PUBLIC_ORIGIN`, `KALAMDB_SERVER_WORKERS`
- `KALAMDB_DATA_DIR`
- `KALAMDB_LOG_LEVEL`, `KALAMDB_LOG_FORMAT`, `KALAMDB_LOGS_DIR`, `KALAMDB_LOG_TO_CONSOLE`
- `KALAMDB_JWT_SECRET`, `KALAMDB_JWT_TRUSTED_ISSUERS`, `KALAMDB_JWT_EXPIRY_HOURS`, `KALAMDB_COOKIE_SECURE`, `KALAMDB_ALLOW_REMOTE_SETUP`
- `KALAMDB_AUTH_LOCAL_ENABLED`
- `KALAMDB_AUTH_OIDC_ENABLED`, `KALAMDB_AUTH_OIDC_DISPLAY_NAME`, `KALAMDB_AUTH_OIDC_ISSUER`, `KALAMDB_AUTH_OIDC_CLIENT_ID`, `KALAMDB_AUTH_OIDC_CLIENT_SECRET`
- `KALAMDB_AUTH_OIDC_SCOPES`, `KALAMDB_AUTH_OIDC_DEVICE_AUTHORIZATION_ENDPOINT`, `KALAMDB_AUTH_OIDC_BROKER_DEVICE_FLOW_ENABLED`
- `KALAMDB_AUTH_OIDC_AUTO_PROVISION`, `KALAMDB_AUTH_OIDC_DEFAULT_ROLE`, `KALAMDB_AUTH_OIDC_AUDIENCE`
- `KALAMDB_SECURITY_CORS_ALLOWED_ORIGINS`, `KALAMDB_SECURITY_TRUSTED_PROXY_RANGES`
- `KALAMDB_RATE_LIMIT_AUTH_REQUESTS_PER_IP_PER_SEC`
- `KALAMDB_TOPIC_VISIBILITY_TIMEOUT_SECS`, `KALAMDB_TOPIC_DEFAULT_RETENTION_SECONDS`, `KALAMDB_TOPIC_DEFAULT_RETENTION_MAX_BYTES`
- `KALAMDB_CLUSTER_ID`, `KALAMDB_NODE_ID`, `KALAMDB_CLUSTER_RPC_ADDR`, `KALAMDB_CLUSTER_API_ADDR`, `KALAMDB_CLUSTER_PEERS`
- `KALAMDB_RPC_TLS_ENABLED`, `KALAMDB_RPC_TLS_CA_CERT`, `KALAMDB_RPC_TLS_SERVER_CERT`, `KALAMDB_RPC_TLS_SERVER_KEY`
- `KALAMDB_ROOT_PASSWORD` for root bootstrap and `KALAMDB_TOKIO_WORKER_THREADS` for Tokio runtime sizing

Compatibility aliases still accepted today:

- `KALAMDB_CLUSTER_NODE_ID` for `KALAMDB_NODE_ID`
- `KALAMDB_VISIBILITY_TIMEOUT_SECS` for `KALAMDB_TOPIC_VISIBILITY_TIMEOUT_SECS`
- `KALAMDB_TRUSTED_PROXY_RANGES` for `KALAMDB_SECURITY_TRUSTED_PROXY_RANGES`

For OTEL-specific override behavior, see [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel).
For IdP issuer setup, see [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc).
For the full, complete override table, use [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced).

## Cluster Override Format

`KALAMDB_CLUSTER_PEERS` uses this format:

```text
node_id@rpc_addr@api_addr[@rpc_server_name];node_id@rpc_addr@api_addr[@rpc_server_name]
```

Example:

```text
2@kalamdb-node2:2910@http://kalamdb-node2:2900;3@kalamdb-node3:2910@http://kalamdb-node3:2900
```

================================================================================
# Docker Deployment
URL: https://kalamdb.org/docs/server/getting-started/docker
Source: content/server/getting-started/docker.mdx

> Run KalamDB with Docker or Docker Compose in single-node or 3-node cluster mode. Includes copy-paste commands, health checks, and volume configuration.
# Docker Deployment

KalamDB publishes [`jamals86/kalamdb:latest`](https://hub.docker.com/r/jamals86/kalamdb) on Docker Hub. The image includes `kalamdb-server`, `kalam-cli`, and a `kalam` symlink. The server listens on port **2900** inside the container.

> **Note:** Some older Docker Hub overview text references port `8080` or the `kalamstack` GitHub org. The commands below match the current image. Canonical Hub readme source: [`docker/REPO-README.md`](https://github.com/kalamdb/KalamDB/blob/main/docker/REPO-README.md) in the KalamDB repo.

## Quick start — copy-paste

Run a single-node server on host port **2900**, bootstrap auth, and log in:

```bash
docker run -d --name kalamdb -p 2900:2900 \
  -e KALAMDB_SERVER_HOST=0.0.0.0 \
  -e KALAMDB_JWT_SECRET="$(openssl rand -base64 32)" \
  -e KALAMDB_ALLOW_REMOTE_SETUP=true \
  -v kalamdb_data:/data \
  jamals86/kalamdb:latest

sleep 5
curl -sS http://127.0.0.1:2900/v1/api/auth/status

curl -sS -X POST http://127.0.0.1:2900/v1/api/auth/setup \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!","root_password":"RootPass123!"}'

curl -sS -X POST http://127.0.0.1:2900/v1/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!"}'
```

Save the `access_token` from the login response.

Verify from the host with `auth/status` (recommended). The `/v1/api/healthcheck` endpoint is localhost-only when reached through Docker port publishing — use this from inside the container instead:

```bash
docker exec kalamdb wget -qO- http://127.0.0.1:2900/v1/api/healthcheck
```

Admin UI: `http://127.0.0.1:2900/ui` — log in as `admin` / `AdminPass123!`.

Useful environment variables for local Docker runs:

| Variable | Purpose |
|----------|---------|
| `KALAMDB_SERVER_HOST` | Bind address inside the container — use `0.0.0.0` when publishing ports |
| `KALAMDB_JWT_SECRET` | JWT signing secret — use at least 32 characters in shared environments |
| `KALAMDB_ALLOW_REMOTE_SETUP` | Allow first-time auth setup from the host when running in Docker |
| `KALAMDB_SECURITY_CORS_ALLOWED_ORIGINS` | CORS origins for the Admin UI (compose files default to `*` for local demos) |

Avoid `KALAMDB_ROOT_PASSWORD` for first-time Docker setup on the host: it can mark the server as configured while host login still fails. Use `POST /v1/api/auth/setup` instead (as in the quick start above).

## Single node with Docker Compose

Download the compose file, create a compatible `server.toml` (the checked-in `docker/run/single/server.toml` in the KalamDB repo is missing required sections for current server builds), then start:

```bash
mkdir kalamdb-docker && cd kalamdb-docker
curl -fsSL https://raw.githubusercontent.com/kalamdb/KalamDB/main/docker/run/single/docker-compose.yml -o docker-compose.yml

cat > server.toml <<'EOF'
[server]
host = "0.0.0.0"
port = 2900

[storage]
data_path = "/data"

[limits]

[logging]
level = "info"
logs_path = "/data/logs"
log_to_console = true
format = "json"

[performance]

[auth]
jwt_secret = "docker-compose-placeholder-secret-32c"

[security.cors]
allowed_origins = ["*"]

[rate_limit]
max_queries_per_sec = 1000000
max_messages_per_sec = 10000000
max_subscriptions_per_user = 100000
max_auth_requests_per_ip_per_sec = 20000000
max_connections_per_ip = 100000
max_requests_per_ip_per_sec = 20000000
request_body_limit_bytes = 104857600
ban_duration_seconds = 300
enable_connection_protection = true
EOF

KALAMDB_JWT_SECRET="$(openssl rand -base64 32)" KALAMDB_PORT=2900 docker compose up -d
```

By default the compose file maps `${KALAMDB_PORT:-8088}:2900`. Setting `KALAMDB_PORT=2900` publishes the API at `http://127.0.0.1:2900`.

Bootstrap and verify:

```bash
curl -sS http://127.0.0.1:2900/v1/api/auth/status
curl -sS -X POST http://127.0.0.1:2900/v1/api/auth/setup \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!","root_password":"RootPass123!"}'
```

Friendly alias for the compose file only: [kalamdb.org/docker-compose.yml](https://kalamdb.org/docker-compose.yml) redirects to the GitHub source. You still need a compatible `server.toml` beside it.

## 3-node cluster

```bash
mkdir kalamdb-cluster && cd kalamdb-cluster
curl -fsSL https://raw.githubusercontent.com/kalamdb/KalamDB/main/docker/run/cluster/docker-compose.yml -o docker-compose.yml
KALAMDB_JWT_SECRET="$(openssl rand -base64 32)" docker compose up -d
```

Default host ports:

| Node | HTTP API | Raft gRPC |
|------|----------|-----------|
| node 1 | `8081` | `9091` |
| node 2 | `8082` | `9092` |
| node 3 | `8083` | `9093` |

Verify node 1 from the host:

```bash
curl http://127.0.0.1:8081/v1/api/auth/status
```

## Windows PowerShell

`docker run` with bootstrap:

```powershell
docker run -d --name kalamdb -p 2900:2900 `
  -e KALAMDB_SERVER_HOST=0.0.0.0 `
  -e KALAMDB_JWT_SECRET="$([Convert]::ToBase64String((1..32 | ForEach-Object { Get-Random -Maximum 256 })))" `
  -e KALAMDB_ALLOW_REMOTE_SETUP=true `
  -v kalamdb_data:/data `
  jamals86/kalamdb:latest

Start-Sleep -Seconds 5
Invoke-RestMethod http://127.0.0.1:2900/v1/api/auth/status

Invoke-RestMethod -Method Post http://127.0.0.1:2900/v1/api/auth/setup `
  -ContentType "application/json" `
  -Body '{"user":"admin","password":"AdminPass123!","root_password":"RootPass123!"}'
```

Compose (after creating `docker-compose.yml` and the `server.toml` from the single-node section above):

```powershell
$env:KALAMDB_JWT_SECRET = [Convert]::ToBase64String((1..32 | ForEach-Object { Get-Random -Maximum 256 }))
$env:KALAMDB_PORT = "2900"
Invoke-WebRequest https://raw.githubusercontent.com/kalamdb/KalamDB/main/docker/run/single/docker-compose.yml -UseBasicParsing -OutFile docker-compose.yml
docker compose up -d
```

## CLI inside the container

The image ships with `kalam` and `kalam-cli`:

```bash
docker exec kalamdb kalam --version
docker exec -it kalamdb kalam --url http://127.0.0.1:2900 --user admin --password
```

CLI state defaults to `/data/.kalam` inside the container so credentials persist with the `/data` volume.

## Persistence and cleanup

Data is stored under `/data` in the container. The quick-start `docker run` command uses the named volume `kalamdb_data`.

```bash
docker logs -f kalamdb          # view logs
docker stop kalamdb             # stop without deleting data
docker rm kalamdb               # remove container; volume keeps data
docker rm -f kalamdb && docker volume rm kalamdb_data   # full reset
```

## Notes

- Keep `KALAMDB_JWT_SECRET` non-default in shared environments.
- For first login/setup in Docker, review [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication).
- For OIDC issuer trust, review [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc).
- Official image: [jamals86/kalamdb on Docker Hub](https://hub.docker.com/r/jamals86/kalamdb)

================================================================================
# Quick Start Guide
URL: https://kalamdb.org/docs/server/getting-started
Source: content/server/getting-started/index.mdx

> Run KalamDB, bootstrap authentication, create your first table, and continue into the TypeScript SDK, HTTP API, or PostgreSQL extension.
# Quick Start

This guide takes you from zero to a working KalamDB flow:

1. Run the server
2. Bootstrap authentication
3. Create a namespace and table
4. Execute SQL
5. Connect from the TypeScript SDK or CLI

Use this path if you want the fastest route into a working local KalamDB setup before moving into the TypeScript SDK, HTTP API, PostgreSQL extension, or production guides.

## Prerequisites

Pick the path that matches how you want to run the server:

| Option | You need |
|--------|----------|
| **1. Docker** | Docker Engine or Docker Desktop |
| **2. Kalam CLI** | [Kalam CLI](https://kalamdb.org/docs/cli/setup) installed (`install.sh` or npm) and, for TypeScript starters, Node.js plus a package manager (npm, pnpm, yarn, or bun) |
| **3. GitHub binaries** | `curl` or a browser — no Rust toolchain required |
| **4. Build from source** | Git and Rust `1.92+` |

## Option 1: Docker

The fastest way to run KalamDB locally is the published image [`jamals86/kalamdb:latest`](https://hub.docker.com/r/jamals86/kalamdb). Copy-paste the commands below — they start the server, bootstrap auth, and log you in as the initial DBA user.

```bash
docker run -d --name kalamdb -p 2900:2900 \
  -e KALAMDB_SERVER_HOST=0.0.0.0 \
  -e KALAMDB_JWT_SECRET="$(openssl rand -base64 32)" \
  -e KALAMDB_ALLOW_REMOTE_SETUP=true \
  -v kalamdb_data:/data \
  jamals86/kalamdb:latest

sleep 5
curl -sS http://127.0.0.1:2900/v1/api/auth/status

curl -sS -X POST http://127.0.0.1:2900/v1/api/auth/setup \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!","root_password":"RootPass123!"}'

curl -sS -X POST http://127.0.0.1:2900/v1/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!"}'
```

What this does:

- Starts `kalamdb-server` with data persisted in the named volume `kalamdb_data`
- Generates a JWT signing secret (required when binding to `0.0.0.0`)
- Allows first-time setup from your host machine (`KALAMDB_ALLOW_REMOTE_SETUP=true`)
- Creates DBA user `admin` and returns a JWT when login succeeds

Save the `access_token` from the login response for the SQL steps below.

> **Docker note:** `/v1/api/healthcheck` is localhost-only inside the container. From your host, use `/v1/api/auth/status` (as above) or run `docker exec kalamdb wget -qO- http://127.0.0.1:2900/v1/api/healthcheck`.

Admin UI: `http://127.0.0.1:2900/ui` — log in as `admin` with `AdminPass123!`.

For Docker Compose (single node or 3-node cluster), volume overrides, and Windows PowerShell examples, see [Docker Deployment](https://kalamdb.org/docs/server/getting-started/docker).

## Option 2: Kalam CLI (`kalam init` + `kalam dev`)

Use this path when you want a project scaffold (`kalam.toml`, `schema.sql`, migrations) and a single command that starts the local server, applies schema, and supervises your app.

### Step 1 — Install the CLI

```bash
curl -fsSL https://kalamdb.org/install.sh | sh
```

Or with npm:

```bash
npm install -g @kalamdb/cli
```

Verify:

```bash
kalam --version
kalam doctor
```

### Step 2 — Scaffold a project with `kalam init`

Create a directory and run the interactive wizard:

```bash
mkdir my-app && cd my-app
kalam init
```

The wizard asks for:

1. **Project name** — written to `[project].name` in `kalam.toml`
2. **Schema mode** — choose **SQL file** (`schema.sql`)
3. **Language targets** — TypeScript and/or Dart for generated SDK types
4. **Template** — built-in TypeScript starter (for example `simple-live`)
5. **Package manager** — npm, pnpm, yarn, or bun (TypeScript only)
6. **Server mode** — choose **local** so `kalam dev` can start `kalamdb-server` for you
7. **Server URL** — defaults to `http://localhost:2900` for local mode

Non-interactive equivalent (CI or scripted setup):

```bash
kalam init --yes \
  --name my-app \
  --languages typescript \
  --template simple-live \
  --package-manager pnpm \
  --server-mode local
```

`kalam init` creates at minimum:

| File / directory | Purpose |
|------------------|---------|
| `kalam.toml` | Project config: environments, schema paths, dev orchestration |
| `schema.sql` | Your schema source |
| `kalam/migrations/` | Migration history |
| `kalam/server/server.toml` | Local server config (includes `auth.root_password`) |
| `src/generated/kalam.ts` | Generated TypeScript types (after first dev run) |

### Step 3 — Start everything with `kalam dev`

From the project root:

```bash
kalam dev
```

On startup, `kalam dev`:

1. Resolves the dev environment from `kalam.toml` (`http://localhost:2900` by default)
2. **Starts or reuses** a local `kalamdb-server` — if no binary is installed, the CLI can download the matching release into `~/.kalam/bin` (interactive terminals only)
3. **Bootstraps auth** — logs in as local root using `kalam/server/server.toml` and saves credentials to `~/.kalam/`
4. **Applies schema** — runs the migration pipeline against `schema.sql`
5. **Regenerates types** — refreshes SDK artifacts when configured
6. **Supervises app processes** — starts commands from `[dev.processes]` (for example `pnpm dev` for TypeScript starters)

To wipe local database files and start over, stop the dev session and run `kalam db reset` (use `--yes` when reusing a non-project server or in non-interactive shells), then `kalam dev` again.

Press `Ctrl+C` to shut down managed processes cleanly.

Useful flags:

```bash
kalam dev --env dev --namespace app --project-dir .
kalam dev --force   # skip draft confirmation prompts on startup
```

Check project state in another terminal:

```bash
kalam status
curl http://127.0.0.1:2900/v1/api/auth/status
```

While `kalam dev` is running, sign in to the local server with **`root` / `kalamdb123`**:

- **Admin UI:** [http://localhost:2900/ui](http://localhost:2900/ui)
- **CLI:** `kalam --url http://127.0.0.1:2900 --user root --password kalamdb123`

If port **2900** is already in use, `kalam dev` reuses that server instead of starting a new one. Use a free port when scaffolding:

```bash
kalam init --yes --name my-app --languages typescript --server-mode local --server-url http://localhost:2933
```

Full references: [Project Init](https://kalamdb.org/docs/cli/init) and [Local Development](https://kalamdb.org/docs/cli/dev).

## Option 3: GitHub release binaries

Download prebuilt `kalamdb-server` and `kalam` archives from [KalamDB Releases](https://github.com/kalamdb/KalamDB/releases). Each release publishes platform archives plus a `SHA256SUMS` file — verify checksums before extracting.

### Choose your platform

The latest release ships these server and CLI archives (version shown dynamically):

| Platform | Server archive | CLI archive |
|----------|----------------|-------------|
| Linux x86_64 | `kalamdb-server--linux-x86_64.tar.gz` | `kalamcli--linux-x86_64.tar.gz` |
| Linux ARM64 | `kalamdb-server--linux-aarch64.tar.gz` | `kalamcli--linux-aarch64.tar.gz` |
| macOS Apple Silicon | `kalamdb-server--macos-aarch64.tar.gz` | `kalamcli--macos-aarch64.tar.gz` |
| Windows x86_64 | `kalamdb-server--windows-x86_64.zip` | `kalamcli--windows-x86_64.zip` |

Replace `<version>` with the bare release tag from GitHub (for example `0.5.3-rc.1`). macOS Intel builds are not published — use Docker, Rosetta with a Linux binary, or build from source on Intel Macs.

Release archives contain **versioned binary names** (not `kalamdb-server` / `kalam`). You also need a `server.toml` config file — the binary does not start without one.

### Linux (x86_64) example

<ReleaseCodeBlock
	code={`# 1. Download both archives and SHA256SUMS from the release page, then verify:
sha256sum -c SHA256SUMS --ignore-missing

# 2. Extract (creates versioned binary names in the current directory)
mkdir -p kalamdb-bin && cd kalamdb-bin
tar -xzf ../kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64.tar.gz
tar -xzf ../kalamcli-{{RELEASE_VERSION}}-linux-x86_64.tar.gz

# 3. Optional convenience symlinks
ln -sf kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64 kalamdb-server
ln -sf kalamcli-{{RELEASE_VERSION}}-linux-x86_64 kalam

# 4. Create a minimal local config
cat > server.toml <<'EOF'
[server]
host = "127.0.0.1"
port = 2900

[storage]
data_path = "./data"

[limits]

[logging]
logs_path = "./logs"
log_to_console = true

[performance]

[auth]
jwt_secret = "local-binary-test-secret-32chars-min"
root_password = "kalamdb123"
EOF

# 5. Start the server
./kalamdb-server-{{RELEASE_VERSION}}-linux-x86_64 server.toml`}
/>

In another terminal, confirm the server is up and run the CLI:

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
./kalamcli-<version>-linux-x86_64 --help   # or ./kalam --help if you created the symlink
```

### macOS (Apple Silicon) example

<ReleaseCodeBlock
	code={`shasum -a 256 -c SHA256SUMS

mkdir -p kalamdb-bin && cd kalamdb-bin
tar -xzf ../kalamdb-server-{{RELEASE_VERSION}}-macos-aarch64.tar.gz
tar -xzf ../kalamcli-{{RELEASE_VERSION}}-macos-aarch64.tar.gz
ln -sf kalamdb-server-{{RELEASE_VERSION}}-macos-aarch64 kalamdb-server
ln -sf kalamcli-{{RELEASE_VERSION}}-macos-aarch64 kalam

cat > server.toml <<'EOF'
[server]
host = "127.0.0.1"
port = 2900

[storage]
data_path = "./data"

[limits]

[logging]
logs_path = "./logs"
log_to_console = true

[performance]

[auth]
jwt_secret = "local-binary-test-secret-32chars-min"
root_password = "kalamdb123"
EOF

./kalamdb-server-{{RELEASE_VERSION}}-macos-aarch64 server.toml`}
/>

### Windows (x86_64) example

Then verify from PowerShell or Command Prompt:

```powershell
curl http://127.0.0.1:2900/v1/api/auth/status
.\kalamcli-<version>-windows-x86_64.exe --help
```

More platform notes: [GitHub Binaries](https://kalamdb.org/docs/server/getting-started/binaries).

## Option 4: Build from source

Clone the repository and run the server with Cargo (requires Rust `1.92+`):

```bash
git clone https://github.com/kalamdb/KalamDB.git
cd KalamDB/backend
cargo run --release --bin kalamdb-server -- server.example.toml
```

The example config binds to `http://127.0.0.1:2900` by default. Copy and edit `server.example.toml` if you need different paths or secrets.

To build the CLI as well:

```bash
cd ../cli
cargo build --release
./target/release/kalam --help
```

## Verify Health

After starting the server with any option above:

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
```

For a native or CLI-managed local server you can also use:

```bash
curl http://127.0.0.1:2900/v1/api/healthcheck
```

Docker users: the health endpoint is localhost-only from inside the container. Use `auth/status` from the host, or `docker exec kalamdb wget -qO- http://127.0.0.1:2900/v1/api/healthcheck`.

## Bootstrap Authentication

Check whether the server still needs first-time setup:

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
```

If it returns `"needs_setup": true`, initialize root + DBA user:

```bash
curl -X POST http://127.0.0.1:2900/v1/api/auth/setup \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!","root_password":"RootPass123!"}'
```

If the server is already configured, this endpoint returns conflict; in that case, go directly to login.

Then log in:

```bash
curl -X POST http://127.0.0.1:2900/v1/api/auth/login \
  -H 'Content-Type: application/json' \
  -d '{"user":"admin","password":"AdminPass123!"}'
```

Keep the returned `access_token` for API calls.

## Create Namespace And Table

```bash
TOKEN="<ACCESS_TOKEN>"

curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"sql":"CREATE NAMESPACE IF NOT EXISTS app;"}'

curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{"sql":"CREATE TABLE app.messages (id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(), sender TEXT NOT NULL, content TEXT NOT NULL, created_at TIMESTAMP DEFAULT NOW()) WITH (TYPE='USER', FLUSH_POLICY='rows:1000,interval:60');"}
JSON
```

## Insert And Query

```bash
curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d @- <<'JSON'
{"sql":"INSERT INTO app.messages (sender, content) VALUES ('alice', 'Hello KalamDB');"}
JSON

curl -X POST http://127.0.0.1:2900/v1/api/sql \
  -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"sql":"SELECT * FROM app.messages ORDER BY created_at DESC LIMIT 10;"}'
```

## Connect With TypeScript SDK (`@kalamdb/client`)

Install the package first:

```bash
npm install @kalamdb/client
```

Package reference: 

```ts

const client = createClient({
  url: 'http://127.0.0.1:2900',
  authProvider: async () => Auth.jwt('<ACCESS_TOKEN>'),
});

const res = await client.query('SELECT * FROM app.messages LIMIT 5');
console.log(res.results[0]);
```

## Connect With Kalam CLI

Install the CLI with npm or the curl installer:

```bash
npm install -g @kalamdb/cli

# or
curl -fsSL https://kalamdb.org/install.sh | sh
```

Log in and run a query:

```bash
kalam login --instance local --url http://127.0.0.1:2900 --user admin --password
kalam -c "SELECT * FROM app.messages ORDER BY created_at DESC LIMIT 5;"
kalam doctor
```

If you want to test the OIDC flow instead of local password login:

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser
```

Full CLI reference: [Kalam CLI](https://kalamdb.org/docs/cli) — start with [Project Init](https://kalamdb.org/docs/cli/init) and [Local Development](https://kalamdb.org/docs/cli/dev)

## Use KalamDB Skills With Coding Agents

Install the official KalamDB skill for Codex, Claude Code, OpenCode, and Agent Skills-compatible tools:

```bash
npx skills add kalamdb/kalamdb-skills
```

## Choose Your Next Track

### Building an app or AI-agent backend in TypeScript

Continue with [TypeScript Setup](https://kalamdb.org/docs/ts-sdk/setup), [Authentication](https://kalamdb.org/docs/ts-sdk/auth), [Querying & DML](https://kalamdb.org/docs/ts-sdk/querying), and [Realtime Subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions).

### Building background workers and agent automation

Continue with [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers), [Consumer Runtime](https://kalamdb.org/docs/ts-sdk/agents), and [AI Agent Coding Guidelines](https://kalamdb.org/docs/server/getting-started/ai-agent-guidelines).

### Integrating through PostgreSQL

Continue with [PostgreSQL Extension Getting Started](https://kalamdb.org/docs/pg-kalam/getting-started), [SQL Syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax), and [Data Type Conversions](https://kalamdb.org/docs/pg-kalam/type-conversions).

### Hardening for production

Continue with [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication), [Configuration](https://kalamdb.org/docs/server/getting-started/configuration), [Security](https://kalamdb.org/docs/server/security), and [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc).

### Advanced features and integrations

Continue with [SQL Reference](https://kalamdb.org/docs/server/sql-reference), [Vector Search](https://kalamdb.org/docs/server/architecture/vector-search), [Dex](https://kalamdb.org/docs/server/integrations/dex), [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc), [MinIO (S3-Compatible)](https://kalamdb.org/docs/server/integrations/minio), [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger), and [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel).

================================================================================
# Server Docs
URL: https://kalamdb.org/docs/server
Source: content/server/index.mdx

> Run KalamDB, configure authentication, use SQL, operate storage, and deploy production-ready realtime database services.
# KalamDB Server Docs

Use these server docs to run KalamDB, build a SQL-first backend for AI agents and realtime apps, configure authentication, operate storage, and deploy production services.

If you are comparing backend-as-a-service options, KalamDB gives you a self-hosted realtime data layer with SQL, authentication, tenant isolation, live subscriptions, topics, and vector search in one runtime.

## Start Here

1. [/docs/server/getting-started](https://kalamdb.org/docs/server/getting-started)
2. [/docs/server/sql-reference](https://kalamdb.org/docs/server/sql-reference)
3. [/docs/server/getting-started/authentication](https://kalamdb.org/docs/server/getting-started/authentication)
4. [/docs/server/configurations](https://kalamdb.org/docs/server/configurations)
5. [/docs/server/security](https://kalamdb.org/docs/server/security)

## Choose Your Path

### I am building AI agents or a chat application

Start with [/docs/server/getting-started](https://kalamdb.org/docs/server/getting-started), then continue to
[/docs/ts-sdk/setup](https://kalamdb.org/docs/ts-sdk/setup),
[/docs/ts-sdk/subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions), and
[/docs/ts-sdk/consumers](https://kalamdb.org/docs/ts-sdk/consumers).

### I want a backend-as-a-service alternative I can control

Start with [/docs/server/getting-started](https://kalamdb.org/docs/server/getting-started), then read
[/docs/server/getting-started/authentication](https://kalamdb.org/docs/server/getting-started/authentication),
[/docs/server/sql-reference](https://kalamdb.org/docs/server/sql-reference),
[/docs/server/security](https://kalamdb.org/docs/server/security), and
[/docs/server/configurations/oidc](https://kalamdb.org/docs/server/configurations/oidc).

### I need PostgreSQL compatibility

Start with [/docs/pg-kalam/getting-started](https://kalamdb.org/docs/pg-kalam/getting-started), then read
[/docs/pg-kalam/architecture](https://kalamdb.org/docs/pg-kalam/architecture),
[/docs/pg-kalam/sql-syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax), and
[/docs/pg-kalam/type-conversions](https://kalamdb.org/docs/pg-kalam/type-conversions).

### I need advanced data and operations guidance

Start with [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types),
[/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search),
[/docs/server/topic-pubsub](https://kalamdb.org/docs/server/topic-pubsub),
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers),
[/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering), and
[/docs/server/configurations/otel](https://kalamdb.org/docs/server/configurations/otel).

## Questions New Users Ask

### What should I read first?

If you are new, start at [/docs/server/getting-started](https://kalamdb.org/docs/server/getting-started). It takes you
from running KalamDB locally to creating your first table and sending your first SQL query.

### Can frontend clients execute SQL directly against KalamDB?

Yes. Use [/docs/ts-sdk](https://kalamdb.org/docs/ts-sdk) for browser and app-facing clients. USER tables keep the same
SQL scoped to the signed-in user, and live subscriptions push updates over WebSockets.

### Where do I start for workers, automation, and advanced agent flows?

Go from [/docs/ts-sdk/setup](https://kalamdb.org/docs/ts-sdk/setup) to
[/docs/ts-sdk/consumers](https://kalamdb.org/docs/ts-sdk/consumers),
[/docs/ts-sdk/agents](https://kalamdb.org/docs/ts-sdk/agents), and
[/docs/server/getting-started/ai-agent-guidelines](https://kalamdb.org/docs/server/getting-started/ai-agent-guidelines).

### Where do I learn production deployment and observability?

Read [/docs/server/security](https://kalamdb.org/docs/server/security),
[/docs/server/configurations/logging](https://kalamdb.org/docs/server/configurations/logging),
[/docs/server/configurations/advanced](https://kalamdb.org/docs/server/configurations/advanced),
[/docs/server/configurations/oidc](https://kalamdb.org/docs/server/configurations/oidc),
[/docs/server/integrations/dex](https://kalamdb.org/docs/server/integrations/dex),
[/docs/server/integrations/jaeger](https://kalamdb.org/docs/server/integrations/jaeger), and
[/docs/server/configurations/otel](https://kalamdb.org/docs/server/configurations/otel).

## Core Capabilities

KalamDB combines:

- SQL query and DDL workflows
- user-isolated and shared table models
- live query subscriptions over WebSocket
- topic consume/ack worker patterns
- vector search with `EMBEDDING(n)` columns and cosine ranking
- hot + cold storage lifecycle in one runtime

| Capability | What it gives you |
|------------|-------------------|
| `USER` tables | per-user/tenant isolation |
| `SHARED` tables | global app data scope |
| `STREAM` tables | ephemeral realtime state with TTL |
| live subscriptions | push-based UI/state updates |
| topics + consumers | queue-like worker processing |
| vector search | semantic retrieval with cosine distance |
| storage commands | explicit flush/compact/manifest tooling |

## Release Information

For the latest published versions and artifacts, use:

- [KalamDB Releases](https://github.com/kalamdb/KalamDB/releases)

================================================================================
# Dex
URL: https://kalamdb.org/docs/server/integrations/dex
Source: content/server/integrations/dex.mdx

> Use Dex as a local or test OpenID Connect provider for KalamDB, including server.toml config, Admin UI login, and CLI browser/device flows.
# Dex

Dex is the simplest supported way to test KalamDB's current OIDC flow locally.

Use Dex when you want:

- a small standards-compliant OIDC issuer for local development
- repeatable browser and device-flow testing for the Admin UI and CLI
- a test IdP that matches KalamDB's single-provider OIDC model

## Local Dex Setup

The KalamDB repo already includes a Dex container and config:

```bash
cd KalamDB/docker/utils
docker compose up -d dex
```

Default local Dex values from the shared development config:

- issuer: `http://127.0.0.1:5556`
- public client ID: `client`
- Admin UI callback: `http://127.0.0.1:2900/ui/oauth/callback`
- CLI browser callback: `http://127.0.0.1:8787/callback`
- test user: `alice@example.org`
- test password: `kalamdb123`

## KalamDB `server.toml` Example

```toml
[auth]
jwt_secret = "replace-with-a-strong-random-secret-at-least-32-chars"
jwt_trusted_issuers = "kalamdb,http://127.0.0.1:5556"
allow_remote_setup = false
cookie_secure = false

[auth.local]
enabled = true

[auth.oidc]
enabled = true
display_name = "Dex"
issuer = "http://127.0.0.1:5556"
client_id = "client"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
broker_device_flow_enabled = false
device_authorization_endpoint = "http://127.0.0.1:5556/device/code"
```

For the public Dex client used in local development, leave `client_secret` unset.

## Admin UI Login

Make sure the Dex client allows this redirect URI:

```text
http://127.0.0.1:2900/ui/oauth/callback
```

Then open the Admin UI, choose the OIDC login button, and sign in as `alice@example.org` with password `kalamdb123`.

## CLI Login

### Browser login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc
```

This uses the local browser callback at `http://127.0.0.1:8787/callback`.

### Direct device login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser
```

Use this when you want to stay headless and let the CLI complete provider device flow directly.

### Brokered device login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser --brokered
```

Enable brokered mode only when the CLI host cannot reach Dex directly but can reach KalamDB.

## Notes On Production Use

Dex can be used beyond local development, but production use is an operational choice, not a KalamDB requirement.

Dex is a good fit when:

- you control the deployment environment
- you manage TLS, connector configuration, storage, and backups yourself
- you want a lightweight self-hosted OIDC layer

Many teams still prefer an existing enterprise IdP for production. From KalamDB's perspective, that is fine because the runtime only cares about one configured OIDC issuer.

## Related

- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)
- [Kalam CLI](https://kalamdb.org/docs/cli)
- [OIDC Model](https://kalamdb.org/docs/server/auth/providers)

================================================================================
# Firebase
URL: https://kalamdb.org/docs/server/integrations/firebase
Source: content/server/integrations/firebase.mdx

> Use Firebase as the single configured OIDC issuer for KalamDB. This page shows the current generic OIDC model, not the removed provider-specific config surface.
# Firebase

KalamDB no longer has Firebase-specific auth configuration. Firebase now fits the same single-provider OIDC model as Dex, Keycloak, Okta, Auth0, Entra ID, Google, and other standards-compliant issuers.

Use the generic OIDC docs and set the issuer to your Firebase project issuer.

## Example `server.toml`

```toml
[auth]
jwt_trusted_issuers = "kalamdb,https://securetoken.google.com/YOUR_PROJECT_ID"

[auth.oidc]
enabled = true
display_name = "Firebase"
issuer = "https://securetoken.google.com/YOUR_PROJECT_ID"
client_id = "YOUR_PROJECT_ID"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
```

If you need explicit audience validation, keep `client_id` set to the Firebase project ID.

## What Changed

Do not use or document these removed shapes anymore:

- `[oauth]`
- `[oauth.providers.firebase]`
- provider-coded identities such as `oidc:fbs:<subject>`

Current external identities use the Firebase/OIDC `sub` claim directly as the KalamDB `user_id`.

If you need explicit persisted users or elevated roles, create them with `issuer` and `subject`:

```sql
CREATE USER 'FIREBASE_UID'
  WITH OIDC '{"issuer":"https://securetoken.google.com/YOUR_PROJECT_ID","subject":"FIREBASE_UID"}'
  ROLE dba
  EMAIL 'user@example.com';
```

## Related

- [Authentication Overview](https://kalamdb.org/docs/server/auth)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)
- [Auto-Provisioning Users](https://kalamdb.org/docs/server/auth/auto-provisioning)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [OIDC Model](https://kalamdb.org/docs/server/auth/providers)

================================================================================
# Integrations
URL: https://kalamdb.org/docs/server/integrations
Source: content/server/integrations/index.mdx

> Integration guides for external services used with KalamDB. Learn how to use Dex for local OIDC testing and how to connect MinIO and Jaeger.
# Integrations

This section covers external tools and identity/storage integrations.

- [Dex](https://kalamdb.org/docs/server/integrations/dex)
- [MinIO (S3-Compatible)](https://kalamdb.org/docs/server/integrations/minio)
- [Jaeger](https://kalamdb.org/docs/server/integrations/jaeger)

KalamDB's current auth surface supports one external OIDC provider at a time. Dex is the recommended local and CI-friendly test issuer in the current docs set.
*** Add File: /Users/jamal/git/KalamSite/content/server/integrations/dex.mdx
---
title: "Dex"
description: "Use Dex as a local or test OpenID Connect provider for KalamDB, including server.toml config, Admin UI login, and CLI browser/device flows."
---

# Dex

Dex is the simplest supported way to test KalamDB's current OIDC flow locally.

Use Dex when you want:

- a small standards-compliant OIDC issuer for local development
- repeatable browser and device-flow testing for the Admin UI and CLI
- a test IdP that matches KalamDB's single-provider OIDC model

## Local Dex Setup

The KalamDB repo already includes a Dex container and config:

```bash
cd KalamDB/docker/utils
docker compose up -d dex
```

Default local Dex values from the shared development config:

- issuer: `http://127.0.0.1:5556`
- public client ID: `client`
- Admin UI callback: `http://127.0.0.1:2900/ui/oauth/callback`
- CLI browser callback: `http://127.0.0.1:8787/callback`
- test user: `alice@example.org`
- test password: `kalamdb123`

## KalamDB `server.toml` Example

```toml
[auth]
jwt_secret = "replace-with-a-strong-random-secret-at-least-32-chars"
jwt_trusted_issuers = "kalamdb,http://127.0.0.1:5556"
allow_remote_setup = false
cookie_secure = false

[auth.local]
enabled = true

[auth.oidc]
enabled = true
display_name = "Dex"
issuer = "http://127.0.0.1:5556"
client_id = "client"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
broker_device_flow_enabled = false
device_authorization_endpoint = "http://127.0.0.1:5556/device/code"
```

For the public Dex client used in local development, leave `client_secret` unset.

## Admin UI Login

Make sure the Dex client allows this redirect URI:

```text
http://127.0.0.1:2900/ui/oauth/callback
```

Then open the Admin UI, choose the OIDC login button, and sign in as `alice@example.org` with password `kalamdb123`.

## CLI Login

### Browser login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc
```

This uses the local browser callback at `http://127.0.0.1:8787/callback`.

### Direct device login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser
```

Use this when you want to stay headless and let the CLI complete provider device flow directly.

### Brokered device login

```bash
kalam login --instance local --url http://127.0.0.1:2900 --oidc --no-browser --brokered
```

Enable brokered mode only when the CLI host cannot reach Dex directly but can reach KalamDB.

## Notes On Production Use

Dex can be used beyond local development, but production use is an operational choice, not a KalamDB requirement.

Dex is a good fit when:

- you control the deployment environment
- you manage TLS, connector configuration, storage, and backups yourself
- you want a lightweight self-hosted OIDC layer

Many teams still prefer an existing enterprise IdP for production. From KalamDB's perspective, that is fine because the runtime only cares about one configured OIDC issuer.

## Related

- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)
- [Kalam CLI](https://kalamdb.org/docs/cli)
- [OIDC Model](https://kalamdb.org/docs/server/auth/providers)

================================================================================
# Jaeger
URL: https://kalamdb.org/docs/server/integrations/jaeger
Source: content/server/integrations/jaeger.mdx

> Set up Jaeger for distributed tracing with KalamDB. Configure OTEL exports via gRPC or HTTP, visualize traces, and debug production workloads.
# Jaeger

Use Jaeger as an OTLP-compatible backend for KalamDB distributed tracing.

## Download Compose Example

- [Download docker-compose.yml (Jaeger)](https://kalamdb.org/downloads/otel/docker-compose.yml)

## Docker Compose Details

The compose file runs:

```yaml
jaeger:
  image: jaegertracing/jaeger:latest
  environment:
    COLLECTOR_OTLP_ENABLED: "true"
  ports:
    - "16686:16686"   # Jaeger UI
    - "4317:4317"     # OTLP gRPC receiver
    - "4318:4318"     # OTLP HTTP receiver
```

Start it:

```bash
docker compose up -d
```

Jaeger UI: `http://127.0.0.1:16686`

## Connect KalamDB → Jaeger

### Via gRPC (default, port 4317)

```toml
[logging.otlp]
enabled = true
endpoint = "http://127.0.0.1:4317"
protocol = "grpc"
service_name = "kalamdb-server"
timeout_ms = 3000
```

### Via HTTP (port 4318)

```toml
[logging.otlp]
enabled = true
endpoint = "http://127.0.0.1:4318"
protocol = "http"
service_name = "kalamdb-server"
timeout_ms = 3000
```

The OTLP HTTP exporter appends `/v1/traces` automatically.

### Via Environment Variables

```bash

```

## What to Expect in Jaeger UI

After starting KalamDB with OTEL enabled:

1. Open `http://127.0.0.1:16686`.
2. Select the service name from the dropdown (default: `kalamdb-server`).
3. Click **Find Traces**.
4. You should see traces for API requests, SQL execution, and storage operations.

Each trace shows the full request lifecycle — from API entry through SQL parsing, query execution, and storage I/O.

## Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| No service in Jaeger dropdown | OTEL not enabled or no traffic yet | Set `enabled = true`, execute some SQL, then refresh |
| Service appears but no traces | Endpoint mismatch | Verify `protocol` matches the port: `grpc` → 4317, `http` → 4318 |
| Connection refused | Jaeger not running or wrong port | Run `docker compose ps` to verify Jaeger is up |
| Traces delayed | Batch export timeout | Default timeout is 3000ms; increase `timeout_ms` if network is slow |

## Related

For all OTEL configuration options and environment variable overrides, see [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel).

================================================================================
# Keycloak
URL: https://kalamdb.org/docs/server/integrations/keycloak
Source: content/server/integrations/keycloak.mdx

> Use Keycloak as the single configured OIDC issuer for KalamDB. This page shows the current generic OIDC model, not the removed provider-specific config surface.
# Keycloak

KalamDB no longer has Keycloak-specific auth configuration. Keycloak now fits the same single-provider OIDC model as Dex, Okta, Auth0, Entra ID, Google, and other standards-compliant issuers.

Use the generic OIDC docs and set the issuer to your Keycloak realm URL.

## Example `server.toml`

```toml
[auth]
jwt_trusted_issuers = "kalamdb,https://keycloak.example.com/realms/myrealm"

[auth.oidc]
enabled = true
display_name = "Keycloak"
issuer = "https://keycloak.example.com/realms/myrealm"
client_id = "kalamdb"
scopes = ["openid", "email", "profile"]
auto_provision = true
default_role = "user"
```

The issuer URL must exactly match the `iss` claim in Keycloak tokens.

## What Changed

Current external identities use the OIDC `sub` claim directly as the KalamDB `user_id`.

If you need explicit persisted users or elevated roles, create them with `issuer` and `subject`:

```sql
CREATE USER 'provider-subject'
  WITH OIDC '{"issuer":"https://keycloak.example.com/realms/myrealm","subject":"provider-subject"}'
  ROLE dba
  EMAIL 'alice@example.com';
```

## Related

- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)
- [OIDC Model](https://kalamdb.org/docs/server/auth/providers)

================================================================================
# MinIO (S3-Compatible)
URL: https://kalamdb.org/docs/server/integrations/minio
Source: content/server/integrations/minio.mdx

> Use MinIO with KalamDB S3 storage. Covers CONFIG fields, storage factory behavior, Docker Compose setup, timeout tuning, and bucket configuration.
# MinIO (S3-Compatible)

KalamDB uses the S3 storage backend for MinIO. You configure MinIO through `CREATE STORAGE ... TYPE 's3'` and the storage `CONFIG` JSON.

## Quick Start with Docker Compose

The KalamDB repository includes a MinIO service in `docker/utils/docker-compose.yml`:

```yaml
minio:
  image: minio/minio:latest
  environment:
    MINIO_ROOT_USER: minioadmin
    MINIO_ROOT_PASSWORD: minioadmin
    MINIO_CI: "on"
  ports:
    - "9120:9000"   # S3 API
    - "9121:9001"   # Console UI
  volumes:
    - ./minio-data:/data
```

Start it:

```bash
docker compose up minio -d
```

MinIO Console: `http://127.0.0.1:9121` (login: `minioadmin` / `minioadmin`)  
S3 API Endpoint: `http://127.0.0.1:9120`

The compose entrypoint auto-creates a `kalamdb-test` bucket and sets public anonymous access for development.

## S3 CONFIG Fields

For S3-compatible backends (including MinIO), KalamDB supports these `CONFIG` JSON fields:

| Field | Type | Default | Notes |
|---|---|---|---|
| `type` | string | — | Must be `"s3"` |
| `region` | string | `"us-east-1"` | Falls back to `us-east-1` in factory if omitted |
| `endpoint` | string | `null` | Custom endpoint for MinIO / S3-compatible |
| `allow_http` | boolean | `false` | Must be `true` for non-TLS endpoints |
| `access_key_id` | string | `null` | Static credentials |
| `secret_access_key` | string | `null` | Static credentials |
| `session_token` | string | `null` | Temporary session credentials |

## Example: Register MinIO Storage

```sql
CREATE STORAGE s3_minio
  TYPE 's3'
  NAME 'MinIO Local Storage'
  DESCRIPTION 'MinIO bucket for local development'
  BASE_DIRECTORY 's3://kalamdb-test/'
  CONFIG '{
    "type": "s3",
    "region": "us-east-1",
    "endpoint": "http://127.0.0.1:9120",
    "allow_http": true,
    "access_key_id": "minioadmin",
    "secret_access_key": "minioadmin"
  }'
  SHARED_TABLES_TEMPLATE '{namespace}/{tableName}'
  USER_TABLES_TEMPLATE '{namespace}/{tableName}/{userId}';
```

## S3 Factory Behavior

When a custom `endpoint` is set (as with MinIO), the factory applies special handling:

- **Path-style requests**: `virtual_hosted_style_request` is set to `false` automatically. This is required for MinIO and most S3-compatible services.
- **Timeout skip**: `ClientOptions` (request/connect timeouts from `[storage.remote_timeouts]`) are **not applied** to S3-compatible endpoints. This avoids timeout conflicts with non-AWS backends. Timeouts only apply to standard AWS S3.
- **Region fallback**: If `region` is omitted, it defaults to `"us-east-1"`.

## Notes on CONFIG

- Use an `s3://...` `BASE_DIRECTORY` for S3-compatible storage.
- `allow_http = true` is required for any non-TLS endpoint (local MinIO, test environments).
- The `type` field in the CONFIG JSON must match the `TYPE` in the SQL command.

## Remote Timeout Tuning (`server.toml`)

Remote object store timeouts are configured globally in `server.toml`:

```toml
[storage.remote_timeouts]
request_timeout_secs = 60
connect_timeout_secs = 10
```

> **Important**: These timeouts apply to **standard AWS S3 only**. They are skipped for S3-compatible endpoints (MinIO, etc.) to avoid conflicts.

## Credential Placement

Recommended options:

- **Development**: Put credentials in `CONFIG` JSON directly.
- **Production (AWS S3)**: Use IAM instance roles or environment-based AWS credential chains — omit `access_key_id`/`secret_access_key` from CONFIG.
- **Production (MinIO)**: Use scoped credentials; rotate regularly.
- Avoid committing production secrets to source control.

## Verify Storage Health

After creating the storage, verify connectivity:

```sql
STORAGE CHECK s3_minio;
STORAGE CHECK s3_minio EXTENDED;
```

## Other Cloud Storage Backends

KalamDB also supports GCS and Azure Blob Storage. See
[/docs/server/sql-reference/storage](https://kalamdb.org/docs/server/sql-reference/storage) for all backend types and
CONFIG formats.

## Related Resources

- [/docs/server/sql-reference/storage](https://kalamdb.org/docs/server/sql-reference/storage)
- [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests)

================================================================================
# v0.5.3-rc.1 Release
URL: https://kalamdb.org/docs/server/release-v0-5-3-rc-1
Source: content/server/release-v0-5-3-rc-1.mdx

> KalamDB v0.5.3-rc.1 pre-release notes for the June 17 CLI workflow update, benchmark improvement, and docs destinations.
# KalamDB v0.5.3-rc.1 Release

KalamDB v0.5.3-rc.1 is the June 17 pre-release for the local CLI project workflow: `kalam init`, `kalam dev`, schema apply, generated SDK artifacts, and supervised app logs.

## v0.5.3-rc.1 Performance Benchmark

The synced benchmark artifact for `0.5.3-rc.1` reports **38/38 benchmarks passed** with **213,667.71 ms measured benchmark time**.

That is **8.0% faster measured benchmark time** than `0.5.2-rc.1`, which reported 232,155.85 ms.

- [Open the v0.5.3-rc.1 HTML benchmark report](https://kalamdb.org/benchmark/results/bench-2026-06-17-131609-0-5-3-rc-1.html)
- [View all synced benchmark reports](https://kalamdb.org/docs/server/benchmarking)
- [Open the GitHub release](https://github.com/kalamdb/KalamDB/releases/tag/v0.5.3-rc.1)

## CLI Project Workflow

The release makes the CLI project loop the recommended path for local development:

1. Run [Project Init](https://kalamdb.org/docs/cli/init) to create `kalam.toml`, `schema.sql`, migrations, optional starter files, and package-manager wiring.
2. Run [Local Development](https://kalamdb.org/docs/cli/dev) to start or reuse a KalamDB server, apply schema changes, regenerate SDK code, and stream supervised process logs.
3. Use [Migrations](https://kalamdb.org/docs/cli/migrations) when schema changes need a committed rollout path.
4. Run `kalam db reset` to clear local server data (`kalam/server/data`, `kalam/server/logs`) when you need a fresh database on the next `kalam dev`.

Local projects scaffolded with `--server-mode local` use `root` / `kalamdb123` in `kalam/server/server.toml` and `.env`.

```bash
mkdir my-agent-app && cd my-agent-app
kalam init --yes --name my-agent-app --languages typescript --package-manager pnpm
kalam dev
```

To start over:

```bash
kalam db reset
kalam dev
```

## Release Post

For the narrative announcement with the CLI workflow image, benchmark summary, and release links, read the [KalamDB v0.5.3-rc.1 pre-release blog post](https://kalamdb.org/blog/kalamdb-v0-5-3-rc-1-pre-release).

================================================================================
# v0.5.4-rc.1 Release
URL: https://kalamdb.org/docs/server/release-v0-5-4-rc-1
Source: content/server/release-v0-5-4-rc-1.mdx

> KalamDB v0.5.4-rc.1 pre-release notes for INSERT RETURNING, ON CONFLICT upsert, Live OKF Context Sync, and CLI and auth hardening.
# KalamDB v0.5.4-rc.1 Release

KalamDB v0.5.4-rc.1 is the June 25 pre-release for PostgreSQL-style `INSERT ... RETURNING` and `ON CONFLICT` upsert, the Live OKF Context Sync example, and CLI, auth, and test hardening.

## SQL: INSERT ... RETURNING

`INSERT` statements can now return affected rows directly in the SQL response.

**Supported syntax:**

```sql
INSERT INTO users (id, name, age) VALUES (1, 'Nader', 3) RETURNING *;
INSERT INTO users (id, name, age) VALUES (1, 'Nader', 3) RETURNING id, name;
```

**Behavior:**

- `RETURNING *` returns all table columns in schema order
- `RETURNING col1, col2, ...` returns a selected projection
- Works for single-row and multi-row `INSERT`
- API responses include the returned rows in the standard `results[].rows` shape
- Parameterized inserts support `RETURNING` over HTTP and through the SDKs

## SQL: INSERT ... ON CONFLICT (upsert)

PostgreSQL-style upsert is supported on the primary key conflict target.

```sql
INSERT INTO users (id, name, age) VALUES (1, 'Nader Updated', 5)
ON CONFLICT (id) DO UPDATE
  SET name = EXCLUDED.name, age = EXCLUDED.age
RETURNING *;

INSERT INTO users (id, name, age) VALUES (1, 'Ignored', 99)
ON CONFLICT (id) DO NOTHING
RETURNING *;
```

| Feature | Behavior |
| --- | --- |
| `ON CONFLICT (pk) DO UPDATE SET ...` | Updates existing row; unassigned columns are preserved |
| `EXCLUDED.column` | References the proposed insert values |
| `ON CONFLICT (pk) DO NOTHING` | Skips conflicting rows |
| `DO UPDATE ... WHERE` | Applies update only when the predicate is true |
| `RETURNING` with `DO NOTHING` | Returns zero rows for skipped conflicts |
| Multi-row upsert | One returned row per input row (insert or update) |
| User tables | Conflicts are scoped per user (RLS isolation) |
| Shared tables | Conflicts are global on the primary key |

**Limitations (current release):**

- Conflict target must be the table's **primary key** column
- `ON CONFLICT ON CONSTRAINT` is not supported
- Non-primary-key unique targets are rejected with a clear error

## TypeScript ORM (`@kalamdb/orm`)

Drizzle-style upsert and returning now work end-to-end:

```typescript
const rows = await db
  .insert(users)
  .values({ id: 1, name: 'Nader', age: 3 })
  .returning();

const updated = await db
  .insert(users)
  .values({ id: 1, name: 'Nader Updated', age: 5 })
  .onConflictDoUpdate({
    target: users.id,
    set: { name: sql`excluded.name`, age: sql`excluded.age` },
  })
  .returning();
```

FILE uploads through Drizzle use `kalamFile(name, blob)` in `.values()` / `.set()`. `kalamDriver()` detects upload params and routes to multipart `queryWithFiles()` — no separate wrapper per insert:

```typescript

await db.insert(attachments).values({
  id: 'att_1',
  file_data: kalamFile('upload', selectedFile),
});
```

See [Live OKF Context Sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync) for a folder-sync example that upserts FILE columns with `onConflictDoUpdate`.

## Live OKF Context Sync example

A new showcase app under `examples/live-okf-context-sync/` demonstrates syncing a **Google Open Knowledge Format (OKF)** folder into KalamDB for multi-client AI agent context.

- Per-user OKF Markdown folders synced into KalamDB via a Node sync worker
- File metadata in a user-scoped SQL table; bytes via KalamDB `FILE` upload/download
- Local folder watch (push) + `liveTable()` pull with a SQLite hash cache under `.index/`
- Schema-first workflow: `kalam/schema.sql` is the source of truth; TypeScript types are generated via `@kalamdb/orm`
- Live subscriptions so every sync worker sees metadata changes in realtime
- Delete local `data/` and restart — files restore from the server (KalamDB is the source of truth)

Docs: [Live OKF Context Sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync)

## CLI and dev workflow

- **`kalam dev` improvements** — stronger prechecks for auth, paths, and server readiness before starting dev processes
- **Project init refactor** — cleaner scaffolding for TypeScript projects and connection URL handling
- **Self-update** — more reliable version comparison (version first, then build date for same-version rebuilds)
- **Update-check tests** — version-bump-proof; no longer require editing hardcoded release numbers on each release
- **`run-tests.sh`** — improved running-server credential validation and supplementary suite orchestration

## Security and auth

- **OIDC bearer auth fix** — closes a repository bypass where OIDC bearer tokens could skip expected authorization checks
- **Auth endpoint hardening** — login, setup, and OIDC exchange paths tightened for consistency

## Upgrade notes

1. **Rebuild or update** server and CLI to `0.5.4-rc.1`
2. **TypeScript projects** — bump `@kalamdb/*` packages together; run `npm install` to refresh lockfiles
3. **Upsert adoption** — replace read-then-write patterns with `ON CONFLICT ... DO UPDATE` where appropriate
4. **RETURNING** — use instead of a follow-up `SELECT` after inserts when you need the written row
5. **OKF example** — requires `kalam dev` (starts server, applies schema, generates ORM types)

- [Open the GitHub release](https://github.com/kalamdb/KalamDB/releases/tag/v0.5.4-rc.1)
- [Full changelog](https://github.com/kalamdb/KalamDB/compare/v0.5.3-rc.1...v0.5.4-rc.1)

## Release post

For the narrative announcement with SQL examples, the OKF context sync summary, and release links, read the [KalamDB v0.5.4-rc.1 pre-release blog post](https://kalamdb.org/blog/kalamdb-v0-5-4-rc-1-pre-release).

================================================================================
# SDK Version Compatibility
URL: https://kalamdb.org/docs/server/sdk/compatibility
Source: content/server/sdk/compatibility.mdx

> Compare KalamDB server, pg_kalam, TypeScript, Dart, Flutter, and Rust SDK versions to choose supported release pairings for apps.
# SDK Version Compatibility

Use this SDK version compatibility table to choose the KalamDB server, pg_kalam extension, and client SDK versions that belong together for a deployment.

## Compatibility Matrix

## Selection Guidance

- For new apps, start with the latest server docs and latest SDK docs unless you are pinned to a specific release candidate.
- Keep the server, pg_kalam extension, and SDKs on the same row when you need repeatable staging or production builds.
- When upgrading one component across rows, run a SQL request, an authenticated request, and a realtime subscription smoke test before rolling forward.

## Related Docs

- [TypeScript SDK setup](https://kalamdb.org/docs/ts-sdk/setup)
- [Dart / Flutter SDK setup](https://kalamdb.org/docs/dart-sdk/setup)
- [pg_kalam SQL syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)
- [HTTP API reference](https://kalamdb.org/docs/server/api/http-reference)

================================================================================
# SDKs & Client Libraries
URL: https://kalamdb.org/docs/server/sdk
Source: content/server/sdk/index.mdx

> Official KalamDB client libraries. Connect with TypeScript, Dart/Flutter, Rust, Python SDK docs, or the HTTP and WebSocket APIs.
# SDKs & Client Libraries

> **Status:** TypeScript, Dart/Flutter, and Rust SDKs are documented and available today. The Rust crate [`kalam-client`](https://crates.io/crates/kalam-client) is published on crates.io (beta). Python SDK docs exist for async SQL and live queries, but package publishing may still be in progress.

KalamDB client libraries:

- [TypeScript SDK](https://kalamdb.org/docs/ts-sdk) for browser/Node.js apps and worker services
- [Dart / Flutter SDK](https://kalamdb.org/docs/dart-sdk) for mobile and cross-platform app clients
- [Rust SDK](https://kalamdb.org/docs/rust-sdk) — [`kalam-client` on crates.io](https://crates.io/crates/kalam-client) for native async Rust services and topic workers
- [Python SDK](https://kalamdb.org/docs/python-sdk) for async SQL and live query examples; see [Python SDK status](https://kalamdb.org/docs/server/sdk/python) before relying on package installation

## Which One To Start With

- Building frontend + backend JS/TS apps: start at [TypeScript Setup](https://kalamdb.org/docs/ts-sdk/setup)
- Building Dart/Flutter apps: start at [Dart Setup](https://kalamdb.org/docs/dart-sdk/setup)
- Building Rust workers/services: start at [Rust SDK](https://kalamdb.org/docs/rust-sdk)
- Building Python services today: review [Python SDK](https://kalamdb.org/docs/python-sdk), then fall back to [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference) if you need a stable package install path

## Shared Concepts Across SDKs

- JWT bearer auth for SQL and realtime flows
- SQL execution over `/v1/api/sql`
- WebSocket realtime subscriptions over `/v1/ws`
- Topic consume/ack for worker processing
- FILE column uploads and downloads (`FileRef`, `BoundFileRef`, `TableId` in Rust; `BoundFileRef` in TypeScript)
- Vector search through regular SQL (`EMBEDDING`, `COSINE_DISTANCE`, and indexed queries)

For protocol-level integration details, see [API Reference](https://kalamdb.org/docs/server/api).

================================================================================
# Python SDK Status
URL: https://kalamdb.org/docs/server/sdk/python
Source: content/server/sdk/python/index.mdx

> Check the KalamDB Python SDK status, current docs, and fallback HTTP/WebSocket API paths before relying on package installation.
# Python SDK Status

KalamDB has Python SDK docs for async SQL helpers and live query streams:

- [Python SDK overview](https://kalamdb.org/docs/python-sdk)
- [Python Live Queries](https://kalamdb.org/docs/python-sdk/subscriptions)

Release metadata lists a `kalamdb` Python SDK package for the current `0.5.3-rc.1` line,
but the public package install path is not linked from the site config yet. Treat Python
package publishing as in-progress until the docs include a concrete install command.

## Current Recommendation For Python Services

For production integrations that need a stable install path, use the protocol docs
directly:

- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)
- [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol)
- [Authentication & Bootstrap](https://kalamdb.org/docs/server/getting-started/authentication)

For Python-facing examples, use the Python SDK docs above as the API shape reference,
then verify package availability before shipping.

## Expected Scope

- SQL execution over the HTTP API
- JWT authentication helpers
- realtime subscriptions
- topic consume/ack worker APIs
- FILE column upload helpers

================================================================================
# Security & Best Practices
URL: https://kalamdb.org/docs/server/security
Source: content/server/security/index.mdx

> Operational security baseline for KalamDB deployments. Learn about auth hardening, TLS, CORS controls, and abuse protection.
# Security

This chapter is for operators deploying KalamDB in staging/production.

## Security Baseline Checklist

1. Serve API behind HTTPS (TLS at edge proxy/load balancer).
2. Set strong `auth.jwt_secret` and keep it out of source control.
3. Keep `auth.allow_remote_setup = false` after initial bootstrap.
4. Restrict CORS and WebSocket origins to known domains.
5. Keep rate limiting and request-size limits enabled.
6. Restrict setup/health/admin-sensitive routes to trusted networks.
7. For clusters, enable `cluster.rpc_tls` (mTLS between nodes).

## Baseline Config Example

```toml
[auth]
jwt_secret = "replace-with-strong-random-secret-32-plus-chars"
cookie_secure = true
allow_remote_setup = false

[rate_limit]
enable_connection_protection = true
max_auth_requests_per_ip_per_sec = 20
max_requests_per_ip_per_sec = 200
max_connections_per_ip = 100

[security]
max_request_body_size = 10485760
max_ws_message_size = 1048576
strict_ws_origin_check = true
allowed_ws_origins = ["https://app.example.com"]

[security.cors]
allowed_origins = ["https://app.example.com", "https://admin.example.com"]
allow_credentials = true
```

## High-Risk Misconfigurations To Avoid

- Wildcard browser origins in production
- Disabled rate-limit middleware in public deployments
- Short/static JWT secrets shared across environments
- Leaving remote setup enabled permanently
- Exposing cluster RPC ports publicly

## Incident Response Priorities

1. Rotate compromised JWT secrets/certificates
2. Disable compromised users/service accounts
3. Tighten ingress and rate limits during active abuse
4. Preserve logs for forensic review

## Related Docs

- [Production Checklist](https://kalamdb.org/docs/server/security/production-checklist)
- [Authentication Providers](https://kalamdb.org/docs/server/auth)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)
- [Auto-Provisioning Users](https://kalamdb.org/docs/server/auth/auto-provisioning)
- [Firebase](https://kalamdb.org/docs/server/integrations/firebase)
- [Rate Limiting](https://kalamdb.org/docs/server/security/rate-limits)
- [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Keycloak](https://kalamdb.org/docs/server/integrations/keycloak)
- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)

================================================================================
# Production Security Checklist
URL: https://kalamdb.org/docs/server/security/production-checklist
Source: content/server/security/production-checklist.mdx

> Step-by-step checklist and hardened configuration reference for running KalamDB safely in production. Covers secrets, tokens, CORS, TLS, RBAC, rate limits, cluster mTLS, and incident response.
# Running KalamDB in Production

This guide walks you through every setting, secret, and network boundary you need to lock down before exposing a KalamDB server to the public internet or an untrusted network.

Work through the checklist top-to-bottom. Each section is independently actionable — you can turn a single item on and restart the server between steps.

> **Threat model assumption**: attackers can reach your HTTP and cluster RPC ports. Your defenses must not rely on network isolation alone.

---

## Quick Checklist

Copy this into your deploy runbook. Every box must be ticked before you open the port.

### Network & TLS
- [ ] `server.host = "127.0.0.1"` (or explicitly a private interface); never `0.0.0.0` unless you need remote access
- [ ] Terminate TLS at an edge proxy (nginx, Caddy, ALB, Cloudflare) — KalamDB itself serves plain HTTP
- [ ] HTTP server sits behind a firewall/security-group that only allows the edge proxy
- [ ] Cluster RPC port (`cluster.rpc_addr`) is **not** reachable from the public internet
- [ ] For multi-node clusters: `rpc_tls.enabled = true` with `require_client_cert = true`

### Secrets
- [ ] `auth.jwt_secret` is at least 32 bytes, random, unique per environment
- [ ] JWT secret is injected via env var or secret manager, **never** committed to source
- [ ] Root/admin password is changed from any default and stored in a secret manager
- [ ] `KALAMDB_ROOT_PASSWORD` env var is cleared from the shell history after seeding
- [ ] OAuth/OIDC client secrets are in env vars, not the config file

### Auth & RBAC
- [ ] `auth.allow_remote_setup = false` after first-run bootstrap
- [ ] Every human user has a named account, not the built-in `root`
- [ ] Service accounts use the `service` role, never `dba`/`system`
- [ ] Password policy tuned for local auth (`auth.local.min_password_length`, `auth.local.bcrypt_cost`)
- [ ] Access token expiry ≤ 1 hour; refresh token expiry ≤ 7 days

### Origins & Cookies
- [ ] `security.cors.allowed_origins` is an explicit allowlist — no `"*"`
- [ ] `security.cors.allow_credentials = true` **only** when origins are an explicit allowlist
- [ ] `security.strict_ws_origin_check = true`
- [ ] Auth cookies served over HTTPS: `auth.cookie_secure = true`, `SameSite=Strict`, `HttpOnly`

### Abuse Controls
- [ ] `rate_limit.enable_connection_protection = true`
- [ ] `rate_limit.max_auth_requests_per_ip_per_sec` tuned (start at 10–20)
- [ ] `rate_limit.max_connections_per_ip` tuned (start at 100)
- [ ] `security.max_request_body_size` tuned to your real upload size
- [ ] `security.max_ws_message_size` tuned to your real message size

### Observability
- [ ] Audit logs shipped to a separate write-only sink (e.g., SIEM, S3 with object lock)
- [ ] Metrics endpoint exposed only to the internal monitoring network
- [ ] Alerts configured on: failed logins/min, 401/403 rate, new user creation, role changes

---

## 1. Bind Addresses & TLS

KalamDB does not terminate TLS itself. Run it behind a reverse proxy that provides HTTPS and forwards to KalamDB on localhost.

### Recommended topology

```
[ client ] ──HTTPS──► [ nginx/Caddy/ALB ] ──HTTP(loopback)──► [ kalamdb :2900 ]
                                                              [ kalamdb :2910 cluster mTLS ]
```

### Config

```toml
[server]
host = "127.0.0.1"     # loopback only; proxy handles remote clients
port = 2900

[cluster]
# Bind cluster RPC to the private cluster network interface only.
rpc_addr = "10.0.1.15:2910"

[rpc_tls]
enabled = true
require_client_cert = true
ca_cert = "/etc/kalamdb/tls/cluster-ca.pem"
server_cert = "/etc/kalamdb/tls/node.pem"
server_key = "/etc/kalamdb/tls/node.key"
```

If your cluster spans more than loopback nodes, KalamDB will refuse to start without `rpc_tls.enabled = true`. Do not work around this check.

---

## 2. JWT Secrets & Token Hygiene

### Secret requirements
- **Length**: minimum 32 bytes (64+ recommended)
- **Source**: cryptographic RNG (`openssl rand -base64 48`), not a passphrase
- **Rotation**: rotate on compromise, personnel change, or at least annually
- **Scope**: unique per environment (dev, staging, prod never share)

Generate and inject via environment:

```bash

```

```toml
[auth]
# Leave unset in the file when you set KALAMDB_AUTH_JWT_SECRET in the env.
# jwt_secret = "..."
access_token_expiry_hours = 1
refresh_token_expiry_hours = 168   # 7 days
```

KalamDB refuses to start on non-loopback binds if the secret is empty, a known-weak placeholder, or shorter than 32 bytes.

### Rotation procedure

1. Generate a new secret.
2. Deploy it to every node simultaneously.
3. Bounce the service.
4. All outstanding tokens will fail verification — clients must re-login.

> Do not re-use an old secret. There is no "grace period" key ring; rotation is abrupt by design.

### Where tokens are exposed

| Surface | Carries token? | Notes |
|---|---|---|
| `Authorization: Bearer …` header | Yes | Preferred for server-to-server |
| Auth cookies | Yes | Browser flows; always set `Secure`, `HttpOnly`, `SameSite=Strict` |
| WebSocket subprotocol | Yes | Sent once during upgrade; never in URL query |
| Logs | **No** | Log redaction is enabled by default; keep it enabled |

---

## 3. Admin Bootstrap & Setup

On first boot, KalamDB seeds a `root` user. What happens next depends on your configuration.

### Option A — Seeded password (recommended for remote deployments)

```bash

# run the server once so the password is hashed and persisted
# then unset the env var and store the password in your secret manager
unset KALAMDB_ROOT_PASSWORD
```

### Option B — Remote setup endpoint (one-time)

If you cannot set the env var, allow the setup endpoint for the first call only:

```toml
[auth]
allow_remote_setup = true   # flip back to false immediately after setup
```

**Set `allow_remote_setup = false` and restart after you complete setup.** Leaving it enabled lets anyone who can reach the endpoint attempt to seed credentials.

### Option C — Localhost-only root

If you only administer via an SSH tunnel, leave the password empty. KalamDB will refuse remote logins for `root` and only accept localhost calls. Create named admin users via `CREATE USER` for remote access.

---

## 4. Password Policy

Minimum controls enforced by KalamDB:
- Minimum length (configurable via `auth.local.min_password_length`)
- Max length (configurable via `auth.local.max_password_length`; bcrypt still uses the first 72 bytes)
- Bcrypt hash with configurable cost (`auth.local.bcrypt_cost`, default `12`)
- Optional complexity rules via `auth.local.enforce_password_complexity`
- Rejected common-password list
- Constant-time verification
- Generic error messages (no user enumeration)

Recommended runtime:

```toml
[auth.local]
enabled = true
min_password_length = 12
bcrypt_cost = 12
enforce_password_complexity = false
```

Push stricter composition requirements (for example character classes) through `auth.local.enforce_password_complexity` or your external identity provider. KalamDB keeps complexity off by default to avoid false rejections of passphrases.

---

## 5. CORS & WebSocket Origins

### CORS for the browser admin UI

```toml
[security.cors]
allowed_origins = [
    "https://admin.example.com",
    "https://app.example.com",
]
allow_credentials = true
allowed_methods  = ["GET", "POST", "OPTIONS"]
allowed_headers  = ["Authorization", "Content-Type"]
max_age          = 600
```

Hard rules:
- Never combine `allowed_origins = ["*"]` with `allow_credentials = true`. Browsers reject it, and KalamDB will refuse to start with that combination.
- Use scheme + host + port — origin matching is exact.
- Never include localhost origins in production config.

### WebSocket origins

```toml
[security]
strict_ws_origin_check = true
allowed_ws_origins = ["https://app.example.com"]
```

`strict_ws_origin_check = true` rejects connections that omit the `Origin` header (non-browser tooling must set it explicitly). Leave empty `allowed_ws_origins` only on loopback dev installs.

---

## 6. Rate Limiting & Connection Protection

Even with auth in place, rate limits protect against credential stuffing, SQL abuse, and connection exhaustion.

```toml
[rate_limit]
enable_connection_protection = true

# Auth endpoints (login, refresh, setup, WS auth)
max_auth_requests_per_ip_per_sec = 10

# Per-IP concurrent connection cap
max_connections_per_ip = 100

# Pre-auth request flood protection
max_requests_per_ip_per_sec = 200

# Per-user query rate (SQL endpoint)
max_queries_per_sec = 100

# WebSocket message flood protection
max_messages_per_sec = 50

# Temporary ban window for abusive IPs
ban_duration_seconds = 300
```

If you run behind a reverse proxy, configure `security.trusted_proxy_ranges` so that rate limits attribute requests to the real client IP via `X-Forwarded-For`. **Do not** set this to `0.0.0.0/0` — that lets any caller spoof any IP and bypass the limits.

```toml
[security]
trusted_proxy_ranges = ["10.0.0.0/8", "172.16.0.0/12"]
```

---

## 7. RBAC & Least Privilege

KalamDB has four roles. Use the lowest one that works.

| Role | Typical use | Can do |
|---|---|---|
| `user` | App end-users, SDK clients | Read/write their own namespaces, run DML |
| `service` | Backend services, pub/sub consumers | DML + topic consume/ack |
| `dba` | Database administrators | DDL, manage users except `system` |
| `system` | Reserved for the server itself | Everything (do not create new ones) |

Rules of thumb:
- Application tokens should be `user` or `service`, never `dba`.
- Create one `dba` per human administrator; audit their usage.
- `EXECUTE AS '<user_id>'` follows the role hierarchy: system can target system/DBA/service/user, DBA can target DBA/service/user, service can target service/user, and regular users can only target themselves.
- System tables (`system.*`) are read/write restricted to `dba` and `system`. This is enforced inside the query planner, including through subqueries, CTEs, and views.

### On role demotion
When you demote or lock a user, rotate their tokens. KalamDB re-validates the DB role on each token refresh and invalidates tokens whose `token_generation` is older than the DB row — but already-issued access tokens remain valid until `access_token_expiry_hours`. Keep access-token TTLs short.

---

## 8. Request Size Guards & File Uploads

```toml
[security]
max_request_body_size = 10485760   # 10 MiB; tune to real payload size
max_ws_message_size   = 1048576    # 1 MiB
```

For file-accepting endpoints (multipart `FILE("name")` placeholders, exports), KalamDB:
- Validates path components against an allowlist (alphanumeric, `-`, `_`, `.`)
- Canonicalises paths and rejects symlink escape
- Returns 403 to non-owners trying to download another user's files or exports

Operators should additionally:
- Put an object-storage lifecycle on the exports bucket (e.g., delete after N days)
- Scan uploaded files with your AV/virus pipeline before re-serving them to users

---

## 9. Cluster RPC Hardening

For anything beyond a single-node loopback cluster:

```toml
[rpc_tls]
enabled = true
require_client_cert = true
ca_cert = "/etc/kalamdb/tls/cluster-ca.pem"
server_cert = "/etc/kalamdb/tls/node-1.pem"
server_key = "/etc/kalamdb/tls/node-1.key"

[cluster]
rpc_addr = "10.0.1.15:2910"   # private interface only
```

Notes:
- The cluster RPC port carries Raft consensus, `ForwardSql`, `Ping`, and node-info RPCs. Treat it as equally sensitive to the data port.
- `ForwardSql` additionally re-validates the caller's Bearer JWT on the receiving node — mTLS is defense-in-depth, not the only layer.
- Use a private CA unique to the cluster. Do not reuse your edge-TLS certificate chain for cluster mTLS.
- Rotate node certificates independently of JWT secrets.

---

## 10. Logging, Audit & Observability

- KalamDB emits structured JSON logs. SQL statements are redacted before logging; do not re-enable raw SQL in log formatters.
- Enable the audit stream to capture user logins, user creation/modification, role changes, impersonation events, and admin export downloads.
- Ship audit logs to an append-only sink (S3 with object lock, SIEM).
- Expose Prometheus metrics on the internal network only — never on the public API port.

### Suggested alerts
- `auth_failed_logins_per_minute > 20`
- `auth_403_forbidden_per_minute > 50`
- `user_role_change_events > 0` (human review on every change)
- `impersonation_events` grouped by actor
- Cluster RPC connection count deviation > 3σ

---

## 11. Incident Response Playbook

When credentials, a token secret, or a node are suspected compromised:

1. **Contain** — rotate `auth.jwt_secret`. All outstanding tokens become invalid immediately.
2. **Lock down** — set `auth.allow_remote_setup = false`, revoke compromised user accounts (`ALTER USER … LOCK`).
3. **Rotate** — regenerate cluster mTLS material, redeploy, force clients to re-authenticate.
4. **Narrow** — temporarily lower `max_auth_requests_per_ip_per_sec` and `max_connections_per_ip`.
5. **Preserve** — snapshot logs, audit stream, and RocksDB backup before further changes.
6. **Review** — audit `system.users` for unexpected role changes, `system.jobs` for unexpected admin jobs, exports directory for unexpected downloads.
7. **Post-mortem** — document the breach path, fix the configuration gap, add a regression check.

---

## 12. High-Risk Misconfigurations To Avoid

KalamDB will refuse to start on several of these. The rest are human errors we see most often.

- `server.host = "0.0.0.0"` with no edge proxy and default JWT secret
- `security.cors.allowed_origins = ["*"]` combined with `allow_credentials = true`
- `security.strict_ws_origin_check = false` on a public WebSocket endpoint
- `auth.allow_remote_setup = true` left on after bootstrap
- `auth.jwt_secret` shared across environments, or shorter than 32 bytes
- `security.trusted_proxy_ranges` containing `0.0.0.0/0` or `::/0`
- Running a multi-node cluster with `rpc_tls.enabled = false`
- Handing out `dba` or `system` role tokens to applications
- Storing secrets in `server.toml` committed to Git

---

## Baseline Production `server.toml`

Drop-in starting point. Fill in real values, inject secrets via env.

```toml
[server]
host = "127.0.0.1"
port = 2900
enable_http2 = true

[auth]
# jwt_secret intentionally omitted — supplied via KALAMDB_JWT_SECRET
allow_remote_setup = false
cookie_secure = true
jwt_expiry_hours = 24

[auth.local]
enabled = true
min_password_length = 12
bcrypt_cost = 12
enforce_password_complexity = false

[rate_limit]
enable_connection_protection = true
max_auth_requests_per_ip_per_sec = 10
max_connections_per_ip = 100
max_requests_per_ip_per_sec = 200
max_queries_per_sec = 100
max_messages_per_sec = 50
ban_duration_seconds = 300

[security]
max_request_body_size = 10485760
max_ws_message_size = 1048576
strict_ws_origin_check = true
allowed_ws_origins = ["https://app.example.com"]
trusted_proxy_ranges = ["10.0.0.0/8"]

[security.cors]
allowed_origins = ["https://admin.example.com", "https://app.example.com"]
allow_credentials = true
allowed_methods = ["GET", "POST", "OPTIONS"]
allowed_headers = ["Authorization", "Content-Type"]
max_age = 600

[rpc_tls]
enabled = true
require_client_cert = true
ca_cert = "/etc/kalamdb/tls/cluster-ca.pem"
server_cert = "/etc/kalamdb/tls/node.pem"
server_key = "/etc/kalamdb/tls/node.key"

[cluster]
rpc_addr = "10.0.1.15:2910"
```

---

## Related Docs

- [Security Overview](https://kalamdb.org/docs/server/security)
- [Rate Limiting](https://kalamdb.org/docs/server/security/rate-limits)
- [Authentication](https://kalamdb.org/docs/server/auth)
- [Token Auth](https://kalamdb.org/docs/server/auth/token-auth)
- [OIDC & Issuer Trust](https://kalamdb.org/docs/server/configurations/oidc)
- [Advanced Configuration](https://kalamdb.org/docs/server/configurations/advanced)
- [OpenTelemetry (OTEL)](https://kalamdb.org/docs/server/configurations/otel)

================================================================================
# Rate Limiting
URL: https://kalamdb.org/docs/server/security/rate-limits
Source: content/server/security/rate-limits.mdx

> How to tune KalamDB rate-limit settings for SQL queries, auth endpoints, WebSocket connections, and connection abuse controls in production.
# Rate Limiting

KalamDB applies rate limiting at SQL, auth, WebSocket, and per-IP connection layers.

## Key Settings

| Key | Scope |
|-----|-------|
| `rate_limit.max_queries_per_sec` | per-user SQL request rate |
| `rate_limit.max_messages_per_sec` | per-WebSocket incoming message rate |
| `rate_limit.max_subscriptions_per_user` | active live subscriptions per user |
| `rate_limit.max_auth_requests_per_ip_per_sec` | login/setup/refresh abuse protection |
| `rate_limit.max_connections_per_ip` | concurrent socket cap per IP |
| `rate_limit.max_requests_per_ip_per_sec` | pre-auth request flood protection |
| `rate_limit.request_body_limit_bytes` | request payload size guard |
| `rate_limit.ban_duration_seconds` | temporary ban window |
| `rate_limit.enable_connection_protection` | master switch for connection guard |

## Practical Starting Point

```toml
[rate_limit]
max_queries_per_sec = 100
max_messages_per_sec = 50
max_subscriptions_per_user = 10
max_auth_requests_per_ip_per_sec = 20
max_connections_per_ip = 100
max_requests_per_ip_per_sec = 200
request_body_limit_bytes = 10485760
ban_duration_seconds = 300
enable_connection_protection = true
cache_max_entries = 100000
cache_ttl_seconds = 600
```

## Environment Strategy

- development: permissive but enabled (so behavior stays realistic)
- staging: mirror production values
- production: strict thresholds + alerts on violations

## Tuning Workflow

1. collect baseline traffic per endpoint and user role
2. set limits just above normal peaks
3. alert on near-limit and over-limit events
4. adjust independently for auth, SQL, and WebSocket traffic

## Signals To Monitor

- spikes in `429` responses
- repeated bans for same source ranges
- auth endpoint bursts
- large WebSocket fan-out patterns with rising rejection counts

================================================================================
# Backup & Restore
URL: https://kalamdb.org/docs/server/sql-reference/backup
Source: content/server/sql-reference/backup.mdx

> Database backup and restore commands in KalamDB. Learn how to create namespace backups, schedule exports, and restore data from backup archives.
# Backup & Restore

KalamDB provides built-in SQL commands for backing up and restoring the **entire
database**. A backup captures everything in one compressed archive — RocksDB
data, Parquet storage files, Raft snapshots, and the server configuration — so
you always have a consistent, self-contained snapshot you can restore from.

> **Role requirement**: `BACKUP DATABASE` and `RESTORE DATABASE` require the
> **DBA** or **System** role.

---

## BACKUP DATABASE

Create a full database backup. The entire data directory and `server.toml` are
compressed into a single `.tar.gz` archive at the specified path.

```sql
BACKUP DATABASE TO '<backup_path>';
```

The command enqueues a background backup job and returns immediately with a
**Job ID** you can use to monitor progress.

### Parameters

| Parameter | Description |
|---|---|
| `backup_path` | Absolute path for the output archive. Must end in `.tar.gz`. Quotes (single or double) are required. |

### Archive contents

| Path inside archive | Description |
|---|---|
| `data/rocksdb/` | RocksDB write-path data |
| `data/storage/` | Flushed Parquet segment files |
| `data/snapshots/` | Raft snapshots |
| `data/streams/` | Stream commit log data |
| `server.toml` | Server configuration (if present) |

### Example

```sql
-- Back up the entire database to a timestamped archive
BACKUP DATABASE TO '/backups/kalamdb_20260224.tar.gz';

-- Store to a date-named path
BACKUP DATABASE TO '/var/backups/kalamdb/2026-02-24.tar.gz';
```

**Response**

```
Database backup started to '/backups/kalamdb_20260224.tar.gz'. Job ID: BK-0001
```

---

## RESTORE DATABASE

Restore the entire database from a previously created `.tar.gz` backup archive.
The command stages the restored data under `<db_path>_restore_pending/` without
touching the live database. **A server restart is required to complete the
restore** — on startup, KalamDB detects the pending directory and swaps it in.

```sql
RESTORE DATABASE FROM '<backup_path>';
```

> **Warning**: Once the server restarts after staging, all data written since the
> backup was taken will be lost. Take a fresh backup of the current state before
> running a restore if you need a rollback option.

### Parameters

| Parameter | Description |
|---|---|
| `backup_path` | Absolute path to the `.tar.gz` backup archive to restore from. Quotes (single or double) are required. |

### Restore lifecycle

1. Issue `RESTORE DATABASE FROM '<path>';` — returns a **Job ID**.
2. The job extracts the archive to `<db_path>_restore_pending/`.
3. Job status transitions to `Completed` with the message *"Restore staged from '…'. RocksDB restore is pending server restart."*
4. **Restart the KalamDB server** — it detects the pending directory, swaps it in, and starts fresh.

### Example

```sql
-- Stage a restore from a specific timestamped backup
RESTORE DATABASE FROM '/backups/kalamdb_20260224.tar.gz';
```

**Response**

```
Database restore started from '/backups/kalamdb_20260224.tar.gz'. Job ID: RS-0001
```

After the job completes:

```sql
-- Confirm the restore job reached Completed before restarting
SELECT job_id, status, message FROM system.jobs WHERE job_id = 'RS-0001';
```

Then restart the server process to activate the restored data.

---

## Path security rules

KalamDB validates the backup path on both `BACKUP` and `RESTORE` to prevent
path traversal and sensitive file access:

- `..` sequences are **not allowed** (blocks path traversal)
- Null bytes (`\0`) are **not allowed**
- Writing to `/etc/`, `/root/`, `/var/log/`, or `C:\Windows\` is **blocked**
- The path must be quoted (bare paths are rejected)

```sql
-- These will be rejected:
BACKUP DATABASE TO '../../../tmp/evil.tar.gz';   -- path traversal
BACKUP DATABASE TO '/etc/shadow';                -- sensitive directory
BACKUP DATABASE TO /backups/app.tar.gz;          -- unquoted path
```

---

## Async job execution

Backup and restore operations run as **background jobs** managed by the
`UnifiedJobManager`. The SQL command returns immediately with a job ID; the
work happens asynchronously so the server remains responsive.

You can monitor job status via the `system.jobs` table:

```sql
SELECT job_id, status, message
FROM system.jobs
WHERE job_id = 'BK-0001';
```

| `status` value | Meaning |
|---|---|
| `Queued` | Job is waiting to start |
| `Running` | Backup/restore is in progress |
| `Completed` | Finished successfully |
| `Failed` | An error occurred — see `message` for details |

---

## Backup strategy

For production deployments, consider:

1. **Scheduled backups** — Run `BACKUP DATABASE` on a cron or application-level
   schedule (daily, hourly, etc.)
2. **Timestamped archives** — Include a date/time in the path so each backup is
   uniquely named and old ones are not silently overwritten
3. **Remote storage** — Copy the `.tar.gz` archive to S3, GCS, or Azure Blob
   Storage after creation for off-site durability
4. **Snapshot + backup** — Combine `CLUSTER SNAPSHOT` with `BACKUP DATABASE`
   for a fully consistent cluster-wide state capture
5. **Pre-restore snapshot** — Take a backup of the current state before running
   `RESTORE DATABASE` so you can roll back if needed

================================================================================
# Cluster Operations
URL: https://kalamdb.org/docs/server/sql-reference/cluster
Source: content/server/sql-reference/cluster.mdx

> CLI-first cluster inspection and administration in KalamDB, including list, snapshots, elections, purging, rebalance, join, and snapshot cleanup.
# Cluster Operations

Cluster inspection and administration are documented on the CLI page first. The CLI renders the structured SQL results returned by the server, which keeps operational output out of backend string-formatting paths.

See [Interactive Commands — cluster](https://kalamdb.org/docs/cli/interactive-commands)
for the supported operator commands and
[SQL Workflows — follower write forwarding](https://kalamdb.org/docs/cli/workflows)
for the end-to-end flow when a write lands on a follower node. For the Multi-Raft architecture,
see [/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering).

## Inspect cluster state

Use the CLI for node and group views:

```bash
kalam --command "\\cluster list"
kalam --command "\\cluster list groups"
```

For raw tabular access, query the system views directly:

```sql
SELECT * FROM system.cluster ORDER BY is_leader DESC, node_id ASC;
SELECT * FROM system.cluster_groups ORDER BY group_id ASC;
```

`CLUSTER LIST`, `CLUSTER STATUS`, and `CLUSTER LS` are no longer documented SQL commands. Use `\cluster list` or the `system.cluster` views instead.

## Run cluster operations

Use the CLI cluster meta-commands documented in
[Interactive Commands](https://kalamdb.org/docs/cli/interactive-commands).
Those commands still use the SQL transport under the hood, but the CLI owns the operator-facing rendering.

## Automation note

The underlying SQL transport is still what the CLI calls, so automation can keep using the cluster SQL statements when you want the raw row payload instead of CLI rendering.

## Unsupported or removed commands

```sql
CLUSTER LEAVE; -- unsupported
```

`CLUSTER LEAVE` is still unsupported. For list-style inspection, use `\cluster list` or the system views instead of old `CLUSTER LIST`/`STATUS`/`LS` SQL forms.

`CLUSTER TRANSFER LEADER` is still routed by the backend, but some builds can report it as unsupported at runtime depending on the OpenRaft version in use.

## Related Architecture

- [/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)

================================================================================
# Schema & Query Diagnostics
URL: https://kalamdb.org/docs/server/sql-reference/diagnostics
Source: content/server/sql-reference/diagnostics.mdx

> Inspect table schemas with DESCRIBE and inspect query plans with EXPLAIN, ANALYZE, and VERBOSE. Includes KalamDB hot/cold storage scan metrics.
# Schema & Query Diagnostics

Use these commands to inspect table schemas and query execution plans. They are intended for
operators, DBAs, and developers tuning SQL against KalamDB tables.

For storage-tier background on plan metrics, see
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers).

## DESCRIBE — table schema

KalamDB supports two DESCRIBE styles. Both answer “what columns does this table have?”, but they
use different syntax, authorization, and result shapes.

### `DESCRIBE TABLE` (KalamDB native)

```sql
DESCRIBE TABLE [<namespace>.]<table_name>;
DESC TABLE [<namespace>.]<table_name>;

-- Schema version history
DESCRIBE TABLE [<namespace>.]<table_name> HISTORY;
```

**Access:** authenticated users with an elevated role (DBA or System). Regular `User` role
statements are rejected at execution time.

**Returns:** rich column metadata from the schema registry:

| Column | Description |
|--------|-------------|
| `column_name` | Column name |
| `ordinal_position` | 1-based position in the table |
| `column_id` | Parquet field id |
| `data_type` | Kalam SQL type name (for example `TEXT`, `BIGINT`, `TIMESTAMP`) |
| `is_nullable` | Whether the column allows `NULL` |
| `is_primary_key` | Whether the column is part of the primary key |
| `column_default` | Default expression, if any |
| `column_comment` | Column comment, if any |
| `schema_version` | Schema version when the column was added |

**Example:**

```sql
DESCRIBE TABLE dba.notifications;
```

### `DESCRIBE <table>` / `DESC <table>` (DataFusion shorthand)

```sql
DESCRIBE [<namespace>.]<table_name>;
DESC [<namespace>.]<table_name>;
```

**Access:** DBA or System only (admin meta command).

**Behavior:** KalamDB rewrites the statement to query `information_schema.columns` and returns
Kalam SQL type names (`kdb_data_type`) instead of Arrow physical types:

| Column | Description |
|--------|-------------|
| `column_name` | Column name |
| `data_type` | Kalam SQL type (`kdb_data_type`) |
| `is_nullable` | `YES` / `NO` |

Unqualified table names resolve against the session’s current namespace (`USE NAMESPACE`, CLI
`ns:<name>` context, or API `namespace_id`).

**Example:**

```sql
USE NAMESPACE dba;
DESCRIBE notifications;
```

### CLI shortcut

In the interactive shell, `\d <table>` and `\describe <table>` run an
`information_schema.columns` query with Kalam type names — the same columns as the shorthand
above, plus `namespace`, `column_default`, and `position`:

```sql
\d dba.notifications
\describe notifications
```

See [CLI querying](https://kalamdb.org/docs/cli/querying) for output formats.

---

## EXPLAIN — query plans

`EXPLAIN` shows how DataFusion plans a query without executing it (unless `ANALYZE` is present).

**Access:** DBA or System only.

### Syntax (keyword order matters)

DataFusion expects this order:

```sql
EXPLAIN [ANALYZE] [VERBOSE] [FORMAT <indent|tree|pgjson>] <query>;
```

| Keyword | Position | Effect |
|---------|----------|--------|
| `EXPLAIN` | required | Start the diagnostic command |
| `ANALYZE` | optional, **before** `VERBOSE` | Execute the query and attach runtime metrics |
| `VERBOSE` | optional, after `ANALYZE` | Add extra result rows (full metrics, row count, duration) when combined with `ANALYZE` |
| `FORMAT` | optional | Plan rendering style (`indent` default, `tree`, `pgjson`) |

**Common mistake:** `EXPLAIN VERBOSE ANALYZE` is **not** valid. Use
`EXPLAIN ANALYZE VERBOSE` instead. With the wrong order, the parser treats `ANALYZE` as a
separate statement and fails.

### `EXPLAIN` (plan only)

Returns two rows — logical plan and physical plan — without running the query:

```sql
EXPLAIN
SELECT id, user_id
FROM dba.notifications
WHERE user_id = 'test'
LIMIT 3;
```

Example output columns:

| plan_type | plan |
|-----------|------|
| `logical_plan` | Optimized logical plan tree |
| `physical_plan` | Physical operators (`FilterExec`, `GlobalLimitExec`, `DeferredBatchExec`, …) |

### `EXPLAIN ANALYZE` (plan + runtime metrics)

Executes the query, discards result rows to the client, and returns an annotated physical plan
with per-operator metrics:

```sql
EXPLAIN ANALYZE
SELECT id, user_id, _deleted
FROM dba.notifications
WHERE user_id = 'test'
LIMIT 3;
```

Returns one primary row:

| plan_type | plan |
|-----------|------|
| `Plan with Metrics` | Physical plan tree with timing and counter metrics |

On `USER` and `SHARED` tables, KalamDB enables **scan diagnostics** for `EXPLAIN ANALYZE`.
Look for `DeferredBatchExec` nodes and metrics such as:

| Metric | Meaning |
|--------|---------|
| `hot_rows_scanned` | Rows read from the hot tier (RocksDB) |
| `cold_rows_scanned` | Rows read from cold Parquet files |
| `cold_files_total` | Cold files considered by manifest pruning |
| `cold_files_skipped` | Cold files skipped by pruning |
| `cold_files_scanned` | Cold files actually read |
| `cold_file_visited{file=…}` | Individual Parquet files touched (up to 16 recorded) |

Plan text also includes static scan context on deferred MVCC sources, for example:
`storage_tiers=[hot=rocksdb,cold=parquet], mvcc=true`.

### `EXPLAIN ANALYZE VERBOSE` (full metrics + summary)

Adds extra rows after the annotated plan:

```sql
EXPLAIN ANALYZE VERBOSE
SELECT id, user_id, _deleted
FROM dba.notifications
WHERE user_id = 'test'
LIMIT 3;
```

| plan_type | plan |
|-----------|------|
| `Plan with Metrics` | Annotated plan with standard metrics |
| `Plan with Full Metrics` | Plan with all available metric detail |
| `Output Rows` | Rows produced by the query |
| `Duration` | Total execution time |

### Reading plans for KalamDB tables

Typical physical plans on application tables include:

- **`DeferredBatchExec`** — deferred MVCC scan (hot RocksDB + cold Parquet merge)
- **`FilterExec`** — predicate evaluation (pushdown where possible)
- **`ProjectionExec`** — column selection
- **`GlobalLimitExec`** / **`LocalLimitExec`** — `LIMIT` handling
- **`CooperativeExec`** — stream table scans

Use plain `EXPLAIN` to compare logical vs physical shape without execution cost. Use
`EXPLAIN ANALYZE` when you need to see whether filters hit the hot tier, how many cold files
were scanned, and operator timings.

---

## CLI output tips

In default **table** format, the CLI renders `plan_type` / `plan` result sets specially:

- Multi-line plan text is expanded across table rows (continuation lines leave `plan_type` blank)
- The `plan` column uses the full terminal width instead of the generic 32-character cap

For machine-readable output, use JSON:

```bash
kalam --json -c "EXPLAIN ANALYZE SELECT 1;"
```

Or in the interactive shell:

```sql
\format json
EXPLAIN ANALYZE VERBOSE SELECT id FROM dba.notifications LIMIT 3;
```

## Related

- [Table DDL — DESCRIBE TABLE](https://kalamdb.org/docs/server/sql-reference/tables#describe-table)
- [Query Data (SELECT & JOIN)](https://kalamdb.org/docs/server/sql-reference/query-data)
- [Storage tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [CLI querying](https://kalamdb.org/docs/cli/querying)

================================================================================
# Data Manipulation
URL: https://kalamdb.org/docs/server/sql-reference/dml
Source: content/server/sql-reference/dml.mdx

> KalamDB data manipulation language (DML) reference. Learn INSERT, UPDATE, DELETE, PostgreSQL-style upsert with ON CONFLICT DO UPDATE, and RETURNING syntax for USER, SHARED, and STREAM tables.
# Data Manipulation (DML)

Standard SQL data manipulation commands for reading and writing data.

DML permission and routing behavior depends on table type. For `USER`, `SHARED`, `STREAM`, and
`SYSTEM` access rules, see [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## INSERT

Insert one or more rows into a table:

```sql
INSERT INTO [<namespace>.]<table_name> (<column1>, <column2>, ...)
VALUES (<value1>, <value2>, ...);
```

### Batch Insert

```sql
INSERT INTO [<namespace>.]<table_name> (<column1>, <column2>, ...)
VALUES
  (<value1a>, <value2a>, ...),
  (<value1b>, <value2b>, ...);
```

### Examples

```sql
-- Single insert with auto-generated ID
INSERT INTO chat.messages (conversation_id, sender, content)
VALUES (1, 'alice', 'Hello!');

-- Batch insert
INSERT INTO chat.messages (conversation_id, sender, role, content)
VALUES
  (1, 'alice', 'user', 'What is KalamDB?'),
  (1, 'assistant', 'assistant', 'KalamDB is a SQL-first realtime database.');
```

## Upsert with `ON CONFLICT`

KalamDB supports PostgreSQL-style upsert for literal `INSERT ... VALUES` statements on
`USER`, `SHARED`, and `STREAM` tables:

```sql
INSERT INTO [<namespace>.]<table_name> (<column1>, <column2>, ...)
VALUES (<value1>, <value2>, ...)
ON CONFLICT (<primary_key_column>)
DO UPDATE SET
  <column1> = EXCLUDED.<column1>,
  <column2> = <literal_value>;
```

If the primary key already exists, KalamDB updates the row with the `DO UPDATE SET` assignments.
If the primary key is new, KalamDB inserts the row from `VALUES`.

### Examples

```sql
-- Update an existing shared row by primary key
INSERT INTO app.items (id, name)
VALUES (1, 'beta')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name;

-- Insert when the primary key is missing, update when it already exists
INSERT INTO app.items (id, name)
VALUES (42, 'gamma')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name;
```

`DO UPDATE SET` assignments support:

- `EXCLUDED.<column>` to read the value from the attempted insert
- literal constants such as `'published'`, `42`, or `NULL`

### Upsert inside explicit transactions

Upsert works inside `BEGIN` / `COMMIT` blocks and sees rows staged earlier in the same
transaction, including on `SHARED` tables:

```sql
BEGIN;

INSERT INTO app.items (id, name) VALUES (2, 'alpha');

INSERT INTO app.items (id, name) VALUES (2, 'gamma')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name;

COMMIT;
```

Outside an explicit transaction, KalamDB wraps each upsert in its own internal transaction
and commits it automatically.

### Upsert rules and limits

- Requires a single-column primary key. The conflict target must be that primary key column.
- Literal `VALUES` inserts only. `INSERT ... SELECT` upserts are not supported on this path.
- `ON CONFLICT DO NOTHING`, `ON CONFLICT ON CONSTRAINT`, and `ON CONFLICT DO UPDATE WHERE`
  are not supported.
- Tuple assignments in `DO UPDATE SET` are not supported.
- `system.*` tables are rejected.

## `RETURNING` on upsert

Add `RETURNING` to an upsert to get back the inserted or updated row instead of only an
affected-row count. This follows PostgreSQL's `INSERT ... RETURNING` shape: the clause lists
the output columns, and aliases are supported.

```sql
INSERT INTO [<namespace>.]<table_name> (<columns>)
VALUES (<values>)
ON CONFLICT (<primary_key_column>)
DO UPDATE SET <assignments>
RETURNING <column> [, <column> AS <alias> ...];
```

### Examples

```sql
-- Return the final row after an update
INSERT INTO app.items (id, name)
VALUES (1, 'beta')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name
RETURNING id, name;

-- Return a column alias
INSERT INTO app.items (id, name)
VALUES (1, 'beta')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name
RETURNING id, name AS returned_name;

-- Return the inserted row when no conflict occurred
INSERT INTO app.items (id, name)
VALUES (42, 'alpha')
ON CONFLICT (id) DO UPDATE SET name = EXCLUDED.name
RETURNING id, name;
```

### Response shape

- Without `RETURNING`, the SQL API reports an insert result with `rows_affected`.
- With `RETURNING`, the SQL API returns a query result with `schema`, `rows`, and `row_count`,
  the same success shape as `SELECT`.

`RETURNING` is currently supported on the upsert path above. Plain `INSERT ... RETURNING`
without `ON CONFLICT`, and `UPDATE` / `DELETE ... RETURNING`, are not supported yet.

## UPDATE

```sql
UPDATE [<namespace>.]<table_name>
SET <column1> = <value1>, <column2> = <value2>
WHERE <condition>;
```

### Example

```sql
UPDATE chat.messages
SET content = 'Updated message content'
WHERE id = 42;
```

## DELETE

```sql
DELETE FROM [<namespace>.]<table_name>
WHERE <condition>;
```

### Example

```sql
DELETE FROM chat.messages
WHERE conversation_id = 1 AND sender = 'bot';
```

## Explicit Transactions

KalamDB stays in autocommit mode by default. Use explicit transaction blocks when you need PostgreSQL-style atomic multi-statement writes.

### Syntax

```sql
BEGIN;
START TRANSACTION;

COMMIT;
COMMIT WORK;

ROLLBACK;
ROLLBACK WORK;
```

### Commit example

```sql
BEGIN;

INSERT INTO chat.messages (conversation_id, sender, content)
VALUES (7, 'alice', 'draft');

UPDATE chat.messages
SET content = 'published'
WHERE conversation_id = 7 AND sender = 'alice';

COMMIT;
```

### Rollback example

```sql
BEGIN;

INSERT INTO chat.messages (conversation_id, sender, content)
VALUES (8, 'alice', 'temporary');

SELECT *
FROM chat.messages
WHERE conversation_id = 8;

ROLLBACK;
```

Within the open transaction, reads see the staged writes from that same transaction. After `ROLLBACK`, those staged changes are discarded.

### Rules and limits

- Explicit transactions currently support `USER` and `SHARED` tables.
- `STREAM` tables and `system.*` tables are rejected inside explicit transactions.
- DDL statements are not supported inside explicit transactions.
- Nested `BEGIN` and savepoints are not supported in this phase.
- A single `/v1/api/sql` request can contain multiple sequential `BEGIN ... COMMIT` or `BEGIN ... ROLLBACK` blocks.
- If a `/v1/api/sql` request ends with an open transaction, KalamDB rolls it back automatically.

To inspect live transaction state, query `system.transactions`. For pg-extension-specific session
state, use `system.sessions`. See
[/docs/server/sql-reference/system-views](https://kalamdb.org/docs/server/sql-reference/system-views).

For deeper read-query patterns (joins, CTEs, and DataFusion-compatible advanced SELECT usage), see
[/docs/server/sql-reference/query-data](https://kalamdb.org/docs/server/sql-reference/query-data).

## Related Architecture

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)

================================================================================
# FILE Datatype
URL: https://kalamdb.org/docs/server/sql-reference/file-datatype
Source: content/server/sql-reference/file-datatype.mdx

> How to define FILE columns in KalamDB tables, upload files with the FILE() function, and read or download stored file references using SQL and the HTTP API.
# FILE Datatype

KalamDB supports a `FILE` datatype for storing file references in table rows while the binary content is managed by the file service.

Use this page to understand SQL usage. For architecture and storage-path layout, see
[/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype).

## Define FILE Columns

Create a table with a `FILE` column:

```sql
CREATE TABLE app.documents (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  title TEXT NOT NULL,
  attachment FILE,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (TYPE = 'USER');
```

`FILE` columns are supported on `USER` and `SHARED` tables.

## Insert with FILE(...)

Use `FILE("placeholder")` in SQL and send multipart form parts named `file:<placeholder>`.

```sql
INSERT INTO app.documents (id, title, attachment)
VALUES (1001, 'Contract', FILE("contract"));
```

Multipart body parts:

- `sql`
- `params` (optional JSON array string)
- `namespace_id` (optional)
- `file:contract` (binary part for `FILE("contract")`)

Backend behavior (API + file utils):

- SQL multipart parser enforces file limits (`max_files_per_request`, `max_size_bytes`).
- Placeholder names in `FILE("name")` must match multipart keys `file:name`.
- On SQL failure after staging/finalization, cleanup routines attempt to remove uploaded files.

## Query FILE Columns

Querying returns file-reference metadata in the row.

```sql
SELECT id, title, attachment
FROM app.documents
WHERE id = 1001;
```

In SDKs, parse the returned file reference and bind table context before downloading:

- **TypeScript:** `queryRows(..., 'app.documents')` then `row.file('attachment')` for a `BoundFileRef` with no-arg `downloadUrl()`
- **Rust:** `TableId::from_strings("app", "documents")` then `cell.as_bound_file(&table_id)` and `download_bound_file()`
- **Dart:** `KalamFileRef.getDownloadUrl(...)` with explicit namespace and table (see [Dart SDK querying](https://kalamdb.org/docs/dart-sdk/querying#file-columns))

## Download Files

Use the file endpoint:

`GET /v1/files/{namespace}/{table_name}/{subfolder}/{file_id}`

Notes:

- Bearer token is required.
- `SYSTEM` and `STREAM` tables are rejected for file paths.
- `user_id` may be required for user-table scope resolution.

Path model summary:

- `USER` table downloads resolve using user scope (`user_id` effective context).
- `SHARED` table downloads resolve without user scope.
- Effective folder layout is controlled by storage templates (see architecture page).

## Best Practices

- Use `USER` tables for tenant-private files.
- Use `SHARED` tables for global/shared assets.
- Store descriptive metadata in additional columns (e.g., `mime`, `label`, `tags`).
- Fetch only required columns where possible (for example `SELECT id, attachment`).

## Related Docs

- [/docs/server/api/http-reference](https://kalamdb.org/docs/server/api/http-reference)
- [/docs/ts-sdk/file-columns](https://kalamdb.org/docs/ts-sdk/file-columns)
- [/docs/rust-sdk/file-columns](https://kalamdb.org/docs/rust-sdk/file-columns)
- [/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)

================================================================================
# Built-in Functions
URL: https://kalamdb.org/docs/server/sql-reference/functions
Source: content/server/sql-reference/functions.mdx

> KalamDB built-in SQL functions plus DataFusion-compatible scalar, aggregate, window, and PostgreSQL-style JSON query support. Includes SNOWFLAKE_ID, NOW, COSINE_DISTANCE, and more.
# Built-in Functions

KalamDB provides built-in functions for ID generation, user context, and timestamps.

KalamDB query execution also supports DataFusion SQL functions. For advanced query-time function usage, see:

- [DataFusion SQL Functions Overview](https://datafusion.apache.org/user-guide/sql/index.html)
- [DataFusion Scalar Functions](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
- [DataFusion Aggregate Functions](https://datafusion.apache.org/user-guide/sql/aggregate_functions.html)
- [DataFusion Window Functions](https://datafusion.apache.org/user-guide/sql/window_functions.html)

The functions on this page are KalamDB-specific functions you will use most often.

## ID Generation

### SNOWFLAKE_ID()

Generate a unique 64-bit snowflake ID (time-ordered, globally unique):

```sql
SELECT SNOWFLAKE_ID();
-- Returns: 7302451094528000001
```

Commonly used as the default for primary keys:

```sql
CREATE TABLE app.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  content TEXT NOT NULL
);
```

### UUID_V7()

Generate a UUIDv7 (time-ordered UUID):

```sql
SELECT UUID_V7();
-- Returns: 0193a5e0-7a1b-7000-8000-000000000001
```

### ULID()

Generate a ULID (Universally Unique Lexicographically Sortable Identifier):

```sql
SELECT ULID();
-- Returns: 01HQJK5M5R3YJQWXHN4QWXYN01
```

## Context Functions

### CURRENT_USER()

Returns the username of the currently authenticated user:

```sql
SELECT CURRENT_USER();
-- Returns: 'alice'
```

Useful for audit columns:

```sql
CREATE TABLE app.audit_log (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  action TEXT NOT NULL,
  performed_by TEXT DEFAULT CURRENT_USER(),
  created_at TIMESTAMP DEFAULT NOW()
);
```

## Date & Time

### NOW()

Returns the current timestamp:

```sql
SELECT NOW();
-- Returns: 2026-02-18T10:30:00.000Z
```

Commonly used as a default for timestamp columns:

```sql
CREATE TABLE app.events (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  event_type TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
);
```

## Vector Search

### COSINE_DISTANCE()

Use `COSINE_DISTANCE(embedding_column, '[...]')` to rank rows by vector similarity.
This is the primary KalamDB pattern for `EMBEDDING(n)` columns: pass the stored column
as the first argument and a JSON array literal as the query vector.

```sql
SELECT id, title
FROM rag.documents_vectors
ORDER BY COSINE_DISTANCE(doc_embedding, '[1.0,0.0,0.0]')
LIMIT 5;
```

Typical flow:

1. Store embeddings in an `EMBEDDING(n)` column.
2. Create a cosine index with `ALTER TABLE ... CREATE INDEX ... USING COSINE`.
3. Use `ORDER BY COSINE_DISTANCE(...) LIMIT k` to fetch nearest matches.

Notes:

- The query vector must match the column dimension.
- Lower distance means a closer match.
- JSON query literals (for example `'[1.0,0.0,0.0]'`) use KalamDB's vector-search path and return `FLOAT`.
- When both arguments are array/list values, KalamDB delegates to DataFusion 54's native `cosine_distance` and returns `DOUBLE`.
- This is the main SQL entry point for vector search in KalamDB today.

## JSON Operators and Functions

KalamDB registers DataFusion's JSON function package in every SQL session.
That gives JSON columns the common PostgreSQL-style query syntax most users expect for extraction and existence checks.

```sql
SELECT
  doc->'profile' AS profile_json,
  doc->>'status' AS status_text,
  doc->'items'->0 AS first_item,
  doc ? 'customer_id' AS has_customer_id
FROM app.orders
WHERE doc->>'status' = 'paid';
```

Supported PostgreSQL-style operators in KalamDB SQL:

| Syntax | Equivalent helper | Result |
|---|---|---|
| `json_col -> 'key'` or `json_col -> 0` | `json_get(...)` | JSON value |
| `json_col ->> 'key'` or `json_col ->> 0` | `json_as_text(...)` | text |
| `json_col ? 'key'` | `json_contains(...)` | `BOOLEAN` |

Available JSON helpers from the registered DataFusion package:

| Function | Use |
|---|---|
| `json_get` | Return a JSON value at a path |
| `json_as_text` | Return a value at a path as text |
| `json_get_json` | Return nested JSON as raw JSON text |
| `json_get_str` | Return a string value |
| `json_get_int` | Return an integer value |
| `json_get_float` | Return a floating-point value |
| `json_get_bool` | Return a boolean value |
| `json_get_array` | Return an array value |
| `json_contains` | Test whether a key or index exists |
| `json_length` | Return the length of an array or object |
| `json_object_keys` | Return the keys of a JSON object as an array |
| `json_from_scalar` | Convert a scalar into a JSON value for composed expressions |

Scope note:

- KalamDB plans PostgreSQL-style `->`, `->>`, and `?` operators natively through DataFusion 54 and `datafusion-functions-json`. No SQL rewrite pass is applied for these operators.
- Do not assume full PostgreSQL `jsonb` operator parity for operators such as `#>`, `#>>`, or `@>` yet.

## DataFusion 54 Query Extensions

KalamDB runs on DataFusion 54 and enables additional query-time SQL beyond the KalamDB-specific functions above.

### Lambda array functions

Use SQL lambda syntax with higher-order array functions:

```sql
SELECT array_transform([1, 2, 3, 4, 5], x -> x * 10) AS scaled;

SELECT array_transform(
  array_filter([1, 2, 3, 4, 5], x -> x > 2),
  x -> x * 10
) AS filtered_scaled;

SELECT array_any_match([1, 2, 3], x -> x > 2) AS has_large;
```

These work in general `SELECT` queries. They are not supported in `SUBSCRIBE TO` / live-query SQL, which keeps a narrower surface.

### Lateral joins

DataFusion 54 adds basic lateral join support for correlated subqueries in the `FROM` clause:

```sql
SELECT u.id, expanded.value
FROM app.users AS u
CROSS JOIN LATERAL (
  SELECT array_transform(u.tags, t -> upper(t)) AS value
) AS expanded;
```

### Comparison and subquery semantics

DataFusion 54 also changes a few comparison behaviors worth knowing when porting SQL:

- Numeric columns compared to string literals (for example `amount > '100'`) are coerced numerically, not lexicographically.
- Uncorrelated scalar subqueries that return more than one row fail at execution with a cardinality error.

For the full upstream function catalog, use the [DataFusion SQL docs](https://datafusion.apache.org/user-guide/sql/index.html).

## Function Summary

| Function | Return Type | Description |
|----------|-------------|-------------|
| `SNOWFLAKE_ID()` | `BIGINT` | Time-ordered 64-bit unique ID |
| `UUID_V7()` | `TEXT` | Time-ordered UUID v7 |
| `ULID()` | `TEXT` | Lexicographically sortable unique ID |
| `CURRENT_USER()` | `TEXT` | Authenticated username |
| `NOW()` | `TIMESTAMP` | Current server timestamp |
| `COSINE_DISTANCE(vector, query)` | `FLOAT` or `DOUBLE` | Vector similarity distance. JSON query literals return `FLOAT`; array/array inputs use native DataFusion 54 and return `DOUBLE`. |

## DataFusion Function Support

KalamDB supports DataFusion SQL function execution for SELECT/query workloads, including scalar, aggregate, and window functions documented in DataFusion.

If you need advanced expression behavior beyond KalamDB-specific built-ins, use the DataFusion function references linked above as the canonical guide.

For end-to-end schema and index examples, see
[/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search).

================================================================================
# Execute As
URL: https://kalamdb.org/docs/server/sql-reference/impersonation
Source: content/server/sql-reference/impersonation.mdx

> Use KalamDB EXECUTE AS '<user_id>' wrapper syntax with the role-based delegation matrix.
# Execute As

`EXECUTE AS` is wrapper syntax for a single SQL statement. It switches USER-table or STREAM-table execution to a resolved target user ID only when the authenticated actor role is allowed to target that user's role.

## EXECUTE AS

```sql
EXECUTE AS '<user_id>' (
  <single_statement>
);
```

### Rules

1. The wrapper must contain **exactly one** SQL statement
2. The target user ID must be **single-quoted**
3. System users may target system, DBA, service, and user accounts
4. DBA users may target DBA, service, and user accounts
5. Service users may target service and user accounts
6. Regular users may only target themselves
7. The wrapper is valid for USER and STREAM tables; shared tables use their table policy directly
8. Legacy inline `... AS USER 'name'` syntax is **not supported**

### Examples

**Authorized delegated query:**
```sql
EXECUTE AS 'user_123' (
  SELECT * FROM app.messages WHERE conversation_id = 42
);
```

**Authorized delegated insert:**
```sql
EXECUTE AS 'user_123' (
  INSERT INTO app.messages (conversation_id, sender, role, content)
  VALUES (42, 'user_123', 'assistant', 'Processing complete')
);
```

**Authorized delegated delete:**
```sql
EXECUTE AS 'user_123' (
  DELETE FROM app.messages WHERE id = 123
);
```

## Use Cases

| Scenario | Description |
|----------|-------------|
| **Service workers** | Write rows into a user's USER-table or STREAM-table partition through an explicit delegation boundary |
| **Testing** | Assert allowed and denied role-matrix edges consistently |
| **Audit trails** | Record the actor and target user for delegated operations |

================================================================================
# SQL Reference
URL: https://kalamdb.org/docs/server/sql-reference
Source: content/server/sql-reference/index.mdx

> Complete SQL syntax reference for KalamDB. Learn how to manage namespaces, tables, DML, realtime subscriptions, users, and storage.
# SQL Reference

KalamDB is SQL-first with additional commands for realtime subscriptions, storage lifecycle, topics, and cluster operations.

This chapter is organized by command category so you can jump directly to implementation details.

For the architecture behind common SQL choices, start with
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types),
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers), and
[/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query).

## SQL Categories

| Category | Description |
|----------|-------------|
| [/docs/server/sql-reference/namespaces](https://kalamdb.org/docs/server/sql-reference/namespaces) | create/drop/select namespace context |
| [/docs/server/sql-reference/tables](https://kalamdb.org/docs/server/sql-reference/tables) | create/alter/drop tables and views |
| [/docs/server/sql-reference/dml](https://kalamdb.org/docs/server/sql-reference/dml) | `INSERT`, `UPDATE`, `DELETE`, `SELECT`, upsert (`ON CONFLICT DO UPDATE`), `RETURNING` |
| [/docs/server/sql-reference/query-data](https://kalamdb.org/docs/server/sql-reference/query-data) | query patterns, joins, CTEs, and DataFusion-compatible SELECT usage |
| [/docs/server/sql-reference/diagnostics](https://kalamdb.org/docs/server/sql-reference/diagnostics) | `DESCRIBE` / `DESC` schema inspection and `EXPLAIN` / `ANALYZE` / `VERBOSE` query plans |
| [/docs/server/sql-reference/system-views](https://kalamdb.org/docs/server/sql-reference/system-views) | `system.*` observability views including `system.sessions` and `system.transactions` |
| [/docs/server/sql-reference/subscriptions](https://kalamdb.org/docs/server/sql-reference/subscriptions) | `SUBSCRIBE TO`, `KILL LIVE QUERY` |
| [/docs/server/sql-reference/topic-pubsub](https://kalamdb.org/docs/server/sql-reference/topic-pubsub) | topic pub/sub SQL syntax for `CREATE TOPIC`, `CONSUME`, `ACK`, and consumer groups |
| [/docs/server/sql-reference/users](https://kalamdb.org/docs/server/sql-reference/users) | create/alter/drop users and roles |
| [/docs/server/sql-reference/storage](https://kalamdb.org/docs/server/sql-reference/storage) | storage backends, flush, compact, manifests |
| [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage) | table-to-storage mapping patterns |
| [/docs/server/sql-reference/cluster](https://kalamdb.org/docs/server/sql-reference/cluster) | raft and node operations |
| [/docs/server/sql-reference/backup](https://kalamdb.org/docs/server/sql-reference/backup) | namespace backup/recovery commands |
| [/docs/server/sql-reference/impersonation](https://kalamdb.org/docs/server/sql-reference/impersonation) | `EXECUTE AS '<user_id>'` wrapper syntax |
| [/docs/server/sql-reference/functions](https://kalamdb.org/docs/server/sql-reference/functions) | KalamDB built-ins plus DataFusion SQL function support |

## Statement Separator

```sql
SELECT 1;
SELECT 2;
```

## Quick Example

```sql
CREATE NAMESPACE IF NOT EXISTS chat;

CREATE TABLE chat.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  sender TEXT NOT NULL,
  content TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (TYPE = 'USER', FLUSH_POLICY = 'rows:1000');

INSERT INTO chat.messages (sender, content)
VALUES ('alice', 'Hello world!');

SELECT * FROM chat.messages ORDER BY created_at DESC LIMIT 20;
```

================================================================================
# Namespaces
URL: https://kalamdb.org/docs/server/sql-reference/namespaces
Source: content/server/sql-reference/namespaces.mdx

> SQL commands for managing KalamDB namespaces. Learn how to create, drop, list, and select namespace context for organizing tables and data.
# Namespace Commands

Namespaces provide logical grouping for tables, similar to schemas in other databases.

## CREATE NAMESPACE

```sql
CREATE NAMESPACE <namespace_name>;
CREATE NAMESPACE IF NOT EXISTS <namespace_name>;
```

**Example:**
```sql
CREATE NAMESPACE IF NOT EXISTS chat;
CREATE NAMESPACE IF NOT EXISTS analytics;
```

## DROP NAMESPACE

```sql
DROP NAMESPACE <namespace_name>;
DROP NAMESPACE IF EXISTS <namespace_name>;
DROP NAMESPACE <namespace_name> CASCADE;
DROP NAMESPACE IF EXISTS <namespace_name> CASCADE;
```

The `CASCADE` option drops all tables within the namespace.

**Example:**
```sql
-- Drop only if empty
DROP NAMESPACE analytics;

-- Drop with all contained tables
DROP NAMESPACE IF EXISTS analytics CASCADE;
```

## ALTER NAMESPACE

```sql
ALTER NAMESPACE <namespace_name>
  SET DESCRIPTION '<description>';
```

**Example:**
```sql
ALTER NAMESPACE chat
  SET DESCRIPTION 'Chat application tables';
```

## USE / SET NAMESPACE

Set the default namespace for the current request or multi-statement batch:

```sql
USE <namespace_name>;
USE NAMESPACE <namespace_name>;
SET NAMESPACE <namespace_name>;
```

In the interactive CLI, a successful `USE` also updates the CLI's local
namespace so later requests automatically send `namespace_id`.

**Example:**
```sql
USE chat;

-- Now you can reference tables without the namespace prefix
SELECT * FROM messages;
```

## SHOW NAMESPACES

List all available namespaces:

```sql
SHOW NAMESPACES;
```

================================================================================
# Query Data (SELECT & JOIN)
URL: https://kalamdb.org/docs/server/sql-reference/query-data
Source: content/server/sql-reference/query-data.mdx

> How to read data in KalamDB with SELECT queries, WHERE filters, GROUP BY, JOINs, CTEs, PostgreSQL-style JSON queries, and DataFusion-compatible advanced query patterns.
# Query Data (SELECT & JOIN)

Use this page for query-only SQL patterns in KalamDB.

KalamDB supports the DataFusion SELECT query surface. For advanced query syntax and planner behavior, use the DataFusion SQL docs as the primary reference:

- [https://datafusion.apache.org/user-guide/sql/select.html](https://datafusion.apache.org/user-guide/sql/select.html)
- [https://datafusion.apache.org/user-guide/sql/index.html](https://datafusion.apache.org/user-guide/sql/index.html)

The visible rows and join targets still follow KalamDB table-type authorization. `USER` and `STREAM`
tables are scoped by effective user, `SHARED` tables use `ACCESS_LEVEL`, and `system.*` access is
restricted. See [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## DataFusion Statement Scope

KalamDB uses DataFusion as the engine for query-time SQL, especially read/query workloads:

- `SELECT` and projection/filter/order/limit patterns
- joins (`INNER`, `LEFT`, `RIGHT`, `FULL`, `CROSS`, `USING`)
- subqueries and `WITH` (CTE)
- set operations such as `UNION` / `UNION ALL`
- scalar/aggregate/window function execution in query plans

KalamDB-specific operational SQL surfaces are implemented by KalamDB handlers instead of raw DataFusion SQL semantics, including:

- DDL (`CREATE/ALTER/DROP` namespaces, tables, users, storage, topics)
- topic/consumer commands (`CREATE TOPIC`, `ALTER TOPIC`, `CONSUME`, `ACK`, `RESET CONSUMER GROUP`)
- live-query controls (`SUBSCRIBE TO`, `KILL LIVE QUERY`)

## Basic SELECT

```sql
SELECT <columns>
FROM [<namespace>.]<table_name>
[WHERE <condition>]
[ORDER BY <expr> [ASC|DESC]]
[LIMIT <n>];
```

Example:

```sql
SELECT id, sender, content, created_at
FROM chat.messages
WHERE conversation_id = 42
ORDER BY created_at DESC
LIMIT 50;
```

## Filtering, Grouping, and Aggregation

```sql
SELECT sender, COUNT(*) AS message_count
FROM chat.messages
WHERE conversation_id = 42
GROUP BY sender
HAVING COUNT(*) >= 10
ORDER BY message_count DESC;
```

## JOIN Support

KalamDB supports standard DataFusion join patterns, including:

- `INNER JOIN`
- `LEFT JOIN`
- `RIGHT JOIN`
- `FULL JOIN`
- `CROSS JOIN`
- `JOIN ... USING (...)`

Example (`INNER JOIN`):

```sql
SELECT m.id, m.content, u.email
FROM chat.messages m
INNER JOIN system.users u ON m.user_id = u.user_id
WHERE m.conversation_id = 42;
```

Example (`LEFT JOIN`):

```sql
SELECT c.id, c.title, m.content
FROM chat.conversations c
LEFT JOIN chat.messages m ON c.id = m.conversation_id
ORDER BY c.id;
```

## Subqueries and CTEs

Common DataFusion query constructs are supported, including subqueries and CTEs.

```sql
WITH ranked AS (
  SELECT
    sender,
    COUNT(*) AS total_messages
  FROM chat.messages
  GROUP BY sender
)
SELECT *
FROM ranked
WHERE total_messages >= 100
ORDER BY total_messages DESC;
```

## Set Operations

You can use DataFusion set operations such as `UNION` and `UNION ALL`.

```sql
SELECT sender FROM chat.messages_2025
UNION ALL
SELECT sender FROM chat.messages_2026;
```

## PostgreSQL-style JSON Queries

`JSON` and `JSONB` columns can be queried with the same common extraction syntax PostgreSQL users already know.

```sql
SELECT
  doc->>'status' AS status,
  doc->'items'->0 AS first_item
FROM app.orders
WHERE doc->>'status' = 'paid'
  AND doc ? 'customer_id';
```

Use `->` when you want a JSON value, `->>` when you want text, and `?` when you only need to test key existence.

KalamDB also exposes DataFusion JSON helpers such as `json_get`, `json_as_text`, `json_length`, and `json_object_keys` for function-style access.
For the full JSON function list and current operator coverage, see
[/docs/server/sql-reference/functions](https://kalamdb.org/docs/server/sql-reference/functions).

## DataFusion 54 Query Features

KalamDB 0.5.x runs on DataFusion 54. In addition to the core `SELECT` / `JOIN` / CTE surface above, general SQL queries can use:

- **Lambda array functions** such as `array_transform`, `array_filter`, and `array_any_match` with `x -> expr` syntax
- **Lateral joins** such as `CROSS JOIN LATERAL`, `INNER JOIN LATERAL`, and `LEFT JOIN LATERAL`
- **Native JSON operators** (`->`, `->>`, `?`) planned directly by DataFusion without a KalamDB rewrite pass

Subscription and live-query SQL intentionally keeps a narrower surface. Advanced DataFusion 54 features such as lambda arrays, lateral joins, `GROUP BY`, and joins are not available in `SUBSCRIBE TO` statements. See [/docs/server/sql-reference/subscriptions](https://kalamdb.org/docs/server/sql-reference/subscriptions).

When porting SQL from other engines, note that DataFusion 54 compares numeric columns to string literals numerically (for example `5 > '100'` is false) and rejects uncorrelated scalar subqueries that return multiple rows.

## Advanced Query Reference

For advanced SELECT features, rely on DataFusion SQL documentation:

- [https://datafusion.apache.org/user-guide/sql/select.html](https://datafusion.apache.org/user-guide/sql/select.html)
- [https://datafusion.apache.org/user-guide/sql/scalar_functions.html](https://datafusion.apache.org/user-guide/sql/scalar_functions.html)
- [https://datafusion.apache.org/user-guide/sql/index.html](https://datafusion.apache.org/user-guide/sql/index.html)

## Related Architecture

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/datatypes](https://kalamdb.org/docs/server/architecture/datatypes)
- [/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search)

================================================================================
# Storage ID Usage
URL: https://kalamdb.org/docs/server/sql-reference/storage-id-usage
Source: content/server/sql-reference/storage-id-usage.mdx

> How to register storage backends and assign STORAGE_ID in CREATE TABLE for shared and user tables. Includes S3, local, and multi-backend configuration.
# Storage ID Usage

Use this guide when you want to control where table data is stored, pick storage per table, and understand user-table routing behavior.

For MinIO-specific S3-compatible setup, see
[/docs/server/integrations/minio](https://kalamdb.org/docs/server/integrations/minio). For the storage architecture,
see [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers),
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests), and
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## Create storage backends first

Before using `STORAGE_ID` in `CREATE TABLE`, the storage must exist in `system.storages`.

### Filesystem storage

```sql
CREATE STORAGE local_archive
    TYPE filesystem
    NAME 'Local Archive'
    DESCRIPTION 'Filesystem storage for archived tables'
    PATH './data/storage/local-archive'
    SHARED_TABLES_TEMPLATE 'shared/{namespace}/{tableName}'
    USER_TABLES_TEMPLATE 'users/{namespace}/{tableName}/{userId}';
```

`PATH` is supported for filesystem storage. `BASE_DIRECTORY` is also accepted.

### S3 storage

```sql
CREATE STORAGE s3_prod
    TYPE s3
    NAME 'Production S3'
    DESCRIPTION 'S3 bucket for production data'
    BUCKET 'my-kalamdb-prod-bucket'
    REGION 'us-east-1'
    SHARED_TABLES_TEMPLATE 'shared/{namespace}/{tableName}'
    USER_TABLES_TEMPLATE 'users/{namespace}/{tableName}/{userId}';
```

For S3, you can use either:

- `BUCKET 'bucket-name'` (optionally with `REGION`)
- `BASE_DIRECTORY 's3://bucket/prefix'`

Useful checks:

```sql
SHOW STORAGES;
SELECT storage_id, storage_type, storage_name FROM system.storages;
```

## Use STORAGE_ID in CREATE TABLE

KalamDB accepts `STORAGE_ID` in `WITH (...)` table options.

### Shared table with explicit storage

```sql
CREATE TABLE app.config (
    key TEXT PRIMARY KEY,
    value TEXT
) WITH (
    TYPE = 'SHARED',
    STORAGE_ID = 's3_prod'
);
```

### User table with explicit storage and flush policy

```sql
CREATE TABLE app.messages (
    id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
    content TEXT,
    created_at TIMESTAMP DEFAULT NOW()
) WITH (
    TYPE = 'USER',
    STORAGE_ID = 'local_archive',
    FLUSH_POLICY = 'rows:5000'
);
```

### If STORAGE_ID is omitted

If omitted, KalamDB resolves storage to `local` by default.

```sql
CREATE TABLE app.metrics (
    id BIGINT PRIMARY KEY,
    value DOUBLE
) WITH (TYPE = 'SHARED');
```

## Verify table storage metadata

Inspect table metadata in `system.schemas`:

```sql
SELECT namespace_id, table_name, table_type, storage_id, use_user_storage
FROM system.schemas
WHERE namespace_id = 'app' AND is_latest = true
ORDER BY table_name;
```

## Change table storage metadata

For `USER` and `SHARED` tables, admins can update `STORAGE_ID` with `ALTER TABLE ... SET TBLPROPERTIES`:

```sql
ALTER TABLE app.messages
    SET TBLPROPERTIES (STORAGE_ID = 's3_prod');
```

The new storage ID must already exist in `system.storages`.

## Per-user storage routing (USE_USER_STORAGE)

For `USER` tables, you can enable per-user storage routing metadata:

```sql
CREATE TABLE app.geo_events (
    event_id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
    payload JSON,
    created_at TIMESTAMP DEFAULT NOW()
) WITH (
    TYPE = 'USER',
    STORAGE_ID = 's3_prod',
    USE_USER_STORAGE = true
);
```

Rules:

- `USE_USER_STORAGE` is valid only for `TYPE = 'USER'`
- `STORAGE_ID` remains fallback storage for the table
- User preference fields exist in `system.users` (`storage_mode`, `storage_id`)

You can also toggle it after creation:

```sql
ALTER TABLE app.geo_events
    SET TBLPROPERTIES (USE_USER_STORAGE = false);
```

## Changing storage per user (current status)

Current SQL behavior:

- DML against `system.*` tables is blocked
- `ALTER USER` currently supports only `SET PASSWORD`, `SET ROLE`, and `SET EMAIL`

So there is no public SQL command yet to directly update a user's `storage_mode` or `storage_id`.

Current developer workflow:

- Set table-level routing with `STORAGE_ID` (optionally `USE_USER_STORAGE`)
- For user-level overrides, use internal/admin backend flows

## Related Architecture

- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)

## Common failures

### Unknown storage ID

```sql
CREATE TABLE app.bad_example (
    id INT PRIMARY KEY
) WITH (TYPE = 'SHARED', STORAGE_ID = 'does_not_exist');
```

Expected error pattern:

```text
Storage 'does_not_exist' does not exist
```

### Invalid USE_USER_STORAGE on non-user table

```sql
CREATE TABLE app.bad_shared (
    id INT PRIMARY KEY
) WITH (TYPE = 'SHARED', USE_USER_STORAGE = true);
```

Expected error pattern:

```text
USE_USER_STORAGE is only supported for USER tables
```

================================================================================
# Storage Commands
URL: https://kalamdb.org/docs/server/sql-reference/storage
Source: content/server/sql-reference/storage.mdx

> Managing storage backends in KalamDB. Learn commands for registering storage, flushing hot data to cold tier, compacting Parquet files, and manifests.
# Storage Commands

KalamDB supports multiple storage backends for cold-tier data. These commands manage storage configuration, health checks, and maintenance.

For practical table-level storage selection (`STORAGE_ID`, `USE_USER_STORAGE`, and verification
queries), see [/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage).
For S3-compatible local deployment setup, see
[/docs/server/integrations/minio](https://kalamdb.org/docs/server/integrations/minio).
For the underlying hot/cold architecture and manifest state machine, see
[/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers) and
[/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests).

## CREATE STORAGE

```sql
CREATE STORAGE <storage_id>
  TYPE '<filesystem|s3|gcs|azure>'
  [NAME '<storage_name>']
  [DESCRIPTION '<description>']
  [PATH '<path>']
  [BUCKET '<bucket_or_s3_url>']
  [REGION '<region>']
  [BASE_DIRECTORY '<path_or_url>']
  [SHARED_TABLES_TEMPLATE '<template>']
  [USER_TABLES_TEMPLATE '<template>']
  [CREDENTIALS '<json_credentials>']
  [CONFIG '<json_config>'];
```

### Examples

**Local filesystem:**
```sql
CREATE STORAGE local
  TYPE 'filesystem'
  PATH './data';
```

**Amazon S3:**
```sql
CREATE STORAGE s3_prod
  TYPE 's3'
  BUCKET 'my-bucket'
  REGION 'us-west-2'
  CREDENTIALS '{"access_key_id":"...","secret_access_key":"..."}';
```

**Google Cloud Storage:**
```sql
CREATE STORAGE gcs_prod
  TYPE 'gcs'
  BUCKET 'my-gcs-bucket'
  CREDENTIALS '{"service_account_key":"..."}';
```

**Azure Blob Storage:**
```sql
CREATE STORAGE azure_prod
  TYPE 'azure'
  BUCKET 'my-container'
  CREDENTIALS '{"account_name":"...","account_key":"..."}';
```

### Storage Type Aliases

| SQL Type | Alias | Base Directory Prefix |
|---|---|---|
| `filesystem` | — | Local path (e.g., `./data/storage`) |
| `s3` | — | `s3://bucket/prefix` |
| `gcs` | `gs` | `gs://bucket/prefix` |
| `azure` | `az` | `az://container/prefix` |

### CONFIG JSON Fields by Backend

#### S3 CONFIG

```json
{
  "type": "s3",
  "region": "us-east-1",
  "endpoint": "http://127.0.0.1:9120",
  "allow_http": true,
  "access_key_id": "...",
  "secret_access_key": "...",
  "session_token": "..."
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `region` | string | `"us-east-1"` | AWS region or custom region for S3-compatible |
| `endpoint` | string | `null` | Custom endpoint for MinIO / S3-compatible |
| `allow_http` | boolean | `false` | Must be `true` for non-TLS endpoints |
| `access_key_id` | string | `null` | Static credentials (omit for IAM roles) |
| `secret_access_key` | string | `null` | Static credentials |
| `session_token` | string | `null` | Temporary session credentials |

For S3-compatible local setup, see [/docs/server/integrations/minio](https://kalamdb.org/docs/server/integrations/minio).

#### GCS CONFIG

```json
{
  "type": "gcs",
  "service_account_json": "{...}"
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `service_account_json` | string | `null` | Service account JSON key. Omit to use Application Default Credentials (ADC). |

#### Azure CONFIG

```json
{
  "type": "azure",
  "account_name": "mystorageaccount",
  "access_key": "...",
  "sas_token": "..."
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `account_name` | string | `null` | Azure storage account name |
| `access_key` | string | `null` | Account access key |
| `sas_token` | string | `null` | Shared Access Signature token (alternative to access_key) |

#### Local/Filesystem CONFIG

```json
{
  "type": "local",
  "root": "./data/storage"
}
```

| Field | Type | Default | Notes |
|---|---|---|---|
| `root` | string | `null` | Informational; actual path comes from `BASE_DIRECTORY` or `PATH`. |

## ALTER STORAGE

```sql
ALTER STORAGE <storage_id>
  [SET NAME '<new_name>']
  [SET DESCRIPTION '<new_description>']
  [SET SHARED_TABLES_TEMPLATE '<new_template>']
  [SET USER_TABLES_TEMPLATE '<new_template>']
  [SET CONFIG '<json_config>'];
```

## DROP STORAGE

```sql
DROP STORAGE <storage_id>;
DROP STORAGE IF EXISTS <storage_id>;
```

## SHOW STORAGES

```sql
SHOW STORAGES;
```

## STORAGE CHECK

Verify storage backend health:

```sql
STORAGE CHECK <storage_id>;
STORAGE CHECK <storage_id> EXTENDED;
```

## STORAGE FLUSH

Force flushing data from hot tier (RocksDB) to cold tier (Parquet):

```sql
-- Flush a specific table
STORAGE FLUSH TABLE <namespace>.<table_name>;

-- Flush all tables in a namespace
STORAGE FLUSH ALL IN <namespace>;
STORAGE FLUSH ALL IN NAMESPACE <namespace>;

-- Flush everything
STORAGE FLUSH ALL;
```

## STORAGE COMPACT

Merge and optimize cold-tier Parquet segments:

```sql
-- Compact a specific table
STORAGE COMPACT TABLE <namespace>.<table_name>;

-- Compact all tables in a namespace
STORAGE COMPACT ALL IN <namespace>;
STORAGE COMPACT ALL IN NAMESPACE <namespace>;

-- Compact everything
STORAGE COMPACT ALL;
```

## SHOW MANIFEST

Display the storage manifest (schema + segment index):

```sql
SHOW MANIFEST;
```

## Related Architecture

- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)

================================================================================
# Subscriptions & Live Queries
URL: https://kalamdb.org/docs/server/sql-reference/subscriptions
Source: content/server/sql-reference/subscriptions.mdx

> Real-time SQL subscriptions over WebSocket in KalamDB. Learn SUBSCRIBE TO syntax, live query lifecycle, KILL LIVE QUERY, and event streaming patterns.
# Subscriptions & Live Queries

KalamDB's real-time engine pushes data changes to connected clients over WebSocket. This eliminates the need for polling and separate pub/sub infrastructure.

For the runtime architecture, table-type delivery behavior, and cluster fan-out path, see
[/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query).

## SUBSCRIBE TO

Subscribe to a table for real-time change notifications:

```sql
SUBSCRIBE TO <namespace>.<table_name>
[WHERE <condition>]
[OPTIONS (last_rows=<n>, batch_size=<n>, from=<n>)];
```

### Options

| Option | Description |
|--------|-------------|
| `last_rows=<n>` | Receive the last N rows as initial data |
| `batch_size=<n>` | Maximum rows per notification batch |
| `from=<n>` | Resume from a specific sequence ID |

### Examples

```sql
-- Subscribe to all messages in a conversation
SUBSCRIBE TO chat.messages
WHERE conversation_id = 1
OPTIONS (last_rows=50);

-- Subscribe to typing events with batching
SUBSCRIBE TO chat.typing_events
WHERE conversation_id = 1
OPTIONS (last_rows=10, batch_size=5);

-- Resume from a specific point
SUBSCRIBE TO chat.messages
WHERE conversation_id = 1
OPTIONS (from=1234);
```

## KILL LIVE QUERY

Cancel an active subscription:

```sql
KILL LIVE QUERY '<subscription_id>';
```

## Using Subscriptions via SDK

The recommended way to use real-time subscriptions is via the TypeScript SDK:

```typescript

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!')
});

// Open live events with a SQL WHERE clause
const unsub = await client.liveEvents(
  'SELECT * FROM chat.messages WHERE conversation_id = 1',
  (event) => {
    if (event.type === 'change') {
      console.log('New message:', event.rows);
    }
  },
  { batchSize: 50 }
);

// Simple table event stream
const unsubTyping = await client.liveEvents(
  'SELECT * FROM chat.typing_events',
  (event) => {
    console.log('Typing:', event.change_type, event.rows);
  }
);

// Check active subscription count
console.log(`Active: ${client.getSubscriptionCount()}`);

// Cleanup
await unsub();
await unsubTyping();
await client.disconnect();
```

## Using Subscriptions via CLI

In the interactive CLI, use `\live` or `\subscribe`:

```bash
kalam> \live SELECT * FROM chat.messages WHERE conversation_id = 1 OPTIONS (last_rows=20);
```

Both forms start the same live query flow. Press `Ctrl+C` to stop the subscription.

### Supported subscription SQL shape

Live subscriptions support a narrow SQL surface:

```sql
SELECT <columns> | *
FROM <namespace>.<table>
[WHERE <simple predicates>];
```

The following are **not** supported in `SUBSCRIBE TO` / live-query SQL today:

- `JOIN`, `GROUP BY`, `HAVING`, `ORDER BY`, `LIMIT`
- lateral joins and lambda array functions
- subqueries, CTEs, and set operations

Use regular `SELECT` queries for DataFusion 54 advanced features. Use subscriptions only for simple table watches with optional `WHERE` filters.

## How It Works

1. Client sends a `SUBSCRIBE TO` statement (via SQL, SDK, or WebSocket)
2. KalamDB registers the subscription with filter conditions
3. On every `INSERT`, `UPDATE`, or `DELETE` matching the filter, a change event is pushed
4. Events are delivered over the persistent WebSocket connection
5. Client can cancel with `KILL LIVE QUERY` or disconnect

### UPDATE notification payload

For UPDATE events, the notification includes:

- **`rows`**: All non-null columns from the updated row, plus the primary key and `_seq`. This provides a full snapshot of the current row state without requiring a re-query.
- **`old_values`**: Only the columns that actually changed (plus PK and `_seq`), with their previous values.

This design makes tables usable as **change triggers** — when any column is updated, subscribers receive every non-null value, not just the diff.

## Related Architecture

- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/clustering](https://kalamdb.org/docs/server/architecture/clustering)

================================================================================
# System Views
URL: https://kalamdb.org/docs/server/sql-reference/system-views
Source: content/server/sql-reference/system-views.mdx

> Inspect KalamDB runtime state with system views, including system.sessions for pg bridge sessions and system.transactions for active explicit transactions.
# System Views

KalamDB exposes computed `system.*` views for metadata and runtime observability. This page focuses on the new session and transaction views added for explicit transaction support.

`system.*` access follows the `SYSTEM` table rules described in
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types#system-tables).

## `system.sessions`

`system.sessions` shows active PostgreSQL gRPC bridge sessions tracked by the server.

It is intentionally pg-session-focused:

- `pg_kalam` sessions appear here while they are active.
- Native `/v1/api/sql` requests do not create rows here.
- The CLI `\sessions` command runs this view directly.

### Common query

```sql
SELECT *
FROM system.sessions
ORDER BY last_seen_at DESC, session_id;
```

### Active-transaction query

```sql
SELECT session_id, transaction_id, transaction_state, transaction_has_writes
FROM system.sessions
WHERE transaction_id IS NOT NULL
ORDER BY last_seen_at DESC, session_id;
```

### Columns

| Column | Meaning |
|---|---|
| `session_id` | Server-side pg bridge session identifier. |
| `backend_pid` | Parsed PostgreSQL backend PID when the session id uses the `pg-<pid>-...` form. |
| `current_schema` | Current schema reported by the pg extension session. |
| `state` | Session state, similar to PostgreSQL `pg_stat_activity` semantics. |
| `transaction_id` | Active remote transaction id, if one is open. |
| `transaction_state` | Current remote transaction lifecycle state. |
| `transaction_has_writes` | Whether the active transaction has staged writes. |
| `client_addr` | Observed client socket address for the gRPC session. |
| `transport` | Transport used by the session. |
| `opened_at` | When the server first observed the session. |
| `last_seen_at` | Most recent RPC activity timestamp. |
| `last_method` | Most recent gRPC method observed for the session. |

## `system.transactions`

`system.transactions` shows active explicit transactions across all server origins.

Use it to observe transaction state regardless of whether the transaction came from `pg_kalam` or native KalamDB SQL.

- Common `origin` values include `PgRpc` for the PostgreSQL extension and `SqlBatch` for native `/v1/api/sql` execution.
- Rows disappear immediately after `COMMIT`, `ROLLBACK`, timeout cleanup, or request-end cleanup.

### Common query

```sql
SELECT transaction_id, owner_id, origin, state, write_count
FROM system.transactions
ORDER BY origin, transaction_id;
```

### Capacity-oriented query

```sql
SELECT transaction_id, origin, state, age_ms, idle_ms, write_bytes, touched_tables_count
FROM system.transactions
ORDER BY age_ms DESC;
```

### Columns

| Column | Meaning |
|---|---|
| `transaction_id` | Canonical explicit transaction identifier. |
| `owner_id` | Human-readable owner identifier, such as a pg session id or request-scoped SQL owner id. |
| `origin` | Source of the transaction, such as `PgRpc` or `SqlBatch`. |
| `state` | Current lifecycle state. |
| `age_ms` | Transaction age in milliseconds. |
| `idle_ms` | Milliseconds since the transaction last performed work. |
| `write_count` | Number of staged mutations currently buffered. |
| `write_bytes` | Approximate in-memory size of the staged write set. |
| `touched_tables_count` | Number of tables referenced by the transaction. |
| `snapshot_commit_seq` | Committed snapshot boundary captured at `BEGIN`. |

## Key Distinction

- `system.sessions` shows live PostgreSQL bridge sessions only.
- `system.transactions` shows active explicit transactions across all supported origins.
- A native `/v1/api/sql` transaction appears in `system.transactions`, but not in `system.sessions`.

## Related

- [/docs/server/sql-reference/dml](https://kalamdb.org/docs/server/sql-reference/dml)
- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [Kalam CLI](https://kalamdb.org/docs/cli)
- [/docs/pg-kalam/sql-syntax](https://kalamdb.org/docs/pg-kalam/sql-syntax)

================================================================================
# Table DDL
URL: https://kalamdb.org/docs/server/sql-reference/tables
Source: content/server/sql-reference/tables.mdx

> KalamDB table DDL reference. Learn CREATE TABLE, ALTER TABLE, DROP TABLE, and VIEW commands with table type options, flush policies, and constraints.
# Table DDL

KalamDB supports three application table types: `USER`, `SHARED`, and `STREAM`. For the full
architecture, access matrices, storage behavior, and when to choose each type, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## CREATE TABLE

### Syntax

```sql
CREATE [USER|SHARED|STREAM] TABLE [IF NOT EXISTS] [<namespace>.]<table_name> (
  <column_name> <data_type> [NOT NULL|NULL] [DEFAULT <expr>] [PRIMARY KEY],
  ...,
  [CONSTRAINT <name> PRIMARY KEY (<column_name>)]
)
[WITH (
  TYPE = '<USER|SHARED|STREAM>',
  STORAGE_ID = '<storage_id>',
  USE_USER_STORAGE = <TRUE|FALSE>,
  FLUSH_POLICY = '<rows:N|interval:N|rows:N,interval:N>',
  TTL_SECONDS = <seconds>,
  ACCESS_LEVEL = '<PUBLIC|PRIVATE|RESTRICTED|DBA>',
  EVICTION_STRATEGY = '<time_based|size_based|hybrid>',
  MAX_STREAM_SIZE_BYTES = <bytes>,
  COMPRESSION = '<none|snappy|zstd>'
)];
```

Table options are type-specific:

- `USER`: `STORAGE_ID`, `USE_USER_STORAGE`, `FLUSH_POLICY`, `COMPRESSION`
- `SHARED`: `STORAGE_ID`, `ACCESS_LEVEL`, `FLUSH_POLICY`, `COMPRESSION`
- `STREAM`: `TTL_SECONDS`, `EVICTION_STRATEGY`, `MAX_STREAM_SIZE_BYTES`

`ACCESS_LEVEL` is enforced only on `SHARED` tables. `PUBLIC` allows regular users to read but not
write; `PRIVATE` and `RESTRICTED` allow service, dba, and system roles; `DBA` allows only dba and
system roles. The canonical shared-table access matrix is maintained at
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types#access_level-semantics).

For storage-specific options such as `STORAGE_ID` and `USE_USER_STORAGE`, see
[/docs/server/sql-reference/storage-id-usage](https://kalamdb.org/docs/server/sql-reference/storage-id-usage). For
flush behavior, see [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers).

`COMPRESSION` accepts only `none`, `snappy`, and `zstd`, and is valid only for `USER` and `SHARED`
tables. It controls the Apache Parquet codec used when table data is flushed or compacted into
cold-storage segments. `none` writes uncompressed Parquet pages, `snappy` is the default and
optimizes for low CPU overhead, and `zstd` uses Zstandard level 1 for better compression density with
modest extra CPU. This option is not WebSocket gzip and does not change RocksDB hot-tier
compression. `STREAM` tables use hot stream log storage and do not accept table Parquet compression.

### Examples

**User table** (per-user isolated):

```sql
CREATE TABLE app.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  conversation_id BIGINT NOT NULL,
  sender TEXT NOT NULL,
  role TEXT NOT NULL DEFAULT 'user',
  content TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
) WITH (
  TYPE = 'USER',
  STORAGE_ID = 'local',
  USE_USER_STORAGE = false,
  FLUSH_POLICY = 'rows:1000,interval:60',
  COMPRESSION = 'snappy'
);
```

**Shared table** (global access):

```sql
CREATE SHARED TABLE app.config (
  key TEXT PRIMARY KEY,
  value TEXT NOT NULL,
  updated_at TIMESTAMP DEFAULT NOW()
) WITH (
  ACCESS_LEVEL = 'PUBLIC',
  COMPRESSION = 'zstd'
);
```

**Stream table** (ephemeral with TTL):

```sql
CREATE STREAM TABLE app.events (
  event_id TEXT PRIMARY KEY,
  payload TEXT,
  created_at TIMESTAMP DEFAULT NOW()
) WITH (
  TTL_SECONDS = 30,
  EVICTION_STRATEGY = 'hybrid',
  MAX_STREAM_SIZE_BYTES = 1048576
);
```

## ALTER TABLE

```sql
-- Add a column
ALTER TABLE [<namespace>.]<table_name>
  ADD COLUMN <name> <type> [NOT NULL|NULL] [DEFAULT <value>];

-- Drop a column
ALTER TABLE [<namespace>.]<table_name>
  DROP COLUMN <name>;

-- Modify a column type
ALTER TABLE [<namespace>.]<table_name>
  MODIFY COLUMN <name> <type> [NOT NULL|NULL];

-- Change table properties
ALTER TABLE [<namespace>.]<table_name>
  SET TBLPROPERTIES (<table_option> = <value>, ...);

-- Equivalent shorthand for shared access changes
ALTER TABLE [<namespace>.]<table_name>
  SET ACCESS LEVEL <public|private|restricted|dba>;

-- Create a vector index on an embedding column
ALTER TABLE [<namespace>.]<table_name>
  CREATE INDEX <embedding_column> USING COSINE;
```

`SET TBLPROPERTIES` supports the same type-specific persisted options as `CREATE TABLE`.
Use `FLUSH_POLICY = NULL` to clear a user/shared flush policy.

Examples:

```sql
ALTER TABLE app.config
  SET TBLPROPERTIES (ACCESS_LEVEL = 'PUBLIC', COMPRESSION = 'zstd');

ALTER TABLE app.messages
  SET TBLPROPERTIES (FLUSH_POLICY = 'rows:5000', USE_USER_STORAGE = true);

ALTER TABLE app.events
  SET TBLPROPERTIES (
    TTL_SECONDS = 3600,
    EVICTION_STRATEGY = 'size_based',
    MAX_STREAM_SIZE_BYTES = 1048576
  );
```

### Vector index example

```sql
CREATE TABLE rag.documents_vectors (
  id BIGINT PRIMARY KEY,
  embedding EMBEDDING(384)
) WITH (TYPE = 'USER');

ALTER TABLE rag.documents_vectors
  CREATE INDEX embedding USING COSINE;
```

After the index exists, rank nearest rows with:

```sql
SELECT id
FROM rag.documents_vectors
ORDER BY COSINE_DISTANCE(embedding, '[0.12,0.03,0.98]')
LIMIT 10;
```

## DROP TABLE

```sql
DROP TABLE [IF EXISTS] [<namespace>.]<table_name>;
DROP USER TABLE [IF EXISTS] [<namespace>.]<table_name>;
DROP SHARED TABLE [IF EXISTS] [<namespace>.]<table_name>;
DROP STREAM TABLE [IF EXISTS] [<namespace>.]<table_name>;
```

## CREATE VIEW

```sql
CREATE VIEW [<namespace>.]<view_name> AS <select_query>;
CREATE VIEW [<namespace>.]<view_name> (<column1>, <column2>, ...) AS <select_query>;
```

**Example:**
```sql
CREATE VIEW chat.recent_messages AS
  SELECT id, sender, content, created_at
  FROM chat.messages
  WHERE created_at > NOW() - INTERVAL '1 hour'
  ORDER BY created_at DESC;
```

## SHOW TABLES

```sql
SHOW TABLES;
SHOW TABLES IN <namespace>;
SHOW TABLES IN NAMESPACE <namespace>;
```

## DESCRIBE TABLE

```sql
DESCRIBE TABLE [<namespace>.]<table_name>;
DESC TABLE [<namespace>.]<table_name>;

-- Show table history
DESCRIBE TABLE [<namespace>.]<table_name> HISTORY;
```

Returns full schema-registry metadata (`column_id`, `is_primary_key`, `schema_version`, Kalam
`data_type` names, and more).

For the DataFusion-style shorthand (`DESCRIBE namespace.table`), CLI `\d` behavior, and
`EXPLAIN` / `EXPLAIN ANALYZE VERBOSE` plan inspection, see
[/docs/server/sql-reference/diagnostics](https://kalamdb.org/docs/server/sql-reference/diagnostics).

## SHOW STATS

```sql
SHOW STATS FOR TABLE [<namespace>.]<table_name>;
```

## More Table Architecture

- [/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types)
- [/docs/server/architecture/storage-tiers](https://kalamdb.org/docs/server/architecture/storage-tiers)
- [/docs/server/architecture/manifests](https://kalamdb.org/docs/server/architecture/manifests)
- [/docs/server/architecture/live-query](https://kalamdb.org/docs/server/architecture/live-query)

================================================================================
# Topic Pub/Sub SQL
URL: https://kalamdb.org/docs/server/sql-reference/topic-pubsub
Source: content/server/sql-reference/topic-pubsub.mdx

> Topic Pub/Sub SQL syntax for KalamDB. Create topics, attach table sources, consume messages, acknowledge offsets, and reset groups.
# Topic Pub/Sub SQL

Topic Pub/Sub SQL in KalamDB is documented in the Topic Pub/Sub section because the same workflow covers setup, inspection, and consumer-group behavior.

For the canonical command reference, see [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics).

## Syntax Covered

The topic SQL reference covers:

- `CREATE TOPIC ... PARTITIONS ...`
- `ALTER TOPIC ... SET RETENTION ...`
- `ALTER TOPIC ... ADD SOURCE ... ON INSERT|UPDATE|DELETE ...`
- `CONSUME FROM ... GROUP ... FROM ... LIMIT ...`
- `ACK ... GROUP ... UPTO OFFSET ...`
- `RESET CONSUMER GROUP ... ON ... TO ...`
- `CLEAR TOPIC` and `DROP TOPIC`

## Quick Example

```sql
CREATE TOPIC app.user_events PARTITIONS 1;

ALTER TOPIC app.user_events
ADD SOURCE app.users
ON INSERT
WITH (payload = 'full');

CONSUME FROM app.user_events
GROUP 'ai-worker'
FROM EARLIEST
LIMIT 10;

ACK app.user_events
GROUP 'ai-worker'
UPTO OFFSET 9;
```

Continue with [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics) for retention options, payload modes, consumer group offset rules, ACK behavior, and admin reset commands.

================================================================================
# Export User Data
URL: https://kalamdb.org/docs/server/sql-reference/user-data
Source: content/server/sql-reference/user-data.mdx

> Export personal user data from KalamDB with EXPORT USER DATA. Generate a downloadable ZIP archive containing all user-scoped records for compliance.
# Export User Data

KalamDB lets any authenticated user export a snapshot of all their own data as a
downloadable `.zip` archive. The export captures every user-owned Parquet file
(all rows the user has written), so you always have a portable, offline copy of
your data.

> **Role requirement**: Any authenticated user can export their **own** data.
> No elevated role is required.

---

## EXPORT USER DATA

Trigger an asynchronous export job that flushes all user tables, copies the
Parquet files, and bundles them into a ZIP archive.

```sql
EXPORT USER DATA;
```

The command returns immediately with a **Job ID**. The export runs in the
background and can take up to a few minutes on large data sets (the executor
first flushes all buffered writes to Parquet before copying).

### Idempotency

Only **one export per user per calendar day** can be active at a time. If you
run `EXPORT USER DATA` more than once on the same day, the second call returns
the existing job ID rather than creating a duplicate job.

### Response

```
User data export started. Job ID: UE-abc123. Use SHOW EXPORT to check status and get the download link.
```

### Example

```sql
-- Export all your data
EXPORT USER DATA;
```

---

## SHOW EXPORT

Check the status of your export jobs and, once complete, retrieve the download
URL.

```sql
SHOW EXPORT;
```

Returns the most recent 20 export jobs for the calling user, sorted by creation
time descending.

### Result columns

| Column | Description |
|---|---|
| `job_id` | The export job ID (prefix `UE-`) |
| `status` | Job status: `Queued`, `Running`, `Completed`, `Failed` |
| `created_at` | When the export job was created |
| `completed_at` | When the job finished (null if not yet done) |
| `download_url` | Full URL to download the ZIP (populated only when `status = Completed`) |

### Example

```sql
-- Check status of your latest export
SHOW EXPORT;
```

**Sample output (completed)**

| job_id | status | created_at | completed_at | download_url |
|---|---|---|---|---|
| UE-abc123 | Completed | 2026-02-25 13:00:00 | 2026-02-25 13:01:30 | http://localhost:2900/v1/exports/alice/export-alice-20260225-130000.zip |

---

## Downloading the export

Once `SHOW EXPORT` shows `Completed`, use the `download_url` to fetch your ZIP
with any HTTP client. Authenticate as the exporting user using a **Bearer
token**.

### Using curl

```bash
# 1. Get access token via Basic auth
TOKEN=$(curl -s -u alice:Password123 http://localhost:2900/v1/auth/token \
  | jq -r '.access_token')

# 2. Download the ZIP with Bearer auth
curl -H "Authorization: Bearer $TOKEN" \
     -o my_data.zip \
     "http://localhost:2900/v1/exports/alice/export-alice-20260225-130000.zip"
```

### Access control

The download endpoint enforces strict ownership:

- The authenticated user **must** match the `{user_id}` in the URL path.
- Any other authenticated user receives **403 Forbidden**.
- Unauthenticated requests receive **401 Unauthorized**.

---

## ZIP archive structure

The downloaded ZIP contains one Parquet file per table that belongs to the
exporting user, laid out as:

```
export-<user_id>-<timestamp>.zip
└── <namespace>/<table_name>/
    └── <parquet_segment_file>.parquet
    └── ...
```

Each Parquet file contains all rows written by the user for that table (after
the flush-first pass ensures buffered writes are included).

---

## Async job execution

Export jobs run as background `UserExport` jobs in the `UnifiedJobManager`.
Monitor progress via SQL:

```sql
SELECT job_id, status, message
FROM system.jobs
WHERE job_type = 'UserExport'
ORDER BY created_at DESC
LIMIT 5;
```

| `status` value | Meaning |
|---|---|
| `Queued` | Export is waiting to start |
| `Running` | Flushing tables and copying Parquet files |
| `Completed` | ZIP file is ready to download |
| `Failed` | An error occurred — see `message` column for details |

---

## GDPR / data portability

`EXPORT USER DATA` is designed to support data portability requirements. The

written, in the open Parquet columnar format — compatible with DuckDB, Apache
Spark, pandas, and most data tools.

================================================================================
# User Management
URL: https://kalamdb.org/docs/server/sql-reference/users
Source: content/server/sql-reference/users.mdx

> User management SQL commands in KalamDB. Learn CREATE USER, ALTER USER, DROP USER syntax with role-based access control (RBAC) and permission settings.
# User Management

KalamDB includes built-in authentication and role-based access control (RBAC).

## CREATE USER

```sql
CREATE USER '<username>'
  WITH <PASSWORD '<password>' | OIDC '<json-auth-data>'
  ROLE <user|service|dba|system>
  [EMAIL '<email>'];
```

### Roles

| Role | Description |
|------|-------------|
| `user` | Standard user — access to own tables |
| `service` | Service account — for bots and integrations |
| `dba` | Database administrator — full table management |
| `system` | System-level — unrestricted access |

### Examples

```sql
-- Create a regular user with password
CREATE USER 'alice'
  WITH PASSWORD 'SecurePass123!'
  ROLE user
  EMAIL 'alice@example.com';

-- Create a service account
CREATE USER 'chat_bot'
  WITH PASSWORD 'BotSecret456!'
  ROLE service;

-- Create an OIDC-mapped user (issuer + subject are required)
CREATE USER 'google-user-123'
  WITH OIDC '{"issuer":"https://accounts.google.com","subject":"google-user-123"}'
  ROLE user
  EMAIL 'alice@example.com';

-- Create a DBA
CREATE USER 'admin'
  WITH PASSWORD 'AdminPass789!'
  ROLE dba
  EMAIL 'admin@example.com';
```

## ALTER USER

```sql
-- Change password
ALTER USER '<username>' SET PASSWORD '<new_password>';

-- Change role
ALTER USER '<username>' SET ROLE <user|service|dba|system>;

-- Update email
ALTER USER '<username>' SET EMAIL '<new_email>';
```

### Examples

```sql
ALTER USER 'alice' SET PASSWORD 'NewSecurePass!';
ALTER USER 'alice' SET ROLE dba;
ALTER USER 'alice' SET EMAIL 'alice.new@example.com';
```

## DROP USER

```sql
DROP USER '<username>';
DROP USER IF EXISTS '<username>';
```

Dropping a user also removes their per-user storage directory, providing GDPR-compliant data deletion.

```sql
DROP USER IF EXISTS 'alice';
```

================================================================================
# Getting Started with Topics
URL: https://kalamdb.org/docs/server/topic-pubsub/getting-started
Source: content/server/topic-pubsub/getting-started.mdx

> Build a simple USER-table chat flow with KalamDB topics, preserve producer user identity in topic messages, and write assistant replies back into the same table.
# Getting Started with Topics

Use this page to build the smallest useful pub/sub flow in KalamDB.

We will keep it single-partition and minimal, but this version uses a `USER` table because that is the right starting point for per-user chat and agent workflows.

## Why use pub/sub here?

Without pub/sub, an AI worker has to keep polling a table to discover new prompts.

With topics:

- your app writes a normal SQL row
- KalamDB turns that row change into a durable topic message
- the AI worker consumes the message
- the worker writes the reply back with normal SQL

That gives you a clean handoff between user-facing writes and background AI work.

## Why use a USER table here?

`demo.messages` is a logical table name shared by all users, but each authenticated user writes into their own isolated USER-table scope.

That means:

- Alice writes her own `demo.messages` rows
- Bob writes his own `demo.messages` rows
- the agent can see which user produced the topic message
- the agent can write the assistant reply back into the same USER table for that same user

There is no separate `tenant` field in the current topic message envelope. For USER tables, the producing `user` is the ownership signal.

## 1. Create a namespace and a USER table

This table keeps both user and assistant messages together.

```sql
CREATE NAMESPACE demo;

CREATE USER TABLE demo.messages (
  id TEXT PRIMARY KEY DEFAULT ULID(),
  author TEXT NOT NULL,
  msg TEXT NOT NULL,
  attachment FILE,
  created TIMESTAMP NOT NULL DEFAULT NOW()
);
```

Notes:

- `ULID()` gives you a sortable text ID
- `attachment FILE` is optional
- the owning user does not need a separate column because USER-table scope already tracks that

## 2. Create a single-partition topic

```sql
CREATE TOPIC demo.message_events PARTITIONS 1;
```

Use `PARTITIONS 1` for now. Multi-partition topics still have experimental edges in the current implementation.

## 3. Route message writes into the topic

```sql
ALTER TOPIC demo.message_events
ADD SOURCE demo.messages
ON INSERT
WITH (payload = 'full');
```

Now every inserted row in `demo.messages` becomes one topic message in `demo.message_events`.

If you also want edit and delete events, add optional routes:

```sql
ALTER TOPIC demo.message_events ADD SOURCE demo.messages ON UPDATE WITH (payload = 'full');
ALTER TOPIC demo.message_events ADD SOURCE demo.messages ON DELETE WITH (payload = 'full');
```

When those routes are active, the topic envelope changes `op` to `Update` or `Delete` accordingly.

## 4. Let a user write a message

```sql
INSERT INTO demo.messages (author, msg)
VALUES ('user', 'Write a short summary of this support ticket.');
```

If `user_123` runs that insert while authenticated as `user_123`, the new row belongs to `user_123`'s USER-table scope.

That insert is the only thing the app has to do. KalamDB handles topic publication inside the write path.

## 5. Inspect the topic in SQL

```sql
CONSUME FROM demo.message_events
FROM EARLIEST
LIMIT 1;
```

The result contains `topic`, `partition`, `offset`, `key`, `payload`, `timestamp_ms`, and `op`.

In this example, the binary `payload` contains the inserted row as JSON bytes plus `_table: "demo.messages"` because the route uses `payload = 'full'`.

The current SQL result columns are `topic_id`, `partition_id`, `offset`, `key`, `payload`, `timestamp_ms`, `user_id`, and `op`.

SQL `CONSUME` is useful for inspection, and it now includes the producing `user_id` when the message came from a USER-table scoped write.

Because this example omits `GROUP`, it is stateless. Running it again reads from `FROM EARLIEST` again and does not move any worker cursor.

## 6. What the agent actually needs from the topic message

The current topic message design does preserve the producing user for USER-table writes.

When the agent consumes through the HTTP topic API or the SDK consumer, the message envelope includes:

```json
{
  "topic_id": "demo.message_events",
  "partition_id": 0,
  "offset": 0,
  "user": "user_123",
  "op": "Insert"
}
```

And the decoded `payload` looks like:

```json
{
  "id": "01HS7Y6W0R5J4Q1M9A1Q7X2ABC",
  "author": "user",
  "msg": "Write a short summary of this support ticket.",
  "attachment": null,
  "created": "2026-04-18T12:00:00Z",
  "_table": "demo.messages"
}
```

This is the current identity story in topic messages:

- `user` tells the agent which USER-table scope owns the row
- `op` tells the agent whether the event came from an insert, update, or delete route
- `_table` tells the agent which table produced the message
- there is no separate tenant field beyond that user scope today

## 7. Let the agent write back into the same USER table

Once the agent knows the producing `user`, it can write the assistant reply into that same user's `demo.messages` table scope by using a client authenticated as that user:

```sql
EXECUTE AS 'user_123' (
  INSERT INTO demo.messages (author, msg)
  VALUES ('assistant', 'Here is a short summary of the ticket...')
);
```

That is the key reason the producing `user` field matters for USER-table topic consumers.

## 8. Read the conversation back

```sql
SELECT id, author, msg, created
FROM demo.messages
ORDER BY created;
```

## What this pattern gives you

- the app keeps using simple SQL writes
- each user keeps their own isolated `demo.messages` rows
- the AI worker gets a durable queue of new work
- the worker can identify the producing user from the topic message
- the assistant reply stays in the same USER table conversation history

## Important notes for this simple flow

- keep the topic at one partition
- because the agent writes back into the same table, its assistant insert will also create a topic message
- today, route `WHERE` filters are stored but not enforced in the publisher path, so do not rely on `WHERE author = 'user'` yet
- instead, make the agent ignore messages where the decoded payload already has `author = 'assistant'`
- if you need the producing `user`, consume through the [HTTP Topic API](https://kalamdb.org/docs/server/topic-pubsub/http-topic-api) or an SDK consumer, not SQL `CONSUME`
- for the full command surface and current caveats, see [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics)

================================================================================
# HTTP Topic API
URL: https://kalamdb.org/docs/server/topic-pubsub/http-topic-api
Source: content/server/topic-pubsub/http-topic-api.mdx

> Consume and acknowledge KalamDB topic messages over HTTP with code-accurate details on offsets, partition targeting, claims, and current limitations.
# HTTP Topic API

Use the HTTP topic endpoints when you want explicit consume and ack control from a worker runtime or service.

The endpoints live under `/v1/api/topics` and currently expose:

- `POST /consume`
- `POST /ack`

## Auth and roles

Both endpoints require:

- `Authorization: Bearer <JWT_TOKEN>`
- role in `{service, dba, system}`

`user` role is rejected.

## `POST /v1/api/topics/consume`

Request:

```json
{
  "topic_id": "app.user_events",
  "group_id": "ai-worker",
  "start": "Earliest",
  "limit": 100,
  "partition_id": 0,
  "timeout_seconds": 10
}
```

`start` accepts:

- `"Latest"` (default)
- `"Earliest"`
- `{ "Offset": 123 }`

Current consume behavior:

- one request reads one `topic_id` + one `partition_id`
- omit `group_id` for stateless inspection; the request honors `start` on every call and does not create group claims or offsets
- include `group_id` for durable consumer-group reads
- if the group already has a committed offset for that partition, the handler resumes from `last_acked_offset + 1` and ignores `start`
- if the group has no committed offset yet, `start` decides where reading begins
- the HTTP handler does not auto-ack the returned batch
- the request model accepts `timeout_seconds`, but the current handler does not wait on that field yet; clients should poll explicitly
- use `partition_id: 0` unless you are deliberately experimenting with multi-partition topics

SQL `CONSUME ... GROUP` follows the same resume rule for existing groups. Both SQL and HTTP grouped consumers require explicit `ACK` to persist progress.

Response:

```json
{
  "messages": [
    {
      "topic_id": "app.user_events",
      "partition_id": 0,
      "offset": 10,
      "payload": "<base64>",
      "key": "optional-key",
      "timestamp_ms": 1730000000000,
      "user": "user_123",
      "op": "Insert"
    }
  ],
  "next_offset": 11,
  "has_more": false
}
```

Response notes:

- `payload` is base64 because the stored message body is binary in the API contract
- `op` is `Insert`, `Update`, or `Delete`
- `has_more` is `true` when the batch size hit the requested `limit`

## `POST /v1/api/topics/ack`

Request:

```json
{
  "topic_id": "app.user_events",
  "group_id": "ai-worker",
  "partition_id": 0,
  "upto_offset": 10
}
```

This commits offset `10` inclusively. The next HTTP consume for the same topic, group, and partition resumes from `11`.

Response:

```json
{
  "success": true,
  "acknowledged_offset": 10
}
```

Current ack behavior:

- `ACK` is persisted in `system.topic_offsets`
- acknowledgements are monotonic, so lower offsets do not move a group backward
- acknowledgements are tracked per topic, group, and partition

## Claim and redelivery behavior

When HTTP consume returns messages for a group, KalamDB creates an in-memory claim for that `(topic, group, partition)` range.

- another consumer in the same group and partition does not get that same range immediately
- if the batch is not acknowledged before the visibility timeout expires, the claim can expire and the range can be re-delivered
- the default visibility timeout is `60` seconds and comes from `topics.visibility_timeout_secs`

Configure it in `server.toml`:

```toml
[topics]
visibility_timeout_secs = 60
```

You can override it at startup with `KALAMDB_TOPIC_VISIBILITY_TIMEOUT_SECS`. `KALAMDB_VISIBILITY_TIMEOUT_SECS` is also accepted as a legacy alias.

Treat topic consumption as at-least-once delivery and keep handlers idempotent.

## Current limitations

- single-partition topics are the safe default today
- there is no built-in `consume every partition` request
- `timeout_seconds` is not active long-polling yet
- for topic setup and inspection, use [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics)

## Error Shape

```json
{
  "error": "...",
  "code": "FORBIDDEN|NOT_FOUND|INTERNAL_ERROR"
}
```

Use SQL-native topic flows in [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics).

================================================================================
# Topic Pub/Sub
URL: https://kalamdb.org/docs/server/topic-pubsub
Source: content/server/topic-pubsub/index.mdx

> Understand how KalamDB topics work today: route metadata, synchronous publish-on-write, RocksDB message storage, consumer offsets, and current limits.
# Topic Pub/Sub

KalamDB topics are a durable table-change feed. Topic definitions live in `system.topics`, matching table writes are turned into topic messages inside the write path, and consumer progress is persisted in `system.topic_offsets`.

If you want an app write to trigger background work without polling a table, topics are the pub/sub layer for that.

A consumer group is just a named cursor in `system.topic_offsets`. Use a group when a worker should resume where it left off, and omit `GROUP` when you want a stateless read for debugging or replay.

## Why topics exist

Use topics when you need:

- normal SQL writes from your app
- durable delivery to workers or services
- replay from stored offsets
- consumer-group progress tracking
- a clean boundary between request-time writes and async work

A common pattern is:

1. your app inserts a row
2. KalamDB publishes a topic message for that row change
3. an agent or worker consumes the message
4. the worker writes results back with SQL

See [Getting Started](https://kalamdb.org/docs/server/topic-pubsub/getting-started) for the smallest SQL-only version of that flow.

## How topics work inside KalamDB

1. `CREATE TOPIC` stores topic metadata and routes in `system.topics`.
2. On startup, KalamDB reloads persisted topics into an in-memory route cache and restores next offsets from stored topic messages.
3. When a table write happens, the table provider calls the topic publisher inline in the write path.
4. In cluster mode, only the leader publishes topic messages so the same row change is not published twice.
5. Each matching row change becomes one topic message.
6. Messages are appended to the internal RocksDB `topic_messages` partition with a `(topic_id, partition_id, offset)` key.
7. Consumer-group acknowledgements are stored in `system.topic_offsets`.
8. While a group is processing a batch, KalamDB keeps an in-memory claim for that `(topic, group, partition)` range. If it is not acknowledged before the visibility timeout expires, it can be delivered again.
9. Retention policies are stored per topic and can prune old messages by age or per-partition byte cap without rewriting offsets.

The claim visibility timeout is controlled by `topics.visibility_timeout_secs` in `server.toml`. Keep it longer than normal worker processing time, but low enough that crashed workers recover quickly. The default is `60` seconds; local smoke tests and fast worker loops often use `10` seconds.

## What a topic message contains

Each message carries:

- the topic id
- the partition id
- the offset inside that partition
- the payload bytes
- an optional key
- the publish timestamp
- the producing user id when available
- the triggering operation: `Insert`, `Update`, or `Delete`

SQL `CONSUME` exposes that field as `user_id`, while the HTTP topic API resolves it to `user` in JSON responses.

## What works well today

- durable topic metadata in `system.topics`
- routing by source table and operation
- row-local `WHERE` filters on topic routes
- synchronous publish-on-write from table mutations
- persisted message log in RocksDB
- consumer-group offsets in `system.topic_offsets`
- HTTP consume plus explicit ack
- startup restoration of routes and next offsets
- topic retention by age or per-partition byte cap
- background cleanup after `DROP TOPIC` or `CLEAR TOPIC`

## Current limits and experimental areas

Use `PARTITIONS 1` in production for now.

Current code still has unfinished edges:

- multi-partition topics have internal plumbing, but the full user-facing story is incomplete
- `partition_key_expr` is stored in route metadata but is not used by the publisher
- SQL `CONSUME` reads partition `0` only
- HTTP consume reads one partition per request
- `payload = 'full'` is the only payload mode you should rely on today
- `payload = 'diff'` is still in development; the parser accepts it, but current publishing code emits the same full-row payload as `full`
- `payload = 'key'` is still in development; current publishing code serializes row JSON instead of a stable primary-key-only payload shape

Retention is now part of the supported topic surface. You can configure it with `CREATE TOPIC ... WITH (...)`, update it with `ALTER TOPIC ... SET RETENTION`, and disable it with `ALTER TOPIC ... CLEAR RETENTION`. See [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics) for the exact syntax and how retention affects `FROM EARLIEST` and consumer-group recovery.

## Choose the right page

- [Getting Started](https://kalamdb.org/docs/server/topic-pubsub/getting-started): simple SQL-only topic flow for an AI worker.
- [SQL Topics & Consumers](https://kalamdb.org/docs/server/topic-pubsub/sql-topics): create topics, attach routes, inspect queue state, and use the SQL consume surface.
- [HTTP Topic API](https://kalamdb.org/docs/server/topic-pubsub/http-topic-api): explicit consume and ack flow for services and workers.

================================================================================
# SQL Topics & Consumers
URL: https://kalamdb.org/docs/server/topic-pubsub/sql-topics
Source: content/server/topic-pubsub/sql-topics.mdx

> Use SQL to create topics, attach source tables, inspect consumer offsets, and understand the current topic semantics and limits in KalamDB.
# SQL Topics & Consumer Groups

Use SQL to create topics, attach source tables, inspect offsets, and do simple topic reads inside KalamDB.

For production workers, the HTTP or SDK consume APIs are still a better fit when you need explicit ack timing and partition selection. The SQL surface is best for setup, inspection, debugging, and simple single-partition flows. SQL consumer groups resume from persisted offsets too, and grouped SQL consumes require an explicit `ACK` after processing.

## Why use groups

Use `GROUP` when a worker should keep its place in a topic.

Without `GROUP`, a consume is stateless. Every call starts from the `FROM` position you requested, so it is good for inspection and replay but not for a durable worker cursor.

With `GROUP`, KalamDB stores progress in `system.topic_offsets` for `(topic_id, group_id, partition_id)`. That gives you:

- resume-after-restart behavior for the same logical worker or worker fleet
- independent read positions for different consumers on the same topic
- a way to replay with one group while production workers keep their own offsets

There is no separate `CREATE GROUP` command. A group is created the first time you consume or acknowledge offsets with a new `group_id`.

## Stateless reads vs durable reads

Use no group when you want a pure viewer:

```sql
CONSUME FROM demo.message_events FROM EARLIEST LIMIT 100;
```

That query does not store progress. Run it again and you can replay the same range again.

Use a group when you want a named cursor:

```sql
CONSUME FROM demo.message_events
GROUP 'ui-debug-admin'
FROM LATEST
LIMIT 100;
```

That query creates or resumes the `ui-debug-admin` cursor for partition `0`.

## Recommended starting point

Use a single-partition topic until multi-partition behavior is fully finished:

```sql
CREATE TOPIC app.user_events PARTITIONS 1;
```

`CREATE TOPIC` requirements today:

- topic names must be namespace-qualified, like `app.user_events`
- the namespace must already exist
- `CREATE TOPIC IF NOT EXISTS` is not currently supported
- topic creation requires `dba` or `system` role

## CREATE TOPIC

```sql
CREATE TOPIC <topic_name>
[PARTITIONS <count>]
[WITH (retention_seconds = <seconds|NULL>, retention_max_bytes = <bytes|NULL>)];
```

When a topic is created, KalamDB persists it in `system.topics` and adds it to the in-memory route cache immediately.

Retention options on `CREATE TOPIC` are optional:

- omit an option to use the server `[topics]` default
- set an option to `NULL` to disable that limit for this topic
- `retention_max_bytes` is enforced per partition, not across the whole topic

Examples:

```sql
CREATE TOPIC app.user_events PARTITIONS 1;

CREATE TOPIC app.user_events
PARTITIONS 1
WITH (retention_seconds = 86400, retention_max_bytes = 268435456);

CREATE TOPIC app.audit_events
PARTITIONS 1
WITH (retention_seconds = 3600, retention_max_bytes = NULL);
```

In the last example, messages are deleted once they are older than one hour, and there is no byte cap for the topic partitions.

## Topic retention

Topic retention is a topic-level policy stored in `system.topics`. It is independent of consumer group ack state.

Current behavior:

- `retention_seconds` deletes messages older than the configured age
- `retention_max_bytes` trims the oldest retained messages until the partition is back under the byte cap
- if both limits are active, age-based pruning runs first and byte-based pruning runs after that
- if both limits are `NULL`, retention is disabled for the topic
- retained message deletion does not rewrite offsets or move consumer group cursors

The default server configuration is 7 days and 1 GiB per partition unless you override it in the `[topics]` config block:

```toml
[topics]
default_retention_seconds = 604800
default_retention_max_bytes = 1073741824
retention_check_interval_seconds = 3600
retention_batch_size = 10000
```

## ALTER TOPIC retention

Use `ALTER TOPIC` when you want to change retention after the topic already exists.

```sql
ALTER TOPIC <topic_name>
SET RETENTION WITH (retention_seconds = <seconds|NULL>, retention_max_bytes = <bytes|NULL>);

ALTER TOPIC <topic_name> CLEAR RETENTION;
```

`SET RETENTION` only changes the options you specify. Omitted options keep their current value.

Examples:

```sql
ALTER TOPIC app.user_events
SET RETENTION WITH (retention_seconds = 604800, retention_max_bytes = 1073741824);

ALTER TOPIC app.user_events
SET RETENTION WITH (retention_seconds = NULL);

ALTER TOPIC app.user_events
SET RETENTION WITH (retention_max_bytes = 536870912);

ALTER TOPIC app.user_events CLEAR RETENTION;
```

Use `CLEAR RETENTION` when you want to disable both limits in one step.

## How retention affects reads

Retention removes stored messages, but it does not renumber topic offsets.

- `CONSUME FROM ... FROM EARLIEST` starts at the earliest currently retained offset
- explicit offsets below that low watermark fail because those messages are no longer available
- a consumer group with a committed offset below the earliest retained offset must be reset before it can continue

Example recovery flow for a lagged group:

```sql
RESET CONSUMER GROUP 'backfill-worker' ON app.user_events TO 250;

CONSUME FROM app.user_events
GROUP 'backfill-worker'
FROM EARLIEST
LIMIT 100;
```

That reset changes the next offset for the group. The `FROM` clause in later grouped reads does not override an already-committed group cursor.

## ADD SOURCE

```sql
ALTER TOPIC <topic_name>
ADD SOURCE <table_name_or_namespace.table_name>
ON <INSERT|UPDATE|DELETE>
[WHERE <filter_expression>]
[WITH (payload = '<key|full|diff>')];
```

The parser accepts `key`, `full`, and `diff`, but only `full` is supported today. `key` and `diff` are still in development.

Current route behavior:

- a route matches by source table and operation
- an optional row-local `WHERE` predicate is evaluated before a message is published
- one changed row becomes one topic message
- route registration updates `system.topics` and the in-memory route cache immediately
- `ALTER TOPIC ... ADD SOURCE` requires `dba` or `system` role

You can add separate routes for `INSERT`, `UPDATE`, and `DELETE`:

```sql
ALTER TOPIC app.user_events ADD SOURCE app.users ON INSERT WITH (payload = 'full');
ALTER TOPIC app.user_events ADD SOURCE app.users ON UPDATE WITH (payload = 'full');
ALTER TOPIC app.user_events ADD SOURCE app.users ON DELETE WITH (payload = 'full');
```

## Payload modes

Only `payload = 'full'` has behavior you should rely on today. `key` and `diff` are still in development:

```sql
ALTER TOPIC app.user_events
ADD SOURCE app.users
ON INSERT
WITH (payload = 'full');
```

What those modes mean in the current code:

| Mode | Current behavior |
| --- | --- |
| `key` | In development. The parser accepts it, but current publishing code still serializes row JSON without a stable primary-key-only payload shape. |
| `full` | Publishes the row JSON and injects `_table` with the source table id. |
| `diff` | In development. The parser accepts it, but current publishing code still publishes the same full-row payload as `full`. |

If you want predictable payloads today, use `payload = 'full'`. Treat `key` and `diff` as in development.

## `WHERE` filters

Use `WHERE` when only some row changes should publish to the topic. The predicate is evaluated against the row routed for the selected operation.

That is useful for worker queues that should only wake up on a subset of writes. For example, route only cancelled tasks into a cleanup worker topic:

```sql
ALTER TOPIC app.task_cancellations
ADD SOURCE app.tasks
ON INSERT
WHERE cancelled = true
WITH (payload = 'full');

ALTER TOPIC app.task_cancellations
ADD SOURCE app.tasks
ON UPDATE
WHERE cancelled = true
WITH (payload = 'full');
```

With those two routes:

- a task inserted with `cancelled = true` publishes immediately
- a task that starts active and is later updated to `cancelled = true` publishes on that update
- rows where `cancelled = false` are ignored by these routes

Keep the predicate row-local. Subqueries are rejected for route filters.

## CONSUME FROM

```sql
CONSUME FROM <topic_name>
[GROUP '<group_id>']
[FROM <LATEST|EARLIEST|offset>]
[LIMIT <count>];
```

Examples:

```sql
CONSUME FROM app.user_events FROM EARLIEST LIMIT 10;

CONSUME FROM app.user_events
GROUP 'ai-worker'
FROM EARLIEST
LIMIT 10;
```

Current SQL consume behavior:

- SQL `CONSUME` reads partition `0` only
- the result columns are `topic_id`, `partition_id`, `offset`, `key`, `payload`, `timestamp_ms`, `user_id`, and `op`
- `payload` is a binary column containing the stored message bytes
- `user_id` is the canonical producing user id when the message came from a USER-table scoped write
- without `GROUP`, the read is stateless
- with `GROUP`, KalamDB first checks `system.topic_offsets` and resumes from the committed offset when one exists
- if a group has no committed offset yet, the `FROM` clause decides the initial position
- with `GROUP`, the returned range is claimed in memory until it is acknowledged or the visibility timeout expires
- use `ACK` to persist progress after processing a grouped batch

If you need explicit ack timing or partition selection, use the [HTTP Topic API](https://kalamdb.org/docs/server/topic-pubsub/http-topic-api).

## New group vs existing group

The important rule is:

- for a brand-new group, `FROM` decides where the first read starts
- for a group that already has a committed offset, KalamDB resumes from that offset and ignores `FROM`

Examples:

New group that tails only future messages:

```sql
CONSUME FROM demo.message_events
GROUP 'ui-debug-admin'
FROM LATEST
LIMIT 100;
```

Use that when you create a fresh debug group and only want messages written after the group starts.

New group that replays the backlog:

```sql
CONSUME FROM demo.message_events
GROUP 'backfill-worker'
FROM EARLIEST
LIMIT 100;
```

Use that when a new worker should process existing messages from the beginning.

Existing group that already consumed part of the topic:

```sql
CONSUME FROM demo.message_events
GROUP 'backfill-worker'
FROM LATEST
LIMIT 100;
```

Even though this query says `FROM LATEST`, KalamDB resumes `backfill-worker` from its committed offset. The group already has a cursor, so the `FROM` clause no longer changes the starting point.

If you created a new group on the fly and expected to see existing messages, do not use `FROM LATEST`. Use `FROM EARLIEST` or `FROM <offset>` for that first consume.

## How to use groups safely

Use the same group id for one logical worker fleet:

```sql
CONSUME FROM app.user_events GROUP 'email-worker' FROM EARLIEST LIMIT 50;
```

Use a different group when you want an independent cursor:

```sql
CONSUME FROM app.user_events GROUP 'audit-replay' FROM EARLIEST LIMIT 50;
```

Use no group when you want a viewer that never advances offsets:

```sql
CONSUME FROM app.user_events FROM EARLIEST LIMIT 50;
```

Use a throwaway group for one-off debugging when you want group semantics without affecting a real worker group:

```sql
CONSUME FROM app.user_events
GROUP 'debug-2026-04-24-admin'
FROM EARLIEST
LIMIT 50;
```

That lets you inspect with a separate durable cursor while production groups keep their own positions.

## ACK

```sql
ACK <topic_name>
GROUP '<group_id>'
[PARTITION <partition_id>]
UPTO OFFSET <offset>;
```

Current ack behavior:

- `ACK` writes to `system.topic_offsets`
- acknowledgements are monotonic, so a lower offset does not move a group backward
- `ACK` is per topic, group, and partition
- `ACK` requires `service`, `dba`, or `system` role

If you are using SQL `CONSUME ... GROUP`, issue `ACK` after you finish processing the returned rows. If the caller does not ACK before the topic visibility timeout, the range can be delivered again.

## RESET CONSUMER GROUP

```sql
RESET CONSUMER GROUP '<group_id>'
ON <topic_name>
[PARTITION <partition_id>]
TO <next_offset>;
```

Use reset when you deliberately want to move a durable group cursor. This is an admin-only operation and affects only the named topic, group, and partition.

Replay a debug group from the beginning:

```sql
RESET CONSUMER GROUP 'ui-debug-admin1' ON blog.summarizer TO 0;

CONSUME FROM blog.summarizer
GROUP 'ui-debug-admin1'
FROM EARLIEST
LIMIT 100;
```

Move a group so the next read starts at offset `250`:

```sql
RESET CONSUMER GROUP 'backfill-worker' ON app.user_events PARTITION 0 TO 250;
```

For inspection without moving or resetting any group, omit `GROUP`:

```sql
CONSUME FROM blog.summarizer FROM 0 LIMIT 100;
```

## CLEAR TOPIC

```sql
CLEAR TOPIC <topic_name>;
```

`CLEAR TOPIC` keeps the topic definition but schedules a background cleanup job to delete stored messages and consumer offsets.

## DROP TOPIC

```sql
DROP TOPIC <topic_name>;
```

`DROP TOPIC` removes the topic definition and route cache entry immediately, then schedules the same background cleanup job for stored messages and offsets.

## Inspect topic state

Topic metadata and consumer progress are queryable:

```sql
SELECT topic_id, partitions, routes, created_at, updated_at
FROM system.topics
ORDER BY topic_id;

SELECT topic_id, group_id, partition_id, last_acked_offset, updated_at
FROM system.topic_offsets
ORDER BY topic_id, group_id, partition_id;
```

The message log itself is internal. You inspect messages with `CONSUME`, not by querying a `system.topic_messages` table.

## Practical guidance

- use `PARTITIONS 1`
- use `payload = 'full'`; it is the only payload mode with fully documented behavior today
- use a separate source table and result table for agent flows so you do not create loops
- treat `key` and `diff` as still in development, and treat route `WHERE` and custom partitioning as incomplete for now

================================================================================
# Authentication
URL: https://kalamdb.org/docs/cli/auth
Source: content/cli/auth.mdx

> Kalam CLI login, logout, whoami, token create, named instances, OIDC modes, and credential management.
# Authentication

## Top-level auth commands

```bash
kalam login --instance prod --url https://db.example.com --user root --password
kalam login --instance prod --url https://db.example.com --oidc
kalam login --instance prod --url https://db.example.com --oidc --no-browser
kalam whoami --instance prod
kalam logout --instance prod
kalam token create --name ci-prod --instance prod
```

| Command | Behavior |
|---------|----------|
| `kalam login` | Saves access and refresh tokens for `--instance` unless `--no-save` |
| `kalam whoami` | Calls `/v1/api/auth/me` with resolved credentials |
| `kalam logout` | Deletes credentials for one instance; `--all` clears every stored instance |
| `kalam token create` | Creates a service user and prints access/refresh tokens for automation |

When `kalam login` succeeds in an interactive terminal, the CLI drops into the SQL shell. With non-interactive stdin/stdout, login exits after saving credentials.

## Connection options

| Option | Description |
|--------|-------------|
| `-u, --url <URL>` | Full server URL or bare host. Loopback defaults to `http://`; other bare hosts default to `https://`. |
| `-H, --host <HOST>` | Host-only alternative; combines with `--port`. |
| `-p, --port <PORT>` | Port with `--host` (default `3000`). |
| `--token <JWT>` | JWT auth token. |
| `--user <USER>` | User/password login identifier. |
| `--password [PASS]` | Basic auth password; prompts when omitted in interactive mode. |
| `--instance <NAME>` | Credential profile name (default `local`). |

URL resolution order:

1. Normalized `--url` if set (rejects embedded credentials, query params, fragments)
2. `http://<host>:<port>` when `--host` is set
3. Stored credentials URL for the selected `--instance`
4. Fallback `http://localhost:2900`

Bare `--url` inputs like `localhost:2900` default to `http://`; hosts like `kalam.masky.app` default to `https://`.

## Named instances

Use separate profiles for `dev`, `staging`, and `prod`:

```bash
kalam login --instance dev --url http://127.0.0.1:2900 --user admin --password
kalam login --instance staging --url https://staging-db.example.com --user admin --password
kalam login --instance prod --url https://db.example.com --user admin --password

kalam whoami --instance prod
```

## OIDC login modes

KalamDB exposes one configured external OIDC provider.

### Browser login with PKCE

```bash
kalam login --instance prod --url https://db.example.com --oidc
```

Opens the provider login page, listens on `http://127.0.0.1:8787/callback`, and exchanges the code through KalamDB.

### Direct device login

```bash
kalam login --instance prod --url https://db.example.com --oidc --no-browser
```

For providers with a device authorization endpoint when you do not want a local browser callback.

### Brokered device login

```bash
kalam login --instance prod --url https://db.example.com --oidc --no-browser --brokered
```

When the CLI host can reach KalamDB but not the OIDC provider directly. `--brokered` requires `--no-browser`.

`kalam login --no-save ...` skips writing credentials to disk.

## Credential flags (scripting)

| Option | Description |
|--------|-------------|
| `--list-instances` | List stored credential instances. |
| `--show-credentials` | Show stored credentials for `--instance`. |
| `--update-credentials` | Login and refresh JWT/refresh token for `--instance`. |
| `--delete-credentials` | Delete credentials for `--instance`. |
| `--save-credentials` | Save credentials after successful user/password login. |

```bash
kalam --list-instances
kalam --show-credentials --instance dev
kalam --update-credentials --instance dev --url http://localhost:2900 --user admin
kalam --delete-credentials --instance dev
```

Interactive equivalents: `\show-credentials`, `\update-credentials`, `\delete-credentials`.

## Project workflow (`kalam init` / `kalam dev`)

`kalam init` writes `.env` with `KALAM_PROFILE=kalam-dev` (pattern: `kalam-<env>` from `[project].default_env`). Workflow commands load that profile from `~/.kalam/` — not from `kalam.toml`.

| Scenario | What happens |
|----------|----------------|
| **Local `kalam dev`** (new server) | After the server is healthy, the CLI logs in as `root` using `kalam/server/server.toml` (`auth.root_password`, default `kalamdb123` for local init) or `kalamdb123` on loopback when no password is configured, then saves JWT credentials for `kalam-dev`. You can also open the [admin UI](http://localhost:2900/ui) or run `kalam --url http://127.0.0.1:2900 --user root --password kalamdb123` while dev is running. |
| **Local `kalam dev`** (server already running) | Precheck verifies saved credentials for `KALAM_PROFILE` |
| **Remote `kalam dev`** (`auto_start_db = false`) | Precheck requires valid saved credentials — run `kalam login --instance kalam-dev` (or your env profile) first |

Override the profile per shell with `KALAM_PROFILE` in `.env` or the environment.

## Related

- [Project Init](https://kalamdb.org/docs/cli/init)
- [Local Development](https://kalamdb.org/docs/cli/dev)
- [Server authentication](https://kalamdb.org/docs/server/getting-started/authentication)
- [Querying & SQL](https://kalamdb.org/docs/cli/querying)

================================================================================
# Topic Consumers
URL: https://kalamdb.org/docs/cli/consumers
Source: content/cli/consumers.mdx

> Consume KalamDB topics from the terminal with kalam --consume and the interactive \\consume command.
# Topic Consumers

Use topic consumer mode to read topic records directly from a terminal.

## Flags

| Option | Description |
|--------|-------------|
| `--consume` | Enable topic consumer mode. |
| `--topic <TOPIC>` | Topic name (required with `--consume`). |
| `--group <GROUP_ID>` | Consumer group (offsets committed when set). |
| `--from <earliest\|latest\|OFFSET>` | Start position; supports numeric offset. |
| `--consume-limit <N>` | Maximum messages before exit. |
| `--consume-timeout <SECONDS>` | Stop after idle window. |

`--consume` takes precedence over `--file` and `--command`.

## Examples

```bash
# Read newest events and keep running
kalam --consume --topic blog.summarizer --from latest

# Replay from earliest with bounded run
kalam --consume --topic blog.summarizer --group summarizer-agent --from earliest --consume-limit 50 --consume-timeout 60
```

## Interactive command

```sql
\consume blog.summarizer --group summarizer-agent --from earliest --limit 25 --timeout 45
```

## SQL alternative

For explicit offset control, use SQL:

```sql
CREATE TOPIC app.new_messages PARTITIONS 4;
ALTER TOPIC app.new_messages ADD SOURCE app.messages ON INSERT WITH (payload = 'full');
CONSUME FROM app.new_messages GROUP 'worker-1' FROM EARLIEST LIMIT 100;
ACK app.new_messages GROUP 'worker-1' UPTO OFFSET 99;
```

`--consume` and `\consume` are convenience shells around topic consumption. Use `CONSUME FROM` and `ACK` when scripting.

## Related

- [Topic pub/sub](https://kalamdb.org/docs/server/topic-pubsub)
- [TypeScript consumers](https://kalamdb.org/docs/ts-sdk/consumers)

================================================================================
# Local Development
URL: https://kalamdb.org/docs/cli/dev
Source: content/cli/dev.mdx

> Run kalam dev to start the local server, apply schema, regenerate SDK types, watch schema.sql, and multiplex frontend/agent logs.
# Local Development

`kalam dev` is the canonical local orchestration entry point for KalamDB projects. It reads `kalam.toml`, prepares the development database, runs the schema pipeline, regenerates language artifacts, watches `schema.sql` for changes, and supervises configured child processes.

## Quick start

```bash
cd my-app
kalam dev
```

With environment overrides:

```bash
kalam dev --env dev --namespace app --project-dir .
```

Retry a paused schema pipeline on startup (skips draft confirmation prompts):

```bash
kalam dev --force
```

## What `kalam dev` does

On startup, in order:

1. **Resolve environment** — URL, namespace, and env name from flags, environment variables, then `kalam.toml`
2. **Prechecks** — schema source exists, codegen target dirs, migrations dir, `kalam/.gitignore`, CLI log dir, server reachability or binary availability
3. **Authentication (early)** — for **remote** servers (`auto_start_db = false`) or when a **local server is already running** at the configured URL; verifies saved credentials or prompts you to run `kalam login`
4. **Local server** — when `[dev].auto_start_db = true` and nothing is listening yet, download (interactive) / locate `kalamdb-server`, spawn it, wait for health, then **log in as local root** and save credentials to `~/.kalam/` for profile `kalam-<env>` (for example `kalam-dev`)
5. **Schema pipeline** — diff `schema.sql`, update `_draft.sql`, apply numbered migrations, regenerate types when configured
6. **Draft prompt** — if `_draft.sql` remains and you did not pass `--force`, show Apply / Reset / Cancel
7. **Child processes** — start commands from `[dev.processes]` when configured
8. **File watch loop** — poll the schema source every 2 seconds when both `[dev].watch` and `[schema].watch` are `true`

Press `Ctrl+C` to shut down managed processes cleanly.

If you have no `[dev.processes]`, no local server was spawned, and file watch is disabled, the session may exit after the schema pipeline completes.

## Dev flags

| Flag | Description |
|------|-------------|
| `--project-dir <PATH>` | Project root containing `kalam.toml` |
| `--env <NAME>` | Environment to use (default from `[project].default_env`) |
| `--namespace <NAME>` | Override namespace for this session |
| `--force` | Skip draft confirmation prompts; use forceful migration apply options; retry a failed schema pipeline once on startup |
| `--progress` | **Deprecated** — ignored; `kalam dev` always uses append-only multiplexed logs |

### Environment resolution

| Setting | Resolution order (first match wins) |
|---------|-------------------------------------|
| Environment name | `--env` → `KALAM_ENV` → `[project].default_env` → `dev` |
| Server URL | `KALAM_URL` → `[connection.<env>].url` |
| Namespace | `--namespace` → `KALAM_NAMESPACE` → `[connection.<env>].namespace` |

Credential profile for workflow auth comes from `.env` (`KALAM_PROFILE`, default `kalam-dev`) or `kalam login --instance kalam-<env>`.

## Configuring `kalam dev`

See [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml) for the full field list. Key sections:

```toml
[project]
package_manager = "pnpm"

[dev]
auto_start_db = true   # start or reuse local KalamDB (false for remote-only projects)
apply_schema = true    # run migration pipeline on startup and on schema changes
generate_types = true  # run kalam schema gen after schema apply
watch = true           # enable dev-side schema polling

# TypeScript init writes `app` automatically; add more keys as needed.
[dev.processes]
app = "pnpm dev"
agent = "dart run bin/agent.dart"
```

Both `[schema].watch` and `[dev].watch` must be `true` for file watching. If either is `false`, `kalam dev` does not poll the schema file.

Package-manager dev commands written by init:

| Manager | `[dev.processes].app` value |
|---------|----------------------------|
| npm | `npm run dev` |
| pnpm | `pnpm dev` |
| yarn | `yarn dev` |
| bun | `bun run dev` |

## Local KalamDB server

When `[dev].auto_start_db = true` (the default for `--server-mode local` during init):

1. **Health check** — if the configured URL already responds, reuse that server (auth is verified during prechecks)
2. **Start binary** — otherwise spawn `kalamdb-server` with config at `kalam/server/server.toml`
3. **Bootstrap auth** — log in as `root` using `auth.root_password` from `kalam/server/server.toml` (default `kalamdb123` for projects scaffolded with `--server-mode local`), or `kalamdb123` on loopback when no password is configured, then save JWT credentials for `kalam-<env>`
4. **Data & logs** — server storage under `kalam/server/data` and `kalam/server/logs`

### Accessing the local server

While `kalam dev` is running, the managed server listens at your configured dev URL (default **`http://localhost:2900`**). Sign in with:

| Field | Value |
|-------|--------|
| User | `root` |
| Password | `kalamdb123` |

**Admin UI** — open [http://localhost:2900/ui](http://localhost:2900/ui) and log in with `root` / `kalamdb123`.

**CLI shell** (another terminal, from any directory):

```bash
kalam --url http://127.0.0.1:2900 --user root --password kalamdb123
```

Or save credentials once:

```bash
kalam login --instance local --url http://127.0.0.1:2900 --user root --password kalamdb123
kalam --instance local
```

**Health check:**

```bash
curl http://127.0.0.1:2900/v1/api/auth/status
```

These credentials come from `auth.root_password` in `kalam/server/server.toml` (default `kalamdb123` for local init). Change the password there before the first `kalam dev` run if you need a different value.

### Reset local database state

To wipe local project state and reset schema on the server:

```bash
# Stop kalam dev first (Ctrl+C)
kalam db reset
kalam dev
```

If `kalam dev` reuses an existing server on localhost (no `kalam/server` directory in the project), the CLI prompts before dropping the namespace. Use `kalam db reset --yes` to confirm without prompting, or in CI/non-interactive shells.

See [Migrations — `kalam db reset`](https://kalamdb.org/docs/cli/migrations#kalam-db-reset) for local vs server behavior and the confirmation rules.

### App environment variables

Scaffolded `.env` / `.env.example` set workflow credentials for the CLI:

```bash
KALAM_PROFILE=kalam-dev
KALAM_NAMESPACE=chat_app
KALAM_USER=root
KALAM_PASSWORD=kalamdb123
```

Browser and agent processes should use the `KALAM_*` / `VITE_KALAM_*` names:

| Process | Variables |
|---------|-----------|
| Node agent (`[dev.processes].agent`) | `KALAM_URL`, `KALAM_USER`, `KALAM_PASSWORD` |
| Vite app (`[dev.processes].app`) | `VITE_KALAM_URL`, `VITE_KALAM_USER`, `VITE_KALAM_PASSWORD` |

Defaults match the local server: `http://127.0.0.1:2900`, user `root`, password `kalamdb123`.

### Finding `kalamdb-server`

The CLI resolves the server binary in this order:

1. `KALAMDB_SERVER_BIN` — absolute path to the binary (use `kalamdb-server.exe` on Windows)
2. Binary colocated next to the `kalam` executable
3. `~/.kalam/bin/kalamdb-server` (managed install directory)
4. `kalamdb-server` on `PATH`

On first `kalam dev` in an interactive terminal, the CLI can **download** the matching server release into `~/.kalam/bin` when no binary is found. Non-interactive environments (CI) must preinstall the server or set `KALAMDB_SERVER_BIN`.

```bash
# Point at a specific binary

kalam dev
```

See [Server binaries](https://kalamdb.org/docs/server/getting-started/binaries) for manual downloads.

## Managed service logs

`kalam dev` multiplexes logs from the local server and every `[dev.processes]` entry into one append-only console stream:

- Each line is prefixed with a stable source label: `[cli]` for workflow messages, `[server]` for the database, and `[<process-key>]` for each `[dev.processes]` entry (for example `[app]`)
- Sources use distinct colors when the terminal supports it
- When `[logging].file = true`, the same lines are mirrored to `kalam/cli/logs/kalam.log` (secrets redacted)

If schema apply fails, the failure is shown prominently and **only the schema pipeline pauses**. Server and frontend logs keep streaming so you can debug without losing context.

## Schema pipeline during dev

When `apply_schema` is enabled and `schema.mode = "sql"`:

1. Diff `schema.sql` against `kalam/.schema-baseline.sql` (when no numbered migrations are pending)
2. Write or update `kalam/migrations/_draft.sql` when changes exist
3. Apply numbered migrations that are not yet on the server (`--force` uses forceful apply options)
4. Update the baseline after a successful apply **unless** a draft is still pending confirmation
5. Run `kalam schema gen` when `generate_types = true`

With `--force`, numbered migrations still apply immediately, but `_draft.sql` is **not** auto-sealed — you still confirm draft application unless you seal/apply through migration commands.

See [Migrations](https://kalamdb.org/docs/cli/migrations) for draft sealing, `kalam db migrate`, and recovery commands.

## Draft migration prompt

When `_draft.sql` exists and you did not pass `--force`, `kalam dev` pauses streaming output and shows a modal selector:

| Choice | Behavior |
|--------|----------|
| **Apply changes** | Seal and apply the draft migration, then refresh baseline/types |
| **Reset and rebuild** | Drop the namespace on the server, clear local numbered migrations + baseline, rebuild from `schema.sql` |
| **Cancel** | Stop `kalam dev` without applying |

In non-interactive terminals, send `y` (apply), `r` (reset), or any other key (cancel).

## File watch behavior

When watch is enabled, `kalam dev` polls the schema file every **2 seconds**. On change:

1. **Debounce** — wait until the file metadata stabilizes (editors often save in bursts; up to ~2s)
2. **Diff** — update `_draft.sql` from the new schema vs baseline when no numbered migrations are pending
3. **Apply** — run the schema pipeline (numbered migrations first; draft stays pending until you confirm)
4. **Regenerate** — refresh TypeScript/Dart outputs when configured
5. **Prompt** — show the draft modal again when `_draft.sql` remains (unless `--force`)

This is **file-based** watch — it reacts to edits on `schema.sql` in your repo. It does not poll `system.tables` on the server.

> **Note:** The legacy `kalam --watch-schema` server-polling mode is [deprecated](https://kalamdb.org/docs/cli/schema-watch#deprecated-kalam---watch-schema). Use `kalam dev` with `schema.sql` instead.

## Local vs remote server mode

Set during `kalam init` or in `kalam.toml`:

- **Local** (`--server-mode local`, `[dev].auto_start_db = true`) — `kalam dev` may download, spawn, or reuse `kalamdb-server`
- **Remote** (`--server-mode remote`, `[dev].auto_start_db = false`) — `kalam dev` connects to `[connection.<env>].url` and never starts a local binary

## Troubleshooting

### `kalamdb-server was not found`

Install or download the server, or set an explicit path:

```bash

kalam dev
```

In CI, preinstall the binary before running `kalam dev`. Interactive download requires a TTY.

### `KALAMDB_SERVER_BIN points to a file that does not exist`

Unset the variable to let the CLI auto-locate the server, or fix the path (include `.exe` on Windows).

### Dev process failed to start (`[dev.processes]`)

The CLI prints the shell command and actionable hints:

- Verify the program exists on `PATH` (or use an absolute path in `kalam.toml`)
- Run the command manually from the project root
- On **Windows**, npm/pnpm/yarn are often `*.cmd` shims — `kalam dev` runs process commands through the system shell (`cmd /C`)

Example fix when pnpm is recorded in `kalam.toml`:

```toml
[dev.processes]
app = "pnpm dev"
```

### Schema pipeline paused

A migration or codegen step failed. Child processes and the local server may keep running while schema work is paused. Retry once at startup:

```bash
kalam dev --force
```

Check `[server]` and `schema` log lines, then fix `schema.sql` or migration files before rerunning.

### Authentication / login errors

For **remote** servers, run `kalam login --instance kalam-<env>` and ensure `.env` sets `KALAM_PROFILE` to that profile.

For **local** servers, `kalam dev` tries saved credentials first, then root login using `kalam/server/server.toml`. If that fails, run `kalam login` or fix `auth.root_password` in the server config. See [Authentication](https://kalamdb.org/docs/cli/auth).

### Schema diff fails on second `kalam dev` with `CREATE TOPIC`

Use Kalam CLI **v0.5.3-rc.1** or newer. Older CLI builds may apply the first migration but fail on the second dev run when diffing schemas that contain `CREATE TOPIC`. Update with `kalam self-update` or build from source.

### Stale local data

When migrations or namespaces look wrong after many dev iterations, reset local project state:

```bash
kalam db reset          # prompts if reusing a non-project server on localhost
kalam db reset --yes    # non-interactive / confirm namespace drop
kalam dev
```

## Related commands

```bash
kalam status              # project + migration summary
kalam schema gen          # regenerate types without dev session
kalam migration status    # applied / pending / failed migrations
kalam db migrate          # apply committed migrations explicitly
kalam db reset            # clear local kalam/server + optional namespace drop (see migrations doc)
```

> **Note:** `kalam deploy` is not supported yet. Use `kalam db migrate` and your own rollout process for production.

## Next

- [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml)
- [Migrations](https://kalamdb.org/docs/cli/migrations)
- [Project Init](https://kalamdb.org/docs/cli/init)

================================================================================
# Kalam CLI
URL: https://kalamdb.org/docs/cli
Source: content/cli/index.mdx

> SQL-first terminal client for KalamDB. Install kalam, scaffold projects, run kalam dev, migrations, queries, subscriptions, and topic consumers.
# Kalam CLI

The Kalam CLI (`kalam`) is the SQL-first terminal client for KalamDB. It ships as a native Rust binary and as the npm package `@kalamdb/cli`.

Use it for:

- **project workflows** — `kalam init`, `kalam dev`, migrations, and schema codegen
- one-shot SQL and interactive shells
- named instance profiles (`dev`, `staging`, `prod`)
- live query subscriptions and topic consumers
- cluster and storage meta-commands

## June 17 v0.5.3-rc.1 CLI Workflow

The June 17 `v0.5.3-rc.1` pre-release makes `kalam init` and `kalam dev` the main local project workflow. Read the [release note](https://kalamdb.org/docs/server/release-v0-5-3-rc-1), then start with [Project Init](https://kalamdb.org/docs/cli/init) and [Local Development](https://kalamdb.org/docs/cli/dev).

## Recommended reading order

### Project workflow

1. [Setup & Install](https://kalamdb.org/docs/cli/setup)
2. [Project Init](https://kalamdb.org/docs/cli/init)
3. [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml)
4. [Local Development](https://kalamdb.org/docs/cli/dev) — includes schema file watch
5. [Migrations](https://kalamdb.org/docs/cli/migrations)

### SQL client

6. [Authentication](https://kalamdb.org/docs/cli/auth)
7. [Querying & SQL](https://kalamdb.org/docs/cli/querying)
8. [Live Subscriptions](https://kalamdb.org/docs/cli/subscriptions)
9. [Topic Consumers](https://kalamdb.org/docs/cli/consumers)
10. [Interactive Commands](https://kalamdb.org/docs/cli/interactive-commands)
11. [SQL Workflows](https://kalamdb.org/docs/cli/workflows)
12. [Options & Timeouts](https://kalamdb.org/docs/cli/options)

## Execution modes

When you run `kalam` without a subcommand, these modes apply (in precedence order):

1. `--subscribe` / `--list-subscriptions` — subscription management
2. `--consume --topic ...` — topic consumer mode
3. SQL execution:
   - `--file <path>` — execute file and exit
   - `--command <sql>` — execute one statement and exit
   - no flags — interactive shell

Project commands (`init`, `dev`, `migration`, `db`, `schema`, `status`, `deploy`) are always top-level subcommands.

## Quick command map

```bash
# Project workflow
kalam init
kalam init --yes --name my-app --languages typescript --package-manager pnpm
kalam dev
kalam migration status
kalam migration seal && kalam db migrate
kalam db reset
kalam schema gen
kalam status

# Interactive SQL shell
kalam

# One-shot SQL
kalam -c "SELECT * FROM system.tables LIMIT 5;"

# Live subscription
kalam --subscribe "SELECT * FROM app.messages"

# Topic consumer
kalam --consume --topic blog.summarizer --group summarizer-agent --from latest

# Login and doctor
kalam login --instance prod --url https://db.example.com --user admin --password
kalam doctor -v
```

## Related docs

- [Server getting started](https://kalamdb.org/docs/server/getting-started)
- [SQL reference](https://kalamdb.org/docs/server/sql-reference)
- [TypeScript ORM](https://kalamdb.org/docs/ts-sdk/orm)
- [TypeScript consumers](https://kalamdb.org/docs/ts-sdk/consumers)

================================================================================
# Project Init
URL: https://kalamdb.org/docs/cli/init
Source: content/cli/init.mdx

> Scaffold a KalamDB project with kalam init — kalam.toml, schema.sql, migrations, TypeScript/Dart templates, package manager selection, and environment linking.
# Project Init

`kalam init` scaffolds a KalamDB project in the current directory (or `--project-dir`). It creates `kalam.toml`, a schema source, `kalam/migrations/`, generated-output paths, optional starter code from embedded templates, or a full example downloaded from the KalamDB repository. When TypeScript is selected, it runs your JavaScript package manager to install SDK dependencies.

Run this once per repository before `kalam dev`, `kalam migration`, or production rollout workflows.

For the full configuration reference, see [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml).

## Quick start

```bash
mkdir my-app && cd my-app
kalam init
kalam dev
```

While `kalam dev` runs, open [http://localhost:2900/ui](http://localhost:2900/ui) and sign in as **`root` / `kalamdb123`**, or use `kalam --url http://127.0.0.1:2900 --user root --password kalamdb123` in another terminal.

For TypeScript projects, init detects npm, pnpm, yarn, or bun on `PATH`, prompts when more than one is installed, writes the choice to `kalam.toml`, and runs `<manager> install` after scaffolding.

Non-interactive bootstrap:

```bash
kalam init --yes \
  --name chat-app \
  --schema-mode sql \
  --languages typescript \
  --template simple-live \
  --package-manager pnpm \
  --server-mode local
```

## What `kalam init` creates

| Artifact | Purpose |
|----------|---------|
| `kalam.toml` | Project config: environments, schema mode, codegen targets, dev orchestration, optional `package_manager` |
| `schema.sql` | Active schema source when `schema.mode = "sql"` (from template or a starter stub) |
| `kalam/migrations/` | Migration history directory (default path; omitted from `kalam.toml` when unchanged) |
| `kalam/.schema-baseline.sql` | Created on first successful schema apply during `kalam dev` |
| `kalam/.gitignore` | Ignores local KalamDB state under `kalam/` (server data, CLI logs, baselines) |
| `kalam/server/server.toml` | Local server config when `--server-mode local` (includes `auth.root_password`) |
| Generated outputs | Paths from `[schema.targets.<language>]` (for example `src/generated/kalam.ts`) |
| Template or example files | Embedded TypeScript starter, or downloaded repository example when selected (default with `--yes`: `simple-live`) |
| `package.json` + lockfile | TypeScript starter dependencies (when TypeScript is selected) |
| `.env` / `.env.example` | Sets `KALAM_PROFILE=kalam-dev`, `KALAM_USER=root`, and `KALAM_PASSWORD=kalamdb123` for local server mode |

`kalam.toml` stores URLs, namespaces, and toolchain choices only — **no secrets**. Authenticate with `kalam login` for remote servers, or let `kalam dev` bootstrap local root credentials on loopback.

### Namespace naming

`[connection.dev].namespace` is derived from the project name using KalamDB identifier rules (for example `chat-app` → `chat_app`). If normalization changes the name, init prints the namespace it chose.

## Init flags

| Flag | Description |
|------|-------------|
| `--name <NAME>` | Project name (written to `[project].name`) |
| `--schema-mode <sql\|remote>` | Active schema source mode (`remote` is rejected until supported) |
| `--languages <typescript,dart>` | Comma-separated codegen targets. Aliases: `ts` → `typescript` |
| `--template <id>` | Embedded TypeScript template or repository example id (default `simple-live` with `--yes`) |
| `--package-manager <npm\|pnpm\|yarn\|bun>` | JavaScript package manager for TypeScript projects |
| `--server-mode <local\|remote>` | Whether `kalam dev` starts a local server or uses an existing URL |
| `--server-url <URL>` | KalamDB URL for remote mode (local defaults to `http://localhost:2900`) |
| `--project-dir <PATH>` | Directory to initialize (default: current directory) |
| `--yes` | Skip prompts; use defaults for unspecified values |

### Defaults with `--yes`

When a flag is omitted and `--yes` is set:

| Setting | Default |
|---------|---------|
| Schema mode | `sql` |
| Languages | `typescript` |
| Template | `simple-live` (TypeScript only) |
| Package manager | First available in order: invoking tool from `npm_config_user_agent` (when present), then `pnpm`, `bun`, `yarn`, `npm` |
| Server mode | `local` |
| Server URL | `http://localhost:2900` |
| Project name | Current directory name, or `my-app` |

## Interactive flow

When run in a TTY without `--yes`, `kalam init` walks through:

1. **Project name**
2. **Schema mode** — SQL file (`schema.sql`) or remote (coming soon)
3. **Language targets** — multi-select TypeScript and/or Dart
4. **Project template** — embedded starters plus repository examples such as `chat-with-ai`, `react-ai-chat`, and `summarizer-agent`
5. **Package manager** — when TypeScript is selected and multiple of npm/pnpm/yarn/bun are on `PATH`
6. **Server mode** — local (`kalam dev` starts or reuses a server) or remote
7. **Server URL** — when server mode is remote

Menus support `Up`/`Down`, `Space` (multi-select), `Enter` to confirm, and `Esc` to cancel.

In CI or piped shells without a TTY, pass `--yes` and every flag you need (including `--package-manager` when multiple managers are installed).

## TypeScript package managers

TypeScript scaffolding lives under the CLI's `ts/` SDK module. Init only runs a package manager when `typescript` is in `--languages` (or selected interactively).

| Behavior | Detail |
|----------|--------|
| Detection | Scans `PATH` for `npm`, `pnpm`, `yarn`, and `bun` |
| Single manager installed | Selected automatically |
| Multiple installed | Interactive prompt; with `--yes`, uses preference order above |
| Explicit flag | `--package-manager pnpm` (or `npm`, `yarn`, `bun`) |
| Persistence | Written to `[project].package_manager` in `kalam.toml` |
| Install step | Runs `<manager> install` in the project root after files are created |
| Dart-only projects | No package manager field and no install step |

If dependency install fails, project files remain on disk. The CLI prints command output and recovery steps (for example `cd <project> && pnpm install`, then `kalam dev`).

## Example `kalam.toml`

This matches the default TypeScript scaffold (local server, SQL schema). TypeScript init **automatically** writes `[dev.processes].app` with the selected package manager's dev command. See [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml) for every field.

```toml
[project]
name = "chat-app"
default_env = "dev"
package_manager = "pnpm"

[connection.dev]
url = "http://localhost:2900"
namespace = "chat_app"

[schema]
mode = "sql"
path = "schema.sql"
watch = true
languages = ["typescript"]

[schema.targets.typescript]
output = "src/generated/kalam.ts"

[migrations]
auto_create = true

[dev]
auto_start_db = true
apply_schema = true
generate_types = true
watch = true

# kalam dev starts this alongside the DB server and schema pipeline.
# Logs are prefixed [app] to distinguish from [cli] and [server].
[dev.processes]
app = "pnpm dev"

[logging]
file = true
capture_process_output = true
```

Add more supervised processes with additional keys under `[dev.processes]` (for example `worker = "cargo run --bin worker"`). The log prefix always matches the key name.

## Link an environment

After init, add or update named environments without re-running init:

```bash
kalam link --env staging --url https://staging-db.example.com --namespace app
```

`kalam link` updates `[connection.<env>]` in `kalam.toml` only. Login credentials stay in `~/.kalam/`.

## Schema commands (no database mutation for gen)

```bash
# Regenerate SDK artifacts from schema.sql
kalam schema gen

# Limit to one language
kalam schema gen --languages typescript

# Pull live schema into schema.sql (remote mode only)
kalam schema pull
```

## Inspect project state

```bash
kalam status
kalam status --env prod --namespace app
```

Reports resolved URL, namespace, schema mode, codegen targets, and migration counts.

## Starters And Examples

TypeScript projects can start from embedded samples or repository examples.

Embedded samples live under `cli/templates/typescript/*` and are compiled into the CLI:

```bash
kalam init --languages typescript --template simple-live
```

`simple-live` is a live-subscription starter with sample `schema.sql`, `src/index.ts`, and SDK dependencies. Init scaffolds `[dev.processes].app` so `kalam dev` can supervise the starter automatically. List available samples interactively during `kalam init`, or pass `--template` with `--yes`.

Repository examples live under [`examples/`](https://github.com/kalamdb/KalamDB/tree/main/examples). They appear in the same `kalam init` template menu even when you installed only the CLI. When selected, the CLI downloads the repository archive and copies the chosen example into your project directory:

```bash
kalam init --yes --languages typescript --template chat-with-ai
kalam dev
```

Available repository example ids include `live-okf-context-sync`, `realtime-ops-feed`, `chat-with-ai`, `react-ai-chat`, and `summarizer-agent`.

## Troubleshooting

### `interactive setup needs a terminal (TTY)`

Run in a regular terminal, or pass non-interactive flags:

```bash
kalam init --yes --name my-app --languages typescript --package-manager npm
```

### No JavaScript package manager on PATH

Install Node.js (includes npm), [pnpm](https://pnpm.io/installation), or [bun](https://bun.sh), open a new terminal, then rerun init. For TypeScript with `--yes`, add `--package-manager <name>` once a manager is available.

### Explicit package manager not found

If you pass `--package-manager npm` but only pnpm is installed, either install npm or choose an installed manager:

```bash
kalam init --yes --languages typescript --package-manager pnpm
```

### Dependency install failed after scaffolding

Project files are already created. Finish manually:

```bash
cd my-app
pnpm install   # or npm / yarn / bun install
kalam dev
```

The CLI surfaces network, permission, Node version, and registry errors when it can infer them from the package manager output.

### Project already exists

If `kalam.toml` is already present in the directory, init refuses to overwrite. Use a different `--project-dir`, or open the existing project and run `kalam dev`.

## Next steps

1. [Local Development](https://kalamdb.org/docs/cli/dev) — run `kalam dev`
2. [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml) — full config field reference
3. [Migrations](https://kalamdb.org/docs/cli/migrations) — draft, seal, and apply schema history

================================================================================
# Interactive Commands
URL: https://kalamdb.org/docs/cli/interactive-commands
Source: content/cli/interactive-commands.mdx

> Backslash meta-commands in the Kalam CLI interactive shell: session, cluster, export/import, credentials, and more.
# Interactive Commands

Launch interactive mode with `kalam` (no `--command` or `--file`). Type SQL directly or use backslash meta-commands. `--command` also accepts meta-commands for one-shot cluster and admin operations.

## Core shell and inspection

| Command | Description |
|---------|-------------|
| `\help`, `\?` | Show help. |
| `\quit`, `\q` | Exit CLI. |
| `\info`, `\session` | Session, server URL, health, version, cluster, config. |
| `\sessions` | Active PostgreSQL gRPC bridge sessions from `system.sessions`. |
| `\history`, `\h` | Open command history menu. |
| `\health` | Unauthenticated public health probes. |
| `\stats`, `\metrics` | Query system stats metrics. |

`\health` uses `/health` and `/v1/api/healthcheck` without authenticated SQL. On remote deployments those endpoints may be localhost-only.

`\sessions` runs:

```sql
SELECT *
FROM system.sessions
ORDER BY last_seen_at DESC, session_id;
```

For active transactions across `pg_kalam` and `/v1/api/sql`:

```sql
SELECT transaction_id, owner_id, origin, state, write_count
FROM system.transactions
ORDER BY origin, transaction_id;
```

See [system views](https://kalamdb.org/docs/server/sql-reference/system-views) for columns and patterns.

## Namespace, schema, impersonation

| Command | Description |
|---------|-------------|
| `\dt`, `\tables` | List tables. |
| `\d <table>`, `\describe <table>` | Describe columns. See [Schema & Query Diagnostics](https://kalamdb.org/docs/server/sql-reference/diagnostics). |
| `\as <user_id> <SQL>` | Wrapper for `EXECUTE AS '<user_id>'`. |
| `\format table\|json\|csv` | Change output format. |
| `\refresh-tables`, `\refresh` | Refresh autocomplete metadata. |

```sql
USE NAMESPACE chat;
SELECT * FROM messages WHERE conversation_id = 42;
\flush table messages
\as user_123 SELECT * FROM app.messages WHERE conversation_id = 42;
```

`\flush` shortcuts `STORAGE FLUSH`:

- `\flush` or `\flush all` — `STORAGE FLUSH ALL` in current namespace
- `\flush table messages` — `STORAGE FLUSH TABLE chat.messages` when namespace is `chat`
- `\flush table billing.invoices` — preserves explicit namespace

See [impersonation](https://kalamdb.org/docs/server/sql-reference/impersonation) for delegation rules.

## Table transfer

| Command | Description |
|---------|-------------|
| `\export <namespace.table> [--user-id <id>] [--output <file.zip>]` | Export table to local ZIP. |
| `\import <namespace.table> <file.zip> [--user-id <id>]` | Import ZIP into target table. |

- Runs async server jobs with progress polling
- User tables require `--user-id` when applicable
- `<namespace.table>` is required; unqualified names are rejected

```sql
\export app.orders --output ./orders-backup.zip
\import app.orders_restored ./orders-backup.zip
\export app.user_profile --user-id user_123 --output ./profile-user123.zip
```

## Credentials

| Command | Description |
|---------|-------------|
| `\show-credentials`, `\credentials` | Show stored credentials for current instance. |
| `\update-credentials <user> <pass>` | Update credentials. |
| `\delete-credentials` | Delete credentials for current instance. |

## Live queries

| Command | Description |
|---------|-------------|
| `\live <SQL>`, `\subscribe <SQL>` | Start live subscription. |

See [Live Subscriptions](https://kalamdb.org/docs/cli/subscriptions).

## Topic consumer

| Command | Description |
|---------|-------------|
| `\consume <topic> [--group NAME] [--from earliest\|latest\|OFFSET] [--limit N] [--timeout SECONDS]` | Consume topic messages. |

See [Topic Consumers](https://kalamdb.org/docs/cli/consumers).

## Cluster and ingest

| Command | Description |
|---------|-------------|
| `\flush [all\|table <table>]` | `STORAGE FLUSH` in current namespace context. |
| `\cluster snapshot` | Trigger cluster snapshot. |
| `\cluster purge --upto <index>` | Purge Raft logs up to index. |
| `\cluster trigger-election` | Trigger elections across groups. |
| `\cluster transfer-leader <node_id>` | Request leader transfer. |
| `\cluster rebalance` | Rebalance data-group leaders. |
| `\cluster stepdown` | Request leader stepdown. |
| `\cluster clear` | Clear older snapshot files. |
| `\cluster list`, `\cluster ls` | Node overview from `system.cluster`. |
| `\cluster list groups` | Raft group details. |
| `\cluster join <node_id> <rpc_addr> <api_addr>` | Add a node at runtime. |

One-shot examples:

```bash
kalam --command "\\cluster list"
kalam --command "\\cluster rebalance"
kalam --command "\\cluster join 2 10.0.0.2:2910 http://10.0.0.2:2900"
```

Cluster routing details: [SQL Workflows](https://kalamdb.org/docs/cli/workflows).

================================================================================
# kalam.toml Reference
URL: https://kalamdb.org/docs/cli/kalam-toml
Source: content/cli/kalam-toml.mdx

> Full kalam.toml configuration reference — project metadata, connections, schema, migrations, dev orchestration, logging, and dev.processes.
# `kalam.toml` Reference

`kalam.toml` is the project configuration file at the repository root. Workflow commands (`kalam init`, `kalam dev`, `kalam migration`, `kalam schema`, `kalam db`, `kalam status`, `kalam link`) read it to resolve environments, schema sources, codegen targets, and dev orchestration.

**Secrets do not belong in `kalam.toml`.** URLs and namespaces are fine; passwords and JWTs stay in `~/.kalam/` (credential profiles like `kalam-dev`).

## Project layout

```text
my-app/
├── kalam.toml
├── schema.sql
├── .env / .env.example
├── kalam/
│   ├── migrations/
│   ├── .schema-baseline.sql
│   ├── .gitignore
│   ├── cli/logs/kalam.log
│   └── server/
│       ├── server.toml
│       ├── data/
│       └── logs/
├── src/generated/kalam.ts    # TypeScript
└── lib/generated/kalam.dart  # Dart
```

The default project data directory is `kalam/` (override with `[project].kalam_dir`).

## Full example

```toml
[project]
name = "my-app"
default_env = "dev"
package_manager = "pnpm"

[connection.dev]
url = "http://localhost:2900"
namespace = "my_app"

[connection.prod]
url = "https://db.example.com"
namespace = "my_app"

[schema]
mode = "sql"
path = "schema.sql"
watch = true
languages = ["typescript", "dart"]

[schema.targets.typescript]
output = "src/generated/kalam.ts"

[schema.targets.dart]
output = "lib/generated/kalam.dart"

[migrations]
auto_create = true

[dev]
auto_start_db = true
apply_schema = true
generate_types = true
watch = true

[dev.processes]
app = "pnpm dev"
worker = "cargo run --bin worker"

[logging]
file = true
capture_process_output = true
```

## `[project]`

| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `name` | yes | — | Project name |
| `default_env` | no | `dev` | Default environment for workflow commands |
| `package_manager` | no | — | `npm`, `pnpm`, `yarn`, or `bun` — written by `kalam init` for TypeScript projects |
| `kalam_dir` | no | `kalam` | Relative path for migrations, local server config, CLI logs, and schema baselines |

## `[connection.<env>]`

| Field | Required | Description |
|-------|----------|-------------|
| `url` | yes | KalamDB HTTP URL |
| `namespace` | yes | Namespace identifier |

Add or update environments without re-running init:

```bash
kalam link --env staging --url https://staging.example.com --namespace my_app
```

## `[schema]`

| Field | Required | Default | Description |
|-------|----------|---------|-------------|
| `mode` | yes | — | `sql` (local file) or `remote` |
| `path` | when `mode = "sql"` | — | Schema source path (typically `schema.sql`) |
| `watch` | no | `true` | Enable file watch together with `[dev].watch` |
| `languages` | no | `["typescript"]` | Codegen targets: `typescript`, `dart` |

Each language in `languages` requires a matching `[schema.targets.<language>]` block.

### `[schema.targets.<language>]`

| Field | Required | Description |
|-------|----------|-------------|
| `output` | yes | Path for generated SDK/schema artifacts (must be unique per language) |

## `[migrations]`

| Field | Default | Description |
|-------|---------|-------------|
| `auto_create` | `true` | Automatically maintain `kalam/migrations/_draft.sql` from schema diffs |
| `dir` | `kalam/migrations` | Migration directory (default; usually omitted from the written file) |

## `[dev]`

| Field | Default | Description |
|-------|---------|-------------|
| `auto_start_db` | `true` | Start or reuse a local `kalamdb-server` (`false` for remote-only) |
| `apply_schema` | `true` | Run the migration pipeline on startup and on schema file changes |
| `generate_types` | `true` | Run `kalam schema gen` after schema apply |
| `watch` | `true` | Poll the schema file during `kalam dev` (requires `[schema].watch = true`) |

## `[dev.processes]`

Optional map of **process name → shell command**. `kalam dev` supervises each entry alongside the database server and schema pipeline.

| Behavior | Detail |
|----------|--------|
| **Init default** | TypeScript projects get `app = "<manager> dev"` automatically (`npm run dev`, `pnpm dev`, `yarn dev`, or `bun run dev`) |
| **Log prefix** | Each key becomes a stable prefix — `[app]`, `[worker]`, etc. — distinct from `[cli]` workflow output and `[server]` database logs |
| **Execution** | Commands run from the project root through the system shell |
| **Multiple processes** | Add any number of entries |
| **Dart-only projects** | Section is omitted; scaffold includes a commented example |

```toml
[dev.processes]
app = "pnpm dev"
agent = "dart run bin/agent.dart"
```

## `[logging]`

| Field | Default | Description |
|-------|---------|-------------|
| `file` | `true` | Write workflow logs to disk |
| `capture_process_output` | `true` | Mirror managed process stdout/stderr into the log file (secrets redacted) |
| `path` | `kalam/cli/logs/kalam.log` | Log file path (default; usually omitted from the written file) |

## Environment resolution

| Setting | Resolution order (first match wins) |
|---------|-------------------------------------|
| Environment name | `--env` → `KALAM_ENV` → `[project].default_env` → `dev` |
| Server URL | `KALAM_URL` → `[connection.<env>].url` |
| Namespace | `--namespace` → `KALAM_NAMESPACE` → `[connection.<env>].namespace` |

Credential profiles follow the pattern `kalam-<env>` in `~/.kalam/`. After `kalam init`, `.env` sets `KALAM_PROFILE=kalam-dev`.

## Validation

The CLI validates `kalam.toml` on load:

- `project.name` must be non-empty
- `project.kalam_dir` must be a relative in-project path
- `schema.path` is required when `schema.mode = "sql"`
- Every `schema.languages` entry needs a matching `[schema.targets.<lang>]`
- Generated output paths must be unique
- `package_manager` must be one of `npm`, `pnpm`, `yarn`, `bun` when set

## Related

- [Project Init](https://kalamdb.org/docs/cli/init) — scaffolding and flags
- [Local Development](https://kalamdb.org/docs/cli/dev) — orchestration and log multiplexing
- [Migrations](https://kalamdb.org/docs/cli/migrations) — draft, seal, and apply

================================================================================
# Migrations
URL: https://kalamdb.org/docs/cli/migrations
Source: content/cli/migrations.mdx

> KalamDB project migrations — draft diffs, seal, apply with kalam db migrate, status, retry, and repair.
# Migrations

KalamDB projects store schema history under `kalam/migrations/` (configured in `[migrations].dir`). The CLI diffs `schema.sql` against a local baseline, maintains a working `_draft.sql`, and tracks applied migrations on the server per namespace.

## Migration files

| File pattern | Meaning |
|--------------|---------|
| `YYYYMMDDHHMMSS_<name>.sql` | Committed, ordered migration with `-- UP` and `-- DOWN` sections |
| `_draft.sql` | Working draft generated from schema changes not yet sealed |

Draft files are **not** applied to production deployments until sealed into a numbered migration.

## Typical development flow

```bash
# 1. Edit schema.sql
vim schema.sql

# 2. Let kalam dev detect changes (or run pipeline manually)
kalam dev

# 3. When prompted, apply the draft — or seal and migrate explicitly:
kalam migration seal
kalam db migrate

# 4. Check state
kalam migration status

# 5. Start fresh local database state (stop kalam dev first)
kalam db reset
kalam dev
```

During `kalam dev`, edits to `schema.sql` update `_draft.sql` automatically. Numbered migrations apply on startup and after each watched change; the draft waits for your confirmation unless you use `kalam dev --force` (which skips the prompt but still leaves sealing/applying the draft to the Apply path or explicit `kalam migration seal`).

## Commands

### `kalam migration create <name>`

Create a numbered migration immediately from the current diff (without going through the draft file):

```bash
kalam migration create add_messages_table
# creates kalam/migrations/20260611120000_add_messages_table.sql
```

### `kalam migration status`

Report migration state for the resolved environment:

```bash
kalam migration status
kalam migration status --env staging
```

Groups: **Applied**, **Pending**, **Failed**, **Applying**.

### `kalam migration seal`

Promote `_draft.sql` into the next numbered migration file (rename only — does not apply to the database):

```bash
kalam migration seal
```

Run `kalam db migrate` afterward to apply the sealed migration.

### `kalam db migrate`

Apply all pending **numbered** migrations to the linked database:

```bash
kalam db migrate
kalam db migrate --env prod
```

- Does not include `_draft.sql` unless you sealed it first
- Ensures the target namespace exists before applying
- Validates checksums for migrations already marked applied

### `kalam db reset`

Clear local dev project state and, when appropriate, drop the linked namespace on the server so the next `kalam dev` can re-apply migrations cleanly.

```bash
kalam db reset
kalam db reset --yes   # confirm namespace drop without prompting
kalam dev
```

**Options**

- `--project-dir <PATH>`
- `--env <ENV>`
- `--yes` — drop the namespace on a remote or non-project server without prompting (required in non-interactive shells when the server is not this project's `kalam/server`)

**Removes locally** (when present):

- `kalam/server/` — entire local server directory (data, logs, and `server.toml`)
- `kalam/.schema-baseline.sql` — schema diff baseline

**Keeps**:

- `kalam/migrations/` — migration SQL files on disk

**Server namespace drop** (when the linked KalamDB server is reachable):

| Server | Namespace drop |
|--------|----------------|
| This project's `kalam/server` on localhost (directory existed before reset) | Automatic — runs `DROP NAMESPACE ... CASCADE` |
| Another KalamDB process on localhost (for example `kalam dev` reusing `:2900` with no local `kalam/server`) | Prompts for confirmation (default **No**) |
| Remote URL (non-loopback) | Prompts for confirmation (default **No**) |

Dropping the namespace removes tables **and** server-side migration records for that namespace (including failed or stuck migrations in `system.migrations`).

If you decline the prompt, or run in a non-interactive shell without `--yes`, local files are still cleared but the server namespace is left unchanged. You may see `migration failed previously` on the next `kalam dev` until you run `kalam db reset --yes` or drop the namespace manually.

Stop `kalam dev` before reset when wiping `kalam/server/` so RocksDB files are not locked.

After reset, run `kalam dev` again. Pending numbered migrations re-apply on startup.

The CLI prints each removed path, reports when nothing was present, and shows whether the namespace was dropped or skipped.

### `kalam migration retry <id>`

Re-queue a **failed** migration for another apply attempt:

```bash
kalam migration retry 20260610120000_add_users
```

### `kalam migration repair <id> --mark-applied`

Manually mark a migration as applied on the server (use when you've fixed state out of band):

```bash
kalam migration repair 20260610120000_add_users --mark-applied
```

## How drafting works

1. **Baseline** — `kalam/.schema-baseline.sql` mirrors the last successfully applied schema
2. **Diff** — the CLI compares `schema.sql` to the baseline and generates UP/DOWN SQL
3. **Draft** — changes land in `kalam/migrations/_draft.sql`
4. **Seal** — `kalam migration seal` renames the draft to `timestamp_name.sql`
5. **Apply** — `kalam db migrate` or the dev pipeline executes UP statements and records checksums on the server
6. **Baseline update** — after apply, baseline is refreshed from `schema.sql`

If the diff is empty, a stale `_draft.sql` is removed automatically.

## Dev vs production behavior

| Context | Draft auto-apply | Auto-create drafts |
|---------|------------------|-------------------|
| `kalam dev` | Prompted (or `--force`) | Yes, when `migrations.auto_create = true` |
| `kalam db migrate` | No — numbered files only | No |

> **Note:** `kalam deploy` is **not supported yet**. For production, seal migrations locally, run `kalam db migrate --env prod`, and use your own rollout process.

## Stuck or failed migrations

During apply, the CLI may prompt when a migration was interrupted or failed previously:

- **Retry** — attempt apply again
- **Skip / mark applied** — for failed migrations when you've verified state manually
- **Abort** — stop the apply run

Pass `--force` on `kalam dev` to retry automatically in some stuck states.

For manual recovery:

```bash
kalam migration status
kalam migration retry <migration_id>
# or
kalam migration repair <migration_id> --mark-applied
```

## Global flags

All migration commands accept:

| Flag | Description |
|------|-------------|
| `--project-dir <PATH>` | Project root with `kalam.toml` |
| `--env <NAME>` | Target environment |

## Migration file format

```sql
-- Migration: add_messages_table
-- Created: 2026-06-11T12:00:00Z

-- UP
CREATE TABLE app.messages (
  id BIGINT PRIMARY KEY,
  body TEXT
);

-- DOWN
DROP TABLE app.messages;
```

## Related

- [Local Development](https://kalamdb.org/docs/cli/dev) — watch loop and draft prompts
- [Project Init](https://kalamdb.org/docs/cli/init) — `kalam.toml` and migrations directory
- [SQL reference](https://kalamdb.org/docs/server/sql-reference)

================================================================================
# Options & Timeouts
URL: https://kalamdb.org/docs/cli/options
Source: content/cli/options.mdx

> Kalam CLI timeout presets, config path, verbose logging, and global utility flags.
# Options & Timeouts

## Timeout and runtime tuning

| Option | Description |
|--------|-------------|
| `--timeout <SECONDS>` | HTTP request timeout (default `30`). |
| `--connection-timeout <SECONDS>` | Connection/TLS handshake timeout (default `10`). |
| `--receive-timeout <SECONDS>` | Receive timeout (default `30`). |
| `--auth-timeout <SECONDS>` | WebSocket auth timeout (default `5`). |
| `--fast-timeouts` | Preset for local development. |
| `--relaxed-timeouts` | Preset for high-latency networks. |
| `--config <PATH>` | Config file (default `~/.kalam/config.toml`). |
| `-v, --verbose` | Verbose logging. |

## Global utility options

| Option | Description |
|--------|-------------|
| `-h, --help` | Print command help. |
| `-V, --version` | Version, commit, branch, build timestamp (`kalam version`). |

## Self-update and diagnostics

```bash
kalam version
kalam doctor
kalam doctor -v
kalam update
```

`kalam doctor` checks binary path, PATH, config, stored credentials, healthcheck reachability, and authenticated identity.

When interactive mode starts, the CLI checks for updates. If a newer release exists, it prints a notice and may add `update:<version>` to the prompt.

## Related

- [Setup & Install](https://kalamdb.org/docs/cli/setup)
- [Server configuration](https://kalamdb.org/docs/server/getting-started/configuration)

================================================================================
# Querying & SQL
URL: https://kalamdb.org/docs/cli/querying
Source: content/cli/querying.mdx

> Run SQL with kalam --command, --file, and interactive mode. Output formats, namespace state, and table footers.
# Querying & SQL

The CLI is SQL-first: if a statement works through `/v1/api/sql`, send it unchanged via `kalam --command`, `kalam --file`, or the interactive shell.

## Execution options

| Option | Description |
|--------|-------------|
| `-c, --command <SQL>` | Execute one statement and exit. |
| `-f, --file <PATH>` | Execute SQL from file and exit. |
| `--format <table\|json\|csv>` | Output format (default `table`). |
| `--json` | Shorthand for `--format json`. |
| `--csv` | Shorthand for `--format csv`. |
| `--no-color` | Disable ANSI colors. |
| `--no-spinner` | Disable spinner animations. |
| `--loading-threshold-ms <MS>` | Spinner threshold in milliseconds. |

```bash
kalam --command "SELECT * FROM system.tables LIMIT 5;"
kalam --command "SELECT * FROM system.tables LIMIT 5;" --json
kalam --file ./setup.sql --csv
kalam --url kalam.masky.app --command "SELECT 1;"
```

## Common patterns

```bash
# Read system metadata
kalam -c "SELECT * FROM system.tables ORDER BY namespace_id, table_name LIMIT 20;"

# DDL / DML
kalam -c "CREATE NAMESPACE chat;"
kalam -c "CREATE TABLE chat.messages (id BIGINT PRIMARY KEY, body TEXT) WITH (TYPE='USER');"
kalam -c "INSERT INTO chat.messages (id, body) VALUES (1, 'hello');"

# Run a whole SQL file
kalam -f ./migrations/001-init.sql
```

## Namespace state in interactive mode

Namespace switching is SQL, not a backslash command:

```sql
USE NAMESPACE chat;
SELECT * FROM messages ORDER BY id DESC LIMIT 20;
\flush table messages
```

After `USE NAMESPACE chat`, `SET NAMESPACE chat`, or `USE chat`, the CLI stores `chat` locally and sends it as `namespace_id` on later requests. Unqualified table names resolve to `chat.<table>` until you switch again. The prompt shows the active namespace as `ns:<name>`.

## Table footer metadata

In `table` format, results end with row count and footer metadata:

```text
(50 rows)

As: alice
Took: 2.146 ms
```

- `As:` — effective user (includes `EXECUTE AS` and `\as`)
- `Took:` — server-reported execution time in milliseconds

## Interactive formatting

| Command | Description |
|---------|-------------|
| `\dt`, `\tables` | List tables. |
| `\d <table>`, `\describe <table>` | Describe columns (`<table>` or `<namespace.table>`). |
| `\format table\|json\|csv` | Change output format for the session. |
| `\refresh-tables`, `\refresh` | Refresh autocomplete table metadata. |

## Schema and plan diagnostics

Inspect schemas and query plans from the shell with SQL meta commands (DBA/System for
`EXPLAIN`; see the diagnostics page for full access rules):

```sql
-- Native schema (rich metadata)
DESCRIBE TABLE dba.notifications;

-- Shorthand (Kalam types via information_schema)
DESCRIBE dba.notifications;

-- Plan without executing
EXPLAIN SELECT id FROM dba.notifications WHERE user_id = 'test' LIMIT 3;

-- Plan with hot/cold scan metrics (keyword order: ANALYZE before VERBOSE)
EXPLAIN ANALYZE VERBOSE
SELECT id, user_id FROM dba.notifications WHERE user_id = 'test' LIMIT 3;
```

In **table** format, `EXPLAIN` results expand multi-line plans to the full terminal width. Use
`\format json` when you need the complete plan string for tooling.

Full reference: [Schema & Query Diagnostics](https://kalamdb.org/docs/server/sql-reference/diagnostics).

## Related

- [SQL reference](https://kalamdb.org/docs/server/sql-reference)
- [Interactive Commands](https://kalamdb.org/docs/cli/interactive-commands)
- [SQL Workflows](https://kalamdb.org/docs/cli/workflows)

================================================================================
# Schema Watch
URL: https://kalamdb.org/docs/cli/schema-watch
Source: content/cli/schema-watch.mdx

> Watch schema.sql during kalam dev. There is no kalam schema watch subcommand; kalam --watch-schema is deprecated legacy server polling.
# Schema Watch

There is **no** `kalam schema watch` subcommand. The `kalam schema` command only exposes `gen` and `pull`.

For local development, schema watching is built into **`kalam dev`**. It polls `schema.sql` on disk, runs the migration pipeline, and regenerates SDK artifacts when configured.

## File watch (`kalam dev`) — recommended

Enable both flags in `kalam.toml`:

```toml
[schema]
watch = true

[dev]
watch = true
apply_schema = true
generate_types = true
```

Then run:

```bash
kalam dev
```

### Behavior

1. Poll `schema.sql` every **2 seconds**
2. On modification, **debounce** until the file is stable (handles editor save bursts)
3. Diff against `kalam/.schema-baseline.sql` and update `kalam/migrations/_draft.sql`
4. Apply pending numbered migrations
5. Regenerate TypeScript/Dart artifacts when `generate_types = true`
6. Prompt to apply the draft (unless `kalam dev --force`, which skips the modal — you still seal/apply through the prompt's Apply path or `kalam migration seal`)

The watch loop runs alongside managed processes (`[dev.processes]`). Schema failures pause only the schema pipeline — other service logs keep streaming.

See [Local Development](https://kalamdb.org/docs/cli/dev) and [Migrations](https://kalamdb.org/docs/cli/migrations) for the full pipeline.

## Deprecated: `kalam --watch-schema`

> **Deprecated.** `kalam --watch-schema` is legacy server-side polling. It is not part of the `kalam schema` subcommand and is not the recommended workflow. Prefer `kalam dev` with a `schema.sql` project. This flag may be removed in a future CLI release.

The old mode polled `system.tables` on the server and reran a shell command when remote DDL changed (for example via the admin UI). It required `--run`:

```bash
kalam --watch-schema --run "npm run schema:gen"
```

| Option | Description |
|--------|-------------|
| `--watch-schema` | Enable remote polling mode (**deprecated**) |
| `--namespace <NAME>` | Limit to namespaces (repeatable) |
| `--table <namespace.table>` | Limit to tables (repeatable) |
| `--run <COMMAND>` | Shell command after changes (**required**) |
| `--run-on-start` | Run command once before polling |
| `--interval <DURATION>` | Poll interval (default `5s`; `500ms`, `2s`, `1m`) |

This mode does **not** read `kalam.toml`, manage migrations, or start child processes. Migrate to `kalam init` + `kalam dev` when possible.

## Related

- [Local Development](https://kalamdb.org/docs/cli/dev)
- [Project Init](https://kalamdb.org/docs/cli/init)
- [Migrations](https://kalamdb.org/docs/cli/migrations)
- [TypeScript ORM](https://kalamdb.org/docs/ts-sdk/orm)

================================================================================
# Setup & Install
URL: https://kalamdb.org/docs/cli/setup
Source: content/cli/setup.mdx

> Install the Kalam CLI via npm, install.sh, GitHub binaries, or from source. Verify with kalam doctor and kalam version.
# Setup & Install

## Option 1: npm global install

```bash
npm install -g @kalamdb/cli
```

Use this when your machine already manages developer tools through Node/npm. The package downloads the matching native release binary during install.

## Option 2: `install.sh`

```bash
curl -fsSL https://kalamdb.org/install.sh | sh
```

Pin an exact release:

```bash
curl -fsSL https://kalamdb.org/install.sh | sh -s -- --version 0.5.1-beta.2
```

Install the latest GitHub prerelease:

```bash
curl -fsSL https://kalamdb.org/install.sh | sh -s -- --pre-release
```

Environment variables for CI:

```bash
curl -fsSL https://kalamdb.org/install.sh | KALAM_VERSION=0.5.1-beta.2 sh
curl -fsSL https://kalamdb.org/install.sh | KALAM_PRE_RELEASE=1 sh
curl -fsSL https://kalamdb.org/install.sh | KALAM_INSTALL_DIR=/usr/local/bin sh
curl -fsSL https://kalamdb.org/install.sh | KALAM_NO_MODIFY_PATH=1 sh
```

| Input | Effect |
|-------|--------|
| `--version <version>` or `KALAM_VERSION` | Install an exact release. Leading `v` is accepted. |
| `--pre-release` or `KALAM_PRE_RELEASE=1` | Install the latest GitHub prerelease. |
| `KALAM_INSTALL_DIR` | Custom install directory (default `$HOME/.kalam/bin`). |
| `KALAM_NO_MODIFY_PATH=1` | Skip shell profile `PATH` edits. |

## Option 3: GitHub release binaries

See [Server binaries](https://kalamdb.org/docs/server/getting-started/binaries) for prebuilt downloads without a Rust toolchain.

## Option 4: Build from source

```bash
cd cli
cargo build --release
./target/release/kalam --help
```

## Verify installation

```bash
kalam --help
kalam version
kalam doctor
```

- `kalam version` — packaged version plus commit, branch, and build timestamp
- `kalam doctor` — compact status by default; pass `-v` for full detail

## Self-update

```bash
kalam update
```

Downloads a verified release archive, verifies `SHA256SUMS`, and replaces the installed native binary.

When interactive mode starts, the CLI checks for updates. If a newer release exists, it prints a one-time notice and may add an `update:<version>` segment to the prompt.

## KalamDB Skills for coding agents

Load KalamDB commands, SQL syntax, and SDK patterns into Codex, Claude Code, OpenCode, and Agent Skills-compatible tools:

```bash
npx skills add kalamdb/kalamdb-skills
```

## JavaScript package managers (TypeScript projects)

`kalam init` supports **npm**, **pnpm**, **yarn**, and **bun** for installing TypeScript starter dependencies. Install any one of them before scaffolding a TypeScript project, or pass `--package-manager` with `kalam init --yes`. The choice is stored in `kalam.toml` as `[project].package_manager`.

## Next

- [Project Init](https://kalamdb.org/docs/cli/init) — scaffold `kalam.toml` and start with `kalam dev`
- [kalam.toml Reference](https://kalamdb.org/docs/cli/kalam-toml) — full project configuration reference
- [Authentication](https://kalamdb.org/docs/cli/auth)
- [Querying & SQL](https://kalamdb.org/docs/cli/querying)
- [Options & Timeouts](https://kalamdb.org/docs/cli/options)

================================================================================
# Live Subscriptions
URL: https://kalamdb.org/docs/cli/subscriptions
Source: content/cli/subscriptions.mdx

> Kalam CLI live query subscriptions with --subscribe, \\live, OPTIONS clauses, and batching behavior.
# Live Subscriptions

## Flags

| Option | Description |
|--------|-------------|
| `--subscribe <SQL>` | Run a live query subscription. |
| `--subscription-timeout <SECONDS>` | Idle timeout after initial data (`0` = no timeout). |
| `--initial-data-timeout <SECONDS>` | Max wait for initial data batch (`0` = no timeout). |
| `--list-subscriptions` | Print current subscription model/capabilities. |

## Interactive commands

| Command | Description |
|---------|-------------|
| `\live <SQL>`, `\subscribe <SQL>` | Start live subscription (`\subscribe` is an alias). |

Both routes send a `SELECT` plus optional trailing `OPTIONS (...)` to the WebSocket subscription path.

## OPTIONS clauses

| Clause | Effect |
|--------|--------|
| `OPTIONS (batch_size=<n>)` | Limits each initial snapshot batch to at most `n` rows. |
| `OPTIONS (last_rows=<n>)` | Rewinds the newest `n` rows as one startup batch. |
| `OPTIONS (from=<seq_id>)` | Starts live delivery after a sequence id (`from_seq_id` alias accepted). |

Combine options: `OPTIONS (last_rows=20, batch_size=5, from=1234)`.

## Examples

```bash
kalam --subscribe "SELECT * FROM app.messages WHERE conversation_id = 7 OPTIONS (batch_size=5)"
kalam --subscribe "SELECT * FROM app.messages OPTIONS (last_rows=20)" --subscription-timeout 15
kalam --subscribe "SELECT * FROM app.messages OPTIONS (last_rows=20, batch_size=5, from=1234)"
```

```text
kalam> \live SELECT * FROM app.messages OPTIONS (batch_size=5);
kalam> \subscribe SELECT * FROM app.messages OPTIONS (last_rows=20);
kalam> \live SELECT * FROM app.messages OPTIONS (last_rows=20, batch_size=5, from_seq_id=1234);
```

## Verifying batching

- The CLI prints `BATCH <n>` for each startup snapshot page.
- With `OPTIONS (batch_size=5)` and 20 rows, expect four startup batches of 5.
- Startup batches run oldest-to-newest in normal batch mode.
- `last_rows` is a single-batch rewind of the newest rows, not paginated history.

Example output:

```text
[12:00:00.123] ✓ SUBSCRIBED [sub_...] 0 total rows, batch 1 (loading...), 3 columns
[12:00:00.124] BATCH 1 [sub_...] 5 rows (more pending)
  {"id":1,"message":"hello 1"}
[12:00:00.130] BATCH 2 [sub_...] 5 rows (more pending)
...
[12:00:00.145] BATCH 4 [sub_...] 5 rows (complete)
```

## Related

- [TypeScript subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [SQL Workflows](https://kalamdb.org/docs/cli/workflows)

================================================================================
# SQL Workflows
URL: https://kalamdb.org/docs/cli/workflows
Source: content/cli/workflows.mdx

> Cluster routing, impersonation, backup, restore, export, and topic pub/sub SQL workflows in the Kalam CLI.
# SQL Workflows

Many CLI workflows are plain SQL through `-c`, `-f`, or the interactive shell. Backslash commands cover CLI-managed operations; this page covers SQL-first workflows.

## Follower write forwarding

KalamDB uses Multi-Raft groups — a client can send a write to any reachable node.

```bash
kalam --url http://node-2:2900 --command "INSERT INTO app.messages (id, body) VALUES (101, 'hello')"
```

Assume effective user `user-42` hashes to `DataUserShard(7)`:

1. Request lands on node 2 even if it is a follower for that shard.
2. KalamDB classifies SQL and derives the target Raft group from table type and `user_id`.
3. User/stream tables route by hashing `user_id` into `cluster.user_shards` groups.
4. Non-leader nodes forward SQL, params, auth, and request id over gRPC to the group leader.
5. Leader executes, replicates, commits, and returns the result.

The client does not need to discover the leader first.

### Multi-Raft routing today

- One metadata Raft group plus multiple user data groups.
- User/stream data routes by `user_id` for write-path locality.
- Shared tables route to a single shared data group today; partition-by-key is planned.

See [clustering](https://kalamdb.org/docs/server/architecture/clustering) and [table types](https://kalamdb.org/docs/server/architecture/table-types).

## Impersonation

```sql
EXECUTE AS 'user_123' (
  SELECT * FROM app.messages ORDER BY id DESC LIMIT 20
);
```

Or the `\as` shortcut — see [Interactive Commands](https://kalamdb.org/docs/cli/interactive-commands).

## Backup and restore

```sql
BACKUP DATABASE TO '/var/backups/kalamdb-nightly.tar.gz';
RESTORE DATABASE FROM '/var/backups/kalamdb-nightly.tar.gz';
```

- Paths resolve on the **server** filesystem, not the CLI machine.
- `.tar.gz` / `.tgz` → single archive; otherwise directory layout.
- Requires DBA or System role.
- Track status: `SELECT * FROM system.jobs WHERE job_id = 'backupjob';`

```bash
kalam -c "BACKUP DATABASE TO '/var/backups/kalamdb-nightly.tar.gz'"
kalam -c "RESTORE DATABASE FROM '/var/backups/kalamdb-nightly.tar.gz'"
```

## Export user data

```sql
EXPORT USER DATA;
SHOW EXPORT;
```

`SHOW EXPORT` returns `download_url` paths like `/v1/exports/<user_id>/<export_id>`. Prefix with your server base URL when downloading.

```bash
kalam -c "EXPORT USER DATA;"
kalam -c "SHOW EXPORT;" --json
```

## Table export/import (meta-commands)

```bash
kalam -c "\\export app.orders --output ./orders-backup.zip"
kalam -c "\\import app.orders_restored ./orders-backup.zip"
```

## Live queries

Same `SELECT` you would run normally — no separate live-query keyword:

```bash
kalam --subscribe "SELECT * FROM app.messages WHERE conversation_id = 7"
```

## Topic pub/sub SQL

```sql
CREATE TOPIC app.new_messages PARTITIONS 4;
ALTER TOPIC app.new_messages ADD SOURCE app.messages ON INSERT WITH (payload = 'full');
CONSUME FROM app.new_messages GROUP 'worker-1' FROM EARLIEST LIMIT 100;
ACK app.new_messages GROUP 'worker-1' UPTO OFFSET 99;
DROP TOPIC app.new_messages;
```

Use `--consume` / `\consume` for terminal convenience; use SQL when scripting explicit offset control.

## Related

- [Topic pub/sub](https://kalamdb.org/docs/server/topic-pubsub)
- [Cluster SQL reference](https://kalamdb.org/docs/server/sql-reference/cluster)

================================================================================
# Consumer Runtime
URL: https://kalamdb.org/docs/ts-sdk/agents
Source: content/ts-sdk/agents.mdx

> Build resilient row-processing workers with @kalamdb/consumer runConsumer(), retries, failure hooks, typed TopicOp metadata, and LangChain adapters.
# Consumer Runtime

`@kalamdb/consumer` exposes a high-level worker runtime on top of topic consumers:

- `runConsumer()` for row/change-driven handlers with retry/failure hooks
- `runAgent()` as a deprecated compatibility alias for the same lifecycle model
- `createLangChainAdapter()` for plugging in LangChain chat models

These APIs are exported directly from `@kalamdb/consumer`.

The `client` you pass to these helpers should be created with `createConsumerClient()` or otherwise satisfy the worker-side `ConsumerClientLike` contract.

## `runConsumer()` quick start

```ts

const client = createConsumerClient({
  url: 'http://127.0.0.1:2900',
  authProvider: async () => Auth.basic('root', 'kalamdb123'),
  onConnectionError: ({ message, recoverable }) => {
    console.error('client connection issue:', message, { recoverable });
  },
});

await runConsumer<Record<string, unknown>>({
  client,
  name: 'blog-summarizer',
  topic: 'blog.summarizer',
  groupId: 'blog-summarizer-agent',
  start: 'earliest',
  batchSize: 20,
  retry: {
    maxAttempts: 3,
    initialBackoffMs: 250,
    maxBackoffMs: 1500,
    multiplier: 2,
  },
  onChange: async (ctx, change) => {
    const row = change.data;
    const blogId = row.blog_id;
    if (typeof blogId !== 'string' && typeof blogId !== 'number') {
      return;
    }

    await ctx.sql(
      'UPDATE blog.blogs SET updated = NOW() WHERE blog_id = $1',
      [String(blogId)],
    );
  },
  onFailed: async (ctx, change) => {
    await ctx.sql(
      'INSERT INTO blog.summary_failures (run_key, blog_id, error, created, updated) VALUES ($1, $2, $3, NOW(), NOW())',
      [ctx.runKey, String(change.data.blog_id ?? 'unknown'), String(ctx.error ?? 'unknown')],
    );
  },
  ackOnFailed: true,
});
```

## Handler context (`ConsumerRunContext`)

`onChange(ctx, change)` receives:

- runtime metadata: `ctx.name`, `ctx.runKey`, `ctx.attempt`, and `ctx.maxAttempts`
- source data: `change.data` is the decoded row/event payload
- change metadata: `change.user`, typed `change.op`, `change.key`, `change.timestampMs`, `change.partitionId`, `change.offset`, `change.topic`, `change.groupId`, and metadata-only `change.message`
- helpers: `sql()`, `queryOne()`, `queryAll()`, `ack()`
- LLM helper: `llm` (when `llm` adapter + optional `systemPrompt` are configured)

Use `change.data` for the decoded changed row/event and use the other `change.*` fields for metadata about that same event. High-level `ctx` is intentionally reserved for runtime execution state and helpers, so it does not expose `message`, `change`, `user`, `op`, or `offset` duplicates. The high-level `change.message` object intentionally omits `payload`, deprecated `value`, and raw transport `change` fields, so the decoded row lives in exactly one place: `change.data`. `change.user` is populated only when the topic event carries a subject and is expected to be `undefined` for shared-table routes such as the summarizer example. The older `onRow(ctx, row)` alias remains deprecated; use `onChange(ctx, change)` or `onFailed(ctx, change)` for the stable shape.

Generated ORM row types use the same interface. Pass the generated row type as the first `runConsumer<T>()` generic, read the row from `change.data`, and read event metadata from `change.op`, `change.user`, `change.offset`, or `change.message`. No ORM-specific wrapper is needed.

### `runKey`

Default format is:

`<name>:<topic>:<partition_id>:<offset>`

Override with `runKeyFactory` when integrating custom idempotency keys.

## Retry and ACK behavior

`runConsumer()` does not expose an `auto_ack` option. Internally it creates the underlying low-level consumer with `auto_ack: false` so the runtime can control retries and ack timing explicitly:

- `onChange` succeeds: message is acked.
- `onChange` throws: retried based on `retry` policy.
- retries exhausted:
  - if `onFailed` is missing, message is not acked.
  - if `onFailed` succeeds and `ackOnFailed !== false`, message is acked.
  - if `onFailed` throws, message is not acked.

If you need to choose `auto_ack` yourself, use `client.consumer()` directly instead of `runConsumer()`.

Hooks:

- `onRetry({ error, attempt, maxAttempts, backoffMs, runKey, message })`
- `onError({ error, runKey, message })`
- `onConnect()`
- `onConnectionRetry({ error, attempt, maxAttempts, backoffMs })`
- `onConnectionRestored({ attempt })`
- `onConnectionError({ error, message, recoverable, attempt, backoffMs })`

### Retry policy options

The `retry` object supports additional tuning:

- `jitterRatio` — add randomness to backoff (0 disables jitter)
- `shouldRetry(error, attempt)` — classify retryable failures (defaults to retrying)

```ts
retry: {
  maxAttempts: 5,
  initialBackoffMs: 250,
  maxBackoffMs: 5_000,
  multiplier: 2,
  jitterRatio: 0.1,
  shouldRetry: (error) => {
    // Example: don't retry on validation errors
    return !String(error).includes('invalid-input');
  },
}
```

Other useful `runConsumer` options include `partitionId`, `timeoutSeconds`, and `stopSignal` for graceful shutdown.

`runConsumer()` also supervises the underlying consumer loop. If the server is temporarily down or the connection drops, it retries with exponential backoff and jitter until `stopSignal` aborts. Configure this with `connectionRetry`:

```ts
connectionRetry: {
  initialBackoffMs: 500,
  maxBackoffMs: 30_000,
  multiplier: 1.8,
  jitterRatio: 0.2,
}
```

Use the connection hooks when you want operators to see the worker lifecycle clearly in logs:

```ts
await runConsumer({
  client,
  name: 'blog-summarizer',
  topic: 'blog.summarizer',
  groupId: 'blog-summarizer-agent',
  onConnect: () => {
    console.log('worker connected and first healthy poll completed');
  },
  onConnectionRetry: ({ error, attempt, maxAttempts, backoffMs }) => {
    console.warn('worker cannot reach KalamDB:', String(error));
    console.warn(`retrying in ${backoffMs}ms (attempt ${maxAttempts ? `${attempt}/${maxAttempts}` : attempt})`);
  },
  onConnectionRestored: ({ attempt }) => {
    console.log(`worker reconnected after ${attempt} retry attempt${attempt === 1 ? '' : 's'}`);
  },
  onConnectionError: ({ error, message, recoverable, attempt }) => {
    console.error(`worker stopped reconnecting after ${attempt} attempts:`, error);
    console.error(`final status: ${recoverable ? 'retryable' : 'fatal'} - ${message}`);
  },
  onChange: async () => {},
});
```

`onConnect()` fires once when `runConsumer()` reaches its first healthy poll, and then again only after a later outage has been recovered through the same runtime. `onConnectionRestored()` remains the explicit recovery-only hook.

`onConnectionError()` now mirrors the same connection classification used by `createConsumerClient()`: fatal configuration/auth/bootstrap failures are surfaced with `recoverable: false`, while transient reachability failures stay retryable.

`onConnectionRestored()` fires once after a retriable connection failure when the runtime successfully reaches KalamDB again. With the default `createConsumerClient()` implementation, this happens after the first successful poll, even if that poll returns no messages.

## Row parsing

By default, the runtime parses KalamDB topic rows like this:

- it reads `message.payload`
- if `message.payload` has a nested `row` object, it uses `payload.row` for backward compatibility
- otherwise it uses `message.payload` directly

That means apps using the ORM generator can pass the generated row type directly and skip parser boilerplate:

```ts

await runConsumer<ChatDemoMessages>({
  client,
  name: 'chat-demo-agent',
  topic: 'chat_demo.ai_inbox',
  groupId: 'chat-demo-agent',
  onChange: async (_ctx, change) => {
    const row = change.data;
    if (row.role !== 'user') return;
    console.log(row.content);
  },
});
```

You can override with `changeParser(message)`:

```ts
changeParser: (message) => {
  const payload = message.payload as Record<string, unknown> | null;
  if (!payload) return null;
  if (typeof payload.row === 'object' && payload.row) return payload.row as Record<string, unknown>;
  return payload;
}
```

Return `null` only when you intentionally want to skip agent handling for a message.

## Message-Level Handling

Use `onMessage` only for older single-argument handlers that still want the retry/failure lifecycle:

```ts

type OrderEventPayload = {
  order_id: string;
  status: string;
  amount: number;
  _table: string;
};

await runConsumer<OrderEventPayload>({
  client,
  name: 'orders-worker',
  topic: 'orders.events',
  groupId: 'orders-group',
  onMessage: async (_ctx, change) => {
    console.log(change.offset, change.op, change.data.status);
  },
});
```

`onMessage` remains as a deprecated compatibility hook. New workers should use `onChange(ctx, change)` so change data and per-change metadata stay together.

`runConsumer()` acks after successful handler completion. Call `ctx.ack()` only if you explicitly want to ack before returning.

## LangChain integration

`llm` and `systemPrompt` are still supported optional `RunConsumerOptions` fields. If you do not need model-assisted workers, leave them undefined and `ctx.llm` will be `null`.

`createLangChainAdapter()` accepts a duck-typed chat model with `invoke()` and optional `stream()`:

```ts

const llm = createLangChainAdapter(
  new ChatOpenAI({ apiKey: process.env.OPENAI_API_KEY, model: 'gpt-4o-mini' }),
);

await runConsumer({
  client,
  name: 'summary',
  topic: 'blog.summarizer',
  groupId: 'summary-group',
  llm,
  systemPrompt: 'Write one concise sentence.',
  onChange: async (ctx, _change) => {
    const summary = await ctx.llm?.complete('Summarize this row');
    console.log(summary);
  },
});
```

`ctx.llm.complete()` accepts either a plain string prompt or structured message input.

## Production notes

- Prefer idempotent writes keyed by `runKey`.
- Keep `retry.maxAttempts` bounded and tune backoff for your latency profile.
- Use `stopSignal` for graceful shutdown in containerized workers.
- Leave the default connection retry enabled for long-running workers so transient restarts do not exit the process.
- Persist terminal failures (`onFailed`) to a table for replay/inspection.

## Related docs

- [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers)
- [TypeScript Summarizer Example](https://kalamdb.org/docs/use-cases/summarizer-agent)

================================================================================
# Transport & Endpoints
URL: https://kalamdb.org/docs/ts-sdk/api
Source: content/ts-sdk/api.mdx

> Learn how @kalamdb/client, @kalamdb/orm, and @kalamdb/consumer map to KalamDB HTTP and WebSocket endpoints. Covers transport routing, connection lifecycle, and package boundaries.
# Transport & Endpoints

`@kalamdb/client`, `@kalamdb/orm`, and `@kalamdb/consumer` share auth and SQL concepts, but they do not own the same transport surface.

- `@kalamdb/client` owns HTTP SQL, auth endpoints, and the shared WebSocket used for realtime queries.
- `@kalamdb/orm` layers Drizzle schemas, table helpers, generated schema files, and table-level live helpers on top of `@kalamdb/client`.
- `@kalamdb/consumer` layers topic polling and ACK APIs on top of the base client, using its own worker-focused WASM bundle.

Understanding that split helps when debugging networking, tracing, and deployment issues.

## Base URL

```text
http://localhost:2900
```

## HTTP endpoints used by `@kalamdb/client`

| Method | Path | Used by |
|--------|------|---------|
| `POST` | `/v1/api/sql` | `query`, `queryWithFiles`, convenience DML helpers |
| `POST` | `/v1/api/auth/login` | `login()` |
| `POST` | `/v1/api/auth/refresh` | `refreshToken()` |
| `GET` | `/v1/files/{namespace}/{table}/{sub}/{stored_name}` | `FileRef.getDownloadUrl()`, `KalamCellValue.asFileUrl()`, `row.file(...).downloadUrl()` |

FILE download helpers return owner-scoped URLs. For user tables, the server
allows the row owner to fetch the file directly; only `dba` and `system` roles
can append `user_id` for an authorized cross-user byte download.

## HTTP endpoints used by `@kalamdb/consumer`

| Method | Path | Used by |
|--------|------|---------|
| `POST` | `/v1/api/topics/consume` | `consumeBatch()`, `consumer().run()`, `runConsumer()` |
| `POST` | `/v1/api/topics/ack` | `ack()`, `consumer().run()`, `runConsumer()` |

## WebSocket endpoint used by `@kalamdb/client`

| Method | Path | Used by |
|--------|------|---------|
| `GET` | `/v1/ws` | internal shared-socket connection for `live`, `liveTable`, and `liveEvents` |

`@kalamdb/consumer` does not add a second application-level WebSocket API. Its worker surface is topic-oriented and uses the topic HTTP endpoints.

## `@kalamdb/client` method-to-transport mapping

| SDK method | Transport |
|------------|-----------|
| `query()` | WASM-backed HTTP call to `POST /v1/api/sql` |
| `queryWithFiles()` | direct multipart HTTP `POST /v1/api/sql` |
| `queryOne()` / `queryAll()` | wrapper over `query()` plus local `KalamCellValue` wrapping |
| `queryRows()` | wrapper over `query()` plus local `KalamRow` / bound FILE helpers |
| `insert()` / `update()` / `delete()` | SQL helper methods backed by `POST /v1/api/sql` |
| `executeAsUser()` | SQL wrapper over `query()` |
| `login()` | direct HTTP `POST /v1/api/auth/login` and in-memory auth upgrade to JWT |
| `refreshToken()` | HTTP `POST /v1/api/auth/refresh` |
| `liveEvents()` | shared WebSocket low-level live event stream |
| `live()` / `liveTable()` | shared WebSocket live-row materialization |
| `connect()` | explicitly opens or restores the shared WebSocket |
| `disconnect()` | closes the shared WebSocket |
| `shutdown()` | disconnects, frees the WASM client, and resets SDK state |

## `@kalamdb/orm` method-to-transport mapping

| SDK method | Transport |
|------------|-----------|
| `kalamDriver(client)` query execution | delegated to `@kalamdb/client` `query()` for normal SQL |
| `kalamDriver(client)` FILE uploads | delegated to `@kalamdb/client` `queryWithFiles()` when `.values()` / `.set()` include `kalamFile()`, `File`, or `Blob` on a `file()` column |
| `queryWithFiles(client, builder, files?)` | compiles Drizzle SQL, merges upload params, then multipart `POST /v1/api/sql` |
| `compileQuery(builder)` | local SQL normalization only (no network) |
| `executeAsUser(client, builder, user)` | compiled SQL wrapper over `@kalamdb/client` `executeAsUser()` |
| `generateSchema(client)` | delegated to `query('SHOW TABLES')` and best-effort `DESCRIBE <table>` calls |
| `liveTable()` | delegated to the provided `@kalamdb/client` live-row API |
| `kalamFile()` / `file()` / `bytes()` / `embedding()` | local Drizzle helpers; `kalamFile()` triggers multipart routing in the driver |

## `@kalamdb/consumer` method-to-transport mapping

| SDK method | Transport |
|------------|-----------|
| `query()` / `queryOne()` / `queryAll()` | delegated to the internal `@kalamdb/client` SQL client |
| `executeAsUser()` | delegated to the internal `@kalamdb/client` SQL client |
| `login()` / `refreshToken()` | delegated to the internal `@kalamdb/client` auth flow |
| `consumeBatch()` | HTTP `POST /v1/api/topics/consume` |
| `ack()` | HTTP `POST /v1/api/topics/ack` |
| `consumer().run()` | repeated `consumeBatch()` calls plus optional `ack()` |
| `runConsumer()` | built on `consumer().run()` with retry and failure hooks |
| `disconnect()` | clears topic auth cache and disconnects the internal base client |

## Connection and auth flow

### `@kalamdb/client`

1. Create the client with `Auth.basic(...)`, `Auth.jwt(...)`, or `Auth.none()` through `authProvider`.
2. The client initializes lazily on first use and resolves `authProvider`.
3. Optional `login()` exchanges basic credentials for JWT early.
4. The first `live()`, `liveTable()`, or `liveEvents()` call opens the shared WebSocket automatically.
5. `query()` and `queryWithFiles()` continue to run over HTTP.

Connection lifecycle notes:

- `onConnect()` fires when the shared realtime socket becomes healthy.
- `onConnectionError()` receives both WebSocket failures and pre-connect bootstrap failures such as invalid URLs, auth-provider errors, login failures, and refused connections.
- `onError()` remains supported as an alias for older code.

### `@kalamdb/consumer`

1. Create the worker client with the same `authProvider` contract.
2. Topic requests resolve auth on demand and build the correct `Authorization` header.
3. `consumeBatch()` and `ack()` call the topic HTTP endpoints directly.
4. `runConsumer()` keeps ack timing in the worker runtime, not in the topic transport.

Consumer lifecycle notes:

- `createConsumerClient()` exposes `onConnect()` and `onConnectionError()` with worker-oriented connection metadata.
- `runConsumer()` exposes the same high-level connection lifecycle and delegates to the consumer client underneath, so the worker runtime and client do not drift into different error models.

The high-level TypeScript app client exposes `connect()` for eager connection and manual reconnect flows. Most applications can still rely on lazy WebSocket connection from the first live call.

## References

- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)
- [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol)
- [Realtime Subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm)
- [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers)

================================================================================
# Authentication
URL: https://kalamdb.org/docs/ts-sdk/auth
Source: content/ts-sdk/auth.mdx

> Type-safe authentication modes in @kalamdb/client. Learn about login and refresh flows, JWT handling, auth headers, and external identity provider tokens.
# Authentication

`@kalamdb/client` models auth as a tagged union with type-safe constructors.

The same `Auth` and `authProvider` model is reused by `createConsumerClient()` in `@kalamdb/consumer`.

## Auth modes

```ts

Auth.basic('user', 'password');     // { type: 'basic', user, password }
Auth.jwt('<token>');                // { type: 'jwt', token }
```

---

## `authProvider` — the only auth option

All clients must provide an `authProvider` callback. The client resolves it
during initialization, and the WASM transport uses it again for WebSocket
connect/reconnect auth resolution, so token refresh stays centralized in one
place.

```ts

const authProvider: AuthProvider = async () => {
  const token = await myApp.getOrRefreshJwt(); // your token store
  return Auth.jwt(token);
};

const client = createClient({
  url: 'http://localhost:2900',
  authProvider,
});
```

### With user/password bootstrap

Return `Auth.basic(user, pass)` when your app starts from user/password
credentials. The SDK only uses those credentials on `POST /v1/api/auth/login`
and exchanges them for a JWT before the first authenticated query or WebSocket
connection:

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('alice', 'Secret123!'),
});
```

### With OAuth / SSO

Fetch and return a fresh JWT from your token endpoint:

```ts
const authProvider: AuthProvider = async () => {
  // Fetch a fresh JWT from your token endpoint
  const res = await fetch('/api/auth/token');
  const { access_token } = await res.json();
  return Auth.jwt(access_token);
};
```

---

## User/password → JWT login upgrade

`login()` exchanges user/password credentials for a JWT. When the client was created
with `authProvider` returning `Auth.basic(...)`, the SDK calls this
automatically before the first query or WebSocket connection. You can also call
it manually:

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('alice', 'Secret123!'),
});

const login = await client.login();
console.log(login.access_token, login.refresh_token);
// client auth is now JWT internally
```

## Refresh token

```ts
const refreshed = await client.refreshToken('<refresh_token>');
console.log(refreshed.access_token);
```

`refreshToken()` updates the internal auth to the new JWT token.

---

## Auth helpers

```ts

  buildAuthHeader,
  encodeBasicAuth,
  isBasicAuth,
  isJwtAuth,
  isAuthenticated,
} from '@kalamdb/client';
```

`buildAuthHeader()` returns:

- basic → throws, because user/password credentials are only valid for `POST /v1/api/auth/login`
- jwt → `Bearer <token>`

For `basic`, `buildAuthHeader()` throws because user/password credentials are
only valid for `POST /v1/api/auth/login`.

Used internally by `queryWithFiles()` (including FILE uploads routed from `kalamDriver()`) after the login exchange has produced JWT auth.

================================================================================
# Cell Values (KalamCellValue)
URL: https://kalamdb.org/docs/ts-sdk/cell-values
Source: content/ts-sdk/cell-values.mdx

> Learn how to use KalamCellValue for type-safe access to query and subscription row values in the KalamDB TypeScript SDK, including file URLs and null handling.
# Cell Values

`KalamCellValue` is the SDK's typed wrapper around every individual cell value
returned in a query result or subscription event.

Instead of receiving plain `Record<string, unknown>` rows (where you must guess
the type and cast manually), you get `RowData = Record<string, KalamCellValue>`
rows with named accessor methods that mirror the Rust implementation.

Topic consumer payloads stay raw JSON until you explicitly wrap them with `wrapRowMap()`.

## Quick Example

```ts

const client = createClient({ url: 'http://localhost:2900', authProvider: async () => Auth.basic('admin', 'kalamdb123') });

// queryAll() returns RowData[] — every cell is a KalamCellValue
const rows = await client.queryAll('SELECT id, name, score, avatar FROM users');

for (const row of rows) {
  const id     = row.id.asString();       // string | null
  const name   = row.name.asString();     // string | null
  const score  = row.score.asFloat();     // number | null
  const born   = row.born_at.asDate();    // Date | null
  const uuid   = row.public_id.asUuid();   // string | null
  const amount = row.total.asDecimal();    // exact string | null
  const avatar = row.avatar.asFile();     // FileRef | null
  const url    = row.avatar.asFileUrl('http://localhost:2900', 'default', 'users');
  console.log(id, name, score, born?.toISOString(), url);
}

// Single row:
const row = await client.queryOne('SELECT * FROM users WHERE id = $1', [42]);
if (row) console.log(row.name.asString());
```

## Creating a `KalamCellValue`

### `KalamCellValue.from(raw)`

Wrap any raw JS value as a `KalamCellValue`. Pass `null` / `undefined` for SQL NULL.

```typescript
const cell = KalamCellValue.from(someRawValue);
```

### `client.queryAll(sql, params?)`

Execute a SQL query and return all rows with every cell pre-wrapped as
`KalamCellValue`. This is the recommended method for typed cell access.

```typescript
const rows = await client.queryAll('SELECT * FROM orders');
for (const row of rows) {
  console.log(row.status.asString(), row.total.asFloat());
}
```

### `client.queryOne(sql, params?)`

Single-row variant of `queryAll()` — returns the first row, or `null`.

```typescript
const row = await client.queryOne('SELECT * FROM users WHERE id = $1', [42]);
console.log(row?.name.asString());
```

### `wrapRowMap(raw)`

Convert a raw object row into `RowData` when you are handling row-shaped JSON outside `queryOne()` / `queryAll()`:

```ts

const typedRow = wrapRowMap({ id: '1', name: 'Alice' });
```

## `RowData` Type

```typescript
type RowData = Record<string, KalamCellValue>;
```

Use `RowData` as the type annotation for typed result rows throughout your
application.

## Type Guards

| Method | Returns |
|---|---|
| `cell.isNull()` | `boolean` — true if this cell is SQL NULL |
| `cell.isString()` | `boolean` — true if underlying value is a JSON string |
| `cell.isNumber()` | `boolean` — true if underlying JSON is a number |
| `cell.isBool()` | `boolean` — true if underlying JSON is a boolean |
| `cell.isObject()` | `boolean` — true if underlying JSON is an object |
| `cell.isArray()` | `boolean` — true if underlying JSON is an array |

## Typed Accessors

### `.asString(): string | null`

Returns the value as a string, or `null` for SQL NULL.  
Handles `{ "Utf8": "..." }` Rust-side string envelopes automatically.

```typescript
const name = row.name.asString();           // "Alice"
const nullCell = row.deleted_at.asString(); // null
```

### `.asInt(): number | null`

Returns the value as an integer (`Math.trunc`-ed), or `null`.  
Parses string-encoded integers (`"42"` → `42`).

```typescript
const count = row.item_count.asInt(); // 7
```

### `.asBigInt(): bigint | null`

Returns the value as a native `bigint`. Use this for `Int64` / `UInt64` columns
that may exceed `Number.MAX_SAFE_INTEGER`.

```typescript
const id = row.snowflake_id.asBigInt();
```

### `.asSeqId(): SeqId | null`

Returns the value as a `SeqId` wrapper when the cell contains a KalamDB sequence ID such as `_seq`.

```ts
const seq = row._seq.asSeqId();
console.log(seq?.timestampMillis());
```

### `.asFloat(): number | null`

Returns the value as a floating-point number, or `null`.  
Parses string-encoded floats (`"3.14"` → `3.14`).

```typescript
const price = row.price.asFloat(); // 19.99
```

### `.asBool(): boolean | null`

Returns the value as a boolean, or `null`.  
Handles string-encoded booleans: `"true"`, `"false"`, `"1"`, `"0"`.

```typescript
const active = row.is_active.asBool(); // true
```

### `.asDate(): Date | null`

Returns the value as a `Date`, or `null` for SQL NULL / unparseable values.

Handles:
- Unix milliseconds (number): `1704067200000` → `Date`
- ISO 8601 strings: `"2024-01-01T00:00:00Z"` → `Date`
- Numeric timestamp strings: `"1704067200000"` → `Date`

```typescript
const created = row.created_at.asDate(); // Date object
console.log(created?.toLocaleDateString());
```

### `.asDateOnly(): Date | null`

Returns a UTC midnight `Date` for KalamDB `DATE` columns. Handles Date32 day
offsets from the wire and ISO date strings.

```ts
const invoiceDate = row.invoice_date.asDateOnly();
console.log(invoiceDate?.toISOString().slice(0, 10));
```

### `.asTimeString(): string | null`

Returns a KalamDB `TIME` value as `HH:mm:ss` or `HH:mm:ss.ffffff`. Handles the
server wire value of microseconds since midnight.

```ts
const startsAt = row.starts_at.asTimeString();
```

### `.asUuid(): string | null`

Validates and lowercases UUID values.

```ts
const id = row.public_id.asUuid();
```

### `.asDecimal(): string | null`

Returns `DECIMAL(p,s)` values as exact strings. This avoids precision loss from
coercing fixed-point values into JavaScript numbers.

```ts
const total = row.total.asDecimal();
```

### `.asBytes(): Uint8Array | null`

Returns `BYTES` values as `Uint8Array`. The accessor accepts the server's byte
array wire format and base64 strings when interoperating with older tooling.

```ts
const payload = row.payload.asBytes();
```

### `.asEmbedding(): number[] | null`

Returns `EMBEDDING(n)` vectors as a number array.

```ts
const vector = row.doc_embedding.asEmbedding();
```

### `.asObject(): Record<string, JsonValue> | null`

Returns the raw value as a plain object, or `null`.

### `.asArray(): JsonValue[] | null`

Returns the raw value as an array, or `null`.

## FILE Column Support

### `.asFile(): FileRef | null`

Parse a FILE column value and return a `FileRef` instance (or `null`).

```typescript
const ref = row.attachment.asFile();
if (ref) {
  console.log(ref.name, ref.mime, ref.formatSize());
  console.log(ref.isImage()); // true for image/* MIME types
}
```

See [FILE Columns & Uploads](https://kalamdb.org/docs/ts-sdk/file-columns) for the full `FileRef` API.

### `.asFileUrl(baseUrl, namespace, table): string | null`

Convenience method — parse a FILE column **and** return the download URL in one
call.

```typescript
const url = row.avatar.asFileUrl('http://localhost:2900', 'default', 'users');
// e.g. "http://localhost:2900/v1/files/default/users/f0001/12345-photo.png"
img.src = url ?? '/placeholder.png';
```

The generated URL uses KalamDB's stored filename path, so it is safe to fetch
directly with the caller's bearer token.

## Serialisation

### `.asJson(): JsonValue`

Returns the underlying raw JSON value. This is the preferred JSON accessor.

### `.toJson(): JsonValue`

Compatibility alias of `.asJson()`.

Returns the underlying raw JSON value. Use this when you need to pass the value
to code that expects plain JSON (e.g. `FileRef.from(cell.asJson())`).

### `.toString(): string`

Human-readable display:
- SQL NULL → `"NULL"`
- strings → value as-is
- objects / arrays → compact JSON
- other → `String(value)`

## Using with Live Rows

`live()` callbacks receive rows as `RowData`, where each column is already a
`KalamCellValue`. Use the same typed accessors as query results:

```ts

const stop = await client.live<RowData>('SELECT * FROM default.users', (rows) => {
  for (const row of rows) {
    console.log(row.name.asString(), row.score.asFloat());

    // FILE columns still work via KalamCellValue helpers.
    const avatar = row.avatar?.asFile();
    if (avatar) console.log(avatar.downloadUrl());
  }
});
```

### `KalamRow.cell(column): KalamCellValue`

Access a single column as a `KalamCellValue`. `KalamRow` is used by `queryRows()`;
live rows expose the same cell values directly on `RowData`.

### `KalamRow.typedData: RowData`

All column values wrapped as `KalamCellValue` in a `Record<string, KalamCellValue>`.
Cached on first access for performance.

## Using with Consumers

```ts

const worker = createConsumerClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('worker', 'Secret123!'),
});

await worker.consumer({ topic: 'events', group_id: 'events-reader', auto_ack: true }).run(async (ctx) => {
  const row = wrapRowMap(ctx.message.value as Record<string, unknown>);
  console.log(row.event_type.asString(), row.created_at.asDate());
});
```

## Full API Reference

| Member | Signature | Description |
|---|---|---|
| `KalamCellValue.from` | `(raw: unknown) => KalamCellValue` | Wrap a raw value |
| `client.queryAll` | `(sql, params?) => Promise<RowData[]>` | Query with typed cells |
| `client.queryOne` | `(sql, params?) => Promise<RowData \| null>` | Single row query |
| `KalamRow.cell` | `(column) => KalamCellValue` | Access column as KalamCellValue |
| `KalamRow.typedData` | `RowData` (getter) | All columns as KalamCellValue map |
| `asJson` | `() => JsonValue` | Raw underlying value (preferred) |
| `toJson` | `() => JsonValue` | Raw underlying value |
| `isNull` | `() => boolean` | True if SQL NULL |
| `isString` | `() => boolean` | True if JSON string |
| `isNumber` | `() => boolean` | True if JSON number |
| `isBool` | `() => boolean` | True if JSON boolean |
| `isObject` | `() => boolean` | True if JSON object |
| `isArray` | `() => boolean` | True if JSON array |
| `asString` | `() => string \| null` | Value as string |
| `asInt` | `() => number \| null` | Value as integer |
| `asBigInt` | `() => bigint \| null` | Value as bigint |
| `asSeqId` | `() => SeqId \| null` | Value as typed sequence ID |
| `asFloat` | `() => number \| null` | Value as float |
| `asBool` | `() => boolean \| null` | Value as boolean |
| `asDate` | `() => Date \| null` | Value as Date |
| `asDateOnly` | `() => Date \| null` | DATE value as UTC midnight Date |
| `asTimeString` | `() => string \| null` | TIME value as `HH:mm:ss[.fraction]` |
| `asUuid` | `() => string \| null` | Validated UUID string |
| `asDecimal` | `() => string \| null` | Exact DECIMAL string |
| `asBytes` | `() => Uint8Array \| null` | BYTES value as byte array |
| `asEmbedding` | `() => number[] \| null` | EMBEDDING vector |
| `asObject` | `() => Record<string, JsonValue> \| null` | Value as object |
| `asArray` | `() => JsonValue[] \| null` | Value as array |
| `asFile` | `() => FileRef \| null` | FILE column as FileRef |
| `asFileUrl` | `(baseUrl, namespace, table) => string \| null` | FILE column URL |
| `toString` | `() => string` | Display string |

================================================================================
# Chat Quickstart (Docs-Only)
URL: https://kalamdb.org/docs/ts-sdk/chat-quickstart
Source: content/ts-sdk/chat-quickstart.mdx

> Build a minimal TypeScript chat and worker pipeline with @kalamdb/client for the app client and @kalamdb/consumer for the worker runtime.
# Chat Quickstart (Docs-Only)

This page mirrors the smaller `examples/chat-with-ai` flow using only documented `@kalamdb/client` and `@kalamdb/consumer` APIs:

- `createClient`, `Auth.basic`
- `query`, `live`
- `createConsumerClient()`, `runConsumer()`

This path is for local development without external auth setup.

## Prerequisites

- Node.js `>= 18`
- running KalamDB server
- a user that can run SQL and consume topic messages

Install:

```bash
npm i @kalamdb/client @kalamdb/consumer
```

## 1) Create a client

```ts

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const worker = createConsumerClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});
```

## 2) Apply chat schema + topic

```ts
await client.query('CREATE NAMESPACE IF NOT EXISTS chat_demo');

await client.query(`
CREATE SHARED TABLE chat_demo.messages (
  id         BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  room       TEXT NOT NULL DEFAULT 'main',
  role       TEXT NOT NULL,
  author     TEXT NOT NULL,
  content    TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT NOW()
)
`);

await client.query('CREATE TOPIC chat_demo.ai_inbox');
await client.query(`
ALTER TOPIC chat_demo.ai_inbox
ADD SOURCE chat_demo.messages
ON INSERT
WITH (payload = 'full')
`);
```

## 3) Start the worker with `runConsumer()`

```ts
await runConsumer<Record<string, unknown>>({
  client: worker,
  name: 'chat-demo-agent',
  topic: 'chat_demo.ai_inbox',
  groupId: 'chat-demo-agent',
  onChange: async (ctx, change) => {
    const row = change.data;
    if (String(row.role ?? '') !== 'user') {
      return;
    }

    const room = String(row.room ?? 'main');
    const content = String(row.content ?? '').trim();
    if (!content) {
      return;
    }

    const reply = `AI reply: KalamDB stored "${content}" and this worker wrote the response.`;

    await ctx.sql(
      'INSERT INTO chat_demo.messages (room, role, author, content) VALUES ($1, $2, $3, $4)',
      [room, 'assistant', 'KalamDB Copilot', reply],
    );
  },
});
```

## 4) Subscribe in the chat client

Use `live()` so the UI always receives the latest materialized row set.

```ts
const room = 'main';

const unsubscribe = await client.live(
  `SELECT * FROM chat_demo.messages
   WHERE room = '${room}'`,
  (rows) => {
    // This callback receives the current chat timeline after KalamDB has
    // already reconciled inserts and updates for you.
    renderRows(rows);
  },
  {
    limit: 200,
    subscriptionOptions: { last_rows: 200 },
  },
);
```

If you switch this table to `WITH (TYPE = 'USER')`, the same query text still works for every user and KalamDB will only return that authenticated user's rows.

Insert a user message:

```ts
await client.query(
  'INSERT INTO chat_demo.messages (room, role, author, content) VALUES ($1, $2, $3, $4)',
  [room, 'user', 'demo-user', 'hello'],
);
```

## 5) Cleanup

```ts
await unsubscribe();
await client.disconnect();
```

## Next

- [Examples](https://kalamdb.org/docs/ts-sdk/examples)
- [Realtime Subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers)
- [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai)

================================================================================
# Client Lifecycle
URL: https://kalamdb.org/docs/ts-sdk/client-lifecycle
Source: content/ts-sdk/client-lifecycle.mdx

> KalamDBClient lifecycle management. Learn about initialization, WebSocket connection behavior, automatic reconnection controls, and resource cleanup.
# Client Lifecycle

This page documents the lifecycle methods implemented in `src/client.ts`.

## Construction

```ts

const authProvider: AuthProvider = async () => {
  const token = await myApp.getOrRefreshJwt();
  return Auth.jwt(token);
};

const client = createClient({
  url: 'http://localhost:2900',
  authProvider,
  onConnect: () => {
    console.log('realtime socket ready');
  },
  onConnectionError: (error) => {
    console.error('cannot reach KalamDB:', error.message);
  },
});
```

Validation at construction time:

- `url` is required
- `authProvider` is required

See [Authentication](https://kalamdb.org/docs/ts-sdk/auth) for the full auth guide.

## WebSocket connection behavior

The SDK manages the WebSocket connection lifecycle automatically. `connect()` is
public, but most applications only need it for eager connection or manual
reconnect flows.

- `query()` runs over HTTP and does not require a WebSocket connection.
- `live()`, `liveTable()`, and `liveEvents()` automatically establish the WebSocket
  connection on first use.

## Connect

```ts
await client.connect();
```

`connect()` explicitly establishes or restores the shared WebSocket connection.
It is safe to call more than once and deduplicates concurrent connection work.
Existing subscription state is preserved in the underlying client so reconnecting
the same instance can resume live queries.

## Lifecycle handlers

`createClient()` and the `KalamDBClient` instance support three connection-lifecycle hooks:

- `onConnect(callback)` fires when the shared realtime connection becomes healthy.
- `onConnectionError(callback)` fires for connection and auth/bootstrap failures.
- `onError(callback)` remains as a compatibility alias for connection-error handling.

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider,
  onConnect: () => {
    console.log('connected to KalamDB');
  },
  onConnectionError: (error) => {
    console.error(
      `[${error.recoverable ? 'retryable' : 'fatal'}] ${error.message}`,
    );
  },
});

client.onConnect(() => {
  console.log('socket restored or connected');
});

client.onConnectionError((error) => {
  console.error('connection/auth problem:', error.message);
});
```

Use `onConnectionError` for new code. `onError` still works, but it exists mainly so older code does not break.

If you do not register either handler, the SDK now logs connection/auth failures to the console by default instead of failing silently.

### What reaches `onConnectionError`

The callback is used for both WebSocket connection failures and pre-connect failures that happen before the socket opens:

- invalid or malformed server URLs
- auth provider resolution failures
- login or token refresh failures used during connection/bootstrap
- unreachable server / refused connection cases

The event shape comes from the shared WASM client and includes:

- `message`: normalized error message
- `recoverable`: `true` when the failure can reasonably be retried, `false` for fatal configuration/auth issues

### `wsLazyConnect` (default: `true`)

By default, the WebSocket connection is **lazy**: it is deferred until the
first `live()`, `liveTable()`, or `liveEvents()` call. This avoids unnecessary
connections when the client is only used for HTTP queries.

Set `wsLazyConnect: false` to establish the connection eagerly during
initialization:

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.jwt('<JWT_TOKEN>'),
  wsLazyConnect: false, // connect immediately
});
```

## `pingIntervalMs`

Use `pingIntervalMs` to tune the WebSocket ping interval (default `5000`).

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider,
  pingIntervalMs: 15_000,
});
```

## `disableCompression`

By default the server sends gzip-compressed binary WebSocket frames. During
development you can disable compression to inspect raw JSON frames in browser
DevTools or `wscat`:

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider,
  disableCompression: true, // appends ?compress=false to the WS URL
});
```

When enabled the client appends `?compress=false` to the WebSocket URL and
the server responds with plain-text JSON frames for every message.

> **Do not use in production.** Compression significantly reduces bandwidth
> for subscription payloads. Only enable for local debugging.

## Initialization behavior

`initialize()`:

- loads WASM runtime (`init(...)`)
- resolves initial credentials from `authProvider`
- creates underlying WASM `KalamClient`
- wires `authProvider` callback into WASM for WebSocket auth resolution and reconnects
- calls `setDisableCompression(true)` on WASM client when option is set
- if `wsLazyConnect: false`, establishes the WebSocket connection immediately

If eager initialization hits a connection/auth/bootstrap failure, the SDK emits `onConnectionError` before rethrowing.

You usually don’t call it manually; most methods initialize lazily.

## Disconnect

```ts
await client.disconnect();
```

Notes:

- `disconnect()` closes the WebSocket and clears SDK-side subscription tracking.
- After disconnecting, the next `subscribe()` call will re-establish the connection.

## Full teardown with `shutdown()`

```ts
await client.shutdown();
```

Use `shutdown()` when you want to free the underlying WASM client and fully
reset local SDK state. This is the stronger cleanup option for long-running
Node.js processes, test suites, and explicit resource-management flows.

## Reconnection controls

```ts
client.setAutoReconnect(true);
client.setReconnectDelay(1000, 30000);
client.setMaxReconnectAttempts(10); // 0 means infinite

console.log(client.getReconnectAttempts());
console.log(client.isReconnecting());
```

`setReconnectDelay(initial, max)` is passed as `BigInt` to WASM.

## Subscription state helpers

```ts
client.getSubscriptionCount();
client.getSubscriptions();
client.isSubscribedTo('SELECT * FROM app.messages');
client.getLastSeqId('<subscription-id>');
```

These are SDK-level convenience methods around subscription bookkeeping.

## Cleanup pattern

```ts
try {
  // work
} finally {
  await client.unsubscribeAll();
  await client.disconnect();
}
```

Use this pattern in long-running services to avoid leaked subscriptions.

================================================================================
# Topic Consumers & ACK
URL: https://kalamdb.org/docs/ts-sdk/consumers
Source: content/ts-sdk/consumers.mdx

> Use @kalamdb/consumer for reliable topic processing. Learn createConsumerClient(), consumer().run(), consumeBatch(), explicit ack(), and error handling patterns.
# Topic Consumers & ACK

`@kalamdb/consumer` is the worker package. Use it for topic consumption, ACKs, and retrying workers. Keep it out of browser code.

```mermaid
flowchart LR
  A[Table write] --> B[KalamDB topic]
  B --> C["@kalamdb/consumer worker"]
  C --> D[process payload]
  D --> E[ack offset]
  D --> F[write result]
```

The consumer client wraps an internal `@kalamdb/client` instance, so workers still have `query()`, `queryOne()`, `queryAll()`, role-matrix `executeAsUser()`, `login()`, `refreshToken()`, and `disconnect()` available alongside the topic APIs.

For high-level row-processing workers with retries and failure hooks, see:

- [/docs/ts-sdk/agents](https://kalamdb.org/docs/ts-sdk/agents)

`runConsumer()` already deserializes standard KalamDB topic row payloads and unwraps legacy `{ row: ... }` envelopes. Use a custom `changeParser` only for non-row payload modes or custom envelopes.

Create a worker client with the same auth options you would use for `@kalamdb/client`:

```ts

const client = createConsumerClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('worker', 'Secret123!'),
  onConnect: () => {
    console.log('worker client is ready');
  },
  onConnectionError: (event) => {
    console.error(event.message, { recoverable: event.recoverable, attempt: event.attempt });
  },
});
```

The consumer client now exposes the same connection lifecycle hooks as the base app client:

- `onConnect(callback)` for the first healthy connection and later recoveries
- `onConnectionError(callback)` for topic/auth/transport failures
- `onError(callback)` as a compatibility alias

If you do not register an error callback, `createConsumerClient()` logs connection failures to the console by default.

`onConnectionError` receives a richer worker-oriented event than the base client callback:

- `message`
- `recoverable`
- `attempt`
- `backoffMs` when a retry is scheduled
- `context` when the client knows which step failed
- raw `error`

Fatal configuration failures such as invalid URLs are marked `recoverable: false`. Unreachable servers and transient transport failures are reported as recoverable.

## Consumer builder API

Copy this when you want a long-running worker loop.

```ts
const handle = client.consumer({
  topic: 'orders',
  group_id: 'billing',
  auto_ack: true,
  batch_size: 10,
});

await handle.run(async (ctx) => {
  console.log(
    ctx.message.partition_id,
    ctx.message.offset,
    ctx.message.op,
    ctx.message.user,
    ctx.message.payload,
  );
});

handle.stop();
```

Source behavior (`client.ts`):

- loops `consume(options)` until `stop()` requested
- if `auto_ack` and handler didn’t manually `ctx.ack()`, SDK auto-acks
- when no more messages, sleeps ~1s before polling again
- `handle.run(handler, hooks?)` accepts optional lifecycle hooks such as `onBatchSuccess({ nextOffset, hasMore, messageCount })` after each successful poll

The consumer client delegates base connection handling to its internal `@kalamdb/client` instance, so connection/auth failures during login, refresh, or topic auth normalization also flow through `onConnectionError`.

## Manual ACK mode

Use manual ACK when the message should be committed only after your handler finishes.

```ts
const handle = client.consumer({
  topic: 'orders',
  group_id: 'billing-manual',
  auto_ack: false,
});

await handle.run(async (ctx) => {
  await processOrder(ctx.message.payload);
  await ctx.ack();
});
```

`ctx` contains:

- `user` (typed branded user identifier)
- `message` (topic, group, partition, offset, key, op, timestamp, decoded payload)
- `ack(): Promise<void>`

## Topic payload contract in handlers

`ctx.message.payload` is the decoded JSON payload from the topic source. In the current backend, `WITH (payload = 'full')` publishes the changed row object directly and injects `_table` with the source table id.

Most handlers can work with the payload directly:

- direct row object for current `full` payloads
- optional legacy wrapper object with row under `payload.row` if you are still consuming older message shapes

For lower-level `consumer().run()` handlers, use a defensive parser when you need to normalize multiple payload shapes yourself:

```ts
function extractRow(payload: unknown): { conversation_id: number; role: string; content: string } | null {
  if (!payload || typeof payload !== 'object') return null;

  const record = payload as Record<string, unknown>;
  const source = (record.row && typeof record.row === 'object'
    ? record.row
    : record) as Record<string, unknown>;

  const conversationId = Number(source.conversation_id);
  const role = source.role;
  const content = source.content;

  if (!Number.isInteger(conversationId)) return null;
  if (role !== 'user' && role !== 'assistant') return null;
  if (typeof content !== 'string') return null;

  return { conversation_id: conversationId, role, content };
}
```

If your topic source uses another payload mode (`key` / `diff`), treat both as still in development. If you are testing them anyway, align parser logic with the current payload shape and remember that `_table` lives inside the decoded payload, not in a separate `source_table` field.

## One-shot batch consume

```ts
const batch = await client.consumeBatch({
  topic: 'orders',
  group_id: 'billing',
  batch_size: 20,
  start: 'earliest',
});

console.log(batch.messages, batch.next_offset, batch.has_more);
```

If you know the payload shape ahead of time, type it once and keep the whole consume flow strongly typed:

```ts
type OrderEventPayload = {
  order_id: string;
  status: string;
  amount: number;
  _table: string;
};

const batch = await client.consumeBatch<OrderEventPayload>({
  topic: 'orders',
  group_id: 'billing',
  batch_size: 20,
  start: 'earliest',
});

for (const message of batch.messages) {
  console.log(message.payload.status);
}

const handle = client.consumer<OrderEventPayload>({
  topic: 'orders',
  group_id: 'billing',
});

await handle.run(async (ctx) => {
  console.log(ctx.message.payload.status);
});
```

Each `message` includes:

- `topic`, `group_id`, `partition_id`, `offset`
- optional `key`
- optional `user`
- optional `op`
- optional `timestamp_ms`
- decoded `payload`

`message.value` still exists as a deprecated alias for `message.payload`.

## Low-level ack API

```ts
await client.ack('orders', 'billing', 0, 42);
```

This acknowledges up to and including offset `42`.

## ConsumeRequest options

```ts
{
  topic: string;
  group_id: string;
  start?: string;
  partition_id?: number;
  batch_size?: number;
  timeout_seconds?: number;
  auto_ack?: boolean;
  concurrency_per_partition?: number;
}
```

`start` accepts backend-supported start modes (for example `earliest` or `latest`) and can include offset-style values depending on server routing.

Current backend notes:

- the HTTP topic API reads one partition per request
- `partition_id: 0` is the safe default today
- `timeout_seconds` is accepted by the API shape but does not long-poll yet; the SDK keeps polling explicitly

## USER and STREAM table write pattern in workers

When consuming events and writing into `TYPE = 'USER'` or `TYPE = 'STREAM'` tables, use `executeAsUser()` from an authorized service, DBA, or system account:

```ts
await client.executeAsUser(
  'INSERT INTO chat.messages (conversation_id, role, content) VALUES ($1, $2, $3)',
  String(ctx.user ?? ''),
  [conversationId, 'assistant', reply],
);
```

This preserves per-user isolation in service-side writes through KalamDB's explicit delegation boundary.

For the table-type and impersonation rules behind this pattern, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types) and
[/docs/server/sql-reference/impersonation](https://kalamdb.org/docs/server/sql-reference/impersonation).

## When to use which API

- **SQL subscriptions**: row-level table/query reactivity.
- **consumer()** from `@kalamdb/consumer`: explicit queue or partition processing with ack semantics.
- **runConsumer()** from `@kalamdb/consumer`: higher-level runtime with retry, `onConnect`, `onConnectionError`, and `onFailed` lifecycle hooks. It owns ack timing internally, so there is no `auto_ack` option on `RunConsumerOptions`.

================================================================================
# Cookbook
URL: https://kalamdb.org/docs/ts-sdk/cookbook
Source: content/ts-sdk/cookbook.mdx

> Production-style @kalamdb/client, @kalamdb/orm, and @kalamdb/consumer recipes for KalamDB. Includes chat pipelines, vector search, multi-tenant patterns, and real-world integration code.
# SDK Cookbook

This page is a practical recipe collection built from real SDK source usage and the `examples/chat-with-ai` implementation.

## Recipe 1: Chat with AI pipeline (topic + consumer + subscriptions)

This mirrors the `chat-with-ai` architecture:

1. User message inserted into `chat.messages`
2. CDC routes inserts into topic `chat.ai_processing`
3. Background service consumes topic and writes AI reply back to table
4. Frontend receives both user and AI rows instantly via live subscription

### 1) SQL setup

```sql
CREATE NAMESPACE IF NOT EXISTS chat;

CREATE TABLE chat.messages (
  id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
  conversation_id BIGINT NOT NULL,
  sender TEXT NOT NULL,
  role TEXT NOT NULL DEFAULT 'user',
  content TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL DEFAULT NOW()
) WITH (TYPE = 'USER');

CREATE TOPIC chat.ai_processing;
ALTER TOPIC chat.ai_processing
  ADD SOURCE chat.messages ON INSERT WITH (payload = 'full');
```

### 2) Frontend subscription (React)

```ts
const chatSql = `
  SELECT * FROM chat.messages
  WHERE conversation_id = ${conversationId}
`;

const unsub = await client.live(
  chatSql,
  (rows) => {
    // Let the SDK hand the UI the current materialized conversation state.
    setMessages(rows);
  },
  {
    subscriptionOptions: { last_rows: 200 },
  },
);
```

### 3) Node.js background processor (consumer)

```ts

const worker = createConsumerClient({
  url: process.env.KALAMDB_URL ?? 'http://localhost:2900',
  authProvider: async () => Auth.basic(process.env.KALAMDB_USER!, process.env.KALAMDB_PASSWORD!),
});

const handle = worker.consumer({
  topic: 'chat.ai_processing',
  group_id: 'ai-processor-service',
  auto_ack: true,
  batch_size: 1,
});

await handle.run(async (ctx) => {
  const payload = ctx.message.value as Record<string, unknown>;
  const row = (payload.row ?? payload) as {
    conversation_id: string | number;
    role: string;
    content: string;
  };

  if (row.role !== 'user') return;

  const aiReply = await generateReply(row.content);
  const user = String(ctx.user ?? '');
  if (!user) return;
  const conversationId = Number(row.conversation_id);
  if (!Number.isInteger(conversationId)) return;

  await worker.executeAsUser(
    'INSERT INTO chat.messages (conversation_id, sender, role, content) VALUES ($1, $2, $3, $4)',
    user,
    [conversationId, 'AI Assistant', 'assistant', aiReply],
  );
});
```

### 4) Operational notes from source

- `executeAsUser(...)` follows KalamDB's role matrix; service workers can target service/user accounts, while direct USER-table queries remain scoped to the authenticated account.
- `executeAsUser(...)` is valid for USER and STREAM tables; shared tables still rely on their table policy.
- Typing indicator updates are done via a stream table (`chat.typing_indicators`).
- Service loads WASM explicitly in Node contexts where needed.

The server-side table policy details are documented at
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types).

## Recipe 2: Resumable subscriptions with sequence checkpoints

Use `from` plus the typed `lastSeqId` checkpoints returned by `getSubscriptions()`.

```ts

let savedSeqId: SeqId | undefined;
const eventsSql = "SELECT * FROM app.events WHERE source = 'system'";

const unsub = await client.live(
  eventsSql,
  (rows) => {
    // process the fully reconciled event feed
    renderEvents(rows);

    const active = client.getSubscriptions().find((sub) => sub.tableName === eventsSql);
    if (active?.lastSeqId) {
      savedSeqId = active.lastSeqId;
    }
  },
  {
    subscriptionOptions: {
      ...(savedSeqId ? { from: savedSeqId } : {}),
    },
  },
);
```

## Recipe 2.1: Payload parser guard for consumer handlers

When your topic source uses `WITH (payload = 'full')`, parse both wrapper and direct row forms:

```ts
function extractMessageRow(value: unknown) {
  if (!value || typeof value !== 'object') return null;
  const record = value as Record<string, unknown>;
  const source =
    record.row && typeof record.row === 'object'
      ? (record.row as Record<string, unknown>)
      : record;

  const conversationId = Number(source.conversation_id);
  if (!Number.isInteger(conversationId)) return null;
  if (source.role !== 'user' && source.role !== 'assistant') return null;
  if (typeof source.content !== 'string') return null;

  return {
    conversation_id: conversationId,
    role: source.role,
    content: source.content,
  };
}
```

## Recipe 3: Typed query helpers for safer app code

```ts
interface Conversation {
  id: string;
  title: string;
  created_at: string;
}

const conversations = await client.queryAll(
  'SELECT id, title, created_at FROM chat.conversations ORDER BY updated_at DESC'
);

const firstConversation = await client.queryOne(
  'SELECT id, title, created_at FROM chat.conversations LIMIT 1'
);

conversations.forEach((row) => {
  console.log(row.id.asString(), row.title.asString(), row.created_at.asString());
});

console.log(firstConversation?.id.asString());
```

Why this pattern helps:

- keeps SQL in one place
- maps array-row response into typed objects
- reduces unsafe indexing in UI code

## Recipe 4: File attachments in messages

With Drizzle (`@kalamdb/orm`), pass upload bytes in `.values()` — the driver routes to multipart SQL automatically:

```ts

await db.insert(messages).values({
  conversation_id: conversationId,
  sender: 'user',
  role: 'user',
  content: messageText,
  file_data: kalamFile('attachment', selectedFile),
});
```

For raw SQL without Drizzle, use `@kalamdb/client` directly:

```ts
const result = await client.queryWithFiles(
  'INSERT INTO chat.messages (conversation_id, sender, role, content, file_data) VALUES ($1, $2, $3, $4, FILE("attachment"))',
  { attachment: selectedFile },
  [conversationId, 'user', 'user', messageText],
  (progress) => {
    setUploadPercent(progress.percent);
  }
);
```

Then parse FILE metadata:

```ts

const ref = parseFileRef(row.file_data);
if (ref) {
  const downloadUrl = ref.getDownloadUrl('http://localhost:2900', 'chat', 'messages');
  console.log(downloadUrl, ref.formatSize(), ref.getTypeDescription());
}
```

## Recipe 5: Connection bootstrap for apps

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic(username, password),
  wasmUrl: '/wasm/kalam_client_bg.wasm',
});

await client.login();
client.setAutoReconnect(true);
client.setReconnectDelay(1000, 30000);
client.setMaxReconnectAttempts(0);
```

This mirrors the provider pattern in `chat-with-ai`: Basic credentials are exchanged for JWT automatically before the realtime connection. Most apps can let the first live/subscription call connect lazily; call `connect()` only when you want to establish the shared WebSocket eagerly.

## Recipe 6: Graceful shutdown for worker services

```ts
const handle = worker.consumer({ topic: 'orders', group_id: 'order-worker', auto_ack: true });

const runPromise = handle.run(async (ctx) => {
  await processMessage(ctx.message.value);
});

process.on('SIGINT', async () => {
  handle.stop();
  await runPromise.catch(() => undefined);
  await worker.disconnect();
  process.exit(0);
});
```

## Recipe 7: Vector search for semantic retrieval

This matches the new vector-search flow tested in KalamDB core: embeddings live in regular SQL tables, vector indexes are created with DDL, and ranking happens in standard SQL.

### 1) Schema

```sql
CREATE TABLE rag.documents (
  id BIGINT PRIMARY KEY,
  title TEXT NOT NULL,
  body TEXT NOT NULL
) WITH (TYPE = 'USER');

CREATE TABLE rag.documents_vectors (
  id BIGINT PRIMARY KEY,
  doc_embedding EMBEDDING(384)
) WITH (TYPE = 'USER');

ALTER TABLE rag.documents_vectors
  CREATE INDEX doc_embedding USING COSINE;
```

### 2) Insert or update embeddings

```ts
await client.query(
  `INSERT INTO rag.documents_vectors (id, doc_embedding)
   VALUES ($1, $2)`,
  [docId, JSON.stringify(embedding)],
);
```

### 3) Query nearest documents

```ts
const matches = await client.queryAll(
  `SELECT d.id, d.title, d.body
   FROM rag.documents AS d
   JOIN rag.documents_vectors AS v ON v.id = d.id
   ORDER BY COSINE_DISTANCE(v.doc_embedding, $1)
   LIMIT 5`,
  [JSON.stringify(queryEmbedding)],
);
```

Why this pattern helps:

- keeps semantic retrieval inside the same SQL surface as the rest of your app
- works with `TYPE = 'USER'` for tenant-safe agent memory
- lets you combine vectors, files, and structured rows without a separate vector service

## Related docs

- [/docs/ts-sdk/subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [/docs/ts-sdk/orm](https://kalamdb.org/docs/ts-sdk/orm)
- [/docs/ts-sdk/consumers](https://kalamdb.org/docs/ts-sdk/consumers)
- [/docs/ts-sdk/file-columns](https://kalamdb.org/docs/ts-sdk/file-columns)
- [/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search)
- [/docs/ts-sdk/types](https://kalamdb.org/docs/ts-sdk/types)

================================================================================
# Examples
URL: https://kalamdb.org/docs/ts-sdk/examples
Source: content/ts-sdk/examples.mdx

> TypeScript SDK examples for React live-query screens, ORM-backed workers, browser subscriptions, and vector search workflows in KalamDB.
# TypeScript SDK Examples

The KalamDB repo currently ships four runnable TypeScript examples. Start with the React one if you want the current app pattern.

They live in the repository under:

- `examples/react-ai-chat`
- `examples/simple-typescript`
- `examples/chat-with-ai`
- `examples/summarizer-agent`

Each runnable example has its own setup path and its own test.

## 1. React AI Chat

Repository path: `examples/react-ai-chat`

What it demonstrates:

- `@kalamdb/react` with `KalamProvider` and `LiveQueries`
- one React screen reading conversations, messages, and typing tokens together
- file attachments, streamed typing tokens, final assistant inserts, and approval actions
- browser demo mode and the server-backed topic-agent path

Simple multi-table component:

```tsx

const client = getExampleClient();

  const queries = {
    conversations: {
      table: conversations,
      orderBy: (table: typeof conversations) => desc(table.updatedAt),
      limit: 50,
    },
    messages: {
      table: messages,
      where: (table: typeof messages) => eq(table.conversationId, selectedConversationId),
      orderBy: (table: typeof messages) => asc(table.createdAt),
      deps: [selectedConversationId],
    },
    typingTokens: {
      table: typingTokens,
      where: (table: typeof typingTokens) => eq(table.conversationId, selectedConversationId),
      orderBy: (table: typeof typingTokens) => asc(table.createdAt),
      deps: [selectedConversationId],
    },
  };

  return (
    
  );
}
```

Why it matters:

This is the current React app pattern in the repo. One provider owns the client, one `LiveQueries` block owns the live datasets, and the UI renders rows directly instead of syncing copies into local state.

Run it:

```bash
cd examples/react-ai-chat
npm install
npm run setup
npm run agent
npm run dev
```

Test it:

```bash
npm test
```

`npm run setup` imports `chat-app.sql`, clears existing example topics, seeds the default conversation, and writes `.env.local`. For browser-only demo mode, skip setup and set `VITE_KALAMDB_DEMO_MODE=true`.

## 2. Realtime Ops Feed

Repository path: `examples/simple-typescript`

What it demonstrates:

- `live()` for live browser updates
- `query()` for writes from the browser
- browser auth with `Auth.basic(...)`
- cross-tab realtime sync from one write

Why it matters:

This is the smallest browser example that still feels real. It avoids wrapper layers, keeps the full flow inside one React component, and shows the recommended pattern: one live query returns the current rows while USER-table isolation keeps each signed-in user inside their own data partition.

Run it:

```bash
cd examples/simple-typescript
npm install
npm run setup
npm run dev
```

Test it:

```bash
npm test
```

The Playwright test opens two tabs, inserts one row, and verifies both tabs update.

## 3. Chat With AI

Repository path: `examples/chat-with-ai`

What it demonstrates:

- generated ORM tables in both the browser and the worker
- topic fan-out from `ALTER TOPIC ... ADD SOURCE ...`
- `runConsumer<T>()` with a generated row type
- `kalamDriver(client)` and `executeAsUser(...)` for typed writes in the worker

Simple worker snippet:

```ts

  chat_demo_messages as chatMessages,
  type ChatDemoMessages,
} from './schema.generated';

const client = createConsumerClient({
  url: 'http://127.0.0.1:2900',
  authProvider: async () => Auth.basic('admin', 'kalamdb123'),
});

const db = drizzle(kalamDriver(client));

await runConsumer<ChatDemoMessages>({
  client,
  name: 'chat-ai-agent',
  topic: 'chat_demo.ai_inbox',
  groupId: 'chat-ai-agent',
  onChange: async (_ctx, change) => {
    const row = change.data;
    const user = String(row.sender_username ?? '').trim();
    if (row.role !== 'user' || !user) return;

    await executeAsUser(
      client,
      db.insert(chatMessages).values({
        room: row.room,
        role: 'assistant',
        author: 'KalamDB Copilot',
        sender_username: user,
        content: `AI reply: ${row.content}`,
      }),
      user,
    );
  },
});
```

Why it matters:

This example shows the full table → topic → worker → browser loop with the current typed SDK path. The browser writes with generated tables, the worker consumes generated row types, and the reply insert stays typed as well.

Run it:

```bash
cd examples/chat-with-ai
npm install
npm run setup
npm run agent
npm run dev
```

Test it:

```bash
npm test
```

The Playwright test starts the agent, opens two tabs, sends a message, and waits for the assistant reply in both tabs.

## 4. Summarizer Agent

Repository path: `examples/summarizer-agent`

What it demonstrates:

- a worker-only `runConsumer()` example
- row enrichment back into the source table
- a failure sink table for exhausted retries

Why it matters:

If you do not need a browser at all, this is the cleanest place to start. It shows how little code is required to build a useful background worker around KalamDB topics.

Run it:

```bash
cd examples/summarizer-agent
npm install
npm run setup
npm run start
```

Test it:

```bash
npm test
```

The integration test inserts a row and waits until the worker writes the summary back.

## 5. Vector Search Pattern

Repository reference: `backend/tests/scenarios/scenario_14_vector_rag.rs`

What it demonstrates:

- `EMBEDDING(n)` columns for document and attachment vectors
- `ALTER TABLE ... CREATE INDEX ... USING COSINE`
- nearest-neighbor SQL with `ORDER BY COSINE_DISTANCE(...) LIMIT k`
- joining vector hits back to FILE-backed document rows

Why it matters:

This is the current KalamDB pattern for semantic retrieval and RAG. You keep rich document rows in one table, keep embeddings in a keyed companion table, and run vector search with normal SQL. The same flow works with `TYPE = 'USER'`, so each signed-in user only searches their own embeddings.

TypeScript query example:

```ts
const rows = await client.queryAll(`
  SELECT d.id, d.title, d.body
  FROM rag.documents AS d
  JOIN rag.documents_vectors AS v ON v.id = d.id
  ORDER BY COSINE_DISTANCE(v.doc_embedding, '[1.0,0.0,0.0]')
  LIMIT 5
`);
```

SQL setup example:

```sql
CREATE TABLE rag.documents_vectors (
  id BIGINT PRIMARY KEY,
  doc_embedding EMBEDDING(384)
) WITH (TYPE = 'USER');

ALTER TABLE rag.documents_vectors
  CREATE INDEX doc_embedding USING COSINE;
```

Read next:

- [Vector Search](https://kalamdb.org/docs/server/architecture/vector-search)
- [SDK Cookbook](https://kalamdb.org/docs/ts-sdk/cookbook)

================================================================================
# FILE Columns & Uploads
URL: https://kalamdb.org/docs/ts-sdk/file-columns
Source: content/ts-sdk/file-columns.mdx

> Work with FILE datatype columns in @kalamdb/client and @kalamdb/orm. Learn kalamFile(), queryWithFiles(), upload progress, FileRef utilities, and multi-part file handling.
# FILE Columns & Uploads

`@kalamdb/client` supports file upload and retrieval patterns for KalamDB `FILE` columns.

## Upload through Drizzle (`@kalamdb/orm`)

When you use Drizzle, pass upload bytes in `.values()` or `.set()`. You do not need a separate `queryWithFiles()` wrapper for ORM builders:

```ts

const client = createClient({
  url: 'http://localhost:2900',
  namespace: 'app',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const db = drizzle(kalamDriver(client));

await db.insert(users).values({
  id: 'user_123',
  avatar: kalamFile('avatar', avatarFile),
});
```

`kalamFile(name, blob)` marks an upload for the driver. You can also pass a `File` or `Blob` directly to a generated `file()` column. `kalamDriver()` compiles the query, rewrites upload params to `FILE("name")`, and calls `client.queryWithFiles()` automatically.

Upserts work the same way — include `kalamFile(...)` in both `.values()` and `.onConflictDoUpdate().set()` when the FILE column changes. See [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm#file-uploads-through-drizzle).

## Upload with raw SQL (`@kalamdb/client`)

Use `queryWithFiles(sql, files, params?, onProgress?)` when you are not going through Drizzle:

```ts
const avatarFile = fileInput.files?.[0];
if (!avatarFile) throw new Error('No file selected');

await client.queryWithFiles(
  'INSERT INTO app.users (id, avatar) VALUES ($1, FILE("avatar"))',
  { avatar: avatarFile },
  ['user_123']
);
```

## Insert + read FileRefs (end-to-end)

```ts
// 1) Insert a FILE via multipart upload
await client.queryWithFiles(
  'INSERT INTO app.users (id, avatar) VALUES ($1, FILE("avatar"))',
  { avatar: avatarFile },
  ['user_123'],
);

// 2) Read rows with FILE-aware wrappers
const rows = await client.queryRows<{ id: string; avatar: unknown }>(
  'SELECT id, avatar FROM app.users WHERE id = $1',
  'app.users',
  ['user_123'],
);

const avatar = rows[0]?.file('avatar');
if (avatar) {
  console.log(avatar.name, avatar.formatSize());
  console.log(avatar.downloadUrl());
}
```

`downloadUrl()` returns the authenticated file-download endpoint for the stored
file path:

```text
/v1/files/{namespace}/{table}/{sub}/{stored_name}
```

Fetch it with the same bearer token you use for SQL calls:

```ts
const response = await fetch(avatar.downloadUrl(), {
  headers: {
    Authorization: `Bearer ${token}`,
  },
});

const bytes = await response.arrayBuffer();
```

For user tables, SDK helpers intentionally generate an owner-scoped URL without
`user_id`. A normal user can download only files in their own user-table scope.
Service-role clients may upload or write through authorized `EXECUTE AS USER`
flows, but raw cross-user FILE downloads are reserved for `dba` and `system`
tokens that explicitly add an authorized `user_id` query parameter.

### Progress callback

In browser environments (`XMLHttpRequest` available), pass `onProgress`:

```ts
await client.queryWithFiles(sql, files, params, (progress) => {
  console.log(progress.file_name, progress.percent);
});
```

## Auth behavior

From source:

- SDK builds auth header via `buildAuthHeader(auth)`
- sends multipart request to `POST /v1/api/sql`
- falls back to `fetch` when XHR progress is unavailable

## FileRef model

`file_ref.ts` exposes `FileRef` and helpers:

```ts

```

### Parse and use file metadata

```ts
const ref = parseFileRef(row.attachment);
if (ref) {
  console.log(ref.name, ref.size, ref.mime);
  console.log(ref.formatSize());
  console.log(ref.relativePath());
}
```

When rows come from `queryOne/queryAll`, FILE values are `KalamCellValue`:

```ts
const row = await client.queryOne('SELECT avatar FROM app.users WHERE id = $1', ['user_123']);
const ref = row?.avatar.asFile();
if (ref) {
  console.log(ref.getDownloadUrl('http://localhost:2900', 'app', 'users'));
}
```

### Build download URL

```ts
const url = ref.getDownloadUrl('http://localhost:2900', 'app', 'users');
```

That URL points at `/v1/files/.../{stored_name}` and uses the sanitized stored
filename, not the raw file id alone.

### Type helpers

```ts
ref.isImage();
ref.isVideo();
ref.isAudio();
ref.isPdf();
ref.getTypeDescription();
```

## Storage-path internals documented by source

`FileRef` includes:

- `id`, `sub`, `name`, `size`, `mime`, `sha256`, optional `shard`
- filename sanitization and extension normalization
- shared-table shard-aware `relativePath()` generation

This is useful for building file galleries, attachment viewers, and audit tooling.

================================================================================
# TypeScript SDK
URL: https://kalamdb.org/docs/ts-sdk
Source: content/ts-sdk/index.mdx

> Use the KalamDB TypeScript packages: @kalamdb/client for apps, @kalamdb/orm for Drizzle schemas, @kalamdb/consumer for workers, and @kalamdb/cli for npm-distributed terminal installs.
# TypeScript SDK Packages

KalamDB has separate TypeScript packages for separate jobs. Pick one package path, copy the first example, then move to the package docs when you need more detail.

```mermaid
flowchart LR
  A[App or service] --> C["@kalamdb/client"]
  R[React app] --> X["@kalamdb/react"]
  X --> C["@kalamdb/client"]
  O[Typed schema or Drizzle query] --> M["@kalamdb/orm"]
  M --> C
  W[Worker or agent] --> K["@kalamdb/consumer"]
  K --> C
  C --> S[KalamDB server]
```

## Which Package Should I Use?

| Job | Install | Start here |
|---|---|---|
| Query SQL, login, upload files, subscribe to rows | `@kalamdb/client` | [Setup & Quick Start](https://kalamdb.org/docs/ts-sdk/setup) |
| Use Drizzle tables or generate `schema.ts` | `@kalamdb/client @kalamdb/orm drizzle-orm` | [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm) |
| Build React live-query hooks and components | `@kalamdb/client @kalamdb/react` plus `@kalamdb/orm drizzle-orm` for typed mode | [React Live Queries](https://kalamdb.org/docs/ts-sdk/react) |
| Consume topics, ACK messages, run agents | `@kalamdb/client @kalamdb/consumer` | [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers) |
| Install the terminal CLI through npm | `npm install -g @kalamdb/cli` | [Kalam CLI](https://kalamdb.org/docs/cli) |

`@kalamdb/cli` is published from the TypeScript package tree because npm is one of the supported
distribution channels, but it is documented with the server guides because it is an operational
client rather than an in-app SDK dependency.

## Fast Start: App Client

Use `@kalamdb/client` in browser apps, Next.js apps, scripts, and API services.

```bash
npm i @kalamdb/client
```

Copy this smoke test:

```ts

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const result = await client.query('SELECT CURRENT_USER()');
console.log(result.status, result.results?.[0]);

await client.disconnect();
```

Use it for:

- `query()`, `queryOne()`, `queryAll()`, and `queryRows()`
- `login()` and `refreshToken()`
- `live()`, `liveTable()`, and `liveEvents()`
- `queryWithFiles()` for raw SQL FILE uploads (Drizzle apps use `kalamFile()` via [ORM](https://kalamdb.org/docs/ts-sdk/orm#file-uploads-through-drizzle))

Realtime subscriptions have two documented usage modes in the client API:

- `simple`: use `live()` or `liveTable()` and let the SDK materialize the latest row set for you. Read: [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple)
- `advanced`: use `liveEvents()` when you want raw events, `change_type`, and batch metadata. Read: [Advanced Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-advanced)

## Fast Start: ORM Package

Use `@kalamdb/orm` when you want typed tables and Drizzle-style queries. It sits on top of `@kalamdb/client`; it is not a second connection layer.

```bash
npm i @kalamdb/client @kalamdb/orm drizzle-orm
```

Generate a schema from a running KalamDB server:

```bash
npx kalamdb-orm \
  --url http://localhost:2900 \
  --user admin \
  --password AdminPass123! \
  --namespace app \
  --out src/db/schema.ts
```

Keep that file fresh in local dev with `kalam dev` (watches `schema.sql` and runs `kalam schema gen` when `[dev].generate_types = true`):

```bash
kalam dev
```

Query with the generated table:

```ts

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const db = drizzle(kalamDriver(client));
const rows = await db.select().from(messages).limit(20);
console.log(rows);
```

Read next: [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm).

## Fast Start: Consumer Package

Use `@kalamdb/consumer` only in workers or agents that consume topics. Keep this code out of browser bundles.

```bash
npm i @kalamdb/client @kalamdb/consumer
```

Copy this worker skeleton:

```ts

const worker = createConsumerClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('worker', 'Secret123!'),
});

const handle = worker.consumer({
  topic: 'orders.events',
  group_id: 'billing-worker',
  auto_ack: false,
  batch_size: 10,
});

await handle.run(async (ctx) => {
  await processOrder(ctx.message.payload);
  await ctx.ack();
});
```

Read next: [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers) and [Consumer Runtime](https://kalamdb.org/docs/ts-sdk/agents).

## Fast Start: React Package

Use `@kalamdb/react` when a React app wants live rows, typed mutations, aggregate multi-query state, or assistant workflow composition.

```bash
npm i @kalamdb/client @kalamdb/react react react-dom
npm i @kalamdb/orm drizzle-orm
```

```tsx

```

Read next: [React Live Queries](https://kalamdb.org/docs/ts-sdk/react).

## How Packages Work Together

```mermaid
sequenceDiagram
  participant UI as App UI
  participant Client as KalamDB client package
  participant Server as KalamDB
  participant Worker as KalamDB consumer package

  UI->>Client: query() or live()
  Client->>Server: SQL over HTTP / WebSocket
  UI->>Client: INSERT row
  Client->>Server: write row
  Server-->>Worker: topic message available
  Worker->>Server: consumeBatch()
  Worker->>Server: write result + ack()
  Server-->>Client: live rows update
```

Use this split in real projects:

- UI code imports `@kalamdb/client`.
- Typed table code imports `@kalamdb/orm` plus the app client.
- Worker code imports `@kalamdb/consumer`.
- React helpers live in `@kalamdb/react` so the base client stays framework-neutral.

## Common Paths

| I want to... | Read |
|---|---|
| Run the first query | [Setup & Quick Start](https://kalamdb.org/docs/ts-sdk/setup) |
| Subscribe to rows for a UI | [Realtime Subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions), [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple), and [Advanced Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-advanced) |
| Build a React live-query screen | [React Live Queries](https://kalamdb.org/docs/ts-sdk/react) |
| Generate typed tables | [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm) |
| Build a topic worker | [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers) |
| Build a retrying worker | [Consumer Runtime](https://kalamdb.org/docs/ts-sdk/agents) |
| Copy full example flows | [Examples](https://kalamdb.org/docs/ts-sdk/examples) and [Cookbook](https://kalamdb.org/docs/ts-sdk/cookbook) |

================================================================================
# Drizzle ORM & Generator
URL: https://kalamdb.org/docs/ts-sdk/orm
Source: content/ts-sdk/orm.mdx

> Use @kalamdb/orm with @kalamdb/client for Drizzle queries, generated schema.ts files, KalamDB-specific column helpers, and typed live table helpers.
# Drizzle ORM & Generator

`@kalamdb/orm` is the Drizzle package for KalamDB. Use it when you want generated tables, typed queries, or table-level live helpers. It runs on top of `@kalamdb/client`.

```mermaid
flowchart LR
  A[KalamDB server] --> B[generate schema.ts]
  B --> C[Drizzle tables]
  C --> D[kalamDriver client]
  D --> A
```

Install it with the client and Drizzle:

```bash
npm i @kalamdb/client @kalamdb/orm drizzle-orm
```

## Driver quick start

Copy this after you have a generated `schema.ts` file.

```ts

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const db = drizzle(kalamDriver(client));
const recent = await db.select().from(messages).limit(20);
```

The driver uses KalamDB SQL over HTTP and normalizes temporal values for Drizzle columns. `timestamp(..., { mode: 'date' })`, `date(..., { mode: 'date' })`, and `time(...)` can read KalamDB's numeric wire values without app-side parsing.

Pass a full `KalamDBClient` to `kalamDriver()` so FILE uploads can route to `queryWithFiles()` when needed.

## Table helpers

Use `kTable` instead of raw `pgTable` so the schema retains KalamDB table type metadata.

```ts

  id: bigint('id', { mode: 'bigint' }).primaryKey(),
  owner_id: uuid('owner_id').notNull(),
  title: text('title').notNull(),
  metadata: jsonb('metadata'),
  attachment: file('attachment'),
  raw_bytes: bytes('raw_bytes'),
  doc_embedding: embedding('doc_embedding', 384),
  created_at: timestamp('created_at', { mode: 'date' }).notNull(),
});
```

Available table helpers:

| Helper | Use for |
|---|---|
| `kTable.shared(name, columns, options?)` | `WITH (TYPE = 'SHARED')` tables |
| `kTable.user(name, columns, options?)` | per-user isolated tables |
| `kTable.stream(name, columns, options?)` | stream tables |
| `kTable.system(name, columns, options?)` | system metadata tables |
| `kSystemColumns([...])` | `_seq`, `_deleted`, and `_commit_seq` typed hidden columns |

KalamDB-specific columns:

| Helper | Runtime value |
|---|---|
| `file(name)` | `FileRef | null` on read; pass `kalamFile()`, `File`, or `Blob` on write |
| `bytes(name)` | `Uint8Array | null` |
| `embedding(name, dimensions)` | `number[] | null` |

## Generate `schema.ts`

Generate a Drizzle schema from a running KalamDB server:

```bash
npx kalamdb-orm \
  --url http://localhost:2900 \
  --user admin \
  --password AdminPass123! \
  --namespace app \
  --include-system-columns \
  --out src/db/schema.ts
```

The generator reads `SHOW TABLES`, calls `DESCRIBE <namespace.table>` when column metadata is incomplete, and emits only the imports needed by the generated tables. It preserves primary keys, not-null constraints, system-column options, comments, table config constants such as `chat_messagesConfig`, and `$inferSelect` / `$inferInsert` aliases.

Options:

| Option | Description |
|---|---|
| `--namespace <name>` | Limit output to one or more namespaces. Repeat it or pass comma-separated names. |
| `--include-system` | Include `system` and `dba` tables. |
| `--include-system-columns <mode>` | Add typed hidden columns to generated table definitions. Use `all`, `_seq`, or `_deleted`. |
| `--bigint-mode <mode>` | Choose how generated `BIGINT` columns are emitted. Accepted values are `string`, `bigint`, or `number`; default is `string`. |
| `--no-type-aliases` | Skip generated `$inferSelect` and `$inferInsert` aliases. |

`BIGINT` defaults to `text()` because KalamDB transports Int64 values as strings to preserve precision. Choose `--bigint-mode bigint` when you want Drizzle to coerce values to native `bigint`, or `--bigint-mode number` only when values fit safely in JavaScript numbers.

## Keep `schema.ts` fresh during local development

Add a generator script to your app so the command stays short:

```json
{
  "scripts": {
    "schema:gen": "kalamdb-orm --url http://localhost:2900 --user admin --password AdminPass123! --namespace app --out src/db/schema.ts"
  }
}
```

In a `kalam.toml` project, `kalam dev` watches `schema.sql` and reruns `kalam schema gen` when `[dev].generate_types = true`:

```bash
kalam dev
```

See [Local Development](https://kalamdb.org/docs/cli/dev). The legacy `kalam --watch-schema` server-polling flag is [deprecated](https://kalamdb.org/docs/cli/schema-watch#deprecated-kalam---watch-schema).

## Exact KalamDB datatype mapping

| KalamDB type | Generated Drizzle helper | Read behavior |
|---|---|---|
| `BOOLEAN` | `boolean()` | boolean |
| `INT` | `integer()` | number |
| `SMALLINT` | `smallint()` | number |
| `BIGINT` | `text()` by default | string for Int64 precision |
| `DOUBLE` | `doublePrecision()` | number |
| `FLOAT` | `real()` | number |
| `TEXT` | `text()` | string |
| `TIMESTAMP` | `timestamp(..., { mode: 'date' })` | `Date` when using date mode |
| `DATETIME` | `timestamp(..., { mode: 'date' })` | `Date` when using date mode |
| `DATE` | `date(..., { mode: 'date' })` | UTC-date `Date` |
| `TIME` | `time()` | `HH:mm:ss[.fraction]` string |
| `JSON` | `jsonb()` | JSON value |
| `BYTES` | `bytes()` | `Uint8Array` |
| `EMBEDDING(n)` | `embedding(name, n)` | `number[]` |
| `UUID` | `uuid()` | UUID string |
| `DECIMAL(p,s)` | `numeric()` | exact decimal string |
| `FILE` | `file()` | `FileRef | null` on read; `kalamFile()` / `File` / `Blob` on write |

## Upsert and returning

Drizzle `returning()` and `onConflictDoUpdate()` compile to KalamDB SQL and run through the same driver:

```ts

const inserted = await db
  .insert(users)
  .values({ id: 1, name: 'Nader', age: 3 })
  .returning();

const updated = await db
  .insert(users)
  .values({ id: 1, name: 'Nader Updated', age: 5 })
  .onConflictDoUpdate({
    target: users.id,
    set: { name: sql`excluded.name`, age: sql`excluded.age` },
  })
  .returning();
```

## FILE uploads through Drizzle

You do **not** wrap every query. Normal `select`, `insert`, `update`, and `delete` calls go through `db` directly. Only FILE bytes need multipart SQL, and the driver handles that when upload values appear in `.values()` or `.set()`:

```ts

await db.insert(attachments).values({
  id: 'att_1',
  file_data: kalamFile('upload', selectedFile),
});

await db
  .insert(context_files)
  .values({
    path: 'notes/readme.md',
    file_ref: kalamFile('upload', uploadFile),
    updated_at: new Date(),
  })
  .onConflictDoUpdate({
    target: context_files.path,
    set: {
      file_ref: kalamFile('upload', uploadFile),
      updated_at: new Date(),
    },
  });
```

You can also pass a `File` or `Blob` directly to a generated `file()` column; the driver uses the filename (or `upload` for plain blobs) as the multipart field name.

`kalamDriver()` compiles the Drizzle builder, normalizes quoted identifiers and `DEFAULT` columns for KalamDB, replaces upload params with `FILE("name")`, and calls `client.queryWithFiles()` when bytes are present.

For raw SQL strings (no Drizzle builder), use `queryWithFiles()` from `@kalamdb/orm` or `@kalamdb/client`. Use `compileQuery()` when you need normalized SQL and params without executing.

See [FILE Columns & Uploads](https://kalamdb.org/docs/ts-sdk/file-columns) for download URLs, `FileRef`, and progress callbacks.

## Execute as a user

Agents and service workers can compile a Drizzle builder and run it through KalamDB's `EXECUTE AS USER` path:

```ts

await executeAsUser(
  client,
  db.insert(messages).values({
    room: 'main',
    role: 'assistant',
    author: 'KalamDB Copilot',
    sender_username: 'alice',
    content: 'Done.',
  }),
  'alice',
);
```

Only pass a user id that your service account is authorized to impersonate.

## Live table helpers

Use `liveTable()` for React state, dashboards, and admin UI screens that want the current materialized row set.

```ts

const stop = await liveTable(client, messages, (rows) => {
  setRows(rows);
}, {
  lastRows: 100,
});

await stop();
```

`liveTable()` reuses the same `@kalamdb/client` connection and normalizes timestamp/date/time fields from the Drizzle table metadata. It accepts the same row-oriented options as `client.live()`, including `lastRows`, `from`, `limit`, `getKey`, and `onCheckpoint`. Use `client.liveEvents()` directly when you need low-level change events.

## UI and example fit

Use this package in UI code or service code that wants typed tables. Worker code should import `@kalamdb/consumer` only when it needs topic consumption.

## Copy This Project Shape

```text
src/
  db/
    schema.ts      # generated by kalamdb-orm
    client.ts      # createClient() + drizzle(kalamDriver(client))
  app/
    messages.ts    # Drizzle queries against generated tables
```

================================================================================
# Querying & DML
URL: https://kalamdb.org/docs/ts-sdk/querying
Source: content/ts-sdk/querying.mdx

> Execute SQL queries with @kalamdb/client. Learn query(), queryOne(), queryAll(), queryRows(), executeAsUser(), and the DML convenience methods.
# Querying & DML

## Raw SQL execution

```ts
const res = await client.query('SELECT * FROM app.messages LIMIT 10');
console.log(res.status, res.results?.[0]?.named_rows);
```

With parameters:

```ts
const res = await client.query(
  'SELECT * FROM app.messages WHERE conversation_id = $1',
  ['conv_42'],
);
```

From `client.ts`, params are sent through `queryWithParams(sql, JSON.stringify(params))`.

## Vector search queries

Vector search uses the same `query()` and helper APIs as any other SQL workflow.

```ts
const matches = await client.queryAll(
  `SELECT d.id, d.title
   FROM rag.documents AS d
   JOIN rag.documents_vectors AS v ON v.id = d.id
   ORDER BY COSINE_DISTANCE(v.doc_embedding, $1)
   LIMIT 5`,
  [JSON.stringify(queryEmbedding)],
);
```

Typical SQL setup:

```sql
CREATE TABLE rag.documents_vectors (
  id BIGINT PRIMARY KEY,
  doc_embedding EMBEDDING(384)
) WITH (TYPE = 'USER');

ALTER TABLE rag.documents_vectors
  CREATE INDEX doc_embedding USING COSINE;
```

See [/docs/server/architecture/vector-search](https://kalamdb.org/docs/server/architecture/vector-search) for the full
schema and retrieval pattern.

## Row helpers (`named_rows`)

`query()` returns a raw `QueryResponse`. Results are exposed via `results?.[0]?.named_rows` as plain named-column maps from the server.

For ergonomic access, use:

- `queryOne(sql, params?)` → `RowData | null`
- `queryAll(sql, params?)` → `RowData[]`
- `queryRows<T>(sql, tableName, params?)` → `KalamRow<T>[]`

`queryOne()` and `queryAll()` are intentionally not generic. They return `RowData` with `KalamCellValue` wrappers. Use `queryRows<T>()` when you want typed application objects plus FILE-aware row helpers.

```ts
const one = await client.queryOne('SELECT id, name FROM app.users WHERE id = $1', ['u1']);
console.log(one?.id.asString(), one?.name.asString());

const many = await client.queryAll('SELECT id, active FROM app.users');
for (const row of many) {
  console.log(row.id.asString(), row.active.asBool());
}
```

For FILE-aware row wrappers (`.file()` / `.downloadUrl()`):

```ts
interface MessageRow {
  id: string;
  sender: string;
  content: string;
}

const rows = await client.queryRows<MessageRow>(
  'SELECT id, sender, content, avatar FROM app.messages',
  'app.messages',
);
console.log(rows[0]?.data.sender);
```

The helper methods above wrap each value as `KalamCellValue` (`.asString()`, `.asInt()`, `.asFile()`, etc.).
See [/docs/ts-sdk/cell-values](https://kalamdb.org/docs/ts-sdk/cell-values) for the full API.

## executeAsUser

```ts
await client.executeAsUser(
  'SELECT * FROM app.messages LIMIT 1',
  'alice',
);
```

Source behavior:

- wraps SQL as `EXECUTE AS '<user_id>' ( <statement> )`
- strips trailing semicolons in inner SQL
- escapes `'` in the target user id
- requires one non-empty statement and a non-empty user id
- is valid for USER and STREAM tables; SHARED tables use their table policy directly
- KalamDB authorizes cross-user targets with its role matrix: system can target any role, DBA can target DBA/service/user, service can target service/user, and regular users can only target themselves

For the underlying table policy and role matrix, see
[/docs/server/architecture/table-types](https://kalamdb.org/docs/server/architecture/table-types) and
[/docs/server/sql-reference/impersonation](https://kalamdb.org/docs/server/sql-reference/impersonation).

## Convenience DML methods

```ts
await client.insert('app.messages', {
  id: 1,
  sender: 'alice',
  content: 'hello',
});

await client.update('app.messages', 1, { content: 'updated' });
await client.delete('app.messages', 1);
```

### Important caveat

`update()` builds SQL string values directly (with quote escaping for strings). For complex/untrusted inputs, prefer `query()` with parameter placeholders where possible.

## Type shape notes from generated declarations

Generated SDK types mark some fields optional:

```ts
interface QueryResponse {
  status: 'success' | 'error';
  results?: QueryResult[];
  took?: number;
  error?: ErrorDetail;
}

interface QueryResult {
  schema?: SchemaField[];
  rows?: unknown[][];
  named_rows?: Record<string, unknown>[];
  row_count: number;
  message?: string;
}
```

Handle missing `results`/`schema` defensively in generic tooling; prefer `named_rows` when present.

================================================================================
# React Package
URL: https://kalamdb.org/docs/ts-sdk/react
Source: content/ts-sdk/react.mdx

> Use @kalamdb/react for React live queries, typed hooks, render-prop components, mutation state, and assistant app examples.
# React Package

`@kalamdb/react` adds React hooks and render-prop components for KalamDB live rows. Use it when a React app needs realtime tables, typed mutations, or a screen with many live datasets.

## Install

```bash
npm i @kalamdb/react @kalamdb/client react react-dom
npm i @kalamdb/orm drizzle-orm
```

`@kalamdb/orm` and `drizzle-orm` are needed for typed table mode. Raw SQL mode only needs `@kalamdb/react` and `@kalamdb/client`.

## What It Adds

| Export | Type | Use it for |
|---|---|---|
| `KalamProvider` | component | Put one KalamDB client in React context. |
| `LiveQuery` | component | Render one live query with typed rows or raw SQL rows. |
| `LiveQueries` | component | Render several named live queries together. |
| `useKalamClient` | hook | Read the client from context, or pass one directly. |
| `useLiveQuery` | hook | Read one live query and call `insert`, `update`, `remove`, or `refetch`. |
| `useLiveQueries` | hook | Read several Drizzle live queries with one aggregate state. |
| `useLiveSelection` | hook | Derive view data from live rows. |
| `useMutationState` | hook | Track local insert, update, delete, and error state. |
| `useMutationActions` | hook | Use mutation helpers without opening a live query. |

The live query context includes `rows`, `state`, `insert`, `update`, `remove`, `clearError`, and `refetch`. `state.updating` and `state.deleting` are `Set<string | number>` values keyed by row id.

## App Setup

Create one client and wrap your app.

```tsx filename="src/App.tsx"

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

  return (
    
  );
}
```

## Typed Table Schema

Use `kTable.user(...)` when you want typed rows and typed mutation payloads.

```ts filename="src/schema.ts"

  id: text('id').primaryKey(),
  roomId: text('room_id').notNull(),
  body: text('body').notNull(),
  authorName: text('author_name'),
  createdAt: timestamp('created_at'),
});

  id: text('id').primaryKey(),
  value: integer('value').notNull(),
  isFavorite: boolean('is_favorite').notNull(),
});
```

## One Live Query

`LiveQuery` is the simplest UI component. The child receives rows, connection state, mutation state, and mutation helpers.

```tsx filename="src/MessagesPane.tsx"

  return (
    
  );
}
```

Use `deps` when a query depends on props or state. The live controller restarts when those values change.

## Hook Form

`useLiveQuery` returns the same context without a render-prop wrapper.

```tsx filename="src/MessageCount.tsx"

  const { rows, state, refetch } = useLiveQuery({
    table: messages,
    where: (table) => eq(table.roomId, roomId),
    deps: [roomId],
  });

  return (
    
  );
}
```

Use `select` when a component only needs derived data.

```tsx
const summary = useLiveQuery({
  table: messages,
  where: (table) => eq(table.roomId, roomId),
  deps: [roomId],
  select: (context) => ({
    total: context.rows.length,
    first: context.rows[0]?.body ?? '',
    loading: context.state.loading,
  }),
});
```

## Raw SQL Mode

Raw SQL mode is useful for simple views that do not use Drizzle tables. Rows are `RowData`, so convert cell values before rendering.

```tsx filename="src/SqlMessagesPane.tsx"

  const query = `SELECT id, body FROM chat.messages WHERE room_id = '${roomId}' ORDER BY created_at ASC LIMIT 100`;

  return (
    
  );
}
```

Use `getKey` when the row key is not obvious, or when the query returns a computed key.

## Many Live Queries

`LiveQueries` and `useLiveQueries` open one controller per named query. The aggregate state is useful for dashboards and assistant screens.

```tsx filename="src/Dashboard.tsx"

  return (
    
  );
}
```

## Assistant Screen Pattern

The newest package example uses named datasets for messages, tool calls, tool results, typing, presence, and approvals.

```tsx filename="src/AssistantWorkspace.tsx"

  const live = useLiveQueries({
    queries: {
      messages: { table: messages, where: (table) => eq(table.threadId, threadId), deps: [threadId] },
      toolCalls: {
        table: toolCalls,
        where: (table) => eq(table.threadId, threadId),
        orderBy: (table) => desc(table.createdAt),
        deps: [threadId],
      },
      toolResults: {
        table: toolResults,
        where: (table) => eq(table.threadId, threadId),
        orderBy: (table) => desc(table.createdAt),
        deps: [threadId],
      },
      typing: { table: typing, where: (table) => eq(table.threadId, threadId), deps: [threadId] },
      presence: { table: presence, where: (table) => eq(table.threadId, threadId), deps: [threadId] },
      approvals: { table: approvals, where: (table) => eq(table.threadId, threadId), deps: [threadId] },
    },
    deps: [threadId],
  });

  const view = useLiveSelection(live, (context) => ({
    messages: context.messages.rows,
    activeToolCalls: context.toolCalls.rows.filter((row) => row.status !== 'completed'),
    latestToolResults: context.toolResults.rows,
    typingUsers: context.typing.rows.map((row) => row.userName),
    onlineUsers: context.presence.rows.filter((row) => row.status === 'online'),
    pendingApprovals: context.approvals.rows.filter((row) => row.status === 'pending'),
    approve: (id: string) => context.update(approvals, id).set({ status: 'approved' }),
    reject: (id: string) => context.update(approvals, id).set({ status: 'rejected' }),
  }));

  return (
    
  );
}
```

## Mutation Helpers

Mutations work with typed tables or string table names.

```tsx
await insert(messages).values({
  id: crypto.randomUUID(),
  roomId: 'main',
  body: 'hello',
  createdAt: new Date(),
});

await update(messages, messageId).set({ body: 'edited' });
await remove(messages, messageId);

await insert('chat.messages', {
  id: crypto.randomUUID(),
  room_id: 'main',
  body: 'raw sql payload',
});
```

For typed tables, camel-case keys map to database column names and Drizzle encoders run before the payload is sent.

## Examples

### Package Examples

The React package includes small source examples in `link/sdks/typescript/react/example`:

- `messages-pane.tsx`: typed `LiveQuery` with insert state.
- `sql-messages-pane.tsx`: raw SQL `LiveQuery` with `getKey`.
- `assistant-workspace.tsx`: `useLiveQueries` plus `useLiveSelection` for messages, tools, results, typing, presence, and approvals.

### React AI Chat App

The main runnable example is `examples/react-ai-chat`. It shows `KalamProvider`, `LiveQueries`, conversation navigation, message history, file attachments, streamed typing tokens, final assistant inserts, and approval actions.

```bash
cd examples/react-ai-chat
npm install
npm run setup
npm run agent
npm run dev
```

Open the Vite URL, usually `http://127.0.0.1:5176`.

For browser-only demo mode, skip `npm run setup` and set this in `.env.local`:

```bash
VITE_KALAMDB_DEMO_MODE=true
```

Validate the example:

```bash
npm run build
npm test
```

Read next: [TypeScript Examples](https://kalamdb.org/docs/ts-sdk/examples) and [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm).

================================================================================
# Setup & Quick Start
URL: https://kalamdb.org/docs/ts-sdk/setup
Source: content/ts-sdk/setup.mdx

> Install @kalamdb/client for app-facing TypeScript code, add @kalamdb/orm for Drizzle schemas, and add @kalamdb/consumer when you need topic workers or agents.
# Setup & Quick Start

> **Status:** The TypeScript SDK is in **beta**.

This page gets you from install to first query. Pick the package set that matches the code you are writing.

```mermaid
flowchart LR
  A[App code] --> B["@kalamdb/client"]
  C[Typed tables] --> D["@kalamdb/orm"]
  D --> B
  E[Worker process] --> F["@kalamdb/consumer"]
  F --> B
```

## Pick the right package

| If you are writing... | Install |
|---|---|
| UI, service, or script that runs SQL | `@kalamdb/client` |
| Typed Drizzle table code | `@kalamdb/client @kalamdb/orm drizzle-orm` |
| Background worker or agent | `@kalamdb/client @kalamdb/consumer` |

Looking for Rust instead? Use the [Rust SDK](https://kalamdb.org/docs/rust-sdk).

## Install

### App client only

- 

```bash
npm i @kalamdb/client
```

### App client plus worker runtime

- 
- 

```bash
npm i @kalamdb/client @kalamdb/consumer
```

### App client plus Drizzle ORM

- 
- 

```bash
npm i @kalamdb/client @kalamdb/orm drizzle-orm
```

Runtime requirements:

- Node.js `>= 18` (or a modern browser for `@kalamdb/client`)
- running KalamDB server (`http://localhost:2900` by default)

## Quick start: `@kalamdb/client`

Copy this into a script after KalamDB is running locally.

```ts

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const result = await client.query('SELECT CURRENT_USER()');
console.log(result.status, result.results?.[0]);

await client.disconnect();
```

Notes:

- `query()` works over HTTP and does not require a WebSocket connection.
- When using `Auth.basic(user, password)`, the SDK only sends those credentials to `POST /v1/api/auth/login`, then switches authenticated requests and WebSocket traffic to JWT automatically.
- The high-level SDK manages the WebSocket lifecycle automatically. `connect()` is public for eager or manual reconnect flows, but most apps do not need to call it directly.
- With the default `wsLazyConnect: true`, the first subscription call opens the shared socket automatically.

### Start a live query

```ts
const inboxSql = `
  SELECT id, room, role, body, created_at
  FROM support.inbox
  WHERE room = 'main'
`;

const stop = await client.live(
  inboxSql,
  (rows) => {
    renderInbox(rows);
  },
  {
    lastRows: 100,
  },
);

await stop();
await client.disconnect();
```

Notes:

- `live()` is the recommended UI API because it gives you the current materialized row set directly.
- Use `liveEvents()` only when you need raw low-level events such as `subscription_ack` or `change` frames.
- Keep live SQL in `SELECT ... FROM ... WHERE ...` form without `ORDER BY` or `LIMIT`.

### Parameterized queries

Use `$1`, `$2`, ... placeholders:

```ts
const filtered = await client.query(
  'SELECT * FROM app.messages WHERE conversation_id = $1 AND is_deleted = $2',
  ['conv_42', false],
);
```

### Optional WASM path override

In environments where auto-WASM resolution fails, pass `wasmUrl`:

```ts
const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
  wasmUrl: '/wasm/kalam_client_bg.wasm',
});
```

## Quick start: `@kalamdb/consumer`

Use a separate worker client when you need topic polling or agents. This package is for Node.js worker processes, not browser bundles.

```ts

const worker = createConsumerClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('worker', 'Secret123!'),
});

const handle = worker.consumer({
  topic: 'orders',
  group_id: 'billing',
  auto_ack: true,
  batch_size: 10,
});

await handle.run(async (ctx) => {
  console.log(ctx.message.offset, ctx.message.value);
});
```

The consumer client reuses the same auth model as `@kalamdb/client`, but adds topic-specific APIs:

- `consumer({...}).run(...)`
- `consumeBatch(...)`
- `ack(...)`
- `runConsumer(...)`

If WASM auto-resolution fails in the worker runtime, use `consumerWasmUrl`.

## Quick start: `@kalamdb/orm`

Use this path when you want generated tables or Drizzle queries.

Generate a Drizzle schema from a running KalamDB server:

```bash
npx kalamdb-orm \
  --url http://localhost:2900 \
  --user admin \
  --password AdminPass123! \
  --namespace app \
  --out src/db/schema.ts
```

For local development in a `kalam.toml` project, run `kalam dev` so edits to `schema.sql` rerun codegen automatically:

```bash
kalam dev
```

Then use the generated tables with the driver:

```ts

const client = createClient({
  url: 'http://localhost:2900',
  authProvider: async () => Auth.basic('admin', 'AdminPass123!'),
});

const db = drizzle(kalamDriver(client));
const rows = await db.select().from(messages).limit(20);
```

FILE uploads through Drizzle use `kalamFile(name, blob)` in `.values()` / `.set()`; see [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm#file-uploads-through-drizzle).

## Common startup pitfalls

- For local dev auth, confirm server credentials and URL first.
- If browser bundling cannot find the client WASM file, provide `wasmUrl` explicitly.
- If the worker runtime cannot find the consumer WASM file, provide `consumerWasmUrl` explicitly.
- Set `wsLazyConnect: false` only when you want the app client's WebSocket established eagerly.

## Next

### App-facing docs

- [Examples](https://kalamdb.org/docs/ts-sdk/examples)
- [Chat Quickstart (Docs-Only)](https://kalamdb.org/docs/ts-sdk/chat-quickstart)
- [Authentication](https://kalamdb.org/docs/ts-sdk/auth)
- [Drizzle ORM & Generator](https://kalamdb.org/docs/ts-sdk/orm)
- [Client Lifecycle](https://kalamdb.org/docs/ts-sdk/client-lifecycle)
- [Querying & DML](https://kalamdb.org/docs/ts-sdk/querying)
- [Realtime Subscriptions](https://kalamdb.org/docs/ts-sdk/subscriptions)

### Worker docs

- [Topic Consumers & ACK](https://kalamdb.org/docs/ts-sdk/consumers)
- [Consumer Runtime](https://kalamdb.org/docs/ts-sdk/agents)
- [Cookbook](https://kalamdb.org/docs/ts-sdk/cookbook)

================================================================================
# Advanced Realtime Mode
URL: https://kalamdb.org/docs/ts-sdk/subscriptions-advanced
Source: content/ts-sdk/subscriptions-advanced.mdx

> Use liveEvents() when you want raw subscription events, change types, batch metadata, and explicit checkpoint handling.
# Advanced Realtime Mode

Use advanced mode when you want protocol events instead of a pre-materialized row set.

If your UI only needs current rows, use `live()` or `liveTable()` first. They expose `onCheckpoint` too, so durable resume does not require raw event handling.

`liveEvents(sql, onEvent, options?)` is the low-level API.

## Low-level SQL events

```ts

const stop = await client.liveEvents(
  'SELECT * FROM app.messages WHERE conversation_id = 42',
  (event) => {
    if (event.type !== MessageType.Change) {
      return;
    }

    if (event.change_type === ChangeType.Insert) {
      console.log(event.rows);
    }
  },
  {
    batchSize: 100,
    lastRows: 50,
  },
);

await stop();
```

Use this path for protocol tooling, debugging, custom reconciliation, or event processors that need direct access to `subscription_ack`, `initial_data_batch`, `change`, and `error` frames.

## Options

```ts
{
  batchSize?: number;
  lastRows?: number;
  from?: SeqId | number | string;
  autoFetchBatches?: boolean;
  onCheckpoint?: ({ lastSeqId, subscriptionId }) => void;
  onError?: (event) => void;
}
```

- `batchSize` chunks the initial snapshot from the server.
- `lastRows` includes the last N rows before live changes begin.
- `from` resumes from a known `SeqId` checkpoint.
- `autoFetchBatches` controls whether the client asks for remaining startup batches automatically.
- `onCheckpoint` fires whenever the SDK observes a newer applied sequence ID.
- `onError` receives protocol error frames without forcing your event callback to branch on every error.

## Checkpoints

```ts
let latestSeq;

const stop = await client.liveEvents(
  'SELECT id, body FROM app.messages WHERE conversation_id = 42',
  (event) => {
    if (event.type === MessageType.Change) {
      applyChange(event.change_type, event.rows, event.old_values);
    }
  },
  {
    lastRows: 20,
    from: latestSeq,
    onCheckpoint: ({ lastSeqId }) => {
      latestSeq = lastSeqId;
      localStorage.setItem('messages.seq', lastSeqId.toString());
    },
  },
);
```

The checkpoint is a single `SeqId`. Snapshot and commit boundaries stay server-owned.

## Event model

Use runtime enums from `@kalamdb/client`:

```ts

```

Common sequence:

1. `subscription_ack`
2. one or more `initial_data_batch`
3. `change` events with `insert`, `update`, or `delete`
4. optional `error`

## SQL safety for dynamic filters

`liveEvents()` accepts a SQL string. For dynamic values:

- Prefer validated numeric IDs before interpolation.
- Do not interpolate raw user text directly into SQL.

```ts
function buildConversationSql(conversationId: number): string {
  if (!Number.isInteger(conversationId) || conversationId <= 0) {
    throw new Error('conversationId must be a positive integer');
  }

  return `SELECT * FROM app.messages WHERE conversation_id = ${conversationId}`;
}
```

## Related docs

- [Live Queries](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple)

================================================================================
# Simple Realtime Mode
URL: https://kalamdb.org/docs/ts-sdk/subscriptions-simple
Source: content/ts-sdk/subscriptions-simple.mdx

> Use live() and liveTable() when you want @kalamdb/client to materialize the latest row set for your UI.
# Simple Realtime Mode

Use simple mode when your application wants the current row set, not raw subscription protocol events.

Start with `live()`.

It materializes the latest rows in the client and applies `insert`, `update`, and `delete` for you.

## Recommended: materialized live rows

```ts
const inboxSql = `
  SELECT id, room, role, body, created_at
  FROM support.inbox
  WHERE room = 'main'
`;

const unsub = await client.live(
  inboxSql,
  (rows) => {
    // USER tables are tenant-private.
    // The same SQL can run for every signed-in account, but the callback only
    // receives rows from that caller's own table partition.
    renderInbox(rows);
  },
  {
    limit: 100,
    lastRows: 100,
    onError: (event) => {
      console.error(event.code, event.message);
    },
  },
);
```

Use `liveTable()` for the same behavior with table-name sugar:

```ts
const unsub = await client.liveTable('support.inbox', (rows) => {
  renderInbox(rows);
});
```

This path is implemented in the shared Rust core, so TypeScript and Dart use the same row materialization behavior by default.

## Resume from a specific `SeqId`

Persist the latest applied `SeqId` and feed it back into `from` on the next session.

```ts

const inboxSql = `
  SELECT id, room, role, body, created_at
  FROM support.inbox
  WHERE room = 'main'
`;

const savedSeqText = localStorage.getItem('support.inbox.seq');
const startFrom = savedSeqText ? SeqId.from(savedSeqText) : undefined;

const unsub = await client.live(
  inboxSql,
  (rows) => {
    renderInbox(rows);
  },
  {
    limit: 100,
    lastRows: 100,
    ...(startFrom ? { from: startFrom } : {}),
    onCheckpoint: ({ lastSeqId }) => {
      localStorage.setItem('support.inbox.seq', lastSeqId.toString());
    },
  },
);
```

This is the recommended pattern for chat timelines, activity feeds, audit streams, and reconnect-heavy mobile sessions.

## Table subscription sugar

If you only need `SELECT * FROM table`, use the shorthand APIs:

```ts
const lowLevel = await client.liveEvents('SELECT * FROM app.messages', (event) => {
  console.log(event.type);
});

const highLevel = await client.liveTable('support.inbox', (rows) => {
  renderInbox(rows);
});
```

## Custom row identity in TypeScript

By default, the high-level API reconciles rows using the row `id` field in the Rust core.

If your query does not expose a stable `id`, prefer declarative `getKey` column names so reconciliation still stays inside the shared Rust core:

```ts
const unsub = await client.live(
  "SELECT room, message_id, body, created_at FROM chat.messages WHERE room = 'main'",
  (rows) => {
    console.log(rows);
  },
  {
    getKey: ['room', 'message_id'],
  },
);
```

Use `getKey` only when the identity must be derived by arbitrary JavaScript code:

```ts
const unsub = await client.live(
  "SELECT room, created_at, body FROM chat.messages WHERE room = 'main'",
  (rows) => {
    console.log(rows);
  },
  {
    getKey: (row) => `${row.room.asString()}:${row.created_at.asString()}`,
  },
);
```

When `getKey` is provided, reconciliation falls back to the TypeScript SDK layer because arbitrary JavaScript callbacks cannot be shared through the Rust core.

## What simple mode gives you

- `live()` or `liveTable()` for ready-to-render row arrays
- SDK-managed row reconciliation for `insert`, `update`, and `delete`
- `limit` to cap the materialized row set after startup
- the simplest path for UI state in dashboards, feeds, and chat windows

## Related docs

- [Live Queries](https://kalamdb.org/docs/ts-sdk/subscriptions)
- [Advanced Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-advanced)

================================================================================
# Live Queries
URL: https://kalamdb.org/docs/ts-sdk/subscriptions
Source: content/ts-sdk/subscriptions.mdx

> Choose between simple materialized live rows and advanced low-level SQL live control in the @kalamdb/client API.
# Live Queries

`@kalamdb/client` supports realtime subscriptions via WebSocket.

The client has exactly three realtime entry points:

- `live(sql, onRows, options?)` for materialized rows from any supported `SELECT`.
- `liveTable(tableName, onRows, options?)` for materialized rows from `SELECT * FROM table`.
- `liveEvents(sql, onEvent, options?)` for low-level protocol events.

## Which Mode Should I Choose?

| Mode | Start with | Best for | You manage |
|---|---|---|---|
| `simple` | `live()` or `liveTable()` | UI state, feeds, timelines, dashboards, chat windows | almost nothing; the SDK materializes the row set for you |
| `advanced` | `liveEvents()` | custom event handling and protocol-aware realtime logic | raw events, `change_type`, `rows`, `old_values`, and batch metadata |

Use `simple` when you want the latest rows.

Use `advanced` when you want to own the subscription lifecycle and event handling.

## Simple Mode

- Read: [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple)
- Start with `live()` or `liveTable()`.
- The SDK reconciles inserts, updates, and deletes into one current row set.

```ts
const unsub = await client.live(
  "SELECT id, room, body, created_at FROM support.inbox WHERE room = 'main'",
  (rows) => renderInbox(rows),
  {
    limit: 100,
    lastRows: 100,
  },
);
```

## Advanced Mode

- Read: [Advanced Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-advanced)
- Start with `liveEvents()` for raw events.
- This mode exposes `change_type`, `rows`, `old_values`, `_seq`, `subscription_id`, and `batch_control` directly.

```ts
const stop = await client.liveEvents(
  'SELECT * FROM app.messages WHERE conversation_id = 42',
  (event) => {
    if (event.type === MessageType.InitialDataBatch) {
      renderBatch(event.rows);
    }
  },
  {
    batchSize: 5,
    autoFetchBatches: false,
  },
);
```

## Shared Subscription Rules

- Live SQL must stay within the strict supported shape: `SELECT ... FROM ... WHERE ...`.
- Do not include `ORDER BY` or `LIMIT` inside `live()` or `liveEvents()` SQL.
- `batchSize` chunks the initial snapshot coming from the server.
- `lastRows` controls how much history to rewind before live changes begin.
- `from` resumes from a known `SeqId` checkpoint.
- `limit` applies only to the high-level materialized row set that `live()` keeps after startup.

## Next Reads

- [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple)
- [Advanced Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-advanced)
- [TypeScript SDK Overview](https://kalamdb.org/docs/ts-sdk)

================================================================================
# Types & Enums
URL: https://kalamdb.org/docs/ts-sdk/types
Source: content/ts-sdk/types.mdx

> Core TypeScript types, runtime enums, and helper utilities exported by @kalamdb/client, plus the worker-specific types that live in @kalamdb/consumer.
# Types & Enums

`@kalamdb/client` exports the app-facing runtime enums and rich TypeScript types.

## Runtime enums

```ts

```

```ts
enum MessageType {
  SubscriptionAck = 'subscription_ack',
  InitialDataBatch = 'initial_data_batch',
  Change = 'change',
  Error = 'error',
}

enum ChangeType {
  Insert = 'insert',
  Update = 'update',
  Delete = 'delete',
}
```

## Auth types

```ts
type AuthCredentials =
  | { type: 'basic'; user: string; password: string }
  | { type: 'jwt'; token: string }
  | { type: 'none' };
```

## ClientOptions (construction)

`createClient({ ... })` accepts a `ClientOptions` object. Common fields:

- `url: string` — server base URL
- `authProvider: () => Promise<AuthCredentials>` — required auth source for the high-level client
- `authProviderMaxAttempts?: number` — retry attempts for transient auth-provider failures
- `authProviderInitialBackoffMs?: number` — initial auth-provider retry backoff
- `authProviderMaxBackoffMs?: number` — max auth-provider retry backoff
- `wsLazyConnect?: boolean` — default `true`; defer the WebSocket until the first subscription/live call
- `pingIntervalMs?: number` — default `5000`; application-level keepalive interval
- `disableCompression?: boolean` — dev-only debugging (`?compress=false`)
- `wasmUrl?: string | BufferSource` — override WASM loading
- `onConnect?`, `onDisconnect?`, `onError?`, `onReceive?`, `onSend?` — connection lifecycle callbacks
- `logLevel?`, `logListener?` — SDK logging controls

There is no high-level `auth` or `autoConnect` option on `createClient(...)`.

`createConsumerClient({ ... })` from `@kalamdb/consumer` accepts the same base options plus `consumerWasmUrl?: string | BufferSource` for worker-only WASM resolution.

## Query response model

```ts
interface QueryResponse {
  status: 'success' | 'error';
  results?: QueryResult[];
  took?: number;
  error?: ErrorDetail;
}

interface QueryResult {
  schema?: SchemaField[];
  rows?: unknown[][];
  named_rows?: Record<string, unknown>[];
  row_count: number;
  message?: string;
}
```

Prefer `named_rows` when present. Helper methods `queryOne()/queryAll()` wrap those rows as `RowData`, while `queryRows()` wraps them as FILE-aware `KalamRow<T>` values.

## Subscription and consumer types

- `ServerMessage`
- `SubscriptionOptions`
- `SubscriptionCallback`

The topic worker types moved into `@kalamdb/consumer`:

- `ConsumeRequest`
- `ConsumeResponse`
- `ConsumeMessage`
- `ConsumeContext`
- `ConsumerHandle`
- `AckResponse`

`ServerMessage` from generated declarations includes:

- auth events (`auth_success`, `auth_error`)
- subscription events (`subscription_ack`, `initial_data_batch`, `change`, `error`)

## UserId branded type

`types.ts` defines a branded `UserId`:

```ts
type UserId = string & { readonly __brand: unique symbol };
function UserId(value: string): UserId;
```

Use this in service pipelines when you want stricter domain typing.

## KalamDataType

`KalamDataType` mirrors the server's canonical column type enum and schema
metadata. The SDK supports both scalar and parameterized variants:

| KalamDB type | TypeScript metadata value | Query row wire value |
|---|---|---|
| `BOOLEAN` | `'Boolean'` | boolean |
| `INT` | `'Int'` | number |
| `SMALLINT` | `'SmallInt'` | number |
| `BIGINT` | `'BigInt'` | string for Int64 precision |
| `DOUBLE` | `'Double'` | number |
| `FLOAT` | `'Float'` | number |
| `TEXT` | `'Text'` | string |
| `TIMESTAMP` | `'Timestamp'` | numeric timestamp or ISO string |
| `DATE` | `'Date'` | Date32 day offset or ISO date |
| `DATETIME` | `'DateTime'` | numeric timestamp or ISO string |
| `TIME` | `'Time'` | microseconds since midnight or time string |
| `JSON` | `'Json'` | JSON value |
| `BYTES` | `'Bytes'` | byte-number array |
| `EMBEDDING(n)` | `{ Embedding: n }` | number array |
| `UUID` | `'Uuid'` | UUID string |
| `DECIMAL(p,s)` | `{ Decimal: { precision, scale } }` | exact decimal string |
| `FILE` | `'File'` | FILE reference JSON |

Use `KalamCellValue` accessors for row reads and `@kalamdb/orm` for Drizzle
column generation from this metadata.

## Row helper methods

Use client helpers instead of manual parsing:

```ts
const one = await client.queryOne('SELECT * FROM app.users WHERE id = $1', ['u1']);
const all = await client.queryAll('SELECT * FROM app.users');

// FILE-aware wrappers
const rows = await client.queryRows<{ id: string; avatar: unknown }>(
  'SELECT id, avatar FROM app.users',
  'app.users',
);
const avatar = rows[0]?.file('avatar');
```

## KalamCellValue & RowData

Every cell in a `RowData` row is wrapped in a `KalamCellValue` — a typed
accessor class that mirrors the Rust `KalamCellValue` newtype.

```ts

const row: RowData = wrapRowMap({ id: '1', score: 9.5, active: true });

row.id.asString();    // "1"
row.score.asFloat();  // 9.5
row.active.asBool();  // true
row.avatar.asFile();  // FileRef | null
row.total.asDecimal(); // exact string | null
```

See [Cell Values (KalamCellValue)](https://kalamdb.org/docs/ts-sdk/cell-values) for the full API
including all accessor methods and FILE column support.

## Type-only imports

Prefer `import type` for large type sets in app code:

```ts

```

================================================================================
# WASM Entrypoint
URL: https://kalamdb.org/docs/ts-sdk/wasm
Source: content/ts-sdk/wasm.mdx

> Advanced guide to using @kalamdb/client/wasm for browser-native KalamDB connections. Covers the low-level KalamClient API, bundler configuration, and setup.
# WASM Entrypoint

Most users should consume `@kalamdb/client` high-level APIs. This page covers advanced WASM access.

> Warning: this page documents the low-level generated WASM bindings, not the normal `@kalamdb/client` package API.
> If you want the supported high-level client, use `createClient(...)` from `@kalamdb/client` and follow the package docs instead of constructing `KalamClient` from `@kalamdb/client/wasm` directly.

Recommended high-level docs:

- `/docs/ts-sdk/setup`
- `/docs/ts-sdk/subscriptions`
- `/docs/ts-sdk/client-lifecycle`

## Import low-level bindings

```ts

  KalamClient,
  WasmTimestampFormatter,
  parseIso8601,
  timestampNow,
  initSync,
} from '@kalamdb/client/wasm';
```

## Initialization patterns

```ts
// async init
await init();

// or explicit module/path
await init({ module_or_path: '/wasm/kalam_client_bg.wasm' });
```

`initSync(...)` is available for explicit bytes/module workflows.

## Using `KalamClient` directly

Use this only when you intentionally need the raw WASM interface.

```ts
const client = KalamClient.withJwt('http://localhost:2900', '<jwt-token>');
await client.connect();

const out = await client.query('SELECT 1');
console.log(out);
```

You can also create an anonymous low-level client and supply auth dynamically:

```ts
const client = KalamClient.anonymous('http://localhost:2900');
client.setAuthProvider(async () => ({ jwt: { token: await getJwt() } }));
await client.connect();
```

When using raw WASM methods, some callbacks/events may come as JSON strings requiring manual parse.

## Packaging notes

From SDK package metadata:

- package exports include `./wasm`
- build copies WASM artifacts into `dist/wasm` and `dist/.wasm-out`

Repository builds target a size-first WASM pipeline:

- `wasm-pack build --profile release-dist ...`
- `wasm-opt -Oz --all-features` as a post-build pass when Binaryen is installed

If your bundler fails to serve WASM automatically, ensure `kalam_client_bg.wasm` is accessible and provide explicit module/path.

## Timestamp utilities

- `parseIso8601(isoString)`
- `timestampNow()`
- `WasmTimestampFormatter`

Useful for low-level time conversions aligned with Rust implementation.

================================================================================
# Transport & Endpoints
URL: https://kalamdb.org/docs/rust-sdk/api
Source: content/rust-sdk/api.mdx

> How kalam-client maps to KalamDB HTTP and WebSocket endpoints, with debugging tips for compression, retries, timeouts, and auth flows.
# Transport & Endpoints

`kalam-client` is a high-level wrapper over the same public endpoints described in the server API reference.

## SQL execution (HTTP)

`execute_query(...)` → `POST /v1/api/sql`

- See: [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)
- See: [SQL Reference](https://kalamdb.org/docs/server/sql-reference)

The client sends `params` as JSON for `$1`, `$2`, … placeholders and optionally sets a namespace header for unqualified table names.

Multipart file uploads (`execute_with_files`) use the same SQL endpoint with multipart bodies when `file-uploads` is enabled.

## File downloads (HTTP)

| SDK method | HTTP endpoint |
|------------|---------------|
| `download_bound_file(&BoundFileRef, ...)` | `GET /v1/files/{namespace}/{table}/{sub}/{stored_name}` |
| `download_file(&FileRef, namespace, table, ...)` | same path with explicit namespace/table strings |

Downloads reuse the client's JWT (or basic-auth exchange). Optional `target_user_id` maps to the `user_id` query parameter for cross-user USER-table access by privileged roles.

See [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns).

## Authentication endpoints

Used by the SDK when `auth-flows` is enabled (default via `native-sdk`):

| SDK method | HTTP endpoint |
|------------|---------------|
| `login(user, password)` | `POST /v1/api/auth/login` |
| `refresh_access_token(refresh_token)` | `POST /v1/api/auth/refresh` |

Basic credentials on `AuthProvider::basic_auth` are exchanged through `login` before authenticated SQL or WebSocket traffic.

## Live subscriptions (WebSocket)

`live()` / `live_events()` use the shared WebSocket realtime endpoint.

- See: [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol)
- See: [Architecture: Live Query](https://kalamdb.org/docs/server/architecture/live-query)

Subscriptions multiplex over one connection per `KalamLinkClient`. Call `connect()` to open the socket explicitly, or rely on lazy connect (default) on the first subscription.

### Compression

`ConnectionOptions` controls WebSocket compression. For local debugging you can disable compression so frames are easier to inspect — do not disable in production.

### Message serialization

The native Rust client uses the shared `link-common` protocol. Debug hooks `on_receive` / `on_send` on `EventHandlers` can log wire traffic during development.

## Topic consume (HTTP)

With the `consumer` feature, `TopicConsumer::poll()` and `commit_sync()` call the topic consume/ack HTTP APIs documented under [Topic Pub/Sub](https://kalamdb.org/docs/server/topic-pubsub).

## Diagnostics endpoints (optional features)

| Feature | SDK method | Endpoint |
|---------|------------|----------|
| `healthcheck` | `health_check()` | `GET /v1/api/healthcheck` |
| `cluster` | `cluster_health_check()` | `GET /v1/api/cluster/health` |
| `setup` | `check_setup_status()` | `GET /v1/api/auth/status` |
| `setup` | `server_setup(...)` | `POST /v1/api/auth/setup` |

See [Server Setup & Diagnostics Boundary](https://kalamdb.org/docs/rust-sdk/health-setup).

## Timeouts and retries

Configured on `KalamLinkClient::builder()`:

```rust
let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::jwt_token(token))
    .timeout(Duration::from_secs(10))
    .max_retries(2)
    .build()?;
```

- `timeout` — per HTTP request deadline
- `max_retries` — idempotent query retries
- `timeouts(KalamLinkTimeouts::...)` — granular WebSocket and HTTP phases

## Environment variables (examples and local dev)

| Variable | Typical use |
|----------|-------------|
| `KALAMDB_SERVER_URL` | Base URL in workspace examples |
| `KALAMDB_ROOT_PASSWORD` | `system_user_auth` in examples |

## Next

- [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle)
- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)
- [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol)

================================================================================
# Auth-Aware Client
URL: https://kalamdb.org/docs/rust-sdk/auth-aware-client
Source: content/rust-sdk/auth-aware-client.mdx

> Build a Rust Kalam client that refreshes credentials on connect and reconnect with DynamicAuthProvider and shared client state.
# Auth-Aware Client

If your service uses expiring tokens or loads user context separately from raw auth, your Kalam client should follow the same lifecycle: build once, refresh credentials when auth changes, and start subscriptions only when the session is ready.

This avoids a common failure mode — opening live subscriptions before you know which user or tenant the worker operates as.

## Recommended lifecycle

1. Build a shared `KalamLinkClient` (or `Arc<KalamLinkClient>`) at service startup.
2. Resolve auth with `AuthProvider::jwt_token(...)` or a `DynamicAuthProvider`.
3. Call `connect()` before live work when you want the WebSocket warm.
4. Start `live()` or `consumer()` loops only after credentials and namespace context are known.
5. Call `disconnect()`, close subscriptions/consumers, and drop the client on shutdown.

See also [Authentication](https://kalamdb.org/docs/rust-sdk/auth), [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions), and [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle).

## Dynamic auth provider

Use `DynamicAuthProvider` when credentials must be fetched lazily on every connect or reconnect — OAuth/OIDC flows, secret managers, or rotating refresh tokens.

```rust
use std::{future::Future, pin::Pin, sync::Arc};

use kalam_client::{AuthProvider, DynamicAuthProvider, KalamLinkClient, Result};

struct TokenStore {
    // e.g. Arc<RwLock<Option<String>>>
}

impl DynamicAuthProvider for TokenStore {
    fn get_auth(
        &self,
    ) -> Pin<Box<dyn Future<Output = Result<AuthProvider>> + Send + '_>> {
        Box::pin(async move {
            let token = load_fresh_jwt().await?;
            Ok(AuthProvider::jwt_token(token))
        })
    }
}

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth_provider(Arc::new(TokenStore { /* ... */ }))
    .build()?;
```

The provider is invoked when the client needs fresh credentials for connect or reconnect.

## Shared client in a Tokio service

```rust
use std::sync::Arc;
use kalam_client::{AuthProvider, KalamLinkClient};

async fn run_service() -> Result<(), Box<dyn std::error::Error>> {
    let client = Arc::new(
        KalamLinkClient::builder()
            .base_url("http://localhost:2900")
            .auth_provider(Arc::new(my_token_store))
            .build()?,
    );

    client.connect().await?;

    let worker = {
        let client = Arc::clone(&client);
        tokio::spawn(async move {
            run_live_loop(&client).await
        })
    };

    worker.await??;
    client.disconnect().await;
    Ok(())
}
```

## Refresh without rebuilding the client

When you control token rotation in application code:

```rust
let fresh = client.fresh_auth().await?; // re-runs DynamicAuthProvider if configured
client.update_shared_auth(fresh);
```

Or push a known token:

```rust
client.update_shared_auth(AuthProvider::jwt_token(new_token));
```

Pair with `auth_refresher` on the builder for automatic `TOKEN_EXPIRED` recovery during queries — see [Authentication](https://kalamdb.org/docs/rust-sdk/auth).

## When to call `connect()`

- **HTTP-only workers** — skip `connect()` if you only call `execute_query`.
- **Live-query services** — call `connect()` before the first `live()` / `live_events()` so all subscriptions share one socket.
- **Eager warm-up** — set `ConnectionOptions::default().with_ws_lazy_connect(false)` on the builder if the first subscription must not pay socket setup latency.

## Next

- [Authentication](https://kalamdb.org/docs/rust-sdk/auth)
- [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle)
- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)

================================================================================
# Authentication
URL: https://kalamdb.org/docs/rust-sdk/auth
Source: content/rust-sdk/auth.mdx

> Use AuthProvider for JWT, basic login bootstrap, system-user auth, or anonymous access in kalam-client. Covers login and refresh_access_token.
# Authentication

KalamDB supports user/password bootstrap and JWT bearer tokens. For long-lived services, prefer JWT with refresh or a dynamic auth provider.

## Auth types

```rust
use kalam_client::AuthProvider;

AuthProvider::basic_auth("admin".into(), "kalamdb123".into()); // login bootstrap
AuthProvider::jwt_token("eyJhbGci...".into());               // bearer token
AuthProvider::system_user_auth("kalamdb123".into());         // root system user
AuthProvider::none();                                          // localhost bypass
```

Pass credentials to the builder:

```rust
let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::basic_auth("admin".into(), "kalamdb123".into()))
    .build()?;
```

## Basic auth bootstrap

`AuthProvider::basic_auth(user, password)` does **not** send HTTP Basic on every request. The client exchanges credentials on `POST /v1/api/auth/login` and then uses the returned JWT for SQL, files, topics, and WebSocket traffic.

You never call `apply_to_request` with basic credentials directly — that method returns an error for `BasicAuth` by design.

## JWT auth

When you already have a token:

```rust
let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::jwt_token(access_token))
    .build()?;
```

Or use the builder shortcut:

```rust
KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .jwt_token(access_token)
    .build()?;
```

## Explicit login and refresh

`login` and `refresh_access_token` require the `auth-flows` feature (included in default `native-sdk`).

```rust
let bootstrap = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .build()?;

let login = bootstrap.login("admin", "kalamdb123").await?;

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::jwt_token(login.access_token))
    .build()?;
```

Refresh:

```rust
let refreshed = client.refresh_access_token(&refresh_token).await?;
```

## Update auth on an existing client

```rust
client.set_auth(AuthProvider::jwt_token(new_token));
// or, for shared clones:
client.update_shared_auth(AuthProvider::jwt_token(new_token));
```

For dynamic providers, see [Auth-Aware Client](https://kalamdb.org/docs/rust-sdk/auth-aware-client).

## TOKEN_EXPIRED recovery

Pass an `auth_refresher` callback when building the client to recover from expired tokens on query or subscription paths:

```rust
use std::{future::Future, pin::Pin, sync::Arc};
use kalam_client::{AuthProvider, AuthRefreshCallback, Result};

let refresher: AuthRefreshCallback = Arc::new(|| {
    Box::pin(async {
        let token = fetch_fresh_token().await?;
        Ok(AuthProvider::jwt_token(token))
    }) as Pin<Box<dyn Future<Output = Result<AuthProvider>> + Send>>
});

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::jwt_token(initial_token))
    .auth_refresher(refresher)
    .build()?;
```

## System user convenience

`AuthProvider::system_user_auth(password)` is shorthand for basic auth as user `root`. Common in examples and internal tools:

```rust
let auth = AuthProvider::system_user_auth(
    std::env::var("KALAMDB_ROOT_PASSWORD").unwrap_or_else(|_| "kalamdb123".into()),
);
```

## Next

- [Auth-Aware Client](https://kalamdb.org/docs/rust-sdk/auth-aware-client)
- [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle)
- [Server authentication](https://kalamdb.org/docs/server/getting-started/authentication)

================================================================================
# Cell Values (KalamCellValue)
URL: https://kalamdb.org/docs/rust-sdk/cell-values
Source: content/rust-sdk/cell-values.mdx

> Use KalamCellValue for type-safe access to query and subscription row values in kalam-client, including null handling and numeric precision.
# Cell Values

`KalamCellValue` is the typed wrapper around each cell in query and subscription results.

Instead of parsing raw `serde_json::Value` at every call site, use the accessor methods on `KalamCellValue`.

## Quick example

```rust
use kalam_client::KalamCellValue;

let response = client
    .execute_query("SELECT id, name, score FROM users", None, None, None)
    .await?;

for row in response.rows_as_maps() {
    let id = row.get("id").and_then(|v| v.as_text());
    let name = row.get("name").and_then(|v| v.as_text());
    let score = row.get("score").and_then(|v| v.as_double());
    println!("{id:?} {name:?} {score:?}");
}
```

## Row shapes

| Context | Type |
|---------|------|
| `QueryResponse::rows_as_maps()` | `Vec<HashMap<String, KalamCellValue>>` |
| `QueryResponse::rows()` | `Vec<Vec<KalamCellValue>>` (positional) |
| `LiveRowsEvent::Rows` | `Vec<HashMap<String, KalamCellValue>>` |
| Low-level `ChangeEvent` rows | Named or positional depending on event variant |

Prefer named maps (`rows_as_maps`, live row maps) for application code.

## Creating values

```rust
let cell = KalamCellValue::text("hello");
let cell = KalamCellValue::int(42);
let cell = KalamCellValue::boolean(true);
let cell = KalamCellValue::null();
let cell = KalamCellValue::from(serde_json::json!({"key": "value"}));
```

## Type guards

`KalamCellValue` derefs to `serde_json::Value`, so you can use standard JSON checks:

```rust
cell.inner().is_null();
cell.inner().is_string();
cell.inner().is_number();
```

Prefer the typed accessors below for SQL column semantics.

## Typed accessors

| Method | SQL types |
|--------|-----------|
| `as_text()` | `Text` |
| `as_small_int()` | `SmallInt` |
| `as_int()` | `Int` |
| `as_big_int()` | `BigInt` (handles string-encoded JSON for precision) |
| `as_float()` / `as_double()` | `Float` / `Double` |
| `as_decimal()` | `Decimal` |
| `as_boolean()` | `Boolean` |
| `as_timestamp()` | `Timestamp` (epoch value) |

`BigInt` and large integers are often serialized as JSON strings — accessors handle both numeric and string forms.

## Raw access

```rust
let raw: &serde_json::Value = cell.inner();
let owned: serde_json::Value = cell.clone().into_inner();
```

## Live subscription rows

```rust
if let LiveRowsEvent::Rows { rows, .. } = event? {
    for row in rows {
        let body = row.get("body").and_then(KalamCellValue::as_text);
        println!("{body:?}");
    }
}
```

## FILE columns

FILE values deserialize to structured JSON. Use file-specific helpers instead of parsing raw JSON by hand.

### `.as_file() -> Option<FileRef>`

Parse a FILE column into table-agnostic metadata:

```rust
if let Some(file_ref) = cell.as_file() {
    println!("{} ({})", file_ref.name, file_ref.format_size());
    assert!(file_ref.is_image());
}
```

`FileRef` alone does not include namespace/table — you need those to build download URLs.

### `.as_bound_file(&TableId) -> Option<BoundFileRef>`

Parse and attach table context in one step. Reuse one `TableId` when parsing many rows from the same table:

```rust
use kalam_client::TableId;

let table_id = TableId::from_strings("docs", "files");
let bound = cell.as_bound_file(&table_id).expect("FILE column");

println!("{}", bound.relative_url());
let download = client.download_bound_file(&bound, None).await?;
```

See [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns) for uploads, `BoundFileRef`, and download APIs.

## Accessor summary

| Method | Returns | Use for |
|--------|---------|---------|
| `as_text()` | `Option<&str>` | Text columns |
| `as_int()` / `as_big_int()` | numeric options | Integer columns |
| `as_file()` | `Option<FileRef>` | FILE metadata without table context |
| `as_bound_file(&TableId)` | `Option<BoundFileRef>` | FILE metadata + download context |

## Next

- [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns)
- [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)
- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)
- [Types & Models](https://kalamdb.org/docs/rust-sdk/types)

================================================================================
# Client Lifecycle
URL: https://kalamdb.org/docs/rust-sdk/client-lifecycle
Source: content/rust-sdk/client-lifecycle.mdx

> KalamLinkClient builder options, ws_lazy_connect, connect and disconnect, EventHandlers, and shutdown cleanup in kalam-client.
# Client Lifecycle

## Creating a client

```rust
use std::{sync::Arc, time::Duration};
use kalam_client::{AuthProvider, EventHandlers, KalamLinkClient};

let handlers = EventHandlers::new()
    .on_connect(|| println!("connected"))
    .on_disconnect(|reason| println!("disconnected: {reason}"))
    .on_error(|error| eprintln!("connection error: {}", error.message));

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::jwt_token(token))
    .timeout(Duration::from_secs(30))
    .max_retries(3)
    .event_handlers(handlers)
    .build()?;
```

Notes:

- `build()` returns a ready client handle. HTTP queries work immediately.
- Live subscriptions additionally use the shared WebSocket.
- `base_url` is required; everything else has defaults.

## WebSocket lazy connect (default: `true`)

`ConnectionOptions::ws_lazy_connect` defaults to `true`. The shared WebSocket opens on the first `live()` or `live_events()` call, not at `build()` time.

Eager connect:

```rust
use kalam_client::ConnectionOptions;

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::jwt_token(token))
    .connection_options(ConnectionOptions::default().with_ws_lazy_connect(false))
    .build()?;

client.connect().await?; // opens socket immediately
```

## Explicit connect and disconnect

```rust
client.connect().await?;
assert!(client.is_connected().await);

client.disconnect().await;
```

Call `connect()` before multiple subscriptions so they share one socket and one auth context.

## Auth refresh model

- Static `AuthProvider` on the builder is resolved at build time.
- `DynamicAuthProvider` re-runs on connect/reconnect via `fresh_auth()`.
- `auth_refresher` handles `TOKEN_EXPIRED` during query/subscription operations.
- `set_auth` / `update_shared_auth` push new credentials without rebuilding the client.

See [Authentication](https://kalamdb.org/docs/rust-sdk/auth) and [Auth-Aware Client](https://kalamdb.org/docs/rust-sdk/auth-aware-client).

## Event handlers

`EventHandlers` callbacks:

| Callback | When it fires |
|----------|----------------|
| `on_connect` | Shared WebSocket becomes ready |
| `on_disconnect` | Connection closes (`DisconnectReason`) |
| `on_error` | Connection or protocol errors (`ConnectionError`) |
| `on_receive` / `on_send` | Optional debug hooks for wire messages |

Handlers are `Arc`-backed and safe to share across cloned clients.

## Timeouts and retries

```rust
use kalam_client::KalamLinkTimeouts;

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .timeout(Duration::from_secs(10))
    .max_retries(2)
    .timeouts(KalamLinkTimeouts::builder().receive_timeout(Duration::from_secs(15)).build())
    .build()?;
```

- `timeout` — HTTP request timeout (default 30s)
- `max_retries` — retries for idempotent HTTP queries (default 3)
- `timeouts(...)` — finer-grained send/receive limits for HTTP and WebSocket

## Shutdown checklist

1. `live.close().await?` / `events.close().await?` for each open subscription
2. `consumer.close().await?` for each topic consumer
3. `client.disconnect().await` to close the shared WebSocket

Long-running services should always close streams before dropping the client.

## Cloning and sharing

`KalamLinkClient` is cloneable (`Arc` internally). Clone handles share connection state — useful for spawning multiple subscription tasks from one service.

## Next

- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)
- [Transport & Endpoints](https://kalamdb.org/docs/rust-sdk/api)
- [Auth-Aware Client](https://kalamdb.org/docs/rust-sdk/auth-aware-client)

================================================================================
# File Attachments
URL: https://kalamdb.org/docs/rust-sdk/examples/file-attachments
Source: content/rust-sdk/examples/file-attachments.mdx

> Upload FILE columns with execute_with_files, bind TableId on query rows, and download bytes with download_bound_file in kalam-client.
# File Attachments

End-to-end FILE column workflow: create a table with a `FILE` column, upload bytes with multipart SQL, parse the returned metadata, and download the file through the same client.

**Source:** [`link/sdks/rust/examples/file-attachments/`](https://github.com/kalamdb/KalamDB/tree/main/link/sdks/rust/examples/file-attachments)

Requires the `file-uploads` feature on `kalam-client` (included in the example's `Cargo.toml`).

## What it does

1. Creates a temporary namespace and `documents` table with an `attachment FILE` column
2. Inserts a row using `execute_with_files` and `FILE("attachment")`
3. Selects the row and parses the FILE cell with `as_bound_file(&table_id)`
4. Downloads bytes with `download_bound_file`
5. Verifies the payload matches and drops the namespace

## Run it

```bash
cd link/sdks/rust

cargo run -p file-attachments
```

Expected output (values vary per run):

```text
row id: doc1
file name: note.txt
stored size: 46 bytes
relative url: /v1/files/rust_file_ex_.../documents/f0001/...-note.txt
downloaded: "Hello from the Rust SDK file-attachments example"
content-type: Some("text/plain")
```

## Key APIs

```rust
use kalam_client::{FileUpload, QueryParam, TableId};

let table_id = TableId::from_strings(&namespace, "documents");

let files = vec![
    FileUpload::new("attachment", "note.txt", bytes).with_mime("text/plain"),
];

client
    .execute_with_files(
        r#"INSERT INTO docs.documents (id, attachment)
           VALUES ($1, FILE("attachment"))"#,
        files,
        Some(vec![QueryParam::from("doc1")]),
        None,
    )
    .await?;

let bound = row[attachment_index]
    .as_bound_file(&table_id)
    .expect("FILE column");

let download = client.download_bound_file(&bound, None).await?;
```

Reuse one `TableId` when parsing many rows from the same table. See [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns) for the full API reference.

## Next

- [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns)
- [Quickstart](https://kalamdb.org/docs/rust-sdk/examples/quickstart)
- [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)

================================================================================
# Examples
URL: https://kalamdb.org/docs/rust-sdk/examples
Source: content/rust-sdk/examples/index.mdx

> Runnable kalam-client examples from link/sdks/rust/examples. Quickstart, live inbox, and topic consumer.
# Examples

Runnable Rust projects that ship with the KalamDB repository. Each example is a small Cargo crate you can run against a local server.

**Repository:** [`link/sdks/rust/examples/`](https://github.com/kalamdb/KalamDB/tree/main/link/sdks/rust/examples)

## Prerequisites

- Rust 1.92+
- KalamDB on port 2900
- `KALAMDB_SERVER_URL` (default `http://localhost:2900`)
- `KALAMDB_ROOT_PASSWORD` (default `kalamdb123`)

```bash
cd link/sdks/rust

```

## Example catalog

| Example | What you'll learn | Docs |
|---------|-------------------|------|
| Quickstart | Connect, authenticate, run your first SQL query | [Quickstart](https://kalamdb.org/docs/rust-sdk/examples/quickstart) |
| File Attachments | FILE upload, `TableId`, and `download_bound_file` | [File Attachments](https://kalamdb.org/docs/rust-sdk/examples/file-attachments) |
| Live Inbox | Materialized `live()` row subscriptions | [Live Inbox](https://kalamdb.org/docs/rust-sdk/examples/live-inbox) |
| Topic Consumer | Poll a topic, process records, commit offsets | [Topic Consumer](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer) |

## Which example should I pick?

- **First time with the Rust SDK?** [Quickstart](https://kalamdb.org/docs/rust-sdk/examples/quickstart)
- **Working with FILE columns?** [File Attachments](https://kalamdb.org/docs/rust-sdk/examples/file-attachments)
- **Building a live dashboard or service?** [Live Inbox](https://kalamdb.org/docs/rust-sdk/examples/live-inbox)
- **Background worker on topics?** [Topic Consumer](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer)

Install the crate in your own app: [Setup & Quick Start](https://kalamdb.org/docs/rust-sdk/setup).

================================================================================
# Live Inbox
URL: https://kalamdb.org/docs/rust-sdk/examples/live-inbox
Source: content/rust-sdk/examples/live-inbox.mdx

> Subscribe to materialized live rows in Rust with kalam-client live(). Create a table, watch inserts arrive in realtime.
# Live Inbox

Shows how to use `live()` for materialized row subscriptions — the same pattern used by TypeScript and Dart clients.

**Source:** [`link/sdks/rust/examples/live-inbox/`](https://github.com/kalamdb/KalamDB/tree/main/link/sdks/rust/examples/live-inbox)

## What it does

1. Creates a temporary inbox table
2. Opens a WebSocket and starts a `live()` subscription
3. Receives the initial (empty) row set
4. Inserts a message via SQL
5. Receives the updated row set with the new message
6. Cleans up the table

## Run it

```bash
cd link/sdks/rust

cargo run -p live-inbox
```

Expected output:

```text
initial rows: 0
live rows after insert: 1
first row body: hello from rust sdk
```

## Key API pieces

| Piece | Role |
|-------|------|
| `client.connect()` | Opens the shared WebSocket before subscriptions |
| `SubscriptionConfig` | Names the subscription and sets the live SQL |
| `SubscriptionOptions::with_last_rows(20)` | Rewinds up to 20 recent rows on connect |
| `LiveRowsConfig { limit: Some(20), .. }` | Caps the materialized row set the client keeps |
| `live.next()` | Yields the next `LiveRowsEvent::Rows` batch |

Live SQL must stay in supported form: `SELECT ... FROM ... WHERE ...` (no `ORDER BY` or `LIMIT` in the subscription SQL).

## Core pattern

```rust
use kalam_client::{
    LiveRowsConfig, LiveRowsEvent, SubscriptionConfig, SubscriptionOptions,
};

client.connect().await?;

let mut config = SubscriptionConfig::new("inbox", "SELECT * FROM app.messages");
config.options = Some(SubscriptionOptions::new().with_last_rows(20));

let mut live = client
    .live_with_config(
        config,
        LiveRowsConfig {
            limit: Some(20),
            ..LiveRowsConfig::default()
        },
    )
    .await?;

while let Some(event) = live.next().await {
    if let LiveRowsEvent::Rows { rows, .. } = event? {
        println!("{} rows", rows.len());
    }
}

live.close().await?;
client.disconnect().await;
```

## Next

- [Topic Consumer](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer)
- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)
- [Realtime Ops Feed](https://kalamdb.org/docs/use-cases/realtime-ops-feed) — same pattern in TypeScript

================================================================================
# Quickstart
URL: https://kalamdb.org/docs/rust-sdk/examples/quickstart
Source: content/rust-sdk/examples/quickstart.mdx

> Run your first KalamDB SQL query from Rust with kalam-client. Connect, authenticate, and print the current user.
# Quickstart

The smallest Rust example: connect to KalamDB, authenticate, and run one SQL query.

**Source:** [`link/sdks/rust/examples/quickstart/`](https://github.com/kalamdb/KalamDB/tree/main/link/sdks/rust/examples/quickstart)

## What it does

- Builds a `KalamLinkClient` with a server URL and credentials
- Runs `SELECT CURRENT_USER()`
- Prints the result and disconnects

No WebSocket or subscriptions — just HTTP SQL.

## Run it

```bash
cd link/sdks/rust

cargo run -p quickstart
```

Expected output:

```text
status: Ok
current user row: ...
```

## Source walkthrough

```rust
use std::time::Duration;

use kalam_client::{AuthProvider, KalamLinkClient};

fn server_url() -> String {
    std::env::var("KALAMDB_SERVER_URL").unwrap_or_else(|_| "http://localhost:2900".to_string())
}

fn auth() -> AuthProvider {
    let password =
        std::env::var("KALAMDB_ROOT_PASSWORD").unwrap_or_else(|_| "kalamdb123".to_string());
    AuthProvider::system_user_auth(password)
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = KalamLinkClient::builder()
        .base_url(server_url())
        .auth(auth())
        .timeout(Duration::from_secs(30))
        .build()?;

    let response = client
        .execute_query("SELECT CURRENT_USER()", None, None, None)
        .await?;

    println!("status: {:?}", response.status);
    if let Some(result) = response.results.first() {
        if let Some(rows) = &result.rows {
            for row in rows {
                println!("current user row: {row:?}");
            }
        }
    }

    client.disconnect().await;
    Ok(())
}
```

## Use in your own project

Install from [crates.io](https://crates.io/crates/kalam-client):

```bash
cargo add kalam-client
cargo add tokio --features macros,rt-multi-thread
```

See [Setup & Quick Start](https://kalamdb.org/docs/rust-sdk/setup) for the full install guide.

## Next

- [File Attachments example](https://kalamdb.org/docs/rust-sdk/examples/file-attachments)
- [Live Inbox](https://kalamdb.org/docs/rust-sdk/examples/live-inbox)
- [Topic Consumer](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer)
- [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)

================================================================================
# Topic Consumer
URL: https://kalamdb.org/docs/rust-sdk/examples/topic-consumer
Source: content/rust-sdk/examples/topic-consumer.mdx

> Consume KalamDB topic events in Rust. Create a topic route, poll records, mark processed, and commit offsets.
# Topic Consumer

Shows how to build a background worker that consumes topic events with explicit offset commits.

**Source:** [`link/sdks/rust/examples/topic-consumer/`](https://github.com/kalamdb/KalamDB/tree/main/link/sdks/rust/examples/topic-consumer)

## What it does

1. Creates a table and a topic
2. Routes table inserts to the topic (`ALTER TOPIC ... ADD SOURCE`)
3. Inserts a row (which publishes to the topic)
4. Polls the consumer group for records
5. Marks records processed and commits offsets synchronously

## Run it

```bash
cd link/sdks/rust

cargo run -p topic-consumer
```

Expected output:

```text
polled 1 record(s)
offset=0 payload={"row":{...}}
committed up to offset 0
```

Requires the `consumer` feature (enabled in the example crate).

## Topic routing

```sql
CREATE TOPIC default.my_topic;
ALTER TOPIC default.my_topic ADD SOURCE default.my_table ON INSERT WITH (payload = 'full');
```

## Consumer pattern

```rust
use std::time::Duration;
use kalam_client::AutoOffsetReset;

let mut consumer = client
    .consumer()
    .group_id("my-worker")
    .topic("default.my_topic")
    .auto_offset_reset(AutoOffsetReset::Earliest)
    .enable_auto_commit(false)
    .poll_timeout(Duration::from_secs(5))
    .build()?;

let records = consumer.poll().await?;
for record in &records {
    // process record.payload
    consumer.mark_processed(record);
}
consumer.commit_sync().await?;
consumer.close().await?;
```

Install with the consumer feature:

```bash
cargo add kalam-client --features native-sdk,consumer
```

## Next

- [Summarizer Agent](https://kalamdb.org/docs/use-cases/summarizer-agent) — same worker pattern in TypeScript
- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)
- [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai)

================================================================================
# FILE Columns & Uploads
URL: https://kalamdb.org/docs/rust-sdk/file-columns
Source: content/rust-sdk/file-columns.mdx

> Work with FILE datatype columns in kalam-client. Upload with execute_with_files, parse FileRef metadata, bind table context with TableId, and download bytes with download_bound_file.
# FILE Columns & Uploads

`kalam-client` supports multipart SQL uploads and authenticated downloads for KalamDB `FILE` columns.

Enable uploads with the `file-uploads` feature (included in `native-full`):

```toml
kalam-client = { version = "0.5", features = ["native-sdk", "file-uploads"] }
```

## Upload files with SQL

Use `FILE("placeholder")` in SQL and pass matching `FileUpload` payloads:

```rust
use kalam_client::{FileUpload, QueryParam};

let bytes = tokio::fs::read("./photo.jpg").await?;
let files = vec![
    FileUpload::new("photo", "photo.jpg", bytes).with_mime("image/jpeg"),
];

client
    .execute_with_files(
        r#"INSERT INTO docs.files (id, attachment) VALUES ($1, FILE("photo"))"#,
        files,
        Some(vec![QueryParam::from("doc1")]),
        None,
    )
    .await?;
```

Use `execute_with_files_with_progress(...)` when you need upload progress callbacks.

## Insert + read FileRefs (end-to-end)

```rust
use kalam_client::{FileUpload, QueryParam, TableId};

// 1) Insert a FILE via multipart upload
let files = vec![
    FileUpload::new("photo", "photo.jpg", std::fs::read("photo.jpg")?)
        .with_mime("image/jpeg"),
];
client
    .execute_with_files(
        r#"INSERT INTO docs.files (id, attachment) VALUES ($1, FILE("photo"))"#,
        files,
        Some(vec![QueryParam::from("doc1")]),
        None,
    )
    .await?;

// 2) Read rows and bind table context once
let table_id = TableId::from_strings("docs", "files");

let response = client
    .execute_query(
        "SELECT id, attachment FROM docs.files WHERE id = $1",
        None,
        Some(vec![QueryParam::from("doc1")]),
        None,
    )
    .await?;

let cell = &response.rows().unwrap()[0][1];
let bound = cell.as_bound_file(&table_id).expect("FILE column");

println!("{} ({})", bound.name, bound.format_size());
println!("{}", bound.relative_url());

// 3) Download bytes through the same client
let download = client.download_bound_file(&bound, None).await?;
std::fs::write("downloaded.jpg", &download.bytes)?;
```

For many rows from the same table, reuse one `TableId`:

```rust
let table_id = TableId::from_strings("docs", "files");

for row in response.rows().unwrap() {
    if let Some(bound) = row[1].as_bound_file(&table_id) {
        let download = client.download_bound_file(&bound, None).await?;
        // ...
    }
}
```

## Why table context is separate

FILE column JSON is **table-agnostic** — it stores `id`, `sub`, `name`, `size`, `mime`, `sha256`, and optional `shard`. Namespace and table name are **not** embedded in stored JSON.

Download URLs require table identity:

```text
/v1/files/{namespace}/{table}/{sub}/{stored_name}
```

The SDK binds that context at parse time:

| Type | Role |
|------|------|
| `FileRef` | Wire/storage metadata from the FILE column JSON |
| `TableId` | Type-safe `namespace` + `table` pair from `kalamdb-commons` |
| `FileRefContext` | Wraps a `TableId` for download context |
| `BoundFileRef` | `FileRef` + `FileRefContext` — URLs and downloads need no extra args |

## FileRef model

`FileRef` is re-exported from `kalamdb-commons`. Parse from query cells:

```rust
let file_ref = cell.as_file().expect("FILE column");
assert!(file_ref.is_image());
println!("{}", file_ref.stored_name());
println!("{}", file_ref.relative_path());
```

Build download URLs when you already know namespace and table strings:

```rust
let url = file_ref.download_url("http://localhost:2900", "docs", "files");
let path = file_ref.relative_url("docs", "files");
```

Prefer `BoundFileRef` when table identity is known — see below.

### Fields and helpers

- `id`, `sub`, `name`, `size`, `mime`, `sha256`, optional `shard`
- `stored_name()` — sanitized filename used in HTTP paths
- `relative_path()` — storage layout (`sub/id-name.ext`, shard-aware for shared tables)
- `format_size()`, `is_image()`, `is_pdf()`, `type_description()`, and related MIME helpers

## BoundFileRef and TableId

Attach table context when parsing cells:

```rust
use kalam_client::{BoundFileRef, FileRefContext, TableId};

let table_id = TableId::from_strings("docs", "files");
let bound = cell.as_bound_file(&table_id).expect("FILE column");

// Deref to FileRef fields
println!("{}", bound.name);

// Table identity
assert_eq!(bound.namespace(), "docs");
assert_eq!(bound.table(), "files");
assert_eq!(bound.table_id(), &table_id);

// URLs without repeating namespace/table
let url = bound.download_url(client.base_url());
let path = bound.relative_url();
```

Construct manually when needed:

```rust
let ctx = FileRefContext::from_strings("docs", "files");
// or: FileRefContext::new(table_id)
let bound = BoundFileRef::new(file_ref, ctx);
```

## Download APIs

| Method | When to use |
|--------|-------------|
| `download_bound_file(&BoundFileRef, target_user_id)` | Preferred — context already bound |
| `download_file(&FileRef, namespace, table, target_user_id)` | Legacy explicit namespace/table strings |

Both return `FileDownload`:

- `bytes: Vec<u8>`
- `content_type: Option<String>`
- `content_disposition: Option<String>`

For USER tables, downloads default to the authenticated caller's scope. Pass `Some("user_id")` as `target_user_id` when a DBA or system role needs another user's file (same rules as the HTTP API's `user_id` query parameter).

## Auth behavior

Downloads use the same JWT (or basic-auth exchange) as SQL queries. The client builds the request URL, applies auth headers, and returns the response body as bytes.

## Related server docs

- [FILE Datatype (SQL)](https://kalamdb.org/docs/server/sql-reference/file-datatype)
- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)

## Example

Runnable source: [File Attachments](https://kalamdb.org/docs/rust-sdk/examples/file-attachments) (`cargo run -p file-attachments` from `link/sdks/rust`).

## Next

- [Cell Values (KalamCellValue)](https://kalamdb.org/docs/rust-sdk/cell-values)
- [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)
- [Types & Models](https://kalamdb.org/docs/rust-sdk/types)

================================================================================
# Server Setup & Diagnostics Boundary
URL: https://kalamdb.org/docs/rust-sdk/health-setup
Source: content/rust-sdk/health-setup.mdx

> Optional kalam-client features for health checks, cluster diagnostics, and first-run server setup.
# Server Setup & Diagnostics Boundary

Unlike the Dart SDK, `kalam-client` exposes **optional** server diagnostics and bootstrap helpers behind Cargo features. They are not enabled in the default `native-sdk` install.

## Default app surface

With default features, a typical Rust app uses:

- `execute_query` / `execute_with_files` for SQL
- `live` / `live_events` for realtime
- `login` / `refresh_access_token` for auth (via `auth-flows` in `native-sdk`)
- `client.consumer()` when the `consumer` feature is enabled

Health, setup, and cluster helpers are opt-in.

## Health check (`healthcheck` feature)

```toml
kalam-client = { version = "0.5", features = ["native-sdk", "healthcheck"] }
```

```rust
let health = client.health_check().await?;
println!("{:?}", health);
```

Calls `GET /v1/api/healthcheck`. Responses are cached briefly inside the client to avoid hammering the endpoint.

For a one-off check without the feature, use `curl http://localhost:2900/health` or `execute_query("SELECT 1", ...)`.

## Cluster health (`cluster` feature)

```toml
kalam-client = { version = "0.5", features = ["native-sdk", "cluster"] }
```

```rust
let cluster = client.cluster_health_check().await?;
```

Calls `GET /v1/api/cluster/health` for per-node runtime metrics. Intended for operators and admin tooling — often localhost-restricted.

## First-run setup (`setup` feature)

```toml
kalam-client = { version = "0.5", features = ["native-sdk", "setup"] }
```

```rust
let status = client.check_setup_status().await?;
if status.needs_setup {
    let response = client.server_setup(server_setup_request).await?;
}
```

| Method | Endpoint |
|--------|----------|
| `check_setup_status()` | `GET /v1/api/auth/status` |
| `server_setup(request)` | `POST /v1/api/auth/setup` |

`login` returns `KalamLinkError::SetupRequired` (HTTP 428) when the server still needs initial configuration.

For most deployments, use documented server flows instead of embedding setup in app code:

- [Getting Started: Authentication](https://kalamdb.org/docs/server/getting-started/authentication)
- [Getting Started: Configuration](https://kalamdb.org/docs/server/getting-started/configuration)

## CLI-oriented bundle

`native-full` enables healthcheck, setup, cluster, consumer, and file-uploads together — useful for internal tools, not typical production microservices.

```toml
kalam-client = { version = "0.5", features = ["native-full"] }
```

## What stays outside the SDK

- Database schema design and migrations
- Admin UI operations
- Server process management and deployment

Use the [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference) for protocol details.

## Next

- [Transport & Endpoints](https://kalamdb.org/docs/rust-sdk/api)
- [Setup & Quick Start](https://kalamdb.org/docs/rust-sdk/setup)
- [Server Getting Started](https://kalamdb.org/docs/server/getting-started)

================================================================================
# Rust SDK
URL: https://kalamdb.org/docs/rust-sdk
Source: content/rust-sdk/index.mdx

> Official kalam-client Rust SDK for KalamDB. Install from crates.io, authenticate, run SQL, subscribe to live rows, and consume topics.
# Rust SDK

[`kalam-client`](https://crates.io/crates/kalam-client) is the official KalamDB client for Rust. It wraps the shared `link-common` implementation with a Tokio-native API for services, CLIs, and background workers.

> **Status:** Beta — the API may still change between releases.

| Resource | Link |
|----------|------|
| crates.io | [kalam-client](https://crates.io/crates/kalam-client) |
| API reference | [docs.rs/kalam-client](https://docs.rs/kalam-client) |
| Source | [github.com/kalamdb/KalamDB](https://github.com/kalamdb/KalamDB/tree/main/link/kalam-client) |

**Current version:**  · **MSRV:** Rust 1.92+

## What you can do

`kalam-client` focuses on the core app and worker workflows:

- Execute SQL over HTTP (`execute_query`, `execute_with_files`)
- Upload and download FILE column attachments (`FileRef`, `BoundFileRef`, `download_bound_file`)
- Open materialized live rows over WebSocket (`live`, `live_with_config`)
- Consume low-level subscription frames (`live_events`, `live_events_with_config`)
- Authenticate with JWT, basic login bootstrap, system-user auth, or dynamic providers (`AuthProvider`, `login`, `refresh_access_token`)
- Observe connection lifecycle (`EventHandlers`)
- Inspect active subscriptions (`subscriptions`)
- Control the shared WebSocket explicitly (`connect`, `disconnect`, `is_connected`)
- Poll and commit topic records when the `consumer` feature is enabled (`client.consumer()`)

Optional feature-gated helpers:

- `health_check()` with the `healthcheck` feature
- `check_setup_status()` / `server_setup()` with the `setup` feature
- `cluster_health_check()` with the `cluster` feature

## Installation

```bash
cargo add kalam-client
cargo add tokio --features macros,rt-multi-thread
```

```toml
[dependencies]
kalam-client = "0.5"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
```

Enable optional capabilities as needed:

```toml
# Topic workers
kalam-client = { version = "0.5", features = ["native-sdk", "consumer"] }

# Multipart SQL file uploads
kalam-client = { version = "0.5", features = ["native-sdk", "file-uploads"] }
```

| Feature | Description |
|---------|-------------|
| `native-sdk` (default) | Tokio runtime, HTTP SQL, auth flows, live subscriptions |
| `consumer` | `TopicConsumer`, `ConsumerBuilder`, poll/commit APIs |
| `file-uploads` | `execute_with_files` multipart uploads and FILE downloads |
| `healthcheck` | Cached `health_check()` helper |
| `setup` | `check_setup_status()` and `server_setup()` |
| `cluster` | `cluster_health_check()` |
| `native-full` | All native features (CLI-oriented bundle) |

## Quick start

```rust
use std::time::Duration;
use kalam_client::{AuthProvider, KalamLinkClient};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = KalamLinkClient::builder()
        .base_url("http://localhost:2900")
        .auth(AuthProvider::basic_auth("admin".into(), "kalamdb123".into()))
        .timeout(Duration::from_secs(30))
        .build()?;

    let response = client
        .execute_query("SELECT CURRENT_USER()", None, None, None)
        .await?;

    if response.success() {
        println!("rows: {:?}", response.rows_as_maps());
    }

    client.disconnect().await;
    Ok(())
}
```

Runnable examples: [Examples](https://kalamdb.org/docs/rust-sdk/examples).

For `live()` and `live_events()`, keep SQL strict: `SELECT ... FROM ... WHERE ...`. Do not put `ORDER BY` or `LIMIT` in live subscription SQL.

## Recommended reading order

1. [Setup & Quick Start](https://kalamdb.org/docs/rust-sdk/setup)
2. [Authentication](https://kalamdb.org/docs/rust-sdk/auth)
3. [Auth-Aware Client](https://kalamdb.org/docs/rust-sdk/auth-aware-client)
4. [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)
5. [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns)
6. [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)
7. [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle)
8. [Types & Models](https://kalamdb.org/docs/rust-sdk/types)
9. [Cell Values (KalamCellValue)](https://kalamdb.org/docs/rust-sdk/cell-values)
10. [Server Setup & Diagnostics Boundary](https://kalamdb.org/docs/rust-sdk/health-setup)
11. [Transport & Endpoints](https://kalamdb.org/docs/rust-sdk/api)
12. [Examples](https://kalamdb.org/docs/rust-sdk/examples)

## Examples

Runnable crates ship under `link/sdks/rust/examples/`:

| Example | Docs |
|---------|------|
| Quickstart — first SQL query | [Quickstart](https://kalamdb.org/docs/rust-sdk/examples/quickstart) |
| File Attachments — FILE upload and download | [File Attachments](https://kalamdb.org/docs/rust-sdk/examples/file-attachments) |
| Live Inbox — `live()` subscriptions | [Live Inbox](https://kalamdb.org/docs/rust-sdk/examples/live-inbox) |
| Topic Consumer — poll and commit offsets | [Topic Consumer](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer) |

See [Examples overview](https://kalamdb.org/docs/rust-sdk/examples) for run commands and prerequisites.

## Why `live()` first

Most UIs want the latest reconciled row list, not raw insert/update/delete frames.

- `live()` / `live_with_config()` — materialized rows (recommended default)
- `live_events()` / `live_events_with_config()` — low-level protocol events

Use `SubscriptionOptions::with_last_rows()` to rewind recent history. Use `LiveRowsConfig { limit: Some(n), .. }` to cap the materialized set the client keeps.

## Builder options

| Method | Description |
|--------|-------------|
| `base_url(...)` | Required KalamDB server URL |
| `auth(...)` / `jwt_token(...)` / `auth_provider(...)` | Static or dynamic credentials |
| `timeout(...)` | HTTP request timeout (default 30s) |
| `max_retries(...)` | Retries for idempotent HTTP queries (default 3) |
| `connection_options(...)` | WebSocket lazy connect, compression, keepalive |
| `timeouts(...)` | Fine-grained receive/send timeouts |
| `event_handlers(...)` | Connect/disconnect/error hooks |
| `auth_refresher(...)` | Custom `TOKEN_EXPIRED` recovery callback |

## Notes

- `base_url(...)` is required on every builder.
- Basic auth is exchanged for JWT before protected SQL or WebSocket work.
- HTTP queries work without `connect()`. Call `connect()` before live subscriptions.
- Default `ConnectionOptions` uses `ws_lazy_connect: true` — the socket opens on the first subscription.
- Close live streams and consumers with `close()` during shutdown.

================================================================================
# Querying & DML
URL: https://kalamdb.org/docs/rust-sdk/querying
Source: content/rust-sdk/querying.mdx

> Execute SQL with params and namespaces using kalam-client. Read QueryResponse rows, run DML, and upload files with execute_with_files.
# Querying & DML

`execute_query(...)` runs SQL over the KalamDB HTTP API and returns a `QueryResponse`.

```rust
let response = client
    .execute_query("SELECT 1 AS ok", None, None, None)
    .await?;

println!("success: {}", response.success());
```

HTTP queries do not require `connect()`.

## Parameterized queries

Use `$1`, `$2`, … placeholders and pass `serde_json::Value` parameters:

```rust
let params = vec![
    serde_json::json!("conv_42"),
    serde_json::json!(false),
];

let response = client
    .execute_query(
        "SELECT * FROM app.messages WHERE conversation_id = $1 AND archived = $2",
        None,
        Some(params),
        None,
    )
    .await?;
```

Parameters are JSON-encoded. Use explicit SQL casts when you need richer server-side types.

## Namespaces

Pass the namespace as the fourth argument when SQL uses unqualified table names:

```rust
client
    .execute_query(
        "SELECT * FROM messages ORDER BY created_at DESC LIMIT 10",
        None,
        None,
        Some("chat"),
    )
    .await?;
```

Background: [SQL Reference: Namespaces](https://kalamdb.org/docs/server/sql-reference/namespaces).

## Reading results

### Named rows (recommended)

```rust
if !response.success() {
    if let Some(err) = &response.error {
        eprintln!("{} ({})", err.message, err.code);
    }
    return Ok(());
}

for row in response.rows_as_maps() {
    let id = row.get("id").and_then(|v| v.as_text());
    let body = row.get("body").and_then(|v| v.as_text());
    println!("id={id:?} body={body:?}");
}
```

`rows_as_maps()` returns `Vec<HashMap<String, KalamCellValue>>`.

### Convenience helpers

```rust
response.row_count();
response.first_row_as_map();
response.get_value("column_name");
response.get_i64("big_id_column");
```

### Multiple statements

```rust
let response = client
    .execute_query("SELECT 1; SELECT 2;", None, None, None)
    .await?;

for (i, result) in response.results.iter().enumerate() {
    println!("statement[{i}] rows={}", result.row_count);
}
```

### Positional rows

`response.rows()` returns `Vec<Vec<KalamCellValue>>` when you prefer index-based access.

## DML: INSERT / UPDATE / DELETE

All SQL goes through `execute_query`:

```rust
client
    .execute_query(
        "INSERT INTO app.messages (conversation_id, role, content) VALUES ($1, $2, $3)",
        None,
        Some(vec![
            serde_json::json!("conv_42"),
            serde_json::json!("user"),
            serde_json::json!("hello"),
        ]),
        None,
    )
    .await?;
```

Check `response.success()` and inspect `response.error` on failure.

## File uploads and downloads

Enable the `file-uploads` feature (or `native-full`):

```toml
kalam-client = { version = "0.5", features = ["native-sdk", "file-uploads"] }
```

Upload with `FileUpload` and `execute_with_files`:

```rust
use kalam_client::{FileUpload, QueryParam};

let bytes = tokio::fs::read("./avatar.png").await?;
let files = vec![
    FileUpload::new("avatar", "avatar.png", bytes).with_mime("image/png"),
];

client
    .execute_with_files(
        r#"INSERT INTO app.users (name, avatar) VALUES ($1, FILE("avatar"))"#,
        files,
        Some(vec![QueryParam::from("Alice")]),
        None,
    )
    .await?;
```

After querying FILE columns, bind table context with `TableId` and download bytes:

```rust
use kalam_client::TableId;

let table_id = TableId::from_strings("app", "users");
let bound = cell.as_bound_file(&table_id).expect("FILE column");
let download = client.download_bound_file(&bound, None).await?;
```

See [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns) for the full `FileRef`, `BoundFileRef`, and download API.

## Progress callbacks

`execute_query_with_progress(...)` reports query execution progress for long-running statements when the server emits progress frames.

## Next

- [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns)
- [Cell Values (KalamCellValue)](https://kalamdb.org/docs/rust-sdk/cell-values)
- [Types & Models](https://kalamdb.org/docs/rust-sdk/types)
- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)

================================================================================
# Setup & Quick Start
URL: https://kalamdb.org/docs/rust-sdk/setup
Source: content/rust-sdk/setup.mdx

> Install kalam-client from crates.io, build a KalamLinkClient, run your first SQL query, and try a live subscription.
# Setup & Quick Start

> **Status:** Beta — the API may still change between releases.

The Rust SDK is the [`kalam-client`](https://crates.io/crates/kalam-client) crate on crates.io. One crate covers SQL, live rows, auth, optional file uploads, and optional topic consumers behind Cargo features.

## Install

```bash
cargo add kalam-client
cargo add tokio --features macros,rt-multi-thread
```

**MSRV:** Rust 1.92+

You need a running KalamDB server. Quick Docker start:

```bash
docker run -d --name kalamdb -p 2900:2900 \
  -e KALAMDB_SERVER_HOST=0.0.0.0 \
  -e KALAMDB_ROOT_PASSWORD=kalamdb123 \
  -e KALAMDB_JWT_SECRET=local-dev-secret-please-change-32chars \
  -v kalamdb_data:/data jamals86/kalamdb:latest
```

Verify: `curl http://127.0.0.1:2900/health`

## Create a client

`KalamLinkClient::builder()` requires `base_url`. Prefer `AuthProvider::basic_auth(...)` or `AuthProvider::jwt_token(...)` for normal apps.

```rust
use std::time::Duration;
use kalam_client::{AuthProvider, KalamLinkClient};

let client = KalamLinkClient::builder()
    .base_url("http://localhost:2900")
    .auth(AuthProvider::basic_auth("admin".into(), "kalamdb123".into()))
    .timeout(Duration::from_secs(30))
    .build()?;
```

For local tooling against a root password:

```rust
let auth = AuthProvider::system_user_auth(
    std::env::var("KALAMDB_ROOT_PASSWORD").unwrap_or_else(|_| "kalamdb123".into()),
);
```

`system_user_auth` logs in as the `root` system user.

If auth is disabled locally, you can omit `.auth(...)` and use `AuthProvider::none()`.

By default, `ConnectionOptions::ws_lazy_connect` is `true`, so the WebSocket opens on the first live subscription rather than at build time.

## First query

```rust
let response = client
    .execute_query("SELECT CURRENT_USER()", None, None, None)
    .await?;

if !response.success() {
    if let Some(err) = &response.error {
        eprintln!("query failed: {} ({})", err.message, err.code);
    }
    return Ok(());
}

for row in response.rows_as_maps() {
    println!("{row:?}");
}
```

`QueryResponse` can contain multiple `QueryResult` values when you send multiple statements. `rows_as_maps()` reads the first result set.

Rows are `HashMap<String, KalamCellValue>`. Use typed accessors such as `as_text()` instead of assuming raw JSON shapes.

## First live stream

Call `connect()` before subscriptions. Prefer `live()` when you want materialized rows.

```rust
use kalam_client::{
    LiveRowsConfig, LiveRowsEvent, SubscriptionConfig, SubscriptionOptions,
};

client.connect().await?;

let mut config = SubscriptionConfig::new(
    "inbox",
    "SELECT * FROM app.messages WHERE room = 'main'",
);
config.options = Some(SubscriptionOptions::new().with_last_rows(20));

let mut live = client
    .live_with_config(
        config,
        LiveRowsConfig {
            limit: Some(20),
            ..LiveRowsConfig::default()
        },
    )
    .await?;

while let Some(event) = live.next().await {
    if let LiveRowsEvent::Rows { rows, .. } = event? {
        println!("{} rows", rows.len());
        break;
    }
}

live.close().await?;
```

Live SQL must be `SELECT ... FROM ... WHERE ...` only.

## Cleanup

```rust
client.disconnect().await;
```

## Runnable examples

From the KalamDB repository:

```bash
cd link/sdks/rust

cargo run -p quickstart
cargo run -p live-inbox
cargo run -p topic-consumer   # requires consumer feature in the example crate
```

Runnable walkthroughs: [Examples](https://kalamdb.org/docs/rust-sdk/examples) — [Quickstart](https://kalamdb.org/docs/rust-sdk/examples/quickstart), [Live Inbox](https://kalamdb.org/docs/rust-sdk/examples/live-inbox), [Topic Consumer](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer).

## Develop from source

```toml
kalam-client = { path = "../KalamDB/link/kalam-client", features = ["native-sdk", "consumer"] }
```

## Next

- [Authentication](https://kalamdb.org/docs/rust-sdk/auth)
- [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)
- [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns)
- [Realtime Subscriptions](https://kalamdb.org/docs/rust-sdk/subscriptions)
- [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle)

================================================================================
# Realtime Subscriptions
URL: https://kalamdb.org/docs/rust-sdk/subscriptions
Source: content/rust-sdk/subscriptions.mdx

> Use live(), live_with_config(), and live_events() for realtime Rust apps with kalam-client.
# Realtime Subscriptions

The Rust SDK supports live queries over a shared WebSocket connection.

Ownership boundary:

- `link-common` owns the shared connection, reconnect loop, replay filtering, and `SeqId` checkpoint tracking.
- Your app owns the async loop over `live.next()` or `events.next()`.

Live SQL must follow the strict supported shape: `SELECT ... FROM ... WHERE ...`.

Do not include `ORDER BY` or `LIMIT` in live subscription SQL. Apply ordering or row caps in application code after rows arrive, or use `LiveRowsConfig::limit`.

Call `client.connect().await?` before opening subscriptions so they multiplex over one socket (unless you rely on lazy connect on first subscription).

## Which API to use

| API | Use when |
|-----|----------|
| `live()` / `live_with_config()` | You want the current materialized row list (recommended for UI and most services) |
| `live_events()` / `live_events_with_config()` | You need raw `ChangeEvent` frames for custom reconciliation or debugging |

Unlike `execute_query`, live APIs take the final SQL string directly — there is no separate params argument. Embed constants in the `WHERE` clause or use separate subscriptions per filter.

## Materialized live rows

```rust
use kalam_client::{
    LiveRowsConfig, LiveRowsEvent, SubscriptionConfig, SubscriptionOptions,
};

client.connect().await?;

let mut config = SubscriptionConfig::new(
    "inbox",
    "SELECT id, room, body FROM app.messages WHERE room = 'main'",
);
config.options = Some(
    SubscriptionOptions::new()
        .with_last_rows(20)
        .with_batch_size(50),
);

let mut live = client
    .live_with_config(
        config,
        LiveRowsConfig {
            limit: Some(20),
            ..LiveRowsConfig::default()
        },
    )
    .await?;

while let Some(event) = live.next().await {
    match event? {
        LiveRowsEvent::Rows { rows, last_seq_id, .. } => {
            println!("{} rows, last_seq_id={last_seq_id:?}", rows.len());
            for row in rows {
                println!("{}", row.get("body").and_then(|v| v.as_text()).unwrap_or(""));
            }
        }
        LiveRowsEvent::Error { code, message, .. } => {
            eprintln!("live error {code}: {message}");
            break;
        }
    }
}

live.close().await?;
```

Shorthand when defaults are enough:

```rust
let mut live = client.live("SELECT * FROM app.tasks WHERE status = 'open'").await?;
```

### Control knobs

| Knob | Layer | Effect |
|------|-------|--------|
| `with_batch_size(n)` | Server snapshot | Chunks initial rows from the server |
| `with_last_rows(n)` | Server rewind | How much history to fetch on connect |
| `LiveRowsConfig { limit: Some(n), .. }` | Client materializer | Caps rows kept in memory after reconciliation |
| `with_from(seq_id)` | Resume | Continue from a persisted `SeqId` |

```rust
use kalam_client::{SeqId, SubscriptionOptions};

let options = SubscriptionOptions::new()
    .with_last_rows(200)
    .with_from(SeqId::from(42_i64));
```

Each `LiveRowsEvent::Rows` includes `last_seq_id` for durable checkpoints.

## Low-level change events

```rust
use kalam_client::{ChangeEvent, SubscriptionConfig};

let config = SubscriptionConfig::new(
    "raw-inbox",
    "SELECT * FROM app.messages WHERE room = 'main'",
);

let mut events = client.live_events_with_config(config).await?;

while let Some(change) = events.next().await {
    match change? {
        ChangeEvent::Insert { rows, .. } => println!("inserted {}", rows.len()),
        ChangeEvent::Update { rows, .. } => println!("updated {}", rows.len()),
        ChangeEvent::Delete { old_rows, .. } => println!("deleted {}", old_rows.len()),
        other => println!("event: {:?}", other),
    }
}

events.close().await?;
```

## Inspect active subscriptions

```rust
let subs = client.subscriptions().await;
for info in subs {
    println!("{} — {}", info.id, info.query);
}
```

Cancel one explicitly:

```rust
client.cancel_subscription("subscription-id").await?;
```

## Topic consumers (optional)

Topic workers are behind the `consumer` feature — not a separate crate.

```toml
kalam-client = { version = "0.5", features = ["native-sdk", "consumer"] }
```

```rust
use std::time::Duration;
use kalam_client::AutoOffsetReset;

let mut consumer = client
    .consumer()
    .group_id("billing-workers")
    .topic("orders.events")
    .auto_offset_reset(AutoOffsetReset::Earliest)
    .enable_auto_commit(false)
    .poll_timeout(Duration::from_secs(10))
    .build()?;

let records = consumer.poll().await?;
for record in &records {
    // process record.payload
    consumer.mark_processed(record);
}
consumer.commit_sync().await?;
consumer.close().await?;
```

Full walkthrough: [Topic Consumer example](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer).

## Next

- [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle)
- [Cell Values (KalamCellValue)](https://kalamdb.org/docs/rust-sdk/cell-values)
- [Live Inbox example](https://kalamdb.org/docs/rust-sdk/examples/live-inbox)

================================================================================
# Types & Models
URL: https://kalamdb.org/docs/rust-sdk/types
Source: content/rust-sdk/types.mdx

> Reference key kalam-client models including QueryResponse, QueryResult, LoginResponse, ChangeEvent, and EventHandlers.
# Types & Models

This page documents the most important public types re-exported by `kalam-client` from `link-common`.

## Query models

### `QueryResponse`

Returned by `execute_query`, `execute_with_files`, and related SQL helpers.

- `status: ResponseStatus` — `Success` or `Error`
- `results: Vec<QueryResult>` — one per executed statement
- `took: Option<f64>` — execution time in milliseconds
- `error: Option<ErrorDetail>` — populated when `status` is error

Helpers:

- `success()` → `bool`
- `is_token_expired()` → `bool`
- `rows_as_maps()` → first result as `Vec<HashMap<String, KalamCellValue>>`
- `rows()` → first result as positional `Vec<Vec<KalamCellValue>>`
- `row_count()`, `first_row_as_map()`, `get_value(...)`, `get_i64(...)`

### `QueryResult`

- `schema` — column metadata (`SchemaField`)
- `rows: Option<Vec<Vec<KalamCellValue>>>` — positional row storage
- `row_count: usize`
- `message: Option<String>`

`row_as_map(index)` builds a named row map for one index.

### `SchemaField`

Column metadata with flag helpers for primary key, nullability, and uniqueness.

## Authentication models

### `LoginResponse`

Returned by `login` and `refresh_access_token` (requires `auth-flows`):

- `access_token: String`
- `refresh_token: Option<String>`
- `expires_at` / `refresh_expires_at` — server-provided timestamps
- `user` — `LoginUserInfo`

### `AuthProvider`

Enum: `BasicAuth`, `JwtToken`, `None`.

Construct with `basic_auth`, `jwt_token`, `system_user_auth`, or `none`.

### `ResolvedAuth`

Internal resolved state: static provider or dynamic `ArcDynAuthProvider`.

## Subscription models

### `SubscriptionConfig`

- `id` — subscription name
- `query` — live SQL string
- `options: Option<SubscriptionOptions>` — `last_rows`, `batch_size`, `from` (`SeqId`)

### `SubscriptionOptions`

Builder-style helpers: `with_last_rows`, `with_batch_size`, `with_from`.

### `LiveRowsConfig`

Client-side materializer settings — primarily `limit: Option<usize>`.

### `LiveRowsEvent`

- `Rows { rows, last_seq_id, .. }` — materialized snapshot
- `Error { code, message, .. }` — subscription failure

### `ChangeEvent`

Low-level protocol events from `live_events()`:

- `Insert`, `Update`, `Delete` with row payloads
- Additional control/ack variants for protocol tooling

### `SubscriptionInfo`

Returned by `client.subscriptions()` — active subscription id and query text.

## Consumer models (feature `consumer`)

- `ConsumerBuilder` — fluent builder from `client.consumer()`
- `TopicConsumer` — `poll()`, `mark_processed()`, `commit_sync()`, `close()`
- `ConsumerConfig`, `AutoOffsetReset`
- `ConsumeMessage`, `ConsumeRequest`, `ConsumeResponse`, `AckResponse`

## Connection models

### `EventHandlers`

Lifecycle callbacks — see [Client Lifecycle](https://kalamdb.org/docs/rust-sdk/client-lifecycle).

### `ConnectionOptions`

HTTP/WebSocket behavior including `ws_lazy_connect`, compression, keepalive, and `HttpVersion`.

### `KalamLinkTimeouts`

Receive/send timeouts for HTTP and WebSocket operations.

## Error type

`KalamLinkError` — library error enum. `Result<T>` is `std::result::Result<T, KalamLinkError>`.

Common variants: configuration errors, authentication errors, server HTTP errors, subscription errors, `SetupRequired` (HTTP 428 on login).

## FILE column models

### `FileRef`

Table-agnostic metadata stored in FILE column JSON (from `kalamdb-commons`):

- `id`, `sub`, `name`, `size`, `mime`, `sha256`, optional `shard`
- `stored_name()`, `relative_path()`, `format_size()`, MIME type helpers
- `download_url(base_url, namespace, table)` and `relative_url(namespace, table)` when namespace/table are known as strings

Parse from cells with `KalamCellValue::as_file()`.

### `TableId`

Type-safe namespace + table identity (from `kalamdb-commons`, re-exported by `kalam-client`):

```rust
let table_id = TableId::from_strings("docs", "files");
let ns = table_id.namespace_id().as_str();
let table = table_id.table_name().as_str();
let qualified = table_id.full_name(); // "docs.files"
```

Use one `TableId` when binding many FILE cells from the same table.

### `FileRefContext`

Wraps a `TableId` for download context. Construct with `FileRefContext::new(table_id)` or `FileRefContext::from_strings(namespace, table)`.

### `BoundFileRef`

`FileRef` plus table context. Derefs to `FileRef`.

- `table_id()`, `namespace()`, `table()`
- `download_url(base_url)`, `relative_url()` — no extra namespace/table args
- `file_ref()` — borrow underlying `FileRef`

Parse from cells with `KalamCellValue::as_bound_file(&table_id)`.

### `FileUpload`

Multipart payload for `FILE("placeholder")` SQL — `FileUpload::new(placeholder, filename, bytes)` plus optional `with_mime`.

### `FileDownload`

Response from `download_bound_file()` / `download_file()`:

- `bytes`, `content_type`, `content_disposition`

## IDs

- `SeqId` — subscription resume cursor (from `kalamdb_commons`)
- `TableId` — namespace + table composite key
- `UserId`, `Role` — re-exported domain types

## Next

- [FILE Columns & Uploads](https://kalamdb.org/docs/rust-sdk/file-columns)
- [Cell Values (KalamCellValue)](https://kalamdb.org/docs/rust-sdk/cell-values)
- [Querying & DML](https://kalamdb.org/docs/rust-sdk/querying)
- [docs.rs/kalam-client](https://docs.rs/kalam-client)

================================================================================
# Transport & Endpoints
URL: https://kalamdb.org/docs/dart-sdk/api
Source: content/dart-sdk/api.mdx

> How kalam_link maps to KalamDB HTTP and WebSocket endpoints, with practical debugging tips for compression, retries, timeouts, and auth flows in Dart/Flutter.
# Transport & Endpoints

The Dart SDK is a high-level wrapper over the same public endpoints described in the API reference.

## SQL execution (HTTP)

`query(...)` executes SQL over the HTTP SQL endpoint.

- See: [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)
- See: [SQL Reference](https://kalamdb.org/docs/server/sql-reference)

The SDK passes `params` as JSON (for `$1`, `$2`, … placeholders).

Related auth endpoints used by the SDK:

- `login(...)` → `POST /v1/api/auth/login`
- `refreshToken(...)` → `POST /v1/api/auth/refresh`

FILE download helpers use the shared KalamDB file endpoint:

| Method | Endpoint | Dart helpers |
|---|---|---|
| `GET` | `/v1/files/{namespace}/{table}/{sub}/{stored_name}` | `KalamFileRef.getDownloadUrl(...)`, `KalamCellValue.asFileUrl(...)` |

The last path segment is the stored filename generated by the shared
`link-common` `FileRef` model. It is not the raw file id by itself.

For user tables, these helpers generate owner-scoped URLs. A regular user can
download only their own files; `service` tokens can write through authorized
`EXECUTE AS USER` flows, but direct cross-user FILE byte downloads require a
`dba` or `system` token with an authorized `user_id` query parameter.

## Live subscriptions (WebSocket)

`subscribe(...)` uses the WebSocket realtime endpoint.

- See: [WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol)
- See: [Architecture: Live Query](https://kalamdb.org/docs/server/architecture/live-query)

On native Dart and Flutter targets, the shared subscription connection defaults
to MessagePack serialization. Browser/WASM behavior is separate and may differ
by release.

### Debugging compression

By default, the server may send gzip-compressed frames. For local debugging, disable compression:

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  disableCompression: true,
);
```

This appends `?compress=false` to the WS URL so you can more easily inspect frames.

> Do not enable in production. Compression reduces bandwidth and improves performance.

## Timeouts and retries

- `timeout` controls HTTP request timeout for queries.
- `maxRetries` controls retry count for idempotent queries (typically `SELECT`).

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  timeout: const Duration(seconds: 10),
  maxRetries: 2,
);
```

## Next

- [Setup & Quick Start](https://kalamdb.org/docs/dart-sdk/setup)
- [Authentication](https://kalamdb.org/docs/dart-sdk/auth)

================================================================================
# Auth-Aware Client
URL: https://kalamdb.org/docs/dart-sdk/auth-aware-client
Source: content/dart-sdk/auth-aware-client.mdx

> Build a Flutter Kalam client that reacts cleanly to auth and user-session changes with Riverpod, without blocking app startup or realtime sync.
# Auth-Aware Kalam Client for Flutter

If your Flutter app uses expiring tokens and loads app state separately from raw auth,
your Kalam client should follow the same lifecycle. Initialize the runtime once, connect
when auth is ready, and only start subscriptions after the app user/session is ready.

This pattern keeps the SDK aligned with real app state and avoids a common failure mode:
starting realtime work before your app knows which user it is operating as.

## Recommended lifecycle

Use this sequence in production apps:

1. Call `KalamClient.init()` once before `runApp()`.
2. Prime a shared client controller early so the SDK is ready to connect.
3. Connect or refresh auth when the login state or JWT changes.
4. Start subscriptions only after your app user/session has loaded.
5. Dispose the client on logout or app shutdown.

This is especially useful when your app has two readiness stages:

- Authentication is complete, so you can mint a JWT.
- App session data is complete, so you know which subscriptions to start.

See also [Authentication](https://kalamdb.org/docs/dart-sdk/auth), [Realtime Subscriptions](https://kalamdb.org/docs/dart-sdk/subscriptions), and [Client Lifecycle](https://kalamdb.org/docs/dart-sdk/client-lifecycle).

## Why not connect everything at startup?

`KalamClient.init()` belongs at startup. Full client connection usually does not.

Even with `wsLazyConnect: true`, a `connect()` call can still perform auth-related async
work. If you await that before first render, you add boot latency. If you start
subscriptions before your app user exists, you risk wiring realtime state to an
incomplete session.

The safer pattern is:

- initialize the SDK before `runApp()`
- keep one shared client/controller alive while the app is open
- react to auth and app-session changes with explicit lifecycle methods

## Riverpod example

The example below uses Riverpod to split those stages cleanly.

```dart

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await KalamClient.init();
  runApp(const ProviderScope(child: MyApp()));
}

class AppUser {
  const AppUser({required this.id});

  final String id;
}

final authTokenProvider = StreamProvider<String?>((ref) {
  return authRepository.idTokenChanges();
});

final appUserProvider = StreamProvider<AppUser?>((ref) {
  return userRepository.sessionChanges();
});

final kalamControllerProvider = Provider<KalamController>((ref) {
  final controller = KalamController(url: 'https://db.example.com');
  ref.onDispose(controller.dispose);
  return controller;
});

final kalamLifecycleProvider = Provider<void>((ref) {
  final controller = ref.watch(kalamControllerProvider);

  ref.listen<AsyncValue<String?>>(authTokenProvider, (_, next) {
    unawaited(controller.syncAuth(next.valueOrNull));
  });

  ref.listen<AsyncValue<AppUser?>>(appUserProvider, (_, next) {
    final user = next.valueOrNull;
    if (user == null) {
      unawaited(controller.stopRealtime());
      return;
    }

    unawaited(controller.ensureRealtimeReady(userId: user.id));
  });
});

class KalamController {
  KalamController({required this.url});

  final String url;
  KalamClient? _client;
  String? _currentToken;
  String? _subscribedUserId;
  StreamSubscription<ChangeEvent>? _messagesSubscription;

  Future<void> syncAuth(String? token) async {
    if (token == _currentToken) {
      return;
    }

    _currentToken = token;

    if (token == null) {
      _subscribedUserId = null;
      await _messagesSubscription?.cancel();
      _messagesSubscription = null;
      await _client?.dispose();
      _client = null;
      return;
    }

    if (_client == null) {
      _client = await KalamClient.connect(
        url: url,
        wsLazyConnect: true,
        authProvider: () async => Auth.jwt(_currentToken!),
      );
      return;
    }

    await _client!.refreshAuth();
  }

  Future<void> ensureRealtimeReady({required String userId}) async {
    final client = _client;
    if (client == null || _subscribedUserId == userId) {
      return;
    }

    await _messagesSubscription?.cancel();
    _subscribedUserId = userId;

    _messagesSubscription = client
        .subscribe(
          "SELECT id, body, sent_at FROM messages WHERE recipient_id = '$userId'",
        )
        .listen((event) {
          switch (event) {
            case InsertEvent(:final row):
              debugPrint('new row: $row');
            default:
              break;
          }
        });
  }

  Future<void> stopRealtime() async {
    _subscribedUserId = null;
    await _messagesSubscription?.cancel();
    _messagesSubscription = null;
  }

  Future<void> dispose() async {
    await _messagesSubscription?.cancel();
    await _client?.dispose();
  }
}
```

The current `subscribe(...)` API takes a SQL string directly, not a separate
`params` list, so build the final live-query SQL from trusted app state before
starting the subscription.

## Boot the lifecycle once

Create the controller once and let it react to auth/session changes for the lifetime of
the app.

```dart
class MyApp extends ConsumerWidget {
  const MyApp({super.key});

  @override
  Widget build(BuildContext context, WidgetRef ref) {
    ref.watch(kalamLifecycleProvider);

    return const Directionality(
      textDirection: TextDirection.ltr,
      child: Placeholder(),
    );
  }
}
```

This keeps the SDK lifecycle close to your app shell instead of scattering connect logic
across screens.

## Token refresh and auth changes

In most apps, your token can change after the client has already been created. When that
happens:

- return the latest token from `authProvider`
- call `refreshAuth()` after the token changes
- dispose the client entirely on logout

That gives the SDK one stable client instance while still letting credentials rotate.

```dart
await controller.syncAuth(freshToken);
```

Inside the controller above, the first authenticated state creates the client and later
token changes call `refreshAuth()`.

## Common mistakes

- Awaiting `KalamClient.connect()` before the first frame.
- Creating a new client in every screen or feature provider.
- Starting subscriptions before your app user/session has loaded.
- Keeping subscriptions alive after logout.
- Treating raw auth state and app session readiness as the same signal.

## Where the SDK can reduce boilerplate further

This pattern works today, but it also shows where the Dart SDK could get more ergonomic:

1. A small auth-aware client controller in the SDK that handles `connect`, `refreshAuth`, and `dispose` around token changes.
2. A first-class `setAuth()` API so apps do not have to model token replacement indirectly.
3. A lightweight cookbook package or example app using Riverpod for auth-driven lifecycle wiring.
4. A subscription group helper that can start and stop a set of realtime queries as app session readiness changes.

If you are contributing to the SDK, these are high-leverage places to remove repetitive app code without hiding how Kalam auth and subscriptions work.

================================================================================
# Authentication
URL: https://kalamdb.org/docs/dart-sdk/auth
Source: content/dart-sdk/auth.mdx

> Use authProvider for JWT, user/password bootstrap, or anonymous auth in the Dart/Flutter SDK. Covers login, refreshToken, and refreshAuth rotation patterns.
# Authentication

KalamDB supports user/password bootstrap and JWT bearer tokens. For long-lived apps, prefer JWT + refresh flows.

## Auth types

```dart

Auth.basic('user', 'password');     // User/password bootstrap
Auth.jwt('eyJhbGci...');            // JWT bearer token
Auth.none();                         // No auth (localhost bypass)
```

---

## `authProvider` — recommended

Use `authProvider` as the source of credentials. The callback is invoked
when the client is created, and you can re-run it with `refreshAuth()` when
your token changes or before an explicit reconnect path:

```dart
final client = await KalamClient.connect(
  url: 'https://db.example.com',
  authProvider: () async {
    final token = await myApp.getOrRefreshJwt();
    return Auth.jwt(token);
  },
);
```

The `AuthProvider` typedef is:

```dart
typedef AuthProvider = Future<Auth> Function();
```

### Provider return guidance

Return `Auth.jwt(...)` for normal deployments.

The Dart SDK also supports:

- `Auth.basic(...)` when you want the client to exchange user/password credentials for a JWT on `POST /v1/api/auth/login` before the first query or WebSocket connect
- `Auth.none()` for local anonymous access

There is no separate `auth:` parameter on `KalamClient.connect(...)` in the Dart SDK. Auth flows go through `authProvider`, `login(...)`, and `refreshToken(...)`.

### `refreshAuth()`

Call `refreshAuth()` to proactively push fresh credentials to the Rust layer
without waiting for the next reconnect — useful for scheduled token rotation:

```dart
// Refresh tokens every 55 minutes
Timer.periodic(const Duration(minutes: 55), (_) => client.refreshAuth());
```

`refreshAuth()` re-runs the configured `authProvider` and pushes the refreshed credentials into the Rust client.

---

## `disableCompression`

Disable gzip compression on WebSocket messages for easier local debugging:

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getToken()),
  disableCompression: true, // appends ?compress=false to the WS URL
);
```

The server will respond with plain-text JSON frames instead of compressed
binary frames, making inspection in Wireshark or proxy tools straightforward.

> **Do not use in production.** Compression significantly reduces bandwidth.
> Only enable for local debugging.

---

## Login (user/password → JWT upgrade)

If your server requires an explicit login step, create a temporary client,
perform `login()`, then reconnect with the returned access token:

```dart
final bootstrap = await KalamClient.connect(
  url: 'https://db.example.com',
  authProvider: () async => Auth.none(),
);

final session = await bootstrap.login('alice', 'Secret123!');
await bootstrap.dispose();

final client = await KalamClient.connect(
  url: 'https://db.example.com',
  authProvider: () async => Auth.jwt(session.accessToken),
);
```

If you return `Auth.basic(...)` from `authProvider`, the SDK will also perform the login exchange automatically before the first query or WebSocket connection.

## Refresh token

If you have a refresh token and want a new access token:

```dart
final fresh = await client.refreshToken(refreshToken);
print('new access token: ${fresh.accessToken}');
```

================================================================================
# Cell Values (KalamCellValue)
URL: https://kalamdb.org/docs/dart-sdk/cell-values
Source: content/dart-sdk/cell-values.mdx

> Learn how to use KalamCellValue for type-safe access to query and subscription row values in the KalamDB Dart SDK, including file URLs and null handling.
# Cell Values

`KalamCellValue` is the typed wrapper around each individual cell value returned
in a Dart SDK query result or subscription event.

Instead of accessing raw `dynamic` values from positional rows, you use
typed accessor methods to extract values safely.

## Quick Example

```dart

final response = await client.query('SELECT id, name, score, avatar FROM users');

// Typed row maps: Map<String, KalamCellValue>
for (final row in response.rows) {
  final id     = row['id']?.asString();      // String?
  final name   = row['name']?.asString();    // String?
  final score  = row['score']?.asDouble();   // double?
  final born   = row['born_at']?.asDate();   // DateTime?
  final url    = row['avatar']?.asFileUrl(
    'http://localhost:18080', 'default', 'users',
  );                                          // String?
  print('$id $name $score $url');
}
```

## Creating a `KalamCellValue`

### `KalamCellValue.from(raw)`

Wrap any raw dynamic value. Pass `null` for SQL NULL.

```dart
final cell = KalamCellValue.from(someRawValue);
```

## Accessing Typed Rows

### `QueryResult.rows`

Returns `List<Map<String, KalamCellValue>>` — each row as a map with typed cell
values.

```dart
for (final row in result.rows) {
  print(row['email']?.asString());
}
```

### Convenience on `QueryResponse`

```dart
final response = await client.query('SELECT * FROM orders');
final rows = response.rows; // List<Map<String, KalamCellValue>>
```

## Type Guards

| Getter | Type | Description |
|---|---|---|
| `cell.isNull` | `bool` | `true` if this cell is SQL NULL |
| `cell.isString` | `bool` | `true` if underlying value is a `String` |
| `cell.isNumber` | `bool` | `true` if underlying value is a `num` |
| `cell.isBool` | `bool` | `true` if underlying value is a `bool` |
| `cell.isObject` | `bool` | `true` if underlying value is a `Map` |
| `cell.isArray` | `bool` | `true` if underlying value is a `List` |

## Typed Accessors

### `.asString(): String?`

Returns the value as a `String`, or `null` for SQL NULL.  
Handles `{"Utf8": "..."}` Rust-side string envelopes automatically.

```dart
final name = row['name']?.asString(); // "Alice"
```

### `.asInt(): int?`

Returns the value as an `int`, or `null`.  
Truncates doubles. Parses string-encoded integers.

```dart
final count = row['item_count']?.asInt(); // 7
```

### `.asDouble(): double?`

Returns the value as a `double`, or `null`.  
Parses string-encoded floats.

```dart
final price = row['price']?.asDouble(); // 19.99
```

### `.asBool(): bool?`

Returns the value as a `bool`, or `null`.  
Handles string-encoded booleans: `"true"`, `"false"`, `"1"`, `"0"`.

```dart
final active = row['is_active']?.asBool(); // true
```

### `.asDate(): DateTime?`

Returns the value as a `DateTime`, or `null` for SQL NULL / unparseable values.

Handles:
- Unix milliseconds (`int`)
- ISO 8601 strings (`"2024-01-01T00:00:00Z"`)
- Numeric timestamp strings (`"1704067200000"`)

```dart
final created = row['created_at']?.asDate();
print(created?.toLocal());
```

### `.asObject(): Map<String, dynamic>?`

Returns the value as a `Map<String, dynamic>`, or `null`.

### `.asArray(): List<dynamic>?`

Returns the value as a `List<dynamic>`, or `null`.

## FILE Column Support

### `.asFile(): KalamFileRef?`

Parse a FILE column value and return a `KalamFileRef`, or `null`.

```dart
final ref = row['attachment']?.asFile();
if (ref != null) {
  print('${ref.name} (${ref.formatSize()})');
  print(ref.isImage()); // true for image/* MIME types
}
```

See the [FILE Columns](https://kalamdb.org/docs/dart-sdk/querying#file-columns) section for the full
`KalamFileRef` API.

### `.asFileUrl(baseUrl, namespace, table): String?`

Convenience method — parse a FILE column **and** return the download URL.

```dart
final url = row['avatar']?.asFileUrl(
  'http://localhost:18080',
  'default',
  'users',
);
// e.g. "http://localhost:18080/v1/files/default/users/f0001/12345-photo.png"
```

The generated URL uses KalamDB's stored filename path and should be fetched
with the caller's bearer token.

## Serialisation

### `.asJson(): dynamic`

Returns the underlying raw JSON value. This is the preferred JSON accessor.

### `.toJson(): dynamic`

Compatibility alias of `.asJson()`.

Returns the underlying raw dynamic value. Use this when you need to pass the
value to code that expects plain JSON data.

### `.toString(): String`

Human-readable display:
- SQL NULL → `"NULL"`
- `String` → value as-is
- everything else → `_raw.toString()`

## Using with Live Events

Live event streams (`InsertEvent`, `UpdateEvent`, `DeleteEvent`,
`InitialDataBatch`) expose rows as typed maps:

```dart
client.liveEvents('SELECT * FROM orders').listen((event) {
  switch (event) {
    case InsertEvent():
      final name = event.row['customer']?.asString();
      final total = event.row['total']?.asDouble();
      print('New order: $name \$$total');

    case UpdateEvent():
      final newName = event.row['customer']?.asString();
      final oldName = event.oldRow?['customer']?.asString();
      print('Order updated: $oldName → $newName');

    case DeleteEvent():
      print('Deleted: ${event.row['id']?.asInt()}');

    case InitialDataBatch():
      for (final row in event.rows) {
        print(row['customer']?.asString());
      }

    default:
      break;
  }
});
```

### Available typed accessors on ChangeEvent subclasses

| Class | Property | Type |
|---|---|---|
| `InitialDataBatch` | `.rows` | `List<Map<String, KalamCellValue>>` |
| `InsertEvent` | `.rows` | `List<Map<String, KalamCellValue>>` |
| `InsertEvent` | `.row` | `Map<String, KalamCellValue>` |
| `UpdateEvent` | `.rows` | `List<Map<String, KalamCellValue>>` |
| `UpdateEvent` | `.row` | `Map<String, KalamCellValue>` |
| `UpdateEvent` | `.oldRows` | `List<Map<String, KalamCellValue>>` |
| `UpdateEvent` | `.oldRow` | `Map<String, KalamCellValue>?` |
| `DeleteEvent` | `.oldRows` | `List<Map<String, KalamCellValue>>` |
| `DeleteEvent` | `.row` | `Map<String, KalamCellValue>` |

## Full API Reference

| Member | Signature | Description |
|---|---|---|
| `KalamCellValue.from` | `(dynamic) → KalamCellValue` | Wrap a raw value |
| `asJson` | `() → dynamic` | Raw underlying value (preferred) |
| `toJson` | `() → dynamic` | Raw underlying value |
| `isNull` | `bool` getter | True if SQL NULL |
| `isString` | `bool` getter | True if String |
| `isNumber` | `bool` getter | True if num |
| `isBool` | `bool` getter | True if bool |
| `isObject` | `bool` getter | True if Map |
| `isArray` | `bool` getter | True if List |
| `asString` | `() → String?` | Value as String |
| `asInt` | `() → int?` | Value as int |
| `asDouble` | `() → double?` | Value as double |
| `asBool` | `() → bool?` | Value as bool |
| `asDate` | `() → DateTime?` | Value as DateTime |
| `asObject` | `() → Map<String, dynamic>?` | Value as Map |
| `asArray` | `() → List<dynamic>?` | Value as List |
| `asFile` | `() → KalamFileRef?` | FILE column as KalamFileRef |
| `asFileUrl` | `(baseUrl, namespace, table) → String?` | FILE column URL |
| `toString` | `() → String` | Display string |

================================================================================
# Client Lifecycle
URL: https://kalamdb.org/docs/dart-sdk/client-lifecycle
Source: content/dart-sdk/client-lifecycle.mdx

> Learn KalamClient.init(), connect() factory, wsLazyConnect, auth refresh with authProvider, connectionHandlers lifecycle events, and safe cleanup with dispose() in Dart/Flutter.
# Client Lifecycle

## One-time initialization

Call `KalamClient.init()` once at app startup. It initializes the underlying Rust runtime.

```dart
await KalamClient.init();
```

Do not call `KalamClient.connect(...)` before `init()`.

## Creating a client

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  timeout: const Duration(seconds: 30),
  maxRetries: 3,
  disableCompression: false,
  logLevel: Level.debug,
  logListener: (entry) => debugPrint(entry.toString()),
);
```

Notes:

- `connect(...)` returns a ready-to-use handle. The SDK manages the WebSocket
  connection lifecycle automatically.
- Most methods use HTTP; subscriptions additionally use WebSocket under the hood.
- With `wsLazyConnect: true` (the default), `connect()` does not open the shared WebSocket immediately.
- On native Dart targets, the shared subscription connection defaults to MessagePack serialization for lower overhead on first subscription and live updates.

### `wsLazyConnect` (default: `true`)

By default, the WebSocket connection is **lazy**: it is deferred until the
first `subscribe()` call. This avoids unnecessary connections when the client
is only used for HTTP queries.

Set `wsLazyConnect: false` to establish the connection eagerly:

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  wsLazyConnect: false, // connect WebSocket immediately
);
```

### Auth refresh model

- `authProvider` is the connect-time auth API.
- `connect()` resolves auth once up front, and the first lazy `subscribe()` reuses those already-resolved credentials.
- Use `refreshAuth()` to proactively re-run `authProvider` and push updated credentials into the client.
- Return `Auth.basic(...)` if you want the SDK to perform a Basic-to-JWT exchange before first use.

## Refreshing auth in place

If you provided an `authProvider`, you can re-fetch and push updated credentials into the client:

```dart
await client.refreshAuth();

Timer.periodic(const Duration(minutes: 55), (_) => client.refreshAuth());
```

With the default `Auth.none()` provider, `refreshAuth()` simply re-resolves anonymous auth.

## Manual WebSocket controls

The client also exposes a few explicit shared-socket controls:

- `isConnected` returns whether the shared WebSocket is currently open.
- `disconnectWebSocket()` closes the shared WebSocket.
- `reconnectWebSocket()` first tries a fast reconnect with the current auth already held in memory. If the server rejects that reconnect for auth reasons, the SDK refreshes auth once and retries.

```dart
final connected = await client.isConnected;
if (!connected) {
  await client.reconnectWebSocket();
}
```

### Mobile live-connection pattern

For Flutter apps with live screens, keep one `KalamClient` alive for the whole signed-in session and reconnect it when the app returns to the foreground.

```dart
class LiveClientController with WidgetsBindingObserver {
  final KalamClient client;

  LiveClientController(this.client);

  @override
  void didChangeAppLifecycleState(AppLifecycleState state) {
    if (state == AppLifecycleState.resumed) {
      client.reconnectWebSocket();
    }
  }
}
```

That avoids waiting for a stale socket to be detected passively and is the fastest path back to a live subscription after resume.

## Connection handlers

Use `connectionHandlers` to receive lifecycle events (connect/disconnect/errors) and optional raw message debugging.

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  connectionHandlers: ConnectionHandlers(
    onConnect: () => print('connected'),
    onDisconnect: (reason) => print('disconnected: ${reason.message}'),
    onError: (error) => print('error: ${error.message}'),
    onReceive: (message) => print('[recv] $message'),
    onSend: (message) => print('[send] $message'),
  ),
);
```

Handlers are only active when you pass at least one callback.

Connection events are buffered on the native side while Dart pulls them. The buffer is bounded to protect mobile memory; lifecycle events keep the newest signals when the buffer is full, and high-volume `onReceive` / `onSend` debug messages may be dropped under sustained load. Use `onReceive` / `onSend` only for local debugging; avoid logging sensitive data (tokens, PII) in production.

## SDK logging level and redirection

`KalamClient.connect(...)` accepts:

- `logLevel: Level?` (from `package:logger/logger.dart`)
- `logListener: LogListener?`

Example:

```dart

final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  logLevel: Level.info,
  logListener: (entry) {
    myLogger.i('[${entry.tag}] ${entry.message}');
  },
);
```

If no `logListener` is provided, logs are emitted with `print(...)` and appear in `flutter run`/Logcat.

## Disposing

Call `dispose()` when the client is no longer needed:

```dart
await client.dispose();
```

After disposing, do not call `query()` or `subscribe()` on the same instance.

If you keep a single client for the whole app lifetime, dispose it only when the app is shutting down.

## Next

- [Health & Diagnostics](https://kalamdb.org/docs/dart-sdk/health-setup)
- [Authentication](https://kalamdb.org/docs/dart-sdk/auth)

================================================================================
# Server Setup & Diagnostics Boundary
URL: https://kalamdb.org/docs/dart-sdk/health-setup
Source: content/dart-sdk/health-setup.mdx

> Understand which server setup and diagnostics flows are intentionally outside the kalam_link Dart/Flutter SDK surface.
# Server Setup & Diagnostics Boundary

The current Dart SDK intentionally focuses on app-facing query, auth, and live-subscription flows.

It does **not** expose a dedicated `healthCheck()` helper, topic consumer APIs, or initial server bootstrap helpers.

## How to verify server status instead

Use one of the existing server-facing tools instead:

- the Admin UI / SQL Studio
- the documented HTTP API endpoints
- the server CLI and deployment flows
- your app's normal authenticated query path (`SELECT 1`, namespace reads, etc.)

## Initial server setup

The current Dart SDK does not expose `checkSetupStatus()` or `serverSetup()` helpers.

For first-time server bootstrap, use one of the documented server-side flows instead:

- [Getting Started: Authentication](https://kalamdb.org/docs/server/getting-started/authentication)
- [Getting Started: Configuration](https://kalamdb.org/docs/server/getting-started/configuration)
- [HTTP API Reference](https://kalamdb.org/docs/server/api/http-reference)

## Next

- [Authentication](https://kalamdb.org/docs/dart-sdk/auth)
- [Getting Started: Authentication](https://kalamdb.org/docs/server/getting-started/authentication)
- [Getting Started: Configuration](https://kalamdb.org/docs/server/getting-started/configuration)

================================================================================
# Dart / Flutter SDK
URL: https://kalamdb.org/docs/dart-sdk
Source: content/dart-sdk/index.mdx

> Official kalam_link Dart/Flutter SDK for KalamDB. Learn setup, auth, typed querying, realtime subscriptions, and lifecycle hooks.
# Dart / Flutter SDK

`kalam_link` is the official KalamDB client for Dart and Flutter. It wraps the
compiled Rust core via Flutter Rust Bridge v2, giving you native-speed queries
and live subscriptions with a simple Dart API.

> **Status:** This SDK is in **beta** and may still change between releases.

## What you can do

`kalam_link` currently focuses on the core app-client workflows:

- Execute SQL over HTTP (`query`)
- Open any `SELECT` query live over WebSocket (`liveEvents`)
- Consume reconciled materialized live rows (`live`, `liveTable`)
- Authenticate with JWT, user/password bootstrap, or anonymous access (`authProvider`, `login`, `refreshToken`, `refreshAuth`)
- Observe connection lifecycle (`connectionHandlers`)
- Inspect active subscriptions (`getSubscriptions`)
- Control the shared WebSocket explicitly (`isConnected`, `disconnectWebSocket`, `reconnectWebSocket`)

The current Dart SDK does not expose Topic consumer / ACK worker APIs.

## Installation

Add to your `pubspec.yaml`:

Then run:

```bash
flutter pub get
# or
dart pub get
```

## Initialization

Call `KalamClient.init()` once at app startup before creating any clients:

```dart
void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await KalamClient.init();
  runApp(const MyApp());
}
```

## Quick start

```dart

Future<void> run() async {
  final client = await KalamClient.connect(
    url: 'https://db.example.com',
    authProvider: () async {
      final token = await myApp.getOrRefreshJwt();
      return Auth.jwt(token);
    },
    logLevel: Level.debug,
    logListener: (entry) => debugPrint(entry.toString()),
  );

  final result = await client.query('SELECT id, email FROM users LIMIT 10');
  for (final row in result.rows) {
    print('${row['id']?.asString()} ${row['email']?.asString()}');
  }

  // Low-level live events
  final stream = client.liveEvents("SELECT * FROM support.messages WHERE room = 'main'");
  await for (final event in stream) {
    switch (event) {
      case InsertEvent(:final row):
        print('New message: $row');
      default:
        break;
    }
  }

  // Materialized live rows
  final rowsStream = client.liveTable<Map<String, KalamCellValue>>('app.tasks');
  await for (final rows in rowsStream) {
    print('rows=${rows.length}');
    break;
  }

  await client.dispose();
}
```

For `liveEvents()` and materialized live-query APIs, keep the SQL strict: `SELECT ... FROM ... WHERE ...`. Do not include `ORDER BY` or `LIMIT` inside the live SQL.

## Recommended reading order

1. [Setup & Quick Start](https://kalamdb.org/docs/dart-sdk/setup)
2. [Authentication](https://kalamdb.org/docs/dart-sdk/auth)
3. [Auth-Aware Client](https://kalamdb.org/docs/dart-sdk/auth-aware-client)
4. [Querying & DML](https://kalamdb.org/docs/dart-sdk/querying)
5. [Realtime Subscriptions](https://kalamdb.org/docs/dart-sdk/subscriptions)
6. [Client Lifecycle](https://kalamdb.org/docs/dart-sdk/client-lifecycle)
7. [Types & Models](https://kalamdb.org/docs/dart-sdk/types)
8. [Server Setup & Diagnostics Boundary](https://kalamdb.org/docs/dart-sdk/health-setup)

## Supported platforms

This package is a Flutter FFI plugin (via Flutter Rust Bridge) and is configured for:

- Android, iOS
- macOS, Windows, Linux

## Notes

- Call `KalamClient.init()` exactly once at startup.
- Prefer `authProvider` for expiring tokens.
- Cancel live streams by cancelling your `StreamSubscription`.
- Use `liveEvents()` for realtime SQL change streams.
- Prefer `live()` or `liveTable()` when your UI wants the latest reconciled row list.
- Use `getSubscriptions()` and `SeqId` if you need resume-aware client bookkeeping.
- Native Dart subscriptions use MessagePack by default on the shared Rust transport.

## Connection options

| Parameter | Type | Default | Description |
|---|---|---|---|
| `url` | `String` | required | Server URL |
| `authProvider` | `AuthProvider` | `Auth.none()` | Async callback for fresh credentials (recommended) |
| `disableCompression` | `bool` | `false` | Disable WS compression — dev only |
| `wsLazyConnect` | `bool` | `true` | Defer WebSocket connect until the first live stream |
| `timeout` | `Duration` | 30s | HTTP request timeout |
| `maxRetries` | `int` | `3` | Retry count for idempotent queries |
| `connectionHandlers` | `ConnectionHandlers?` | — | Lifecycle event callbacks |
| `keepaliveInterval` | `Duration?` | server default | WS keepalive ping interval |
| `logLevel` | `Level?` | `Level.warning` | SDK logging threshold (`logger` package level) |
| `logListener` | `LogListener?` | — | Redirect SDK logs to your own logger/sink |
| `authProviderMaxAttempts` | `int` | `3` | Retry attempts for transient auth callback failures |
| `authProviderInitialBackoff` | `Duration` | `250ms` | Initial auth retry backoff |
| `authProviderMaxBackoff` | `Duration` | `2s` | Maximum auth retry backoff |

## SDK logging

The Dart SDK uses `Level` from `package:logger/logger.dart` for log severity.

```dart

final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwt()),
  logLevel: Level.debug,
  logListener: (entry) {
    debugPrint('[${entry.tag}] ${entry.message}');
  },
);
```

When `logListener` is not set, the SDK falls back to `print(...)`, which is visible in Flutter/Android consoles.

================================================================================
# Querying & DML
URL: https://kalamdb.org/docs/dart-sdk/querying
Source: content/dart-sdk/querying.mdx

> Execute SQL with params and namespaces using kalam_link for Dart/Flutter. Read rows from QueryResult.rows, handle multi-statement responses, and work with FILE refs.
# Querying & DML

`KalamClient.query(...)` executes SQL over the KalamDB HTTP API and returns a `QueryResponse`.

```dart
final response = await client.query('SELECT 1');
print(response.success);
```

## Parameterized queries

Use `$1`, `$2`, … placeholders in SQL and pass `params` as a Dart list.

```dart
final res = await client.query(
  r'SELECT * FROM app.messages WHERE conversation_id = $1 AND is_deleted = $2',
  params: ['conv_42', false],
);
```

`params` is encoded to JSON and sent to the server, so values should be JSON-compatible (strings, numbers, booleans, null, lists, and maps).

If you need richer server-side typing (timestamps, decimals, JSON columns), keep your SQL explicit (e.g. `CAST(...)`) and validate the returned values.

## Namespaces

KalamDB supports per-tenant namespaces. The SDK exposes this as an optional `namespace` argument.

```dart
final res = await client.query(
  'SELECT * FROM messages ORDER BY created_at DESC LIMIT 10',
  namespace: 'alice',
);
```

This is most useful when your SQL references unqualified tables (e.g. `messages` instead of `alice.messages`).

For background, see: [SQL Reference: Namespaces](https://kalamdb.org/docs/server/sql-reference/namespaces).

## Reading results

A `QueryResponse` can contain multiple `QueryResult` values (one per statement). The most common pattern is to read the first one.

### Primary row access (`QueryResult.rows`)

```dart
if (!res.success) {
  print(res.error);
  return;
}

final first = res.results.first;
print(first.columns.map((c) => c.name).toList());

for (final row in first.rows) {
  final id = row['id']?.asInt();
  final name = row['name']?.asString();
  final active = row['active']?.asBool();
  print('id=$id name=$name active=$active');
}
```

`rows` is `List<Map<String, KalamCellValue>>` (named columns), not `List<List<dynamic>>`.

### Multiple statements

```dart
final res = await client.query('SELECT 1; SELECT 2;');

for (final (i, r) in res.results.indexed) {
  print('statement[$i] columns=${r.columns.length} rows=${r.rowCount}');
}
```

### DML: INSERT / UPDATE / DELETE

All SQL goes through `query(...)`:

```dart
await client.query(
  r"INSERT INTO app.messages (conversation_id, role, content) VALUES ($1, $2, $3)",
  params: [42, 'user', 'hello'],
);

await client.query(
  r"UPDATE app.messages SET content = $1 WHERE message_id = $2",
  params: ['edited', 'msg_123'],
);

await client.query(
  r"DELETE FROM app.messages WHERE message_id = $1",
  params: ['msg_123'],
);
```

## FILE refs: insert and read

### Insert a FILE reference value

If you already have file metadata (for example from a previous read), insert it as JSON:

```dart
final ref = KalamFileRef(
  id: '1234567890123456789',
  sub: 'f0001',
  name: 'avatar.png',
  size: 20480,
  mime: 'image/png',
  sha256: 'abc123...',
);

await client.query(
  r'INSERT INTO app.users (id, avatar) VALUES ($1, $2)',
  params: ['user_1', ref.toMap()],
);
```

### Read a FILE reference

```dart
final res = await client.query('SELECT id, avatar FROM app.users WHERE id = $1', params: ['user_1']);
final row = res.rows.first;

final fileRef = row['avatar']?.asFile();
if (fileRef != null) {
  final url = fileRef.getDownloadUrl('http://localhost:18080', 'app', 'users');
  print('${fileRef.name} ${fileRef.formatSize()} $url');
}
```

`getDownloadUrl(...)` delegates to the shared Rust `link-common` `FileRef`
model, the same source used by the TypeScript SDK. The returned path is:

```text
/v1/files/{namespace}/{table}/{sub}/{stored_name}
```

Fetch the URL with the authenticated user's bearer token. For user tables, SDK
helpers generate an owner-scoped URL: regular users can download only their own
files. Service clients can upload/write through authorized `EXECUTE AS USER`
flows, but direct cross-user FILE byte downloads require a `dba` or `system`
token and an authorized `user_id` query parameter.

## Error handling

When `success` is `false`, the server returns an error code/message:

```dart
final res = await client.query('SELECT * FROM table_that_does_not_exist');
if (!res.success) {
  throw StateError('${res.error}');
}
```

For user-facing apps, prefer mapping `error.code` to a stable UI message and logging `error.details` for debugging.

## Next

- [Realtime Subscriptions](https://kalamdb.org/docs/dart-sdk/subscriptions)
- [Types & Models](https://kalamdb.org/docs/dart-sdk/types)
- [SQL Reference](https://kalamdb.org/docs/server/sql-reference)

================================================================================
# Setup & Quick Start
URL: https://kalamdb.org/docs/dart-sdk/setup
Source: content/dart-sdk/setup.mdx

> Install the kalam_link Dart/Flutter SDK, initialize the Rust runtime once, then connect, run SQL queries with params, and start realtime subscriptions.
# Setup & Quick Start

> **Status:** This SDK is in **beta** and may still change between releases.

The Dart/Flutter SDK is `kalam_link` (pub package name: `kalam_link`). It is a thin Dart layer over a Rust core bridged via Flutter Rust Bridge. The current package surface covers init, auth, querying, subscriptions, and connection diagnostics. Topic consumer / ACK worker APIs, initial server bootstrap flows, and dedicated health-check helpers stay outside the Dart SDK.

## Install

Add to `pubspec.yaml`:

Then fetch dependencies:

```bash
flutter pub get
# or
dart pub get
```

Supported platforms (per plugin configuration): Android, iOS, macOS, Windows, Linux.

> If you’re evaluating web builds: check the current package release notes first. Flutter FFI plugins may require additional setup for web/WASM targets.
> Repository web builds use the `release-dist` Cargo profile plus a `wasm-opt -Oz` post-pass to keep the shipped WASM artifact small.

## Required initialization

You must call `KalamClient.init()` exactly once at app startup, before any other SDK call.

Flutter:

```dart

Future<void> main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await KalamClient.init();
  // runApp(const MyApp());
}
```

Pure Dart (non-Flutter):

```dart

Future<void> main() async {
  await KalamClient.init();
}
```

## Create a client

`KalamClient.connect(...)` constructs a client handle. Prefer `authProvider` for expiring tokens.

```dart

final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwtFromSecureStorage()),
);
```

If you do not have auth enabled locally, you can omit auth entirely:

```dart
final client = await KalamClient.connect(url: 'http://localhost:2900');
```

By default, `connect()` uses `wsLazyConnect: true`, so the shared WebSocket is opened on the first `subscribe()` call instead of immediately.

On native Dart and Flutter targets, subscription traffic also defaults to MessagePack over the shared Rust transport. That keeps the connect-to-first-batch path leaner than JSON without changing the cross-SDK Rust default.

For mobile apps that are live-query heavy, prefer:

```dart
final client = await KalamClient.connect(
  url: 'http://localhost:2900',
  authProvider: () async => Auth.jwt(await getJwtFromSecureStorage()),
  wsLazyConnect: false,
);
```

That warms the shared WebSocket during `connect()` instead of making the first live screen pay the socket setup cost.

## First query

```dart
final result = await client.query('SELECT CURRENT_USER()');

if (!result.success) {
  throw StateError('Query failed: ${result.error}');
}

print(result.rows.first);
```

`QueryResponse` also includes timing metadata (`tookMs`) and can contain multiple result sets when you send multiple statements.

Rows are typed maps: `Map<String, KalamCellValue>`. Use accessors like `asString()`, `asInt()`, and `asFile()` instead of assuming positional arrays.

## First live event stream

`liveEvents(...)` returns a `Stream<ChangeEvent>`.

```dart
final stream = client.liveEvents("SELECT * FROM app.messages WHERE room = 'main'");

final sub = stream.listen((event) {
  switch (event) {
    case AckEvent(:final subscriptionId):
      print('Opened: $subscriptionId');
    case InsertEvent(:final row):
      print('New row: $row');
    case SubscriptionError(:final code, :final message):
      print('Error [$code]: $message');
    default:
      break;
  }
});

// Later
await sub.cancel();
```

For `subscribe()` and materialized live rows, use only `SELECT ... FROM ... WHERE ...`. Apply ordering or row capping in your app after receiving the rows, and use `lastRows` when you need rewind behavior.

If you are targeting browser/WASM builds, verify the current release notes first. The native MessagePack default described here applies to the Rust-backed Dart client path.

## Cleanup

```dart
await client.dispose();
```

## Next

- [Authentication](https://kalamdb.org/docs/dart-sdk/auth)
- [Querying & DML](https://kalamdb.org/docs/dart-sdk/querying)
- [Realtime Subscriptions](https://kalamdb.org/docs/dart-sdk/subscriptions)
- [Client Lifecycle](https://kalamdb.org/docs/dart-sdk/client-lifecycle)

================================================================================
# Realtime Subscriptions
URL: https://kalamdb.org/docs/dart-sdk/subscriptions
Source: content/dart-sdk/subscriptions.mdx

> Use live(), liveTable(), and liveEvents() for realtime Dart/Flutter data with kalam_link.
# Realtime Subscriptions

The Dart/Flutter SDK supports live queries over WebSocket.

The important ownership boundary is:

- Rust owns the shared connection, reconnect loop, replay filtering, and resume checkpoint tracking.
- Dart owns only the stream wrapper and typed row decoding.

That keeps reconnect behavior consistent with other SDKs and avoids re-implementing replay logic in Flutter apps.

On native Dart targets, subscription traffic uses MessagePack by default on that shared Rust-managed connection, which trims wire size and decode overhead compared with JSON.

Live SQL must follow the strict supported shape: `SELECT ... FROM ... WHERE ...`.
Do not include `ORDER BY` or `LIMIT` in `liveEvents()` or materialized live-query SQL.

Use:

```dart
Stream<ChangeEvent> liveEvents(
  String sql, {
  int? batchSize,
  int? lastRows,
  SeqId? from,
  String? subscriptionId,
})
```

For low-level event handling, `liveEvents(...)` is the right API.

Unlike `query(...)`, the current live-query APIs take the final SQL string
directly and do not expose a separate `params` argument.

The Dart SDK also exposes materialized live-row helpers when you want the SDK
to keep the current row set reconciled for you:

- `live<T>(sql, ...)`
- `liveTable<T>(tableName, ...)`

## Which API to use

Choose the API based on the shape your app actually needs:

- Use `liveEvents(...)` when you need low-level protocol events like `AckEvent`, batched snapshots, insert/update/delete diffs, or exact resume control in your own event loop.
- Use `live(...)` when your UI wants the latest materialized row list for an arbitrary SQL query.
- Use `liveTable(...)` when `SELECT * FROM namespace.table` is enough.

For most widget state and list rendering, prefer the materialized helpers. They remove the need to reconcile `InitialDataBatch`, `InsertEvent`, `UpdateEvent`, and `DeleteEvent` yourself.

## Materialized live rows

`live<T>(...)` keeps the current row set updated inside the SDK
and emits the latest list whenever the query result changes.

```dart
final rowsStream = client.live<Map<String, KalamCellValue>>(
  "SELECT id, room, body, created_at FROM app.messages WHERE room = 'main'",
  lastRows: 20,
  limit: 20,
  onCheckpoint: (checkpoint) {
    persistSeqId(checkpoint.lastSeqId.toString());
  },
);

await for (final rows in rowsStream) {
  for (final row in rows) {
    print('${row['id']?.asInt()} ${row['body']?.asString()}');
  }
}
```

Use `liveTable<T>(...)` when `SELECT * FROM table` is enough:

```dart
final taskRows = client.liveTable<Map<String, KalamCellValue>>('app.tasks');

await for (final rows in taskRows) {
  print('tasks: ${rows.length}');
}
```

If your live query uses a natural or composite key instead of a plain `id`
column, pass `keyColumns` so reconciliation still stays inside the shared Rust
core:

```dart
final messages = client.live<Map<String, KalamCellValue>>(
  "SELECT room_id, message_id, body FROM app.messages WHERE room_id = 'main'",
  keyColumns: ['room_id', 'message_id'],
);
```

You can also cap the in-memory materialized row set with `limit`, map rows
directly into app models with `mapRow`, and persist a durable resume cursor with
`onCheckpoint`.

### Materialized rows vs change events

- `liveEvents(...)` gives you protocol-level events and incremental changes.
- `live(...)` and `liveTable(...)` give you the fully reconciled current row list.
- Materialized helpers are usually the better default for UI code.
- Low-level events are better when you need custom reconciliation, audit logging, or protocol inspection.

### Materialized-row limits

The SQL for materialized live rows still follows the same strict supported shape:

- `SELECT ... FROM ... WHERE ...`
- no `ORDER BY`
- no `LIMIT`

The controls apply at different stages:

- `batchSize` chunks the initial snapshot sent by the server.
- `lastRows` selects how much history to rewind before live changes begin.
- `limit` caps the materialized live row set the SDK keeps after startup.

Keep Dart-side presentation logic for sorting and grouping after rows arrive.
Use `limit` only when the materialized live state itself should stay bounded.

## Concurrency notes

The Dart SDK test suite covers multi-client fan-out and many simultaneous subscriptions on one shared client connection. That means the supported path is not just single-subscriber demos: concurrent writers and multiple active listeners on the same query are part of the exercised contract.

## Basic subscription

```dart
final stream = client.liveEvents(
  "SELECT * FROM app.messages WHERE room = 'main'",
);

await for (final event in stream) {
  switch (event) {
    case AckEvent(:final subscriptionId, :final schema, :final totalRows):
      print('ack $subscriptionId: $totalRows rows, columns=${schema.length}');
    case InitialDataBatch(:final rows, :final hasMore):
      print('snapshot batch: ${rows.length}, hasMore=$hasMore');
    case InsertEvent(:final row):
      print('insert: ${row['id']?.asInt()}');
    case UpdateEvent(:final row, :final oldRow):
      print('update: $oldRow -> $row');
    case DeleteEvent(:final row):
      print('delete: $row');
    case SubscriptionError(:final code, :final message):
      print('subscription error [$code]: $message');
  }
}
```

### Snapshot vs live changes

Most subscriptions follow this pattern:

1. `AckEvent` confirms the subscription and includes schema metadata.
2. One or more `InitialDataBatch` events deliver the initial snapshot.
3. `InsertEvent` / `UpdateEvent` / `DeleteEvent` deliver live changes.

If you only care about live changes, you can ignore `InitialDataBatch` after initial UI hydration.

## Controlling the initial snapshot

### `batchSize`

If the initial query returns many rows, the server can send it in batches. `batchSize` controls the maximum rows per snapshot batch.

```dart
final stream = client.liveEvents(
  'SELECT * FROM app.large_table',
  batchSize: 100,
);
```

You will receive one or more `InitialDataBatch` events with `hasMore=true`, followed by `hasMore=false` and then live changes.

### `lastRows`

`lastRows` asks the server to “rewind” and include the last N rows before live changes begin. This is useful for chat timelines or activity feeds.

```dart
final stream = client.liveEvents(
  'SELECT * FROM chat.messages WHERE room_id = $1',
  lastRows: 50,
);
```

### `subscriptionId`

Provide an explicit `subscriptionId` if you want stable IDs for client-side bookkeeping:

```dart
final stream = client.liveEvents(
  "SELECT * FROM app.messages WHERE room = 'main'",
  subscriptionId: 'messages-feed',
);
```

### `from`

Use `from` to resume a subscription from a known sequence ID.

`from` is the only resume cursor exposed by the Dart API. Commit ordering and
snapshot boundaries stay on the server side.

This is most useful when you:

- persist a checkpoint (last processed seq id) across app restarts
- want to re-subscribe without re-processing older events

```dart
final stream = client.liveEvents(
  "SELECT * FROM app.messages WHERE room = 'main'",
  from: SeqId.parse('123'),
);
```

For materialized live rows, prefer `onCheckpoint` so each applied snapshot gives you the latest `SeqId` directly. `getSubscriptions()` remains available when you need to inspect the whole shared connection state.

When you resume with `from`, the intended contract is strict resume:

- rows with `_seq <= from` must not be replayed
- rows written while the connection was down must still arrive after reconnect
- new live rows after reconnect continue on the same stream

The Dart reconnect and resume test suite covers these cases with duplicate `_seq`
assertions so the SDK keeps a strict “no replay, no missed change” boundary.

## Listing active subscriptions

Use `getSubscriptions()` to inspect all subscriptions on the shared connection.

```dart
final subs = await client.getSubscriptions();
for (final sub in subs) {
  print('${sub.id}: ${sub.query} (lastSeqId=${sub.lastSeqId}, closed=${sub.closed})');
}
```

For server-side observability while debugging, query `SELECT * FROM system.live`.
That view is node-local and reflects the subscriptions currently held in memory
on the node you query. It is not persisted to RocksDB and it is not replicated
through Raft.

## Cancelling a subscription

The SDK returns a `Stream<ChangeEvent>`. Cancel by cancelling your `StreamSubscription`.

```dart
final sub = client.liveEvents('SELECT * FROM app.messages').listen((_) {});

// later
await sub.cancel();
```

Cancellation stays lightweight on the Dart side. The SDK cancels the Rust
subscription handle on the shared connection, waits briefly for the pull loop to
unblock, and then closes the stream. That avoids leaving a hanging Dart-side
pull loop after a widget or provider disposes.

## Data types

All subscription events expose rows as `Map<String, KalamCellValue>` for type-safe access:

- `InitialDataBatch.rows` → `List<Map<String, KalamCellValue>>`
- `InsertEvent.row` / `InsertEvent.rows`
- `UpdateEvent.columnIndexes` / `UpdateEvent.row` / `UpdateEvent.oldRow` / `UpdateEvent.oldRows`
- `DeleteEvent.row` / `DeleteEvent.oldRows`

### UPDATE event behavior

For `UpdateEvent`, `row` and `rows` are **sparse typed maps** reconstructed from the Ack schema.
They contain only the changed columns plus the primary key columns and `_seq`.

`oldRow` contains the previous values for that same sparse column set.

`columnIndexes` exposes the original positional update layout from the wire protocol. It lines up
with the values sent on the Rust bridge raw event if you need exact sparse-column ordering.

```dart
case UpdateEvent(:final row, :final oldRow):
  // row has ALL non-null columns — no need to re-query
  final id = row['id']?.asString();
  final updateType = row['update_type']?.asString();
  final targetId = row['target_id']?.asString();
  print('Trigger: $updateType on $targetId');

  // oldRow has only the changed columns
  for (final key in oldRow.keys) {
    if (!key.startsWith('_') && key != 'id') {
      print('  changed: $key: ${oldRow[key]} → ${row[key]}');
    }
  }
```

Use the typed accessors to extract values:

```dart
case InsertEvent(:final row):
  final id    = row['id']?.asInt();
  final name  = row['name']?.asString();
  final score = row['score']?.asDouble();
  final flag  = row['active']?.asBool();
```

These event accessors are already decoded from JSON row payloads; no manual `List<List<dynamic>>` parsing is needed.

See [Cell Values](https://kalamdb.org/docs/dart-sdk/cell-values) for the full `KalamCellValue` API.

## Next

- [Client Lifecycle](https://kalamdb.org/docs/dart-sdk/client-lifecycle)
- [API: WebSocket Protocol](https://kalamdb.org/docs/server/api/websocket-protocol)
- [Architecture: Live Query](https://kalamdb.org/docs/server/architecture/live-query)

================================================================================
# Types & Models
URL: https://kalamdb.org/docs/dart-sdk/types
Source: content/dart-sdk/types.mdx

> Reference key Dart SDK models in kalam_link, including QueryResponse/QueryResult, SchemaField, LoginResponse, ConnectionHandlers, and ChangeEvent subclasses.
# Types & Models

This page documents the most important public models returned by the Dart/Flutter SDK.

## Query models

### `QueryResponse`

Returned by `client.query(...)`.

- `success: bool`
- `results: List<QueryResult>` (one per statement)
- `tookMs: double?`
- `error: ErrorDetail?`

Convenience helpers:

- `rows` → first result’s `List<Map<String, KalamCellValue>>`
- `columns` → first result’s `List<SchemaField>`

Implementation note: the SDK parses row JSON and keeps schema metadata (`SchemaField`) so you can build higher-level typed mappers in your app.

### `QueryResult`

- `columns: List<SchemaField>`
- `rows: List<Map<String, KalamCellValue>>`
- `rowCount: int`
- `message: String?`

### `SchemaField`

Column metadata includes flag helpers:

- `isPrimaryKey` (flag contains `pk`)
- `isNonNull` (flag contains `nn`)
- `isUnique` (flag contains `uq`)

## Authentication models

### `LoginResponse`

Returned by `login(...)` and `refreshToken(...)`.

- `accessToken: String`
- `refreshToken: String?`
- `expiresAt: String`
- `refreshExpiresAt: String?`
- `user: LoginUserInfo`

`expiresAt` and `refreshExpiresAt` are returned as strings (server-provided timestamps). Treat them as opaque unless you control the formatting.

## Connection handlers

To receive lifecycle events, pass `ConnectionHandlers` to `KalamClient.connect(...)`.

Callbacks include:

- `onConnect()`
- `onDisconnect(DisconnectReason reason)`
- `onError(ConnectionErrorInfo error)`
- `onReceive(String message)`
- `onSend(String message)`
- `onEvent(ConnectionEvent event)`

Lifecycle event types:

- `ConnectEvent`
- `DisconnectEvent`
- `ConnectionErrorEvent`
- `ReceiveEvent` / `SendEvent` (debug hooks)

## Logging types

The SDK logging API uses `Level` from `package:logger/logger.dart`.

- `KalamLogger.level: Level`
- `KalamLogger.listener: LogListener?`
- `LogListener = void Function(LogEntry entry)`
- `LogEntry` fields: `level`, `message`, `tag`, `timestamp`

`KalamClient.connect(...)` also accepts `logLevel` and `logListener` so you can configure SDK logs at construction time.

## Subscription event types

`subscribe(...)` returns a `Stream<ChangeEvent>`.

Event classes:

- `AckEvent` — includes `subscriptionId`, `schema`, and snapshot info
- `InitialDataBatch` — initial snapshot rows, potentially in batches
- `InsertEvent`
- `UpdateEvent` — sparse typed rows plus `columnIndexes`
- `DeleteEvent`
- `SubscriptionError`

All row-bearing events expose typed `Map<String, KalamCellValue>` rows (for example `InsertEvent.row`).
For `UpdateEvent`, those maps are sparse and contain only changed columns plus PK and `_seq`.

`AckEvent` and `InitialDataBatch` include snapshot batching fields (`batchNum`, `hasMore`, `status`).

## Subscription management models

### `SubscriptionInfo`

Returned by `client.getSubscriptions()`.

- `id: String` — subscription ID
- `query: String` — SQL query
- `lastSeqId: SeqId?` — last received sequence ID (use for resume with `from`)
- `lastEventTimeMs: int?` — last event timestamp (millis since epoch)
- `createdAtMs: int` — creation timestamp (millis since epoch)
- `closed: bool` — whether the subscription is closed

## KalamCellValue & typed rows

Every cell in a typed row is a `KalamCellValue` — a wrapper class with safe
accessor methods that mirrors the Rust `KalamCellValue` newtype.

```dart

final rows = response.rows; // List<Map<String, KalamCellValue>>
for (final row in rows) {
  final name  = row['name']?.asString();    // String?
  final score = row['score']?.asDouble();   // double?
  final born  = row['created_at']?.asDate(); // DateTime?
  final url   = row['avatar']?.asFileUrl(
    'http://localhost:18080', 'default', 'users',
  );
}
```

Use the current helper directly:

```dart
final rows = response.rows; // List<Map<String, KalamCellValue>>
```

See [Cell Values (KalamCellValue)](https://kalamdb.org/docs/dart-sdk/cell-values) for the full API
including FILE column support and all accessor methods.

### `KalamFileRef`

Returned by `KalamCellValue.asFile()` for FILE columns.

- `id`, `sub`, `name`, `size`, `mime`, `sha256`, `shard`
- `getDownloadUrl(baseUrl, namespace, table)`
- `relativeUrl(namespace, table)`
- `storedName()`
- `relativePath()`
- `isImage()`
- `formatSize()`

The URL and path helpers call the generated Dart bridge into `link-common`, so
the stored filename and download route are defined in one shared Rust model.

## Next

- [Querying & DML](https://kalamdb.org/docs/dart-sdk/querying)
- [Realtime Subscriptions](https://kalamdb.org/docs/dart-sdk/subscriptions)

================================================================================
# Python SDK
URL: https://kalamdb.org/docs/python-sdk
Source: content/python-sdk/index.mdx

> Use the KalamDB Python SDK for SQL queries and live query streams.
# Python SDK

The Python SDK exposes async SQL helpers and the same realtime naming as the other KalamDB SDKs.

```python
from kalamdb import Auth, KalamClient

async with KalamClient("http://localhost:2900", Auth.basic("admin", "AdminPass123!")) as client:
    rows = await client.query_rows("SELECT * FROM app.messages LIMIT 20")
    print(rows)
```

## Live APIs

Use these three methods for realtime work:

| Method | Returns | Use for |
|---|---|---|
| `live(sql, ...)` | `LiveRows` | Materialized row snapshots for a supported `SELECT` query |
| `live_table(table, ...)` | `LiveRows` | Materialized row snapshots for `SELECT * FROM table` |
| `live_events(sql, ...)` | `LiveEvents` | Low-level protocol events such as `subscription_ack`, `initial_data_batch`, `change`, and `error` |

Read next: [Live Queries](https://kalamdb.org/docs/python-sdk/subscriptions).

================================================================================
# Python Live Queries
URL: https://kalamdb.org/docs/python-sdk/subscriptions
Source: content/python-sdk/subscriptions.mdx

> Use live(), live_table(), and live_events() in the KalamDB Python SDK.
# Python Live Queries

The Python SDK has the same three realtime concepts as TypeScript and Dart, with Pythonic snake_case names.

## Materialized rows

Use `live()` when your app wants current rows instead of raw change events.

```python
live_rows = await client.live(
    "SELECT id, body FROM chat.messages WHERE thread_id = 1",
    last_rows=20,
    limit=20,
    key_columns=["id"],
    on_checkpoint=lambda checkpoint: save_resume_token(checkpoint["last_seq_id"]),
)

async for rows in live_rows:
    render(rows)
```

Use `live_table()` when `SELECT * FROM table` is enough.

```python
tasks = await client.live_table(
    "app.tasks",
    last_rows=20,
    key_columns=["id"],
)

async for rows in tasks:
    render_tasks(rows)
```

## Low-level events

Use `live_events()` when you need the protocol frames directly.

```python
events = await client.live_events(
    "SELECT * FROM chat.messages WHERE thread_id = 1",
    batch_size=100,
    last_rows=50,
    on_checkpoint=lambda checkpoint: save_resume_token(checkpoint["last_seq_id"]),
    on_error=lambda event: log_live_error(event),
)

async for event in events:
    kind = event.get("type")
    if kind == "subscription_ack":
        print("opened", event["subscription_id"])
    elif kind == "initial_data_batch":
        hydrate(event["rows"])
    elif kind == "change":
        apply_change(event["change_type"], event.get("rows"), event.get("old_values"))
    elif kind == "error":
        break
```

## Options

| Option | Applies to | Meaning |
|---|---|---|
| `batch_size` | all live methods | Max rows per initial snapshot batch |
| `last_rows` | all live methods | Include the last N rows before live changes begin |
| `from_` | all live methods | Resume from a known sequence ID |
| `auto_fetch_batches` | all live methods | Ask for remaining startup batches automatically |
| `limit` | `live()`, `live_table()` | Cap the materialized row set kept by the SDK |
| `key_columns` | `live()`, `live_table()` | Stable row identity columns when the query does not expose `id` |
| `on_checkpoint` | all live methods | Persist the latest applied sequence ID |
| `on_error` | `live_events()` | Receive low-level error frames separately |

The checkpoint object contains `subscription_id` and `last_seq_id` strings. Feed `last_seq_id` back into `from_` on the next session to resume.

================================================================================
# Chat With AI
URL: https://kalamdb.org/docs/use-cases/chat-with-ai
Source: content/use-cases/chat-with-ai.mdx

> Vite + React chat app with ORM inserts, topic-driven agent worker, and live subscriptions across browser tabs.
# Chat With AI

A small Vite + React example for the ask-message flow: user sends a message, an agent worker replies, and every open tab sees updates live.

**Source:** [`examples/chat-with-ai/`](https://github.com/kalamdb/KalamDB/tree/main/examples/chat-with-ai)

There is no external AI API call — the agent returns a deterministic demo reply so you can run everything locally without API keys.

## What it does

- The browser writes a user message into `chat_demo.messages` with the ORM
- `chat_demo.ai_inbox` receives those inserts as a topic
- `src/agent.ts` runs `runConsumer()` from `@kalamdb/consumer`
- The agent writes progress rows into `chat_demo.agent_events`
- The agent commits one assistant reply back into `chat_demo.messages`
- Every open browser tab sees the rows through live subscriptions

## Prerequisites

- Node.js 18+
- Kalam CLI

## Quick start

From the example folder:

```bash
cd examples/chat-with-ai
npm install
kalam dev
```

`kalam dev` starts the local server, applies `kalam/schema.sql`, regenerates `src/schema.generated.ts`, and runs both the browser app and agent worker from `[dev.processes]`.

Open the Vite URL (usually `http://127.0.0.1:5173`).

## What to try

Type a short message such as:

```text
latency spike after deploy
```

You should see your user row, live agent progress, and then a final `AI reply:` row. Open a second browser tab to confirm live updates arrive there too.

## The write path (browser)

```ts
await db.insert(chatMessages).values({
    room: ROOM,
    role: 'user',
    author: CHAT_USERNAME,
    sender_username: CHAT_USERNAME,
    content,
});
```

## The agent path (worker)

```ts
await runConsumer<ChatDemoMessages>({
    client,
    name: 'chat-ai-agent',
    topic: 'chat_demo.ai_inbox',
    groupId: 'chat-ai-agent',
    onChange: async (_ctx, change) => {
        const row = change.data;
        if (row.role !== 'user') return;
        // Build and persist the assistant reply here.
    },
});
```

Setup creates the topic route:

```sql
CREATE TOPIC chat_demo.ai_inbox;
ALTER TOPIC chat_demo.ai_inbox ADD SOURCE chat_demo.messages ON INSERT;
```

## Files worth reading

| File | Purpose |
|------|---------|
| `src/App.tsx` | Browser client, ORM insert, live subscriptions |
| `src/schema.generated.ts` | Generated table definitions and row types |
| `src/agent.ts` | Topic consumer and reply worker |
| `kalam.toml` | `kalam dev` project config |
| `kalam/schema.sql` | USER table, STREAM table, topic route, seed data |
| `kalam/migrations/0001_init.sql` | Initial migration |

## Credentials

`kalam dev` uses the local development account:

- user: `root`
- password: `kalamdb123`

Both the browser and worker use this account for the local demo.

## Troubleshooting

**Server health check**

```bash
curl http://127.0.0.1:2900/health
```

**Agent running but no reply**

Reset local state and reapply the migration:

```bash
kalam db reset
kalam dev
```

## Next steps

- [React AI Chat](https://kalamdb.org/docs/use-cases/react-ai-chat) — full `@kalamdb/react` chat with approvals and attachments
- [TypeScript Chat Quickstart](https://kalamdb.org/docs/ts-sdk/chat-quickstart) — docs-only minimal path
- [Summarizer Agent](https://kalamdb.org/docs/use-cases/summarizer-agent) — worker-only enrichment pattern

================================================================================
# Use Cases & Examples
URL: https://kalamdb.org/docs/use-cases
Source: content/use-cases/index.mdx

> Runnable KalamDB example projects from the examples/ folder — TypeScript browser apps and workers you can clone and run locally.
# Use Cases & Examples

Every project listed here lives under [`examples/`](https://github.com/kalamdb/KalamDB/tree/main/examples) in the KalamDB repository. You can clone the repo, or let an installed `kalam` CLI download an example into your own project directory with `kalam init --template <example-id>`.

> For Rust SDK examples, see [Rust SDK — Examples](https://kalamdb.org/docs/rust-sdk/examples). For SDK setup, see [TypeScript SDK](https://kalamdb.org/docs/ts-sdk/setup) or [Rust SDK](https://kalamdb.org/docs/rust-sdk/setup).

## Before you run an example

To start from the CLI without cloning the repo:

```bash
mkdir chat-demo && cd chat-demo
kalam init --template chat-with-ai
kalam dev
```

To run from a cloned checkout, open an example folder:

```bash
npm install
kalam dev
```

`kalam dev` starts the local server, applies the checked-in migration/schema, regenerates TypeScript artifacts when configured, and runs app or worker processes from `[dev.processes]`.

## Example catalog

| Example | What you'll learn | Docs |
|---------|-------------------|------|
| Live OKF Context Sync | One SQL table + `FILE` column + sync worker: mirror an OKF Markdown folder to KalamDB with live pull and local watch push | [Live OKF Context Sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync) |
| Realtime Ops Feed | `queryAll()`, `live()`, and browser auth in one React app | [Realtime Ops Feed](https://kalamdb.org/docs/use-cases/realtime-ops-feed) |
| Chat With AI | User messages, topic-driven agent worker, live chat UI | [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai) |
| React AI Chat | Full chat UI with `@kalamdb/react`, approvals, and file attachments | [React AI Chat](https://kalamdb.org/docs/use-cases/react-ai-chat) |
| Summarizer Agent | Topic consumer that enriches rows and handles failures | [Summarizer Agent](https://kalamdb.org/docs/use-cases/summarizer-agent) |

## Repository paths

| Example | Path |
|---------|------|
| Live OKF context sync | [`examples/live-okf-context-sync/`](https://github.com/kalamdb/KalamDB/tree/main/examples/live-okf-context-sync) |
| Realtime ops feed | [`examples/simple-typescript/`](https://github.com/kalamdb/KalamDB/tree/main/examples/simple-typescript) |
| Chat with AI | [`examples/chat-with-ai/`](https://github.com/kalamdb/KalamDB/tree/main/examples/chat-with-ai) |
| React AI chat | [`examples/react-ai-chat/`](https://github.com/kalamdb/KalamDB/tree/main/examples/react-ai-chat) |
| Summarizer agent | [`examples/summarizer-agent/`](https://github.com/kalamdb/KalamDB/tree/main/examples/summarizer-agent) |

## Which example should I pick?

- **Google Open Knowledge Format (OKF) folder sync for agents?** [Live OKF Context Sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync)
- **Smallest browser app?** [Realtime Ops Feed](https://kalamdb.org/docs/use-cases/realtime-ops-feed)
- **AI chat with a worker?** [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai)
- **Full React chat with approvals?** [React AI Chat](https://kalamdb.org/docs/use-cases/react-ai-chat)
- **Worker-only topic enrichment?** [Summarizer Agent](https://kalamdb.org/docs/use-cases/summarizer-agent)

================================================================================
# Live OKF Context Sync
URL: https://kalamdb.org/docs/use-cases/live-okf-context-sync
Source: content/use-cases/live-okf-context-sync.mdx

> Build and run a KalamDB OKF context-folder sync: Markdown files on disk, FILE rows in SQL, and liveTable updates across clients.
# Live OKF Context Sync

Sync a **Google Open Knowledge Format (OKF)** context folder to KalamDB and back — so the same Markdown files are available on every device or agent process that runs the sync worker.

This example is intentionally small: one `schema.sql`, one Node sync process, and no custom file server. KalamDB stores file bytes as **real objects** under the server storage directory (not as SQL BLOBs); the sync worker watches a local folder with **chokidar** for local edits and mirrors server state with a **single ORM live subscription** for remote edits.

For the full FILE upload and on-disk layout model, see [/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype).

> **Showcase only.** KalamDB does not ship folder sync as a core product feature. This app shows how little code you need to layer OKF context sync on top of SQL + `FILE` + **`liveTable()`** — one subscribe call, typed Drizzle rows, no polling loop.

**Source:** [`examples/live-okf-context-sync/`](https://github.com/kalamdb/KalamDB/tree/main/examples/live-okf-context-sync)

## What is Google Open Knowledge Format (OKF)?

**Google Open Knowledge Format (OKF)** is a folder of Markdown files that agents and tools read as structured context: profiles, project notes, runbooks, policies, and preferences. Files often include YAML frontmatter so tools know what each document is for.

Teams using OKF usually need the same folder everywhere the user works — laptop agent, cloud agent, CI job, or second machine. This example keeps **one canonical copy on the server** and mirrors it to disk on each client.

## What this example includes

| Piece | Role |
|-------|------|
| `kalam/schema.sql` | Single USER table — the only schema source of truth |
| `kalam dev` | Starts KalamDB, applies schema, regenerates TypeScript types, runs the sync worker |
| `src/sync-app.ts` | Tiny runnable entrypoint |
| `src/sync-engine.ts` | Orchestrates push (chokidar), pull (`liveTable`), deletes, and one shared `TaskQueue` |
| `src/folder-watcher.ts` | **chokidar** with `ignoreInitial: false` — rescans on every start |
| `src/remote-files.ts` | ORM upsert (`onConflictDoUpdate`) + delete + FILE download |
| `data/` (default) | Your OKF Markdown files on disk |
| `data/.index/sync.db` | Local SQLite cache (hashes + pending upload queue) — never uploaded |
| `seed/` | Starter files copied into an empty folder on first run only |

There is **no web UI or demo agent** in the current version. The focus is the sync loop itself: disk ↔ KalamDB ↔ **one live query per client** — no HTTP polling, no manual “refresh folder” timer.

## Run it locally

Use three terminals: one for `kalam dev`, then two extra sync clients pointed at different local folders.

### Terminal 1 — start KalamDB and the default sync worker

```bash
cd examples/live-okf-context-sync
npm install
kalam dev
```

`kalam dev` starts the local KalamDB server, applies `kalam/schema.sql`, regenerates ORM types, and runs the configured sync worker on `data/`.

Wait until the logs include:

```text
[sync] initial sync for folder '.../data' completed: ...
[sync] live subscription active
```

### Terminal 2 — start a second folder as Alice

```bash
cd examples/live-okf-context-sync
npm run dev -- data1
```

This command starts only another sync worker. It reuses the KalamDB server that `kalam dev` started in terminal 1.

### Terminal 3 — start a third folder as Alice

```bash
cd examples/live-okf-context-sync
npm run dev -- data2
```

Now you have three local folders (`data/`, `data1/`, `data2/`) syncing against the same user-scoped table in KalamDB.

### Try add, edit, and delete

Run these commands from a fourth terminal, or edit the files in your editor:

```bash
cd examples/live-okf-context-sync

# Add a file in data1. It should appear in data/ and data2.
mkdir -p data1/notes
cat > data1/notes/two-client-test.md <<'EOF'
# Two-client test

This file was created in data1 and should sync through KalamDB.
EOF

# Wait until liveTable has delivered the file to data2.
while [ ! -f data2/notes/two-client-test.md ]; do
  sleep 0.2
done

# Edit it from data2. data/ and data1 should receive the new content.
cat >> data2/notes/two-client-test.md <<'EOF'

Edited from data2.
EOF

# Wait until the edit has reached data/.
while ! { [ -f data/notes/two-client-test.md ] && grep -q "Edited from data2." data/notes/two-client-test.md; }; do
  sleep 0.2
done

# Delete it from data/. data1 and data2 should remove it too.
rm data/notes/two-client-test.md
```

Watch the three sync terminals while you do this:

```text
[sync] pushed path='notes/two-client-test.md' _seq=123456789 size=96 B
[sync] file 'notes/two-client-test.md' downloaded from server (...)
[sync] file 'notes/two-client-test.md' deleted
```

That is the whole point of the example: one local folder change becomes a SQL row/file update, then `liveTable()` pushes the reconciled table state to every other sync process.

If a folder starts empty, it does **not** copy `seed/` once the server already has files. It downloads the server files first, then starts watching for local changes.

## One `liveTable()` subscription, full table state

The pull path is the heart of the example. After startup reconciliation, each sync worker opens **exactly one** live subscription:

```typescript
private async startLivePull(): Promise<void> {
  this.liveUnsub = await liveTable(this.client, context_files, (rows) => {
    this.syncQueue.enqueue(() => this.handleLiveRows(rows));
  });
  console.log('[sync] live subscription active');
}
```

That is the entire subscribe surface for server → disk updates. There is no second subscription for deletes, no per-file poll, and no `setInterval` against the REST API.

### What KalamDB + `@kalamdb/orm` do under this call

`context_files` is a generated Drizzle table from `kalam/schema.sql`:

```typescript

  path: text('path').primaryKey(),
  file_ref: file('file_ref').notNull(),
  created_at: timestamp('created_at', { mode: 'date' }),
  updated_at: timestamp('updated_at', { mode: 'date' }),
});

```

When you call `liveTable(client, context_files, callback)`:

1. **SQL is compiled from the table definition** — `@kalamdb/orm` builds `SELECT * FROM okf_sync.context_files` (namespace comes from the client / `kalam.toml` connection).
2. **Row typing is applied** — WASM/live events are mapped through Drizzle column metadata (`file_ref` → `FileRef`, timestamps → `Date`).
3. **Primary key inference** — `path` is the live reconciliation key (`getKey: ['path']`), so inserts, updates, and deletes merge into one materialized array in the Rust live core.
4. **One WebSocket subscription** — the first `liveTable()` on a client opens the shared `/v1/ws` socket; this query registers once on that connection ([client lifecycle](https://kalamdb.org/docs/ts-sdk/client-lifecycle)).
5. **Callbacks receive the full current row set** — not individual `{ op: 'delete' }` events you must fold yourself in simple mode. Every change re-delivers **all rows that match the query right now**, already deduplicated by primary key.

Conceptually:

```text
  KalamDB                           @kalamdb/client + ORM              sync-engine.ts
  ───────                           ─────────────────────              ───────────
  INSERT / UPDATE / DELETE          live core materializes rows        handleLiveRows(rows)
  on okf_sync.context_files   ──►   mapRow → ContextFiles[]      ──►   diff paths → disk I/O
  (MVCC row versions)               callback(rows)                     (download / unlink)
```

This is [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple): your application code works with **`rows[]`**, the SDK applies `initial_data_batch`, `insert`, `update`, and `delete` internally. For a UI you render the array; for this example we **diff path sets** and mirror files.

### Why a single subscribe is enough for folder sync

A naive design might subscribe per file path, poll `GET /files/:id`, or run a SQL query after every chokidar event. KalamDB instead keeps one **table-level live query** aligned with MVCC:

| Concern | How one `liveTable()` covers it |
|---------|----------------------------------|
| New file on another client | Row appears in `rows`; worker downloads bytes |
| Edit on another client | Row’s `file_ref.sha256` changes; worker re-downloads |
| Delete on another client | Row disappears from `rows`; worker removes local file |
| Bulk delete (10 files) | One or few callbacks with shrinking `rows`; diff removes each path |
| Reconnect | Same subscription SQL resumes; SDK replays from checkpoint optional |

The worker stores the last known path set in `remotePaths`. On each callback:

```typescript
private async handleLiveRows(rows: ContextFiles[]): Promise<void> {
  const nextPaths = new Set(rows.map((row) => row.path).filter(isSafeSyncPath));

  // … skip empty first snapshot (SDK warmup) …

  for (const path of this.remotePaths) {
    if (!nextPaths.has(path)) {
      await this.removeLocalFile(path);   // live delete → local unlink
    }
  }
  this.remotePaths = nextPaths;

  for (const row of rows) {
    await this.pullRemoteRow(row);        // insert/update → download if hash differs
  }
}
```

**Deletes are implied:** if `profile.md` was in `remotePaths` but not in the latest `rows`, the server no longer has that row — remove the local Markdown file. You do not need a separate delete channel.

### ORM vs raw SQL vs raw events

| API | What you write | What the callback receives |
|-----|----------------|----------------------------|
| `client.liveEvents(sql, …)` | SQL string | Low-level batches, acks, per-change events |
| `client.live(sql, …)` | SQL string | **Full materialized row array** |
| `liveTable(client, table, …)` | Drizzle `context_files` table | Same as `live()`, plus typed rows and compiled SQL |

The OKF example uses **`liveTable`** because:

- Schema lives in SQL → `kalam dev` regenerates `schema.generated.ts` → the subscription always matches the applied server schema.
- `ContextFiles` types flow from ORM into `pullRemoteRow` without hand-written row parsers.
- `file('file_ref')` maps to KalamDB’s `FILE` column type, so each live row carries a typed `FileRef` for download URLs.

If you needed raw events (metrics, incremental audit log), you would drop down to `client.liveEvents()`. For “keep this folder equal to this table,” **`liveTable` is the intended API.**

### Serializing live with push (`syncQueue`)

The live callback can fire while chokidar is mid-push. The example shares one `TaskQueue` between:

- chokidar `onUpsert` / `onDelete` → `pushLocalFile` / `deleteRemoteFile`
- `liveTable` callback → `handleLiveRows`

```typescript
// folder watcher enqueues onto the same queue passed from the sync engine
{ taskQueue: this.syncQueue }

// live pull enqueues too
this.syncQueue.enqueue(() => this.handleLiveRows(rows));
```

So a live snapshot never interleaves with a half-finished FILE upload. FILE columns are versioned with MVCC; serializing avoids applying a `file_ref` from row version *N* while version *N+1* is still uploading.

### Multi-client correctness (3+ sync folders)

Run the same example three times — `data/`, `data1/`, `data2/` — all as **alice** on the same server. Each process has its **own** `liveTable()` subscription and its **own** `syncQueue`, but all subscribe to the same logical query scoped to alice’s USER table partition.

When one client deletes many files:

1. Local `unlink` → `deleteRemoteFile` → SQL `DELETE` on KalamDB.
2. Live core broadcasts a new materialized row set to **every** subscriber.
3. Other clients diff `remotePaths` → `nextPaths` and unlink locally.

The example adds **tombstones** (`src/lib/sync-tombstones.ts`) so a stale snapshot or a not-yet-updated folder cannot re-upload deleted bytes before the next live callback. That is example-level safety, not a KalamDB requirement — the important platform guarantee is still **one query, one reconciled row array, pushed to all subscribers.**

## Architecture

```text
  data/                         KalamDB (okf_sync.context_files)
  ├── index.md                  ┌─────────────────────────────────┐
  ├── profile.md                │ path TEXT PRIMARY KEY           │
  └── .index/                   │ created_at, updated_at          │
      └── sync.db (local only)  └─────────────────────────────────┘
           │                              ▲
           │  push (chokidar + upsert)    │  pull (liveTable + download)
           └──────────────────────────────┘
                    sync-engine.ts
```

**Two stores, one logical folder:**

1. **KalamDB** — durable SQL metadata in RocksDB/Parquet plus **file bytes as standalone objects** under `kalam/server/data/storage/`. Each row's `file_ref` column is a `FileRef` JSON pointer (id, sha256, size, mime, subfolder) — not the Markdown bytes themselves. Each INSERT/UPDATE creates a new MVCC row version with a new `_seq`; `file_ref` is a normal column value inside that row.
2. **Local disk** — Markdown files you edit, plus `.index/sync.db` for hashes and offline upload retries.

See [FILE bytes on disk (not in SQL)](#file-bytes-on-disk-not-in-sql) for a sample storage path and how it differs from BLOB columns in other databases.

**Startup order (current worker):**

1. **Pull** — `SELECT` all remote rows once; download any missing files into an empty folder.
2. **Push** — chokidar initial scan + offline reconciliation; upsert local edits to KalamDB.
3. **Subscribe** — `liveTable(context_files)`; all further server changes arrive through the single live callback.

Push-before-subscribe on the **initial scan** prevents a stale first snapshot from overwriting offline edits. Pull-before-push on **empty local folders** restores from the server without waiting for the first live batch.

## FILE bytes on disk (not in SQL)

Most databases store uploaded content **inside** the row — PostgreSQL `BYTEA`, MySQL `LONGBLOB`, or similar. KalamDB does not. A `FILE` column holds a **`FileRef` JSON document**; the server writes the bytes to its configured **storage folder or object store** as a normal file.

Architecture reference: [/docs/server/architecture/file-upload-datatype](https://kalamdb.org/docs/server/architecture/file-upload-datatype)

### What the sync worker sends

When you save `data/profile.md`, the worker upserts through Drizzle with `kalamFile('upload', file)`. The driver sends multipart SQL; KalamDB finalizes the bytes under the table's storage path and stores metadata in the row:

```typescript
await db
  .insert(context_files)
  .values({
    path: 'profile.md',
    file_ref: kalamFile('upload', upload),
    updated_at: now,
  })
  .onConflictDoUpdate({
    target: context_files.path,
    set: { file_ref: kalamFile('upload', upload), updated_at: now },
  });
```

### What you see in SQL vs on disk

After a push, query the row:

```bash
kalam sql "SELECT path, file_ref FROM okf_sync.context_files WHERE path = 'profile.md'"
```

The `file_ref` column looks like JSON metadata (exact id values vary per upload):

```json
{
  "id": "1234567890123456789",
  "sub": "f0001",
  "name": "profile.md",
  "size": 142,
  "mime": "text/markdown",
  "sha256": "7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069"
}
```

The **Markdown bytes are not in that JSON**. They live as a separate object. With `kalam dev`, inspect the local storage tree (alice's numeric `userId` comes from login — list it with `kalam sql "SELECT user_id FROM system.users WHERE username = 'alice'"` if needed):

```text
examples/live-okf-context-sync/kalam/server/data/
├── rocksdb/                         # hot SQL rows (FileRef JSON, not file bytes)
└── storage/                         # FILE objects (actual bytes)
    └── okf_sync/context_files/{userId}/
        └── f0001/
            └── 1234567890123456789-profile.md   # real file on disk
```

You can open that file directly — it is the same content as `data/profile.md` on the client. Other clients download it through the file API using the `FileRef` (`GET /v1/files/okf_sync/context_files/f0001/123…-profile.md`), not by reading a BLOB out of a `SELECT`.

When you edit the same path again, MVCC appends a **new row version** with a new `file_ref` (often a new object path). Old objects may remain until cleanup; the live subscription always reconciles against the latest visible row set.

## Schema

The whole server schema fits in one file:

```sql
CREATE NAMESPACE IF NOT EXISTS okf_sync;

CREATE USER TABLE okf_sync.context_files (
  path TEXT PRIMARY KEY,
  file_ref FILE NOT NULL,
  created_at TIMESTAMP DEFAULT NOW(),
  updated_at TIMESTAMP DEFAULT NOW()
);
```

- **`path`** — relative OKF path (`profile.md`, `notes/two-client-test.md`).
- **`file_ref`** — server-managed `FileRef` JSON (id, sha256, size, mime, download path). Content hash lives on the ref; no duplicate `sha256` column in SQL.
- **`USER TABLE`** — rows and file downloads are scoped per user (Alice cannot read Bob's files).

`kalam.toml` sets `generate_types = true`, so `kalam dev` regenerates `src/models/schema.generated.ts` from this SQL via `@kalamdb/orm`.

## Source layout

```text
examples/live-okf-context-sync/
  data/                         # default sync folder (gitignored)
    .index/
      sync.db                   # local SQLite — never synced
    index.md
    profile.md

  seed/                         # copied into data/ on first run only
    index.md
    profile.md

  kalam/
    schema.sql                  # source of truth
    server/data/                # KalamDB server data (persists across restarts)

  src/
    sync-app.ts                 # tiny runnable entrypoint
    sync-engine.ts              # startup order, push/pull/live orchestration
    folder-watcher.ts           # chokidar (ignoreInitial: false)
    remote-files.ts             # ORM upsert/delete/select + FILE download
    local-cache.ts              # SQLite metadata + pending queue
    helpers.ts                  # disk I/O, waits, TaskQueue
    sync-log.ts                 # structured console logging
    lib/
      paths.ts                  # safe paths, ignore rules
      seed.ts                   # first-run copy from seed/
      file-utils.ts             # sha256, mime type, upload File builder
      sync-tombstones.ts        # block stale re-upload/re-download after delete
    db/                         # Kalam client + local SQLite
    models/schema.generated.ts  # generated — do not edit
    models/schema.local.ts      # Drizzle types for local SQLite
```

**Never synced:** `.index/`, `.git/`, `.DS_Store`, SQLite WAL/SHM files.

## How it works (detailed)

### 1. Startup (`kalam dev`)

`kalam dev` (from `kalam.toml`):

1. Starts KalamDB locally (`kalam/server/data/`)
2. Applies `kalam/schema.sql` and regenerates TypeScript types
3. Runs `npm run dev -- data` — the sync worker as user **alice** by default

On startup the worker:

1. Ensures demo users `alice` / `bob` exist
2. Logs in and loads existing remote paths from KalamDB (`select().from(context_files)`)
3. **Bootstrap:** if `data/` has no user files **and** the server has no rows, copies `seed/` into `data/`
4. **Initial pull:** downloads server files missing on disk (empty folder / new sync dir restore)
5. **Initial push:** chokidar scans `data/` with `ignoreInitial: false`, pushes changed files, reconciles offline deletions, flushes the pending upload queue
6. **Live pull:** one `liveTable(context_files, …)` subscription — see [One liveTable subscription](#one-livetable-subscription-full-table-state)
7. **Watch:** chokidar keeps running; push/delete events share the same `TaskQueue` as live callbacks

Console output during initial sync:

```text
[sync] initial sync for folder '.../data' started ...
[sync] file 'profile.md' downloaded from server (142 B)
[sync] downloaded 1 file(s) from server
[sync] file 'profile.md' added
[sync] initial sync for folder '.../data' completed: 1 added, 0 updated, 0 deleted, 0 pushed, 1 downloaded (1 total changes)
[sync] live subscription active
```

### 2. Push — local edit → server

When you save or add a file under `data/`:

1. chokidar fires `add` or `change`; the watcher logs `file '…' added` or `updated`
2. `pushLocalFile(path)` reads bytes and computes SHA-256
3. If local SQLite cache matches and nothing is pending, skip (no-op)
4. Otherwise the worker upserts through **Drizzle ORM** with `kalamFile('upload', file)` in `.values()` / `.onConflictDoUpdate().set()`
5. KalamDB stores bytes as a **standalone object** under `kalam/server/data/storage/` and appends a **new hot MVCC row** with updated `file_ref` JSON (see [FILE bytes on disk](#file-bytes-on-disk-not-in-sql))
6. Local SQLite records the hash; any pending-upload row is cleared
7. Console logs `pushed path='…' _seq=… size=…`

When you delete a file locally, chokidar fires `unlink`, logs `file '…' deleted`, and removes the row with `db.delete(context_files)`.

If the server is down, the upload is queued in `pending_uploads` and retried on reconnect.

### 3. Offline edits (kalam dev stopped)

While the sync worker is not running you can add, edit, or delete files under `data/`. On the next `kalam dev`:

1. Missing server files are **downloaded first** (initial pull)
2. chokidar replays every file on disk (`ignoreInitial: false`) and pushes local changes
3. `reconcileLocalDeletions` deletes server rows for files removed while offline
4. **`liveTable` starts last** — runtime sync is live-only after this point

### 4. Pull — server change → local disk (live subscription)

After `startLivePull()`, every server-side change to `context_files` flows through the **same** `liveTable` callback:

1. SDK delivers `ContextFiles[]` — full materialized query result
2. Paths gone since last callback → `removeLocalFile` (live delete)
3. For each row, compare local SHA-256 with `file_ref.sha256`
4. Match → update local SQLite cache only
5. Differ → download via `FileRef.getDownloadUrl(...)`, verify hash, write disk
6. Console logs `file '…' downloaded from server (size)` or `file '…' deleted`

The worker ignores the first empty callback (SDK warmup before `initial_data_batch` completes) so an empty `rows` array is never treated as “delete everything.”

See [One liveTable subscription](#one-livetable-subscription-full-table-state) for the diff model and queue serialization.

### 5. Bootstrap rules

| Local `data/` | Server rows | What happens |
|---------------|-------------|--------------|
| Empty | Empty | Copy `seed/` → push to server |
| Empty | Has rows | **Download from server** (no seed); then subscribe |
| Has files | Any | Download missing paths, push local changes, then `liveTable` reconciles |

Deleting `data/` does **not** delete server data. On the next start, the worker pulls everything back from KalamDB.

### 6. Per-user isolation

Default sync user is **alice** (`KALAM_USER` / `KALAM_PASSWORD`). User-scoped rows and FILE downloads are enforced by KalamDB. Run a second worker as Bob:

```bash
KALAM_USER=bob KALAM_PASSWORD=bob123 npm run dev -- data-bob
```

## Prerequisites

- Node.js 20+
- Kalam CLI with `kalam dev` ([CLI dev workflow](https://kalamdb.org/docs/cli/dev))

## Single-folder quick start

```bash
cd examples/live-okf-context-sync
npm install
kalam dev
```

`kalam dev` starts the server and the sync worker on `data/`. Edit Markdown under `data/` and watch the console for push/pull lines.

For the more useful multi-client demo, keep `kalam dev` running and start `npm run dev -- data1` plus `npm run dev -- data2` in two more terminals as shown in [Run it locally](#run-it-locally).

Default credentials:

| Role | User | Password |
|------|------|----------|
| Server admin (`kalam dev`) | `root` | `kalamdb123` |
| Default sync user | `alice` | `alice123` |
| Second demo user | `bob` | `bob123` |

Sync a different folder:

```bash
npm run dev -- test1/
# or
KALAM_SYNC_DIR=test1 npm run dev
```

## Verify it works manually

This is the most important end-to-end check: **local folder is disposable; KalamDB is the source of truth.**

### Step 1 — Start with `kalam dev`

```bash
cd examples/live-okf-context-sync
npm install
kalam dev
```

Wait until you see `[sync] live subscription active` in the logs.

### Step 2 — Make changes under `data/`

In another terminal:

```bash
cd examples/live-okf-context-sync

# Edit an existing file
echo "" >> data/profile.md
echo "Updated at $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> data/profile.md

# Or add a new note
mkdir -p data/notes
cat > data/notes/manual-test.md <<'EOF'
# Manual restore test

This file should come back after deleting data/.
EOF
```

Confirm the sync worker logged push lines (for example `pushed path='profile.md' _seq=… size=…`). You can also query the server:

```bash
kalam sql "SELECT path, updated_at FROM okf_sync.context_files ORDER BY path"
```

### Step 3 — Stop the server

Press **Ctrl+C** in the terminal running `kalam dev`.

This stops the sync worker and the local KalamDB process started by the CLI. Server data remains on disk under `kalam/server/data/`.

### Step 4 — Remove the local sync folder

```bash
rm -rf data/
```

You are deleting **only** the local OKF copy and its `.index/sync.db`. You are **not** deleting `kalam/server/data/` — that is where KalamDB kept the SQL rows **and** the uploaded file objects under `kalam/server/data/storage/` (see [FILE bytes on disk](#file-bytes-on-disk-not-in-sql)).

### Step 5 — Start again and confirm restore

```bash
kalam dev
```

Because the server still has rows (and `seed/` is skipped when the server is non-empty), the worker:

1. Starts with an empty `data/` (except what it creates while pulling)
2. **Initial pull** downloads each server file (`pullInitialRemoteFiles`)
3. Completes chokidar scan (usually no-op if nothing changed locally)
4. Opens **`liveTable`** — one subscription for all future updates

**Expected result:**

- `data/profile.md` contains your edit from step 2
- `data/notes/manual-test.md` exists with the same content
- `data/.index/sync.db` is recreated locally
- Console shows `file '…' downloaded from server (…)` for restored files

### Optional — offline edits while stopped

1. Run `kalam dev`, edit files, then stop with Ctrl+C
2. Add, edit, or delete files under `data/` while the worker is stopped
3. Run `kalam dev` again

**Expected result:** initial sync logs show added/updated/deleted counts; changed files are pushed before live pull starts.

If restore returns **old** content after multiple edits, ensure you are on a KalamDB build that includes the PK/MVCC read-path fixes (latest `main` or current release candidate).

## Automated tests

Unit tests (no server):

```bash
npm test
```

Integration tests (server must be reachable on port 2900):

```bash
# Terminal 1
kalam dev

# Terminal 2
KALAM_INTEGRATION=1 npm test
```

Notable integration cases:

- **`delete local folder and restore from database`** — automates the manual restore walkthrough
- **`local changes while sync stopped are pushed on restart`** — add / edit / delete while `kalam dev` is stopped

## Files worth reading

| File | Purpose |
|------|---------|
| `kalam/schema.sql` | USER table definition — only schema source |
| `kalam.toml` | `kalam dev` wiring: schema, types, sync process |
| `src/sync-app.ts` | Tiny CLI entrypoint: resolve folder + connection, then start the engine |
| `src/sync-engine.ts` | Startup order, `startLivePull`, `handleLiveRows`, tombstones, shared `TaskQueue` |
| `src/folder-watcher.ts` | chokidar with `ignoreInitial: false` |
| `src/remote-files.ts` | ORM upsert (`onConflictDoUpdate`) + delete + download |
| `src/local-cache.ts` | SQLite hashes and pending upload queue |
| `src/helpers.ts` | Disk I/O, waits, serialized task queue |
| `src/sync-log.ts` | Structured console logging |
| `src/lib/seed.ts` | First-run copy from `seed/` |
| `src/lib/paths.ts` | Safe paths, ignore rules (`.index`, `.git`, etc.) |
| `src/lib/file-utils.ts` | sha256, mime types, upload `File` builder |
| `src/models/schema.generated.ts` | Generated ORM types for KalamDB |
| `src/models/schema.local.ts` | Drizzle types for local SQLite |

## KalamDB surface area used

| Feature | Usage in this example |
|---------|----------------------|
| `FILE` column | Store OKF bytes; `file_ref.sha256` for change detection |
| `USER TABLE` | Per-user rows and download ACLs; live query scoped to sync user |
| MVCC rows | Each update = new row version; live materialization reads latest visible rows |
| **`liveTable()` (ORM)** | **One subscription** → typed `ContextFiles[]` on every table change |
| `client.live()` | Underlying transport; ORM compiles SQL + `mapRow` + primary key |
| Shared WebSocket | First live call opens `/v1/ws`; additional queries multiplex on same socket |
| Drizzle ORM | Reads, deletes, upsert (`onConflictDoUpdate`), and live table from one schema |
| `@kalamdb/orm` `kalamFile()` + `kalamDriver()` | FILE upserts through normal Drizzle inserts |
| `kalam dev` | Schema apply, typegen, local server, process supervisor |

### End-to-end live path (reference)

```mermaid
sequenceDiagram
  participant Disk as Local folder
  participant App as Sync engine
  participant ORM as liveTable / Drizzle
  participant WS as KalamDB WebSocket
  participant DB as okf_sync.context_files

  Note over App: startLivePull()
  App->>ORM: liveTable(client, context_files, cb)
  ORM->>WS: SUBSCRIBE SELECT * FROM context_files
  WS->>DB: register live query (USER scope)
  DB-->>WS: initial_data_batch + changes
  WS-->>ORM: materialize rows by path PK
  ORM-->>App: callback(ContextFiles[])
  App->>App: syncQueue → handleLiveRows
  alt path removed from rows
    App->>Disk: unlink file
  else path added or file_ref changed
    App->>DB: GET file via FileRef URL
    App->>Disk: write bytes
  end
```

## Next steps

- [Simple Realtime Mode](https://kalamdb.org/docs/ts-sdk/subscriptions-simple) — `live()` / `liveTable()` materialized row sets
- [TypeScript ORM liveTable](https://kalamdb.org/docs/ts-sdk/orm#live-table-helpers) — Drizzle table → subscription SQL
- [File datatype](https://kalamdb.org/docs/server/sql-reference/file-datatype) — how bytes are stored and downloaded
- [Subscriptions](https://kalamdb.org/docs/server/sql-reference/subscriptions) — server-side live query engine
- [Live query architecture](https://kalamdb.org/docs/server/architecture/live-query) — MVCC + push to subscribers
- [kalam dev](https://kalamdb.org/docs/cli/dev) — one-command local workflow
- [React AI Chat](https://kalamdb.org/docs/use-cases/react-ai-chat) — browser client on the same live patterns

================================================================================
# React AI Chat
URL: https://kalamdb.org/docs/use-cases/react-ai-chat
Source: content/use-cases/react-ai-chat.mdx

> Full chat application with @kalamdb/react — kalam dev workflow, conversations, streaming tokens, file attachments, and approval actions.
# React AI Chat

A chat-application validation surface for `@kalamdb/react`. Browser code lives in `src/app`, the topic worker in `src/agent`.

**Source:** [`examples/react-ai-chat/`](https://github.com/kalamdb/KalamDB/tree/main/examples/react-ai-chat)

## What it does

- Multi-conversation chat UI with live message lists
- WebSocket-confirmed sends and streamed typing tokens
- File attachments and human approval actions
- Deterministic topic-agent worker (no external AI API required)
- `kalam.toml` project with `kalam/schema.sql`, migrations, and supervised `kalam dev` processes

## Prerequisites

- Node.js 18+
- Kalam CLI **v0.5.3-rc.1** or newer (`kalam self-update` or build from source)

## Quick start

```bash
cd examples/react-ai-chat
npm install
kalam dev
```

Open the Vite URL, usually `http://127.0.0.1:5176`.

`kalam dev` starts the local KalamDB server, applies migrations, regenerates TypeScript schema types, and runs both configured processes:

```toml
[dev.processes]
app = "npm run dev"
agent = "npm run agent"
```

Default local credentials are `root` / `kalamdb123` at `http://127.0.0.1:2900`, matching `kalam/server/server.toml`.

**Access the running server** (while `kalam dev` is up):

- **Admin UI:** [http://localhost:2900/ui](http://localhost:2900/ui) — log in as `root` / `kalamdb123`
- **CLI:** `kalam --url http://127.0.0.1:2900 --user root --password kalamdb123`

Override credentials for app processes with `KALAM_*` / `VITE_KALAM_*` env vars (see below).

| Process | Environment variables |
|---------|----------------------|
| Agent | `KALAM_URL`, `KALAM_USER`, `KALAM_PASSWORD` |
| Browser (Vite) | `VITE_KALAM_URL`, `VITE_KALAM_USER`, `VITE_KALAM_PASSWORD` |

## Start fresh

Stop `kalam dev`, then:

```bash
kalam db reset
kalam dev
```

This removes `kalam/server/data` and `kalam/server/logs` so the next dev session starts with an empty database. Migration files and `server.toml` are kept.

## What to try

Send a message like:

```text
customer refund needs approval
```

The agent creates approval rows, streams typing tokens, and continues after you approve or reject.

**Browser-only demo mode:** set `VITE_KALAM_DEMO_MODE=true` in `.env.local` and run `npm run dev` without `kalam dev`.

## Files worth reading

| File | Purpose |
|------|---------|
| `kalam.toml` | `kalam dev` config: schema path, codegen, dev processes |
| `kalam/schema.sql` | Namespace, tables, topics, and seed data |
| `src/app/App.tsx` | `KalamProvider` and `LiveQueries` orchestration |
| `src/app/components/Conversation.tsx` | Message list and composer layout |
| `src/agent/index.ts` | Deterministic topic-agent worker |

## Verify it works

```bash
npm run build
npm test
```

With `kalam dev` running in another terminal:

```bash
npm run test:e2e
```

## Next steps

- [Live OKF Context Sync](https://kalamdb.org/docs/use-cases/live-okf-context-sync) — sync Google Open Knowledge Format (OKF) Markdown across devices and agents
- [Local Development](https://kalamdb.org/docs/cli/dev) — full `kalam dev` reference
- [Migrations](https://kalamdb.org/docs/cli/migrations) — `kalam db migrate` and `kalam db reset`
- [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai) — smaller ask/reply example without `@kalamdb/react`
- [TypeScript SDK setup](https://kalamdb.org/docs/ts-sdk/setup)

================================================================================
# Realtime Ops Feed
URL: https://kalamdb.org/docs/use-cases/realtime-ops-feed
Source: content/use-cases/realtime-ops-feed.mdx

> Smallest TypeScript browser example. SQL reads, live subscriptions, and browser auth in one React component.
# Realtime Ops Feed

The smallest useful browser example for the TypeScript SDK — one React component, no wrapper layers.

**Source:** [`examples/simple-typescript/`](https://github.com/kalamdb/KalamDB/tree/main/examples/simple-typescript)

## What it does

- Reads rows with `queryAll()`
- Subscribes to live updates with `live()`
- Authenticates in the browser with `Auth.basic()` and the SDK's built-in JWT exchange

Open the app in two browser tabs, add an event in one tab, and the second tab updates immediately.

## Prerequisites

- Node.js 18+
- Kalam CLI

## Quick start

```bash
cd examples/simple-typescript
npm install
kalam dev
```

Open `http://127.0.0.1:5173`.

`kalam dev` starts the local server, applies `kalam/schema.sql`, keeps `kalam/migrations/` current, regenerates TypeScript schema artifacts, and runs the Vite app.

Default local credentials are `root` / `kalamdb123`.

## What to try

1. Open the app in two tabs side by side.
2. Submit a new event in one tab.
3. Watch the same row appear in the other tab without refreshing.

## Files worth reading

| File | Purpose |
|------|---------|
| `src/App.tsx` | Complete example — query, live subscription, and form submit |
| `kalam.toml` | `kalam dev` project config |
| `kalam/schema.sql` | Namespace, table, and seed rows |
| `kalam/migrations/0001_init.sql` | Initial migration |

## Verify it works

```bash
npm test
```

The Playwright test opens two pages, submits an event, and confirms both tabs show the same live row.

## Troubleshooting

**Cannot connect to KalamDB**

```bash
curl http://127.0.0.1:2900/health
```

**Missing credentials**

Use the local defaults from `kalam dev`: `root` / `kalamdb123`, or copy `.env.example` to `.env.local` and adjust it.

## Next steps

- [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai) — add a topic worker and agent replies
- [TypeScript SDK setup](https://kalamdb.org/docs/ts-sdk/setup)

================================================================================
# Summarizer Agent
URL: https://kalamdb.org/docs/use-cases/summarizer-agent
Source: content/use-cases/summarizer-agent.mdx

> Worker-only TypeScript example. Consume topic events, enrich rows with summaries, and persist failures.
# Summarizer Agent

The worker-only example in the set — no browser UI, just a topic consumer that enriches database rows.

**Source:** [`examples/summarizer-agent/`](https://github.com/kalamdb/KalamDB/tree/main/examples/summarizer-agent)

There is no model dependency. The summarizer is deterministic so the example is easy to read and the test is stable.

## What it does

1. A row lands in `blog.blogs`
2. KalamDB routes the change into `blog.summarizer`
3. The worker consumes the topic with `runConsumer()`
4. The worker writes `summary` back into the same row
5. Failures are stored in `blog.summary_failures`

Use `change.data` as the blog row inside `onChange()`. Because `blog.blogs` is a `SHARED` table, `change.user` is `undefined` in this example.

## Prerequisites

- Node.js 18+
- Kalam CLI

## Quick start

```bash
cd examples/summarizer-agent
npm install
kalam dev
```

`kalam dev` starts the local server, applies `kalam/schema.sql`, keeps `kalam/migrations/` current, and runs the worker with `npm run start`.

## Schema and topic route

```sql
CREATE NAMESPACE IF NOT EXISTS blog;

CREATE SHARED TABLE IF NOT EXISTS blog.blogs (
    blog_id BIGINT PRIMARY KEY DEFAULT SNOWFLAKE_ID(),
    content TEXT NOT NULL,
    summary TEXT,
    created TIMESTAMP NOT NULL DEFAULT NOW(),
    updated TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE SHARED TABLE IF NOT EXISTS blog.summary_failures (
    run_key TEXT PRIMARY KEY,
    blog_id TEXT NOT NULL,
    error TEXT NOT NULL,
    created TIMESTAMP NOT NULL DEFAULT NOW(),
    updated TIMESTAMP NOT NULL DEFAULT NOW()
);

CREATE TOPIC blog.summarizer;
ALTER TOPIC blog.summarizer ADD SOURCE blog.blogs ON INSERT WITH (payload = 'full');
ALTER TOPIC blog.summarizer ADD SOURCE blog.blogs ON UPDATE WITH (payload = 'full');
```

## Worker pattern

```typescript
await runConsumer({
    client,
    name: 'summarizer',
    topic: 'blog.summarizer',
    groupId: 'summarizer',
    onChange: async (_ctx, change) => {
        const row = change.data;
        const summary = `Summary: ${row.content.slice(0, 80)}`;
        // UPDATE blog.blogs SET summary = ... WHERE blog_id = row.blog_id
    },
});
```

## Files worth reading

| File | Purpose |
|------|---------|
| `src/agent.ts` | Full worker logic |
| `kalam.toml` | `kalam dev` project config |
| `kalam/schema.sql` | Schema, topic route, and seed row |
| `kalam/migrations/0001_init.sql` | Initial migration |

## Verify it works

```bash
npm test
```

The integration test inserts a blog row and waits until the worker writes the summary field back.

## Next steps

- [Topic Consumer (Rust)](https://kalamdb.org/docs/rust-sdk/examples/topic-consumer) — same pattern in Rust
- [Chat With AI](https://kalamdb.org/docs/use-cases/chat-with-ai) — topic worker plus browser UI
- [TypeScript agents guide](https://kalamdb.org/docs/ts-sdk/agents)