Live OKF Context Sync

Sync a Google Open Knowledge Format (OKF) context folder to KalamDB and back — so the same Markdown files are available on every device or agent process that runs the sync worker.

This example is intentionally small: one schema.sql, one Node sync process, and no custom file server. KalamDB stores file bytes as real objects under the server storage directory (not as SQL BLOBs); the sync worker watches a local folder with chokidar for local edits and mirrors server state with a single ORM live subscription for remote edits.

For the full FILE upload and on-disk layout model, see /docs/server/architecture/file-upload-datatype.

Showcase only. KalamDB does not ship folder sync as a core product feature. This app shows how little code you need to layer OKF context sync on top of SQL + FILE + liveTable() — one subscribe call, typed Drizzle rows, no polling loop.

Source: examples/live-okf-context-sync/

What is Google Open Knowledge Format (OKF)?

Google Open Knowledge Format (OKF) is a folder of Markdown files that agents and tools read as structured context: profiles, project notes, runbooks, policies, and preferences. Files often include YAML frontmatter so tools know what each document is for.

Teams using OKF usually need the same folder everywhere the user works — laptop agent, cloud agent, CI job, or second machine. This example keeps one canonical copy on the server and mirrors it to disk on each client.

What this example includes

There is no web UI or demo agent in the current version. The focus is the sync loop itself: disk ↔ KalamDB ↔ one live query per client — no HTTP polling, no manual “refresh folder” timer.

Run it locally

Use three terminals: one for kalam dev, then two extra sync clients pointed at different local folders.

Terminal 1 — start KalamDB and the default sync worker

1cd examples/live-okf-context-sync2npm install3kalam dev

kalam dev starts the local KalamDB server, applies kalam/schema.sql, regenerates ORM types, and runs the configured sync worker on data/.

Wait until the logs include:

1[sync] initial sync for folder '.../data' completed: ...2[sync] live subscription active

Terminal 2 — start a second folder as Alice

1cd examples/live-okf-context-sync2npm run dev -- data1

This command starts only another sync worker. It reuses the KalamDB server that kalam dev started in terminal 1.

Terminal 3 — start a third folder as Alice

1cd examples/live-okf-context-sync2npm run dev -- data2

Now you have three local folders (data/, data1/, data2/) syncing against the same user-scoped table in KalamDB.

Try add, edit, and delete

Run these commands from a fourth terminal, or edit the files in your editor:

1cd examples/live-okf-context-sync2 3# Add a file in data1. It should appear in data/ and data2.4mkdir -p data1/notes5cat > data1/notes/two-client-test.md <<'EOF'6# Two-client test7 8This file was created in data1 and should sync through KalamDB.9EOF10 11# Wait until liveTable has delivered the file to data2.12while [ ! -f data2/notes/two-client-test.md ]; do13  sleep 0.214done15 16# Edit it from data2. data/ and data1 should receive the new content.17cat >> data2/notes/two-client-test.md <<'EOF'18 19Edited from data2.20EOF21 22# Wait until the edit has reached data/.23while ! { [ -f data/notes/two-client-test.md ] && grep -q "Edited from data2." data/notes/two-client-test.md; }; do24  sleep 0.225done26 27# Delete it from data/. data1 and data2 should remove it too.28rm data/notes/two-client-test.md

Watch the three sync terminals while you do this:

1[sync] pushed path='notes/two-client-test.md' _seq=123456789 size=96 B2[sync] file 'notes/two-client-test.md' downloaded from server (...)3[sync] file 'notes/two-client-test.md' deleted

That is the whole point of the example: one local folder change becomes a SQL row/file update, then liveTable() pushes the reconciled table state to every other sync process.

If a folder starts empty, it does not copy seed/ once the server already has files. It downloads the server files first, then starts watching for local changes.

One liveTable() subscription, full table state

The pull path is the heart of the example. After startup reconciliation, each sync worker opens exactly one live subscription:

1private async startLivePull(): Promise<void> {2  this.liveUnsub = await liveTable(this.client, context_files, (rows) => {3    this.syncQueue.enqueue(() => this.handleLiveRows(rows));4  });5  console.log('[sync] live subscription active');6}

That is the entire subscribe surface for server → disk updates. There is no second subscription for deletes, no per-file poll, and no setInterval against the REST API.

What KalamDB + @kalamdb/orm do under this call

context_files is a generated Drizzle table from kalam/schema.sql:

1export const context_files = kTable.user('context_files', {2  path: text('path').primaryKey(),3  file_ref: file('file_ref').notNull(),4  created_at: timestamp('created_at', { mode: 'date' }),5  updated_at: timestamp('updated_at', { mode: 'date' }),6});7export type ContextFiles = typeof context_files.$inferSelect;

When you call liveTable(client, context_files, callback):

1. SQL is compiled from the table definition — @kalamdb/orm builds SELECT * FROM okf_sync.context_files (namespace comes from the client / kalam.toml connection).
2. Row typing is applied — WASM/live events are mapped through Drizzle column metadata (file_ref → FileRef, timestamps → Date).
3. Primary key inference — path is the live reconciliation key (getKey: ['path']), so inserts, updates, and deletes merge into one materialized array in the Rust live core.
4. One WebSocket subscription — the first liveTable() on a client opens the shared /v1/ws socket; this query registers once on that connection (client lifecycle).
5. Callbacks receive the full current row set — not individual { op: 'delete' } events you must fold yourself in simple mode. Every change re-delivers all rows that match the query right now, already deduplicated by primary key.

Conceptually:

1  KalamDB                           @kalamdb/client + ORM              sync-engine.ts2  ───────                           ─────────────────────              ───────────3  INSERT / UPDATE / DELETE          live core materializes rows        handleLiveRows(rows)4  on okf_sync.context_files   ──►   mapRow → ContextFiles[]      ──►   diff paths → disk I/O5  (MVCC row versions)               callback(rows)                     (download / unlink)

This is Simple Realtime Mode: your application code works with rows[], the SDK applies initial_data_batch, insert, update, and delete internally. For a UI you render the array; for this example we diff path sets and mirror files.

Why a single subscribe is enough for folder sync

A naive design might subscribe per file path, poll GET /files/:id, or run a SQL query after every chokidar event. KalamDB instead keeps one table-level live query aligned with MVCC:

The worker stores the last known path set in remotePaths. On each callback:

1private async handleLiveRows(rows: ContextFiles[]): Promise<void> {2  const nextPaths = new Set(rows.map((row) => row.path).filter(isSafeSyncPath));3 4  // … skip empty first snapshot (SDK warmup) …5 6  for (const path of this.remotePaths) {7    if (!nextPaths.has(path)) {8      await this.removeLocalFile(path);   // live delete → local unlink9    }10  }11  this.remotePaths = nextPaths;12 13  for (const row of rows) {14    await this.pullRemoteRow(row);        // insert/update → download if hash differs15  }16}

Deletes are implied: if profile.md was in remotePaths but not in the latest rows, the server no longer has that row — remove the local Markdown file. You do not need a separate delete channel.

ORM vs raw SQL vs raw events

The OKF example uses liveTable because:

• Schema lives in SQL → kalam dev regenerates schema.generated.ts → the subscription always matches the applied server schema.
• ContextFiles types flow from ORM into pullRemoteRow without hand-written row parsers.
• file('file_ref') maps to KalamDB’s FILE column type, so each live row carries a typed FileRef for download URLs.

If you needed raw events (metrics, incremental audit log), you would drop down to client.liveEvents(). For “keep this folder equal to this table,” liveTable is the intended API.

Serializing live with push (syncQueue)

The live callback can fire while chokidar is mid-push. The example shares one TaskQueue between:

• chokidar onUpsert / onDelete → pushLocalFile / deleteRemoteFile
• liveTable callback → handleLiveRows

1// folder watcher enqueues onto the same queue passed from the sync engine2{ taskQueue: this.syncQueue }3 4// live pull enqueues too5this.syncQueue.enqueue(() => this.handleLiveRows(rows));

So a live snapshot never interleaves with a half-finished FILE upload. FILE columns are versioned with MVCC; serializing avoids applying a file_ref from row version N while version N+1 is still uploading.

Multi-client correctness (3+ sync folders)

Run the same example three times — data/, data1/, data2/ — all as alice on the same server. Each process has its own liveTable() subscription and its own syncQueue, but all subscribe to the same logical query scoped to alice’s USER table partition.

When one client deletes many files:

1. Local unlink → deleteRemoteFile → SQL DELETE on KalamDB.
2. Live core broadcasts a new materialized row set to every subscriber.
3. Other clients diff remotePaths → nextPaths and unlink locally.

The example adds tombstones (src/lib/sync-tombstones.ts) so a stale snapshot or a not-yet-updated folder cannot re-upload deleted bytes before the next live callback. That is example-level safety, not a KalamDB requirement — the important platform guarantee is still one query, one reconciled row array, pushed to all subscribers.

Architecture

1  data/                         KalamDB (okf_sync.context_files)2  ├── index.md                  ┌─────────────────────────────────┐3  ├── profile.md                │ path TEXT PRIMARY KEY           │4  └── .index/                   │ created_at, updated_at          │5      └── sync.db (local only)  └─────────────────────────────────┘6           │                              ▲7           │  push (chokidar + upsert)    │  pull (liveTable + download)8           └──────────────────────────────┘9                    sync-engine.ts

Two stores, one logical folder:

1. KalamDB — durable SQL metadata in RocksDB/Parquet plus file bytes as standalone objects under kalam/server/data/storage/. Each row’s file_ref column is a FileRef JSON pointer (id, sha256, size, mime, subfolder) — not the Markdown bytes themselves. Each INSERT/UPDATE creates a new MVCC row version with a new _seq; file_ref is a normal column value inside that row.
2. Local disk — Markdown files you edit, plus .index/sync.db for hashes and offline upload retries.

See FILE bytes on disk (not in SQL) for a sample storage path and how it differs from BLOB columns in other databases.

Startup order (current worker):

1. Pull — SELECT all remote rows once; download any missing files into an empty folder.
2. Push — chokidar initial scan + offline reconciliation; upsert local edits to KalamDB.
3. Subscribe — liveTable(context_files); all further server changes arrive through the single live callback.

Push-before-subscribe on the initial scan prevents a stale first snapshot from overwriting offline edits. Pull-before-push on empty local folders restores from the server without waiting for the first live batch.

FILE bytes on disk (not in SQL)

Most databases store uploaded content inside the row — PostgreSQL BYTEA, MySQL LONGBLOB, or similar. KalamDB does not. A FILE column holds a FileRef JSON document; the server writes the bytes to its configured storage folder or object store as a normal file.

Architecture reference: /docs/server/architecture/file-upload-datatype

What the sync worker sends

When you save data/profile.md, the worker upserts through Drizzle with kalamFile('upload', file). The driver sends multipart SQL; KalamDB finalizes the bytes under the table’s storage path and stores metadata in the row:

1await db2  .insert(context_files)3  .values({4    path: 'profile.md',5    file_ref: kalamFile('upload', upload),6    updated_at: now,7  })8  .onConflictDoUpdate({9    target: context_files.path,10    set: { file_ref: kalamFile('upload', upload), updated_at: now },11  });

What you see in SQL vs on disk

After a push, query the row:

1kalam sql "SELECT path, file_ref FROM okf_sync.context_files WHERE path = 'profile.md'"

The file_ref column looks like JSON metadata (exact id values vary per upload):

1{2  "id": "1234567890123456789",3  "sub": "f0001",4  "name": "profile.md",5  "size": 142,6  "mime": "text/markdown",7  "sha256": "7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069"8}

The Markdown bytes are not in that JSON. They live as a separate object. With kalam dev, inspect the local storage tree (alice’s numeric userId comes from login — list it with kalam sql "SELECT user_id FROM system.users WHERE username = 'alice'" if needed):

1examples/live-okf-context-sync/kalam/server/data/2├── rocksdb/                         # hot SQL rows (FileRef JSON, not file bytes)3└── storage/                         # FILE objects (actual bytes)4    └── okf_sync/context_files/{userId}/5        └── f0001/6            └── 1234567890123456789-profile.md   # real file on disk

You can open that file directly — it is the same content as data/profile.md on the client. Other clients download it through the file API using the FileRef (GET /v1/files/okf_sync/context_files/f0001/123…-profile.md), not by reading a BLOB out of a SELECT.

When you edit the same path again, MVCC appends a new row version with a new file_ref (often a new object path). Old objects may remain until cleanup; the live subscription always reconciles against the latest visible row set.

Schema

The whole server schema fits in one file:

1CREATE NAMESPACE IF NOT EXISTS okf_sync;2 3CREATE USER TABLE okf_sync.context_files (4  path TEXT PRIMARY KEY,5  file_ref FILE NOT NULL,6  created_at TIMESTAMP DEFAULT NOW(),7  updated_at TIMESTAMP DEFAULT NOW()8);

• path — relative OKF path (profile.md, notes/two-client-test.md).
• file_ref — server-managed FileRef JSON (id, sha256, size, mime, download path). Content hash lives on the ref; no duplicate sha256 column in SQL.
• USER TABLE — rows and file downloads are scoped per user (Alice cannot read Bob’s files).

kalam.toml sets generate_types = true, so kalam dev regenerates src/models/schema.generated.ts from this SQL via @kalamdb/orm.

Source layout

1examples/live-okf-context-sync/2  data/                         # default sync folder (gitignored)3    .index/4      sync.db                   # local SQLite — never synced5    index.md6    profile.md78  seed/                         # copied into data/ on first run only9    index.md10    profile.md1112  kalam/13    schema.sql                  # source of truth14    server/data/                # KalamDB server data (persists across restarts)1516  src/17    sync-app.ts                 # tiny runnable entrypoint18    sync-engine.ts              # startup order, push/pull/live orchestration19    folder-watcher.ts           # chokidar (ignoreInitial: false)20    remote-files.ts             # ORM upsert/delete/select + FILE download21    local-cache.ts              # SQLite metadata + pending queue22    helpers.ts                  # disk I/O, waits, TaskQueue23    sync-log.ts                 # structured console logging24    lib/25      paths.ts                  # safe paths, ignore rules26      seed.ts                   # first-run copy from seed/27      file-utils.ts             # sha256, mime type, upload File builder28      sync-tombstones.ts        # block stale re-upload/re-download after delete29    db/                         # Kalam client + local SQLite30    models/schema.generated.ts  # generated — do not edit31    models/schema.local.ts      # Drizzle types for local SQLite

Never synced: .index/, .git/, .DS_Store, SQLite WAL/SHM files.

How it works (detailed)

1. Startup (kalam dev)

kalam dev (from kalam.toml):

1. Starts KalamDB locally (kalam/server/data/)
2. Applies kalam/schema.sql and regenerates TypeScript types
3. Runs npm run dev -- data — the sync worker as user alice by default

On startup the worker:

1. Ensures demo users alice / bob exist
2. Logs in and loads existing remote paths from KalamDB (select().from(context_files))
3. Bootstrap: if data/ has no user files and the server has no rows, copies seed/ into data/
4. Initial pull: downloads server files missing on disk (empty folder / new sync dir restore)
5. Initial push: chokidar scans data/ with ignoreInitial: false, pushes changed files, reconciles offline deletions, flushes the pending upload queue
6. Live pull: one liveTable(context_files, …) subscription — see One liveTable subscription
7. Watch: chokidar keeps running; push/delete events share the same TaskQueue as live callbacks

Console output during initial sync:

1[sync] initial sync for folder '.../data' started ...2[sync] file 'profile.md' downloaded from server (142 B)3[sync] downloaded 1 file(s) from server4[sync] file 'profile.md' added5[sync] initial sync for folder '.../data' completed: 1 added, 0 updated, 0 deleted, 0 pushed, 1 downloaded (1 total changes)6[sync] live subscription active

2. Push — local edit → server

When you save or add a file under data/:

1. chokidar fires add or change; the watcher logs file '…' added or updated
2. pushLocalFile(path) reads bytes and computes SHA-256
3. If local SQLite cache matches and nothing is pending, skip (no-op)
4. Otherwise the worker upserts through Drizzle ORM with kalamFile('upload', file) in .values() / .onConflictDoUpdate().set()
5. KalamDB stores bytes as a standalone object under kalam/server/data/storage/ and appends a new hot MVCC row with updated file_ref JSON (see FILE bytes on disk)
6. Local SQLite records the hash; any pending-upload row is cleared
7. Console logs pushed path='…' _seq=… size=…

When you delete a file locally, chokidar fires unlink, logs file '…' deleted, and removes the row with db.delete(context_files).

If the server is down, the upload is queued in pending_uploads and retried on reconnect.

3. Offline edits (kalam dev stopped)

While the sync worker is not running you can add, edit, or delete files under data/. On the next kalam dev:

1. Missing server files are downloaded first (initial pull)
2. chokidar replays every file on disk (ignoreInitial: false) and pushes local changes
3. reconcileLocalDeletions deletes server rows for files removed while offline
4. liveTable starts last — runtime sync is live-only after this point

4. Pull — server change → local disk (live subscription)

After startLivePull(), every server-side change to context_files flows through the same liveTable callback:

1. SDK delivers ContextFiles[] — full materialized query result
2. Paths gone since last callback → removeLocalFile (live delete)
3. For each row, compare local SHA-256 with file_ref.sha256
4. Match → update local SQLite cache only
5. Differ → download via FileRef.getDownloadUrl(...), verify hash, write disk
6. Console logs file '…' downloaded from server (size) or file '…' deleted

The worker ignores the first empty callback (SDK warmup before initial_data_batch completes) so an empty rows array is never treated as “delete everything.”

See One liveTable subscription for the diff model and queue serialization.

5. Bootstrap rules

Deleting data/ does not delete server data. On the next start, the worker pulls everything back from KalamDB.

6. Per-user isolation

Default sync user is alice (KALAM_USER / KALAM_PASSWORD). User-scoped rows and FILE downloads are enforced by KalamDB. Run a second worker as Bob:

1KALAM_USER=bob KALAM_PASSWORD=bob123 npm run dev -- data-bob

Prerequisites

• Node.js 20+
• Kalam CLI with kalam dev (CLI dev workflow)

Single-folder quick start

1cd examples/live-okf-context-sync2npm install3kalam dev

kalam dev starts the server and the sync worker on data/. Edit Markdown under data/ and watch the console for push/pull lines.

For the more useful multi-client demo, keep kalam dev running and start npm run dev -- data1 plus npm run dev -- data2 in two more terminals as shown in Run it locally.

Default credentials:

Sync a different folder:

1npm run dev -- test1/2# or3KALAM_SYNC_DIR=test1 npm run dev

Verify it works manually

This is the most important end-to-end check: local folder is disposable; KalamDB is the source of truth.

Step 1 — Start with kalam dev

1cd examples/live-okf-context-sync2npm install3kalam dev

Wait until you see [sync] live subscription active in the logs.

Step 2 — Make changes under data/

In another terminal:

1cd examples/live-okf-context-sync2 3# Edit an existing file4echo "" >> data/profile.md5echo "Updated at $(date -u +%Y-%m-%dT%H:%M:%SZ)" >> data/profile.md6 7# Or add a new note8mkdir -p data/notes9cat > data/notes/manual-test.md <<'EOF'10# Manual restore test11 12This file should come back after deleting data/.13EOF

Confirm the sync worker logged push lines (for example pushed path='profile.md' _seq=… size=…). You can also query the server:

1kalam sql "SELECT path, updated_at FROM okf_sync.context_files ORDER BY path"

Step 3 — Stop the server

Press Ctrl+C in the terminal running kalam dev.

This stops the sync worker and the local KalamDB process started by the CLI. Server data remains on disk under kalam/server/data/.

Step 4 — Remove the local sync folder

1rm -rf data/

You are deleting only the local OKF copy and its .index/sync.db. You are not deleting kalam/server/data/ — that is where KalamDB kept the SQL rows and the uploaded file objects under kalam/server/data/storage/ (see FILE bytes on disk).

Step 5 — Start again and confirm restore

1kalam dev

Because the server still has rows (and seed/ is skipped when the server is non-empty), the worker:

1. Starts with an empty data/ (except what it creates while pulling)
2. Initial pull downloads each server file (pullInitialRemoteFiles)
3. Completes chokidar scan (usually no-op if nothing changed locally)
4. Opens liveTable — one subscription for all future updates

Expected result:

• data/profile.md contains your edit from step 2
• data/notes/manual-test.md exists with the same content
• data/.index/sync.db is recreated locally
• Console shows file '…' downloaded from server (…) for restored files

Optional — offline edits while stopped

1. Run kalam dev, edit files, then stop with Ctrl+C
2. Add, edit, or delete files under data/ while the worker is stopped
3. Run kalam dev again

Expected result: initial sync logs show added/updated/deleted counts; changed files are pushed before live pull starts.

If restore returns old content after multiple edits, ensure you are on a KalamDB build that includes the PK/MVCC read-path fixes (latest main or current release candidate).

Automated tests

Unit tests (no server):

1npm test

Integration tests (server must be reachable on port 2900):

1# Terminal 12kalam dev3 4# Terminal 25KALAM_INTEGRATION=1 npm test

Notable integration cases:

• delete local folder and restore from database — automates the manual restore walkthrough
• local changes while sync stopped are pushed on restart — add / edit / delete while kalam dev is stopped

Files worth reading

KalamDB surface area used

End-to-end live path (reference)

Next steps

• Simple Realtime Mode — live() / liveTable() materialized row sets
• TypeScript ORM liveTable — Drizzle table → subscription SQL
• File datatype — how bytes are stored and downloaded
• Subscriptions — server-side live query engine
• Live query architecture — MVCC + push to subscribers
• kalam dev — one-command local workflow
• React AI Chat — browser client on the same live patterns