Skip to Content
ArchitectureFile - Upload & DataType

File - Upload & DataType

KalamDB separates file metadata from file bytes:

  • SQL rows store a FileRef JSON payload in FILE columns (id, size, sha256, mime, subfolder, original name).
  • Binary content is written as a real object in the configured storage backend — a file on the local filesystem or an object in S3, GCS, or Azure Blob Storage.

This is not a BLOB-in-the-row model like PostgreSQL BYTEA, MySQL BLOB, or SQLite BLOB. KalamDB never stores file bytes inside RocksDB row values or inside Parquet column pages. Queries return metadata; downloads stream bytes from the storage path the FileRef points at.

This allows database queries and file transfer to scale independently while preserving table-level security.

File storage follows table-type scope. For the full USER/SHARED/STREAM capability matrix, see /docs/server/architecture/table-types.

Upload and Read Flow

  1. Client sends SQL using FILE("placeholder").
  2. Client attaches multipart file part as file:<placeholder>.
  3. KalamDB validates auth and table type (USER / SHARED only).
  4. Binary is staged briefly, then finalized as a standalone object under the table’s storage path (see Physical storage layout).
  5. The hot-tier row stores a FileRef JSON payload used for future reads/downloads.

For endpoint details, see /docs/server/api/http-reference.

Physical storage layout

FILE uploads land in the base_directory of the table’s assigned storage (from system.storages, default local). For a typical local dev server that directory is under storage.data_path:

TEXT
{data_path}/storage/          # default base_directory for the local storage├── {namespace}/{tableName}/           # SHARED table root│   └── f0001/1234567890123456789-contract.pdf└── {namespace}/{tableName}/{userId}/  # USER table root (per-user scope)    └── f0001/1234567890123456789-profile.md

With the default server templates (shared_tables_template, user_tables_template) and data_path = "./data":

Table typeTemplateExample relative path under storage/
SHARED{namespace}/{tableName}app/documents/f0001/1234567890123456789-contract.pdf
USER{namespace}/{tableName}/{userId}okf_sync/context_files/9876543210987654321/f0001/1234567890123456789-profile.md

Each upload gets:

  1. A snowflake file id (used in the on-disk filename).
  2. A subfolder such as f0001 (rotates as folders fill; configurable via max_files_per_folder).
  3. A stored filename {id}-{sanitized-original-name}.{ext} built from the FileRef.

The full object key / filesystem path is:

TEXT
{base_directory}/{resolved_template}/{subfolder}/{stored_filename}

Remote storages use the same relative layout inside the bucket or container prefix configured in base_directory (for example s3://my-bucket/kalam/).

What lives in SQL vs what lives on disk

LayerHoldsExample
RocksDB hot rowFileRef JSON in the FILE column{"id":"123…","sub":"f0001","name":"profile.md","size":142,"mime":"text/markdown","sha256":"a1b2…"}
Parquet cold segment (after flush)Same FileRef JSON in the column — not the bytesMetadata only
Storage object storeActual file bytes…/okf_sync/context_files/{userId}/f0001/123…-profile.md

Clients download through GET /v1/files/{namespace}/{table}/{sub}/{stored_filename} (or SDK helpers built on that path). The handler reads the object from storage; it does not decode bytes from a SQL BLOB column.

End-to-end finalize path (backend)

From kalamdb-filestore + SQL file handlers:

  1. Multipart upload bytes are written to a temporary staging directory per request.
  2. finalize_file generates a FileRef, computes sha256/mime, allocates a subfolder, and puts the bytes to the table’s storage backend via the object-store API.
  3. The staging directory is removed; only the permanent object and the row’s FileRef remain.
  4. On row delete or FILE replacement, the old object can be garbage-collected separately from MVCC row versions.

See the Live OKF Context Sync use case for a concrete walkthrough with kalam dev paths you can inspect on disk.

USER vs SHARED Pathing Model

KalamDB routes file bytes according to table type:

  • USER tables: file objects are stored under the authenticated user’s tenant scope.
  • SHARED tables: file objects are stored under shared table scope.

This matches KalamDB’s tenant-aware architecture:

  • user-private data stays isolated in user-tenant paths
  • shared assets are accessible through shared-table policy

SYSTEM and STREAM table paths are not valid for file APIs.

Backend-Verified Path Resolution

From backend implementation (kalamdb-filestore + API handlers):

  • For USER tables, file operations are resolved with user_id context.
  • For SHARED tables, file operations are resolved without user_id context.
  • Download handler enforces this behavior and rejects STREAM/SYSTEM file access.

Default storage templates are:

  • shared_tables_template = "{namespace}/{tableName}"
  • user_tables_template = "{namespace}/{tableName}/{userId}"

So for a table chat.attachments:

  • shared object example: chat/attachments/f0001/1234567890123456789-photo.png
  • user object example: chat/attachments/<userId>/f0001/1234567890123456789-photo.png

Templates are configurable per storage with CREATE STORAGE / ALTER STORAGE, but the user-scoped resolution model remains the same. Changing templates affects new path resolution; existing objects keep their stored paths via the FileRef recorded in each row.

Architecture Diagram

Why This Design Helps

  • Keeps tenant boundaries explicit for uploads and reads.
  • Avoids mixing user-private files with shared table assets.
  • Lets applications query metadata with SQL while streaming bytes through file endpoints.
  • Object-store friendly: bytes behave like normal files or S3 keys, so backup, CDN offload, and lifecycle policies apply to file content without rewriting SQL dumps.
  • Smaller hot/cold SQL tiers: Parquet segments and RocksDB hold references and queryable fields, not multi-megabyte blobs per row.
Last updated on
Stream StorageClustering & HA