Storage Commands

KalamDB supports multiple storage backends for cold-tier data. These commands manage storage configuration, health checks, and maintenance.

For practical table-level storage selection (STORAGE_ID, USE_USER_STORAGE, and verification queries), see Storage ID Usage. For S3-compatible local deployment setup, see MinIO (S3-Compatible).

CREATE STORAGE


CREATE STORAGE <storage_id>
  TYPE '<filesystem|s3|gcs|azure>'
  [NAME '<storage_name>']
  [DESCRIPTION '<description>']
  [PATH '<path>']
  [BUCKET '<bucket_or_s3_url>']
  [REGION '<region>']
  [BASE_DIRECTORY '<path_or_url>']
  [SHARED_TABLES_TEMPLATE '<template>']
  [USER_TABLES_TEMPLATE '<template>']
  [CREDENTIALS '<json_credentials>']
  [CONFIG '<json_config>'];

Examples

Local filesystem:


CREATE STORAGE local
  TYPE 'filesystem'
  PATH './data';

Amazon S3:


CREATE STORAGE s3_prod
  TYPE 's3'
  BUCKET 'my-bucket'
  REGION 'us-west-2'
  CREDENTIALS '{"access_key_id":"...","secret_access_key":"..."}';

Google Cloud Storage:


CREATE STORAGE gcs_prod
  TYPE 'gcs'
  BUCKET 'my-gcs-bucket'
  CREDENTIALS '{"service_account_key":"..."}';

Azure Blob Storage:


CREATE STORAGE azure_prod
  TYPE 'azure'
  BUCKET 'my-container'
  CREDENTIALS '{"account_name":"...","account_key":"..."}';

Storage Type Aliases

SQL Type	Alias	Base Directory Prefix
`filesystem`	—	Local path (e.g., `./data/storage`)
`s3`	—	`s3://bucket/prefix`
`gcs`	`gs`	`gs://bucket/prefix`
`azure`	`az`	`az://container/prefix`

CONFIG JSON Fields by Backend

S3 CONFIG


{
  "type": "s3",
  "region": "us-east-1",
  "endpoint": "http://127.0.0.1:9120",
  "allow_http": true,
  "access_key_id": "...",
  "secret_access_key": "...",
  "session_token": "..."
}

Field	Type	Default	Notes
`region`	string	`"us-east-1"`	AWS region or custom region for S3-compatible
`endpoint`	string	`null`	Custom endpoint for MinIO / S3-compatible
`allow_http`	boolean	`false`	Must be `true` for non-TLS endpoints
`access_key_id`	string	`null`	Static credentials (omit for IAM roles)
`secret_access_key`	string	`null`	Static credentials
`session_token`	string	`null`	Temporary session credentials

For S3-compatible local setup, see MinIO (S3-Compatible).

GCS CONFIG


{
  "type": "gcs",
  "service_account_json": "{...}"
}

Field	Type	Default	Notes
`service_account_json`	string	`null`	Service account JSON key. Omit to use Application Default Credentials (ADC).

Azure CONFIG


{
  "type": "azure",
  "account_name": "mystorageaccount",
  "access_key": "...",
  "sas_token": "..."
}

Field	Type	Default	Notes
`account_name`	string	`null`	Azure storage account name
`access_key`	string	`null`	Account access key
`sas_token`	string	`null`	Shared Access Signature token (alternative to access_key)

Local/Filesystem CONFIG


{
  "type": "local",
  "root": "./data/storage"
}

Field	Type	Default	Notes
`root`	string	`null`	Informational; actual path comes from `BASE_DIRECTORY` or `PATH`.

ALTER STORAGE


ALTER STORAGE <storage_id>
  [SET NAME '<new_name>']
  [SET DESCRIPTION '<new_description>']
  [SET SHARED_TABLES_TEMPLATE '<new_template>']
  [SET USER_TABLES_TEMPLATE '<new_template>']
  [SET CONFIG '<json_config>'];

DROP STORAGE


DROP STORAGE <storage_id>;
DROP STORAGE IF EXISTS <storage_id>;

SHOW STORAGES


SHOW STORAGES;

STORAGE CHECK

Verify storage backend health:


STORAGE CHECK <storage_id>;
STORAGE CHECK <storage_id> EXTENDED;

STORAGE FLUSH

Force flushing data from hot tier (RocksDB) to cold tier (Parquet):


-- Flush a specific table
STORAGE FLUSH TABLE <namespace>.<table_name>;
 
-- Flush all tables in a namespace
STORAGE FLUSH ALL IN <namespace>;
STORAGE FLUSH ALL IN NAMESPACE <namespace>;
 
-- Flush everything
STORAGE FLUSH ALL;

STORAGE COMPACT

Merge and optimize cold-tier Parquet segments:


-- Compact a specific table
STORAGE COMPACT TABLE <namespace>.<table_name>;
 
-- Compact all tables in a namespace
STORAGE COMPACT ALL IN <namespace>;
STORAGE COMPACT ALL IN NAMESPACE <namespace>;
 
-- Compact everything
STORAGE COMPACT ALL;

SHOW MANIFEST

Display the storage manifest (schema + segment index):


SHOW MANIFEST;