State storage — LMDB or Postgres

CocoIndex persists target states and memoization results so it can detect changes and apply minimal updates between runs. Pick LMDB (zero-config, local) or Postgres (shared, managed) by the form of `db_path`, tune via `LmdbSettings` / `PgSettings`.

Version

v 1.0.14

CocoIndex persists target states and memoization results so it can detect what changed between runs and apply only the necessary updates. This page covers where that state lives and how to configure the backend.

Backend selection

The backend is selected by the form of db_path:

The simplest way to set db_path is the COCOINDEX_DB environment variable — it accepts either form:

# LMDB — local file (default)
export COCOINDEX_DB=./cocoindex.db

# Postgres — any DSN tokio-postgres accepts
export COCOINDEX_DB=postgres://cocoindex:[email protected]:5432/coco

Programmatic equivalents (override the env var):

# In a lifespan function
@coco.lifespan
def coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:
    builder.settings.db_path = "postgres://localhost/coco"
    yield

# Or directly when constructing Settings
settings = coco.Settings(db_path="postgres://localhost/coco")

If neither db_path nor COCOINDEX_DB is set, CocoIndex raises an error at Environment construction time.

Picking a backend

For most local development and single-machine production deployments, LMDB is the right choice — it has no operational overhead and is fast. Switch to Postgres when you want the engine state to live in a managed external database (centralized backup, separate from the host filesystem, etc.).

LMDB

The default. State lives in a local LMDB directory.

Configuration

The simplest setup — just point COCOINDEX_DB (or Settings.db_path) at any filesystem path:

export COCOINDEX_DB=./cocoindex.db

Programmatic:

@coco.lifespan
def coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:
    builder.settings.db_path = pathlib.Path("./cocoindex.db")
    yield

# or
settings = coco.Settings(db_path=pathlib.Path("./cocoindex.db"))

Setting db_path in the lifespan or Settings takes precedence over COCOINDEX_DB.

Tuning — LmdbSettings

The LMDB backend has two tunable knobs, grouped under coco.LmdbSettings and attached to Settings as db_settings. The defaults work well for most use cases.

When to adjust:

• map_size — most deployments never need to touch it. The map grows on its own and each step is cheap, so it is worth setting only when that growth actually costs you something: a large state re-climbing from 4 GiB on every restart, or resize pauses you can measure. If you do size it up, every enlargement logs LMDB map full, auto-resizing to N bytes and retrying — the largest N you see is the number to use. On 64-bit systems, map_size is a virtual address space reservation, so setting it larger than needed is safe and consumes neither physical memory nor disk.
• max_dbs — raise it if you have an unusually large number of apps sharing a single database directory.

Via environment variables:

export COCOINDEX_LMDB_MAP_SIZE=8589934592   # 8 GiB
export COCOINDEX_LMDB_MAX_DBS=2048

Or programmatically:

settings = coco.Settings(
    db_path=pathlib.Path("./cocoindex.db"),
    db_settings=coco.LmdbSettings(
        map_size=8 * 1024 * 1024 * 1024,  # 8 GiB
        max_dbs=2048,
    ),
)

Settings.from_env() picks up COCOINDEX_LMDB_MAX_DBS / COCOINDEX_LMDB_MAP_SIZE automatically.

Postgres (Plus)

Use the Postgres backend when you want CocoIndex’s internal state in an external, managed database.

Configuration

Set db_path to a Postgres connection URL. Any DSN that tokio-postgres accepts works — including host, port, user, password, dbname, sslmode, and query-string options:

# Via env var
export COCOINDEX_DB=postgres://cocoindex:[email protected]:5432/coco

# With TLS
export COCOINDEX_DB="postgres://cocoindex:[email protected]:5432/coco?sslmode=require"

# Unix-socket / passwordless local
export COCOINDEX_DB=postgresql:///coco

Programmatic:

@coco.lifespan
def coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:
    builder.settings.db_path = "postgres://cocoindex@db/coco"
    yield

# or
settings = coco.Settings(db_path="postgres://cocoindex@db/coco")

Either postgres:// or postgresql:// is recognized; everything else is treated as a filesystem path (LMDB).

Required Postgres privileges

On first use, CocoIndex creates an environment-level metadata table (cocoindex_meta) and per-app tables (app_<sanitized>_<id>__paths, __target, __idseq). The connecting role needs CREATE on the target schema (and USAGE if it’s not public):

-- Minimal setup
CREATE DATABASE coco;
CREATE ROLE cocoindex WITH LOGIN PASSWORD 'secret';
GRANT CONNECT ON DATABASE coco TO cocoindex;
\c coco
GRANT USAGE, CREATE ON SCHEMA public TO cocoindex;

CocoIndex handles table creation, migration (versioned via tables_version on cocoindex_meta), and per-app cleanup on App.drop_app(). The first call against a fresh database performs a one-time bootstrap that’s race-safe under concurrent first-time callers (advisory lock around CREATE TABLE IF NOT EXISTS).

Tuning — PgSettings

settings = coco.Settings(
    db_path="postgres://cocoindex@db/coco",
    db_settings=coco.PgSettings(pool_max_size=32, schema="cocoindex"),
)

schema can also be set via the COCOINDEX_PG_SCHEMA environment variable (picked up by coco.Settings.from_env()):

export COCOINDEX_DB=postgres://cocoindex@db/coco
export COCOINDEX_PG_SCHEMA=cocoindex

Putting CocoIndex’s tables in a dedicated schema keeps them out of public and makes them easy to inspect, back up, or drop as a unit. The connecting role needs CREATE on the database (to create the schema on first use) in addition to the privileges below.

Operational notes

• Schema layout. Per app: app_<sanitized_name>_<hex_id>__paths (tracking info, memoization, child existence, tombstones), __target (reverse-tracking ownership index), __idseq (per-app ID sequencer). Plus a single environment-wide cocoindex_meta ({key TEXT PRIMARY KEY, value JSONB}) holding app:<name> registrations.
• Cleanup. App.drop_app() drops all three per-app tables and removes the cocoindex_meta row in one SERIALIZABLE transaction.
• Concurrency. A single per-environment connection pool serves reads and writes. Concurrent writes are coalesced into a single BEGIN ISOLATION LEVEL SERIALIZABLE transaction; bodies share the transaction with inter-body flushing so the engine sees its own prior writes within the batch.
• Multi-process. A single process per database today. Multi-process support (multiple workers sharing a Postgres backend) is on the roadmap.

See also

• App — configuring db_path programmatically via a lifespan.
• Multiple environments — running isolated state stores side by side.