# State storage — LMDB or Postgres

> **CocoIndex v1.** This page documents CocoIndex **v1** — a ground-up redesign from v0. When writing code, ignore any v0 flow-builder DSL or deprecated decorators.
>
> Source: https://cocoindex.io/docs/programming_guide/state_storage/ · Docs index: https://cocoindex.io/docs/llms.txt · Agent skill: https://cocoindex.io/docs/skill.md
>
> v0→v1 quick map — if you reach for these v0 symbols, stop and use the v1 form: `@cocoindex.flow_def`/`FlowBuilder` → `coco.App` + a `@coco.fn` main function; `add_collector()`/`collect()`/`export()` → declare target states (`declare_row`, `declare_file`); `cocoindex.sources/functions/targets.*` → connector APIs (`localfs.walk_dir`, `coco.ops.*`, `postgres.declare_table_target`). Full mapping + API reference: https://cocoindex.io/docs/skill.md.

CocoIndex persists *target states* and *memoization results* so it can
detect what changed between runs and apply only the necessary updates.
This page covers where that state lives and how to configure the
backend.

## Backend selection

The backend is selected by the form of `db_path`:

| `db_path` value | Backend |
|---|---|
| Filesystem path or `pathlib.Path` (e.g. `./cocoindex.db`) | **LMDB** (embedded, default) |
| `postgres://…` or `postgresql://…` URL | **Postgres** |

The simplest way to set `db_path` is the `COCOINDEX_DB` environment
variable — it accepts either form:

```bash
# LMDB — local file (default)
export COCOINDEX_DB=./cocoindex.db

# Postgres — any DSN tokio-postgres accepts
export COCOINDEX_DB=postgres://cocoindex:secret@db.internal:5432/coco
```

Programmatic equivalents (override the env var):

```python
# In a lifespan function
@coco.lifespan
def coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:
    builder.settings.db_path = "postgres://localhost/coco"
    yield

# Or directly when constructing Settings
settings = coco.Settings(db_path="postgres://localhost/coco")
```

If neither `db_path` nor `COCOINDEX_DB` is set, CocoIndex raises an
error at `Environment` construction time.

### Picking a backend

|  | LMDB (default) | Postgres |
|---|---|---|
| **Setup** | Zero-config — a directory on disk | Provision a Postgres database, set the DSN |
| **Process model** | Embedded, single-process | Single-process today; multi-process planned |
| **Storage location** | Local filesystem | External RDBMS |
| **Scaling ceiling** | Bounded by `map_size` (virtual address space, default 4 GiB) | Bounded by the Postgres instance |
| **Best for** | Local development, single-machine pipelines, CI | Shared deployments, managed RDBMS environments, audit-friendly storage |
| **Tuning struct** | `coco.LmdbSettings` | `coco.PgSettings` |

For most local development and single-machine production deployments,
LMDB is the right choice — it has no operational overhead and is fast.
Switch to Postgres when you want the engine state to live in a managed
external database (centralized backup, separate from the host
filesystem, etc.).

## LMDB

The default. State lives in a local [LMDB](http://www.lmdb.tech/doc/)
directory.

### Configuration

The simplest setup — just point `COCOINDEX_DB` (or
`Settings.db_path`) at any filesystem path:

```bash
export COCOINDEX_DB=./cocoindex.db
```

Programmatic:

```python
@coco.lifespan
def coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:
    builder.settings.db_path = pathlib.Path("./cocoindex.db")
    yield

# or
settings = coco.Settings(db_path=pathlib.Path("./cocoindex.db"))
```

Setting `db_path` in the lifespan or `Settings` takes precedence over
`COCOINDEX_DB`.

### Tuning — `LmdbSettings`

The LMDB backend has two tunable knobs, grouped under
`coco.LmdbSettings` and attached to `Settings` as `db_settings`. The
defaults work well for most use cases.

| Setting | Default | Env Variable | Description |
|---------|---------|-------------|-------------|
| `max_dbs` | `1024` | `COCOINDEX_LMDB_MAX_DBS` | Maximum number of named LMDB databases. Must be ≥ 1. |
| `map_size` | `4294967296` (4 GiB) | `COCOINDEX_LMDB_MAP_SIZE` | Maximum size of the LMDB memory map in bytes. Must be > 0. |

When to adjust:

- **Increase `map_size`** if you encounter LMDB "map full" errors. On
  64-bit systems, `map_size` is a virtual address space reservation —
  setting it larger than needed is safe and does not consume physical
  memory.
- **Increase `max_dbs`** if you have an unusually large number of apps
  sharing a single database directory.

Via environment variables:

```bash
export COCOINDEX_LMDB_MAP_SIZE=8589934592   # 8 GiB
export COCOINDEX_LMDB_MAX_DBS=2048
```

Or programmatically:

```python
settings = coco.Settings(
    db_path=pathlib.Path("./cocoindex.db"),
    db_settings=coco.LmdbSettings(
        map_size=8 * 1024 * 1024 * 1024,  # 8 GiB
        max_dbs=2048,
    ),
)
```

`Settings.from_env()` picks up `COCOINDEX_LMDB_MAX_DBS` /
`COCOINDEX_LMDB_MAP_SIZE` automatically.

**Note — Legacy keyword arguments**
For backward compatibility, `Settings` still accepts `lmdb_max_dbs`
and `lmdb_map_size` as keyword arguments, and exposes them as
attributes (e.g., `settings.lmdb_map_size = ...`). These read and
write the same underlying values as
`settings.db_settings.max_dbs` / `settings.db_settings.map_size`.
Passing both `db_settings=` and the legacy keywords in the same
`Settings(...)` call raises `ValueError`.

## Postgres (Plus)

Use the Postgres backend when you want CocoIndex's internal state in
an external, managed database.

### Configuration

Set `db_path` to a Postgres connection URL. Any DSN that
[`tokio-postgres`](https://docs.rs/tokio-postgres/) accepts works —
including `host`, `port`, `user`, `password`, `dbname`, `sslmode`,
and query-string options:

```bash
# Via env var
export COCOINDEX_DB=postgres://cocoindex:secret@db.internal:5432/coco

# With TLS
export COCOINDEX_DB="postgres://cocoindex:secret@db.internal:5432/coco?sslmode=require"

# Unix-socket / passwordless local
export COCOINDEX_DB=postgresql:///coco
```

Programmatic:

```python
@coco.lifespan
def coco_lifespan(builder: coco.EnvironmentBuilder) -> Iterator[None]:
    builder.settings.db_path = "postgres://cocoindex@db/coco"
    yield

# or
settings = coco.Settings(db_path="postgres://cocoindex@db/coco")
```

Either `postgres://` or `postgresql://` is recognized; everything else
is treated as a filesystem path (LMDB).

### Required Postgres privileges

On first use, CocoIndex creates an environment-level metadata table
(`cocoindex_meta`) and per-app tables (`app_<sanitized>_<id>__paths`,
`__target`, `__idseq`). The connecting role needs `CREATE` on the
target schema (and `USAGE` if it's not `public`):

```sql
-- Minimal setup
CREATE DATABASE coco;
CREATE ROLE cocoindex WITH LOGIN PASSWORD 'secret';
GRANT CONNECT ON DATABASE coco TO cocoindex;
\c coco
GRANT USAGE, CREATE ON SCHEMA public TO cocoindex;
```

CocoIndex handles table creation, migration (versioned via
`tables_version` on `cocoindex_meta`), and per-app cleanup on
`App.drop_app()`. The first call against a fresh database performs a
one-time bootstrap that's race-safe under concurrent first-time
callers (advisory lock around `CREATE TABLE IF NOT EXISTS`).

### Tuning — `PgSettings`

| Setting | Default | Description |
|---------|---------|-------------|
| `pool_max_size` | `16` | Maximum number of connections held open by the connection pool. |
| `schema` | `None` | Postgres schema (namespace) to hold CocoIndex's tables. When set, every connection's `search_path` is pinned to it and the schema is created on first use (`CREATE SCHEMA IF NOT EXISTS`). When unset, tables land in the connection's default `search_path` (usually `public`). Must be a simple lowercase identifier. |

```python
settings = coco.Settings(
    db_path="postgres://cocoindex@db/coco",
    db_settings=coco.PgSettings(pool_max_size=32, schema="cocoindex"),
)
```

`schema` can also be set via the `COCOINDEX_PG_SCHEMA` environment
variable (picked up by `coco.Settings.from_env()`):

```bash
export COCOINDEX_DB=postgres://cocoindex@db/coco
export COCOINDEX_PG_SCHEMA=cocoindex
```

Putting CocoIndex's tables in a dedicated schema keeps them out of
`public` and makes them easy to inspect, back up, or drop as a unit. The
connecting role needs `CREATE` on the database (to create the schema on
first use) in addition to the privileges below.

### Operational notes

- **Schema layout.** Per app: `app_<sanitized_name>_<hex_id>__paths`
  (tracking info, memoization, child existence, tombstones),
  `__target` (reverse-tracking ownership index), `__idseq` (per-app
  ID sequencer). Plus a single environment-wide `cocoindex_meta`
  (`{key TEXT PRIMARY KEY, value JSONB}`) holding `app:<name>`
  registrations.
- **Cleanup.** `App.drop_app()` drops all three per-app tables and
  removes the `cocoindex_meta` row in one `SERIALIZABLE` transaction.
- **Concurrency.** A single per-environment connection pool serves
  reads and writes. Concurrent writes are coalesced into a single
  `BEGIN ISOLATION LEVEL SERIALIZABLE` transaction; bodies share the
  transaction with inter-body flushing so the engine sees its own
  prior writes within the batch.
- **Multi-process.** A single process per database today.
  Multi-process support (multiple workers sharing a Postgres backend)
  is on the roadmap.

## See also

- [App](/docs/programming_guide/app#lifespan-optional) — configuring `db_path` programmatically via a lifespan.
- [Multiple environments](/docs/advanced_topics/multiple_environments) — running isolated state stores side by side.