# Observability (Plus)

> **CocoIndex v1.** This page documents CocoIndex **v1** — a ground-up redesign from v0. When writing code, ignore any v0 flow-builder DSL or deprecated decorators.
>
> Source: https://plus.cocoindex.io/docs/advanced_topics/observability/ · Docs index: https://plus.cocoindex.io/docs/llms.txt · Agent skill: https://plus.cocoindex.io/docs/skill.md
>
> v0→v1 quick map — if you reach for these v0 symbols, stop and use the v1 form: `@cocoindex.flow_def`/`FlowBuilder` → `coco.App` + a `@coco.fn` main function; `add_collector()`/`collect()`/`export()` → declare target states (`declare_row`, `declare_file`); `cocoindex.sources/functions/targets.*` → connector APIs (`localfs.walk_dir`, `coco.ops.*`, `postgres.declare_table_target`). Full mapping + API reference: https://plus.cocoindex.io/docs/skill.md.

**Note**
Observability is a **CocoIndex Plus** feature, available since v1.0.59. The open-source edition
exports no metrics, and `coco.metrics` raises `RuntimeError` there.

CocoIndex Plus records what every processing component did and how long it took, and exports
those measurements over [OpenTelemetry](https://opentelemetry.io/) to the observability stack you
already run. It ships a reference Grafana dashboard and example alert rules that work unmodified
against those metrics. Nothing leaves your network, and nothing is exported unless you configure an
exporter.

## Configuration

Metrics export is configured entirely through the standard OpenTelemetry SDK environment variables;
CocoIndex introduces none of its own.

```bash
# Push metrics over OTLP (HTTP/protobuf) to a collector or a Prometheus OTLP receiver.
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
# Optional: how often to export (milliseconds; default 60000) and the resource identity.
export OTEL_METRIC_EXPORT_INTERVAL=15000
export OTEL_SERVICE_NAME=my-indexer
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod
```

| Variable | Effect |
|---|---|
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | Where to push. Setting either enables export. |
| `OTEL_METRICS_EXPORTER` | `otlp` (default when an endpoint is set) or `none`. `prometheus` (a scrape endpoint) is not supported yet. |
| `OTEL_EXPORTER_OTLP_HEADERS`, `OTEL_EXPORTER_OTLP_PROTOCOL` | Authentication headers and protocol, per the OpenTelemetry specification. Only `http/protobuf` is compiled in. |
| `OTEL_METRIC_EXPORT_INTERVAL`, `OTEL_METRIC_EXPORT_TIMEOUT` | Export cadence and timeout. |
| `OTEL_SERVICE_NAME`, `OTEL_RESOURCE_ATTRIBUTES` | Resource identity; the service name defaults to `cocoindex`. |
| `OTEL_SDK_DISABLED=true` | Disables export whatever else is set. |

A catch-up `app.update()` can finish before the first periodic export. CocoIndex flushes the
exporter when the environment closes (`coco.stop()`, the end of `update_blocking()`, the CLI
exiting), so short runs still deliver their final counters.

## What is exported

Every metric carries the `env` (environment name) and `app` labels. Component-level metrics add
`stats_group` and `processor`: the same two-level grouping the terminal progress display uses, where
`stats_group` is a [`coco.stats_group`](/docs/advanced_topics/progress_monitoring/#scoped-reports-with-stats_group) title
(empty for the root report) and `processor` is the mounted function's name. Component paths, shared
keys and target-state keys are never labels; they are unbounded.

The main series, by question:

- **Is my data fresh?** `cocoindex.runs` (by `mode` and `status`), `cocoindex.run.time_to_ready`,
  `cocoindex.run.duration`, `cocoindex.run.last_ready_time`.
- **How much changed, and why?** `cocoindex.component.runs` by `outcome` (unchanged, added,
  reprocessed, deleted, error) and `cocoindex.component.memo.decisions` by `decision`, which says why
  a run was reused or rebuilt: `hit`, `hit_same_run` (a shared component's later mounts in one run),
  `hit_revalidated`, `hit_after_wait`, `miss_absent`, `miss_changed`, `miss_state_invalid`,
  `bypassed` (`full_reprocess`), `not_memoized`.
- **Where does time go?** `cocoindex.component.process.duration` (your code) and
  `cocoindex.component.submit.duration` by `phase` (`precommit`, `sink_apply`, `commit`).
- **What landed in targets?** `cocoindex.target.reconciles` by `provider`, `container` and
  `result`; `cocoindex.target.sink.duration`, `batch_size`, `splits` and `failures` by `sink`.
- **Is concurrency the limiter?** `cocoindex.admission.available`, `cocoindex.admission.waits`,
  `cocoindex.admission.wait.duration`.
- **What is failing?** `cocoindex.errors` by `exception_type` and `handled` (`swallowed` by an
  exception handler or the default log, or `propagated` to the caller).

The full list with Prometheus names, units and label sets is in the
[metrics reference](/docs/advanced_topics/metrics_reference/).

Quantities that go up and down are not gauged by the engine. Components in progress are
`cocoindex.component.starts` minus `cocoindex.component.runs` at query time, and the two gauges that
exist (`admission.available`, `run.last_ready_time`) are read from engine state when the exporter
collects.

### Label cardinality

`stats_group` titles and target `container` keys become label values, so each distinct value is a
separate set of series. The exporter keeps at most 64 stats-group titles and 256 container keys per
app, reports further values as `__other__`, and logs a warning once. Use titles from a bounded set:
a pipeline phase, or one group per repository when an app indexes fewer than 64 of them. Processor
names are code identifiers and are not capped.

## Reference dashboard

Download the dashboard and the example alert rules:

- [`cocoindex-grafana-dashboard.json`](/docs/observability/cocoindex-grafana-dashboard.json)
- [`cocoindex-grafana-alerts.yaml`](/docs/observability/cocoindex-grafana-alerts.yaml)

Import the dashboard into Grafana (Dashboards, New, Import), pick your Prometheus-compatible
datasource, and the panels populate from the metrics above with no edits. The dashboard is
generated from the engine's metric catalog, so every query names a metric the engine exports;
fork the imported copy in Grafana to customize it. Template variables select environment, app,
stats group and processor; the stats-group row repeats per group with its processor table,
mirroring the `[Stats: <title>]` blocks of the terminal display.

The panels use PromQL, so they presume a Prometheus-compatible store: Prometheus (with its OTLP
receiver), Mimir, Thanos, VictoriaMetrics, or a managed equivalent. Datadog, New Relic and similar
services ingest the same OTLP stream directly; chart the names from the metrics reference there.

The alert rules carry placeholder thresholds and say so in a header comment; set them from your own
service objectives.

### Freshness and how the app runs

The freshness panels (time since last ready, updates by status, time to ready) and the "no recent
ready" alert are driven by updates: each `app.update()` reports when it reaches ready and when it
ends. How well they track freshness depends on how your app runs:

- **A long-lived process that runs catch-up updates repeatedly** (a loop or an in-process
  scheduler calling `app.update()`): every update refreshes them. This is what they are built for.
- **A scheduled job that exits after each update:** the final values are exported when the
  environment closes, but Prometheus marks a series stale once its process is gone, so the alert
  cannot tell a healthy schedule from a stopped one. Drive freshness checks from the scheduler.
- **Live mode** (`app.update(live=True)`, live sources, webhooks): the update reaches ready once,
  at its initial catch-up, and runs until the process stops. Time since last ready keeps growing
  on a healthy worker, so don't use the alert. Watch component runs and errors instead; metrics for
  how long a change takes to land in live mode are planned.

## User counters

Add your own counters, attributed to the pipeline structure automatically:

```python
import cocoindex as coco

_tokens = coco.metrics.counter("llm_tokens")

@coco.fn
async def summarize(text: str) -> str:
    response = await llm(text)
    _tokens.add(response.usage.total_tokens)
    return response.text
```

`coco.metrics.counter(name)` returns a cheap handle; `add(value=1)` adds to a counter exported as
`cocoindex.user.<name>`. Names must match `[a-z][a-z0-9_]*`. A value added inside a processing
component carries that component's `stats_group` and `processor` labels, so a token or cost counter
lines up with the processor table on the dashboard; outside a component it carries no component
labels. Distinct names are capped at 256 per process.

## Overhead

With no exporter configured, the engine keeps a per-update aggregate of counts and duration totals
(two clock reads and a few integer additions per component run) and makes no calls into any
exporter. With an exporter, the cost is building attribute sets per event, bounded by the
cardinality caps above.