Observability (Plus)

Export CocoIndex Plus pipeline metrics over OpenTelemetry, import the reference Grafana dashboard, add your own counters, and understand what each number means.

Version

v 1.0.62

CocoIndex Plus records what every processing component did and how long it took, and exports those measurements over OpenTelemetry to the observability stack you already run. It ships a reference Grafana dashboard and example alert rules that work unmodified against those metrics. Nothing leaves your network, and nothing is exported unless you configure an exporter.

Configuration

Metrics export is configured entirely through the standard OpenTelemetry SDK environment variables; CocoIndex introduces none of its own.

# Push metrics over OTLP (HTTP/protobuf) to a collector or a Prometheus OTLP receiver.
export OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
# Optional: how often to export (milliseconds; default 60000) and the resource identity.
export OTEL_METRIC_EXPORT_INTERVAL=15000
export OTEL_SERVICE_NAME=my-indexer
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=prod

A catch-up app.update() can finish before the first periodic export. CocoIndex flushes the exporter when the environment closes (coco.stop(), the end of update_blocking(), the CLI exiting), so short runs still deliver their final counters.

What is exported

Every metric carries the env (environment name) and app labels. Component-level metrics add stats_group and processor: the same two-level grouping the terminal progress display uses, where stats_group is a coco.stats_group title (empty for the root report) and processor is the mounted function’s name. Component paths, shared keys and target-state keys are never labels; they are unbounded.

The main series, by question:

• Is my data fresh? cocoindex.runs (by mode and status), cocoindex.run.time_to_ready, cocoindex.run.duration, cocoindex.run.last_ready_time.
• How much changed, and why? cocoindex.component.runs by outcome (unchanged, added, reprocessed, deleted, error) and cocoindex.component.memo.decisions by decision, which says why a run was reused or rebuilt: hit, hit_same_run (a shared component’s later mounts in one run), hit_revalidated, hit_after_wait, miss_absent, miss_changed, miss_state_invalid, bypassed (full_reprocess), not_memoized.
• Where does time go? cocoindex.component.process.duration (your code) and cocoindex.component.submit.duration by phase (precommit, sink_apply, commit).
• What landed in targets? cocoindex.target.reconciles by provider, container and result; cocoindex.target.sink.duration, batch_size, splits and failures by sink.
• Is concurrency the limiter? cocoindex.admission.available, cocoindex.admission.waits, cocoindex.admission.wait.duration.
• What is failing? cocoindex.errors by exception_type and handled (swallowed by an exception handler or the default log, or propagated to the caller).

The full list with Prometheus names, units and label sets is in the metrics reference.

Quantities that go up and down are not gauged by the engine. Components in progress are cocoindex.component.starts minus cocoindex.component.runs at query time, and the two gauges that exist (admission.available, run.last_ready_time) are read from engine state when the exporter collects.

Label cardinality

stats_group titles and target container keys become label values, so each distinct value is a separate set of series. The exporter keeps at most 64 stats-group titles and 256 container keys per app, reports further values as __other__, and logs a warning once. Use titles from a bounded set: a pipeline phase, or one group per repository when an app indexes fewer than 64 of them. Processor names are code identifiers and are not capped.

Reference dashboard

Download the dashboard and the example alert rules:

• cocoindex-grafana-dashboard.json
• cocoindex-grafana-alerts.yaml

Import the dashboard into Grafana (Dashboards, New, Import), pick your Prometheus-compatible datasource, and the panels populate from the metrics above with no edits. The dashboard is generated from the engine’s metric catalog, so every query names a metric the engine exports; fork the imported copy in Grafana to customize it. Template variables select environment, app, stats group and processor; the stats-group row repeats per group with its processor table, mirroring the [Stats: <title>] blocks of the terminal display.

The panels use PromQL, so they presume a Prometheus-compatible store: Prometheus (with its OTLP receiver), Mimir, Thanos, VictoriaMetrics, or a managed equivalent. Datadog, New Relic and similar services ingest the same OTLP stream directly; chart the names from the metrics reference there.

The alert rules carry placeholder thresholds and say so in a header comment; set them from your own service objectives.

Freshness and how the app runs

The freshness panels (time since last ready, updates by status, time to ready) and the “no recent ready” alert are driven by updates: each app.update() reports when it reaches ready and when it ends. How well they track freshness depends on how your app runs:

• A long-lived process that runs catch-up updates repeatedly (a loop or an in-process scheduler calling app.update()): every update refreshes them. This is what they are built for.
• A scheduled job that exits after each update: the final values are exported when the environment closes, but Prometheus marks a series stale once its process is gone, so the alert cannot tell a healthy schedule from a stopped one. Drive freshness checks from the scheduler.
• Live mode (app.update(live=True), live sources, webhooks): the update reaches ready once, at its initial catch-up, and runs until the process stops. Time since last ready keeps growing on a healthy worker, so don’t use the alert. Watch component runs and errors instead; metrics for how long a change takes to land in live mode are planned.

User counters

Add your own counters, attributed to the pipeline structure automatically:

import cocoindex as coco

_tokens = coco.metrics.counter("llm_tokens")

@coco.fn
async def summarize(text: str) -> str:
    response = await llm(text)
    _tokens.add(response.usage.total_tokens)
    return response.text

coco.metrics.counter(name) returns a cheap handle; add(value=1) adds to a counter exported as cocoindex.user.<name>. Names must match [a-z][a-z0-9_]*. A value added inside a processing component carries that component’s stats_group and processor labels, so a token or cost counter lines up with the processor table on the dashboard; outside a component it carries no component labels. Distinct names are capped at 256 per process.

Overhead

With no exporter configured, the engine keeps a per-update aggregate of counts and duration totals (two clock reads and a few integer additions per component run) and makes no calls into any exporter. With an exporter, the cost is building attribute sets per event, bounded by the cardinality caps above.