ExtractCodeElements

CocoIndex Plus TreeSitter-based extractor for code declarations (classes, functions, methods) and call/type references, with built-in configs for Python, C#, JavaScript/TypeScript, Go, Rust, Java, Kotlin, Swift, C/C++, and Objective-C / Objective-C++, plus a config DSL for restricting or overriding per-language behavior.

Version

v 1.0.14

The cocoindex.ops.code_ast module wraps a TreeSitter-based extractor that parses source code and returns two structured lists per file:

• Declarations — every class, function, method, struct, etc., with its namespace, fully qualified entity name, normalized kind, raw AST node kind, and source position. Only module-, namespace-, and type-level declarations are indexed; declarations local to a function body (nested functions, local classes) are skipped as noise. References inside those bodies are still captured.
• References — every call site, object instantiation, type parameter, and base type that points at another named entity.

from cocoindex.ops.code_ast import ExtractCodeElements

pip install cocoindex-plus

Usage shape

ExtractCodeElements is constructed once and reused — the extractor caches compiled grammars and exclusion regexes per language. Call .extract(...) inside a @coco.fn to walk one file’s source:

import cocoindex as coco
from cocoindex.ops.code_ast import ExtractCodeElements
from cocoindex.ops.text import detect_code_language
from cocoindex.resources.file import FileLike

_extractor = ExtractCodeElements()

@coco.fn
async def index_file(file: FileLike, target: coco.TableTarget) -> None:
    language = detect_code_language(filename=file.file_path.path.name)
    if language is None:
        return
    elements = _extractor.extract(await file.read_text(), language=language)
    for d in elements.declarations:
        target.declare_row(...)  # one row per declaration
    for r in elements.references:
        target.declare_row(...)  # one row per reference

The end-to-end example in cocoindex-plus-examples/code_elements_indexing walks a GitHub repository, runs the extractor over every supported source file, and writes one Postgres row per declaration and reference.

ExtractCodeElements

ExtractCodeElements(
    *,
    languages: dict[str, CodeElementsLanguageConfig] | None = None,
)

When languages is provided, only the languages named in the map are enabled, and the supplied config replaces the built-in node-kind maps and exclusion patterns. The TreeSitter grammar and language hooks always come from the built-ins, so the map keys must be names with built-in support (any of the names in the Built-in language support table; matched case-insensitively).

extract()

extract(
    code: str,
    *,
    language: str,
    base_namespace: str | None = None,
) -> CodeElements

Built-in language support

Defaults ship for the languages below. The language= keys match the names returned by detect_code_language, so a language detected from a filename can be passed straight to .extract(...).

To restrict or extend the defaults, pass a languages={...} map — for example, to index Python only and drop the built-in class_base and typed_parameter reference kinds:

from cocoindex.ops.code_ast import (
    CodeElementsDeclarationConfig,
    CodeElementsLanguageConfig,
    CodeElementsReferenceConfig,
    ExtractCodeElements,
)

extractor = ExtractCodeElements(
    languages={
        "python": CodeElementsLanguageConfig(
            declaration_node_kinds={
                "class_definition": CodeElementsDeclarationConfig(
                    name_field="name", body_field="body", kind="class"
                ),
                "function_definition": CodeElementsDeclarationConfig(
                    name_field="name", body_field="body", kind="function"
                ),
            },
            reference_node_kinds={
                "call": CodeElementsReferenceConfig(path_expr_field="function"),
            },
            exclude_reference_patterns=[r"print|len|range"],
        ),
    },
)

Config types

CodeElementsLanguageConfig

CodeElementsLanguageConfig(
    declaration_node_kinds: dict[str, CodeElementsDeclarationConfig] = {},
    reference_node_kinds: dict[str, CodeElementsReferenceConfig] = {},
    type_list_node_kinds: dict[str, CodeElementsTypeListConfig] = {},
    namespace_node_kinds: dict[str, CodeElementsNamespaceConfig] = {},
    exclude_reference_patterns: list[str] = [],
)

The four *_node_kinds maps each key an AST node type — as reported by the TreeSitter grammar for that language — to its extraction rule. exclude_reference_patterns is a list of regex patterns; any reference whose referenced_full_path fully matches one of them is dropped (patterns are |-joined and anchored as ^(?:p1|p2|...)$).

CodeElementsDeclarationConfig

CodeElementsDeclarationConfig(
    name_field: str,
    body_field: str | None = None,
    kind: str = "others",
)

CodeElementsReferenceConfig

CodeElementsReferenceConfig(path_expr_field: str)

CodeElementsTypeListConfig

CodeElementsTypeListConfig()

A marker config — node kinds in type_list_node_kinds emit one reference per named child (used for nodes like C#‘s base_list and type_argument_list).

CodeElementsNamespaceConfig

CodeElementsNamespaceConfig(name_field: str)

Output shape

CodeElements

CodeElements(declarations: list[Declaration], references: list[Reference])

Declaration

Declaration(
    namespace: str,
    entity_name: str,
    parent_entity_name: str | None,
    base_name: str,
    kind: str,
    ast_node_kind: str,
    has_body: bool,
    start: CodePosition,
    end: CodePosition,
)

Declaration kinds

kind is a classification that is stable across languages, unlike ast_node_kind (the raw TreeSitter node type, which differs per grammar) — so you can select, say, every method without knowing each language’s node names. The vocabulary is:

class, interface, struct, union, enum, trait, type_alias, function, method, constructor, property, constant, variable, field, extension, and others (catch-all).

Most constructs map to the obvious kind (a class → class, a struct → struct, an enum → enum). Two things are not a one-to-one rename of the syntax:

• method vs function is contextual, and holds for every language: a function declared directly inside a type body is a method; a free function — or one nested inside another function — stays function. The same applies to field vs variable.

• A handful of constructs normalize in a language-specific way, where one language’s keyword is folded into a kind named after another’s. These are the non-obvious cases worth knowing:

  Source construct	Language(s)	kind
  record, record class	C#, Java	class
  record struct	C#	struct
  protocol	Swift, Objective-C	interface
  trait	Rust	trait
  object (singleton)	Kotlin	class
  impl block, extension, category	Rust, Swift, Objective-C	extension

  Whether a record is class-like or struct-like is genuinely a per-language call (C# splits on the struct keyword; Java records are always reference types; a Pascal/Delphi record would be struct-like), so it is spelled out here rather than stated as a universal rule. This is intentionally just the non-obvious cases — not an exhaustive construct-by-language matrix.

field and property cover the data members of a type, split by language. A member value is a property in the languages that have first-class properties — C# (property_declaration), Kotlin (val/var), Swift (stored & computed properties, including protocol requirements), TypeScript (interface members), and Objective-C (@property) — and a field everywhere else: C/C++/Objective-C struct fields, C# fields, Go struct fields, Java fields, Rust struct fields, and JS/TS class fields. A single declaration that binds several names (int x, y;, Go X, Y int) yields one field per name. To enumerate all of a type’s data, union field and property.

variable and constant cover module-, namespace-, and type-level value bindings (locals inside function bodies are dropped by the scope filter). constant is reserved for compile-time constants: Go const, Rust const, C# const, Java static final, and Kotlin const val. variable covers the rest: Go package var, Rust static, and JS/TS module-level const/let/var — JavaScript/TypeScript const is binding immutability, not a compile-time constant, so it is variable, not constant.

A declaration’s enclosing namespace or module is captured in its namespace field rather than emitted as a separate declaration. New kinds are added together with the extraction that produces them, so the list above is exactly what the built-in configs emit.

Reference

Reference(
    namespace: str,
    parent_entity_name: str | None,
    referenced_base_name: str,
    referenced_full_path: str,
    ast_node_kind: str,
    start: CodePosition,
    end: CodePosition,
)

CodePosition

CodePosition(char_offset: int, line: int, column: int)