GitHub connector

Index files from GitHub repositories via a GitHub App installation. Each Git object is memo-keyed by its SHA1, so re-runs of unchanged content hit cache with zero downloads. Supports GitHub.com and GitHub Enterprise Server.

Version
v 1.0.14

The github connector reads files from a GitHub repository through a GitHub App installation.

python
from cocoindex.connectors import github
Dependencies

This connector requires additional dependencies. Install with:

bash
pip install cocoindex-plus[github]

Setup

  1. In GitHub, go to Settings → Developer settings → GitHub Apps → New GitHub App.
  2. Grant the following permissions:
    • Repository permissions → Contents: Read-only
    • Repository permissions → Metadata: Read-only
  3. Create and download a Private key (PEM). Save it to a secure path on your machine.
  4. Note the App ID.
  5. Install the App on the repository (or organization) you want to index.

You pass the App ID and the PEM (either inline or by filesystem path) to GitHubApp (see below).

As source

Quick start

The most common use case is “process every file matching these patterns” — use mount_each_file:

python
import os
import cocoindex as coco
from cocoindex.connectors import github
from cocoindex.resources.file import PatternFilePathMatcher

@coco.fn(memo=True)
async def process_file(file: github.File) -> None:
    text = await file.read_text()
    # ... chunk, embed, declare rows on a target, etc. ...

@coco.fn
async def app_main() -> None:
    async with github.GitHubApp(
        app_id=int(os.environ["GITHUB_APP_ID"]),
        private_key_path=os.environ["GITHUB_PRIVATE_KEY_PATH"],
    ) as app:
        repo = github.GitHubRepo(app=app, owner="cocoindex-io", repo="cocoindex")
        commit = await repo.get_commit(ref="main")
        await github.mount_each_file(
            process_file,
            commit,
            github.WalkOptions(
                path_matcher=PatternFilePathMatcher(
                    included_patterns=["**/*.py", "**/*.md"],
                    excluded_patterns=["**/.*", "**/node_modules"],
                ),
            ),
        )

If you’ve used localfs.walk_dir(...).items() + coco.mount_each(...), the GitHub side is the same shape after the App / repo / commit handshake: each matching file becomes its own component at a hierarchical subpath that mirrors the repo layout. For continuous polling, wrap the walk in coco.auto_refresh.

GitHubApp

The GitHub App — credentials plus the authenticated HTTP connection(s) it opens.

python
@dataclass(frozen=True)
class GitHubApp:
    app_id: int
    private_key_pem: str | None = None
    private_key_path: str | None = None
    rate_limiter: RateLimiter | None = None

    async def aclose(self) -> None: ...
    async def __aenter__(self) -> Self: ...
    async def __aexit__(self, *exc_info) -> None: ...

Parameters:

  • app_id — your GitHub App ID.
  • private_key_pem — the PEM contents as a string (preferred when loading from a secret manager).
  • private_key_path — filesystem path to the PEM file. Used only if private_key_pem is not set.
  • rate_limiter — optional RateLimiter to cap the GitHub API request rate. See Rate limiting below.

Exactly one of private_key_pem / private_key_path must be provided. If both are set, private_key_pem wins.

Lifecycle. The App owns the underlying httpx client. Use it as an async context manager — or call aclose() — so connections are released:

python
async with github.GitHubApp(app_id=..., private_key_path=...) as app:
    repo = github.GitHubRepo(app=app, owner=..., repo=...)
    ...

Auth state (JWT, installation tokens, the HTTP client) and the rate_limiter budget are shared by object identity — reuse one GitHubApp instance across every GitHubRepo (and every walk) that should share them. Distinct instances never share, even with the same app_id. This is the one budget / one connection pool boundary.

Rate limiting

GitHub enforces per-installation API rate limits. To stay under them, attach a RateLimiter to GitHubApp.rate_limiter:

python
from cocoindex.resources.rate_limit import RateLimiter

app = github.GitHubApp(
    app_id=int(os.environ["GITHUB_APP_ID"]),
    private_key_path=os.environ["GITHUB_PRIVATE_KEY_PATH"],
    rate_limiter=RateLimiter(max_rows_per_second=20.0),
)

How it’s counted. One token is acquired per outgoing HTTP request — each GraphQL POST and each REST GET — not per file. The connector coalesces a tree walk into batched GraphQL queries (up to 100 objects per POST), so a 100-file batch costs a single token, matching GitHub’s per-request quota. When a batch is split or an item falls back to REST, each resulting request is counted — except a failed request that is split and retried, which is not double-counted.

Shared budget. A GitHubApp owns one auth context per endpoint, so a single budget covers all three fetchers (commits, trees, blobs) and every GitHubRepo that reuses the same GitHubApp instance. The budget lives in the RateLimiter object itself, so to put multiple Apps under one budget, give them the same RateLimiter instance (a fresh one per App is a separate budget).

Cooperates with batching. While a request waits for a token, in-flight fetches keep accumulating into the next batch — so throttling tends to make batches larger (more efficient) rather than fragmenting them.

GitHubRepo

A lightweight handle for a single repository. It owns no connections of its own — all I/O routes through the owning GitHubApp’s shared client.

python
class GitHubRepo:
    def __init__(
        self,
        *,
        app: GitHubApp,
        owner: str,
        repo: str,
        api_base_url: str | None = None,  # defaults to "https://api.github.com"
    ) -> None: ...

    async def get_commit(self, ref: str | None = None) -> github.Commit: ...

Parameters:

  • app — the GitHubApp to authenticate as.
  • owner — repository owner (user or organization login).
  • repo — repository name.
  • api_base_url — custom GitHub REST API base URL. See GitHub Enterprise Server below.

get_commit

Resolves ref (a branch name, tag, or commit SHA) to a Commit. If ref is None, the repository’s default branch is used.

python
commit = await repo.get_commit(ref="main")
commit = await repo.get_commit(ref="v1.0.6")
commit = await repo.get_commit()  # default branch

Lifecycle

The GitHubApp owns the installation-authenticated httpx.AsyncClient, created lazily on the first API call and shared by every GitHubRepo that reuses the same App instance — so JWT and installation-token refresh is amortized across repos for free. A GitHubRepo holds nothing of its own and needs no cleanup; just construct it. Close the App to release connections.

Commit, Dir, and File objects are only valid while the owning App is open. Reading a blob after the App is closed will fail.

If you don’t close the App (e.g. you construct one inline and never async with it), the underlying client and its connection pool stay alive until process exit.

GitHub Enterprise Server

To target GHES, set api_base_url on GitHubRepo to your instance’s REST API endpoint:

python
async with github.GitHubApp(app_id=..., private_key_pem=...) as app:
    repo = github.GitHubRepo(
        app=app,
        owner="mycompany",
        repo="myrepo",
        api_base_url="https://github.company.com/api/v3",
    )
    ...

The same GitHub App flow applies — install the App on the GHES instance, generate the PEM there, and supply its App ID + key.

Git object model

The connector exposes a small Git object model rooted at a Commit:

Commit
└── Dir (root tree)
    ├── File   (blob)
    ├── File
    └── Dir
        ├── File
        └── ...

Each object is identified by its SHA1 (Git is content-addressed). The memo key for File and Dir is (path, sha1), so unchanged content hits cache across re-runs.

File

A regular file (Git blob). Extends FileLike, so it has the standard async read/size methods:

python
class File(Object, FileLike[str]):
    file_path: GitFilePath  # .path is a PurePosixPath; .resolve() returns the SHA1
    sha1: str

    async def read(self, size: int = -1) -> bytes: ...
    async def read_text(self, encoding=None, errors="replace") -> str: ...
    async def size(self) -> int: ...
    async def content_fingerprint(self) -> bytes: ...

Notes:

  • size() is a cheap attribute read in the normal walk path — Dir.get_members populates each File’s metadata from the parent tree listing (GitHub’s tree endpoint includes the blob size), so no extra HTTP call is needed.
  • content_fingerprint() returns the Git blob SHA1 — the natural fingerprint for a content-addressed object.
  • The memo state is short-circuited: SHA1 in the key fully captures content identity, so no second (mtime, fp) validation pass is needed.

Dir

A directory (Git tree). Carries a list of immediate children, fetched lazily.

python
class Dir(Object):
    file_path: GitFilePath
    sha1: str

    async def get_members(self) -> list[File | Dir]: ...

Notes:

  • get_members() is cached on the Dir instance; calling it twice does not re-fetch.
  • Symlinks (mode 120000) and submodules (type=commit) are skipped from get_members().

Commit

A Git commit. Has the commit SHA and provides access to the root tree.

python
class Commit(Object):
    sha1: str

    async def get_tree(self) -> Dir: ...

Commit is the user-facing entry point — pass it to mount_each_file, or implement a custom RepoVisitor and call await visitor.visit_commit(commit).

WalkOptions

Filter bundle threaded through the walk. Used by mount_each_file and by RepoVisitor.visit_commit.

python
@dataclass(frozen=True)
class WalkOptions:
    path_matcher: FilePathMatcher | None = None
    max_file_size: int | None = None

Fields:

  • path_matcher — filter applied to both directories and files. Use PatternFilePathMatcher for globset-style include/exclude patterns. Excluded directories short-circuit their whole subtree. For sub-directory restriction (v0’s path field), prefix the pattern, e.g. included_patterns=["src/**/*.py"].
  • max_file_size — bytes; Files larger than this are skipped. Checked via the file’s pre-populated metadata, so no extra HTTP call.

The dataclass is frozen=True, so a WalkOptions() default value is safe.

mount_each_file

Convenience helper for the common “process every matching file” case:

python
async def mount_each_file(
    fn: Callable[[File, ...], Awaitable[Any]],
    commit: Commit,
    options: WalkOptions = WalkOptions(),
    /,
    *args,
    **kwargs,
) -> None

Parameters:

  • fn — function to run for each matching file. Receives the File as its first argument, followed by args / kwargs. Use @coco.fn(memo=True) to get SHA1-driven per-file memoization (typical).
  • commit — the Commit whose tree to walk.
  • options — a WalkOptions controlling per-entry filtering. Defaults to an unfiltered walk.
  • *args / **kwargs — extra arguments forwarded to fn after the File.

Each matching file is mounted as fn(file, *args, **kwargs) at a component subpath that mirrors the file’s repo path. When the walk finishes, every file is already mounted and ready — there is no handle to await.

python
await github.mount_each_file(
    process_file,
    commit,
    github.WalkOptions(
        path_matcher=PatternFilePathMatcher(included_patterns=["**/*.py"]),
        max_file_size=1_000_000,
    ),
    target_table,                    # extra args forwarded to process_file
)

Advanced: custom RepoVisitor

When you need per-directory aggregation, custom traversal, or to return non-None values from each node, subclass RepoVisitor[T] and override visit_file (typical) or visit_directory (for aggregation). The default visit_directory walks members via process_dir_members, which filters out excluded entries and mounts each surviving child as a hierarchical component.

python
class RepoVisitor(Generic[T]):
    async def visit_commit(self, commit: Commit, options: WalkOptions = WalkOptions()) -> T: ...
    async def visit_directory(self, directory: Dir, options: WalkOptions) -> T: ...
    async def visit_file(self, file: File) -> T: ...
    async def process_dir_members(self, directory: Dir, options: WalkOptions) -> list[T]: ...

visit_commit’s options defaults to a no-filter WalkOptions(), so the simple case (walk everything) doesn’t need to pass options. The internal recursion methods take it as required.

A common shape: sum a metric across a subtree.

python
class LineCountVisitor(github.RepoVisitor[int]):
    async def visit_file(self, file: github.File) -> int:
        return (await file.read_text()).count("\n")

    async def visit_directory(self, directory, options) -> int:
        sub_counts = await self.process_dir_members(directory, options)
        return sum(sub_counts)

total_lines = await LineCountVisitor().visit_commit(commit)

# Or with filtering:
total_lines = await LineCountVisitor().visit_commit(
    commit, github.WalkOptions(path_matcher=PatternFilePathMatcher(...)),
)

Memoization and change detection

Every Git entry that gets mounted becomes its own component with a memo key that includes its (path, sha1). On a re-run:

  • Unchanged subtrees (same Dir SHA1) hit memo and short-circuit recursion entirely.
  • Unchanged files (same File SHA1 at the same path) hit memo and skip your per-file processing.
  • Paths that disappear from the tree are no longer mounted, and CocoIndex’s standard cleanup removes their target rows.
  • Switching to a different ref (e.g. main → a release tag) only re-processes the diff. Most blobs are byte-identical between adjacent commits, so the second walk is typically 5–10× faster than the cold index.

There is no list_diff or watermark machinery to configure — Git’s content-addressing plus CocoIndex’s component-path model give incremental updates for free. GitHub has no push-style change feed, so for continuous re-polling wrap the walk in coco.auto_refresh.

Previous FalkorDB
CocoIndex Docs Edit this page Report issue