# GitHub connector > **CocoIndex v1.** This page documents CocoIndex **v1** — a ground-up redesign from v0. When writing code, ignore any v0 flow-builder DSL or deprecated decorators. > > Source: https://cocoindex.io/docs/connectors/github/ · Docs index: https://cocoindex.io/docs/llms.txt · Agent skill: https://cocoindex.io/docs/skill.md > > v0→v1 quick map — if you reach for these v0 symbols, stop and use the v1 form: `@cocoindex.flow_def`/`FlowBuilder` → `coco.App` + a `@coco.fn` main function; `add_collector()`/`collect()`/`export()` → declare target states (`declare_row`, `declare_file`); `cocoindex.sources/functions/targets.*` → connector APIs (`localfs.walk_dir`, `coco.ops.*`, `postgres.declare_table_target`). Full mapping + API reference: https://cocoindex.io/docs/skill.md. The `github` connector reads files from a GitHub repository through a GitHub App installation. ```python from cocoindex.connectors import github ``` **Note — Dependencies** This connector requires additional dependencies. Install with: ```bash pip install cocoindex-plus[github] ``` ## Setup 1. In GitHub, go to **Settings → Developer settings → GitHub Apps → New GitHub App**. 2. Grant the following permissions: - **Repository permissions → Contents: Read-only** - **Repository permissions → Metadata: Read-only** 3. Create and download a **Private key (PEM)**. Save it to a secure path on your machine. 4. Note the **App ID**. 5. **Install** the App on the repository (or organization) you want to index. You pass the App ID and the PEM (either inline or by filesystem path) to `GitHubApp` (see below). ## As source ### Quick start The most common use case is "process every file matching these patterns" — use [`mount_each_file`](#mount_each_file): ```python import os import cocoindex as coco from cocoindex.connectors import github from cocoindex.resources.file import PatternFilePathMatcher @coco.fn(memo=True) async def process_file(file: github.File) -> None: text = await file.read_text() # ... chunk, embed, declare rows on a target, etc. ... @coco.fn async def app_main() -> None: async with github.GitHubApp( app_id=int(os.environ["GITHUB_APP_ID"]), private_key_path=os.environ["GITHUB_PRIVATE_KEY_PATH"], ) as app: repo = github.GitHubRepo(app=app, owner="cocoindex-io", repo="cocoindex") commit = await repo.get_commit(ref="main") await github.mount_each_file( process_file, commit, github.WalkOptions( path_matcher=PatternFilePathMatcher( included_patterns=["**/*.py", "**/*.md"], excluded_patterns=["**/.*", "**/node_modules"], ), ), ) ``` If you've used `localfs.walk_dir(...).items()` + `coco.mount_each(...)`, the GitHub side is the same shape after the App / repo / commit handshake: each matching file becomes its own component at a hierarchical subpath that mirrors the repo layout. For continuous polling, wrap the walk in [`coco.auto_refresh`](/docs/programming_guide/live_mode#periodic-refresh-with-cocoauto_refresh). ### GitHubApp The GitHub App — credentials plus the authenticated HTTP connection(s) it opens. ```python @dataclass(frozen=True) class GitHubApp: app_id: int private_key_pem: str | None = None private_key_path: str | None = None rate_limiter: RateLimiter | None = None async def aclose(self) -> None: ... async def __aenter__(self) -> Self: ... async def __aexit__(self, *exc_info) -> None: ... ``` **Parameters:** - `app_id` — your GitHub App ID. - `private_key_pem` — the PEM contents as a string (preferred when loading from a secret manager). - `private_key_path` — filesystem path to the PEM file. Used only if `private_key_pem` is not set. - `rate_limiter` — optional [`RateLimiter`](/docs/common_resources/rate_limiting) to cap the GitHub API request rate. See [Rate limiting](#rate-limiting) below. Exactly one of `private_key_pem` / `private_key_path` must be provided. If both are set, `private_key_pem` wins. **Lifecycle.** The App owns the underlying `httpx` client. Use it as an async context manager — or call `aclose()` — so connections are released: ```python async with github.GitHubApp(app_id=..., private_key_path=...) as app: repo = github.GitHubRepo(app=app, owner=..., repo=...) ... ``` Auth state (JWT, installation tokens, the HTTP client) and the `rate_limiter` budget are shared by **object identity** — reuse one `GitHubApp` instance across every `GitHubRepo` (and every walk) that should share them. Distinct instances never share, even with the same `app_id`. This is the one budget / one connection pool boundary. ### Rate limiting GitHub enforces per-installation API rate limits. To stay under them, attach a [`RateLimiter`](/docs/common_resources/rate_limiting) to `GitHubApp.rate_limiter`: ```python from cocoindex.resources.rate_limit import RateLimiter app = github.GitHubApp( app_id=int(os.environ["GITHUB_APP_ID"]), private_key_path=os.environ["GITHUB_PRIVATE_KEY_PATH"], rate_limiter=RateLimiter(max_rows_per_second=20.0), ) ``` **How it's counted.** One token is acquired per **outgoing HTTP request** — each GraphQL POST and each REST GET — *not* per file. The connector coalesces a tree walk into batched GraphQL queries (up to 100 objects per POST), so a 100-file batch costs a single token, matching GitHub's per-request quota. When a batch is split or an item falls back to REST, each resulting request is counted — except a failed request that is split and retried, which is not double-counted. **Shared budget.** A `GitHubApp` owns one auth context per endpoint, so a single budget covers all three fetchers (commits, trees, blobs) and every `GitHubRepo` that reuses the same `GitHubApp` instance. The budget lives in the `RateLimiter` object itself, so to put **multiple** Apps under one budget, give them the *same* `RateLimiter` instance (a fresh one per App is a separate budget). **Cooperates with batching.** While a request waits for a token, in-flight fetches keep accumulating into the next batch — so throttling tends to make batches *larger* (more efficient) rather than fragmenting them. ### GitHubRepo A lightweight handle for a single repository. It owns no connections of its own — all I/O routes through the owning `GitHubApp`'s shared client. ```python class GitHubRepo: def __init__( self, *, app: GitHubApp, owner: str, repo: str, api_base_url: str | None = None, # defaults to "https://api.github.com" ) -> None: ... async def get_commit(self, ref: str | None = None) -> github.Commit: ... ``` **Parameters:** - `app` — the `GitHubApp` to authenticate as. - `owner` — repository owner (user or organization login). - `repo` — repository name. - `api_base_url` — custom GitHub REST API base URL. See [GitHub Enterprise Server](#github-enterprise-server) below. #### `get_commit` Resolves `ref` (a branch name, tag, or commit SHA) to a [`Commit`](#commit). If `ref` is `None`, the repository's default branch is used. ```python commit = await repo.get_commit(ref="main") commit = await repo.get_commit(ref="v1.0.6") commit = await repo.get_commit() # default branch ``` #### Lifecycle The **`GitHubApp`** owns the installation-authenticated `httpx.AsyncClient`, created lazily on the first API call and shared by every `GitHubRepo` that reuses the same App instance — so JWT and installation-token refresh is amortized across repos for free. A `GitHubRepo` holds nothing of its own and needs no cleanup; just construct it. Close the **App** to release connections. `Commit`, `Dir`, and `File` objects are only valid while the owning App is open. Reading a blob after the App is closed will fail. If you don't close the App (e.g. you construct one inline and never `async with` it), the underlying client and its connection pool stay alive until process exit. #### GitHub Enterprise Server To target GHES, set `api_base_url` on `GitHubRepo` to your instance's REST API endpoint: ```python async with github.GitHubApp(app_id=..., private_key_pem=...) as app: repo = github.GitHubRepo( app=app, owner="mycompany", repo="myrepo", api_base_url="https://github.company.com/api/v3", ) ... ``` The same GitHub App flow applies — install the App on the GHES instance, generate the PEM there, and supply its App ID + key. ### Git object model The connector exposes a small Git object model rooted at a `Commit`: ``` Commit └── Dir (root tree) ├── File (blob) ├── File └── Dir ├── File └── ... ``` Each object is identified by its **SHA1** (Git is content-addressed). The memo key for `File` and `Dir` is `(path, sha1)`, so unchanged content hits cache across re-runs. #### File A regular file (Git blob). Extends [`FileLike`](/docs/common_resources/data_types#filelike), so it has the standard async read/size methods: ```python class File(Object, FileLike[str]): file_path: GitFilePath # .path is a PurePosixPath; .resolve() returns the SHA1 sha1: str async def read(self, size: int = -1) -> bytes: ... async def read_text(self, encoding=None, errors="replace") -> str: ... async def size(self) -> int: ... async def content_fingerprint(self) -> bytes: ... ``` **Notes:** - `size()` is a cheap attribute read in the normal walk path — `Dir.get_members` populates each `File`'s metadata from the parent tree listing (GitHub's tree endpoint includes the blob size), so no extra HTTP call is needed. - `content_fingerprint()` returns the Git blob SHA1 — the natural fingerprint for a content-addressed object. - The memo state is short-circuited: SHA1 in the key fully captures content identity, so no second `(mtime, fp)` validation pass is needed. #### Dir A directory (Git tree). Carries a list of immediate children, fetched lazily. ```python class Dir(Object): file_path: GitFilePath sha1: str async def get_members(self) -> list[File | Dir]: ... ``` **Notes:** - `get_members()` is cached on the `Dir` instance; calling it twice does not re-fetch. - Symlinks (mode `120000`) and submodules (`type=commit`) are skipped from `get_members()`. #### Commit A Git commit. Has the commit SHA and provides access to the root tree. ```python class Commit(Object): sha1: str async def get_tree(self) -> Dir: ... ``` `Commit` is the user-facing entry point — pass it to `mount_each_file`, or implement a custom `RepoVisitor` and call `await visitor.visit_commit(commit)`. ### WalkOptions Filter bundle threaded through the walk. Used by `mount_each_file` and by `RepoVisitor.visit_commit`. ```python @dataclass(frozen=True) class WalkOptions: path_matcher: FilePathMatcher | None = None max_file_size: int | None = None ``` **Fields:** - `path_matcher` — filter applied to both directories and files. Use [`PatternFilePathMatcher`](/docs/common_resources/data_types#filepathmatcher) for globset-style include/exclude patterns. Excluded directories short-circuit their whole subtree. For sub-directory restriction (v0's `path` field), prefix the pattern, e.g. `included_patterns=["src/**/*.py"]`. - `max_file_size` — bytes; `File`s larger than this are skipped. Checked via the file's pre-populated metadata, so no extra HTTP call. The dataclass is `frozen=True`, so a `WalkOptions()` default value is safe. ### mount_each_file Convenience helper for the common "process every matching file" case: ```python async def mount_each_file( fn: Callable[[File, ...], Awaitable[Any]], commit: Commit, options: WalkOptions = WalkOptions(), /, *args, **kwargs, ) -> None ``` **Parameters:** - `fn` — function to run for each matching file. Receives the `File` as its first argument, followed by `args` / `kwargs`. Use `@coco.fn(memo=True)` to get SHA1-driven per-file memoization (typical). - `commit` — the `Commit` whose tree to walk. - `options` — a [`WalkOptions`](#walkoptions) controlling per-entry filtering. Defaults to an unfiltered walk. - `*args` / `**kwargs` — extra arguments forwarded to `fn` after the `File`. Each matching file is mounted as `fn(file, *args, **kwargs)` at a component subpath that mirrors the file's repo path. When the walk finishes, every file is already mounted and ready — there is no handle to `await`. ```python await github.mount_each_file( process_file, commit, github.WalkOptions( path_matcher=PatternFilePathMatcher(included_patterns=["**/*.py"]), max_file_size=1_000_000, ), target_table, # extra args forwarded to process_file ) ``` ### Advanced: custom `RepoVisitor` When you need per-directory aggregation, custom traversal, or to return non-`None` values from each node, subclass `RepoVisitor[T]` and override `visit_file` (typical) or `visit_directory` (for aggregation). The default `visit_directory` walks members via `process_dir_members`, which filters out excluded entries and mounts each surviving child as a hierarchical component. ```python class RepoVisitor(Generic[T]): async def visit_commit(self, commit: Commit, options: WalkOptions = WalkOptions()) -> T: ... async def visit_directory(self, directory: Dir, options: WalkOptions) -> T: ... async def visit_file(self, file: File) -> T: ... async def process_dir_members(self, directory: Dir, options: WalkOptions) -> list[T]: ... ``` `visit_commit`'s `options` defaults to a no-filter `WalkOptions()`, so the simple case (walk everything) doesn't need to pass options. The internal recursion methods take it as required. A common shape: sum a metric across a subtree. ```python class LineCountVisitor(github.RepoVisitor[int]): async def visit_file(self, file: github.File) -> int: return (await file.read_text()).count("\n") async def visit_directory(self, directory, options) -> int: sub_counts = await self.process_dir_members(directory, options) return sum(sub_counts) total_lines = await LineCountVisitor().visit_commit(commit) # Or with filtering: total_lines = await LineCountVisitor().visit_commit( commit, github.WalkOptions(path_matcher=PatternFilePathMatcher(...)), ) ``` ## Memoization and change detection Every Git entry that gets mounted becomes its own component with a memo key that includes its `(path, sha1)`. On a re-run: - Unchanged subtrees (same `Dir` SHA1) hit memo and short-circuit recursion entirely. - Unchanged files (same `File` SHA1 at the same path) hit memo and skip your per-file processing. - Paths that disappear from the tree are no longer mounted, and CocoIndex's standard cleanup removes their target rows. - Switching to a different ref (e.g. `main` → a release tag) only re-processes the diff. Most blobs are byte-identical between adjacent commits, so the second walk is typically 5–10× faster than the cold index. There is no `list_diff` or watermark machinery to configure — Git's content-addressing plus CocoIndex's component-path model give incremental updates for free. GitHub has no push-style change feed, so for continuous re-polling wrap the walk in [`coco.auto_refresh`](/docs/programming_guide/live_mode#periodic-refresh-with-cocoauto_refresh).