Skip to content

Repo Ingestion (Proposal)

Status: draft. Generalises the repo lists built for the Novansa pilot (pilot plan §5.3, decision D1) into a module every Thinklio account can use. The format is published at GET /v1/ingest/spec and described for repo authors in the guide.

1. Why

Teams that work in code keep their plans, registers and decisions in their repos, next to the work and edited by the people (and coding agents) doing it. A separate tracker goes stale; Novansa cancelled Linear for exactly that reason. Thinklio should not ask them to move that content. It should read it: show it read-only in the right space, keep it current on every push, keep a history of what changed and which commit changed it, and put it beside the work that does live in Thinklio (tasks, editable lists, the Inbox, agents).

The rule that makes this safe and simple: the repo owns the content; Thinklio owns the view, the history and the links. Nothing is written back to the repo.

2. What exists (the Novansa prototype)

Built on 2026-10-05 and live for six Novansa repos:

  • A Markdown parser for tables and numbered or bulleted items (no LLM), with done detection, sections and stable keys.
  • thinklio.yml at the repo root naming each list file and its options.
  • Sync: default branch only, unchanged files skipped by blob SHA, rows reconciled by key through the record write path, each change credited to the last commit that touched the file, a hold when a read would archive more than a third of a list.
  • Triggers: a push webhook, an hourly sweep and Refresh now.

What makes it Novansa-only: one platform-wide GitHub token (Andrew's), one webhook secret, webhooks added to each repo by hand, and a manifest that names Thinklio space slugs directly.

3. The published spec (thinklio-ingest, version 1)

One definition in code (convex/ingest/spec.ts) produces everything outward-facing, so the endpoint, the schemas and the guide never drift:

  • GET /v1/ingest/spec: the spec as JSON (version, declaration formats with JSON Schemas and examples, ingest types and their status, parsing rules, limits).
  • GET /v1/ingest/schema/frontmatter.json and GET /v1/ingest/schema/manifest.json: JSON Schemas, usable from editors (# yaml-language-server: $schema=…).
  • GET /v1/ingest/guide.md: the guide for repo authors, generated from the same definition.

These are public (no key), cacheable and served from Convex HTTP actions; they move to api.thinklio.ai when that domain points at Convex.

3.1 Declaring a file: front matter

A Markdown file opts in at its top:

---
thinklio:
  ingest: list            # list | page (page: planned) | true (infer) | false
  name: Tech debt         # default: the first H1, else the file name
  space: ripplebase       # optional: suggested space; the account's mapping decides
  format: table           # list only: table (default) | items
  key: ID                 # list only: the column that identifies a row
  sections: [Summary]     # list only: read only under these headings
  done_sections: [Done]   # list only: rows here count as done
---

The shorthand thinklioIngest: true is the same as thinklio: { ingest: true }. Front matter is self-describing, travels with the file when it moves, and is visible to anyone editing it.

3.2 Declaring a repo: thinklio.yml (optional)

version: 1
space: ripplebase                  # default space for this repo's content
scan: ["docs/**/*.md", "*.md"]     # where to look for front matter
lists:                             # files declared here instead of (or as well as) by front matter
  - name: Tech debt
    file: apps/web/docs/tech-debt.md
    key: ID
    done_sections: [Done]

Front matter wins over a manifest entry for the same file. The six Novansa manifests written on 2026-10-05 stay valid (version 1 is their format, with format: list accepted as the older name for items).

3.3 Versioning

version is an integer. Additions that old files don't use (new ingest types, new options) keep the version; a change in meaning bumps it, and Thinklio reads every version it has published. The spec response lists each type's status (available, preview, planned) so clients can tell what works today.

4. Module structure

convex/ingest/
  spec.ts           the published format: version, JSON Schemas, examples, guide
  frontMatter.ts    split YAML front matter from Markdown           (pure)
  declarations.ts   manifest + front matter → ingest targets        (pure)
  parsers/
    markdownLists.ts  tables and items → rows                       (pure)
    (markdownPage.ts) a document → a page                           (with the wiki)
  sources/
    github.ts       the GitHub adapter: repo, tree, file, last commit
  sync.ts           orchestration: discover, skip unchanged, parse, apply (actions)
  repoLists.ts      connections and applying rows to lists (queries, mutations)

The seams that matter for other accounts and other sources:

  • Source adapter. Sync talks to an interface (default branch, list files with their blob SHAs, read a file, last commit for a path), not to GitHub directly. GitLab, Bitbucket, Gitea and Azure DevOps are further adapters; nothing else changes. This follows ADR-038's provider-neutral capabilities.
  • Credentials per connection. The adapter asks the connection for a token. Today that is the platform token; with the GitHub App it is a short-lived installation token for that account's installation.
  • Targets, not files. Declarations become targets ({ path, as: list | page, name, space, options }). Parsers and appliers only see targets, so a new declaration style (say, a Notion database) reuses everything downstream.
  • Appliers through the write paths. Lists go through lib/recordWrites, pages will go through the page write path. History, undo for local edits, "Changed this week" and agents' provenance all work without ingestion knowing about them.

Component later, not now. The module could become a Convex component (its own tables for connections, targets and file state; an app-facing API; callbacks into the app's write paths). That pays off when a second app needs it (Ripplebase ingesting content repos is a candidate). Until then a folder with these seams is cheaper to change.

5. Multi-tenancy

  • Connections belong to an account. An account admin connects a provider; the connection records the provider, how it authenticates and which repos it may read.
  • GitHub App, not personal tokens. A Thinklio GitHub App with read-only Contents and Metadata permissions and the push event. Installing it on an organisation or account (choosing all or selected repos) gives Thinklio an installation ID. Pushes for every installation arrive at one webhook URL, signed with the app's secret, carrying the installation ID; Thinklio maps that to the connection and so to the account. Tokens are minted per installation for an hour at a time. No webhooks to create by hand, no long-lived tokens.
  • The repo suggests, the account decides. A manifest or front matter may name a space slug; the account's repo mapping (default space per connected repo, overrides per target) decides. A public repo connected by two accounts lands in each account's own spaces.
  • Isolation. Every ingested record and list carries the account; reads go through space access like any other list. A connection can only read the repos its installation allows.

6. Security

  • Content is data, not instructions. Ingested text is untrusted. When agents read it, it is marked as coming from a repo and is never treated as an instruction, which matters most for public repos.
  • Read-only. Thinklio never writes to a connected repo.
  • Limits. Files over 1 MB are refused; a scan reads at most 300 Markdown files per repo per pass; a list holds at most 2,000 rows; the hold stops a malformed file from emptying a list.
  • Secrets. App private key and webhook secret in Convex environment variables; installation tokens are minted per sync and never stored.

7. Roadmap

  1. Now (2026-10-05): the module folder, spec v1 with the endpoint, schemas and guide, front matter declarations with discovery, the source adapter seam, a default space per connected repo.
  2. Pages: ingest: page with the wiki (pilot §5.4): repo docs (plans, specs, runbooks) shown read-only as pages, with links resolved between them.
  3. The Thinklio GitHub App and a Settings → Connections → GitHub screen: install, pick repos, map them to spaces, see what each repo declares, last sync, errors.
  4. Validation: POST /v1/ingest/validate (a manifest, front matter, or a file to preview how it parses), and a small GitHub Action that runs it on pull requests so a reformatted register fails CI instead of being held.
  5. More types: decisions (ADR files), changelogs, releases.
  6. More sources: GitLab next; then Bitbucket, Gitea, Azure DevOps on demand.
  7. Extraction into a Convex component when a second app needs it.

8. Decisions needed

  • I1. Create the Thinklio GitHub App (owner: Novansa's GitHub; permissions as §5) and move Novansa's six repos onto it, retiring the personal token and the hand-made webhooks. Recommended: yes, as roadmap step 3.
  • I2. Front matter key. thinklio: block as the canonical form, thinklioIngest: true as the shorthand. Recommended: yes (built).
  • I3. Scan default. With no scan: in a manifest, scan **/*.md (excluding node_modules, build output and hidden folders), capped at 300 files. Recommended: yes (built).