Model Router & Tiered Inference (Proposal)

Status: draft proposal. Not part of the canonical numbered set. If accepted, the router boundary folds into 02 System Architecture §3–5 and the self-managed-infra topology (§16); the tier contract and escalation loop extend 03 Agent Architecture (the harness's model-selection step); the per-account routing policy extends 07 Security & Governance (governance-as-middleware) and 15 Tenancy & Deployment Topology (data-residency tier); the decision record (ADR-031 below) is appended to the decision log. Do not run the docs index/changelog maintainer against this file until it is promoted. The ADR number is provisional until promotion (latest accepted is ADR-030).

Precis

Thinklio should route inference through a single model router keyed on abstract capability tiers — easy | medium | deep — not on concrete model names. Callers that know their intent (a recurring deep-research task, an email triage) inject the tier at the call site; the router honours it. When no cue is provided, the router falls back to a best-effort classifier that emits the same tier vocabulary, so injected and inferred paths converge on one interface. The mapping from tier → model/server/provider is configuration, and it is per-account policy-overridable, which is what makes self-hosted and cloud backends swappable without touching product code.

The strategic case is twofold. Financially, the major providers' current pricing is subsidised below cost; a router is optionality you can only buy in advance — when the correction comes (price hikes, tier removal, rate-limit tightening) you re-bind a tier instead of rewriting call sites. Architecturally, tiers decouple intent (stable, owned by product) from implementation (volatile, owned by ops), and — uniquely for a multi-tenant platform — let "this account's data may only touch self-hosted models" become a routing policy enforced in the same governance middleware that already gates everything, i.e. data residency as a sellable guarantee rather than a privacy preference.

This proposal also records the learnings from studying Odysseus (PewDiePie's self-hosted AI workspace, ~80k stars), which independently validates the "integrated work-surface" bet ([[adoption-surface-first]]) and supplies a concrete reference for the self-hosted serving layer.

1. Background — what Odysseus tells us

Odysseus is a single-user, self-hosted, privacy-first AI workspace bundling chat + agents, deep research, a document editor, email (IMAP/SMTP triage/summary/draft), notes, tasks, and calendar (CalDAV), plus local-model serving with a hardware-aware "Model Cookbook." It runs on Docker Compose (NVIDIA/AMD GPU variants) on localhost:7000. Its positioning is largely the inverse of Thinklio — single-tenant vs. multi-tenant SaaS, local-first vs. cloud, privacy-via-self-host vs. governance-via-middleware — so its infrastructure choices mostly do not transfer. Its product shape and its serving layer do.

What transfers:

• Breadth of surface is the product. ~80k stars went not to a bare agent framework but to an integrated workspace where the agent acts across chat, docs, email, tasks. Strong external corroboration of Thinklio's surface-first bet ([[adoption-surface-first]]).
• Email as an agent workload, not just a channel. Triage → summarise → draft-reply is a concrete, demoable behaviour that rides Thinklio's existing chats-as-core-abstraction and channel model (06 Events, Channels & Messaging, 16 Chats, Channels & Identity).
• "Deep Research" as a first-class surface, producing a document artifact (fits the docs surface) rather than a chat mode.
• The "Compare" feature (blind side-by-side model outputs + synthesis) is both a user feature and, more importantly, an eval harness — the exact instrument you need to gain confidence to move a task class down a tier (see §5).
• The Model Cookbook — hardware-aware model recommendations and local serving — becomes a genuine reference for the one piece of Thinklio that will not live on Convex: the GPU serving box (§6).

What does not transfer: local-only-as-moat (Thinklio's moat is multi-tenancy + governance + Platform API); raw shell/filesystem access for agents (a governance non-starter in multi-tenant — Thinklio's policy middleware is the correct answer, per 07 Security & Governance); hardware-profiling as such (the transferable kernel is "pick the right model for the job" = routing, not hardware detection).

2. Tiers, not models

The router's public contract is a small enum of capability tiers, not a model catalogue:

Tier	Intent	Rough contract (illustrative)
easy	High-volume, low-stakes; classification, extraction, short replies, triage	Cheapest LLM route; tight latency budget; capability floor = "reliable on bounded, well-specified tasks"
medium	Default general-purpose reasoning and drafting	Balanced cost/quality; the everyday workhorse
deep	Complex reasoning, long-horizon planning, research synthesis, agentic tool loops	Highest capability; latency/cost budget relaxed

Two rules make this abstraction actually buy optionality rather than just relocating the coupling:

1. A tier is defined by what it guarantees, not what it maps to — a latency budget, a cost ceiling, and a capability floor. The concrete model/server/provider is merely the current implementation that satisfies the contract, freely swappable because the contract does not change. Without a written contract, "medium" drifts into meaning five different things to five surfaces within six months.
2. The tier vocabulary is small but versioned. easy | medium | deep is the starting set. Leave room for a tier below easy — deterministic / rules / no-LLM for the truly trivial (a regex classification, a canned response), which is the cheapest possible route and where a surprising amount of "easy" traffic actually belongs. Do not build it now, but do not design the enum as if three is a permanent ceiling: a later caller that only knows three tiers must still resolve to a sane default.

3. Injected classification, with inference as fallback

The caller at the call site almost always knows more about intent than any post-hoc classifier can reconstruct from prompt text — a recurring "deep research" task knows it is deep; email triage knows it is easy. Re-inferring that throws away information you already had.

• Default: the caller injects the tier. Classify at the source; route at the router.
• Fallback: best-effort inference. When no tier is supplied, the router runs a fast classifier that emits the same tier vocabulary, so both paths converge on one interface. The classifier should itself run on the cheapest viable route (ideally a tiny local model), because it sits on every un-hinted request's latency and cost path.
• Per-surface defaults so nobody has to think: email → easy, research → deep, chat → medium. A sensible default per surface is what keeps injected classification from degenerating into "everyone asks for deep to be safe."

4. Escalation, and the floor/ceiling question

The injected tier is an entry point + escalation-allowed-up, capped by policy — not an exact, immutable selection:

• Escalate up on verification failure. A medium result that fails a fast verifier (a small local model, a rubric, or self-consistency) escalates to deep. This inverts the dominant risk of pure upfront classification (misclassify-hard-as-easy → wrong answer the user trusts, often silently): the expensive route only fires when the cheap route demonstrably failed, and every escalation is a free labelled "classifier was wrong" training example.
• The tradeoff is latency — a failed cheap attempt + escalation is slower than routing right the first time. This hurts interactive chat and barely touches the async / recurring / background task workloads Thinklio is leaning into; the asymmetry favours the surface-first bet.
• Downgrade is not automatic. If the caller said deep, respect it — unless a cost policy forces a downgrade, in which case it must be a visible, logged decision, never silent.
• Policy caps the max tier an account or surface may reach (cost governance), and sets the tier → backend binding (the data-residency case, §7).

5. You cannot route what you cannot measure

Two things are infrastructure, not features, and must ship with the router:

1. Per-route telemetry — cost, latency, and a quality proxy, tagged by (surface, requested_tier, resolved_tier, escalated_to, backend), on every call. Without it the classifier is tuned blind. This is a natural fit for Convex: log a routing-decision record per call and the reactive dashboards come for free. It is also the instrument that catches cost creep — a surface systematically over-requesting deep — so it can be fixed or capped.
2. Continuous eval — the mechanism that tells you which task classes a given backend is actually safe for at a given tier. This is where Odysseus's "Compare" stops being a cute UX feature and becomes the eval harness: blind A/B of local-vs-frontier on real traffic is how you earn the confidence to rebind a task class to a cheaper tier.

6. Topology — router on Convex, serving box off it

• Router + classifier + telemetry + policy → Convex functions/actions. Natural home: reactive, multi-tenant-aware, co-located with governance (zero-hop policy checks, per ADR-017 / 07 Security & Governance).
• Self-hosted serving layer (vLLM / Ollama / TGI on GPU) → off Convex. A Convex action calls out to it, exactly like the thinklio-services third-party proxies (ADR-030). This is a new self-managed dependency in the 15 Tenancy & Deployment Topology / 02 §16 picture.
• Health-check and fallback are mandatory. Per the [[api-thinklio-ai-offline-legacy-box]] lesson, a self-managed box that a request path depends on needs real health-checking and an automatic cloud-fallback path, so a dead or saturated GPU node degrades to a frontier provider rather than erroring. The tier abstraction makes this clean: "backend for tier X is unhealthy → resolve tier X to its fallback binding."

7. Routing policy is governance

This is the part that is uniquely Thinklio's vs. Odysseus's. In a single-user box, local-only is a privacy preference. In a multi-tenant platform, "this account may only be served by self-hosted models" is a per-account routing policy, enforced in the same governance middleware that already gates every query and mutation. So the router does not merely optimise cost/quality — it takes a policy input per principal (account / team / agent) that can (a) pin the tier → backend binding (data residency, no-third-party-processing guarantees), and (b) cap the maximum reachable tier (cost governance). Design the policy interface now, even if the initial policy is a no-op; it composes directly with governance-as-middleware and turns self-hosting from a cost lever into a sellable enterprise guarantee.

8. What is cheap now vs deferred

• Cheap / now: the tier enum + contract; the router interface accepting an injected tier; a trivial mapping (all tiers → current frontier models) so nothing regresses; per-route telemetry; the policy input as a no-op.
• Deferred / measured: the fallback classifier; the verifier + escalation loop; the self-hosted serving box and its health-check/fallback; the eval/Compare harness; real per-account policy bindings; the sub-easy deterministic tier.

The ordering matters: ship the interface and telemetry first so every call is already tier-tagged and measured. Everything else (classifier, escalation, local backends, policy) can then be introduced behind a stable contract without re-touching call sites.

9. Draft decision record (draft ADR-031)

Provisional. Append to the decision log on promotion. Latest accepted is ADR-030.

ADR-031: Model Router Is Tier-Based, Policy-Aware, and Injection-First

Date: 2026-07-05 Status: Draft

Context: Thinklio currently sends inference to a single endpoint whose logic classifies the task and routes it. The strategic direction is tiered inference — self-hosted models for the bulk of simple/medium work, frontier cloud models reserved for complex work — driven by (a) the financially untenable, below-cost pricing of the major providers and the need for cost control / repricing insulation, and (b) multi-tenant data-residency requirements that only self-hosted serving can satisfy. The design question is how to structure the router so that models, servers, and providers are swappable without churning product code, and so that classification is accurate without paying a classifier on every request.

Decision: The model router's public contract is a small, versioned enum of capability tiers (easy | medium | deep) — not concrete models. Each tier is defined by a contract (latency budget, cost ceiling, capability floor); the model/server/provider binding is configuration. Callers inject the tier at the call site; the router honours it and runs a best-effort classifier (same tier vocabulary) only as fallback when no tier is supplied. The injected tier is an entry point with escalation allowed up-tier on verification failure, capped by a per-account routing policy enforced in governance middleware; the policy also pins the tier → backend binding for data-residency accounts. The router, classifier, telemetry, and policy live on Convex; self-hosted serving lives off Convex behind a health-checked action with a mandatory cloud-fallback path. Ship the interface + per-route telemetry first; defer classifier, escalation, local backends, eval, and real policy bindings behind the stable contract.

Reasoning: Tiers decouple intent (stable, product-owned) from implementation (volatile, ops-owned), which is what makes backends swappable and converts a future repricing/deprecation from a rewrite into a re-bind. Injection-first uses information available at the call site that a post-hoc classifier would have to reconstruct and often get wrong; escalation-on-failure inverts the asymmetric cost of misclassification (only pay for deep when cheaper demonstrably failed) and yields free training signal. Routing-as-policy composes with governance-as-middleware (ADR-017) and turns self-hosting into a sellable data-residency guarantee, not just a cost lever — the one place this design diverges sharply from single-tenant self-hosted tools like Odysseus. Telemetry-first is non-negotiable because an untuned router is worse than none.

Consequences / follow-ups: new self-managed GPU serving dependency in the topology (02 §16 / ADR-015 tenancy); an eval/"Compare" harness becomes a prerequisite for demoting any task class to a cheaper tier; per-surface default tiers must be defined; cost-creep monitoring via tier-tagged telemetry.

10. Open questions

• Is today's classifier an LLM call or heuristic? If an LLM call, it is the first workload to push onto a tiny local model (it taxes every un-hinted request).
• Is the injected tier best modelled as a floor, a hint, or an exact selection per surface? (This proposal defaults to entry point + escalate-up, policy-capped; some latency-sensitive surfaces may want exact, no-escalation.)
• What is the minimal quality proxy for the verifier that is cheaper than the escalation it prevents?
• Does the sub-easy deterministic tier warrant a distinct enum value, or is it just a backend binding of easy?
• Where does the tier live in the harness call path (03) — chosen by the agent, by the invoking surface, or by the task record?

11. Revision history

• 2026-07-05 — Initial draft. Captures the Odysseus study and the tiered-routing design discussion; provisional ADR-031.