Designing an LLM Router You Can Trust

Say you're running a federated tool hub: several MCP servers behind a gateway like AgentGateway, fronted by a thin HTTP routing proxy that decides which tool a request should dispatch to. That proxy is stateless and serves many different clients, so it can't hold a live MCP session open just to get push-based tools/list_changed invalidation. It doesn't own the hub either, so it can't hardcode or codegen a schema against a tool list that changes on someone else's deploy schedule. Every dispatch still has to name a tool that exists in the hub right now, not whatever the hub looked like the last time the proxy happened to check.

Tool Router is a runnable demo of the pattern that solves this: discover the tool list from the hub, cache it for a short TTL instead of polling before every dispatch or holding a session open for push invalidation, and force the model's tool choice into an enum built from that cache, never a name typed into the router's own code or a shared schema package.

Scope: this is a teaching example distilled out of that scenario, not the federated gateway or the proxy itself. MCP standardizes discovery and invocation between a client and one server; it doesn't give a stateless, multi-tenant routing layer in front of a federation of servers a cheap way to stay current without either polling every request or keeping a live session per instance. This pattern fills that gap by building on top of MCP, not replacing it.

Architecture

router-core only knows a generic GET /tools HTTP contract. It never imports hub-core's types. That boundary mirrors a routing proxy sitting in front of a hub it doesn't own: the router can't cheat by reaching into the hub's internals, so proving the pattern works over HTTP proves it works across a real service boundary.

Discovery: a cache, not a fetch-every-time

Calling the hub before every dispatch is an unnecessary round-trip, and push-based invalidation means holding open a stateful session a stateless, multi-tenant proxy doesn't have. Fetching once at startup reintroduces the exact stale-list problem this exists to fix, just moved one layer down. ToolDiscovery sits in between: a short TTL cache-fill, refreshed lazily.

export class ToolDiscovery {
  private cache: { tools: HubTool[]; fetchedAt: number } | null = null

  constructor(
    private hub: HubClient,
    private ttlMs: number,
    private now: () => number = Date.now,
  ) {}

  async discover(): Promise<HubTool[]> {
    const t = this.now()
    if (this.cache && t - this.cache.fetchedAt < this.ttlMs) {
      return this.cache.tools
    }
    const tools = await this.hub.listTools()
    this.cache = { tools, fetchedAt: t }
    return tools
  }

  getCached(): HubTool[] | null {
    return this.cache?.tools ?? null
  }
}

The injected clock lets tests assert the cache refetches after the TTL window and doesn't refetch inside it, without sleeping in real time.

The mechanism: forced tool-choice over a live enum

This is what actually eliminates hallucination, not just reduces it. Every routing call builds a synthetic route tool whose input schema constrains the tool name to an enum of whatever ToolDiscovery currently has cached, then forces the model into using exactly that tool:

const routeTool = {
  name: 'route',
  description: "Route the user's query to exactly one of the currently available tools.",
  input_schema: {
    type: 'object',
    properties: {
      tool: { type: 'string', enum: toolNames, description: 'Name of the tool to invoke.' },
      input: { type: 'object', description: 'Arguments to pass to the selected tool.' },
    },
    required: ['tool'],
  },
}

const response = await client.messages.create({
  model: 'claude-sonnet-5',
  max_tokens: 1024,
  tools: [routeTool],
  tool_choice: { type: 'tool', name: 'route' },
  messages: [{ role: 'user', content: query }],
})

Two constraints, stacked. tool_choice forces the model to use this tool at all: no plain-text answer, no picking something else. The enum on top constrains which value goes in the tool field. There's no free-text slot anywhere in the response for the model to write a name that doesn't exist. It's not that hallucination is unlikely. The schema has no room for it.

Defense in depth

An enum rebuilt from live state on every call should make an out-of-list selection impossible. selectTool() checks anyway:

export async function selectTool(
  discovery: ToolDiscovery,
  client: ChatClient,
  query: string,
): Promise<ToolSelection> {
  const cached = discovery.getCached()
  if (!cached) {
    throw new DiscoveryNotRunError()
  }

  const toolNames = cached.map((tool) => tool.name)
  const selection = await client.routeQuery(query, toolNames)

  if (!toolNames.includes(selection.toolName)) {
    throw new UnknownToolSelectedError(selection.toolName, toolNames)
  }

  return selection
}

The first guard: you can't route before you've ever discovered what's routable. The second is the one that should never fire in the happy path. It's there for the case the enum constraint doesn't hold: a client bug, an API change, a cache gone stale in some way the TTL didn't catch. The demo proves it works by wrapping the client in a FaultInjectingChatClient that forces a stale tool name, and watching UnknownToolSelectedError catch it.

Watching it adapt

npm run demo starts a real Fastify hub in-process, discovers its tools over HTTP, mutates the hub's tool list mid-run, and re-routes the same query with no code change in between:

[3] Discovering tools from the hub
  - get_weather: Look up the current weather forecast for a city.
  - search_docs: Search internal documentation for a query.
  - send_email: Send an email to a recipient.
  - summarize_text: Summarize a block of text.

[4] Routing query: "Can you check the weather forecast for Austin, TX?"
  Routed to: get_weather

[5] Hub changes at runtime (adds 'lookup_order', removes 'send_email')

[6] TTL window expires, re-discovering
  - send_email
  + lookup_order

[7] Re-routing the same query: "Can you check the weather forecast for Austin, TX?"
  Routed to: get_weather (enum rebuilt from live hub state, not a hardcoded list)

[8] Guard check: model selects a tool that no longer exists
  Caught UnknownToolSelectedError: Model selected "send_email", which is not
  in the current hub tool list: [get_weather, search_docs, summarize_text, lookup_order]

get_weather survives the mutation because there was never a hardcoded list to update. send_email doesn't, and the router notices immediately instead of silently calling a tool that's gone.

Source on GitHub.