Route between models

Supported in ADKTypeScript v1.0.0Experimental

Experimental

Model routing is experimental and may change in future releases. We welcome your feedback!

An LlmAgent uses a single model by default. When you need to dynamically select between different models for each request, you can define a routing function that chooses which model to use. RoutedLlm provides this capability, enabling model fallback on error, A/B testing between models, and auto-routing by input complexity. If the selected model fails before producing any output, the routing function is called again with error context so it can select a different model.

Pass a RoutedLlm as an LlmAgent's model parameter. Use RoutedLlm when only the model varies between routes. If you also need to switch instructions, tools, or sub-agents, use RoutedAgent instead.

How routing works

The LlmRouter function receives the map of available models and the current LlmRequest, and returns the key of the model to use:

TypeScript

type LlmRouter = (
  models: Readonly<Record<string, BaseLlm>>,
  request: LlmRequest,
  errorContext?: { failedKeys: ReadonlySet<string>; lastError: unknown },
) => Promise<string | undefined> | string | undefined;

The models parameter accepts either a Record<string, BaseLlm> with explicit keys, or an array of BaseLlm instances. If an array is provided, each model's name is used as its key.

Failover follows the same rules as RoutedAgent: the router is re-called with errorContext only if the selected model fails before yielding any response. After yielding, errors propagate without retry. The router can return undefined to stop retrying and propagate the last error.

Live connections: RoutedLlm.connect() selects the model at connection time. Once a live connection is established, the model cannot be switched mid-stream.

Basic usage

The following example creates a RoutedLlm that tries a primary model first and falls back to a secondary model if the primary fails. The router checks errorContext.failedKeys to avoid re-selecting the failed model:

TypeScript

import {
  BaseLlm,
  Gemini,
  LlmRequest,
  LlmAgent,
  RoutedLlm,
  InMemoryRunner,
} from '@google/adk';

const primaryModel = new Gemini({ model: 'gemini-flash-latest' });
const fallbackModel = new Gemini({ model: 'gemini-pro-latest' });

const router = (
  models: Readonly<Record<string, BaseLlm>>,
  request: LlmRequest,
  // errorContext is provided when a previously selected model fails
  errorContext?: { failedKeys: ReadonlySet<string>; lastError: unknown },
) => {
  if (!errorContext) {
    return 'primary'; // Try primary first
  }
  if (errorContext.failedKeys.has('primary')) {
    return 'fallback'; // Fall back if primary failed
  }
  return undefined; // No more options, propagate the error
};

const routedLlm = new RoutedLlm({
  models: { primary: primaryModel, fallback: fallbackModel },
  router,
});

// Use RoutedLlm as the model for an LlmAgent
const agent = new LlmAgent({
  name: 'my_agent',
  model: routedLlm,
  instruction: 'You are a helpful assistant.',
});

const runner = new InMemoryRunner({ agent, appName: 'my_app' });

const session = await runner.sessionService.createSession({
  appName: 'my_app',
  userId: 'user_1',
});

const run = runner.runAsync({
  userId: 'user_1',
  sessionId: session.id,
  newMessage: { role: 'user', parts: [{ text: 'Hello!' }] },
});

for await (const event of run) {
  if (event.content?.parts?.[0]?.text) {
    console.log(event.content.parts[0].text);
  }
}