Skip to content

Router API

Complete API reference for the Router class - intelligent routing across multiple backend providers.

Creates a new Router. Backends are not passed to the constructor - register them afterwards with register().

Parameters:

  • config?: Partial<RouterConfig> - Router configuration (see RouterConfig)

Returns: Router instance

Example:

import { Bridge, Router } from '@johnhenry/aimatey-core';
import { OpenAIFrontendAdapter } from '@johnhenry/aimatey-frontend/openai';
import { AnthropicBackendAdapter } from '@johnhenry/aimatey-backend/anthropic';
import { OpenAIBackendAdapter } from '@johnhenry/aimatey-backend/openai';
const router = new Router({
routingStrategy: 'round-robin',
fallbackStrategy: 'sequential',
});
router
.register('anthropic', new AnthropicBackendAdapter({ apiKey: process.env.ANTHROPIC_API_KEY! }))
.register('openai', new OpenAIBackendAdapter({ apiKey: process.env.OPENAI_API_KEY! }));
const bridge = new Bridge(new OpenAIFrontendAdapter(), router);

There is also a factory function, createRouter(config?), with the same signature.


Execute an IR request with automatic backend selection and fallback. This is the BackendAdapter entry point - normally you call bridge.chat() and the bridge calls this for you.

Parameters:

  • request: IRChatRequest - Request in Intermediate Representation form
  • signal?: AbortSignal - Cancellation signal

Returns: Promise<IRChatResponse>


Execute a streaming IR request. It is an async generator method, so the stream is returned synchronously - there is nothing to await before iterating.

Parameters:

  • request: IRChatRequest
  • signal?: AbortSignal

Returns: IRChatStream (AsyncGenerator<IRStreamChunk>)


Register a backend under a name. The name is what you use in fallback chains, model mappings, per-request overrides and statistics.

Parameters:

  • name: string - Backend identifier
  • adapter: BackendAdapter - Backend adapter instance
  • options?: BackendRegistrationOptions - Per-backend settings (see Per-backend circuit breaker)

Returns: Router (for chaining)

Example:

import { GroqBackendAdapter } from '@johnhenry/aimatey-backend/groq';
router.register('groq', new GroqBackendAdapter({ apiKey: process.env.GROQ_API_KEY! }));

replace(name, adapter) / unregister(name, options?)

Section titled “replace(name, adapter) / unregister(name, options?)”

Swap a registered backend for another instance, or remove one. Both return the router for chaining. Unregistering the backend named by defaultBackend clears defaultBackend and emits a routing-config-changed warning through config.onWarning. replace() keeps the per-backend options given to register() (they are policy about the name, like latency history) and resets the health verdict.

router.replace('openai', new OpenAIBackendAdapter({ apiKey: rotatedKey }));
router.unregister('groq');

unregister() is not cancellation. A call already handed to the backend runs to its natural end: an execute() resolves and a stream keeps yielding, because the call holds the adapter it started on. From the moment unregister() returns:

  • no new request is routed to the name;
  • the circuit-breaker recovery timer is cancelled;
  • the in-flight call’s outcome is not accounted - no counters, no stats, no breaker. The backend is gone, so a late failure must not land on an object nobody can read, nor trip the breaker of a different backend registered under the same name afterwards.

To wait for in-flight calls, pass drain. The backend is still removed synchronously; you get a promise that settles when the calls have finished (streams included) or the timeout passed:

router.unregister('desktop'); // Router, synchronously
const result = await router.unregister('desktop', { drain: true });
// { drained: true, inFlight: 0 } -- safe to dispose of the adapter now
await router.unregister('desktop', { drain: 5_000 });
// { drained: false, inFlight: 1 } -- gave up after 5 s; the call still finishes

A stream the consumer never finishes or closes keeps its backend “in flight”, so use a timeout when you cannot guarantee consumers read to the end. The number of calls currently running is getBackendInfo(name).inFlight.

By default unregister() stops new work only. If the answer must not be delivered - a revoked device, a rotated credential - pass { abort: true }:

router.unregister('desktop', { abort: true }); // cut, now
await router.unregister('desktop', { abort: true, drain: 5_000 }); // cut, then wait for unwinding

The router owns an AbortController per call, linked to the caller’s own signal (so the caller keeps its control), and abort fires every one running against that backend - chat, stream, embedding and decision calls alike:

  • the adapter’s AbortSignal aborts with an AbortError, and adapter.cancel?.(requestId, reason) is called once for the call, for a far side that cannot see a signal (best effort; a throwing or rejecting cancel() is ignored);
  • the caller receives that AbortError whatever the adapter made of the abort, even an adapter that ignores its signal. A stream ends by throwing it, and nothing the revoked backend yields afterwards is delivered;
  • it is not failed over. The router does not answer from another backend a request it was just told to cut; the caller decides whether to retry;
  • calls on other backends are untouched, and the cut calls’ late outcomes are not accounted, as for any unregistered backend.

cancel() is sent for revocation only. A caller’s abort is not relayed by the router (a proxy behind a router watches the signal itself; Bridge sends cancel() for a direct backend), so no far side is told twice. Whether a transport really stops is still the adapter’s business; revocation guarantees only that the caller is released and gets nothing further.


Inspect the registry.

router.get('openai'); // BackendAdapter | undefined
router.has('openai'); // boolean
router.listBackends(); // readonly string[]

setFallbackChain(chain) / getFallbackChain()

Section titled “setFallbackChain(chain) / getFallbackChain()”

Set the order backends are tried in after the primary fails. Every name must already be registered, otherwise an AdapterError with ErrorCode.ROUTING_FAILED is thrown. Without a chain, 'sequential' fallback tries every other available backend in registration order.

router.setFallbackChain(['openai', 'groq']);
router.getFallbackChain(); // readonly string[]

setModelMapping(mapping) / getModelMapping()

Section titled “setModelMapping(mapping) / getModelMapping()”

Map model names to backend names, for routingStrategy: 'model-based'.

router.setModelMapping({
'gpt-4': 'openai',
'claude-haiku-4-5-20251001': 'anthropic',
});

Run the routing strategy and return the chosen backend name without executing anything. Useful in tests.

Returns: Promise<string> - throws ErrorCode.NO_BACKEND_AVAILABLE if nothing is available.


Send one request to several backends at once.

Parameters:

  • request: IRChatRequest
  • options?: ParallelDispatchOptions

Returns: Promise<ParallelDispatchResult>

const result = await router.dispatchParallel(irRequest, {
backends: ['openai', 'anthropic'],
strategy: 'first',
});
console.log(result.response, result.successfulBackends, result.totalTimeMs);

Answer typed decision questions via the best available backend. Mirrors embed(): candidates are registered backends that implement decide(), whose circuit is not open, and that can serve the request (a backend whose decisionTypes, decisionLimits or decisionImages rule it out is skipped, not failed). They are tried in fallback-chain order, then the default backend, then registration order.

Parameters:

  • request: IRDecisionRequest
  • signal?: AbortSignal

Returns: Promise<IRDecisionResponse>

const response = await router.decide({
state: ticket,
questions,
metadata: { requestId: 'r1', timestamp: Date.now() },
});

parameters.model is a hint: a candidate that declares decisionModels without it is tried after the rest, not excluded. Throws UNSUPPORTED_FEATURE when no backend supports decisions.


Actively probe backends by calling each adapter’s healthCheck(). An adapter that does not implement healthCheck() is reported healthy.

Returns: Promise<Record<string, boolean>>, or Promise<boolean> when a name is given.

const health = await router.checkHealth();
// { anthropic: true, openai: true, groq: false }
const openaiHealthy = await router.checkHealth('openai');

Set healthCheckInterval in the config to run this automatically in the background.


Report the last known state of each backend - health, circuit breaker, and stats - without probing.

Returns: BackendInfo[], or BackendInfo | undefined when a name is given.

for (const info of router.getBackendInfo()) {
console.log(
info.name,
info.isHealthy ? '✅' : '❌',
info.circuitBreakerState,
`${info.stats.averageLatencyMs}ms`,
`${info.stats.successRate}%`
);
}

router.openCircuitBreaker('openai', 30_000); // force open; 30 s is the rest period of THIS open
router.closeCircuitBreaker('openai');
router.resetCircuitBreaker(); // all backends when name is omitted
router.isCircuitBreakerOpen('openai'); // boolean

openCircuitBreaker(name, timeoutMs) rests for exactly timeoutMs, in either direction: a value longer than the configured timeout is honoured, not capped by it. It applies only to that open.

enableCircuitBreaker, circuitBreakerThreshold, circuitBreakerTimeout and circuitBreakerWindow on RouterConfig are defaults. One router often fronts backends that fail very differently - a cloud API that blips for seconds and a LAN peer that is asleep overnight - so register() takes per-backend overrides; any field you leave out inherits the router-wide value:

const router = new Router({
enableCircuitBreaker: true,
circuitBreakerThreshold: 5, // default for every backend
circuitBreakerTimeout: 60_000,
});
router
// Cloud API: tolerate a few blips, retry soon.
.register('openai', openai, { circuitBreaker: { threshold: 5, timeout: 15_000 } })
// LAN peer / docker container: a refused connection means "asleep", not "flaky".
// Trip fast, and rest for five minutes before probing it again.
.register('desktop', desktopTunnel, { circuitBreaker: { threshold: 2, timeout: 5 * 60_000 } })
// A backend that should never be rested.
.register('local', localModel, { circuitBreaker: { enabled: false } });
router.getBackendInfo('desktop')?.circuitBreaker;
// { enabled: true, threshold: 2, timeout: 300000, window: undefined, countAsFailure: fn,
// source: { enabled: 'router', threshold: 'register', timeout: 'register', ... } }

threshold must be a positive integer and timeout a non-negative number of milliseconds; anything else throws from register() before the backend is added. enabled: true gives one backend a breaker in a router that has none. getBackendInfo() reports the effective values, so you can see which layer won. The overrides survive replace() and clone().

By default the breaker counts consecutive failures with no notion of elapsed time: three refusals in four milliseconds trip it exactly like three failures over three minutes, and a success in between resets the count. That is the backward-compatible default and stays the behaviour unless you set a window.

window (milliseconds, per backend or as RouterConfig.circuitBreakerWindow) changes the question to “threshold failures within window”:

router.register('desktop', tunnel, { circuitBreaker: { threshold: 3, window: 10_000 } });

Timestamps of recent failures are kept per backend and pruned to the window. In windowed mode a success does not reset them (intermittent failure is still failure); they are forgotten when the breaker closes or is reset. A failed half-open probe reopens a windowed breaker at once.

A desktop loading a 7B model has a legitimate 40 s time-to-first-token that is not a failure. No threshold or timeout says so; a signal does:

  • An adapter that can tell reports ErrorCode.MODEL_LOADING (a retryable ProviderError). The default countAsFailure predicate does not count it toward the breaker, though it still appears in failedRequests. The shipped Ollama adapter (a 503 “loading model”, or a deadline that expired while /api/ps shows the model not resident), native-model-runner (a request while start() is waiting for the process) and native-node-llamacpp (a request while another request’s load is running) report it.
  • countAsFailure?: (error) => boolean replaces the default for one backend. It receives the thrown value, or the { code, message } of an in-band stream error chunk. It replaces the default, so one that should still ignore warm-up must say so; a predicate that throws counts the failure.
router.register('desktop', tunnel, {
circuitBreaker: { countAsFailure: (e) => (e as { code?: string }).code !== 'PROVIDER_TIMEOUT' },
});

An adapter distributed as a package can recommend its own policy in AdapterMetadata.circuitBreaker (threshold, timeout, window, countAsFailure; never enabled - whether breakers run at all is the application’s call). Precedence, per field: register() option > adapter metadata > RouterConfig. getBackendInfo(name).circuitBreaker.source reports the layer ('register' | 'adapter' | 'router') each field came from. The recommendation is read from the adapter currently registered, so replace() hands the name the replacement’s. An invalid recommendation throws from register()/replace() like an invalid override.


getStats() / resetStats() / getBackendStats(name)

Section titled “getStats() / resetStats() / getBackendStats(name)”
const stats = router.getStats();
console.log(stats.totalRequests, stats.totalFallbacks, stats.backendStats);
const openaiStats = router.getBackendStats('openai'); // BackendStats | undefined
router.resetStats();

clone() returns a new router with merged config and the same backends registered. dispose() stops the background health-check timer and clears the registry - call it when tearing down.


Configuration options for Router. Every field is optional and readonly.

interface RouterConfig {
routingStrategy?: RoutingStrategy; // default 'explicit'
fallbackStrategy?: FallbackStrategy; // default 'sequential'
defaultBackend?: string;
healthCheckInterval?: number; // ms; 0 disables. default 0
enableCircuitBreaker?: boolean; // default false
circuitBreakerThreshold?: number; // default 5
circuitBreakerTimeout?: number; // ms, default 60000
trackLatency?: boolean; // default true
trackCost?: boolean; // default false
capabilityBasedRouting?: boolean; // default false
optimization?: 'cost' | 'speed' | 'quality' | 'balanced'; // default 'balanced'
optimizationWeights?: { cost: number; speed: number; quality: number };
capabilityCacheDuration?: number; // ms, default 3600000
customRouter?: CustomRoutingFunction;
customFallback?: CustomFallbackFunction;
modelTranslation?: ModelTranslationConfig;
onWarning?: (warning: IRWarning) => void;
}

type RoutingStrategy =
| 'explicit' // Use the backend named on the request (default)
| 'model-based' // Route by model name via setModelMapping()
| 'cost-optimized' // Lowest observed average cost (needs trackCost)
| 'latency-optimized' // Lowest observed average latency (needs trackLatency)
| 'round-robin' // Distribute evenly
| 'random' // Random selection
| 'custom'; // Use config.customRouter

There is no 'priority', 'weighted', 'least-latency' or 'least-cost' strategy. Priority-style failover is 'explicit' + defaultBackend + setFallbackChain().


type FallbackStrategy =
| 'none' // Fail immediately
| 'sequential' // Try backends in order until one succeeds (default)
| 'parallel' // Try all remaining backends at once, take the first success
| 'custom'; // Use config.customFallback

type CustomRoutingFunction = (
request: IRChatRequest,
availableBackends: readonly string[],
context: RoutingContext
) => Promise<string | null>;

Returns: the name of the backend to use, or null to fall through to defaultBackend and then the first available backend. It is async, and it receives backend names - not adapters, and not indices.

Example:

import type { CustomRoutingFunction } from '@johnhenry/aimatey-types';
const selectBackend: CustomRoutingFunction = async (request, availableBackends) => {
const wordCount = request.messages
.map((m) => JSON.stringify(m.content).split(' ').length)
.reduce((a, b) => a + b, 0);
const preferred =
wordCount < 10 ? 'groq' : wordCount < 100 ? 'openai' : 'anthropic';
return availableBackends.includes(preferred) ? preferred : (availableBackends[0] ?? null);
};
const router = new Router({ routingStrategy: 'custom', customRouter: selectBackend });

type CustomFallbackFunction = (
request: IRChatRequest,
failedBackend: string,
error: AdapterError,
attemptedBackends: readonly string[],
availableBackends: readonly string[]
) => Promise<string | null>;

interface RoutingContext {
readonly stats: RouterStats;
readonly metadata: Record<string, unknown>;
readonly preferredBackend?: string;
}

interface BackendInfo {
readonly name: string;
readonly adapter: BackendAdapter;
readonly metadata: AdapterMetadata;
readonly isHealthy: boolean;
readonly lastHealthCheck?: number;
readonly circuitBreakerState: 'closed' | 'open' | 'half-open';
readonly consecutiveFailures: number;
readonly circuitBreaker: { enabled: boolean; threshold: number; timeout: number };
readonly inFlight: number;
readonly stats: BackendStats;
}

interface RouterStats {
readonly totalRequests: number;
readonly successfulRequests: number;
readonly failedRequests: number;
readonly totalFallbacks: number;
readonly parallelRequests: number;
readonly backendStats: Record<string, BackendStats>;
readonly sinceTimestamp: number;
}
interface BackendStats {
readonly totalRequests: number;
readonly successfulRequests: number;
readonly failedRequests: number;
readonly successRate: number; // 0-100
readonly averageLatencyMs: number;
readonly p50LatencyMs: number;
readonly p95LatencyMs: number;
readonly p99LatencyMs: number;
readonly totalCost?: number;
readonly averageCost?: number;
}

ParallelDispatchOptions / ParallelDispatchResult

Section titled “ParallelDispatchOptions / ParallelDispatchResult”
interface ParallelDispatchOptions {
readonly backends?: readonly string[];
readonly strategy?: ParallelStrategy; // default 'first'
readonly timeout?: number;
readonly cancelOnFirstSuccess?: boolean; // default true
readonly customAggregator?: (
responses: Array<{ backend: string; response: IRChatResponse; latencyMs: number }>
) => IRChatResponse;
}
interface ParallelDispatchResult {
readonly response: IRChatResponse;
readonly allResponses?: Array<{
readonly backend: string;
readonly response: IRChatResponse;
readonly latencyMs: number;
}>;
readonly successfulBackends: readonly string[];
readonly failedBackends: Array<{ readonly backend: string; readonly error: AdapterError }>;
readonly totalTimeMs: number;
}

Distributes requests evenly across the available backends.

const router = new Router({ routingStrategy: 'round-robin' });
router
.register('backend1', backend1)
.register('backend2', backend2)
.register('backend3', backend3);
// Request 1 → backend1
// Request 2 → backend2
// Request 3 → backend3
// Request 4 → backend1 (cycles)

Use case: Even load distribution, all providers have similar pricing/performance.


Randomly selects an available backend for each request.

const router = new Router({ routingStrategy: 'random' });
router.register('backend1', backend1).register('backend2', backend2);

Use case: Simple distribution, A/B testing.


The equivalent of a “priority” strategy: always start at one backend, then walk a chain.

const router = new Router({
routingStrategy: 'explicit',
defaultBackend: 'primary',
fallbackStrategy: 'sequential',
});
router
.register('primary', primaryBackend)
.register('secondary', secondaryBackend)
.register('tertiary', tertiaryBackend);
router.setFallbackChain(['secondary', 'tertiary']);

Use case: High availability, disaster recovery, primary + fallback providers.


const router = new Router({ routingStrategy: 'model-based' });
router.register('openai', openaiBackend).register('anthropic', anthropicBackend);
router.setModelMapping({
'gpt-4': 'openai',
'gpt-5.6-luna': 'openai',
'claude-haiku-4-5-20251001': 'anthropic',
});

Use case: One endpoint serving several providers’ model catalogues.


Selects the backend with the lowest observed average latency. Needs trackLatency (on by default) and some traffic history to work from.

const router = new Router({
routingStrategy: 'latency-optimized',
trackLatency: true,
healthCheckInterval: 60_000,
});

Use case: Optimize for response time, latency-sensitive applications.


Selects the backend with the lowest observed average cost. Requires trackCost: true and backends that implement estimateCost(); without it the strategy selects nothing and the router falls through to defaultBackend.

const router = new Router({ routingStrategy: 'cost-optimized', trackCost: true });

Use case: Cost optimization, budget-conscious applications.


import type { CustomRoutingFunction } from '@johnhenry/aimatey-types';
const selectBackend: CustomRoutingFunction = async (request, availableBackends) => {
const wordCount = request.messages
.map((m) => JSON.stringify(m.content).split(' ').length)
.reduce((a, b) => a + b, 0);
if (wordCount < 10) return 'groq'; // Simple → fast/cheap
if (wordCount < 100) return 'openai'; // Medium → balanced
return 'anthropic'; // Complex → powerful
};
const router = new Router({ routingStrategy: 'custom', customRouter: selectBackend });
router
.register('groq', groqBackend)
.register('openai', openaiBackend)
.register('anthropic', anthropicBackend);

Use case: Application-specific logic, multi-factor routing.


Set healthCheckInterval (milliseconds; 0 disables) and the router probes every backend in the background, marking failures unhealthy so routing skips them.

const router = new Router({
routingStrategy: 'round-robin',
healthCheckInterval: 60_000,
enableCircuitBreaker: true,
circuitBreakerThreshold: 5,
circuitBreakerTimeout: 60_000,
});

There is no event emitter on Router; poll getBackendInfo() to observe state.


const health = await router.checkHealth();
for (const [name, healthy] of Object.entries(health)) {
if (!healthy) {
const info = router.getBackendInfo(name);
console.log(`⚠️ ${name} is unhealthy`);
console.log(` Success rate: ${info?.stats.successRate.toFixed(1)}%`);
console.log(` Consecutive failures: ${info?.consecutiveFailures}`);
console.log(` Circuit breaker: ${info?.circuitBreakerState}`);
}
}

import type { CustomRoutingFunction } from '@johnhenry/aimatey-types';
const PRICING: Record<string, number> = {
groq: 0.00027,
deepseek: 0.0002,
anthropic: 0.0008,
openai: 0.0015,
};
const cheapest: CustomRoutingFunction = async (request, availableBackends) => {
const tokens = Math.ceil(JSON.stringify(request.messages).length / 4);
let best: string | null = null;
let lowestCost = Infinity;
for (const name of availableBackends) {
const cost = ((PRICING[name] ?? Infinity) * tokens) / 1000;
if (cost < lowestCost) {
lowestCost = cost;
best = name;
}
}
return best;
};
const router = new Router({ routingStrategy: 'custom', customRouter: cheapest });

availableBackends already excludes unhealthy and circuit-open backends, so a custom router only has to weigh the factors it cares about.

import type { CustomRoutingFunction } from '@johnhenry/aimatey-types';
const intelligentRouting: CustomRoutingFunction = async (request, availableBackends, context) => {
if (availableBackends.length === 0) return null;
const complexity = analyzeComplexity(request);
const hour = new Date().getHours();
const isOffPeak = hour < 6 || hour > 22;
const pick = (name: string) => (availableBackends.includes(name) ? name : null);
if (complexity > 80) return pick('anthropic') ?? availableBackends[0]!;
if (isOffPeak && complexity < 30) return pick('groq') ?? availableBackends[0]!;
// context.stats carries live per-backend statistics if you want to weigh them
return pick('openai') ?? availableBackends[0]!;
};
const router = new Router({
routingStrategy: 'custom',
customRouter: intelligentRouting,
healthCheckInterval: 60_000,
});

const router = new Router({
routingStrategy: 'explicit',
defaultBackend: 'primary',
fallbackStrategy: 'sequential',
});
router.register('primary', primaryBackend).register('fallback', fallbackBackend);
router.setFallbackChain(['fallback']);
const bridge = new Bridge(new OpenAIFrontendAdapter(), router);
// If primary fails, this automatically uses fallback
const response = await bridge.chat(request);
// Afterwards, the counters tell you whether a fallback happened
console.log(router.getStats().totalFallbacks);

Routing failures surface as AdapterError with a routing error code. There is no AllBackendsFailedError class - check error.code instead.

import { AdapterError, ErrorCode } from '@johnhenry/aimatey-errors';
try {
const response = await bridge.chat(request);
} catch (error) {
if (error instanceof AdapterError) {
switch (error.code) {
case ErrorCode.ALL_BACKENDS_FAILED:
console.error('Every backend in the chain failed');
break;
case ErrorCode.NO_BACKEND_AVAILABLE:
console.error('No healthy backend to route to');
break;
case ErrorCode.ROUTING_FAILED:
console.error('Routing rejected the request (e.g. unknown backend name)');
break;
}
}
}

RouterError (a subclass of AdapterError) carries an attemptedBackends array when it is thrown.