Cost Optimization

~8 min readUpdated: October 2026

Agent Token Cost Routing: Cut LLM Spend with Model Gateways

Agent token cost routing across model providers with the Smoke Monkey harness and Canvas spend view

Agent costs are dominated by input tokens re-sent every turn, and the fix is **routing**: send each step to the cheapest model that can handle it. In 2026 this drove the **AI gateway** wave (C1, LLM Gateway, Nutanix) — a proxy that routes, caches, and meters model calls. This guide explains token cost routing and how [Smoke Monkey Harness](/solutions/what-is-an-ai-agent-harness) does it in-process through a multi-provider API, with **Smoke Monkey Canvas** showing per-agent spend.

Technical Review: Smoke Monkey Core Architecture Team

|Tested on Node.js 18+ & Bun|TypeScript 5.x

Quick Answer & Executive Definition

Agent Token Cost Routing: Cut LLM Spend with Model Gateways: Agent costs are dominated by input tokens re-sent every turn, and the fix is **routing**: send each step to the cheapest model that can handle it. In 2026 this drove the **AI gateway** wave (C1, LLM Gateway, Nutanix) — a proxy that routes, caches, and meters model calls. This guide explains token cost routing and how [Smoke Monkey Harness](/solutions/what-is-an-ai-agent-harness) does it in-process through a multi-provider API, with **Smoke Monkey Canvas** showing per-agent spend. Designed as a zero-dependency, open-source TypeScript architecture under the MIT License with native Model Context Protocol (MCP) support and deterministic phase state machines.

Key Architectural Takeaways

Agent spend is mostly re-sent context; routing the right step to the right model is the biggest lever.
An AI gateway centralizes routing, caching, and metering across providers.
Smoke Monkey Harness routes in-process via a multi-provider API — no extra proxy hop required.
Smoke Monkey Canvas reports per-agent token usage so you can catch a costly agent early.

Quick Implementation Examplecost-routing.ts

cost-routing.tstypescript

1import { createAgent } from 'smoke-monkey-harness';
2
3// Planner on a frontier model, executors on a cheap local model
4const planner = createAgent({ provider: 'anthropic', model: 'claude-opus-5', workspacePath: process.cwd() });
5const plan = await planner.run('Decompose the task into cheap, mechanical steps. Output JSON.');
6
7const steps = JSON.parse(plan.output).steps;
8const results = await Promise.all(
9  steps.map((s) =>
10    createAgent({
11      provider: 'ollama',            // local = near-zero marginal cost
12      model: 'qwen3:8b',
13      workspacePath: process.cwd(),
14      onUsage: (u) => cost.add(u),
15    }).run(s.instruction)
16  )
17);
18
19console.log('spend', cost.total());
20// Per-agent spend on the canvas: npx @smoke-monkey/canvas start

Video Guides

Watch: Related Video Guides

Anthropic Just Built an Agentic OS — Open Source Harness Breakdown

Smoke Monkey

What is an AI Gateway and Why Do Your LLM Workloads Need One?

Nutanix | I/O

Where Agent Token Cost Comes From

A chat call sends one prompt. An agent sends the whole running history on every turn, so input tokens — not output — dominate the bill, and cost grows with every iteration. A 40-step run that re-reads a large context can cost far more than the task looks like it should. Two levers fix it: reduce what you send (context compaction) and send it to the right model (routing). The 2026 AI gateway trend formalized the second lever. Smoke Monkey Harness bakes both into the runtime, and Smoke Monkey Canvas shows the resulting spend per agent.

What an AI Gateway Does

An AI gateway — C1, LLM Gateway, and Nutanix's model-routing layer are 2026 examples — sits between your app and the providers. It routes a request to the best model for the job, caches identical calls, enforces rate limits, and meters usage. Centralizing that logic is useful, but a proxy is another hop to operate. Smoke Monkey takes a different route: the harness is [multi-provider by design](/solutions/multi-provider-agent-apis), so routing is a provider and model choice inside your code. You get gateway-style routing without deploying a gateway, and you can still put a gateway in front if your organization standardizes on one.

routing.tstypescript

1import { createAgent } from 'smoke-monkey-harness';
2
3// Route by task shape: cheap models for mechanical steps, frontier for hard ones
4const runStep = (step) => {
5  const hard = step.kind === 'plan' || step.kind === 'review';
6  return createAgent({
7    provider: hard ? 'anthropic' : 'ollama',
8    model: hard ? 'claude-opus-5' : 'qwen3:8b',
9    workspacePath: process.cwd(),
10    onUsage: (u) => cost.add({ model: hard ? 'opus' : 'local', ...u }),
11  }).run(step.instruction);
12};
13
14await Promise.all(steps.map(runStep));

Routing Strategy for Agents

The reliable policy is frontier for planning, cheap for execution. Planning happens once and drives everything, so it pays to use a strong model; the many mechanical steps — edits, test runs, summaries — go to a fast, cheap model. Add caching for identical tool calls and a fallback provider so a rate-limit spike does not stall the run. Combine this with token cost optimization and local models for the mechanical tier, and agent spend drops sharply. Smoke Monkey Canvas shows each agent's model and token usage, so you can see which route is actually paying off.

routing-policy.tstypescript

1import { createAgent } from 'smoke-monkey-harness';
2
3// Caching + fallback completes the routing policy
4const agent = createAgent({
5  provider: 'anthropic',
6  model: 'claude-sonnet-5',
7  workspacePath: process.cwd(),
8  cache: { toolResults: true },       // identical tool calls are not re-billed
9  fallback: [{ provider: 'ollama', model: 'qwen3:8b' }],
10  onUsage: (u) => cost.add(u),
11});
12
13await agent.run('Run the routine maintenance steps for this repo');

Metering and Watching Spend

Routing only works if you can measure it. Smoke Monkey exposes onUsage per turn, which you can aggregate per agent, per phase, or per model — exactly the metering a gateway provides, but in-process. Pipe it to your metrics backend, or just watch it on Smoke Monkey Canvas (npx @smoke-monkey/canvas start), where each agent card reports token spend across 300+ MCP tools. That feedback loop is what lets you tune the routing policy instead of guessing — multi-provider routing you can see, on the open Smoke Monkey Harness.

Google Search Questions & Answers

Frequently Asked Questions

Q:What is agent token cost routing?

It is sending each step of an agent run to the cheapest model that can handle it — frontier models for planning, cheap or local models for mechanical execution — to cut total spend.

Q:Do I need an AI gateway for routing?

Not necessarily. Smoke Monkey Harness is multi-provider in-process, so routing is just a provider and model choice in code. A gateway is still useful if your organization standardizes on one.

Q:How much can routing save on agent costs?

Savings depend on the mix, but moving the many mechanical steps off frontier models onto local or cheap models typically removes the majority of token spend, especially combined with context compaction.

Q:Can I see cost per agent?

Yes. Smoke Monkey reports token usage per turn via onUsage, and Smoke Monkey Canvas shows per-agent spend across the fleet, so you can spot an expensive route quickly.

Related Alternatives & Comparisons

vs Openai Codex Alternative Open Source

Openai Codex Alternative Open Source: Free Open Source AI Agent & Runtime (2026)

Compare

vs ChatGPT (OpenAI)

MCP Server Security Best Practices: Hardening Model Context Protocol Agents in 2026

Read Guide

Open Source

Open Source Coding Agent Harness: Build a Forkable, Local AI Engineering Runtime