The demo is always fast.
You watch the founder's recording: they type a question, and the answer appears almost instantly. The audience is impressed. You are impressed. You go home and build the same thing.
Your version takes four seconds.
You ship anyway. The user types a question. They wait. They wait a little more. They start wondering if something is broken. At second three, they click elsewhere.
This is the latency problem: there is a consistent and significant gap between how fast AI feels in demos and how fast it feels to real users doing real things. That gap is not a bug. It is not a configuration error. It is a structural property of how language models work—and most developers don't account for it until users start complaining.
Why Demos Lie
A demo is almost always the best-case scenario.
The model is already warm—the provider's infrastructure has been warmed up by recent requests, so there's no cold start. The prompt is short and well-crafted, probably after several iterations. The response is short, because the presenter picked a question with a concise answer. The demo runs on a fast connection with no queue.
Production is none of these things. Users ask rambling questions with unclear intent. The response requires 800 tokens to answer correctly. Your serverless function cold-starts on the first request of the day. The provider is under load at 9 AM EST when your East Coast users arrive.
The components of AI latency compound:
- Cold start: Serverless functions and container spin-ups add 500ms–2000ms before your code even runs
- Network RTT: A round trip to a US-East provider from a user in Germany is 80–120ms before the model sees a token
- Time to first token (TTFT): The delay between the API call and when the first output token arrives—typically 300ms–1500ms depending on model size and load
- Generation speed: Tokens arrive at 30–80 tokens/second for most models; a 400-token response takes 5–13 seconds to fully stream
- Tool calls: Each tool invocation is a synchronous round trip—an agent that calls three tools before responding adds 1–3 seconds minimum
- Chaining: Multi-step pipelines multiply every delay—if step one takes 1.5s and step two takes 1.5s, users wait 3s minimum, plus hand-off overhead
A demo with a 0.8-second response becomes a 4-second production response before you've changed a single line of code.
The Perception Gap
Latency is not experienced linearly.
Human perception research (much of it from web performance, now directly applicable to AI interfaces) gives us useful thresholds:
- Under 100ms: Feels instant. No loading state required.
- 100ms–1000ms: Noticeable but tolerable. Users don't interrupt.
- 1000ms–3000ms: Users feel the wait. They start wondering if something is broken.
- Over 3000ms: Users lose focus. Attention wanders. Task abandonment rises significantly.
- Over 10 seconds: Most users will attempt to reload or navigate away.
For AI features, the relevant window is that 1–3 second zone. A response that arrives in 1.2 seconds feels responsive. The same response at 2.8 seconds feels sluggish. At 4 seconds, users start clicking away.
The model output quality is identical. The latency alone changes the experience.
What makes this harder is that users calibrate their expectations to the fastest experience they've had with a similar product. If a competitor's feature answers in 1.5 seconds, your 3.5-second response feels broken even if both are technically "fast" by absolute standards.
The Tool Call Trap
Single-turn responses are hard enough. Agentic workflows are worse.
An agent that browses the web, queries a database, and then synthesizes a response isn't making one API call—it's making four or five. Each one is synchronous by default. Each one has its own TTFT, generation time, and network overhead.
The latency compounds aggressively:
This is for a reasonably fast agent. A slower model or a complex tool stack can push this to 10–15 seconds without any bugs or errors.
Users don't experience this as "the agent is thinking." They experience it as "this doesn't work." Especially if there's no visible progress indicator.
The compounding problem gets worse in multi-agent systems. If agent A calls agent B, which calls agent C, every hop adds its own latency budget. A three-hop pipeline with 2-second average per hop produces a 6-second minimum response time before you've added any real computation.
Streaming Is Not Optional
The most important latency intervention is also the most widely skipped: streaming.
Without streaming, the user sees nothing until the entire response is complete. With streaming, the first token appears almost immediately, and the response fills in progressively.
The actual total latency is identical. But the _perceived_ latency drops dramatically. Users who would abandon a 3-second blank wait will happily watch a streaming response for 5 seconds—because they can see that something is happening.
This is not a trick. It is a genuine improvement in user experience, and it costs almost nothing to implement. Every major AI provider supports streaming. Every modern frontend framework can render streamed content.
If your AI feature is not streaming, add streaming before you do anything else.
The second-order insight: streaming also forces you to think about response structure differently. A streamed response should front-load the most important information—the answer, not the preamble. "I'll help you with that—here's the information you need" front-loads the useful part. "Certainly! I'd be happy to assist you with your question about..." front-loads noise. Users reading a stream read the beginning first.
Async First
Not every AI feature needs a synchronous response.
Ask yourself: does the user need this answer in the next 3 seconds, or in the next 3 minutes?
For many AI use cases, the honest answer is "3 minutes is fine." The user asked the agent to research a topic, draft a report, analyze a dataset, or process a batch of records. None of these need to block the interface while they run.
Async processing with a notification pattern (email, push notification, in-app alert) consistently produces better user experience for tasks over 5 seconds. The user submits the task, goes back to what they were doing, and receives a notification when the work is done. No watching a spinner. No wondering if it crashed.
The critical design pattern: give users something to do while waiting. If the task produces a result they'll want to act on, let them set up that action in advance. "While your report generates, review these suggested sections" is useful. "Processing..." is not.
Most developers default to synchronous because it's easier to build. But synchronous AI features are often worse products than async ones, not better.
Latency-Aware Architecture
Once you've shipped streaming and async patterns, the architectural choices matter.
Model selection: Smaller models are faster. A 70B parameter model may be twice as capable as a 7B model, but it's also 5–10x slower per token. If 80% of your requests don't need that extra capability, you're paying the latency cost for capability you're not using. Route simpler requests to faster, cheaper models. Use the heavy model only when you need it.
Geographic proximity: Latency to the provider matters. If your users are in Europe and your provider's primary endpoint is US-East, add a round trip. Use regional endpoints, deploy your middleware closer to your users, or cache aggressively.
Prompt caching: Long system prompts are expensive to process on every request. Most providers offer prompt caching for repeated system prompts—the first request pays the full cost; subsequent requests reuse the cached computation. For a 2000-token system prompt at 100 requests/hour, caching converts most of those to milliseconds.
Parallel tool calls: Most agents execute tools sequentially by default. If tool call 2 doesn't depend on the output of tool call 1, run them in parallel. Three independent tool calls that each take 1 second become 1 second of latency instead of 3.
Pre-computation: For queries with predictable patterns, compute likely responses before the user asks. A dashboard that loads when the user logs in can begin fetching AI-powered insights before the user clicks "Analyze." The model starts thinking before the user's question is formally submitted.
The Waterfall Audit
If you're debugging latency in a live product, the first step is measurement. You cannot optimize what you cannot see.
Instrument your AI pipeline with spans:
- Time from request received to API call initiated
- Time to first token (TTFT)
- Total generation time
- Time for each tool call
- Total end-to-end latency
Most APM tools support this. If you're using LangSmith, LangFuse, or Helicone, this instrumentation is partially built in.
Once you have data, look for the waterfall. AI latency problems almost always follow a pattern: one step that takes 80% of the total time, hidden inside a chain that looks balanced. The step might be a slow tool call, an unnecessarily large prompt, a model that's oversized for the task, or a dependency on an external API you don't control.
Fix the longest pole first. Everything else is noise until you've addressed the primary bottleneck.
What You Can't Fix
It's worth being honest about the irreducible floor.
Language models have minimum latency. You cannot get below the time it takes to transmit your prompt to the provider, process it through the model, and return the first token. For the largest, most capable models, this floor is 300–800ms under ideal conditions. For models with long input prompts, it's higher.
For applications where sub-100ms response time matters (autocomplete, real-time voice, live translation), current generation language models are architecturally wrong. You need either a small, specialized model running locally or a non-LLM approach.
For most AI features—chat interfaces, agents, document processing, content generation—the floor is workable. 300ms TTFT is acceptable if you stream from there. The problem is usually not the floor; it's the layers stacked on top of it.
The Expectation Gap Is a Design Problem
Ultimately, the latency problem is a design problem as much as an engineering one.
If you show users a demo where responses appear instantly, then ship a product where responses take 4 seconds, you've created an expectation you can't meet. Users experienced the fast version. They now believe the slow version is broken.
This is fixable:
- Set expectations explicitly: "Analysis takes 20–30 seconds." Users who know the duration tolerate it.
- Use skeleton screens and progressive disclosure: show the structure of the response before the content.
- Give interim status: "Searching documentation... found 12 relevant sections... generating summary..." is far less frustrating than a blank spinner.
- Calibrate your demos to production conditions, not ideal conditions.
The best AI products treat latency as a first-class design consideration from day one—not a performance ticket that gets filed after users complain.
The demo was always lying to you a little. The response was fast because everything was lined up in its favor. Production will not be so cooperative.
Plan for the slowdown. Stream. Go async where you can. Audit your waterfalls. Set honest expectations.
Your users will wait—for a second or two. Make it worth their time.
export const social = {
title: 'The Latency Problem',
description:
"AI demos feel instant. AI products feel slow. The gap isn't a bug—it's architecture. Here's how to close it.",
hashtags: ['AI', 'Performance', 'AgentEngineering', 'DeveloperExperience'],
hook: "Your AI demo runs in 0.8 seconds. Your production feature runs in 4. Here's why—and what to do about it.",
};