The Agentic Analytics Playbook: How 7 Companies Built Reliable Analytics Agents

We spent the last few months studying how companies actually build analytics agents that work - not demos, not prototypes, but agents that hundreds or thousands of people use every day.

The result is the Agentic Analytics Playbook: 27 pages covering 7 companies, their architectures, their mistakes, and what actually moved the needle on reliability.

Why we wrote this

Every data team we talk to is exploring analytics agents. Most hit the same wall: the first prototype works great on 5 questions, then falls apart on real usage.

The problem is never the model. It's always the context.

But "do better context engineering" is vague advice. We wanted to show what it actually looks like in practice - across different company sizes, tech stacks, and data maturity levels.

What's in the playbook

A framework for thinking about analytics agents. Every production agent we studied has the same 5 components: a harness, a context layer, an interface, a feedback loop, and an evaluation system. We break down what each one does and how they connect.

7 real case studies. For each company, we extracted key metrics, architecture diagrams, and the 3 most important learnings:

• OpenAI built a bespoke agent with 6 context layers serving 3,500 internal users across 70,000 datasets. Their context reduced query time from 22 minutes to under 2 minutes.

• Anthropic went from 21% to 95% accuracy by investing in "procedural knowledge" - teaching their agent how an analyst would approach a problem, not just what the data means.

• Gorgias reached 84% company adoption (250 of 380 employees) without a single company-wide announcement. Their secret: domain experts own their context sections via GitHub PRs.

• Vercel improved success rate from 80% to 100% by removing 80% of their agent's tools. Less is more.

• Ramp scaled to 1,800+ analytical questions across 500+ beta members.

• Lyft reduced resolution time by 87% with a metric ownership model (2 owners per metric).

• nao improved reliability from 17% to 86% with pure context engineering - simple markdown files, no semantic layer.

Architecture comparisons. Side-by-side diagrams showing how each company structures its context, feedback loops, and evaluation.

3 findings that surprised us

1. Context engineering is mostly data engineering. Across every company, the biggest reliability gains came from fixing the data model - not from better prompts, fancier tools, or model upgrades. Adding missing computed fields, establishing source-of-truth tables, writing disambiguation rules. If your agent is wrong, your data documentation is probably the problem.

2. Semantic layers are optional, evaluation is not. Multiple companies achieved 85%+ reliability without a semantic layer. None achieved it without evaluation. Offline tests (deterministic SQL comparison) and online monitoring (feedback signals, failure tracking) are the non-negotiable.

3. Adoption is a context problem too. Gorgias didn't mandate usage - they let domain experts edit context via GitHub PRs, which created ownership and organic adoption. OpenAI's agent reduced analyst time from 22 minutes to 2 minutes, which made adoption effortless. The best agents don't need a rollout plan.

Get the playbook

The full playbook is free. Get it here.

If you're building an analytics agent and want a framework that works with your existing data stack, try nao - it's open source and gives you most of the scaffolding described in this playbook out of the box.