Agentic Analytics Meetup Paris: Shark Tank edition 🦈

Last week, 5 data teams pitched their analytics agent in 5 minutes each. Then 4 sharks grilled them on reliability, adoption, maintenance and costs.

It was the first Shark Tank edition of the Agentic Analytics meetup: 60 people at Qare's office in Paris, plus a livestream. 5 teams, 5 completely different setups: an in-house agent on Snowflake, Databricks Genie, an in-house agent on top of Cube, Snowflake Cortex, and nao.

Here's what each team built, and what the sharks wanted to know.

Watch the full recording →

About nao

This meetup is organized by nao Labs, the team behind nao: an open-source analytics agent that lets every employee query your data conversationally. nao is built around a context layer that your data team builds and version-controls like a dbt repo, and a chat UI you deploy to your business users. The first edition of the meetup was in Paris in April.

The format and the sharks

Each pitcher had 5 minutes to present their agent: the technical setup, business adoption, and the tests and guardrails they put in place. Then 5 minutes of questions from the jury.

The sharks, and what each of them said they would challenge the pitchers on:

• Yochan Khoi, Staff Context Engineer at Gorgias: the rollout strategy. Having the tool is one thing, getting a whole company to use it is another.
• Christophe Blefari, co-founder & CTO at nao: things he has never seen before, and how teams handle maintenance.
• Florian Drouet, Lead Data Engineer at Qare: data security. Qare is a healthcare company, and health data leaves no room for approximation.
• Juliette Duizabo, Head of Data at Photoroom: reliability. If you ask the same question twice, does the agent fetch the data from the same place, the same way?

Spoiler: every pitch got challenged on reliability.

The pitches

Lirav Duvshani & Alix M., GitGuardian

Stack: Snowflake, in-house multi-agent system exposed as an MCP, used from Claude (Jump to 17:52 in the recording)

GitGuardian is a cybersecurity company of almost 200 people. Lirav is Engineering Manager Data & AI in a team of 4, and Alix, an AI engineer, built the agent.

They had opened everything in Snowflake (bronze, silver and gold) to everyone, then plugged a Snowflake MCP into Claude. Answers were sometimes right, sometimes wrong. Two things were missing: access to the code, to be sure of the business rules implemented, and knowledge scattered across Notion and Slack.

Since every employee, sales included, already had Claude, an MCP was the obvious interface. They benchmarked the existing options first. Adding table descriptions helped, but data engineers with access to the table definition code got even better results. So they built their own:

• An orchestrator picks the right business domain and splits the question if needed.
• Sub-agents per domain get the list of their domain's tables and their own skills. They can search the codebase for table descriptions, lineage and a glossary of business rules ("this column is called cost, but the real cost is this other one").
• Query results get an ID. Instead of each agent rewriting the results (and silently dropping data it judged useless), the full table is injected into Claude at the very end.
• A feedback tool in the MCP lets users report a wrong rule. A scheduled task reviews the feedback every morning and suggests additions to the agent's knowledge.

Building the MCP took about 3 months. The hard part was not technical. It was sitting in meetings where people explained their processes, and convincing them to write it all down. As Lirav put it: if everyone wants a stable answer, you have to make the implicit explicit.

The sharks asked:

• Same question three times, same answer? That was the whole point. Before, three people would ask Claude the same question and show up in the same meeting with three different numbers. RevOps knew which filters to apply, sales didn't. Encoding those business rules is what made answers stable.
• How do you monitor the MCP? Everything is logged in Postgres and analyzed after the fact: follow-up questions that signal an unsatisfying answer, runs where the agent got lost, wrong tables picked.
• What happens when sales change their process next week? Not tested yet. The bet is on the feedback tool: the more people use the agent, the more they want the answers to be right.

Wael Gara, EcoVadis

Stack: Databricks Genie, Claude with the Databricks MCP (Jump to 33:44 in the recording)

Wael is not a data person. He works in RevOps at EcoVadis, a sustainability ratings company, and built everything by himself after attending our first meetup. His goal: answer in one hour (or five seconds) questions that used to take him an afternoon, or two weeks.

He started with Databricks Genie spaces: minimal instructions, example queries written by hand ("I have the keys to the castle because I know which filters to use"), and three months of iteration. Then he got the Databricks MCP and Claude. Instead of building each Genie by hand, he wrote down how he builds them and asked Claude to do it. His first Genie had 27 examples. The one Claude built had 84, and Claude could ask each question three times to check the answer was stable.

• One theme at a time: one agent, one dashboard, one Genie. Leadership wants to see numbers first, then ask a question when they want a deep dive.
• 4 dashboards and 4 apps in production, for questions that range from "why are clients churning?" to rep performance.
• His audience isn't always human: once a month, Claude queries his Genies to check that no data model or integration silently broke.

The sharks asked:

• Do people know which agent to use? They only see one Genie for work, and the others come attached to a dashboard. Yochan's feedback from Gorgias: the more entry points you have, the more people forget they exist. Abstract them behind one.
• Is there a feedback loop? Once a week, Wael shows what he is building to someone from sales or the frontline. "If they don't have the keywords, it's useless, even if it's technically great." And he tells them: this is not the Bible, if you spot something wrong, tell me.
• What does the data team think? They love it, because there is a lot of governance behind it: PRs, certified tables, security reviews. "The governance team loves me because I ask before I build."

Anton Heikinheimo, Hostaway

Stack: SQLMesh, Omni, Cube, an in-house agent behind an MCP (Jump to 45:17 in the recording)

Hostaway has used Omni for about a year. Every time something broke, they added context. Anton shared the problem with that approach in numbers:

• data models grow ~10% month over month
• usage grows ~30% month over month
• context and token spend grow ~60% month over month

"Every time something breaks, you can't just add more context." His example: someone asks for retention, gets net retention, actually wanted gross retention. So you add a rule: retention means gross. Except for the board, where it means net.

The next version is built like software: every correction becomes a test. And not only the wrong answers: every answer someone verified as right becomes a test too. Any change (a dbt model, a Fivetran or dlt source, a semantic layer definition) has to pass those tests before it ships, like a blue-green deployment. To make that affordable:

• Only run the tests downstream of a change. Change the GRR definition, run the tests that touch GRR.
• Use real questions, not generated ones. The first real question in their BI tool was "what's the difference between GRR and NRR?", something no generated test set would contain.
• Test the mapping, not the data. With a semantic layer, you only need to check that the question maps the right parameters into the right slots. No warehouse query needed.
• No reasoning in the agent, just retrieval. A small model maps the question to Cube definitions, fast and cheap. The reasoning happens upstream, in Claude or whatever AI the team uses.

The sharks asked:

• How do you maintain lineage between the semantic layer, the markdown and the tests? At the table level, not the column level. If a test references a table, it runs when that table changes.
• How do you expose all this to Claude? Only through an MCP. "The MCP is the interface."
• Can the agent ask a clarifying question when a term is ambiguous? Yes, and you can write a test where the expected outcome is a clarification. It is still flaky: knowing when to clarify and when to answer is hard. Defining "contested terms" explicitly is the end state.

Opeyemi Fabiyi, Data Culture

Stack: Snowflake semantic views, Cortex Agent, Snowflake MCP in Claude (Jump to 59:00 in the recording)

Opeyemi works at Data Culture, a data consultancy, and presented a deployment for a client where one analyst served the entire organization. Their method:

1. Start as narrow as possible: one high-value domain.
2. Write the test suite with the stakeholders of that domain: easy, medium, advanced and complex questions that mirror the analyses they actually care about.
3. Build an evaluation pipeline in two steps: deterministic checks (exact or approximate match against golden queries) catch ~80% of the cases, and an LLM-as-a-judge evaluates the remaining 20% on groundedness, completeness and reasoning.
4. Measure what each piece of context brings. On raw tables, the agent couldn't even generate SQL for most questions. Column descriptions helped a bit. The big jump came from business rules: the nuanced, internal knowledge native to the organization.
5. Don't deploy below 90%. If the agent doesn't pass 90% of the test suite, it doesn't ship. Above that, it goes to a test group first, then to everyone, with monitoring in production.

They tested with Snowflake Intelligence, then moved to Claude through the Snowflake MCP as the agent reached more business users. Next step: from "chat with your data" to workflows. A domain expert explains how they evaluate a product launch, that rubric goes into the agent's context, and an analysis that took half a day now takes under 10 minutes.

The results: 3 domains in production, 4 users at launch in April to 15 in July, and 4,334 questions between April and October. Every new domain triggers requests for the next one.

The sharks asked:

• How long to add a new domain? About two months for the first one. Now that the pipeline exists, under 4 weeks.
• How do you maintain the semantic views? An LLM-as-a-judge flags weak answers in production, and once a week an engineer reviews them and updates the semantic view.
• What if a question is in the domain but the data isn't there? The agent says it's out of scope rather than guessing. The client then asks for the data to be added.
• Why not just a dashboard? Because the agent lets people ask the follow-up question a dashboard can't answer.

Benjamin Rouif, Getaround

Stack: Snowflake, dbt, semantic views via dbt, nao on DeepSeek (Jump to 1:15:15 in the recording)

Benjamin is Head of Data Analytics at Getaround, and the only pitcher of the night running nao. His slide said it all: your agent works until you start talking to the people who use it.

They moved to nao to cut costs: today they spend around $0.20 in tokens per conversation, running on DeepSeek. With a strong context, a simpler model performs very well. About 30 people use it, a quarter of the company, with 300 questions over the past 4 weeks. In their survey, more than half of the users are promoters, and there are no detractors.

Their rule: it's OK for an agent to be wrong, as long as it says it might be. Agents tend to be people pleasers that hide things under the carpet. Four pillars to prevent that:

• A mature stack: 6-7 years of dbt and Snowflake, tested and running.
• Context as code: every definition and rule is versioned in Git, with golden queries to verify what goes through the agent.
• Opinionated behavior rules: every answer must state a confidence level and list every assumption made. "We don't care if it's long, we don't care if it's hard to read."
• A hierarchy of trust: the agent sees almost the whole stack, but always starts at the semantic layer. If the answer isn't there, it goes one level down, then another. Each level down means lower data quality, more assumptions, and lower confidence.

In practice: "how many rentals last month in France?" resolves in the semantic layer in about 10 seconds, with two simple assumptions (France is stored as FR, what "last month" means) and high confidence. "How many second rentals last month?" triggers a clarifying question, then has to go one level down where there are four ways to count previous rentals. Low confidence: double-check before you show it to the board.

Then came the learning. Stakeholders were happy, but a few odd comments showed up, so the team sat down with them. Users trusted the agent too much. They stopped sanity-checking numbers against the dashboards, and when they did, they didn't check the assumptions. Now Getaround is training stakeholders on how to prompt and how to review a conversation.

The sharks asked:

• How did you roll it out? They started by exposing only the semantic layer to business users and the C-level. Without room for follow-up questions, engagement stayed low. So they opened more data, to more teams: business, then product, then engineering. Ops and sales come later, with specific skills.
• Would you hide low-confidence answers from less data-savvy users? Not for now. Knowing the confidence is low is valuable, and censoring the agent isn't the answer.
• Who were the first users? The most data-savvy people: they asked for it, they have orders of magnitude in mind, and they spot errors. Juliette confirmed Photoroom picked the same strategy.
• Why DeepSeek? It performed best in their benchmark at the time. Switching model or lab means re-testing a context that may be tuned to it, and right now the focus is the stakeholders, not the model.

What we took away

Five teams, five stacks, no consensus on tools. But the conclusions were the same:

• 🧪 Start with tests. Evaluate your agent on a test suite before rollout, and turn every wrong (and right) answer into a future test. Opeyemi won't ship below 90%, and Anton's whole architecture is built around it. If you use nao, that's what nao test is for.
• 📈 Context is what moves the needle, especially business knowledge. At Data Culture, the big jump came from business rules, not column descriptions. At GitGuardian, the long part was getting people to explain their processes. It matches what we measured in our context benchmark. More on the method in our context engineering guide.
• 💸 LLM costs can't be ignored anymore. Anton's token spend grows twice as fast as usage. Getaround runs at $0.20 per conversation on DeepSeek. Throwing the biggest model at your data agent doesn't scale: context work combined with model evaluation does.
• 🗣️ Talk to your users. Wael shows his work to the frontline every week. Benjamin discovered his users trusted the agent too much. A working agent is the start of the job, not the end.

For more on making agents reliable, see our 5 steps to improve analytics agent reliability and how to model your data for AI agents.

Thanks and next events

Thanks to our pitchers Lirav, Alix, Wael, Anton, Opeyemi and Benjamin, to our sharks Juliette, Yochan, Florian and Christophe, and a special thanks to Florian and Qare for hosting us in their office.

Want to pitch your agent at the next edition, or host one in your city? Reach out.

In the meantime, star the nao repo on GitHub: we're 13 stars away from 2,000.