The Analytics Theater - in three acts

I am currently reading The Frequency Era by Chris Walker (who is quite well known in the GTM measurement space for his blunt takes about attribution which I enjoyed very much). Since I have not finished it, I will not go into my takeaways. But I want to pick up his buildup to explain the frequency era.

Chris focuses on the scarce resource that defined each era and which replacement started the next era. Agriculture has scarce land, Industrial has scarce capital, Service era has scarce information.

Not spoiler something - I won’t tell what the new scarce source when AI made information and experience not scarce anymore. It’s not relevant for this post. For this post the scarcity is the guiding lens I want to apply on analytics.

I want to look into three acts to understand the evolution of marketing and product analytics. My focus on these is due to my extended experience in these areas. But I think you can easily apply them to the complete field of analytics. So when I speak of analytics in the following I mean foremost marketing and product analytics.

Act 1 - the Rise of Analytics with Craftsmanship

I was there when it happened. Not with the first steps, but quite after them. When I was working in product I introduced my first analytics setup in 2010.

And it was very welcome and fascinating to see. When you work on a web platform or application it is a black box by default. You can see if servers are running and potentially you get database entries where you can see if accounts have been created or purchases done. But you don’t want to ask a developer to query this for you and for some reason they don’t want to give you access to the database.

Early analytics was the answer for “normal” people to just see what is happening on the website. So analytics started to become an essential and we discovered more use cases that went beyond just making website traffic visible. We got conversion funnels and conversion rates, landing page performance and lastly: marketing campaign performance.

Analytics became a job. It even became important enough that agencies could be created for the sole purpose of it. They are still around.

What was required to make analytics a useful tool?

You really depended on the craftsmanship of the analyst who was responsible for the setup and the analysis. Yes, you also needed a tool. But the people working with the data needed a deep understanding of measurement (and the technical issues that come with it) and how you can analyze the data (what each metric means, how to interpret it correctly and finally understanding what can be derived for it for decisions).

Scaling insights that can support more decisions and impact was and is the real struggle since it technically relies on getting more experienced analysts.

The other struggle is that insights and answering questions has natural levels. You will answer first level questions faster and with fewer resources but each easier question is usually followed by 2-3 harder ones.

Like what is our checkout conversion rate → is the CR different for new and returning customers → is the CR different for new, returning and VIP customers and what attribution model should we use for this.

Higher levels → more experienced analysts needed (with skill sets that go beyond simple analysis: statistics, storytelling, project management and scoping, stakeholder management).

The whole thing led to a whole industry: analytics teams (junior, senior, principals), sub skills: CRO, tag management setups, specialized agencies.

A lot of it built on the promise that with each higher level unlocked the business impact of the insights will get higher so they will finally create a recognizable ROI. The big promise of data-driven (well).

The sad part of the story is that for most setups this moment never came to pass. So it’s an ongoing bet that will be more and more unlikely to pay off.

So the scarce resources are analysts and the more experienced you need them, the scarcer they get. That also means that your analysis options are very limited. If you have 100 questions floating around, you might tackle 10 of them.

This leaves us with an act (or era) that lived off the promises of analytics impact, got it delivered for some companies and depended on the experience and craftsmanship of the analysts and measurement engineers.

This act is coming to an end. Like all things not immediately and everywhere, but gradually it will spread.

Why? Because the scarcity is going away.

Act 2 - Industrialized insights

When we believe LinkedIn updates, conference talks, the analytics agents are here to take the helm. We get into the levels realities in a second. But let’s first sketch out, if this all works, what does it mean for you.

In a gist it means you have one of these scarce resources sitting in your computer just one click and question away. You have a meeting and discuss the upcoming campaigns for fall 2026 and the question comes up if you should create specific landing pages for the different channels or if a catch-all is fine. Someone remembers that one year ago you did a test with these specific landing pages before. In the past, this would now require an analyst to dig out the old numbers, double-check them. Now it would require someone in the meeting to fire up Claude or ChatGPT and just ask it - when you feel happy you can just activate voice mode and chat with it.

It will return some insights about the old campaign and give the team to ask follow-up questions to finally come to the decisions that 2 special pages are great for two channels but the rest can get a generic one. Everyone feels confident because of an educated decision they just made.

Okay, while you look at this either with a dreamy face, a scared one or most likely with the mindset, well, sounds like a sales pitch but in reality…

Let’s look at this reality a bit more to get some better foundation.

Analytics questions have different levels of complexity. So based on that, you need different levels of agents to answer them.

Agent level 1 - MCP with your analytics app. It can answer your basic questions. Most likely the one we just framed above, when you give it enough context (how the landing pages were named). They are basically a junior with some experience working on your analytics setup. So they inherit everything this kind of person would inherit. A data setup that most likely has some data quality flaws. The ones experienced analysts would suspect and spot, but a junior not. So your agent will enthusiastically tell you about a drop it recognized in a funnel, while you as an experienced analyst think, well did you also check 1,2,3 and 4 before coming to this conclusion? Again works great on basic stuff, but needs an experienced person working with it.

Agent level 2 - MCP on top of your semantic layer. This is quite a jump of setup here. In 1 you just buy your access off the shelf and instrument it. For level 2 you need a proper data set up (pipelines, data model and metrics definitions in a semantic layer). But you gain something that was missing in level 1. Solid and reduced metric and dimensional datasets that are tested and approved by the data team. Ideally even pre-joined in entity-based metric tables (one big table). In this setup the agent can give good and solid answers that could be asked by not so experienced people. This level is already possible today but it comes with quite some work and won’t work out of the box.

Agent level 3 - Semantic MCP with context. Level 2 still leaves all data insights just stand alone without any context. Context became quite a hype topic in the AI space and you will hear it all the time (buzz, buzz).

To make context a bit more tangible: One of my big frustration of my consulting life was, when I have a setup ready and produce insights based on foundational work, I usually get shrugs and comments like, “yeah, this is due to 1,2,3” and so on. My missing piece is context. The business context, the organizational history, the small decisions not written down, the insights that are common knowledge but not codified anywhere.

For an outsider it is almost impossible to gain that level of context (there are some tricks I learned over the years, but I refrain from analysis nowadays in these projects). Any agent has the same situation and even worse, it forgets everything quite quickly (context window). So a really good setup will spend a lot of time on fine-tuning the context layer (potentially even more). But the gains are significant. With the right context in place the agent can deliver insights that are on a good analyst level.

So, our takeaway here (and you just have to trust me if you have not spent time on working on level 3), we can make good insights possible in an instant. Or to pick up our initial examples, we industrialize them. What was a handcrafted insight before that took time and experience, is now one question input submit away.

Cool and scary, right?

So, what stays scarce then? For me this is impact. Let’s go to act 3.

Act 3 - Impact becomes the scarce resource

In this act we venture even more into uncertainty. While I am quite sure about the things laid out in act 2, everything here remains a hypothesis. But it has some foundation that is based on my experience.

With insights industrialized, picture this:

You are in charge of the marketing team of a midsize software company. While just 4 years ago, you were annoyed that the data team took 4-5 weeks to get you a campaign deep dive, you now feel like sitting at your desk and team members come into your room every 5m dropping a new analysis on your desk. Sure, you are using AI now to condense these reports but they still remain a constant flow of information. Remember the time when you thought that raw analytics event data is just noise, you are experiencing a whole new level of noise.

So, did marketing get better? Not really. Growth targets are still not met (and no, the management was reasonable about these, no weird AI blown up bets), and the central marketing question on what to spend your budget in the next quarter feels uninformed as before. Even when the information about it is massive. But you feel dreadful.

Well, a dashboard, a deep dive analysis, a decision preparation doc was never driving impact in a company. People are doing it (sure, some people dream of people-less companies). We make plenty of decisions every day. Very few of them we do by looking at data. Or do you check a dashboard before you decide which task you start with today.

And when we break it down to a very simple equation: A successful company is significantly better in making good decisions out of the plenty decisions they take. They are also particularly good in deciding which decisions are more important than others.

When analytics becomes a firehose of insights, the skill of taking a step back and having clarity about strategy and execution of it becomes essential. Only with a direction, the amount of insights can be enriched with strategic context. The kind of context that is the hardest to develop.

While we won’t go into a lesson into strategy here (there are plenty of books to read for you). Let me describe situations that are derived from my work.

You can drown yourself in granular analysis of channel, campaign, landing page performance, how your website funnel looks. Or you focus on the 30% growth target you defined before the beginning of the year. And break it down, step by step in sub goals until you reach a level that is operational and can be monitored by data. This can look like this:

To understand the thinking better, let’s expand that.

30% growth is derived from the business growth goals. Knowing the pipeline from lead to contract with conversion rates and conversion times, you can calculate an assumed growth that you need to achieve with marketing at the top of the funnel. Obviously, the 30% is a hypothesis, but it comes from a much stronger data foundation and can be surfaced by an analytics agent since it is based on consolidated metrics.

Now we are making strategic bets on the next level, but they are not completely made up from thin air. They are a mix of some analytics data, team experience but very important from a clear understanding of potential impact and team resources.

The three bets are in three tiers: straightforward, some experience, new test.

Increase paid search budget is based on existing data and experience. And the marketing operations are easy. It is mostly a coordination topic. Additionally measurement of these campaigns needs to be in place and an operational attribution model. Then the analysis needs to be prepared. Mostly the guardrails and how you determine incremental ROAS. And then it is just execution. But this approach has some nice strategic characteristic. Operationally it does not need a lot of resources and you have a setup to support it. Yes, it needs more budget and it is where 80% of the additional budget will go. And you can use an agentic analyst for consistent monitoring.

Adding 10 more influencers is based on experiments you did in the last 9 months with 3 influencers. You learned a lot around the processes that need to be in place. You have experience with measurement. This is now scaling up. The bet is the impact of it, since it will not be a straight linear projection. Measurement and analytics here is mostly for monitoring. But potentially also to support backlog to shortlist qualification.

The third one is a new test and it might help us with the 30% but it is nice to have in our prediction. You did two events last year and there were some signals that could point into a good direction. But the sample size is far too small. So this will now be tested on a bigger scale and will need most operative resources and focus (while having the lowest budget). This also means we need to develop a measurement strategy that can bring some experiences from other measurements, but will need some new thinking. So agentic analytics can help us to identify blind spots and then brainstorm about the measurement possibilities.

Why does this make a difference? We are narrowing down the playing field significantly and we put directions on it as well. The definition of the playing field is shaped by the strategic direction and principles that you use in your setup (early email capture, programmatic landing pages). But also the clear job that analytics plays within is narrowing down the scope.

With this specific scope you can context your agent much more efficiently and also align the feedback and insights they will provide that fits perfectly into your hypothesis testing setup. So it works seamlessly and there is no need of reading long reports and distill them in it. And yes, there is room for automation.

I hope the example shows my thinking right now what I see for act 3, which is also pointing to skills that are useful to develop.

Impact comes from people making better decisions in a clearer strategic context. And yes, analytics plays a part in it.

Before asking AI to analyze performance, you need to define the decision architecture: the target, the bets, the constraints, the metrics, and the context.

Goal trees are my favourite way to approach this but there are plenty of more tools that can help with it. But with analytics insights coming in a firehose, we need to become really good in being clear what we really need and where impact can be achieved.