How to Build an Experimentation Engine

A difficult market has created the set of conditions for fast-moving brands to win

Growth has never been harder than it is right now. Today’s growth leaders are battling rising ad acquisition costs, murky attribution, a shaky economy and tentative consumers.  It’s seriously tough out there. Gone are the days of growing through one to two channels and basic execution.

While the situation sounds dire, for savvy brands with big ambitions this is a good thing. As the bar to compete continues to increase, the brands that experiment the most and learn the fastest will win. When things are hard, the value of good decision making increases.

Executed correctly, experimentation becomes a repeatable process for learning what works, scaling those learnings and compounding good decision making. This is hard to do well. That’s good. This friction is a moat you can leverage to beat your competition.

Here is a playbook you can follow to get started:

Lay the strategic foundations

Experimentation is only useful if it drives commercial outcomes. Run well, it should operate as a revenue generating machine where with each experiment, your organisation gets smarter. For this to be true, you need to know what the drivers of growth are in your business so that you can tie experiments to commercial results.

Map the Drivers of Growth in your Business

Solving for: “I know what drives growth in my business”

Below is an example of what a driver tree might look like for a mid-market ecom brand. The levers for paid advertising are broken out to provide an illustration of how this expands. There will be equivalent branches for all the other traffic sources.

The shape of the tree will vary from brand to brand depending on your business model, size and north star metrics. The important part is to understand what your shape looks like.

Aggregate Data Sources

Solving for: “I know how each branch of my tree is performing”

You now need to make sure you have the right data sources to see how prospective customers flow through your driver tree. This requires fetching, cleaning and normalising data into a queryable format. This is time consuming, technical and therefore generally expensive. Ecom brands can often get pretty far with just GA4, Shopify and Ad Platforms but it’s much harder for lead gen and app based companies.

The key unlock here is being able to drill down into each branch in order to find interesting insights. 

Ensure Data is High Quality

Solving for: “I trust the information in my model”

This is the perfect time to make sure your conversion events are set up correctly, your UTM tagging nomenclature properly applied and your conversion windows consistent. This is all hygiene stuff but it’s important. If you don’t trust your data, you won’t trust any of the experiment results. 

Steps 2 & 3 should happen in conjunction.

Turn on the engine

You’ve now got an incredible base to experiment off. You know the shape of growth in your company, the inputs that drive it, and can quickly see how each one is performing. You’re ready to start. 

Below is a simple methodology that we use at Fourteen10. There is nothing proprietary here, this is just a slight modification of the scientific method. 

The important part is to embed experimentation as a habit and just get moving. In our experience, this requires:

A Structured Process

Solving for: “I know exactly how we operationalise experimentation”

It’s impossible to embed the experimentation habit without great process. This typically includes:

• Clear meeting cadences to review hypotheses, results and update your driver tree

• A squad of the right people who can all offer a different POV

• A clear prioritisation framework for placing bets

• A method for sharing results across the organisation

Commercial Focus

Solving for: “Our experiments are tied to revenue outcomes”

Hypotheses should be linked to your driver tree and where possible the commercial impact should be quantified. 

Lets say your hypothesis is:
 “If we add a first party data seed audience of high value customers to our prospecting meta campaign then the conversion rate of the campaign will increase by 2% because meta’s algorithm will be better able to target relevant customers.”
You will be able to estimate what a 2% increase in conversion rate would do to revenue based on recent results and use that as a guide to prioritise.

Analytical Firepower

Solving for: “I trust the results from my experiment”

This works on a continuum depending on your level of maturity. Initially simple A/B tests are fine and there are some great free calculators out there to help with calculating stat sig and sample sizes. More advanced companies will leverage AI/ML to run more sophisticated tests that can produce insights faster. 

Bias to Action

Solving for: “a failed experiment is better than no experiment” 

To build the experimentation habit, you need to prioritise momentum. While running lots of bad experiments is a waste of time, good teams will set target numbers of experiments to run each month and get competitive with themselves, looking to increase these volumes over time.

This is a form of marginal gains. If you run five experiments a month and each one makes you 1% smarter, while your main competitor runs one each month, in twelve months your growth team will be 60% smarter than theirs.

A culture of truth seeking

Solving for: “I care about the why, and want to get closer to the truth of what actually works”

The final, and arguably most important point, is creating a culture that prioritises seeking the truth. Too often experimentation can fall down because of conflicting stakeholders. Some people don’t want you to get under the hood. To succeed,  everyone needs to understand that ‘the why’ is just as important as ‘the what’. 

Take Advantage of this Moment

We believe an experimentation engine is the primary form of competitive advantage for most growth teams. If you’re not operating one currently, now is the time to do so.