In our previous post within our ‘Feature Experimentation’ series, we listed some essential best practices when it comes to feature testing and experimentation.
We also mentioned the importance of having a roadmap in place to help guide your team so that they’re clear on the main business objectives so that they can build efficient tests.
This roadmap will essentially be the link between these objectives with a product manager’s ideas to be able to run experiments and track the right metrics.
As a product manager in a modern software development world, you know how critical it is to quickly deliver features or products to keep up with rapidly changing demands.
You also understand how important it is to continuously run experiments to make sure that your releases are ones your customers actually want.
Having a roadmap in place will enable you and your team to plan experiments that lead you to the best outcomes in line with your company’s overall strategy and objectives.
Therefore, in this post, we will guide you on how to build and create an experimentation roadmap so you can determine what you need to optimize and figure out which experiments you need to run.
A well-structured roadmap will help you and your team prioritize which actions to take and which experimentation ideas are viable to achieve your goals and objectives.
It outlines a clear strategy for your team to follow so everyone is on the same page working towards the same objectives.
Why do you need an experimentation roadmap?
In the previous section, we quickly went over the importance of having an experimentation roadmap in place to achieve your goals. Here we will go into more detail about why you need such a roadmap.
We are focusing specifically on an ‘experiment-driven’ roadmap, which is a roadmap that illustrates the experiments a product team is aiming to run in order to achieve certain outcomes in line with business objectives.
Such a roadmap helps to point out which features to add, change or remove depending on the pain point you’re trying to solve and the data generated from experiments and any additional data gathered from other sources such as through the discovery phase process or from interviews or product usage.
Put simply, it will act as a strategic plan of action that communicates to all teams how your short-term goals align with the organization’s long-term goals.
Before embarking on your experimentation journey, having a roadmap should be the first step for various reasons, including:
Better resource planning
Creating a clear plan or roadmap before starting your experiments will help you plan your resources in advance instead of finding out you’re running out of resources halfway through an experiment.
Furthermore, as you run more experiments, you’ll be able to better allocate resources to each of these experiments to obtain more accurate results.
It’s also very likely that as you go further into your experimentation journey, you might want to create more complex experiments and tests. When you have a roadmap in place, you will be able to plan in advance for such tests.
Thus, a roadmap will enable you to see the kind of resources, such as maximum budget, you have at your disposal to plan your experiments more efficiently.
Working towards a common objective
As already mentioned, an experimentation roadmap will clarify for your teams the kind of goals and objectives you’re aiming to achieve so that every team is aligned and experiments can be built around these goals.
For example, if you have a roadmap established, you and your team will know which tests you need to run first so you can start planning these tests and the kind of resources you’ll need, as per the point above. This will end up being a huge time-saver as you have a clear idea of which experiments would fulfil company objectives.
This is a more efficient way than running random tests which might not get you the results you’re looking for and that end up being a waste of time and resources.
Greater visibility and enhanced communication
Because everyone has a clear goal to organize their workflow around, this gives greater visibility to all teams within an organization on the testing processes that are being set up.
This, in turn, allows for better communication as everyone is on the same page on what needs to be done and the information they need to do it. In short, it helps to communicate and lay out to the teams involved what the plan is to achieve objectives.
This also helps establish a culture of experimentation within the organization as different teams share their insights and learnings from the experiments.
We’ve emphasised in our previous post in our Feature Experimentation series the importance of making experimentation a team effort and an experimentation roadmap facilitates this process.
By building this roadmap, you’re basically communicating to your teams that experimentation and optimization are taken seriously and are at the forefront of the business.
In short, from this section, we can conclude that an experimentation roadmap helps align teams, the company vision and its overall goals with the kind of experiments you run.
It helps put experimentation at the core of your business and puts it into perspective within the wider business objectives. In other words, it gives teams purpose and direction in their day-to-day work.
Keep your eye on the goal: Focus on the outcomes
In Agile product development, it is often the outcomes that matter foremost while putting users at the heart of the process to deliver concrete and measurable results.
What this means is that when drawing up your roadmap, you should focus on outcomes rather than just listing the features you’re looking to build within a certain timeframe and hence forcing you down a fixed path.
When you look at outcomes, your main focus is on understanding problems and finding innovative ideas to solve these problems through experimentation.
This is the essence of an experimentation or experiment-driven roadmap. When you run experiments, you are given the opportunity to explore various solutions by testing different features within your experiments and tracking their performance.
Consequently, your main focus will be on achieving outcomes rather than having a predefined list of features you think will be the solution to your pain points. Also referred to as a feature-driven roadmap, this roadmap instead focuses on a fixed list of features to be delivered over a given time period without taking into account the wider consumer or market context.
This means that the goal of an experimentation roadmap is to help you reach specific outcomes within various time frames, outcomes that fulfil both short- and long-term goals.
Thus, you start out with a problem or pain point you’re trying to solve and the experimentation roadmap gives your team insight and a clear strategy on how to solve that problem by testing out your ideas through different experiments till you get the outcome you’re looking for.
An example of a problem-focused roadmap can be seen in the image below, which includes a ‘discover and experiment’ phase to help tackle problems in each quarter:
This approach provides more autonomy for your teams and encourages them to share their own insights and knowledge which helps to instil a true culture of experimentation.
Focusing on the outcomes will enable product teams to be more flexible by making constant changes and reiterating to get to the outcome instead of committing in advance to features that they think will deliver the desired results.
Only once you’ve validated features through experiments should you proceed with releasing a feature that will actually deliver value.
The following image depicts a more simplified and straightforward version of this approach:
How to construct your experimentation roadmap: Step-by-step
In this section, we will discuss the steps to follow when creating your roadmap to help run efficient experiments and campaigns.
1. Outline business mission and vision
We stressed the importance of aligning short-term goals of your experiments with the long-term organizational goals.
For that, you will need to clearly outline the mission and vision in order to create quarterly objectives and key results.
Afterwards, it is imperative to establish your business goals, based on the vision and mission, as these will be the basis for all the future experiments you run (or don’t run).
2. Define goals
Once you have established the main business objectives, you will now need to consider why you are running these experiments in the first place. In other words, what the goals of these experiments are, i.e the why aspect of your experiments.
Your roadmap should contain specific, detailed descriptions of the kind of changes you want to see and define what success looks like for your company so you can set goals based on this information.
Your goals will usually be tied to your brand vision and overall business objectives. In other words, it will focus on the bigger picture so that it communicates clearly to all stakeholders the desired business outcomes.
Ironing out the details will then serve as the foundation of your experimentation framework and roadmap and act as a prerequisite for tactical planning.
Setting experiential goals will then help you formulate hypotheses as a starting point for any test you run. This is especially the case if you’re planning to run A/B tests, which usually start with a hypothesis that includes an assumption and the expected outcome.
3. Define experimentation KPIs
After you’ve defined your goals, you’ll need to determine the primary and secondary KPIs you need to track in order to gauge the success of your experiments.
These KPIs will depend on the kind of results you’re looking to get from these experiments so KPIs can be anything from access rate of the product page to the number of clicks on the CTA.
Other KPIs to consider depending on the goal of your experiment may include product usage, conversion rate and sign-up rate.
4. Outline the key features of your tests
An experimentation roadmap will help you identify key features in each of your tests or experiments including:
- What to test
- When to test
- Who to test on
- Who is running the test
These are all key features to include within your roadmap to help you keep track of your experiments and the people and teams involved in the experiment process.
This is especially important when you’re part of a large organization and different teams are involved in and running their own experiments. This also ensures any knowledge, insights and learnings from experiments are visible throughout the organization.
5. Prioritize tests
You will also need to determine which tests to run first depending on priority, usually by directing your attention to the ones that will bring you the highest return on investment (ROI).
For example, you may decide to run tests depending on your existing resources-you will need to consider how easy or complicated it is to put together a certain experiment depending on existing organizational resources- or you may choose to run experiments based on the results you’re seeking and plan accordingly.
However you choose to prioritize your experiments, try to prioritize them based on your desired outcomes so you don’t waste time or resources running tests on features that don’t give you the data or insights you need to optimize your products.
Remember that being part of an agile product team, it’s important to remain flexible. This means that the roadmap you built will not be set in stone. You should always leave room for changes as often unexpected findings may occur during experiments that may force you to shift gears.
There are many techniques for prioritizing roadmap items. One approach is referred to as the MoSCoW technique, which helps product teams to prioritize features. This approach is especially useful for Agile teams which tend to favor items that carry the highest value.
The below image depicts the requirements of this framework, where the top category ‘Must have’ are features that are absolutely critical to a project all the way to ‘Will not have’, which are features that are not worth the investment.
Experimentation roadmap: Your road to better products and/or features
What we can conclude from this post is that an experimentation roadmap serves as a shared source of truth that outlines the company vision, priorities and planned features and their progress.
Most importantly, a roadmap is a way to empower teams down a path of maximum impact by giving them the information they need to get started on their experiments.
We emphasized the idea of flexibility so it’s ok to introduce changes to your roadmap as you start your experiments.
This is the whole point of experimentation; you are conducting experiments without the certainty of knowing what the solution will be so it’s a learn-as-you-go process and then adjusting your roadmap accordingly.
The only thing that should remain a staple within your roadmap is how the goals of your experiments will achieve long-term business objectives.
Read more: Why you should slot feature flags into your Agile roadmap
