Testing in Production: Yes, Every Product Manager Should Do It

Arthur Charbit
January 22, 2021
7 min read
Flagship.ioRelease ManagementTesting in Production: Yes, Every Product Manager Should Do It

In a perfect world, you release a product that is bug-free and works exactly as it should and so there is no need for further testing.

However, both product managers and developers know that it’s not as simple as that. They need a way to make sure that there is a process in place that reveals any issues in code in a live production environment.

This is where testing in production comes in.

But it’s also one of the highly debated topics out there with those who say you should always test in production, and those who are more wary of the concept and say you never should.

In this article, we’ll look into these two different perspectives and share our own point of view on this controversial topic and we’ll guide you through the best ways to reap the benefits of testing in production.  

What is testing in production?

To keep it short and simple, testing in production is a software development practice of running different tests on your product when it’s in a live environment in real time. 

It allows you to test new code changes on live users rather than a staging environment.

Thus, it refers to the continuous testing of a feature or product in a production environment after deployment.

Testing in production is not meant to be a replacement for your QA team or eliminating a unit test or integration test. In other words, it is not supposed to replace testing before production but to complement these tests.

To do or not to do: That is the real question

These are big benefits, and they are enough to create consensus among many developers and product managers who say “Yes, always!” to the practice.

But there’s also another group of developers and product managers who say “No, never!” to testing in production. 

On the one hand, they admit all of the great benefits that testing in production can deliver. On the other hand, they also believe that the practice carries too many potential downsides and that its benefits just aren’t worth taking on the risks the practice can bring.

Which side are we on?

It shouldn’t come as much of a surprise— we’re all for testing in production.

We believe testing in production is a cornerstone practice for anyone in the development world. And we believe it is particularly important for Product Managers, as it gives them a powerful method to generate real-world feedback and performance data they need to make sure they are always building a viable pipeline of products.

But even though we advocate for testing in production, we still want to consider the point of view of those who are “No, never!” when it comes to this type of testing.

By hearing them out, we can learn some of the biggest potential pitfalls created when you test in production. 

Once we acknowledge these issues, we can start to map out some ways to mitigate the practice’s potential downside and capture the benefits of testing in production.

What are the big risks of testing in production?

To be blunt: A lot of things can go wrong when you test in production.

  • You risk deploying bad code
  • You may accidentally leak sensitive data
  • It can possibly cause system overload
  • You can mess up your tracking and analytics
  • You risk releasing a poorly designed product or feature

The list goes on and on. Anything that can go wrong, could go wrong.

Worst of all— if something does go wrong when you are testing in production, your mistake will have real-world consequences. Your product might crash at a critical moment of real-time usage. 

You might also end up collecting inaccurate KPIs and creating issues with your business stakeholders. 

Worse case scenario: your poorly designed product or feature might result in multiple paying customers leaving your product for a competitor instead.

Those who say “No, never!” to testing in production are correct to consider the practice highly risky, and we understand why they stay away from it.

And yet, while we acknowledge these concerns, when it comes down to it, we still advocate for testing in production because when done right, it can bring you unique benefits.

Why should you still test in production?

When done properly, testing in production gives you some great benefits that you just can’t get through any other method.

Here are the main reasons why you should test in production:

  • Collect real-world data and feedback

Testing in production allows you to collect user data in terms of users’ engagement with your new features. This enables the collection of valuable feedback from the customers that matters the most, which in turn would allow you to optimize the user experience based on this feedback. 

This will also allow you to brainstorm ideas for features that you may not have considered before.

  • Uncover bugs

Since you’re testing on live users, you would be able to discover any bugs or issues that you may have otherwise missed in the development stage. Thus, you can ensure your new products and features are stable and capable of handling a high volume of real-world usage.

It is worth noting that there are certain technical issues that will never show up until you put your product or feature in front of real-world users. 

Therefore, you can monitor the performance of your releases in real life so that developers can analyze performance and optimize the releases accordingly.

  • Higher quality releases

Because you’re receiving continuous feedback from your users, developers can improve the products resulting in high quality releases that meet your customers’ needs and expectations. 

Additionally, you can verify the scalability of your product or feature through load testing in production.

  • Support a larger strategy of incremental—or even continuous—release

Testing in production helps facilitate an environment of continuous delivery. 

This is especially true when you roll out your releases to a certain percentage of users so that they may no longer have to wait long periods of time before they have access to your brand new features. 

This way, you can limit the blast radius as with incremental releases, you would not have affected all of your users.

  • Perhaps, most importantly: you already are testing in production, even if you didn’t know it!

Most of Agile development and product management’s best practices are forms of testing in development. We’re talking about very common practices like:

If you are following any of these practices—and many more like them—then you are already running tests with real-world users in a live production environment. 

You are already testing in production, whether you call it that or not, even if you thought you were in the “No, never!” camp this whole time.

Testing in production done right

Let’s face facts: testing in production is now an essential element of product management. 

Just imagine trying to do your job without A/B testing or without phased rollouts or without the ability to collect any sort of immediate real-world feedback or technical usage data on your new products and features. 

You can’t. You need to test in production. You might as well embrace it.

And that is why we remain firmly behind testing in production, despite recognizing the risks involved in the practice. 

Believe it or not, there is a low-risk way to carry out this type of testing...

Safely test in production with feature flags

If testing in development is inevitable these days, then you should spend less time debating its pros and cons, and more time finding the most effective and responsible way to follow the practice.

We believe in this perspective on testing in development so strongly that we’ve built an entire product suite around helping product developers gain all of the benefits of the practice while minimizing their risks. 

Our product suite is called Flagship, and its core function is the feature flag. Feature flags are a software development practice that allows you to enable or disable functionality without deploying code.

By wrapping your features in a flag and deploying them into production without making them visible to all users, you can safely perform all of the testing in production that you need. 

With feature flags—combined with the rest of Flagship— you can:

  • Deploy smaller releases that minimize the impact of failure.
  • Only test your new features on your most loyal and understanding users.
  • Personalize their tests so they know to expect a few hiccups with the release.
  • Immediately toggle off underperforming features with a single click.

Read The Many Uses of Feature Flags to Control Your Releases for more use cases and examples.

With feature flags and a little planning, you can dramatically reduce the risk and increase the sophistication of the testing in production you are already performing. 

This means more real-world user data, more reliable products & features, and less worry about seeing how your hard work performs outside of the safe confines of development and staging environments.

If you’d like to learn more about Flagship, feature flags, or just how you can perform better testing in production, reach out today.

Flagship by AB Tasty recognized as a leader in The Forrester New Wave: Feature Management And Experimentation, Q2 2021
About the author
Arthur is Product Manager for Flagship, AB Tasty's feature management platform.

You might also like...

Release Management with Jira Software

Find out how Jira software can help enhance your release management processes.

Top 6 Release Management Tools

Here are our top picks of release management tools for faster and safer releases.

Fun with Flags: Short vs Long-lived Feature Flags

A look at the longevity category of feature flags and the difference between short and long-lived flags.

Copy link