A guide to help product managers to construct an experiment- and outcome-driven roadmap for enhanced feature experimentation.
In this article, we discuss the imperative use of feature flags in relation to technical debt. This includes aspects such as:
- What technical debt is
- The different types of technical debt and what types to avoid
- How feature flags can help reduce technical debt
- Feature flags best practices to prevent technical debt
Large software applications utilize feature flags for multiple functions, including the switching on and off of features and functionality, based on the need to modify system functionality without modifying code.
Unfortunately, feature flags can also incur technical debt, primarily when they are not managed and maintained. Each unmanaged flag added to this list directly impacts the testing resources in the form of time and human resources required to reduce the number of unmaintained flags. This then has a knock-on effect, leading to serious issues that will ultimately affect the software development and deployment lifecycles.
Therefore, to solve the challenges of continually occurring technical debt or code debt resulting from unmanaged feature flags, let’s consider the following points.
What is technical debt?
The term "technical debt" was first invented by Ward Cunningham, a signatory of the Agile Manifesto, in the early 1990s. His reason for its name is that technical debt bears direct correlations with financial debt. Software development teams can take shortcuts to satisfy immediate business requirements, but the debt plus accrued interest will have to be paid at a later stage.
Technical debt is the consequence of action taken by software development teams to expedite the delivery of a software application or specific feature which later needs to be refactored or redeveloped. If not attended to, technical debt can spiral out of control, resulting in the total breakdown of the software development and maintenance lifecycle. Therefore, it is critical to ensure that DevOps teams and software development teams pay close attention to technical debt management and technical debt reduction methods.
Before we look at the types of technical debt and technical debt measurement, let’s answer the question: Is technical debt all bad?
The theoretical answer to this question is: it depends. Martin Fowler’s technical debt quadrant demonstrates that software developers aren’t always aware of the technical debt they incur. Suffice to say, as soon as these technical debt-causing issues are highlighted, it is imperative to fix them as quickly as possible.
There are also scenarios where it is prudent to incur the technical debt and fix the issues later. A primary technical debt example of such a scenario is where an MVP (minimum viable product) is launched to get it in the hands of the customers. The benefits of deploying the MVP outweigh the cost of the technical debt incurred up to this point.
Types of technical debt
Generally speaking, there are two overarching types of technical debt: Intentional and unintentional.
Intentional technical debt occurs when software development teams choose to leave their code as it is, without refactoring or improving it, to reduce the time-to-market metrics.
Juxtapositionally, unintentional technical debt occurs when poor code is written and ignored. Thus, the code quality will need to be improved over time. There are two reasons for this. The first reason is a result of poor software development practices the first-time round. Secondly, coding best practices change over time. Therefore, existing code will need to be refactored to reduce technical debt and the risk of technical debt as time passes.
The Martin Fowler technical debt quadrant described above shows four types of technical debt:
- Reckless and deliberate
- Reckless and inadvertent
- Prudent and deliberate
- Prudent and Inadvertent
The inadvertent and deliberate distinctions are similar to intentional and unintentional classifications. Ergo, intended technical debt occurs when DevOps and software development teams choose to incur the tech debt. In contrast, unintentional technical debt occurs after the fact when software engineers need to make changes to existing code.
The reckless and prudent distinctions are unique and are what give this quadrant its value. Prudent technical debt describes the debt incurred because DevOps and developer teams know what they are doing when incurring the debt. Reckless debt occurs when development teams produce sub-standard, sloppy code that must be cleaned up afterwards.
Types of technical debt to be avoided
At this juncture, it is reasonable to conclude that teams should avoid technical debt. Additionally, it is imperative to minimize and eliminate tech debt, particularly reckless and deliberate code debt.
Prudent tech debt is the partial exception to this rule. This form of code debt can benefit software development organizations as part of the reducing time-to-value methodology. In other words, the advantages of delivering a product to market as soon as possible can outweigh the cost incurred by technical debt. However, it is critical to monitor the tech debt to ensure that its value does not spiral out of control, negating the benefits of the reduced time-to-value exercise.
How feature flags can help with technical debt
In summary, feature flags can help reduce the technical debt accumulated during the development, testing, and deployment of a software application. On the other hand, if feature flags are not monitored and maintained, they can increase the application’s technical debt.
Before we look at how feature flags reduce technical debt, let’s take a quick look at what a feature flag is:
“Feature toggles [feature flags] are among the most powerful methods to support continuous integration and continuous delivery (CI/CD)... [They] are a method for modifying features at runtime without modifying code.”
One of the most common sources of technical debt is the pressure to release a version of the software application. The business demands that the software be deployed, and they don't care how the developers make it happen. Feature flags are a valuable tool to help manage the "pressure-cooker" release environment.
There are several benefits to the use of feature flags as a software release and deployment aid, including:
- The risk of deploying a bug-ridden application is substantially reduced. Developers can simply switch off features that are not yet complete or thoroughly tested.
- By implementing a CI/CD methodology (continuous integration/continuous delivery), developers can often use feature flags to deploy new features without waiting for the next release to be deployed. In summary, this functionality reduces the time-to-value and increases customer satisfaction: A win-win for all.
- Implementing feature flags is also a means to negotiate with management about which functionality to complete before specified deadline dates, increasing the flexibility to develop and test features thoroughly before deploying them.
In summary, feature flags help manage and reduce technical debt by helping software development teams manage the development/testing/deployment lifecycle more effectively.
Feature flags are useful for dark launching, a practice of releasing specific features to a subset of your user base to determine what the response is to a new feature or set of new features. As an aside, this is also known as a canary deployment or canary testing.
Testing in production is another form of dark launching. By utilizing this option, you can assess the application's health, collect real-world app usage metrics, and validate that the software application delivers what your customers want.
Note: Feature flags can also create technical debt. While they play a significant role in mitigating technical debt in all other areas of the software development lifecycle, implementing them is usually via a set of complex if-else statements.
Therefore, in practice, a feature flag is an if statement defining the path between at least two different code paths depending on one or more conditions.
The following simple scenario describes how to implement feature flags.
Let’s assume that an eCommerce site offers free shipping for all customers that spend more than a specified minimum amount at one time. This code sample is an example of a feature flag. If the total amount paid is more than $50, then the shipping is free. Otherwise, the shipping amount is the amount spent multiplied by the rate (a percentage of the total amount).
def ShippingYN(amt, rate) : if amt > 50.0 : shipping = 0.0 else : shipping = amount * rate return shipping
Best practices using feature flags to avoid technical debt
As with all aspects of software development and deployment, it is vital to observe the following feature flag best practices:
1. Feature flag management
As your organization matures in its use of feature flags as an integral part of the software development/testing/deployment lifecycle, it is vital to be mindful of the fact that some of the feature flags are short-term and should be removed; otherwise, they will add to the application’s complexity, resulting in more technical debt.
Consequently, it is imperative to have a plan in place to remove the flags before even setting them.
It is also possible, and a good idea, to track and measure different metrics for each feature flag, such as how long it has been active, its states (on/off), different configurations, and how many code iterations it has been through. Once your feature flag has been through the required number of iterations to code and test a feature, this flag must be removed and the code merged into your code repository.
Note: Before removing a feature flag, it is a good idea to evaluate its function and purpose; otherwise, there is a risk, albeit slight, that the flag might still be needed and is erroneously removed.
A vital part of the feature flag management process is to define and implement temporary and permanent flags.
1.1 Temporary feature flags
As highlighted above, if a feature is designed to be rolled out to every application user or you are using the feature as a short-term experiment, it is critical to attach a monitoring ticket to this flag to make sure it is removed once the feature has been deployed or the experiment is concluded.
Examples of these temporary flags which can last weeks, months, or even quarters, include:
- Performance experiments: A performance experiment is similar to A/B testing, where two versions of the feature are deployed with the idea of determining which one performs better. A/B testing employs the same construct in that it deploys two versions of an element to the application’s user base to select which one users prefer.
- Painted-door experiments: These experiments are only used in the early phases of the software development lifecycle and are User-Interface mock-ups to determine any customer interest. Once the consumer interest has been determined, these flags can be removed.
- Large-scale code refactoring: It is a good idea to deploy code refactoring changes behind a feature flag until you are positive that the functionality has not been changed or broken. Once the refactoring exercise is complete, you can remove these feature flags.
1.2 Permanent feature flags
Permanent feature flags are used to implement different features or functionality for different groups of users. As a result, it is reasonable to assume that these flags will remain in the software application indefinitely or at least for a very long time. Therefore, it is vital to ensure that they are monitored, documented, and reviewed regularly. As with the temporary flags, there are several different types, including:
- Permission flags: These feature flags are helpful when your product has different permission levels, such as the ability to create journal entries in an online financial general ledger or whether users can view a list of these entries. A second use case for these flags is your SaaS application has different subscription models like Basic, Professional, and Enterprise.
- Promotional flags: These flags help implement regular promotions. For instance, let’s assume your eCommerce store offers a Mother’s Day promotion every year where specific products bought include the shipping costs.
- Configuration-based software flags: Any software driven by config files will benefit from using feature flags to implement the different possible configurations. A typical use case for config flags is the layout of the User Interface.
- Operational flags: These feature flags help manage a distributed cloud-based application. For example, additional compute engines can be spun up when the workload reaches a specific level.
2. Use a central code repository
Feature flags or toggles are most commonly stored in config files. Another option is to keep them in a database table. However, let’s look at how to manage these config files. Large systems can have many if not hundreds of feature flag settings. Apart from using a database table, the only way to manage these settings is to store them in config files.
The best way to maintain the config files is to upload these files to a feature flag library in a central code repository like Git. Not only is Git good for keeping control of these files, but it is also a valuable version control system. Developers can use it to create feature branches of config files used during the software development process without negatively affecting the production version of these files. Once the config files have been updated and tested, they can be merged back into the Git master branch using a merge request.
3. Adhere to naming conventions
It is absolutely critical to give your flags intuitive, easy-to-understand names, especially for long-term flags, although it is a good idea to include short-term flags in this best practice. Naming the feature flags, flag 1, flag 2… flag 100 will not help people who have to work with these flags in the future.
A good example of wisely named feature flags can be found in the scenario highlighted above. It is reasonable to assume that the flag, AdvancedSearchYN, is one of hundreds of flags used in our eCommerce application. Even if they are the only two flags used, it is still advisable to give them intuitive, related names.
As described throughout this article, feature flags in DevOps and software development play a fundamental role in managing and reducing technical debt. Consequently, it is vital to implement a feature flags framework using feature flags patterns as a foundational part of the software development lifecycle. Cobbling it on afterward can increase the risk of incurring more technical debt, especially once the system grows in scale. Similarly, these feature flags must be carefully maintained and monitored to ensure that they don’t have the converse function and incur additional technical debt.
Finally, it is essential to be mindful that, while technical debt is primarily seen as a negative, there are instances, as described by Martin Fowler’s technical debt quadrant, where incurring prudent and deliberate (or inadvertent) tech debt can be beneficial. It is also worth noting that both Agile and Scrum use the concept of technical debt in a positive way to reduce the time-to-value of a new application or feature release, driving sustainable growth through customer satisfaction.
Flagship.io: feature flag as a service for better flag management
Flagship.io is a cloud based service to remotely manage feature flags and take control over how, when and to who features are deployed. Progressive deployments to user segments, controlled rollbacks or feature experiments allow engineering and product teams to mitigate risk while optimizing user experience.
To help with technical debt management, Flagship.io provides dedicated features to keep control over your feature flags. Two of them are especially useful in this regard:
- The Flag Tracking dashboard lists all flags setup in the platform, with their current values (ex: on/off) and campaigns that reference them. This way, you can easily keep track of every single flag purpose (ex: flag 1 is used in progressive rollout campaign X, while flag 2 is used in feature experiment Y). When you manage hundreds of flags, it turns out to be a real time saver.
- The Flagship Code Analyzer is an advanced Git integration to see where your flags are used in your repository. In conjunction with the flag tracking dashboard, you can quickly find flags in your code that are not referenced in any campaigns. It also deeply integrates with your current CI/CD pipeline. As a CLI and a Docker image, it analyzes your codebase and detects the usage of Flagship flags everytime code is pushed to a specific branch or tag. This way, your flag dashboard is always in sync with your codebase. On one hand, you can safely remove flags if they are not referenced in campaigns, and on the other hand you make sure that flags your campaign is relying on are indeed available in your code. View code on Github.
Faster & Safer Releases through Product Experimentation
This guide will walk you through:
- How continuous development has turned product managers into experimenters.
- How to build an experimentation roadmap to run successful product experiments.
- Why you should test in production.
- The importance of feature flags in your release strategy to mitigate risk.