How to abandon the intermediate development environment and not lose anything

Testing code in an intermediate environment before being released into production is considered an industry standard. This is a time-consuming process, which in the modern world can be completely abandoned in favor of testing in a production environment, on real users. In this article we will tell you what practices need to be implemented so that this transition is painless.

This is an adapted translation of the Delete your Staging Environment article from the Flagsmith company blog. The narration is conducted on behalf of the original author.

A short digression: a few months ago, our platform experienced a forty—minute API failure – the first in 18 months. The update that caused the crash passed all the autotests and worked fine in an intermediate environment. This gave us a false sense of security, because of which users suffered as a result.

The reason for the failure was a data mismatch between the two environments, which led to a database migration error. This made us think: if such critical errors pass through the intermediate environment, why do we need it at all?

Why do we use an intermediate environment

The task of the staging environment is to give development teams the opportunity to safely test updates in conditions close to production. The case described above shows that there is always an opportunity to miss an important detail. Why don’t we test the code right away in the product?

I have been developing software for the last 25 years and during this time I have tried many approaches. In almost all projects, the intermediate environment was used by default: simply because it is accepted. During this time, there have been a number of changes that allow you to abandon the intermediate environment.

Joint development

Now joint development is an industry standard. But let’s imagine (or rather, remember) a world without version control systems.

In the late 90s, I worked in a team developing software for USB drives, and we were constantly faced with the problem of source code management. The senior developer was responsible for the process management, whose task was to combine the programmers’ code. To do this, he collected the code and ran it in an intermediate environment for testing.

Autotests

Testing is a key stage of software assembly before deployment. Now the focus is shifting to testing in production on real users. In my opinion, automated testing has become an important milestone in these changes.

Today, programmers can hardly imagine their work without autotests, although before the advent of Selenium in 2004, unit tests were the standard in the industry. Autotests allow us to focus on the quality of the product for the end user, rather than on the compatibility of individual application components. This trend has become so widespread that now there is a debate in the community about whether to engage in unit testing.

Deployment with minimal downtime

Fortunately, the downtime during the upgrade is a thing of the past. In 1999, I advised a major British credit card manufacturer whose software ran on Sun E10K servers. When deploying a new version of the software, their engineers waited for midnight, went into the server room, which was located in a separate building, physically disconnected the servers from the fiber-optic cable and three hours later connected them back. During this process, many errors could occur, so there could be no question of updating the software several times a day.

Zero downtime when updating software that is performed regularly is a huge achievement. The Heroku and AppEngine platforms have made this process sufficiently reliable — so much so that the need for an intermediate environment is gradually disappearing.

Testing in production (feature flags)

Perhaps historically, the last change in the practices of working with code, which eliminates the need to use an intermediate environment, was the ability to separate the deployment from the release. The concept of feature flags was popularized by Martin Fowler: in his article he demonstrated how they allow testing individual functions in production conditions.

The concept of feature flags boils down to the fact that you test new features in production conditions, but “hide” them until they are ready for release. The main advantage of this approach is that all the code can be tested in a production environment, bypassing the intermediate one. This is the most accurate health check of updates. Many new concepts have emerged from this concept — for example, kill switches and A/B tests.

How to abandon the intermediate environment

There are five basic engineering principles that need to be implemented before getting rid of the intermediate environment. Even if you are not ready to give it up, these are useful practices, from the implementation of which you will get a lot of benefits.

The first principle. Culture of code review and pair programming

Right now, GitHub and GitLab merge requests are critical stages in the development process. They allow team members to analyze the code together — before the advent of version control systems, interaction of this level was impossible.

Writing the main application code without an intermediate environment may seem like a difficult task — errors can lead to downtime and loss of money. Code analysis and pair programming practices can help address these concerns. If you are not sure about your code, write it together with a colleague.

Principles two and three

Autotests and zero-downtime deployment

Releasing releases several times a day requires tools that will reduce the number of errors in production to a minimum. We are talking about covering the entire code with tests: modular, end-to-end, non-functional tests, browser testing and system performance testing.

Separation of release and deployment

Feature flags allow you to test updates in a production environment and maintain control over the release of new features with regular code deployment.

Principles four and five

Monitoring

It is not enough to specify Pingdom for one endpoint in the API: you need to monitor the entire stack at different points.

Test meaningful API endpoints that are linked to data and any other third-party services that you rely on.;

Test non-functional properties: latency, load, overall performance, and portability. For example, DebugBear is suitable for this.