How to Do Production Testing Safely. Lessons From the CrowdStrike/Microsoft Incident

Continuous deployment and rapid iteration are “non-negotiables” for modern software development teams.

However, pushing updates directly into production without proper testing can lead to disastrous outcomes, as seen in recent high-profile incidents like CloudStrike.

Companies must adopt robust strategies like Canary Testing and Blue-Green Deployment to ensure rapid and reliable software releases.

This article delves into these methodologies, their differences, and how they can prevent critical failures like CrowdStrike's recent mishap with Microsoft that caused major financial losses to some of the world's biggest brands.

A smooth and well-managed Continuous Integration/Continuous Deployment (CI/CD) pipeline is the backbone of any effective deployment strategy. 

CI/CD pipelines automate the integration and delivery processes, allowing teams to deploy code changes frequently and reliably. By leveraging CI/CD, organisations can implement sophisticated deployment strategies like Canary Testing and Blue-Green Deployment, ensuring safer and more efficient releases.

Benefits of a Robust CI/CD Pipeline

• Automation: Reduces manual intervention, minimising errors and speeding up the deployment process.

• Consistency: Ensures that deployments follow a repeatable and predictable process.

• Scalability: Supports the deployment of complex applications across various environments.

Feedback Loops: Integrates monitoring and feedback mechanisms to continuously improve the deployment process.

Canary Testing, named after the canaries used in coal mines to detect toxic gases, is a strategy where a new software version is released to a small subset of users before a full-scale rollout. 

Canary Testing allows teams to monitor the latest release for issues in a controlled environment, ensuring potential problems can be addressed before the software reaches a broader audience. It is also used for User Experience (UX) testing to gather feedback on the new features and interface changes from a small group of users.

1. Release to a Subset: The new software version is deployed to a small group of users, typically from 1% to 10% of the total user base.

2. Monitor Performance: Detailed monitoring and logging tools track the behaviour and performance of the new release.

3. Analyse Feedback: Both automated metrics and user feedback are analysed to identify issues.

4. Gradual Rollout: If no significant problems are detected, the update is gradually rolled out to the remaining users.

• Risk Mitigation: Limits the exposure of potential bugs or performance issues to a small user base.

• Real-World Testing: Allows for real-world usage data, providing insights that may not be captured in a staging environment.

• User Feedback: Early feedback from actual users can be invaluable for making last-minute adjustments.

Blue-Green Deployment is a methodology where two identical production environments, termed Blue and Green, are maintained. At any given time, one environment (say Blue) is live, serving all production traffic, while the other (Green) is idle or being updated. When it is ready, a new software version is deployed to the idle environment. Once tested, traffic is switched from the live environment to the updated one.

1. Duplicate Environments: Maintain two identical production environments.

2. Deploy to Idle: Deploy the new software version to the idle environment (Green).

3. Testing: Perform thorough testing in the Green environment without affecting the live users.

4. Switch Traffic: If tests pass, switch the traffic from Blue to Green, making Green the live environment.

5. Monitor and Rollback: Monitor the new environment closely. If issues arise, a quick rollback to the Blue environment is possible.

• Zero Downtime: Enables seamless transitions with no downtime, ensuring continuous availability.

• Instant Rollback: If a critical issue is discovered, switching back to the previous environment is straightforward.

• Comprehensive Testing: Allows exhaustive testing in a production-like environment before going live.

• Granularity: Canary Testing is granular, targeting a small subset of users, whereas Blue-Green Deployment switches the entire user base between two environments.

• Risk Exposure: Canary Testing gradually increases risk exposure, while Blue-Green Deployment involves a complete switch, requiring confidence in the new release's stability.

• Rollback Complexity: Rolling back in Blue-Green Deployment is more straightforward since the previous environment remains intact. At the same time, Canary Testing might require halting the rollout and fixing issues in the existing environment.

Recently, CrowdStrike experienced a significant issue with Microsoft, where a botched update led to widespread disruption. This incident underscores the importance of meticulous production testing. Strategies like Canary Testing or Blue-Green Deployment could have mitigated or entirely avoided CrowdStrike's situation.

By releasing the problematic update to a small user group, CrowdStrike could have detected the issue early and prevented a large-scale impact. Monitoring the initial rollout would have provided insights and allowed for timely corrections.

Using a Blue-Green Deployment strategy, CrowdStrike could have tested the new update in the Green environment without affecting the live users. Any issues discovered would have remained isolated, ensuring the live climate remained stable.

Robust production testing strategies like Canary Testing and Blue-Green Deployment are crucial for maintaining software reliability and user trust. These methodologies mitigate risks, provide valuable real-world feedback, and ensure seamless transitions. The CrowdStrike-Microsoft incident is a powerful reminder of the potential consequences of inadequate production testing. Companies can achieve safer, more reliable software releases by learning from such examples and implementing these best practices.