Safe release, trust increase

You can lose someone's trust with the click of a button. Decades of earning the trust of people can be lost very easily. Software engineering has been there for around 60 years. And yet you still see problems, even in top companies.

One error can cost millions of dollars and, worse, people's jobs. Really? In Quora you’ll see which was the costliest coding error anyone has ever made.

TLDR;

• On August 1, 2012, a software bug led to the collapse of a major stock trading firm. Developers rushed to port their high-frequency trading bot to a new NYSE API, leading to an intense work schedule. They retained existing method signatures to avoid recompilation and client updates, but an error in the manual release process led to a disaster.
• When the market opened, the bot began buying the highest offers at $5 million per second. The developers were in a meeting and unreachable. The CEO had data center servers physically destroyed to stop the bot, but not before over $3 billion in stocks were bought.
• A manual release error caused one server to use test logic, leading to wild purchases. A rollback attempt worsened the issue, increasing losses to over $50 million per second. The total loss was $8.65 billion, with Goldman Sachs mitigating damage by purchasing the stocks for $457 million. The firm went out of business weeks later, highlighting the critical importance of proper software deployment practices.

Please note that: The following is not a bulletproof guide; it is just a quick guide to help you start moving towards Operational Excellence. Each sub-bullet is a very large topic by itself that requires a full chapter to explain it, but, to keep it simple and practical, I summarize each one with key points. This article relates to the Release Management process. This guide mentions the steps before, during, and after a release. I suggest that you go deep into learning about the Release Management process since this article only mentions it briefly.

Proper software deployment practices

1. Plan and Prepare.

Before starting to write code, it is essential to go through a series of preparatory steps to ensure a clear understanding of the requirements, design, and potential challenges. Here’s a typical checklist of activities:

“If I had an hour to solve a problem, I’d spend 55 minutes thinking about the problem and 5 minutes thinking about solutions.” — Albert Einstein

Make sure you thoroughly understand the project before jumping to solutions.

1. Understand the Requirements.

• Read the Story/Task: Thoroughly read the user story, task, or requirements document.
• Clarify Ambiguities: Ask questions to clarify any ambiguities or unclear aspects of the requirements with the product owner, stakeholders, or team members.
• Consider edge cases: - Identify Extremes. Think about the minimum and maximum values, boundary conditions, and unusual inputs. - User Scenarios: Consider how different types of users might interact with the system. - Past Issues: Review past bugs or issues for similar projects. - Collaborate: Discuss with the team and stakeholders to uncover potential edge cases.

2. Analyze the Impact.

• Identify Dependencies: Determine if the story has dependencies on other tasks or components.
• Impact Analysis: Assess how the changes will affect existing features and parts of the system.

3. Define Acceptance Criteria.

• Acceptance Tests: Ensure that the acceptance criteria are well-defined and agreed upon. These criteria will guide the development and testing process.

4. System design.

• High-Level Design: Create a high-level design or architecture diagram if needed.
• Low-Level Design: A diagram of the interaction of the main components or a class diagram.

5. Work structure.

• Break Down Tasks: Break the story into smaller, manageable sub-tasks and/or technical tasks.
• Choose Tools and Libraries: Decide on any specific tools, libraries, or frameworks needed for the implementation.

6. Environment Setup.

• Development Environment: Ensure your development environment is correctly set up with the necessary tools and dependencies.
• Access Permissions: Verify that you have the required access permissions to relevant repositories, databases, and other resources.

7. Research and Prototyping.

• Feasibility Study: Conduct a quick feasibility study or prototype to explore potential technical challenges.
• Review Best Practices: Look up best practices or patterns relevant to the story.

8. Plan for Code Quality.

• Code Reviews: Plan for code reviews by peers to ensure code quality and adherence to standards.
• Coding Standards: Review and follow the team’s coding standards and guidelines.

9. Documentation.

• Technical Documentation: Document the design, any architectural decisions, and how the feature integrates with the existing system.
• Comments: Plan to add comments in your code to explain complex logic or decisions.

10. Risk Management.

• Identify Risks: Identify potential risks and plan mitigation strategies.
• Contingency Plan: Have a contingency plan for handling unexpected issues or rollbacks.

11. Time Estimation and Prioritization.

• Estimate Time: Provide a realistic estimate for completing the story.
• Prioritize Tasks: Prioritize tasks based on dependencies, impact, and urgency.

12. Team Coordination.

• Communicate with Team(s): Communicate your plan with the team to ensure alignment and avoid overlaps.

By thoroughly preparing before starting to write code, you can ensure that the development process is smooth, efficient, and aligned with the overall project goals.

2. Execute and Eject.

1. Version Control and Branching Strategy.

• Feature Branches: Develop new features in separate branches.
• Pull Requests: Use pull requests to review and approve changes before merging.
• Main Branch Protection: Protect the main branch with required reviews and passing CI checks.

2. Write Tests First (Test-Driven Development — TDD).

If it ain’t tested, it’s broken — Bruce Eckel

• Unit Tests: Write unit tests for the functionality you are about to implement.
• Integration Tests: Plan and write integration tests if applicable.

3. Write (Clean) Code.

• Standards and best practices: Adhere at all times to the standards and best practices set by your Team.
• Pair Programming: Consider pair programming for complex tasks or when learning new technologies.
• More about this in this link: ©Lean Code

4. Continuous Integration (CI).

• Automated Testing: Implement unit, integration, and end-to-end tests.
• Build Automation: Automate the build process to ensure consistency.
• Code Quality Tools: Use linters and static code analysis tools to maintain code quality.

5. Continuous Delivery/Continuous Deployment (CD).

• Staging Environment: Deploy to a staging environment identical to production.
• Automated Deployment: Use tools like Jenkins, CircleCI, or GitHub Actions for automated deployments.
• Manual Approval: For critical releases, consider a manual approval step before deploying to production.

6. Feature Toggles.

• Feature Flags: Use feature flags to enable or disable features without redeploying.
• Gradual Rollout: Gradually roll out features to a small percentage of users before full deployment.

3. Monitor and control

1. Monitoring and Logging.

• Application Monitoring: Use tools like New Relic, Datadog, or Prometheus to monitor application health.
• Logging: Implement structured logging with tools like ELK Stack (Elasticsearch, Logstash, Kibana) or Splunk.

2. Rollbacks and Recovery.

• Automated Rollbacks: Implement automated rollback mechanisms in case of failure.
• Database Migrations: Ensure database migrations are reversible.
• Backups: Regularly backup critical data and ensure restore procedures are tested.

3. Blue-Green Deployments.

• Dual Environments: Maintain two identical production environments (blue and green).
• Switch Traffic: Switch traffic to the new environment (green) while keeping the old one (blue) as a fallback.

4. Canary Releases.

• Small Batch Deployment: Deploy the new version to a small subset of users.
• Monitor Feedback: Monitor and collect feedback before a full rollout.

5. Security Checks.

• Vulnerability Scanning: Regularly scan for vulnerabilities using tools like OWASP ZAP or Snyk.
• Dependency Management: Keep dependencies up-to-date and check for security advisories.

4. Communicate and Congratulate.

1. Communication and Documentation.

• Release Notes: Document changes, new features, and known issues.
• Incident Response: Have an incident response plan and clear communication channels.

2. Congratulate others.

• Boosts Morale: Recognizes and rewards hard work, enhancing team spirit.
• Encourages Productivity: Motivates the team to maintain high performance.
• Fosters Positive Culture: Builds a supportive and appreciative work environment.
• Reinforces Success: Highlights effective practices and teamwork, setting a positive example for future projects.

3. Congratulate YOU (last but not least).

• Take pride in your work, knowing it helps people to make their lives better in some way.

5. Retrospect and Refine.

1. Kick off retrospective process.

• Schedule the Meeting: Set a time and date for the retrospective with enough time allocated for thorough discussion.
• Gather Data: Collect relevant data from the project, including metrics, feedback, and notes from throughout the project lifecycle.

2. Set the mood.

• Create a Safe Environment: Ensure that the team feels comfortable sharing honest feedback. Emphasize that the goal is to learn and improve, not to assign blame.
• Icebreaker Activities: Start with a brief icebreaker to set a positive tone and encourage participation.

3. What did just happen?

• What went well?
• What didn’t go well?
• What should we start doing?

4. Decide and refine.

• Action Items: Develop a list of actionable items based on the insights gained. Prioritize and assign responsibilities.
• Process Refinement: Determine which processes need refinement based on the retrospective findings. This might include changes to workflows, communication strategies, or tools used.
• Continuous Feedback Loop: Establish a feedback loop to regularly review and adjust the implemented changes.

5. Close the Retrospective.

• Summarize Discussion: Recap the key points discussed, the action items agreed upon, and the individuals responsible for them.
• Follow-Up: Plan a follow-up meeting to review the progress on action items and the impact of the changes made.

Conclusion.

Safe releases enable teams to innovate fearlessly, maintain user trust, and continuously deliver high-quality, dependable solutions that make a meaningful (valuable) impact. Think of it like this: a release process is like a seatbelt — sure, it might seem unnecessary until you really need it

By following these practices, software teams can significantly reduce the risks associated with deploying new changes to production, ensuring a stable and reliable release process.