How can you improve the quality of cloud-based or distributed solutions?

Cloud native application development is an iterative process that needs periodic review across all aspects of application life cycle. Goal for the initial build is always to meet the business objectives, any trade-offs you made during this process are opportunities to be address in future iterations with ever changing landscape of cloud services offered by Cloud providers. In my experience AWS Well Architected Framework serves as good guiding principle to build a well rounded solution that is high performant, reliable, secured, day 2 ready, cost effective and sustainable.

Once you've pinpointed the trade-offs that align with your solution's goals and engaged all the key players, it's a prudent move to craft a 'Resilience Matrix.' This visual tool not only catalogs trade-offs but also maps out their ripple effects across various facets of the project. By doing so, you empower your team to make informed decisions, having holistically assessed the potential impacts. This matrix becomes an invaluable navigation chart as you steer your solution through uncharted waters.

Cloud-based or distributed solutions rely heavily on 4 key areas for any problem i.e. scalability, reliability, security, performance and cost (the last two often rely on storage and data retrieval and cpu/gpu speed). As the systems grow each of these can be compromised. The scalability,security and reliabililty often has to be applied on all layers in between.

Definite, realistic business/product objectives like SLA, SLO, delivery, budget, and regulations limits would narrow down the alternatives. A minimalistic approach is always best since it entails fewer parts to address.

Go to cloud when 1. There is shortage of hardware or hardware is outdated and you do not wish to add more or 2. You expect unpredictable bursts of load Go for distributed computing (microservices) when 1. You have a bunch of feature owners who have short release cycles and coordination between them is difficult and have unpredictable load pattern What is important is to architect and design the applications such that they are extremely efficient in the usage of resources. Deploy it such that it operates with minimum possible resources. Ensure that it is possible to detect the load accurately so that it is possible to scale up and scale down the infrastructure very quickly. Without these, operating on the cloud will be expensive.

In my experience, always consider designing cloud agnostic solutions where possible. These are solutions that aren't soo dependent on a particular cloud provider that it becomes hard to re-host especially for organisations with a Multi-cloud strategy or have yet to determine on which cloud provider provides the optimal blend of cost, capability and compliance to specific standards & constraints. If ignored, this normally leads to Vendor Lock-in and reduced flexibility especially in highly regulated countries or regions.

Knowing the services that the cloud provider of your choice offers is the most important thing when building cloud native solutions. Having knowledge in architecting solutions using established patterns and practices will guide you in the right direction. Remember each service you use comes with a cost, and knowing on when and how to use it plays a big part of the success of your solution.

One thing I've found helpful is to thoughtfully document and collaborate on design descisions. Having clear and open internal documentation on why certain services were selected based on quality attributes and trade-offs is essential. It allows the best ideas to come forward, provides a record of descisions, and is inclusive of teamwork.

In my experience, a Product Owner with Product Manager(s) should draw up a matrix that revolves around product features. They should then map infra, security, network, and data architectural requirements against it. Depending on what feature requires what components from the matrix, cloud computing and storage can be easily calculated.

Cloud Fitment, Cloud Provider Services fitment analysis for the problem in hand is very required to be assessed by the solution architect of the program in-alignment with the Enterprise Strategy. Based on the business use case, the selection of Cloud services should be done in consideration of Functional and Non-functional requirements of the business. As each of the CSP has similar and specialised service offerings, as i mentioned earlier appropriate selection should be done after due fitment analysis. This is not just limited to the selection of Services but also includes the selection of an architectural style like Microservices etc.,

Especially distributed systems need to be designed for handling failures and recovering gracefully. At the same time, they need to be instrumented to capture every one of these steps taken by the system itself to allow architects to gain insights from the automated recovery situation. This is very helpful in detecting a shift in usage pattern and may bring upon the need for adjusting architecture to handle such scenarios in future. Distributed architectures age like any other architecture because the initial design assumptions may not hold true over time. Many times these shifts show up in component failures or system degradation in terms of performance. By monitoring these automated recoveries and failures causes, designers can mitigate it

In any tech environment, especially the vast cloud, failures are a given. It's not about if they'll happen, but when. This reality demands proactive planning. Instead of aiming for a flawless system, it's wiser to create one that can bounce back from issues. Techniques like retries, circuit breakers, and fallbacks are invaluable. They let systems heal themselves or at least minimize disruptions for users. The cloud's vastness offers unmatched flexibility and scalability. However, it's not without challenges. By adopting a forward-thinking, informed design strategy, we can navigate these challenges, ensuring our systems are not only robust but also resilient and easily recoverable.

Solution Architect should think about resilience, failure and recovery at Service level, infra level, zone level,region level and even Service Provider level. Although Cloud based offerings from each of the SPs assure you of failure and recovery architectures, but careful selection of these features is necessary based on the Service level commitments.

Design must consider network failures and recovery which is an important component for distributed applications on multi cloud platforms. Plan and provision alternate network routes and cloud service provider (CSP) availability zones. Monitoring and health check across the distributed applications is an important design consideration for reliability.

Auto recovery to failure is the only way to measure any design's operational quality. It is delivered via how best you have coded to handle events, business errors, to all the way, even perhaps an entire availability region is in disaster (example AWS Serverless infrastructure failure). Using geographically distributed fault tolerance can be expensive, so chose the solution that best fits your requirements. Since flowing data into cloud is way cheaper, I would consider alternative cloud service/region if possible to store critical business events and backups. There are many solutions to provide scale and agility, keeping performance still key, combining hardware infrastructure and the right tech-stack. Eventual consistency is an example.

In the fast-paced world of cloud and distributed systems, what's effective today might not be tomorrow. Think of regular testing and monitoring as routine health check-ups, ensuring your system stays in top shape. As the tech landscape evolves, it's paramount to keep your tools updated, tapping into the latest advancements. Periodically, challenge your system's robustness with "chaos engineering" to identify hidden vulnerabilities. If any component falls short, it's time for a tweak or replacement. Remember, the journey of testing and monitoring isn't a one-off; it's ongoing, and crucial for maintaining the excellence of cloud or distributed setups.

Other things to consider during testing: • Increased test coverage by 20%. By increasing the test coverage of cloud-based you can improve its quality by identifying and fixing more defects. •Reduced mean time to detection (MTTD) by 50%. By reducing the MTTD, you can fix defects faster, which can help to improve availability and performance. • Improved mean time to recovery (MTTR) by 30%. By improving the MTTR, you can recover from defects faster, which can help to reduce the impact of defects on the users. One of the ways to achieve these: • Automating the tests. Automated testing can help to increase the speed and efficiency of testing process. It can help to achieve higher test coverage and reduce the MTTD.

Distributed solutions are difficult to debug therefore a lot of emphasis should be places on observability of these system. Implementing observability is a fine balancing act and requires designers/architects to carefully define approaches, patterns and find ways to instrument and automated so the developers of the solution can easily implement what they need to without sacrificing observability. This problem get a but more tricky if distributed solutions integrate through other systems. In such cases managing quality across all the integrated systems may end up requiring broader solutions, some of these are available as products. Doing buy versus build analysis is also key to achieving desired quality goals in shorter time.

Improving the quality of cloud-based solutions involves a multifaceted approach. A deep understanding of business requirements is critical. Aligning technical designs with business needs ensures that the solution addresses specific pain points effectively. Leveraging PaaS offerings available in the cloud is another pivotal strategy. PaaS not only expedites deployment but also mitigates operational costs, allowing teams to focus on innovation rather than infra mgmt. Lastly, implementing robust monitoring and testing practices, you gain insights into system behavior, performance bottlenecks, and potential failures. This proactive approach aids in swift issue identification and resolution, ensuring a seamless user experience.

Improve cloud-based solutions with: Security: Data protection, evolving threats, encryption, and access controls. Scalability & Performance: Horizontal and vertical scaling, caching, and CDNs. Cost Optimization: Monitoring, tracking, alerting, and cloud-native practices. Data Management: Storage strategies, backups, and data retention policies. DevOps & CI/CD: Automated testing, deployment, and infrastructure as code. Monitoring & Observability: Robust monitoring and automated alerts. Compliance: Adherence to industry regulations. Disaster Recovery: Business continuity, geographic redundancy, and failover. Education & Feedback: Team training and feedback for continuous improvement. Finally Updated docs and cross-functional collaboration.

Tailor the architectural blueprints by utilising a decision tree that considers scalability, reliability, security, performance and cost. Adhering to project centric best practices is imperative for achieving optimal outcomes.

You may need to add Data and security area for cloud based applications. Data storage and security, access policy across distributed systems is an important aspect for architecting solutions.