Succeeding in Technically Complex Software Projects

Complexity in software projects arises from two major sources: technical and organizational challenges. In this blog, I will share insights based on years of experience designing and leading such projects. Before diving into the strategies for executing technically complex projects, it’s crucial to first understand the factors that make a project technically complex.

Understanding Technical Complexity

Let’s explore two examples to illustrate what makes certain projects technically complex:

Example A: A Dating App Using AI to Match Users

Example B: An AI-Driven App Building Platform

At first glance, Example A doesn’t seem as technically challenging, while Example B clearly involves more complexity. But why is that the case? Here are some key factors that contribute to a project’s complexity:

1. Degree of Variability in Expected Outcomes

In Example A, the goal is clear: users need to be consistently matched with others who align well with their preferences. This involves managing a set of attributes for each user, which can range from a few basic details initially to more complex data in future iterations.

In contrast, Example B requires a platform that can generate virtually any type of app. The diversity of possible applications means that the platform must be highly adaptable. Each generated app may have completely different features, data models, and micro-level behaviors. This variability makes the project inherently more complex.

2. Existence of Similar Products

There are numerous dating apps on the market today. When designing a new one, we can draw from well-established concepts and patterns. However, for an AI-driven app builder, there are fewer precedents. While some no-code tools exist, they largely depend on the user to take most actions. None of them function as a true “virtual developer” capable of autonomously creating fully-fledged apps.

3. Conceivability of Major Flows

In Example A, we can quickly outline the major flows: users create profiles, answer questions, and receive matches based on an AI model trained on user data. However, for Example B, the flow is not so straightforward. If a user specifies what their app is supposed to do, should we keep asking follow-up questions to clarify gaps? How do we identify what gaps exist? How far should we go in refining requirements before implementing the solution? These are challenging questions that make defining flows more difficult.

4. Difficulty in Finding Solution Candidates

Building a prototype for Example A is relatively easy—using a pre-trained zero-shot language model, we can quickly develop a proof of concept. However, Example B requires deeper exploration to even conceptualize the first working prototype. It’s not just about functionality but about building a system that can understand and generalize from a wide range of app requirements.

Strategies for Succeeding in Technically Complex Projects

When working on complex projects, adopting the right strategies can make a significant difference. Here are some key strategies I’ve found effective:

1. Prioritizing Early Prototyping

Creating prototypes is one of the most critical steps in developing complex products. A well-thought-out prototype helps align stakeholders, uncover gaps in requirements, identify areas with no clear solutions, and guide the product and engineering teams. Prototypes provide a shared understanding of the project’s direction and are instrumental in iterative development.

I will cover the topic of prototyping in detail in a future blog post.

2. Clarifying the Tech Roadmap

While features and details may be refined incrementally using an Agile SDLC approach, a high-level tech roadmap should be established early on. This roadmap should outline expected behaviors on a quarterly basis. Stakeholders should review and agree on it to minimize tech debt and reduce ambiguity. An effective roadmap helps keep both product and engineering teams aligned.

3. Embracing Versioning and Simplification

The product team should aim to provide detailed specifications, but the engineering team should collaborate with them to define early versions with simpler features that can be developed quickly. There’s little value in perfecting minor features, like fine-tuning UI elements, during early stages. Instead, releasing simplified versions for demonstration helps clarify concepts and exposes requirement gaps, which is vital for Agile methodologies.

4. Hiring the Right People

Rather than discussing generic hiring practices, I want to focus on traits that are particularly beneficial for complex projects. Candidates who are highly detail-oriented may struggle in environments where frequent trial and error are necessary. Look for people who thrive in ambiguity, can explore and compare multiple solutions, and remain organized as requirements evolve.

5. Demonstrating Progress Frequently

Developing complex products often involves deviations from the original plan as solutions may not work as expected. Regularly demonstrating progress, even if the current solution is not fully functional, can alleviate concerns among non-technical stakeholders and show that the project is moving forward. It also allows team members to participate in refining the solution collaboratively.

6. Releasing to Production Early

While it’s important to avoid releasing a buggy product, delaying the launch to perfect every feature is counterproductive. Sometimes, an early release with some manual intervention behind the scenes can be valuable. It provides real-world feedback on how users interact with the product, allowing for better prioritization and more realistic planning.

7. Prioritizing the Backlog

New features often emerge during development when the product or engineering teams encounter unanticipated cases. Encourage the team to flag such scenarios so that the product team can quickly assess and prioritize them. Keeping the backlog well-organized ensures smoother sprint planning.

8. Documenting the Architecture

A document outlining the high-level components, their integrations, the rationale behind design decisions, flow across components, and the physical architecture is essential. However, documentation should not be excessively detailed at the start; a good rule of thumb is to keep documentation 2 to 3 sprints behind development to avoid rework due to changes in requirements.

Conclusion

Successfully navigating technically complex projects requires a balance of strategic planning, agile execution, and continuous feedback. By focusing on early prototyping, refining roadmaps, embracing iterative releases, and fostering a collaborative team environment, you can significantly reduce the risks associated with such projects.

In future blogs, I will dive deeper into topics like prototyping, creating effective roadmaps, and building scalable architectures.