Building a high-performance tech team in India

I have been receiving a lot of requests lately to share my journey of building a high performance tech team, so here it is...

Background

We wanted to build an All-in-One app with all the frequent use cases covered. Along the way, we knew that we’ll make mistakes, must do a ton of experiments, and improve across so many distinct categories/services at a phenomenal pace.

To achieve this, I took a 2-step approach, first I deep dived into the existing problems where I identified important gaps like P0 bugs in production, tech debt because of no focus on tech design, improper sprint planning, no focus on operational excellence etc. With these insights, I tasked myself to connect with industry experts who have done this before and learn from their experiences and the mistakes they made. In the next few months, I spoke to many companies of all sizes (the biggest was a 400+ people engineering team, $2B market cap and the smallest was a 25 people engineering team) in India, Silicon Valley, and China. The goal was to understand what the best practices were and then try and apply them to our situation. 

Things which worked for me...

1. Technical Design sprint between sprints: We loosely followed the agile methodology. One of the constant complaints was of bugs, technical debt accumulating too frequently, and delays in launch. When I went deeper into these issues, I realized that the key reason was that the dev started coding as soon as they got the product spec instead of thinking about how it should be designed and coded. And the schedule of 14-day sprints wasn’t helping – a delayed launch would cause too many frayed nerves all around and at the beginning of every sprint, instead of spending time thinking, SDEs were fire-fighting. Another psychological reason was that most SDEs felt that meetings were a waste of time. We needed to institutionally support tech design thinking before any sprint.

We introduced a 3-day mini sprint between two 14-day sprints and ensured that all pod leads and their teams sit in conference rooms and think about how to architect and code the requirements given in the product spec. The entire process became a lot faster and stress-free. It also led to a reduction in bugs by 12% MoM, technical debts, and 25% reduction in unexpected launch delays.

2. Small is powerful: One of the most impactful changes we made was to split our tech team into smaller teams of 5-6 people (3-4 SDEs, 1 QAE, 1 FEE) from earlier team sizes of 10-12 people. These smaller teams were now called as pods. Along with the pods, we formed a central dev-ops team and a central infrastructure team (which looked at things like payments, notifications, and email infra layers).

This helped us move faster with more independent teams owning just 1-2 microservices, thus less context switching and meetings. This also meant that new leaders were created in the team, leading to more motivation factor.

3. Business perspective: I figured out that as engineers it was difficult to sometimes grasp the difference in the level of business perspective that we had vs Product Managers. This hindered our ability to understand the impact we were creating for the business. I have seen that the Socratic method of asking engineers questions about business works very well (“What do you think would be the CTR if you do this?”; “What % of our users do you think do X”; “What would be the lost revenue because of this extra screen?”). The look of clarity dawning on people’s faces when they realize the business impact of their work is priceless and it switches on a button in their minds.

This led to engineers giving feedback to product team for new features and prioritization.

4. Training and introspection: As Andy Grove put it in High Output Management, you can either train people or you can motivate them. While small pods and business perspective took care of intrinsic motivation, training is what was left.

- Introspection session: Every fortnight, at the end of each sprint, we sat down for 4 hours – each pod lead, product manager, and co-founders. We talked about what mistakes we made in last 14 days. We encouraged everyone to use “I could have done this better….”. It’s not “we” or “us”. It’s an individual introspection and then openly acknowledging what they could have done better and what 2-3 areas they will focus on improving. It was the most high-leverage meeting because you understood what people were thinking, how they were self-actualizing, and what was holding them back.

Biggest output of this meeting was that same mistakes were not repeated. We concluded these meetings with 10-15 actionable and over the next sprint, we found different ways of accomplishing it.

- Management training sessions: As you break the tech teams into pods, a lot of young people became responsible for other people’s outputs for the first time. This can be more frightening than most of us might realize. To help people make this switch, we took the following training session: Effective communication, how to fail fast, clear expectations setting, doing 1-on-1s etc.

These sessions over 1 year helped us grow 4-6 engineers into successful people managers at Amazon post acquisition.

5. Build tools to ship faster: We also got the dev-ops, QA head, and central infra team to find and put in place tools/techniques/practices which could help us ship code faster and with more consistency. We implemented “Docker Swarm” cluster for our staging environment. Suddenly every dev was able to save at least 20% of his time every sprint because of streamlined staging deployment and environment. Another benefit was that the infrastructure resource utilization was better, which meant ~$15k/month savings for the company. Other optimizations included code quality checks using SonarQube, regression tests using Jenkins, code profiler like New Relic for optimizations, DB level slow query logging, and automated on-call management etc. While it felt like we were slowing down in the short run, in the long run, it helped us run faster.

Conclusion

Doing all the above helped us ship faster than before. The number of P0 bugs reduced by almost 22% every subsequent sprint and deliveries started happening on time 46% of the time. It took us a good 1 year and a lot of effort to learn how to do it and make it a habit. Our tech debt also came down by 40%. We kept improving these mechanisms and became 3X better from where we started by launching new features every couple of weeks vs 60 days earlier.

(Thinking retrospectively after working at Amazon & Reliance, these learnings seem obvious, but at that point it was one of the biggest eureka moments for me and my company).

Lesson Learned

Unlearn and Grow: I think I could have done a better job of introducing this new way of working to the team. People who learned these new ways of working flourished, while some good engineers felt left behind. The realization came in retrospection post some exits which gave me the idea to write an engineering handbook to ensure every new engineer knows our ways of working and what to expect. 

/

I am looking forward for all of you to share your journeys with me in the comments below or ask me questions that you might have. I will share the Engineering Handbook I wrote next week in the hope that it might help you as it helped me.