Scaling Engineering - How Reddit was able to Triple in Size

This is a blogpost to summarize some of the key notes from the InfoQ video: Scaling @reddit Triple Team Size w/o Losing Control. I though this was a interesting video to take notes for as it discusses issues from a process and team size, and how to create a good environment for engineers to work in, and what we should strive towards. Even the best architected project will fail as a product if the management isn't there.

Video Link: https://youtu.be/u6hmMW_6fOw

1. Roles and Responsibilities

The presenter's main point around this was that each person should have a defined scope of responsibility to his team, his peers, and his bosses. By defining ambiguity of roles, you are able to delegate tasks more effectively and specialize tasks for the right person.

RACI (Responsibility Assignment Matrix)

A RACI chart is suppose to relate the task to the position, in what part that position must do for that task.  

R - Responsible, The person that does the task (1)

A - Accountable, The person that approves the task (1)

C - Consulted, A person who may provide information on the topic  (Many)

I - Informed, A person who must be notified of the progress of the task (Many)

Program vs Product vs Engineering Managers

Program Managers are people who may have multiple responsibilities to the product and engineering teams. These are more common where people need to where multiple hats due to limitations.

Product Managers motivate the product vision to answer the following: Who are the customers we are trying to address, What is the vision of the product, and How we are going to execute this vision.

Engineering Managers worth with the (Who) development team to formulate an (What) architecture in order to (When) deliver products.

Trust (But Verify)

This advice was meant for how we should treat new and old members of the team. Trust them that they know how to complete their work, but verify that they can deliver quality work. This enables the person so work at their maximum capacity without being bottle necked by micro-management.

2. (re)Orgs

This discussion was around the impact of your organization has on your product, and vice-versa.

Conway's Law

"Organizations which design systems are constrained to produce designs which are copies of the communication of these organizations".

Which effectively translates to: "You will eventually ship your org chart."

This means that your product will have relationships that are based on how those teams communicate with each other, so no matter the architecture you start with, as the teams evolve it is important to evaluate the relationships between those teams.

"Your organization should reflect your company's most immediate goals and the best people for each goal should be continually moved to where they are most needed...ideally"

Reverse Conway's Law

"Do ship the org chart, but make sure you have the right one!"

This is saying that don't be afraid of this concept, but utilize it to your organization's advantage. You can shape your organization in a way that it will naturally promote the idea you are striving for.

3. People

No matter how smart of people you hire, its ultimately communication that will create a great development organization. This section was dedicated to how you can built culture within your organization, and how to enable communication.

Building Culture

Assume subcultures can form, and let that help you build relationships within your company. When people feel connected, they are more likely to be happier, communicate more, and feel invested to the company. Make sure that your core values are instilled into these groups, and you will be able to promote a good working environment.

Tools for Building Culture

There are a bunch of things you can try, but none of them are the silver bullet you need. You just need to use all of them, as different methods may affect different groups. Be vigilant!

4. Scaling Engineering Problems

This section was going over how increasing the size of your organization adds complexity to your engineering division.

Awareness 

As their are more people added to projects, people have less transparency into components of the product. People will have reduced awareness of what other things are happening within the team, and they will lose context and be unable to make as good as decisions.

Coordination

With more people and teams and more separated responsibilities, it increases your dependence on working with other teams. It becomes a requirement for communication between teams, and increases management complexity to deal with.

Bottlenecks

Increased dependencies cause releases to be more complex. By adding dependencies there becomes a lot more risk for deliveries.

5. Process Advice and Caution

Avoid Boilerplate

Don't rely on a generic methodology to manage your team. Each team must solve a different problem, and by trying to rely on a generic solution, you will most likely introduce inefficiencies and promote less distrust of the system.

Code > Process

Always value the processes that put code delivery over process. Process is a tool for you define a path that allows you to push code quality code.

Kaizen

Kaizen is the methodology created by Toyota for incrementally improving process. In their use case on the assembly line, each line member has an alert they can activate if they see a problem in the manufacturing process. When activated, the whole assembly line will stop and managers will come review the defect. Only then will a new process be created to resolve the issue.

This lightweight approach ensures that you only perform processes that are necessary to your problem space. Not to say that methodologies don't work, but you as a manager need to determine which are effective for your team, and promote those. And don't be afraid to throw away things that don't work.

Process becomes Culture over Time

Don't force process on your people, as they will inevitably disregard those processes when given the chance. Try to ease in these processes into your team, and make sure to get buy-in from your people for why it can be effective. Once your people buy-in to those ideas, they will be the promoters of the idea and it will become accepted.