So far away: Distributed development

If you are a software geek, like me, being the product support technician for your friends and family comes with the territory. While it’s painful to watch your family struggle with software, particularly if you helped write it, at least you can tell them, “Back off, I’m a computer scientist,” and repair whatever is wrong. Sure, you’ll cringe as you undo their failed “fixes,” but in time you’ll set things straight. That is, if you live nearby.

If your mom lives a thousand miles away, the call might sound more like this:

Mom: Honey, they changed my password and now my e-mail doesn’t work.

Me: Okay, log onto your computer and open Outlook.

Mom: Using what password?

Me: Use whatever password you normally use.

Mom: Not the new e-mail one, just my old one.

Me: Yes. Then open Outlook.

Mom: Where do I type in the password?

Me: On the main screen where you click on your name and then type a password.

Mom: What main screen?

Me: Wait, can you just open Outlook right now?

Mom: Yes, I’ve got it open, but you told me to log on.

[Another hour like this moving through menus, dialog boxes, and buttons…]

This experience is far better if you can get a tool like Remote Assistance to work, but getting that up and running behind firewalls and routers can be equally entertaining. What should be a five-minute fix becomes an hour of acute aggravation. This order of magnitude multiplier for time and trouble comes into play any time you work across time and distance, which brings us to the topic of distributed development.

Doesn’t anybody stay in one place anymore?

It’s becoming far more common to run development across global locations for the same project. While the added diversity and talent should be a huge advantage, the results are often frustration, delays, and disconnects in quality and functionality. Why? It’s due to the big, bad Bs—bandwidth, boundaries, and being there.

There’s insufficient bandwidth for clear communication and fast network access to central services. The boundaries between project work at the different locations are poorly defined causing additional communication, conflicts, calamities, and clean-up. And because the different teams are separated, the one-on-ones, drop-ins, hallway conversations, and other daily human interactions you get from being there don’t occur.

With all this trouble you might wonder why we bother with distributed development. No, you fool, it’s not the money. Engineering salaries and costs are converging for many roles in China and India, and distributed development adds overhead. The real reasons for distributed development stem from the talent and the markets. The computer science talent pool in Brazil, Russia, India, and China is growing at nearly 20% per year and will have 1.5 times the number of software engineers in all of the United States by 2010. They can’t all move to Redmond, and we wouldn’t want them to because their home markets are critical to our success.

So if you’re running a team, you’d better learn how to deal with the big, bad Bs. Let’s break them down.

I get so tired when I have to explain

The first big, bad B is bandwidth—bandwidth between people and between computers.

Clear communication between people and teams is critical to success, as I’ve said many times before. The amount of information that is communicated between two people in the same room far exceeds that between people over video teleconference (VTC), which in turn exceeds Live Meeting, which exceeds the telephone (as shown in the exchange with my mother), which exceeds IM, which exceeds e-mail. In other words, e-mail is incredibly lame as a communication vehicle, so naturally it’s the most popular.

Actually, people use e-mail because it’s convenient, asynchronous, and works across time zones. Unfortunately, because e-mail has such little bandwidth, the communication is poor and often several round trips are needed to gain comprehension and answers. Due to time zone differences, a round trip often takes a day; thus e-mail communication proceeds at a glacial pace. Bandwidth between computers can also be quite slow, which means quick source control, file, or database operations in Redmond may take hours overseas.

How do you beat the big, bad bandwidth issue? There are only two ways—increase the bandwidth or cut down on the necessary communication.

§  You can increase the bandwidth by using higher bandwidth communication tools, like VTC and Live Meeting. These work even over low network bandwidth lines. However, the tools are synchronous, so you must reserve overlapping work time for communication with distributed team members. That means if you’re in Redmond and other team members are in China, you should reserve 4 – 5 pm Pacific Time everyday for scheduled and impromptu meetings with your peers in China.

§  You can cut down on necessary communication by crafting far more careful e-mails. Mitigate questions and confusion by providing added information and answering common questions in advance. It will take you a bit longer to write e-mail this way, but it won’t take you days, which is how long the communication will take otherwise.

§  You can cut out the need for a whole class of e-mails and phone calls by keeping a SharePoint wiki with project information. Fill it with how-to topics, discussion archives, checklists, documents, and project mail, schedule, and status. Any time a new team member arrives at any location, assign them to update the site with improved instructions or missing documentation.

§  You can also cut down on necessary communication by creating clear project boundaries between the work that is happening at different locations. Isolate local communication, including caching of source control and databases. Then you need cross-group communication only when people or information cross boundaries. That brings me to the next big, bad B.

Doesn’t help to know that you’re just time away

The second big, bad B is boundaries—boundaries between work at different project locations.

If you’ve never worked on distributed projects or teams before, you might foolishly think that distributed groups are all just one happy team. It’s this kind of naïve thinking that causes managers to allow individual team members to work alone from remote locations. Remember, being ignorant, naïve, and foolish is just one step from being stupid.

A project team residing in three separate sites is not one happy team; it’s three potentially happy teams working on one potentially successful project. By team I mean a group of individuals with a common understanding working in unison toward a common goal.

Due to the bandwidth issues I described earlier, you cannot hold and maintain a common understanding and work in unison unless you are likely to pass by your teammates on the way to the restroom. That’s why shared collaborative spaces are the best. That’s also why teams split across floors experience many of the same problems as teams split across continents.

To run a successful distributed project, you must create clear boundaries between the distributed teams that provide enough isolation to allow them to work independently with only minimal coordination; the clearer the boundaries the better the result. I talk about this more in Blessed isolation.

Typically, boundaries are architectural that isolate components, but they could be boundaries between versions or responsibilities. You could also have distributed teams focus on local market scenarios. Treat each distributed team like a dependency or a vendor and you’ll be on the right track. This doesn’t add unnecessary overhead; that communication overhead is there regardless. Instead, it reduces unnecessary conflicts, catastrophes, and clean-up.

But if your teams are isolated, how do you keep that sense of unity and sharing as you drive toward common goals? That’s where the last big, bad B comes into play.

It would be so fine to see your face at my door

The third big, bad B is being there—being there to create the human bonds necessary to achieve real cooperation and connection.

Making the best use of limited bandwidth and clear boundaries will mitigate much of the difficulties with distributed development. However, you are still one group of people working on the same project with the same goals. There are important relationships you must maintain between teams, peers, and reporting structures that require a human face. Being there is important.

VTC and other live presence tools can help. You should use them regularly for one-on-ones and meetings, perhaps not every time but at least once per month. Remember to reserve the overlap time between locations for these functions.

Of course, nothing can replace actual human contact, so plan to have everyone visit with each other at least twice a year. Plan one visit in October for the company meeting and performance reviews, and plan a second visit in February or March for budgeting and fiscal year planning. Don’t visit for a few days; visit for a week or two. Have morale events, one-on-ones, and training, as well as planning and review meetings. Make the most of your time together.

A number of teams do rotations or exchanges. In these cases, team members from one location spend three months, or even six months, with team members in another location. It can be a swap or an assignment. These rotations provide tremendous knowledge transfer while creating understanding, empathy, and connection across the groups.

One last thought… Sometimes giving the illusion of being there can be very effective at maintaining relationships and improving communication. Some companies have tried continuous video feeds of common areas. A cheaper and fun solution that’s been effective at Microsoft involves big pictures and even life-sized cutouts of team members placed in high traffic areas. As people leave meetings or walk the halls thinking about the project, they see one of these cutouts and are reminded to include that person or team in the conversation.

Where are you when the sun goes down?

With a little effort, you can make distributed development work for your projects and teams. Soon the sun may literally never set on your team. The advantages for you and Microsoft are enormous.

The diversity of experience, culture, and ideas might initially cause discomfort for you and your group, but over time it will make you better, your group better, and your products and services better. Your work will be more relevant in more markets for more people, and you will learn and grow in the process. Take the time to beat the Bs and you, your group, and your work will become worldly wonders.