Switching to DevOps

By now, you’re probably annoyed or infuriated with folks talking to you about DevOps. Annoyed, perhaps, because your team already does DevOps, whether you like it or not. Infuriated, perhaps, because your team tried DevOps, and it was a disaster, or your team doesn’t think you deliver services, so DevOps is superfluous.

I’ve touted the benefits of DevOps specifically in Bogeyman buddy—DevOps and generally in The good stuff. Any self-respecting team that delivers services should use DevOps or be replaced by a team that does. That leaves two unanswered questions: Does your team deliver services, and how does a team smoothly transition to DevOps?

Teams that work on components or tools might believe that they don’t deliver services. Perhaps. How do your components or tools get to customers? If they’re included in package feeds, you deliver services (via those feeds). Who services your components and tools when they need fixes? If it’s your team that gets woken up when there’s a security or reliability issue, you deliver services (though indirectly). These days, nearly every software team delivers one or more services directly or through components and tools used by services. So how can your team smoothly transition to DevOps? Glad you asked.

Shoot first and ask questions later

Teams make several common mistakes when transitioning to DevOps that cripple their productivity and crush their morale. Here are the anti-patterns.

• Flipping the switch. Many teams simply state that they’re now doing DevOps. They create an on-call schedule, cut operations vendors and budgets, and tell the dev team to now handle all deployments and outages. However, if the team hasn’t already automated testing, monitoring, and deployment, flipping the switch typically leads to service downtime, schedule slips, and staff, market, and revenue loss.
• Tossing the testers. Many teams see DevOps as an opportunity to force the dev team to test their product, so there’s no longer a need for testers. With sufficient automated scenario and stress-testing coverage, you could reduce the number of testers to those needed in specialized areas, like penetration testing. However, prematurely tossing your testers leads to service downtime, schedule slips, and staff, market, and revenue loss. (There’s a theme.)
• Delaying the debt. If your service is buggy or poorly architected, using DevOps is an awful experience. You’re constantly dealing with issues, hotfixes, and mitigations instead of sleeping or improving your service. Before you transition to DevOps, you’ve got to deal with your technical debt.
• Maintaining the monolith. Before using DevOps, a separate, large operations team might handle outages and deployments. Since the operations team would cover every aspect of the service, you could maintain one monolithic service instead of a collection of microservices. When you switch to DevOps, maintaining the monolith slows everyone down, reduces accountability and its associated improvement, and hides dependency complexity that exists but is left to fester.
• Unleashing the hordes. Many teams couple transitioning to DevOps with switching to microservices and scrapping their monolith. That’s a great change, but unleashing hordes of engineers will likely leave you with a big mess. Instead, think through the new high-level architecture and how to best manage changes to versions, interfaces, and schemas.

One step at a time

To avoid the common pitfalls and smoothly transition to DevOps, your team can take things a step at a time.

1. Reduce technical debt enough that it isn’t overwhelming. Delaying debt is a disaster for any product or team, but especially for DevOps teams. Get your scale, reliability, security, compliance, telemetry, monitoring, testing, and deployments working smoothly. They don’t need to work perfectly, but well enough that your service works as expected at least three out of five days a week without significant manual touch, including deployments. The tools to do this are now turnkey—it’s hard work, but much easier than even a few years ago.
2. Refactor big services into decoupled microservices, each owned by three to eight people. Some of your technical debt is best removed by refactoring. Other service aspects may be okay, but are hard and/or slow to change due to tight coupling. You must abandon the monolith and embrace microservices. That means rearchitecting your service and thinking through dependency management. This is tough to do, but the work needs doing even without DevOps if your service hopes to remain competitive.

Once your service is operating well much of the time, small teams running decoupled microservices can transition to DevOps smoothly. The change is still significant, and sometimes people on call won’t get a full night’s sleep, but it will be sustainable and feel like a big improvement.

3. Deliver and improve continuously based on customer feedback. Small teams working on decoupled microservices (each team typically owns a few of them) can now deploy those services separately and frequently. Collect aggregate customer data, and use it to continuously improve your services, continuously delivering value to your customers. You can even run experiments and test in production to ensure your services are the best they can be for real customers. There’s a reason why CI/CD (continuous integration/continuous delivery) is so popular.

Can we do that?

The three steps I described seem straightforward enough, but any team that has attempted them has run into issues. How do you convince management to devote resources toward reducing technical debt and refactoring services? How do you deal with the compliance overhead of so many microservices? How do you fund all these new microservices with the right people and teams?

Few managers will allocate time and people to reduce technical debt. However, they will allocate time and people to improving reliability and responsiveness to customers, which turns out to be the same thing. Likewise, refactoring isn’t sexy, but implementing new customer features without causing outages is very sexy and, again, covers the same territory. Go to management and advocate for the outcomes rather than the activity.

Compliance is a business essential and a fundamental requirement for many customers, including banks, hospitals, governments, and commerce. However, it does come with considerable overhead to ensure our customers know that their data and processes meet or exceed what’s expected. Most teams deal with the compliance overhead for microservices by grouping them into larger services that share compliance procedures and data. This approach reduces overhead and better corresponds to how customers and regulators perceive and experience the services.

Funding microservices appropriately might seem overwhelming. Say your one monolithic service had 33 people assigned to it. Now it’s been refactored into 22 microservices. That’s 1.5 people per microservice. Even if you assigned just two people per microservice, you’d still have to grow the team by a third—not happening. Instead, divide your team into six or seven small feature teams with roughly five people per team. Each team should own three or four related microservices, which is quite reasonable. You can even let folks choose which team to join, as I describe in I’m so lazy.

Good morning

DevOps is awesome, but switching prematurely is problematic. Since almost all software is now run like a service, you’ll want to take steps to ensure your transition to DevOps is smooth.

First, reduce your technical debt enough that your service works as expected at least three out of five days a week without significant manual touch, including deployments. Doing so will make your life better and improve your reliability and responsiveness to customers. Also, refactor your huge service into decoupled microservices that small teams of three to eight people can own and run. This restructuring will allow you to deliver more customer features faster without causing as many broad outages. Then, switch to DevOps and continuous deployment based on customer feedback. The change will dramatically improve your business results and remind you of why you got into software engineering in the first place: feeling the incredible satisfaction of seeing customers enjoy something you coded just a day or two earlier.

Avoid the common pitfalls and take a few practical steps. Change is hard but well worth the trouble to join the modern era of DevOps and get back to fulfilling your need for near-instant gratification.