Software engineering—what’s missing?

 To start the new year, my boss gave an all-hands speech to a large group of developers about being an engineer. He equated being an engineer with taking responsibility for quality and using methods that ensure high quality at checkin (Nailing the nominals). Naturally, a developer in the crowd took issue with calling software developers “engineers.” He claimed, “Engineers assemble parts out of tables and charts of proven components. Developers create something new and unproven every time.” Some disputes never get old.

Is software development engineering or craft? I’ve already argued that it’s pointless to argue. Developers must act like engineers or we all lose our jobs. No one bets his company or health on well-crafted software—it had better be engineered. As I claimed in A software odyssey—From craft to engineering, developers need to know instead of guess. They need to use instrumentation data from their products and engineering systems to guide decision making and drive quality and improvement.

But what about all those handy tables and charts “real” engineers use to assemble parts? Does software have an equivalent? Why do new language paradigms claim to solve the reuse problem, only to later fail? Can we be engineers without repositories and reuse? Hey, control yourself! We do have code warehouses, language paradigms are misguided, and we are engineers. Quit waving your rationalization crutches around, hang up your cowboy hat, and be a professional.

Eric Aside

I had an email conversation with the guy who asked the question. He was interested in setting up a repository, which got me thinking.

If at first you don’t succeed

Since the 1960s, there have been numerous attempts to establish software repositories and reuse. Many of the advances in object-oriented languages were accompanied by sincere claims that they finally solved the reuse problem, and soon software development would take on more of the character of other engineering disciplines. Unfortunately, those claims haven’t been realized.

At Microsoft, there have been many failed attempts to establish code repositories. The most recent internal attempt is CodeBox. It enjoys great success as an internal “open-source-like” collaborative development environment for tools, but it’s not a reusable code repository beyond copy/paste (a separate subject I address later).

Why have these attempts failed? Is there no hope? Oh, please. Folks have been thinking about the problem the wrong way. Imagining parts as class instances failed as an analogy—engineering reuse is quite alive and well in software.

I got what you need

There are two proven instances of code repositories and reuse that work and work well—libraries and design patterns.

Software libraries, including DLLs, interfaces, platforms (like .NET), SDKs, web services (mashups), and even JavaScript libraries like jQuery, have been wildly successful. They have achieved broad adoption, driven tremendous savings in development effort, and resulted in significant quality and security improvements to software systems.

Design patterns have also been highly successful, though not quite as broadly adopted as libraries. They provide proven design approaches for common engineering problems that result in more robust and reliable code that is easier to explain and understand. Design patterns are an old idea used in many other engineering disciplines. They aren’t reusable parts—they are approaches to common engineering challenges.

Something borrowed, something blue

There is a third, broadly utilized example of code reuse—copy/paste. Code “borrowed” from another engineer is prevalent in every significant codebase ever studied. Borrowing code responsibly can save you time and effort by using the best accumulated knowledge available. Borrowing code irresponsibly can cost you your career and your company millions of dollars. The specifics of who owns the code and how it is licensed are critical, but that’s a discussion for a different time.

Microsoft is extremely restrictive about borrowed code. Even internal efforts like CodeSearch are secured so that only people working on the same product can search their product code. The areas with the most code “sharing” are engineering tools and appropriately licensed code samples from SDKs.

Look, I love “sharing” and “borrowing.” My kids were taught these concepts in toddler group. “Borrowing” responsibly is part of being a professional, but it shouldn’t be confused with reusing documented, measured, tested, and published libraries and patterns from an established repository. Those proven repositories provide predictability and repeatability to engineering—the attributes that guy was seeking when he questioned whether or not developers are engineers.

A tremendous sense of isolation

Why do libraries and design patterns succeed where other repositories and reuse fail? I think the key is isolation.

• Libraries are isolated from the rest of the code. They form a separate logical layer that developers can program against almost as a black box. There’s no need for the vast majority of developers to change the library, and most never even see the library code. This makes for stable libraries and clear separation of responsibilities—quite sustainable.
• Design patterns are isolated from code by their very nature—they aren’t code. They are abstractions of code structure that defy fixed implementation, which also makes them stable and sustainable in their own way.

Note that while both libraries and design patterns provide abstraction, abstraction isn’t enough—you need isolation. If the developer ever has to mess with the repository for a typical project, then the isolation fails, and the repository becomes unstable and unsustainable for reuse. It’s a bummer, but that’s how things have gone over the past 50 years.

Eric Aside

Note that copy/paste succeeds as a reuse approach for the same reason—isolation. Copying code from another source does not impact the source in any way. This is a blessing and a curse. Should the original source have a serious error, such as a security vulnerability, that error is propagated. However, unlike libraries, a fix to the original code does not propagate to where the code was copied.

What are we doing here?

If software libraries are the only form of proven code repositories (like charts and tables for other engineering disciplines), does software engineering fall short of engineering? No, software development is just like every other form of engineering practice.

If you are an electrical engineer designing circuit boards, each new circuit board is unique. You determine what the circuit board’s inputs and outputs are and what it needs to do. You look through your tables and charts of circuit board components to find the ones that meet your specs. You then use common electrical design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the components on your new board.

If you are a mechanical engineer designing transmissions, each new transmission is unique. You determine what the transmission’s inputs and outputs are and what it needs to do. You look through your tables and charts of gears, shafts, and clutches to find the ones that meet your specs. You then use common mechanical design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the components in your new transmission.

If you are a software engineer designing an application or interface, each new piece of software is unique. You determine what the software’s inputs and outputs are and what it needs to do. You look through your online documentation of software libraries to find the ones that meet your specs. You then use common software design patterns, along with your own best judgment and creativity, to uniquely arrange and connect the software components in your new application or interface.

Living the dream

There are a few lessons to be learned about being a software engineer from analyzing software repositories and reuse.

• We can encourage reuse by designing well-isolated architectural layers, creating robust libraries and interfaces that reflect how developers use them, and communicating effective design patterns that propagate engineering expertise.
• Having great software repositories isn’t enough. We must follow other essential engineering practices, such as basing our decisions on data as well as experience, investigating applicable design patterns and libraries we can use, and taking responsibility for quality by having our designs and code peer reviewed, our code analyzed, and our units and components tested.
• Put those pieces together and there’s nothing missing from being an engineer other than demanding that minimum quality bar from each other and exceeding it ourselves as professionals.

You can be an engineer and you should be an engineer. Stand proud. Be strong. Win the hearts, minds, and well-earned capital of our customers.

Eric Aside

This is my 100^th column. I love that it’s about quality and engineering—two things I really care about.

Many thanks to all the readers who’ve spent a little of their valuable time reading my cathartic rants each month. Ranting to you sure beats talking to myself.