Productivity gains in software engineering are powering innovation

Engineers are the best deal – so stock up on them

Everyone is more productive these days. This has been a consistent trend for at least the past decade, where productivity gains have been particularly strong within the business sector.  According to data from the U.S. Bureau of Labor Statistics, today’s business industry workers are on average 30% more productive than their 1998 counterparts (productivity growth of roughly 2.6% per year).

Growth chart  
(Source: U.S. Bureau of Labor Statistics)

Within the technology industry, productivity has increased more.  Thanks to smartphones, improved search engines, better CRM software, and ever-increasing bandwidth, salesmen and marketers can find, receive and process information faster than ever. 

The most dramatic gains, however, have occurred within software development.

Software engineers today are about 200-400% more productive than software engineers were 10 years ago because of open source software, better programming tools, common libraries, easier access to information, better education, and other factors. This means that one engineer today can do what 3-5 people did in 1999!

3589792404_65192f9038 The advent of open source software makes engineers particularly efficient.  One VP Engineering that I talked to gave me an anecdote about one module where they used open source files with about 500,000 lines of code and then wrote 7,000 lines of code to stitch it all together.  Open source software is also free.  In the company I was running in 1999, “software” was a huge budget line item – we had to buy databases, testing suites, libraries, and more.  Today all that stuff is free … a start-up might spend more money on sodas for the office than it does on software. 

We’re all familiar with Moore’s Law – that the power of computers doubles every 18 months.  In my 15 years of software development, I’ve seen 5x-10x productivity gains in engineers.  Which could mean that the productivity of a well-trained engineer doubles every five years.  (note that this Law is much harder to prove than Moore’s Law – but potentially just as profound).  That would mean that the productivity of an engineer is growing at roughly 14.9% per year!  That’s fast … really fast … much faster than the 2.6% yearly gains the population as a whole is making. 

This means that today’s companies are able to do more software engineering and build more stuff with fewer people.  But should they do more with less? It could be much more prudent for a company, especially for a small company, to do the opposite … and to double-down on engineering.  You can use the productivity gains in software development as a strategic advantage and invest aggressively in engineers. First, doing so contributes the most to progress and also increases the chance for breakthroughs in innovation. Second, engineers – as opposed to salesmen and marketers – can often hit the ground running (assuming you have a good on-boarding system) and have a positive impact within a few weeks. 

Alternatively, many large traditional companies might be able to get by with FEWER but DIFFERENT engineers.  These companies might need to change their approach to engineering to take advantage of the new tools.  The companies that can benefit from fewer engineers are likely ones that haven’t changed their technology platforms radically in the last ten years.

Although engineers contribute more to an organization than ever before, their pay – relative to other functions in a company – hasn’t followed suit.  I’ve polled a few dozen companies and have found that over the last ten years, an engineer’s pay has held the same relative salary to marketing and sales.  This is odd behavior … usually when something outputs more, its cost goes up.  So why have engineers’ wages in the U.S. stayed constant relative to salespeople and marketers?  Here are two contributing factors that lower demand:

1.    Off-shoring.  Because of new technology and higher bandwidth, more companies are off-shoring their software development.  But this does not fully explain the flat salary phenomenon since firms are also off-shoring sales and marketing (though to a lesser degree).

2.    Need for software engineers has decreased.  Because software engineers are so much more productive than they were ten years ago, many firms are opting to hire fewer of them.  If a company is not doing hard-core engineering, it actually needs fewer engineers as a portion of its total workforce than it did ten years ago.  (I personally think this could be a big mistake … but I will get to that later). 

Both the off-shoring and the decreased need for engineers has led to a lowering of the demand which has likely put a check on wages. 
 
One problem, of course, is that measuring “output” of an engineer is a really hard thing to do (as opposed to the output of a salesperson) … so it is really hard to quantify the productivity gains.  And even if you can measure output in engineering, it is sometimes hard to tie that to an increase in profitability.

And, like sales, the quality of engineers varies wildly.  A great engineer is potentially 2-4 times more productive than a good engineer.  Ben Ling from Google pointed out to me that some great engineers are massively compensated – because they tend to be the early hires at a company and get lots of stock (most of Google’s first 50 employees were engineers). 

Let’s recap:  The productivity of a software engineer has increased 2-3 times that of a marketing person in the last ten years.  Yet their relative compensation has remained about the same.  That means if you are a savvy company, you should stock up on engineers.  In fact, you would want as many great engineers as you can get a hold of. 

This engineering productivity boom will only increase and continue to create dislocation and creative destruction.  While the extent of growth and industry makeover are hard to gauge, what is certain is that corporations relying on technology and engineering paradigms from the 1990s or before will find themselves hard-pressed to compete with the new and nimble movers. 

(special thanks to Jonathan Hoffman, Michael Hsu, Ben Ling, Jeremy Lizt, Naghi Prasad, and Dave Selinger for their feedback and edits).

9 thoughts on “Productivity gains in software engineering are powering innovation

  1. niallkennedy

    I think the market friction isn’t just between onshore and offshore, but an extreme bias towards locality. Businesses like to work with developers in their own back yard, who they feel are on-call at any particular time. A new trend is local engineering contractors fulfilling orders through their own offshore practices. Businesses enjoy the competitive bidding structure while trusting a local resource for talent discovery, training, and QA.
    It would be interesting if there was less transaction friction in the engineering space. The change in hourly rate in long-lived open source expert consulting (e.g. MySQL, Sun, Percona) might be a good meter.

    Reply
  2. Jonathan Kibera

    To link the “off-shore” note with Gladwell’s latest “10,000 hour” expert rule: I think US/Bay Area devs have 10K hours now developing in the perfect cauldron, which means they now make much better decisions than, say, 1999. I don’t think off-shore Engineers have 10K hours in the the same cauldron – much less effective.

    Reply
  3. Gary Cook

    One small point if I may. It may be that software engineers are much more productive than in the past (and you make a convincing case in that regard). However, that does not automatically mean that you should “stock-up” on them because their productivity has increased “2-3 times more than a marketing person over the past ten years.”
    In fact, it may be that engineers still aren’t as productive as they should be, relative to marketing people. (or other people, say in sales, for that matter) In fact, given the huge over-runs in cost and time suffered by many software projects in the late 1990s, one could argue that engineers were vastly overpaid for what they did then. Perhaps their productivity is just catching up to what it should be?
    I’m not arguing that in fact engineers are only now being paid what they are truly worth (although that is a conclusion you could draw from the data in your note), but rather that it seems as easy to draw the conclusion above as it is to draw the conclusion you came to.

    Reply
  4. Kyosti K

    Yes, we all know that there are huge differences in developer productivity. The problem is that there’s no scientific method to measure this, mainly because of the inherent complexity of software.
    What was your method of inquiry? Can you show measurable evidence? How many data points you had? What was the reasoning you used? How did you formulate and test the hypotheses?
    The other reason, why measuring developer productivity is impossible, is the significant impact of psychological phenomena (perception, cognition, attention, emotion, motivation, personality, behavior, interpersonal relationships and the unconscious mind) in knowledge-based team work (software development is always teamwork).
    Some tips for further reading:
    http://martinfowler.com/bliki/CannotMeasureProductivity.html
    http://www.usc.edu/dept/ATRIUM/Papers/Software_Productivity.html

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s