Software development: the 80-20 rule

At one place I worked, a high-up guy once remarked how he was puzzled how there could be such wide divergence in the level of effort software developers make. He stated it something like the following:

I look around and I see some people working really, really, hard—putting in extra hours and so on. Then I look around and see some people working not very hard. And I wonder why everyone isn't working hard. If those in the second group worked as hard as those in the first group, then we'd be getting a lot more done.

This guy wasn't the big cheese, but he was up there a good ways on the ladder, so I'm unsure whether his comment was merely an attempt to motivate us or if he really was puzzled about his employees' work ethics. I'm not puzzled why it's in some developers' self-interest to slack-off (because I've been in that position a few times). I've even formulated my own 80-20 rule to explain it.

In a sufficiently large software project, 80% of the developers produce about 20 hours of work, on average, each week.

Those words, “sufficiently large”, can cover a lot, but I mean them in the way that you'd probably expect—a project that's too complex to be understood fully by any one person and where (probably) not all developers know all other developers—at least, not well. In other words, I'm talking about a software project with a non-trivial social hierarchy.

As to why the phenomenon of the 80-20 occurs, here's my theory: Developers show up to work because of the paycheck, but they work hard only when they're mentally rewarded. Stated another way, a paycheck buys compliance but creative ownership earns actual productivity.

Small software projects can hand out the juicy, fun, creative problems fairly evenly among developers, but somehow this doesn't happen when there's a social hierarchy. When not everyone knows everyone else, when not everyone trusts everyone else, there exists perceived incentive to keep those juicy bits for oneself, and somehow, in sufficiently large software projects, the Keepers of the Juicy Bits self-organize to become about 20% of the project's staff. Everyone else is just going through the motions, solving routine problems at an easy-does-it pace and baffling the higher ups with their casual attitudes.

I don't have a solution to the 80-20 rule, and the higher ups may endlessly continue to puzzle why they can't get their developers to care as much as they do. But from the individual's perspective, the 80-20 rule suggests avoiding becoming lost somewhere in the 80%.

Aimful

Maybe the biggest reason I'm quitting my job is that I want to make sure I'm still alive.

The last time I was unemployed was nine years ago. I was twenty-three years old and one year out of college, and I had just been laid off from the startup where I had done unremarkable work. Like many post-college kids at that age, I mistook laziness for existential crisis, and unemployment fit me well in that way.

I soon moved back to Houston to live with my parents, and my chief responsibility that spring and early summer was chauffeuring my dad while he recovered from complications from cataract surgery. My other duties included playing a lot of Scrabble—with good humor Dad blamed his losses on his semi-blindness—and getting in a few good bike rides on the open roads west of town. I also really really intended to get around to doing something about one of my many loose ideas for a software project.

But I never got around to doing much of anything productive in the four months I was unemployed—a fact I later learned to regret once I found a job and lost my free time. Regret has since morphed into a fear that I can't be productive when unemployed—that I must be anchored to a fixed schedule to avoid being aimless. I described this fear to some Phoenix friends, and many of them were surprised because I'm regimented and focused. Yes, but you've never see me unchained from the nine-to-five.

What I mainly remember from those idle days nine years ago was that I enjoyed not doing anything, though there are obvious money issues that go along. I remember being asked by friends and relatives about my job search and being reminded each time that I was suppose to feel sad about my situation. What's sad is feeling sad about not working for someone else.

These days I have no shortage of interests: my bike rack project, teaching myself basic chemistry, learning Go and AJAX and git and other software technologies, becoming a better writer, and so on. I don't know whether I've developed so many interests because I haven't had time to pursue them all or because I've matured and learned the value of hard work. That's the lesson taught in Candide—that we should all tend our garden. I want to believe that I've learned that lesson and I'm ready to tend my garden awhile—now that I won't be tending anyone else's this summer.

I quit, again

Today at work I gave my two weeks' notice. As now planned, my last day will be Friday next week, thus bringing my stay with Astronautics to a total of four years and eight months, the longest I've ever held one job. In answer to the questions that logically follow: No, I don't have a new job lined up, and no, I haven't even begun looking.

Happy employees don't quit without good reason, such as getting a better offer elsewhere or wanting to move to a different city. I could say that my reason is that I'll be spending the summer with Laura, traveling and otherwise taking advantage of what may be the only time we'll ever be able to afford having extended time off from work at the same time. But this reason doesn't capture it fully. If I had wanted to stay at my job while taking off two months, I may have been able to work out something with management. I didn't even try.

I'm thirty-two years old, and despite all of Laura's jokes about me being an old man, the truth is that I'm too young to continue grinding it out at a place where I'm not learning a whole lot. I feel that's my situation at Astronautics—at least when considering the time I'm being asked to put into it. When I'm sixty-two I might be able to get by on what I already know, but at thirty-two, not growing is a slow death, however comfortable and materially rewarding it may be. This second reason won't be easy to describe to the countless people who ask me in the next few months why I quit my job, but it's closer to being the real reason.

Agent Smith Visions

Not to beat it to death, but there's a bigger problem with Isaac Asimov's Malthusian reasoning in Robot Visions, one that I didn't write about on Monday but is worth bringing up because it's a common belief, and it's wrong. I think of it as Agent Smith's fallacy.

There are many reasons not to like the movie The Matrix: Using humans as power sources doesn't jibe well with the law of conservation of energy, for one, and the hand-to-hand combat scenes are silly. But The Matrix is a fun popcorn flick—despite Agent Smith's humanity-is-a-virus speech.

I'd like to share a revelation that I've had during my time here. It came to me when I tried to classify your species and I realized that you're not actually mammals. Every mammal on this planet instinctively develops a natural equilibrium with the surrounding environment but you humans do not. You move to an area and you multiply and multiply until every natural resource is consumed and the only way you can survive is to spread to another area. There is another organism on this planet that follows the same pattern. Do you know what it is? A virus. Human beings are a disease, a cancer of this planet. You're a plague and we are the cure.

Natural equilibrium? This is wrong. Do lions go on hunger strike when their numbers swell too large? Do deer choose not to breed so that they don't have too much impact on their environment? Do bacteria have any instinct at all? There is no such thing as an instinctual will to equilibrium. Every species is doing the best it can to increase its population; they just happen to fail most of the time and instead maintain a stable population. Humans happen to be increasing their numbers right now.

Many people—not just Asimov—think something similar to what Agent Smith thinks. Humans consume too much. Humans destroy too much. Humans are evil. There's a tacit assumption that the misfortunes humanity creates for itself are done so through deliberate choice, whereas other species on the planet are mindlessly behaving according to harmonious instinct. What evidence do we have for this distinction between us and other species? This is a flimsy proposition, and we wouldn't believe it if we didn't want to believe it.

There are many reasons to project sin onto humanity—some of them even useful—but the two reasons that stand out to me are: (1) you call evil only that which has power over you—in this case, humanity and the inhibitive social order it imposes over individuals— and (2) the alternative explanation—that no one is in control—is scarier than believing we're evil. For if we're not in control, then we won't ever choose our equilibrium with the environment, but instead we'll have that equilibrium “chosen” for us.

Exponents are hard

It's the year 2030, and a group of scientists figure out time travel. Due to the particulars of how time travel works, they're unable to travel backwards in time and thus cannot change the past, but they're able to travel into the future and return to the present.

The scientists decide to run an experiment to send someone 200 years into the future to determine whether humanity survives the threats of overpopulation and poverty. They send a robot because they agree it's too dangerous for one of themselves to go. Archie, the robot they choose, sits in his time-traveling contraption and blinks in and out before the scientists' eyes. Though taking no time by the scientists' perceptions, Archie reports that he spent five years 200 years in the future.

All is well with humanity, Archie says. In the future, poverty has been solved for all, and the world's population has been brought down to one billion from the ten billion living in 2030. Furthermore, humans are thriving in bases on the Moon, Mars, and the asteroid belt. In short, the future is magnificent.

—So Asimov wrote over twenty years ago in one of his last robot stories, Robot Visions. The story continues with its unnamed narrator not believing Archie's report and not sharing the scientists' relief of humanity's good fortune. As the narrator figures:

I asked myself if population decreased from ten billion to one billion in the course of two centuries, why did it not decrease from ten billion to zero? There would be so little difference between the two alternatives.

Asimov was a smart guy, but how could he be so wrong about this point? As anyone knows who's had an infestation of cockroaches, ants, or any other pest, there's a big difference between most dying and all dying. In this particular case, for a population of ten billion to reduce to one billion over 200 years requires an annual growth rate of about -1%. That's barely negative and is the consequence of small changes in the mortality rate and the birth rate. Today in 2011, some industrialized countries, such as Japan, already have negative growth rates due only to a decrease in the birth rate.

Whereas, for a population to go extinct over 200 years, the annual growth rate must be about -10%. That's a big difference—like suffering an epidemic every year for two centuries. Or suffering continuous non-stop war as destructive as the world wars. While a -1% rate is historically precedented, a continued rate of -10% is not. Surely, Asimov knew this. I guess you ignore a few details when you average 1,700 published words per day for 40 years.

Where's the Monday Post?

The Monday Post was out sick with a high fever, sore throat, infected sinus, and extreme fatigue. The Thursday Post will be out of town to attend a wedding. But Next Monday's Post will be here as usual.

<title>I don't know SQL either.</title>

Often I wake up Monday or Thursday morning with no more than a notion of what I'm going to blog for the day. Sometimes it's easy to transform that notion into words, sometimes it's hard. Some days I begin without even a notion; today is one of those days.

It's also my 210th post. By now I don't remember all of what I've written about here at JEC. This is problematic because sometimes I wake up Monday or Thursday morning with a notion, begin transforming it into words, and then realize that I've already written a post about that exact thing. That's embarrassing and time-consuming—not to mention it makes me wonder about my chances of avoiding senility. But it's the wasting of time that stings right now, and I've made this mistake enough times to consider it a problem worth solving. Because I don't have an Internet connection at home—not one that I pay for—I can't count on Blogger's search box being available whenever I need it. Instead, I'm working to archive my posts locally, offline.

Blogger has a feature for doing this. It allows you to download your blog as a massive XML file containing all posts, reader comments, and some settings junk. If you're looking for a machine-readable copy of your blog, this is perfect. However, I want a human-readable copy, and that requires some tinkering. The solution I'm working towards is to hand-craft the XML back into separate HTML pages—that's 210 pages and counting. I reason that this solves having a local copy while presenting me with an opportunity to learn HTML. That's right; I don't know HTML.

Sure, I've written a few simple CGI websites, so I know enough of the basics to get by. But I've never used, for example, cascading style sheets or JavaScript, and only recently did I learn <em> is the appropriate tag for making text italic, not <i>. By hand-crafting my JEC pages offline, I expect to fast-forward my HTML knowledge 12 years to the 21st century. This might be a good fallback in case my career in embedded software development doesn't pan out.

As to whether any of the knowledge I acquire will propagate up to the online JEC, that remains to be seen.

Chemical hate

My hatred of chemistry began in the 10th grade, only a month or so into the first and only chemistry class I've ever taken. I still resent being made to memorize that long list of ions. Everything I was suppose to learn in the class after that assignment was noise that I retained just long enough to earn a B grade. Sadly, the only thing I learned in high school chemistry is that I hate chemistry. As an adult I'm confronted with my teenage apathy every time I use any of the mysterious chemicals I store beneath my kitchen sink without understanding how or why they work.

Now that I'm older, I realize that I don't hate chemistry; I just hate the particular way it was taught to me. It's unfortunate that high school science is taught as though every student is going to be a scientist. Professional chemists need to know their ions without taking the time to consult a textbook, but the rest of us don't. While it's good that we all spend at least a little time learning how to balance equations, what we non-scientists need most is a broad, qualitative foundation sufficient for allowing us to teach ourselves the little science we need or want as adults.

Awhile back I bought a used chemistry textbook at Goodwill for a few dollars. It targets liberal arts students who take chemistry as their science elective. “Perfect,” I thought. “This is exactly what I need to understand those chemicals beneath my sink.” If a liberal arts student can understand the book then so can I.

Two weeks ago I began my chemistry self-study program—my attempt to undo the damage I did to my own brain seventeen years ago. I'm even using pencil and paper to answer the review questions and problems at the end of the chapters. Hopefully I'll teach myself that I like chemistry after all.