Metaprogramming

I periodically get calls from firms trying to place me into one of their various pre-determined slots. I know they want to hear “C++ Programmer” or “Java programmer.” But aside from those terms being silly they’re not actually reflective of what I do anyway. I do something a lot more specialized.

The best term for what I do is Metaprogramming.

I have mostly given up attempting to explain what that means to the placement firms who are trying to fill “C++ programmer” or “JEE” slots, but I do tell them “Well, think of me as a force multiplier.”

What is metaprogramming really? Wikipedia’s definition of metaprogramming works for me:

Metaprogramming is the writing of computer programs with the ability to treat programs as their data. It means that a program could be designed to read, generate, analyse and/or transform other programs, and even modify itself while running.

What is the military notion of force multiplication? Wikipedia again:

Force multiplication, in military usage, refers to an attribute or a combination of attributes which make a given force more effective than that same force would be without it. The expected size increase required to have the same effectiveness without that advantage is the multiplication factor. For example, if a certain technology like GPS enables a force to accomplish the same results of a force five times as large but without GPS, then the multiplier is five. Such estimates are used to justify an investment cost for force multipliers. A force multiplier refers to a factor that dramatically increases (hence “multiplies”) the effectiveness of an item or group.

How do metaprogramming and force multiplication work together?

In software development the primary cost is time. Not computer time; computers are cheap. Rather, it’s developer time — the time a human developer spends from when they receive a requirement until they deliver executable software. In business, they put dollar values on the time, but it’s still time that the money acquires.

And often, there’s a very limited set of people available to provide that time.

For example, a team with five developers has effectively 200 man-hours/week of time. Attempts to “get more time” by working tons of overtime eventually fail (where eventually may be as few as two or three calendar weeks).

A naive person might say “Add more people!” Great idea. Try to estimate the time it will take to advertise, receive resumes, interview, negotiate, and finally hire or contract a new person. Then add in the time it takes for that person to come up to speed on what is needed. And subtract the time from your team that is spent bringing that new person up to speed. It can easily be eight to twelve weeks before the new person is as productive as the other five. In that time, the effective team hours has dropped from 200 man-hours to 160 man-hours (because of the one doing training).

Well, if more people can’t be added, how about sending the work out?

Great idea, naive attempt number two!

Have the team halt, take all those “things we know” and convert them into formal requirements so precise that they can be used in court if needed, and then refactor the system to make “hooks” where the external development team’s code can be added later. There’s a dead stop for longer than eight to twelve weeks … and that finally allows you to start searching for an outside team to work.

When it’s not possible to add people or distribute the work, what options are left?

An analogy …

If you have 20 soldiers with swords and you replace those swords with M16A1 rifles, those 20 soldiers are now easily as powerful as 100 swordsmen, perhaps even 200 swordsman. In fact, given the 30 round clip of the M16A1, assuming 50% hits, each of the 20 soldiers will hit 15 swordsman from too far away for the swordsmen to attack. That means 20*15=300 swordsmen disabled by your 20 rifle-wielding troops.

That rifle is a “force multiplier.” You have the same people, but they are now 15 times more deadly (probably more, but it’s just an example after all).

Well, we’re not going to issue developers with rifles to “shoot bugs” and “dominate the requirements.” Not that it wouldn’t be amusing to shoot bugs …

Instead, what a metaprogrammer does is create tools, libraries, and frameworks that let an existing programmer do more in less time.

When the tools mainstream, people forget that in every case the tool was a result of metaprogramming.

Some famous metaprogramming tools:

  • assemblers
  • compilers
  • code generators

When a code-generator reaches critical mass people forget that’s what it is.

Many modern languages such as Java have “annotations” that allow the code to have meaning at runtime as well as tools such as APT that eventually became part of the language and toolset directly. These all support metaprogramming.

It’s routine to have SQL create statements generated — clear metaprogramming at work. Hibernate uses annotations extensively for this.

But what does that mean to our beleaguered team of five developers? They are probably already using all the mainstream tools that can help them. Is there no hope?

Task one of the five with creating new tools just for their problem that offloads some of the burden.

How do they know what to create? Look at:

  • where most of the bugs are
  • where most of the copy/paste/edit is done
  • where the requirements have copy/paste/edit in their text

Here is where craft comes in, though.

It’s been pretty consistent on all projects I’ve worked with that roughly 80% of all code can be code-generated.

This means that most projects have five people doing work that one could be doing if that one had good tools. I’ve had people try to argue the point, but even in my last major project I provided a combination of code-generation and libraries that reduced the next version of an application more than 80% and the resulting program was not only smaller but trivially easy to test.

It’s not enough to have great general purpose tools and languages. Not if you want true force multiplication. You have to step beyond the comfort zone and create tools (and create languages!) that let your project’s team radically exceed the expectations of everyone.

So that, in a nutshell, is what I actually do for most of my clients. I help them turn one person into five.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s