It’s easy to build software today.

Pick a stack, which gives you a framework from web browser through server. Pick the plugins that hook to your preferred SQL or No-SQL persistence using the mapping technology so you don’t have to code storage directly. Use Docker for each microservice and have them expose web services and setup Kafka for the queueing, with Redis for the shared state. Control it all via Kubernetes and deploy it through AWS.

I’ve been told this, by more than one person and team. “It’s easy.” I’ve never seen any of the teams who claimed it was easy ship.

And yet, everything I wrote above is true. There really are full starting points. There are excellent technologies. I’ve written before about the challenges of attempting to build with enterprise stacks before the problem is well understood, but let’s grant that the problem is understood.

Why, then, is it still so hard to ship software that people can use (and harder to ship software that they like to use)?

Why Is It Not Easy To Build Cars?

Let’s take a step out of software for a moment.

Cars have been around for well over one hundred years. There are auto parts stores all over the US (often across the street from each other, which always amazes me). There are (or were before Covid) thousands of used cars to pick up as a starting point.

We can buy engines, transmissions, wheels, used cars for frames. The entire “stack” in software terms is there.

And yet, building a car (or restoring a car) requires a huge amount of work.

New wiring is often needed, which requires disassembly of (often) everything. Just re-wiring (even if starting with a new wiring harness) is hours and hours of detailed work. It’s easy to make a bad connection or worse a short. Unlike software, there’s no undo.

Then there’s body work, engine mounting, transmission mounting, connecting engine and transmission, which might need brackets and adapter kits. Sometimes, a new engine has to be mounted higher or lower, and that requires changing where the air filter goes. And so on. Any experienced mechanic can make a list far larger than mine.

But, why is this a big deal, when all the parts are there?

It’s the pragmatics

What Are Pragmatics?

Now here’s a case where hitting a dictionary doesn’t help!

The first definition of pragmatics (from Dictionary.com) is:

1. Logic, Philosophy. the branch of semiotics dealing with the causal and other relations between words, expressions, or symbols and their users.

That’s not helpful (though in effect, it’s right for reasons I’ll explain below).

The second is about linguistics which I don’t even want to get into.

The third is what I’m talking about: practical considerations.

In effect, the pragmatics are the details that allow us to use parts together. The “relations” between parts. Not words, expressions, or symbols in our case, but software reusable elements.

An Example of a Pragmatic

All production systems require excellent logging. Entire companies exist to help make sense of logs. Frankly, lack of excellent logging is one of the reasons that software remains buggy for so long.

And it turns out, logging (even bad logging) is inconvenient for developers. Excellent logging is not only inconvenient, but very hard.

In the simple case, a logged event should cause a record that allows the consumer of the log (normally a developer or operator) to determine what happened. Most logged events still tend to be text blocks dumped into a stream. Even when different techniques are chosen (such as logging to event monitors) the problems are the same, so I’m going to stick with a simple text stream log for the example.

So, there exists a place to send log messages. In Java, using Log4J (yes, the one that was involved in a major security faux-pas) that can be as simple as this:

log.debug("I've reached magic point X in the code");

I’m ignoring how to get the log object etc. Doesn’t matter. That above log simply indicates that some point was reached. It’s a common sort of message.

More useful of course is to log details:

log.debug("Var x = " + x);

This of course is awful for performance reasons (though easy to write and common).

Newer versions of log4j can do this:

log.debug("Var x = {}", x);

That has the benefit of not generating garbage if it doesn’t need to. Some people will do this:

if(log.isDebugEnabled()) log.debug("Var x = {}", x);

That skips the call, skipping any work if not needed. Of course, inside the debug() call itself it checks to see if it’s needed, so that is superfluous and may slow down the code because it calls the check for debug enabled twice. MAY. Depends on the logger implementation. The isDebugEnabled() is extremely useful when more is done in debug than debug logging (for instance, a debug version may render a very different and expense string representation of a complex object tree, more so than the default toString() method).

There are also pragmatics on when to use debug, trace, info, warn, etc. There are pragmatics on how to configure the logger. There are pragmatics on how to rotate logs.

Basically, something as “simple” as emitting log data actually requires thought and attention per usage and it’s common to have multiple log statements per method.

Conceptually, “log what’s happening” is easy. In reality, the pragmatics require effort and attention.

Why Tooling Doesn’t Help Solve This

There are many attempts to simplify (or, more properly, “make easier”) the pragmatics.

For instance, it would be child’s play to mechanically insert logging trace messages at the start and exit of each method in the system. That could be done via a build tool, via aspect oriented tools, etc. But, ending up with a full trace is overwhelming to the person attempting to make sense of broken code. It would log too much.

And here is the fundamental problem: tools only make things easier when they’re used with attention per usage. In other words, the more tools, the greater the skill needed to take advantage of them.

This is very easy to see with interactive debuggers.

I rarely use interactive debuggers. It’s not a religious issue (“Oh, you use debuggers! REAL programmers use <insert some other method here>!”). It’s pragmatic. Almost everything I do tends to be distributed, asynchronous, and often threaded. The debuggers thus cause effects in the system under test that easily mask problems. I’m sure I’m not the first person who had code that only worked under a debugger, but never in production. The failure wasn’t the debugger, or my use of the debugger. In fact, here is an example of stripped-down code that works under the debugger and fails in production:

Config config("a");
launch(config);
config.set("b");
launch(config);'

It’s a very stupid beginner bug. The system fails because two ‘b’ instances run and no ‘a’ instances. Under the debugger, though, it works perfectly. If you don’t recognize the bug, feel free to ask. Someone will point it out >:)

To understand their programs new developers are often shown how to step through code with their debugger. This can help gain an “intuition” of what’s happening, see the local variable assignments, and so forth. When the programmer is doing single-threaded code without any timing or IO concerns (normal for a beginner’s first programs!) what they see under the debugger is honestly representative.

Now, step them into code deep in the guts of a modern distributed framework pulling messages from Kafka and … yeah. The debuggers can absolutely handle it. But people need to learn about the multi-threaded features of the debuggers and those are not as easy as “step next.”

The tool, thus, requires more experience in order to be used. That’s why tooling makes things easier for those who know enough that they don’t need the tool (I can absolutely use multithreaded debuggers but I can easily use logger techniques) but often makes things harder for those people who desperately need what the tools do but they don’t know how to use the tool to do it.

Why More Libraries and Frameworks Doesn’t Help Solve This

I used log4j as an example (or self4j, it’s really hard to tell since I put so little in the example and it doesn’t really matter). It’s a library. I could have used a Spring example (a framework). Today, the line between framework and library is getting fuzzy anyway … but I’ll still refer to both.

What the reusable code provides is amazing. But, it requires a detailed understanding of what it provides. Notice I said “what” and not “how.”

I don’t have to care how a logging library or a communication framework is implemented (in reality, I do have to care, because they expose me to security risks, dependency risks, resource risks, etc., but we’re not supposed to admit any of those and even ignoring them doesn’t harm my point).

I will stipulate that every bit of reusable code from the internet works perfectly as designed and is 100% bug free. We know this to be false, but it doesn’t matter.

In order to use external code the developer using it must know how to use it (what it offers) and when. What is the order in which things need to be called to work as intended? How do the objects returned from calls into the code need to be cleaned up? Are they thread safe? Do they expire? When does the code need to be initialized?

My examples in log4j all assumed that somewhere in the code the log object was created, initialized, and that there were configuration files setup and in the right places to drive the logger. Those are the pragmatics for that bit of code. If Maven or Gradle are used, each of those requires configuration along with the logging configuration.

So all external code increases the amount of attention and understanding needed at each point of usage.

Elsewhere, I pointed out that well written code has very little code exposed to external code. This is why that’s important. Exposing less external code reduces the cognitive load on the developer reading (and understanding) a block of code.

Does this mean it’s wrong to use all these external dependencies? Sometimes, yes, it is in fact wrong.

I’ve been on projects where I used Java reflection and was told (by an exceptional programmer) that I should have used the Apache reflections library. I had about 12 lines of code that used the Java reflection API raw. I looked at the Apache library. Nice library, very full featured, and well documented. Using it would have taken almost twice as many lines of code and drawn in a dependency to replace what I did.

Every dependency increases the pragmatics that must be learned. Even assuming everything about the library is perfect, that doesn’t (and can’t) change. The more dependencies in any block of code, the more pragmatics must be addressed in that block to understand it, write it, and most of all, maintain it.

What Reduces the Pragmatics?

If pragmatics are a side-effect of code we didn’t write, then it would sound like doing everything by hand from scratch would reduce pragmatics.

It doesn’t … because each thing we build by hand from scratch has to be understood by the parts of the code that call it. That we wrote it doesn’t change that it has expectations. And once we’ve written a lot of code in a project, we’re not going to remember our own code’s expectations. It may as well be foreign.

Doing everything yourself by hand is not a good solution (thankfully — I’ve DONE systems by hand solo with a mix of languages including assembler and I don’t want to go back there).

The only realistic way to reduce pragmatics is through common usage — via abstraction. This is really clear in what Steve Jobs and Apple did when they promoted the original Macintosh computer. Every UX aspect always meant the same thing and worked the same way if the program was done “right.”

That isn’t a tool, or library. It’s a commitment to commonality and consistency. Not just in the code, but in the definitions, declarations, and the usages.

It’s very hard. Worse, people feel “stifled.” Even basic things, like indentation levels, can cause friction within a team. “We should have tabs mean two spaces.” “We should have tabs mean four spaces.” “We should run with no-tab and have only spaces.”

Bless Python. Its use of whitespace for scoping makes it so damned nice to read. It reads well. I can understand Python code more easily skimming it than almost any other language precisely because it enforces common structure.

The opposite (by design) is Perl. Perl lets the developer do anything in any one of a zilion different ways. Even for a very skilled Perl developer reading someone else’s code that uses different standard metaphors is tricky. Worse, I’ve got a Perl certification, and I realized when preparing for the exam that the people who wrote the exam (and the exam preparation guide) didn’t know Perl.

I was so confused by the guide that I took the time to check the Perl official docs and test every claim in the guide in a Perl interpreter. They were (literally) wrong. When I took the exam, I had to figure out the “wrong answer” for how Perl works that they “believed” was the right answer.

I sent a very long (must have been five or six thousand words) email detailing the errors in their preparation guide with code-captures from my having run the examples to prove my point. That exam is no longer used.

If an entire certification organization with international customers can’t figure out enough to make a cert exam, imagine the challenge for programmers picking up someone else’s distributed Perl code-base! (Actually, I don’t have to imagine it, I’ve taken over Perl systems before — I don’t bid such gigs anymore).

Conclusion

I wish I had a silver bullet, but I don’t.

In the end, the only way to achieve the commonality of abstractions and usages is programmer discipline. Good programmers put the attention into each method, class, file, etc. Not only do they put the attention into their code, but they also document their common usages so later developers can understand them.

It’s easy to “bang out” something that works. It’s not easy to have something that can be understood over time by different people.

Failure in discipline is shown by cries to “re-write it!” and success in discipline goes unnoticed with the code working for years.

It’s almost funny: COBOL gets mocked ruthlessly, but COBOL systems have lasted for decades and most “modern” systems seem to be lucky to deploy let alone last for two years.

Almost funny … or perhaps very sad. I wonder what is being lost from those “old fashioned” developers who could build code that withstood the test of time as they retire?

Oh, well, I’m sure that all those COBOL systems will be easily replaced with modern stacks. Right?

Right?

Keep the Light!