Search This Blog


Osteoclasts vs. Osteoblasts

I used to feel as this carmudgeon developer, somewhat angry at all these keyboard monkeys who can’t or won’t write decent code. And then I found myself working with such “keyboard monkeys1”, perhaps even being one myself. At least à la the current carmudgeon developer2.

Being on different side of the fence made me realise that sometimes we must stir things up to make progress. Even at the cost of “breaking stuff”.

Nowadays, instead of an old and wise carmudgeon vs reckless keyboard monkeys, I tend to think of the situation as more of a “osteoblasts vs. osteoclasts”, and actually think this is a healthy situation for a software organization to be in3.

what are “osteoblasts” and “osteoclasts” anyway?

osteoclasts vs. osteoblasts search results summary.

tl;dr: osteoblasts build bone tissue while osteoclasts destroy it.

Where am I going with this analogy?

When software gets enough mileage and accumulate enough “tech debt4”, it tends to become more “rigid”. It gets hard to make changes, introduce new features, etc’…

Similarly, our bones gets more rigid as we grow old. A child’s bones are more flexible and tend to bend or “bow” instead of breaking. An adult’s bones are harder and more brittle. The way our bones grow, is largely thanks to the work of osteoblasts and osteoclasts in our body. To help bones grow as they become more and more rigid, the osteoclasts dissolve and damage the bones, allowing the osteoblasts to mend the bones with new and better bone tissue.

Back to software developement: The “reckless” works of the young and darring might anger the old and wise who feel they need to “clean up” after the monkeys. But if we don’t have our reckless osteoclasts to break the software in the right places, we won’t be able to make important adjustments to our software. We need the old and wise menders to make sure everything is working properly. We also need some recklessness and darring to “get shit done”.

It is a delicate balance, and each organization has a different “right” ratio of preservation vs. innovation oriented developers.


  1. Not really. They are all awesome people and great developers. You’ll see where I’m heading with this soon enough.↩︎

  2. Brilliant fellow TBH. Had been around from the product’s early days, and had very deep and vast knowledge of the system.↩︎

  3. Assuming you have the “right” balance of old and wise carmudgeons to reckless keyboard monkeys ratio.↩︎

  4. I actually don’t like that term. I prefer “software rot” over “tech debt” - but that’s an issue for another post.↩︎

Balance your team skills RPG style

All developers1 feel they are “above average”.
That can’t be true, right?
Or maybe it can?

Well, depends on how you measure.
A while ago I recruited a new team, and in this post I want to share some of my newly acquired insights.

The analogy

Remember those role playing games of the nineties? You had to form a party of heroes for a quest. Where for each chosen character, you could also adjust their stats. Strengthening some of the character’s attributes at the expance of other attributes.

pool of radiance - ruins of myth drannor: character creation; assign ability scores screen

It was not a good strategy to have all your heroes optimize for the same attribute. Enhancing only strength, for example, and neglecting wisdom, dexterity, etc’… would not get you far in the game.

pool of radiance - ruins of myth drannor: character creation; select party screen

It’s also a bad strategy to average all your characters on all attributes, making them mediocre at everything. Diversity was key to advance in these games.

Same goes for engineering teams.
Some people have more of a “hacker mindset”, some are “builders”, other “menders”, or “enablers”, and so on…

And like in these games, where some missions can only be passed with a combination of (for example):
Characters that are very strong and specializes in melee attacks, together with characters that are agile & accurate and specializes in range attacks. We need to assemble an engineering team with different types of “experts”. And much like the games, not all expertises are essential to all tasks.

Optimize your team for the job at hand

When building a new team, ask yourself questions like:
Does the goal leans more towards exploration, where fast feedback & market validation are key to success? Or more towards long term stability and maintainability in a well established product?

Obviously, the perfect team for the job would look different based on the goal. If I want to explore, release POCs, demos and experiment fast, I would lean more towards a “hacker mindset” team that works in a “quick & dirty” style (hopefully more quick than dirty). If the team’s task is to maintain a viable product, then “doing things right” becomes a much more desireable mindset than “quick & dirty” (which is probably even harmful in such scenarios).

You may (probably) still want a “hacker” on your “maintaining a legacy product” team, or a “maintainer” on your “hacking a POC” team, as no task is ever “pure”. Much like in these games, more often than not, you’ll need to asign tasks to the appropriate “expert”. So make sure you got one on your team.

The “like-minded” trap; or how to diversify?

It’s very easy, when building a new team, to favor people that “think” like us. This is a hard one: how to identify the skills & strengths we lack (and need) in candidates? I don’t have the answer, and sometimes I only notice my blindspots post-mortem. Best I can advise is: do your homework! Be explicit on the teams goals. Try to identify the skills needed (& currently lacking). Then, Optimize for those.

Most people are “above average”

It may have sounded a paradoxal claim when you first started reading this post. But it’s true. Most people ARE above average, just not in everything. Some people are better at hacking stuff - à la “move fast and break things”, others might be methodical & thorough, perhaps working TDD style. We all have our strengths and weaknesses. If you focus on people’s strengths, you’ll soon find out that indeed, there is no paradox.


  1. Sure seems so. I never met anyone admitting they’re “below average”↩︎

The aquarium analogy, or: Is it all about the fish?

Recently1 A couple of years ago, I got into the fantastic world of aquariums. At first, I thought there’s not much to know about it. You buy a fish tank, You put water in, some equipment - filters and such, buy some fish, and that’s it. Oh boy, was I wrong…

This is actualy a post about software development. really…!

So why am I blabbering about fish? Well, it occurred to me, that growing a software developement organization is much like maintaining an aquarium. my aquarium

How so?

Well, first I need to bore you with some more “fish talk”2.

A bit more “fish talk”

An aquarium is a closed eco-system. And much of what happens in nature, needs to happen also in your little home aquarium. Experienced fish keepers know this as the nitrogen cycle. In short:

  • fish produce waste
  • waste deteriorates into toxic ammonia
  • bacteria consume ammonia and turns it into (toxic) nitrites
  • nitrates (not very toxic, yet can harmful in very high amounts) are formed from the breakdown of nitrites by nitrifying bacteria
  • plants & algea consume nitrates
  • fish eat the algea
illustration taken from aquacadabra.com, if you want to learn more about the nitrogen cycle, pay them a visit.

OK, back to “software development talk”

In the same way I want to show off colorful fish, and thriving plants in my living room’s aquarium. An organization would want to show off with (and sell!) their wonderful products.

And much like fish, the product (software) “deteriorates” and needs to be constantly maintained. google explains why software deteriorates

How do we deal with software rot? Should we throw in more bacteria hire more developers to clean up the mess? Probably not. The thing is, that much like in an aquarium where you can’t just throw in bacteria and expect everything to just work, you can’t just blindly hire more developers and expect them to be on top of everything from day one.

Cultivating bacteria

In a newly set up aquarium, there is no nitrogen cycle, since there are no bacteria & no algea…
You’ll need to “jump start” the cycle. There are starter kits, where you just pour bacteria into the aquarium. But it’s not a magic solution. The bacteria takes time to acclimate. If you add too many fish too early, they produce too much waste, and there’s not enough bacteria to consume it. The fish will die.
Instead, you’ll need to slowly build the fish population in parallel to the bacteria population that grows. You do this by taking care of water quality (doing frquent water changes) and making sure ammonia/nitrites levels are kept low. Then, bacteria population will slowly increase, which in time will be able to handle larger bio-loads (more fish).

Cultivating developement teams

You can’t rush building a developement team either. A small, already “acclimated” team, may work well in its domain of expertise. You may even “jump start” a new team by hiring a bunch of experienced senior engineers. But it would be unwise to frequently pivot the team from one product to the next. It takes time until a team starts working together efficiently, and it takes time to gain expertise in a new domain.
You can’t overwhelm a new aquarium with little bacteria with too many fish at once.

Cultivating individuals

Its not just the team you’d want to cultivate. It’s important to grow individuals as well. Investing in personal growth means growing the teams capacity without hiring more people. I.e: scale up vs. scale out. Both are important.

Focus

Up until now, I’ve talked mostly about the “capacity” that an organization can burden its engineering department with, and the fact you need to “cultivate” your engineering department in parallel to developing the product. This is one side of the coin (cultivating the bacteria engineering). The other side is making the work itself more “friendly”, by being more focused. The point is, that capacity increses with focus. If the organization is focused on the product vision, it gives the developers the “expertise path” in the domain, which in time scales up (more capacity).

This is true for aquariums as well. You need to focus on your “type of aquarium”. One cannnot just blindly add different species of fish to the same tank. Different species has different temprature ranges they are able to leave in. Different pH (acidity), different KH/GH (hardness), etc’…
But even if all environmental parameters match, you still cannot mix species that are likely to be eaten by a different type of fish you have in the same tank. Same goes for fish that are territorial & aggressive.

No team can handle all products, and no aquarium can handle all fish combinations.
Point is: if you’re focused, you can deliver more. If you’re leaning towards exploration mode (which is totally legit), do it moderately and take into account the hit you’re taking in delivering production grade code.

In summary

  • The organization is the tank
  • The product is the fish
  • The Developers are the bacteria

And you need to take special care balancing everything in order for this eco-system to thrive.


  1. I started writing this post when the aquarium was still a “recent” thing for me…↩︎

  2. I assume many readers are not aware of the fish keeping hobby details, much like the clueless couple-of-years-younger me didn’t. And it is important for the analogy ¯\_(ツ)_/¯↩︎

Face-lift

The blog gets a face lift

Why?

Well… initially the blog was written as simple markdown posts. Markdown is an obvious choice for a blog, easy to write and maintain. I decided to render the markdown ad-hoc, so I used pagedown extra1. I also added syntax highlighting using highlight.js. This was simple and easy. Integrating with blogger was not too much of a fuss, I just wrapped every markdown post with:

<div class="markdown" style="display: none;">
  # markdown content went here
</div>

And added a small js snippet to the theme HTML editor:

// Yup! jquery was still a thing back then.
$(document).ready(function () {
  var converter = new Markdown.Converter();
  Markdown.Extra.init(converter);
  $(".markdown").each(function(index, element) {
    var md = element.textContent;
    var formatted = converter.makeHtml(md);
    element.textContent = "";
    element.innerHTML = formatted;
    $(element).find('code').each(function(i, block) {
      hljs.highlightBlock(block);
    });
    $(element).css('display','inline-block');
  });
});

I also added there the <script> tags with src of the libraries I used, directly hosted from my github account through rawgit. It worked out as expected.

The plot thickens

Throughout the years, blogger made all sorts of changes, that occassionaly caused my blog to break in weird ways. Patching it every couple of years was not such a big deal. But at some point, I decided I want to shoot myself in the foot add \(\LaTeX\) in my posts, so I added MathJax to the mix.

At that point, all the patching I did throughout the years, together with adding MathJax, turned the cute little snippet above into some serious dumpster fire.

Unsuprisingly, the blog became unusable in recent years. Rawgit service has been shutdown. libraries became outdated. And my script broke in weird ways.

Time to refactor

OK then, obviously maintaining the blog with scripts to render markdown + syntax highlighting + \(\LaTeX\) inside blogger is not something I want to continue doing. So I was thinking: what are my options?

  1. Ditch blogger and host the blog somewhere I fully control.
  2. Use blogger with pre-rendered HTML. Without any scripting shenanigans.

I didn’t want to ditch blogger. It’s convenient, support out of the box to some nice stats with google analytics. No need to care about hosting. etc’…

So I tried converting all my posts to HTML.

Attempt #1

I needed \(\LaTeX\) for all the nifty equations and I only used markdown to format the text nicely. Well… that’s exactly what \(\LaTeX\) is for. So I figured I can just reformat all my posts to \(\LaTeX\), and then convert the .tex files to HTML using htlatex. Obviously, I’ll write future posts directly in \(\LaTeX\) instead of markdown. All I needed was to figure out how to do the syntax highlighting for code blocks.

I’ve used syntax highlighting in \(\LaTeX\) before, but needed something simpler and easier to maintain. I went for Listings, as it seemed the most suitable solution. The result was… ok. The scala language2 support was not ideal. The parser/lexer did not recognized stuff like class names, or types. But it’s possible to “help” it a bit, by providing extra tokens with their style. The code blocks themselves were not as nice as I hoped: glitch-lines between the rows But the real deal breaker, was the fact that htlatex was not able to export listings as HTML.

Attempt #2

Even before the last experiment, I had a gut feeling against it. The “pipeline” was not very simple. Export markdown to \(\LaTeX\) → fix \(\LaTeX\) & syntax highlights manually → export to HTML using htlatex → maybe even fix HTML itself?

Well, MathJax already works great, and if I only add MathJax, the integration with blogger should be simple enough. So I went for direct markdown → HTML, and left MathJax inlined in the HTML.

I used pandoc to convert the markdown files to HTML. Unfortunately, syntax highlight for scala code with pandoc isn’t great as well, but I figured I could easily just patch the HTML output directly. I decided this is good enough for now and I should stop shaving this yak. Since I need to touch the HTML output a bit3, I figured I should commit it to the blog repo as well.

What’s next?

So now the blog is up and running again. All posts were converted, and everything works well. I will probably change the theme sometime soon (I chose the first thing just to get on with the tweaks). I also noticed some “inacuracies4” I should fix in some old posts. Perhaps archive the less relevant topics. But I also noticed many drafts I started and never finished over the years5: draft posts I guess we’ll see soon.


  1. I can no longer find the original sources, but I do give credit (with mostly broken links) in the first blog post.↩︎

  2. Which is by far the most frequent on my blog↩︎

  3. Some MathJax expressions contained stuff that pandoc “rendered” as HTML wrongly, and I needed to “undo” these.↩︎

  4. I guess the blog stays true to is name ;)↩︎

  5. There’s some interesting ideas in these drafts I should definately write about.↩︎