Holistic system design

Good software is incredibly difficult to write, mostly subjectively awarded the accolade, focused on some specific thing or things, and the result of lots of time and effort from entire teams of experts.

This post reviews some of my own journey, and then gives some advice for holistic systems design.

A journey into software development

When I started designing and modifying software, it was mostly to scratch an itch I had. I would feel compelled to do a thing I struggled to believe was not provisioned for in the software I used.

I had to work to get my head around some things. It was exciting and empowering to make edits from trivial colour and text changes, to non-trivial additional & alternate behaviours, UI's, even systems eventually. One of my problems was that I thought of each piece of software as one thing.

One of the things I have noticed in nearly 30 years using software is that nothing is perfect. Even the most popular systems have areas for improvement. Sometimes things would work better for smaller, or larger systems. Sometimes software does not fit our mental model or workflow.

The clue is in the name. Software is not meant to be hard. If it is, refactor it!

Iterative design

One of the first movements I came across that I became excited to use was rapid application development. I was lucky enough to get on-board with this movement around the time Visual Basic & MFC were popular. My programs were trivial. My tooling was that of a novice. I can remember initial source control being zip and tarball archives, then CVS & later sites such as sourceforge.net emerged & became popular. These all made better the things that came before them. Making large systems was not immediately within reach. It can take a lot of iteration and dedication to build the delightful experiences some claim to have.

I was already a user of Linux, but I had not got the message on the UNIX philosophy yet. I loved all the things of the day; Tabbed UI, the multiple document interface, which was fairly non-standard and new. Instead of building a list of ingredients, and iteratively approaching things, I kept butting up against how difficult it was to scale a cliff in a continual movement.

Challenges Overcome

Eventually, my adventures included capturing window handles and launching processes and child-processes to composite and orchestrate my software from other software. This made extension easier and even as my software grew, distribution and management were less of a pain point.

Enter libraries & frameworks

The existing software engineers amongst us will note that I had obviously not been coding in a vacuum, but using libraries and frameworks, which at the time I knew only as shared objects and DLL's, which were implementation details.

This fast-food approach to software had left me coupled to vendors I had mistakenly assumed had made the same choices I would. I am not suggesting they were operating in bedlam, or that my initial leanings were or have become the canonical source of how to do a thing; but the more I have learned about software and the motivations behind it, I become convinced the technology driving force was to isolate thought to corporate islands. Whoever owns your language, owns a thing you use to think, and undoubtedly owns a larger part of your future. [Intellectual Property]

Being sensible

I am not advocating we all go back to islands of machine code, or try to build our own island states of NIH. I am also not assuming everyone has reached the level where they can shed their framework, or abandon every library, or write their own. it is more to say that we should be incredibly careful of who we entrust our future to, and how much we trust them.

Lazily accepting that a framework or library approach to a problem is the best, or only approach is a decidedly immature viewpoint, and yet one I hear far too much.

• Your CMS is not the easiest for everyone, or the best for everyone.
• Your database vendor does not have a monopoly on information design or layout.
• That framework which saves you a day getting started might cost you years.

What to do then?

• Regularly map out your problem space.
• Try to fix issues by prevalence & severity.
• Try to discuss technology using facts & data specific to you, not feelings, blogs, phrases from memes.
• Try to do less things.
• Try to do those things you do, better than anyone else.
• Search for information refuting your views, expanding your knowledge.

The last point takes a lot of time. You may be shocked to know system & library vendors are not known for cataloguing the limitations, edge cases, or mostly design considerations of their systems exhaustively. It is okay to be human and assume these people are human.

A good rule of thumb is that unless everything is working as designed, all the time, your assumptions are proven to be eternal, and you can prove that; it is not good enough.

• Okay so it took 5 minutes to set something up in layer 7.
  It might be faster solving in layers 3 or 4.
  It is okay to not write all your software in one language.
• Perhaps your storage library creates many decisions.
  Maybe those trap you in your code to talk to your files.
  Are there other tools or approaches?
• Maybe it will be hard to secure that 1GB software package.
  Maybe it is made of tinier parts,
  Maybe those parts are what you should use.
  Maybe it is fine to use, but only in a deferred environment.

You can break down a problem by iterating, partitioning, discarding. Relinquish parts of a problem you have data to show are not important, simply do not care about or do not have data to support.

You can do this, and you will make better systems if you do.