Wednesday, April 20, 2022

The Agile hypothesis

At recent software architecture training, there was a lot of negative discussion of Agile, which got me thinking about the what the strongest defense of Agile might look like. And by Agile I think more of the practices of Extreme Programming than rituals of Scrum.

IF you accept a few premises, then I think Agile appears reasonable. Those premises look something like this:

  1. Engineering requires a rigid and highly specified design/design process.
  2. Producing a highly specified design is expensive (time, money, effort, training), but necessary to avoid the even more expensive prospect of wasted building materials.
  3. Building materials (code) are virtually free and infinite in software.
  4. Software is inherently more complex than other engineering disciplines, or at least too new for a body of sufficiently reliable engineering practices to have evolved.
  5. There may yet be something unique about software's lack of physical constraints that defy or modify typical engineering practices.
  6. Demand for software, and the pace of change are too fast to wait for the body of knowledge to develop, or for engineering experts to be trained, if it's even possible.
  7. Software requirements are more volatile than other fields because the of the pace of innovation and the lack of physical constraints.
  8. Demand for the software engineers already far exceeds the supply.
  1. Therefore, since software (being new/unique/complex/volatile) is more expensive to specify but the materials are less expensive, the economics are flipped. For software, we can lower the expertise bar, increasing our engineering capacity, and still deliver systems, by churning the building materials with incremental experiments, and switch directions with less sunk cost, in order to arrive at a good enough systems.

Said another way, if you're fumbling in a dark room where the furniture is moving around, the most effective way to get from point a to point b is probably slow incremental experiments. A kind of software Darwinism, iterating through many quick generations of software versions in order to evolve into a good enough design.

That doesn't discount that likelihood that many of the rituals and ceremonies of practices like scrum, don't in fact ad much value or enable agility.

Some of these premises are likely flawed, but it seems like #3 and #7, #8 are the most prominent.

3) Maybe code is free, but labor isn't. Nor are the downstream costs of excessive resource consumption of poorly written systems. It's at least (Material + Labor) which probably evens the score.

8) Demand for engineers is self-reinforcing. More poorly written code released into the world increases demand for engineers.

On the other hand, within #3 and #5 there is a kernel of mystery there. The lack of physical constraints does seem to change the calculation, and Agile does seem to be working. There is an awful lot of software in the world encroaching on every aspect of our lives, and while often buggy and imperfect, we much prefer it to not having it.

Some of this might be explained by a lack of competition. Rather than a binary choice between success and failure, companies operate on a continuum of effectiveness. As long as everyone is building mediocre software behind schedule, you don't fail by similarly creating mediocre software behind schedule. Further, the impacts of poor design are not often felt immediately, but impact the longevity of a company. Maybe decades later, which could be after a company has already failed for other reasons.. Again from a Darwinian perspective, other techniques haven't demonstrated a competitive advantage over (decently run) Agile yet.

The other part of the answer is that the need for engineering and architecture is rare.

The vast majority of all software engineering work is a feature grind, not architecture or re-architecture. Incrementally modifying the existing software within the context of whatever architecture you're stuck with. XP may not be a bad match for the construction phase, if you're incrementally delivering building blocks.

That's also true in a house metaphor. You build a house once, maybe once a decade you need an architect again for an addition, but otherwise it’s a combination of professionals and unprofessional (homeowners) incrementally performing construction on the house.

The worst forms of Agile, emphasizing rituals, just pulling tickets with no particular plan or design, maximizing rework, aren't helpful. But perhaps the disciplined variety with a good enough plan, and incremental delivery work in software where they wouldn't necessarily in other fields.