A.I. Coding assistance?

The illusion of productivity

I’ve been trying out copilot agent-mode in both VS Code and Visual Studio 2022 the past couple of weeks. They are very impressive, fascinating and fun to work with. Curiously observing what it understands, how it responds, the code it writes, and how it reacts to errors and mistakes is hours of entertainment. Its a little bit like playing a video game, issuing commands to see what you can get it to do, genuinely engrossing.

While engaging and fun to work with, at least experimentally, I’m not sure that’s quite delivering on the promise. What we really want is the dramatic productivity and business value gains, right?

Well, that’s nowhere in sight. I’ve set it loose on a number of fairly discreet real refactoring tasks or a real codebase, and its slow, error prone, in constant need of correction, sometimes wildly off-base.

The experience reminded me of off-shoring years ago, and it may be analogous in more than one way. The promise of off-shoring was not unlike A.I. coding assistants. It promised access to a dynamic pool of engineering resources that could be scaled up and down according to demand, cheaply. With those resources being in different time zones we could have around the clock coding taking place, and unlock massive throughput.

However, my experience with off-shoring didn’t match those ideals. While they may have been cheap and “easy” to scale, it was not productive. There were language barriers and cultural differences, unfamiliarity with the domain and the code, skill gaps and work hour mismatches, high turnover. There was even the issue of allotting 5 engineers, but really having more than 5 people rotate in and out of the 5 engineer slots, exacerbating the above issues. To overcome this, we needed to spend more time specifying the work, more time reviewing it, explaining it, requesting rework, quality issues, etc. In many cases we had to explicitly provide a pre-coded solution to the off-shore team showing them exactly what to type. Rather than a software production boon it was an inefficiency quagmire.

Agent-mode coding assistance has had that same character for me. A task that I could implement myself in an hour, often takes several hours of me experimenting with the right incantation to get a reasonable solution. Time spent scrutinizing results, trying to nudge it to correct it mistakes, waiting patiently as its slowly “thinking” and slowly altering files, causing compilation issues, and then trial-n-error fixing its own issues. The occasional lies about what it has done or not done, or going wildly off the rails rippling changes that it can’t figure its way out of, sapping trust and confidence a long the way.

Its early, and the vibe coders are vibing, but maybe the analogy is an allegory.