Is AI-Generated Code Fast, Cheap, AND Good?
Years ago a good friend and colleague, Allen Rogers, re-introduced me to the Iron Triangle. It is a general-purpose framework for thinking about opposing forces or trade-offs and can be applied to just about any project. It works particularly well with software development. As Allen said way back when with a sly grin, “we can build this fast, good, or cheap – but you only get to pick two.”
How does AI-generated code impact the Iron Triangle? Improving speed at low cost sounds amazing and appears viable. If the code is also good, it would represent an incredibly rare step-change improvement in software development velocity. But is AI-generated code good? What about code quality, security, and the protection of personal information? This is particularly important right now as the questions organizations have of their code are at an all-time high. As I discussed in a prior blog post—The Code Understanding Imperative—across security, risk, legal, compliance, and more, most organizations have amassed an array of people, process, and technology whose purpose it is to evaluate the organization’s software. Even prior to gen AI, this collective apparatus was bombarding Engineering with questions to the detriment of productivity and morale. By increasing the pace of development, AI-generated code is increasing this burden. If AI-generated code is of lower quality, this burden will be even more crippling. Two CTOs I recently spoke with echoed precisely this concern now that they’ve made substantial investments in Github Co-Pilot.
What If Faster and Cheaper Isn’t Good?
Existing research on AI-generated code quality suggests these concerns are well placed. For example, the primary finding from a November 2022 Stanford paper on the security of AI-generated code was that “inexperienced developers may readily trust an AI assistant’s output at the risk of introducing security vulnerabilities.” And while a June 2023 report from McKinsey extols the productivity boost of gen AI for developers, it highlights several areas demanding attention, including: 1) “examining code for bugs and errors” and 2) “contributing organizational context.”
It’s probably both fairly obvious and not controversial that good code means code that is of high quality, not overly complex, and adheres to best practices for security, the handling of sensitive data, and so on. But McKinsey’s point about organizational context hints at other important aspects. Good code must also 1) actually achieve the intended outcome and 2) do so in the context of organizational / business requirements. For example, does the code fulfill the acceptance criteria as defined in the tickets the product manager wrote to describe the work? Does it adhere to architectural and other standards to ensure it can easily and appropriately integrate with other systems? Bulletproof, incredibly elegant, etc. code that fails these tests is clearly not good.
These collective forces put Engineering leadership in a precarious position. Gen AI’s lure of increased productivity at the same (or lower) cost is probably irresistible. But without careful oversight, this risks a reduction in quality—with potentially devastating consequences like outages, breaches, and more. And if that weren’t bad enough, team morale would also degrade, not just because of code quality problems, but the increased code evaluation burden. Developers splitting their time between putting out fires and responding to even more compliance audits, etc. aren’t doing much developing and are more likely to quit.
Pick All 3
What if you could ensure “good” without additional developer burden and thus get to pick three? That’s why we built Flux. By applying a compound AI approach to code evaluation and understanding, Flux recruits computers to solve the problems of ensuring code meets quality and security standards, saving developer time. Leadership gains an Engineering Management Co-Pilot that automatically flags risks to “good.” Developers get left alone so they can stay happy and keep on coding. If picking all three sounds interesting to you, I’d love to talk with you.
