Is it worth it to DIY engineering intelligence in the age of AI?
Your board wants to know whether the AI bet is paying off. One of your engineers says they can wire GitHub data into a simple dashboard in the next day or so, and they probably can (so could you, since we CTOs and VPEs are also working with coding harnesses these days). Claude just shipped dashboards, which makes it even easier. Or skip the dashboard entirely: connect Claude to GitHub over MCP and just ask.
For a quick answer, any of those works. However, for the answer you take to your board, I don’t think most software engineering organizations should build their own engineering intelligence, and my case for that position gets stronger as AI writes more of your code.
Everyone wants proof that AI is paying off
Boards and CFOs want proof that your AI investment is delivering. Earlier this year, some companies introduced leaderboards for token use. That didn’t last long. It’s easy to spend money with coding harnesses, but activity is easily gamed and a poor proxy for outcomes.
Heavy AI usage can come with very different results. DORA’s 2025 research, which surveyed nearly 5,000 technology professionals, found that AI makes strong teams stronger and struggling teams worse. CircleCI’s analysis of nearly 29 million workflows found teams producing more changes faster, while main branch success rates fell to 70.8%, the lowest in over five years. By Q2, they’d recovered only to 76.7%: better, but still below the mid-80s of 2023–2024 and well short of CircleCI’s recommended 90% benchmark.
More code in motion doesn’t guarantee more working software. You can’t tell which side of that equation you’re on without analyzing the work itself.
The first version is easy
The cheapest version: ask Claude or Copilot CLI how lead time changed over the last two weeks. One step up: grab commits and pull requests and chart cycle time and DORA metrics. You or someone on your team can build that in a day or two.
For answering a narrow question, or to give you a rough ballpark idea, either is fine. If three people on one team check a chart once a sprint or once a month, build it and move on.
One important piece of advice if you do: start from the code, not your ticketing system. Tickets record what someone planned to do. The code records what happened.
Org-wide measurement that informs resourcing decisions or goes to a board is a different job. The consequences of a wrong answer determine which path you pursue.
Moving past the simple version
The first version is just that: easy and cheap, but effectively a functional prototype. Pulling PRs and computing lead time is plumbing, and explorable data dashboards are getting simple to whip together with agentic coding harnesses.
The bigger challenge is interpretation: is this number real, what’s driving it, and does it mean what you think it means? We learned about this building Flux.
- The spike that isn’t. Your PR lead time jumps. Is it real, or is it holiday overlap, a burst of Dependabot PRs, or is half the team out? A homegrown chart shows a scary line, and someone senior spends a few hours digging, only to find out it was December.
- Baselines. Our first anomaly detection compared each week to the week before, and it was noisy. We replaced it with a comparison against the last 12 weeks, and we do this on a team-by-team basis. That means fewer false alarms, and each team is measured against its own normal.
- Teams. Your GitHub teams and your org chart don’t match who actually works together. Team breakdowns keyed off either one will be wrong, and nobody will notice.
- Review depth. A homegrown tool can tell you how many PRs reviewers approved. It can’t tell you whether a 2,000-line change got the same scrutiny as a 20-line one. Even a fast approval on a big change is only a signal to look closer: sometimes reviewers missed a problem, and sometimes pair programming or an earlier design review already covered the ground.
AI makes every one of these harder. Agents open PRs, so who’s the author? Bot and agent volume inflates PR counts and deployment frequency. Teams adopt new AI tools every quarter, and review patterns shift with them. The logic you wrote when you built the tool drifts as your SDLC workflows and processes change underneath it.
Why not just ask Claude every time?
The version of DIY we’ve heard from some leaders skips the tool entirely: connect your favorite agentic harness to GitHub and your other services over MCP, and ask fresh each time. For a quick one-off answer, that’s fine. As a way to measure and improve your organization over time, it has two problems.
First, the answers might be different every time. Ask how lead time trended last quarter on Monday and again on Thursday, and you can get two numbers computed two different ways. That’s fine when you’re curious. It doesn’t work when you’re tracking a trend.
Second, you’re paying for it each and every time. Every question pulls all the context back in: the PRs, the commits, the review history. Nothing is computed once and kept, so you’re redoing the same work and spending the same tokens every time someone asks.
You shouldn’t be redoing work, and you should be striving for consistency. That means computing the numbers once, keeping them, and pointing the LLM at what’s already there.
And now you own a product
Every fix to that interpretation layer is a product decision: what to measure, how to baseline it, what counts as a bot, which team owns what. Making it robust takes a clear vision, iterative improvements, and a lot of tokens. Even if you’re on a subscription plan or otherwise have subsidized tokens, you’re dedicating cycles to building and improving a product that’s not your product.
Measurement tools also invite requests for more. Leadership asks for another view and another team breakdown. Each request is reasonable. Together they turn that DIY tool you built into a product with no product manager.
Then the builder gets promoted or changes teams, and the context for why the tool computes things the way it does leaves with them. Whoever inherits it starts by reverse-engineering undocumented decisions.
The CFO question
If you’re presenting a chart to your CFO as evidence that an AI investment is working, it has to hold up when someone asks where the data came from and how you calculated it.
For a homegrown tool, the answer is often “someone on the team wrote a query, and nobody has checked it since.” For an LLM you asked directly, it’s worse: you can’t show how it got the number, and you can’t reliably reproduce it. Charts make either one look authoritative anyway, masking the weaknesses.
And there’s one more problem. The team that pushed for the AI spend built the tool that grades it. Even if the numbers are right, that’s a hard position to defend in a board meeting.
What a trustworthy answer needs
If you and your fellow leaders want a trustworthy answer, it needs three things: evidence rooted in the code, numbers computed the same way every time, and the work shown before the conclusion. Code is the source of truth for everything your team actually did. Consistent computation means this quarter compares cleanly with last quarter. Showing the evidence first means anyone can check the answer before acting on it.
That’s the reason we built Flux. It’s a code-first engineering intelligence platform: it analyzes your commits, PRs, and repository history to give you ground-truth intelligence from the code itself. Flux computes metrics from your repos and flags when one moves in a way that’s unusual for that team’s own history. When lead time spikes, it shows the common explanations first with evidence, before an agent writes the root-cause narrative.
And yes, Flux uses agents that we’ve built for exactly this purpose. They reason over evidence that’s already computed and stored, so the same question gets the same answer, and you’re not paying to rebuild the context every time someone asks. Teams come from who actually works together, not just the org chart. There’s nothing to tag and no process change for your team.
So is it worth it to DIY engineering intelligence? For a one-team script or a quick question to Claude, sure. For the questions your board is asking about the impact of AI-powered coding, I’d buy the answer and spend your team’s time on the product you ship.
