Benchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, Wisedocs

The Problem: Scaling and Technical Debt [00:12]

• In 2025, a company experienced poor business scaling, overwhelmed by customers and lacking necessary throughput.
• Key issues included slow response times to customer demand, a complex AI pipeline difficult to update, and a legacy codebase (over 10 repos) that developers were hesitant to work with.
• The decision was made to refactor the system over six months, questioning its immediate value.

  "So, it's not just my AI pipeline that's on fire, but also my PowerPoint. So, it's 2025. Uh we're scaling as a business and things are going poorly."

Use Case: WiseDocs and Complex Claims [00:57]

• The company, WiseDocs, processes complex medical claims, often in PDF files exceeding 10,000 pages.
• This necessitates a robust and scalable application, particularly for its AI pipeline which incorporates multiple ML models.

  "So, the company I work at, WiseDocs, processes complex medical claims, which are PDFs that are more than 10,000 pages in size."

Understanding Technical Debt [01:23]

• Technical debt, like financial debt, can compound in unexpected ways, impacting ROI when additional complexity is introduced.
• While AI engineering has seen rapid progress in shipping velocity and code refactoring, this speed has sometimes come at the cost of code and product quality, leading to degraded maintainability and reliability.

  "If we introduce additional complexity into our code base, we can very quickly outrun the ROI we've generated."

The Refactor Process: Orchestration and Model Serving [03:31]

• The refactor began in April, preceded by some pre-work.
• Two months were spent evaluating five open-source orchestrators for the AI pipeline, benchmarking their effectiveness before major AI research like DeepMind's was widely available.
• Proof-of-concepts were built to ensure correct results, highlighting how tooling and agentic workflows have since accelerated this process.

  "So we spent around 2 months evaluating orchestrators for our AI pipeline. We looked at five open-source projects and we wanted to benchmark and see how effective they were for our use case."

Agentic Coding and Model Improvements [05:01]

• Initial attempts at agentic coding for refactoring, even with tools like Temporal and O3, were manual and error-prone, requiring significant human intervention.
• Rerunning benchmarks with modern models like Sonnet 4.6 and Opus 4.8 showed substantial improvements in speed and accuracy, with Opus achieving a one-shot solution.
• Model interaction has also evolved, with modern harnesses featuring sub-agents, plan calls, and verifications, making the process less manual.

  "This refactor took uh 3 hours of back and forth chatting with Incursr, but it made 10 major mistakes."

The Impact of AI on Software Development [06:50]

• AI models are significantly better now, capable of executing tasks at a high capability level with well-constructed prompts.
• However, a critical flaw often arises when prompts, plans, or requirements are incomplete, leading to frustration and wasted effort.
• The accuracy of AI in completing tasks is crucial; measuring at 90% or 99% accuracy provides a more efficient mental model for development.

  "You construct a plan, you create a spec, [snorts] you hand it off to an agent, and you're pretty sure that it'll get things done, right?"

Monorepo Migration and Productivity Gains [10:08]

• The refactor consolidated 10 repositories into a single monorepo, aiming to build additional features.
• Within six months of the rebuild, parity with the old system was achieved, followed by a steep and sustained increase in shipping velocity.
• Commit rates and developer contributions significantly increased, with almost every developer eventually contributing to the monorepo.

  "So, we had these 10 repositories, we put them into a mono repo, and we wanted to build additional features on top of it."

Was the Refactor Worthwhile? [12:30]

• Despite the argument that refactoring later is becoming easier due to AI advancements, the immediate refactor was deemed worthwhile.
• The project achieved business goals, accelerated development, reduced pipeline time and costs, supported larger files, and enabled rapid feature shipping.
• The cleaner codebase also improved developer experience, with patterns adopted in the monorepo spreading to other company projects.

  "So, if you ask me, was it worthwhile? I'd say yes. Uh we had built out the patterns that we had earlier with the number of different repos in in order to match customer requirements and demands."