Software engineer workflows have been revolutionized in recent years by the introduction of AI coding tools like Cursor and GitHub Copilot. These tools promise to boost productivity by automating tasks such as writing code, fixing bugs, and testing changes. Powered by AI models from top-notch companies like OpenAI and Google DeepMind, these tools have shown significant performance improvements in various software engineering tests.
AI Coding Tools and Productivity
A recent study by the non-profit AI research group METR has raised doubts about the actual productivity gains experienced developers can expect from using AI coding tools. In a randomized controlled trial involving 16 experienced open source developers, it was found that allowing the use of AI tools actually increased completion time by 19%. Surprisingly, developers were slower when using AI tooling, despite initially forecasting a 24% reduction in completion time.
The researchers point out that developers spent more time prompting AI and waiting for responses when using these tools, rather than actually coding. Additionally, AI tends to struggle in large, complex code bases, which was the focus of the study.
Looking Ahead in Tech
While these findings cast doubt on the universal productivity gains promised by AI coding tools in 2025, it’s important to note that AI progress has been substantial in recent years. METR acknowledges that AI tools have improved significantly in completing complex, long-horizon tasks. However, caution is advised as other studies have shown that AI coding tools can introduce errors and security vulnerabilities.
In conclusion, developers should approach AI coding tools with a healthy dose of skepticism and be aware of the potential pitfalls before fully integrating them into their workflows.
