AI coding assistants do not boost productivity or prevent burnout, study finds

A study by Uplevel shows no productivity gains or reduced burnout from using GitHub Copilot, contrary to previous claims.

: Uplevel's study of 800 developers found no significant productivity improvements when using GitHub Copilot. Contrary to expectations, Copilot users introduced 41% more bugs. Claims from GitHub about boosting productivity by 55% were not supported. Contrasting experiences from different companies suggest AI coding tools are still evolving.

Uplevel analyzed the performance of around 800 developers over three-month periods with and without GitHub's Copilot coding assistant. Surprisingly, metrics like pull request cycle time and throughput showed no significant improvement, and Copilot users introduced 41% more bugs.

Matt Hoffman from Uplevel initially expected higher code output and fewer defects, but the findings defied these expectations. Additionally, there was no evidence that using Copilot helped prevent developer burnout, contradicting claims from GitHub and other AI tool proponents.

While some like Travis Rehl from Innovative Solutions reported productivity increases, others like Ivan Gekht from Gehtsoft USA found AI-generated code challenging to debug, often requiring rewrites. Conflicting experiences indicate that AI coding assistants are still in their early stages and rapidly evolving.

AI toolsGitHub Copilotproductivity