A Comparative Evaluation of Large Language Models for Automated Code Generation and Debugging

Abstract

Large language models (LLMs) have moved from text completion systems to practical programming assistants capable of generating functions, explaining code, locating defects, and proposing repairs. Their apparent fluency, however, does not translate uniformly into functional correctness across programming settings. This paper presents a structured comparative evaluation of published benchmark evidence on automated code generation and debugging. The analysis integrates results from HumanEval, MBPP, QuixBugs, DebugBench, SWE-bench, LiveCodeBench, and competition-level code generation studies, with attention to test-based correctness, repair success, task complexity, runtime feedback, and benchmark contamination. The evidence shows a clear capability gradient. LLMs perform strongly on short, self-contained programming tasks, and repeated sampling substantially increases the chance of obtaining a correct solution. Debugging performance is also substantial for syntax and reference errors, while logic defects and multiple simultaneous bugs remain more difficult. Repository-level software engineering creates a sharper challenge because successful repair requires long-context understanding, multi-file reasoning, environment interaction, and reliable test execution. The comparison further shows that benchmark design strongly shapes reported performance, making direct score aggregation inappropriate across datasets. Overall, modern LLMs function effectively as code-generation and debugging assistants, but their outputs still require executable validation, security review, and human oversight before integration into production software