CodeQuality4DevGPT is a project focused on identifying quality issues in code generated by ChatGPT. By utilizing various linters, this project aims to analyze the code for errors ranging from minor stylistic issues to more critical logical and efficiency problems. The analyzed code is sourced from the DevGPT database and GitHub API.
Raw Data: Data
Classified Data: Data
This study aims to evaluate the quality of code generated by ChatGPT across multiple programming languages including Python, Java, and JavaScript. By employing a set of robust linters—Pylint for Python, PMD for Java, and JSHint for JavaScript—the research scrutinizes common issues such as stylistic inconsistencies, logical errors, and severe programming flaws within the code samples sourced from the DevGPT database. This analysis not only identifies prevalent problems but also assesses the strengths and weaknesses of each linter in detecting these issues, offering a comparative insight that informs on potential areas for enhancement in AI-generated code.
Research Question 1: How does ChatGPT Perform Across Programming Languages?
Further, the findings contribute significantly to understanding the limitations and capabilities of AI in coding, which is crucial for integrating AI tools more effectively into software development workflows. By highlighting common pitfalls in AI-generated code and providing empirically driven recommendations for best practices, the study aims to improve the reliability and productivity of AI as a development tool. This research aligns with the objectives of the MSR 2024 Mining Challenge, thereby advancing the broader agenda of enhancing code quality in the era of AI-assisted programming.
Research Question 2: What are the Differences Between AI-Generated and Human-Written Code?
Before you begin, ensure you have the following installed:
- Python 3.7: Download here
- Pip Env
- Java 8 or above
-
Clone the repository:
git clone https://github.com/ahabegger/CodeQuality4DevGPT.git
-
Set up Python packages: Navigate to the
CodeQuality4DevGPTfolder and install the required packages:pip install -r requirements.txt
-
Set up Linters:
- JSHint:
- Set up an NPM environment and install JSHint using the instructions here.
npm install -g jshint npm fund
- PMD:
- Download PMD from PMD GitHub and follow the installation guide here.
- JSHint:
-
Setting Up GitHub API: Within the
CodeQuality4DevGPTfolder, create atoken.txtfile and store your GitHub token:touch token.txt echo "{GITHUB Token}" > token.txt
Navigate to the CodeQuality4DevGPT folder and run the following commands to execute the programs:
python ChatGPTCode
python HumanCodeOutput files will be located in the Output folder.
Follow the classification instructions provided in Appendix A of the Study
Contributions to improve CodeQuality4DevGPT are welcome. Please ensure to update tests as appropriate.
This project is licensed under the MIT License - see the LICENSE file for details.