This study investigates the feasibility and effectiveness of using ChatGPT, a GPT-4 based model, in achieving satisfactory performance on the Fundamentals of Engineering (FE) Environmental Exam. This study further shows a significant improvement in the model's accuracy when answering FE exam questions through noninvasive prompt modifications, substantiating the utility of prompt modification as a viable approach to enhance AI performance in educational contexts. Furthermore, the findings reflect remarkable improvements in mathematical capabilities across successive iterations of ChatGPT models, showcasing their potential in solving complex engineering problems. Our paper also explores future research directions, emphasizing the importance of addressing AI challenges in education, enhancing accessibility and inclusion for diverse student populations, and developing AI-resistant exam questions to maintain examination integrity.
Our study demonstrates the potential of ChatGPT as a valuable tool for addressing engineering problems and preparing for the Fundamentals of Engineering (FE) licensing exam. The GPT-4 base model with no vision achieved a notable overall accuracy of 66.42% on the FE Environmental Exam dataset, indicating its capability to provide satisfactory results across various question types (e.g., multiple-choice, fill in the blank). Our research also highlights the significant impact of noninvasive prompt modifications on the GPT-4 base model (no vision), leading to a mean accuracy increase of 8.95% for the FE Environmental Exam, resulting in an overall accuracy of 75.37%. While limitations were observed in specific subject areas, such as Ground Water Soil and Sediments, these findings emphasize the importance of continuous model development and prompt refinement to effectively address complex questions.
Moreover, the study's comparative analysis of different generations of ChatGPT models underscores the significant improvements in mathematical capabilities for engineering applications. The GPT-4 (no vision) model demonstrated superior performance compared to its predecessors, illustrating the developers' successful efforts in addressing the AI's mathematical limitations to some degree. These advancements in ChatGPT models hold promising implications for their use in educational and professional engineering settings, where accurate problem-solving is of paramount importance.
Related Articles
- Pursnani, V., Sermet, Y., Demir, I., 2023. Performance of ChatGPT on the US Fundamentals of Engineering Exam: Comprehensive Assessment of Proficiency and Potential Implications for Professional Environmental Engineering Practice. Computers and Education.
DOI: https://arxiv.org/abs/2304.12198 (in review)
