ChatGPT's ability to provide conversational answers to any question at anytime makes the chatbot a handy resource for your information needs. A new study finds that you may not want to use ChatGPT for software engineering prompts despite the convenience. Researchers find that ChatGPT answers more than half of software engineering questions incorrectly. Before the rise of AI chatbots, Stack Overflow https://stackoverflow.com was the go-to resource for programmers who needed advice for projects similar to ChatGPT's.[1]
Stack Overflow is a question-and-answer website for programmers. It is the flagship site of the Stack Exchange Network. It was created in 2008 by Jeff Atwood and Joel Spolsky. It features questions and answers on certain computer programming topics. With Stack Overflow, you have to wait for someone to answer your question; with ChatGPT, you do not have to wait for a reply.
As a result, many software engineers and programmers have turned to ChatGPT for their questions. Since no data showed how efficacious ChatGPT is in answering those prompts, a new Purdue University study investigated the dilemma. To find out just how efficient ChatGPT is in answering software engineering prompts, the researchers gave ChatGPT 517 Stack Overflow questions and examined the accuracy and quality of those answers.
The results showed that out of the 512 questions, 259 (52%) of ChatGPT's answers were incorrect, and only 248 (48%) were correct. Moreover, a whopping 77% of the answers were verbose. Despite the significant inaccuracy of the answers, the results did show that the answers were comprehensive 65% of the time and addressed all aspects of the question. To further analyze the quality of ChatGPT responses, the researchers asked 12 participants with different levels of programming expertise to give their insights on the answers.
Although the participants preferred Stack Overflow's responses over ChatGPT's across various categories, as seen by the graph, the participants failed to identify incorrect ChatGPT-generated answers 39.34% of the time correctly. According to the study, the well-articulated responses ChatGPT outputs caused the users to overlook incorrect information in the answers. "Users overlook incorrect information in ChatGPT answers (39.34% of the time) due to the comprehensive, well-articulated, and humanoid insights in ChatGPT answers," the authors wrote.
Generating plausible-sounding incorrect answers is a significant issue across all chatbots because it enables the spread of misinformation. In addition to that risk, the low accuracy scores should be enough to make you reconsider using ChatGPT for these prompts. Just because ChatGPT replies with an answer, you cannot depend on it to be correct.
This article is presented at no charge for educational and informational purposes only.
Red Sky Alliance is a Cyber Threat Analysis and Intelligence Service organization. For questions, comments, or assistance, please get in touch with the office directly at 1-844-492-7225, or feedback@redskyalliance.com
Weekly Cyber Intelligence Briefings:
Reporting: https://www.redskyalliance.org/
Website: https://www.redskyalliance.com/
LinkedIn: https://www.linkedin.com/company/64265941
Weekly Cyber Intelligence Briefings:
REDSHORTS - Weekly Cyber Intelligence Briefings
https://attendee.gotowebinar.com/register/5993554863383553632
[1] https://www.zdnet.com/article/chatgpt-answers-more-than-half-of-software-engineering-questions-incorrectly/
Comments