If you’re turning to ChatGPT to help you with computer programming, you may want to be extra careful to double-check its answers. A new study has found that 52 percent of the popular chatbot’s answers to computer programming questions contain inaccurate information.
The study was conducted by Purdue University and presented at the Computer-Human Interaction Conference in Hawaii this month. The researchers looked at 517 programming questions on Stack Overflow before feeding them to ChatGPT.
In addition to over half of the bot’s answers containing incorrect information, they found that 77 percent of the answers were verbose. What’s more, the programmers who participated in the study did not always catch the bot’s inaccuracies, overlooking “the misinformation in the ChatGPT answers 39 percent of the time.” However, ChatGPT is still a favorite among programmers, with the study participants preferring “ChatGPT answers 35 percent of the time due to their comprehensiveness and well-articulated language style.”
