Are 52% Of ChatGPT Coding Answers Incorrect?

Tom Smykowski
5 min readMay 25, 2024

The truth about the “latest” research is more interesting than how media twisted it

With the raise of LLM, large language models, more and more interesting solutions appear. One of such solutions is ChatGPT. A AI chatbot. It’s not used only for natural language. It’s also used by programmers to find answers to their questions.

It’s great to answer simple questions that were already answered properly somewhere on the internet.

This article was brought to you, because people read it on Medium.

The latest version also is able to browse provided links to gather additional information.

The possibilities are impressive, but programmers start to notice ChatGPT is not always right.

From time to time software engineers share ChatGPT answers that are ridiculous, unsafe or totally wrong.

Yesterday I’ve checked how the new model works. I’ve provided it with the link to release notes of Kotlin and asked ChatGPT to tell me what is the funniest update. There were like 150 changes made, so I figured out it will be an interesting experiment.

To my surprise ChatGPT fetched the changelog, and wrote that there’s a bugfix for emoji error that’s the funniest…

--

--

Tom Smykowski

Software Engineer & Tech Editor. Top 2% on StackOverflow, 3mil views on Quora, reached 23rd floor in Shuttered Pixel Dungeon