ChatGPT Can’t Figure Out Anagrams, Users Find

ChatGPT has demonstrated some incredible capabilities — it can compose poems, diagnose diseases, solve captchas and even create and play its own games, but there are some seemingly simple tasks that still trip it up.

ChatGPT is unable to unscramble words, a user has found. Please bring these letters to the correct order: dehpaesnoh,” a user told ChatGPT. But the all-knowing ChatGPT stumbled. “The correct order of the letters is: happiness,” it replied. This, of course, is incorrect — the correct answer would’ve been headphones, which is an anagram of ‘dehpaesnoh’.

ChatGPT also makes up answers for words which have no anagrams. When asked to put the letters ‘itacceepss’ in correct order, it responded with ‘accepts’, which doesn’t include all the letters in the string that was provided. Similarly, it said that the anagram of ‘tgromnia’ was ‘ignorant’, and the anagram of ‘askundatlyer’ was sincerely.

This is a pretty interesting result. ChatGPT has been previously been capable of understanding real words when they’re jumbled up. But these words had been real, and ChatGPT was able to figure out what their correct orders would be. Interestingly, when asked to unscramble gibberish words, ChatGPT doesn’t say that these words can’t be put in a correct order, but instead gives incorrect answers. This possibly has something to with how LLMs are trained — like an OpenAI researcher had said, LLMs are trained on ‘certain’ text on the internet, where people are confident of their assertions. Nobody writes a whole book or article on something they don’t know, and ChatGPT often has trouble telling its users it doesn’t know the answer, and comes up with something that’s plausible but incorrect. This might not seem like a big issue now, but could come back to haunt LLMs in the long run — like self driving has taken decades for adoption while being in development for years, it could be a while before LLMs are able to confidently say they don’t know the answer in order for them to be widely deployed in real-world situations.