If something is too good to be true, it usually is.
Two weeks ago, a paper published by MIT and Harvard researchers had created a sensation by claiming that GPT-4 could solve 100% of MIT’s Computer Science and Electrical Engineering problems. But after independent researchers discovered that the results of the paper were not reproducible, some of the paper’s authors have now asked for the paper to be withdrawn.
“Iddo Drori posted on arXiv a working paper associated with a dataset of exams and assignments from dozens of MIT courses,” three co-authors of the original paper have now said in an open letter. Iddo Drori was another one of the co-authors. “(Iddo Drori) did so without the consent of many of his co-authors and despite having been told of problems that should be corrected before publication. One of us only learned of the posting on Sunday June 18 after traveling over the weekend,” the letter says.
“In the process of addressing this matter, we discovered that contrary to what Iddo Drori had conveyed to us and to the students collecting data for the project, Iddo did not have permission from all the instructors to collect the assignment and exam questions that made up the dataset that was the subject of the paper. Instructors for some of those courses only learned of the existence of this dataset and the inclusion of their course material in it when the paper appeared on social media and when Iddo posted samples of the data online without permission from anyone.
These are serious matters that are being addressed through institutional channels, so we did not take lightly making such a public statement about them, but we feel it is important to explain why this paper should never have been published and must be withdrawn. We have asked Iddo to withdraw the paper from arXiv and have also contacted arXiv directly explaining the situation,” the letter continues.
“We want to emphasize that all the student authors in this paper worked really hard on what could have been a very interesting and valuable paper had the data been collected with consent. The many problems with the published work were not the fault of the students,” the letter says. “And no, GPT-4 cannot get an MIT degree,” the letter concludes. The letter is signed by Armando Solar-Lezama, Professor in EECS and COO and Associate Director of CSAIL, MIT Tonio Buonassisi, Professor of Mechanical Engineering, MIT Yoon Kim, Assistant Professor in EECS and CSAIL, MIT.
The retraction comes days after it had been discovered that the results of the original paper were not reproducible. “What we found left us surprised and disappointed. Even though the authors of the paper said they manually reviewed the published dataset for quality, we found clear signs that a significant portion of the evaluation dataset was contaminated in such a way that let the model cheat like a student who was fed the answers to a test right before taking it,” MIT students Raunak Chowdhuri, Neil Deshmukh, and David Koplow had written about the paper, while highlighting many inconsistencies in how it had administered the tests and interpreted the results.
The retraction just goes on to show that while AI has made remarkable strides over the last few years, it’s easy to get carried away by the hype, even for sophisticated participants within the field. The paper had been widely shared in AI circles, but it took some MIT students’ skepticism to actually dig deep into the results to figure out that all wasn’t quite what it seemed. And with the paper’s co-authors now asking for the paper’s withdrawal, it might still be a while before AI can ace MIT’s coursework.