There’s been an explosion of open-source models in recent times in response to GPT-4, but OpenAI believes that these models will never be as good as the models they produce.
OpenAI’s Chief Scientist Ilya Sutskever has said that there will always be a gap between open-source models and proprietary models such as those produced by his company. Sutskever was responding to an audience question on a tour to Israel. “Could open-source LLMs potentially match GPT-4’s abilities without additional technical advances?” an audience member asked. “Or is there a secret sauce in GPT-4 unknown to the world that sets it apart from other models? Or am I wasting my time installing Stable Vicuna?” they added.
“To the open-source versus the non-open source models question: you don’t want to think about it in binary black and white terms where there is a secret sauce which will never be rediscovered, or whether GPT-4 will ever be reproduced by open-source models,” Ilya Sutskever replied. “Perhaps one day it will be. But when it will be, there will be a much more powerful model in the companies,” he said.
“So there will always be a gap between the open source models and the private models. And this gap may even increase with time. The amount of engineering and effort and research that takes to produce one such neural net keeps increasing. And so even if there are open-source models, they will less and less (frequently) be produced by small groups dedicated researchers and engineers, and it will only be the providence of the company: a big company,” Sutskever said.
Sutskever comment will be a dampener for thousands of ML researchers, who’ve been working hard over the last few months to build open-source alternatives to GPT-4. There are at least 50 open-source LLMs out there, and while many come close to GPT-4 on many parameters, none of them has actually managed to beat it. But Sutskever seems to suggest that no matter how hard the open-source community tries, they won’t be able to build models as good as state-of-the-art private models.
Now Sukskever is a legend in the field of AI, and he’s helped build GPT-4, so there are few people out there which could have more insight into how the LLM world is shaping up. But Ilya Sutskever is also currently incentivized to say that his company will be hard to beat — OpenAI has raised billions of dollars of funding from Microsoft, and stands to gain if the world at large thinks that its models are unbeatable. It’s hard to say what Sutskever’s motivations behind the comment are, but it’s unlikely that the open-source community will simply give up working on their own models because the company that created the current best model says they’ll be impossible to beat. And if past history is anything to go by, the open-source community loves to rise up to a challenge.