Sam Altman: Size of LLMs won’t matter as much moving forward

When OpenAI co-founder and CEO Sam Altman speaks these days, it makes sense to listen. His latest venture has been on everyone’s lips since the release of GPT-4 and ChatGPT, one of the most sophisticated large language model-based interface created to date. But Altman takes a deliberate and humble approach, and doesn’t necessarily believe that when it comes to large language models, that bigger is always going to be better.

Altman, who was interviewed over Zoom at the Imagination in Action event at MIT yesterday believes we are approaching the limits of LLM size for size’s sake. “I think we’re at the end of the era where it’s gonna be these giant models, and we’ll make them better in other ways,” Altman said.

He sees size as a false measurement of model quality and compares it to the chip speed races we used to see. “I think there’s been way too much focus on parameter count, maybe parameter count will trend up for sure. But this reminds me a lot of the gigahertz race in chips in the 1990s and 2000s, where everybody was trying to point to a big number,” Altman said.

As he points out, today we have much more powerful chips running our iPhones, yet we have no idea for the most part how fast they are, only that they do the job well. “I think it’s important that what we keep the focus on is rapidly increasing capability. And if there’s some reason that parameter count should decrease over time, or we should have multiple models working together, each of which are smaller, we would do that. What we want to deliver to the world is the most capable and useful and safe models. We are not here to jerk ourselves off about parameter count,” he said.

Altman has been such a successful technologist partly because he makes big bets, and then moves deliberately and thinks deeply about his companies and the products they produce – and OpenAI is no different.

“We’ve been working on it for so long, but it’s with gradually increasing confidence that it’s really going to work. We’ve been [building] the company for seven years. These things take a long, long time. I would say by and large in terms of why it worked when others haven’t: It’s just because we’ve been on the grind sweating every detail for a long time. And most people aren’t willing to do that,” he said.

When asked about the letter that requested that OpenAI pause for six months, he defended his company’s approach, while agreeing with some parts of the letter.

“There’s parts of the thrust [of the letter] that I really agree with. We spent more than six months after we finished training GPT-4 before we released it. So taking the time to really study the safety model, to get external audits, external red teamers to really try to understand what’s going on and mitigate as much as you can, that’s important,” he said.

But he believes there are substantial ways in which the letter missed the mark.

“I also agreed that as capabilities get more and more serious that the safety bar has got to increase. But unfortunately, I think the letter is missing most technical nuance about where we need to pause — an earlier version of the letter claimed we were training GPT-5. We are not and we won’t be for some time, so in that sense, it was sort of silly — but we are doing other things on top of GPT-4 that I think have all sorts of safety issues that are important to address and were totally left out of the letter. So I think moving with caution, and an increasing rigor for safety issues is really important.  I don’t think the [suggestions in the] letter is the ultimate way to address it,” he said.

Altman says he’s being open about the safety issues and the limitations of the current model because he believes it’s the right thing to do. He acknowledges that sometimes he and other company representatives say “dumb stuff,” which turns out to be wrong, but he’s willing to take that risk because it’s important to have a dialogue about this technology.

“A big part of our goal at OpenAI is to get the world to engage with us and think about [this technology], and gradually update and build new institutions, or adapt our existing institutions to be able to figure out what the future we all want is. So that’s kind of why we’re here.”