I don't think attention really matters. A statement remaining on their website is like continuously saying/affirming whatever that statement is. Removing it means they have changed their minds about something it contains. Removing it just stops that affirmation, it doesn't really matter if people know what you used to think as long as they know you no longer do.
If you have a statement on your website that promises to never put cats on the moon but then you do, I imagine some shareholders/investors might feel misled. I don't know _if_ that opens someone up to legal action but I could see the argument.
Is the statement on their website or on a third party website? Changing your mind about what you publish is one thing, changing your mind about an interview you just gave is another.
but theres an open question now about whether or not that was MEANT to be an interview, or rather a closed door session to a small group of folks. if Humanloop had not gotten clearance to post what they posted, this would have been a grave overstep of journalistic ethics.
(i am not throwing stones, i've fallen foul of this as a blogger before, excusing myself from commonsense precaution in my eagerness to discuss industry relevant insights. private https://www.swyx.io/private)
> if Humanloop had not gotten clearance to post what they posted, this would have been a grave overstep of journalistic ethics.
Where did you get this idea? You don't "go on record" with a journalist. Everything you do and say around them is on the record by default. There are ethical codes and frameworks on what, when, and how to report but "private conversations are private" is certainly not considered normal.
Re your mea culpa, it's a completely different situation and I don't see how journalistic ethics would be applicable there. If you want to have a more appropriate term, maybe "blogger ethics"?
neither Raza nor myself are journalists, but we behave like them when we write stuff up for obvious HN bait. privacy by default is the overwhelming expected norm for private individuals. particularly at a closed door small group event. we abuse the system when we use our status as industry participants and become citizen journalists without first considering if the expectations are aligned.
> we abuse the system when we use our status as industry participants and become citizen journalists
Agreed. This includes not making statements like your previous one about "journalistic ethics". Consider this: If your now removed post had been containing the same information but based on a separate conversation with their ex-partner, with their consent to publish, would that had made it less problematic? Without having read the post, I imagine it wouldn't. So it turns out that there's more to it.
I'm not saying these posts aren't problematic, nor that journalists lack ethics. I am saying that "private conversations are private" is oversimplifying the matter and not part of journalistic ethics in the general. You're making references to professional code you lack qualifications for.
Consider medicine. Let's say Alex is visiting a sick family member. Alex has no medical training. They bring some drugs which later turn out to cause even worse health issues. "Medical ethics" isn't a useful framing for discussing that situation IMO, for similar reasons.
It's like those pentalobe security screws. Nobody thinks they'll stop a determined attempt to disassemble a product, but they're a signal that you're not meant to pull it apart. Then when you do, and you break it, they can point to the screws and say "well clearly you should have known not to do that."
I don't agree. Go search for HN threads relating to anything Sam Altman has said publicly, interviews he's given etc and you'll see that they always get posted multiple times on HN and the biggest thread always has 100s if not 1000s of comments.
This content was going to be wildly shared and discussed whether deleted or not, by deleting it they have at least a) made an official (yet soft, as it doesn't address any specific points) statement that it's not information they're trying to put out and b) probably reduced SEO / future views beyond these current discussions.
I can't see any argument for it getting more attention because of deletion than it would have got anyway?
I'm pretty sure Sam Altman knows about and understands Streisand effect. There are a myriad other explanations for "This content has been removed at the request of OpenAI."
I don't think this is ironic. They retracted something they said, this is perfectly legit.
I've deleted things I posted, including here on HN. You probably did too. And I've scraped data from the internet countless times. You probably did too. I don't think I'm an irony because of that. You probably aren't either.
This might not be about suppressing this particular bit of information so much about holding the group of journalists they invited to their agreements.
You missed what probably triggered it: "...said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs..."
> OpenAI will avoid competing with their customers — other than with ChatGPT
Even though I interpret this as the author’s summarizing of a statement to the effect of “we want to stay at the platform layer and not the application layer”, it stands out to me as a phrasing that might irk lawyers sensitive to anything that could hurt them in an antitrust lawsuit a decade from now.
> The fact that scaling continues to work has significant implications for the timelines of AGI development. The scaling hypothesis is the idea that we may have most of the pieces in place needed to build AGI and that most of the remaining work will be taking existing methods and scaling them up.
Is this a pretty consensual viewpoint? Or is it hype from OpenAI?
Taking existing methods and scaling them up leads to better LLMs. That's a major part of what OpenAI is doing.
Now the scaling hypothesis is basically just that sophisticated behavior emerges by itself as neural networks get bigger and get trained on more data with more complex problems. LLMs certainly show that so far, but to jump to an AGI I think we need to fix various shortcomings. It's certainly possible that a future version of an LLM will be an AGI, but imho that will also be thanks to current and future improvements like adding an internal dialogue or medium-term memory. Or maybe LLMs just can't scale that far, because experiencing the world through text is insufficient.
Yeah at this point nobody has the authority to say how far the architecture might go. It looks promising, OpenAI has all the reason to hype it as much as possible and it certainly seems plausible, but it could just as likely hit a practical ceiling of some sort.
Within comp sci? Not sure. Within academia writ large? No way. But hey I look to biologists who put biological immortality > 100-200 years out (I've never believed Aubrey de Grey and similars during the 20 years he's been in headlines) and see AGI as in the domain of biology/psychology. We've implicated the brain, that's about it.
It seems absurd to me that someone would try to make a prediction about the state of technology in 100 years. If you asked people in 1920 to make a list of predictions about what we would/wouldn't be able to do today would they have done better than chance?
I guess our understanding of the laws of physics haven't changed that much. So someone predicting that free energy will be impossible is likely correct.
I honestly don’t understand the situation here. Why would Sam sit down for an interview and then request it be removed? Did he accidentally let slip something he didn’t mean to? Isn’t controlling the message supposed to be one of his best skills? I find it hard to believe anything said was really not intended for release. Also, does anyone at OpenAI really think removing the blog post will remove it from the internet? The top comment here is already a link to an archive of the content.
I am interested in hearing the rationale for this action as I really don’t see why they requested the removal.
> Quite a few developers said they were nervous about building with the OpenAI APIs when OpenAI might end up releasing products that are competitive to them. Sam said that OpenAI would not release more products beyond ChatGPT.
I bet this isn't true and he wasn't supposed to say it.
I hope it isn't true! I'd expect them and other LLM orgs to support some plugins where it isn't worth them building the functionality internally but also replacing the ones that they can do better or with less or little effort. Hard to be upfront and honest about what the future looks like this early in the applied statistics arena. Maybe OpenAI should be deferring more to their Head of PR?
This is the usual "Netflix attempting to become HBO faster than HBO can become Netflix" [1] and a critical eye to where the work is being done and the value being delivered.
EDIT: btown mentions "Just because ChatGPT+Plugins isn't going to be the center of people's lives" in their comment, and I think this is somewhat the point folks are concerned about. ChatGPT (LLM interfaces in general, broadly speaking) might be the IDE of the white collar future, versus an API you call. And maybe that is okay if they can deliver a superior experience to where people live today (Excel, email client, word processor, other SaaS tools, a development environment). Half joking, maybe they should acquire Superhuman [2] and rebrand to that versus "ChatGPT", because that is the ability consumers and enterprises are paying for: to give people doing work superhuman ability.
(disclosure: paying OpenAI customer, no other relation to any org mentioned in this comment)
Any lawyers know if this opens them up for legal action if they kill a business in this manner after a public statement like this? It must, if this is the reason they requested the removal, otherwise why would they tip their hand about what they intend to do in the future? And not to mention it makes developers even more concerned. They could have just let the lie alone, instead of drawing attention to it.
Also no a lawyer, but... my understanding was that you're always open to legal action from anybody. They might not have a very good case and they might just get laughed out of court, but they can still sue you. Is that not the case?
Edit: Or I guess as you said in a reply to a sibling comment, you might only count it as "open to legal action" if you have a decent chance of losing.
Setting aside the legalities for a moment, a nontrivial part of OpenAI's advantage in the LLM space has been its ability to incentivize feedback mechanisms - to own the interface in which generated text is presented, own the widget where someone gives it a thumbs-up or thumbs-down, and re-integrate that feedback at scale into its fine-tuning.
Of course they'd want to leave the door open to building their own products, in order to continue to be able to present those widgets and have exclusive access to the resulting data.
Just because ChatGPT+Plugins isn't going to be the center of people's lives (did anyone truly think it would?) doesn't mean that when they have another product idea that they have unique insight into, they won't want to bring it to the world directly.
I was in that meeting. It was an invite-only developers round table. The aim was to discuss the platform from the developers point of view, receive feedback to improve the API and other discussions in general.
This was followed by a public fireside chat hosted by Azeem Azhar and QA session.
From the context, I would believe that OpenAI didn’t want some of the points discussed in a closed room to go to public.
That’s part of what I find hard to believe. I would have figured Sam would be aware of the strategic messaging goals not only of OpenAI but also of MS, their biggest investor.
To be fair, Sam is human and is doing a LOT right now. In addition to leading OpenAI and partnerships, he's having to do a parade lap around the world to console politicians and communities.
"A lot of developers are interested in getting access to ChatGPT plugins via the API but Sam said he didn’t think they’d be released any time soon. The usage of plugins, other than browsing, suggests that they don’t have PMF yet."
There might also be some concern related to the comments on the approach to plugins. Plugins not having a product market fit reflects slightly negatively, on the companies powering the plugin and OpenAI. It becomes harder to get companies to invest time in creating plugins. I am not sure if the there was an option to remove just the issue sections of the post.
Also, why remove it even if requested? If you knowingly sit down for an interview, I don’t believe you have any power, legally speaking, to force that.
It’s hardly bad journalism to say no to such a request.
Access. If you don't play ball, you don't get invited to events like the one the article was based on. It's the same reason journalists adhere to embargoes.
The author knows that it's easily available. It doesn't matter if the original site retracted it. The author was also YC20 Humanloop. So they have a loyalty to Altman that most reporters wouldn't.
This could possibly be a pr move / marketing strategy to attract attention. Kind of Streisand effect — I have no idea what he said in the interview, but seeing it being removed ignites my interest
"OpenAI will avoid competing with their customers — other than with ChatGPT
Quite a few developers said they were nervous about building with the OpenAI APIs when OpenAI might end up releasing products that are competitive to them. Sam said that OpenAI would not release more products beyond ChatGPT.
Almost guaranteed this is it. It’s only a matter of time before Microsoft swallows OpenAI whole. And at that point there will be several competing products like Dynamics Copilot, etc. The lines are blurring, but I probably think the comment was fair and probably needs the right context: the research team within OpenAI will unlikely compete head to head with developers building apps on top of it, but why would it ever restrict itself to such a limitation? It doesn’t make any sense. That said, it means they will focus on their platform and have probably no line of sight to any apps they would build.
Any percent control can be 100% control if OpenAI relies on Microsoft for something critical like compute, and is contractually bound to not go to anyone else for compute. If MS can threaten to take their toys and go home, they can get anything they want.
That is an excellent point, but I have to think that MS wouldn't want OpenAI to walk to one of the other big cloud compute shops at the end of a contract. So, not quite 100% control. But that goes for the 49% limit too.
Classic Weinberg. It's called the "The Pinebrook Ploy".
"...Out here in the Nebraska prairie, where trees are scarcer than water, water is scarcer than hills, and the lay of the land lets you see for miles in every direction, there's no shortage of words describing desirable features. Everywhere you drive, you see signs directing you to Pinebrooks, Chestnut Highlands, Cottonwood Pond, Maple Glade, Willow Knolls, Aspen Meadows, Beechwood Shores, Elm Creek Hills, Oakmont Cascade...
...But why go on? The Pinebrook Ploy is so universally practiced that you can give odds that Oakmont Cascade contains neither oak, nor mountain, nor cascade, unless you count the stream that flows through your cellar. Besides, developers are no more guilty than technologists. Early proponents of gas lighting declared that gas light had the same healthful properties as sunlight. Under this application of The Gilded Rule, anyone forcing children to work twelve hours a day in a gas-lit factory was actually a great humanitarian. Without factory work, those children might not get the benefits of light on cloudy days..."
I don't care what the official line is, something changed with it. It might be the same model but the way it answers questions and the content of those answers is much more dubious now. I think how it manages its conversation context was gimped in the web chat version. I find it routinely completely forgetting things in longer conversations.
It is the text generation speed that got me suspicious. It seems like ChatGPT's GPT4 suddenly got 2-3 times faster at generating text, while the API GPT4 still generates the text at the same pace as it always has.
I was using it to experiment lazily with machine learning and midi generation via CNN-LSTMs and noticed it happening in the middle of a chat with it. The context wasn't that long and it was providing excellent insights, then randomly started telling me how to make a music game in python and lost all previous context. I tried starting a new chat and adding the back context,had a good bit of code I was providing for context, and it could hardly keep track of the provide code context over the course of a few simple questions/changes, when before it had no problem and would actually surprise with drawing from long past context relevantly. After what ever happened it would randomly assume karras was being used if having not been provided any pytorch specific code for a few messages.
API doesn't have these issue, or didn't a week ago when I was using it often, but something happened with the web model, which I guess is understandable, as the main reason I was using it vs the API was cost savings.
I was thinking how I could optimise for longer conversations and the most obvious thing seemed to be to somehow summarise the conversation ever X messages and ask it to save what it things is important isn’t them.
If they’re doing something like this that would probably explain it.
But GPT 3.5 has Turbo but nobody claims it reduced accuracy even though it's much faster.
GPT 4 is now a lot faster and struggles much more. And in theory being faster should be an advantage to an extent, even if the model needs more prodding, as you gain a more seamless back and forth instead of waiting half a minute.
Some people say that people think it's nerfed because when it was slow we associated it with "thinking harder" but I doubt that's the case, and the way to measure that is by loss of ability, i.e. if it is stuck more often as a result of itself and not a bad prompt.
Guess I missed that 3.5 Turbo achieved speed by also downgrading. I didn't hear anyone complain, to be fair I only use GPT 4 because I try to use it to save time and even with 4 you have to verify.
And I guess the trend is that anyone who uses it frequently uses 4 and 3.5 for basic stuff only.
Is it possible this was supposed to be on-background and not published for public consumption and there was a miscommunication? When I read the article last week, I was a little surprised it was published because it did read very much like the sort of meeting you have with people behind closed doors that aren’t secret the same way it would be under and NDA, but that are truly designed to offer background context and not to be published as a transcript. That scenario feels more likely to me than that there was something said that is no longer true, but I don’t know.
From an optics perspective, it would probably have been better to just let the article stand rather than requesting that it get removed (unless there were fundamental misrepresentations of misunderstandings of what was said), but it could also be a situation where no one expected the article's removal to be noticed.
> but it could also be a situation where no one expected the article's removal to be noticed.
This seems least likely.
Hard to tell if it's misrepresentation or revealing secrets. Given how blunt, straightforward, and detailed the article was, I'd say it either revealed operational secrets that Microsoft or OpenAI didn't want to reveal. Especially those first few paragraphs that told specifica about shortcomings and what advancements are needed. Or it said they wouldn't compete in a broad category that they definitely will be competing.
Having been on both sides of these types of meetings, I don’t think the saying they won’t compete thing and that being false is realistic. If you meet with people/partners/influencers/press, you typically don’t lie like this, especially when you’re of the size/attention of OpenAI (there are always exceptions but I just don’t see it here). There are situations where you don’t want what are your legitimate and current plans put out in the world in writing by people other than you, especially if they are things that could change. I think there is a scenario where OpenAI today doesn’t intend to compete with its API users, but doesn’t want to make that a definitive statement when it might not know the future could hold or to account for parallel lines of development (if OpenAI was already working on an area and API users decide to get into that space too, how does that work. That happens at companies that offer both platforms and their own products all the time and it is rarely nefarious (obviously, sometimes it is), but I can see how that wouldn’t want to be a public message OpenAI would want out there, particularly if it isn’t directly from them.
That said, I really do have to think this was a meeting that was designed to be on background and not represented as “OpenAI reveals their roadmap” and that this was a miscommunication from the jump.
Just for reference, having been in these sorts of meetings as both a reporter/analyst and as the person providing information, it isn’t uncommon to have a set of guidelines known as “Chatham House Rules” [1] or something similar, where you can use the information, but not state it verbatim from notes or directly quote. But people that aren’t primarily reporters or analysts or lobbyists whatever, that isn’t always understood. Which is why you need to be explicit in the terms of the publicness of the meeting.
Again, this is only a guess (but it is backed up by 15 years of professional work as both a reporter/analyst as well as someone who is often sent to talk to the press/influencers/customers/users in situations that have varying degrees of on/off-recordness, I have to think this was just a misunderstanding of how publicly the contents of a conversation were designed to be.
Again, I don’t think that asking people to take something like this down is the right move most of the time, especially if the article already got attention and write ups. Just chalk it up to a loss and commit to explaining expectations better in the future (if someone was clear on the rules of disclosure and violates them, that’s entirely different. I have zero reason to believe that happened here). If someone posts something and it doesn’t hit the front page of hacker news and you catch it in the first five or ten minutes, yeah, asking to take it down might be OK — but in general, I say let it go.
That said, I can’t help but wonder if it was a strategic mistake, not realizing that the takedown of the article would be noticed.
The truth is, we won’t know unless Raza or Sam comments, and neither has a real incentive to comment IMHO.
That’s possible for sure, but knowing Sam (and having interviewed him going as far back as 2007 or 2008 when we were both babies), it’s hard for me to believe he’d say something like that in an on-record interview. I could be wrong, obviously, but I just don’t find it likely.
I find it more likely that it was intended to be more on-background and that they wasn’t communicated appropriately — but again, I could be wrong.
> OpenAI will avoid competing with their customers — other than with ChatGPT
This is a nearly impossible promise. The most popular apps are broad platform shortcomings, and hard for the platform to guarantee they won’t encroach and solve a problem better solved natively.
Exactly, he can't commit to it. He could have said "currently, it's not in OpenAI's plans to create any other app on top of GPT other than Chat". But commiting to a non-compete? Nah, this is too much...
they love hearing their own voices and seeing their own thoughts in print, which translates to a severe inability to just be quiet and let the machines grind the meat.
It doesn't matter, they probably retracted for legal reasons. If you store a copy, it's one thing, but they stated publicly now that they don't stand behind those words anymore.
This is slightly off-topic, but I'd love to know what AI companies – or emerging tech companies in general – have founders that say what they mean, and aren't talking out of both sides of their mouth.
Reading between the lines of the original article:
1. there are plenty of - consumer - GPUs, but it wouldn't be advantageous for OpenAI to fine tune the models to run on them, hence the artificial limitation to only use GPUs with 80+ GB. We already see plenty of projects working on adapting these models for 16-24 GB GPUs or even running on CPUs - that is obviously a problem for OpenAI
2. GPT-4 has been nerfed recently - there are plenty of discussions on this and I've also noticed it in my extended usage of both ChatGPT Plus and the OpenAI API.
Also, as a developer (disclaimer) of a project on top of the OpenAI API, I've received multiple messages from users mentioning the issue. Even in the OpenAI discord there are multiple threads of developers complaining about the change to GPT-4.
3. Plugins do have PMF - in the OpenAI discord there are multiple threads about accessing plugins via the API. There are also many projects working specifically on this (AgentGPT, repository loader etc).
My personal experience: as a developer, you can develop plugins but you can't charge for usage. At the same time, plugins are available only to paying users of ChatGPT Plus ...
4. OpenAI will wait and see what 3rd party projects get traction and once they see public interest in a project, will build that project.
I'm willing to bet the ChatGPT iOS app was built in a couple of weeks (at most) and was a direct answer to the multitude of apps on the App Store.
5. OpenAI needs the cover of regulation. They developed their models without it but now regulation could help to prevent others from reaching their levels.
At the same time, any open source projects that gain enough public interest will be copied/absorbed.
In general, the quality of the replies seems to have been affected.
More specifically:
- the context window seems to have been reduced: I only have access to the 4k model, but even when the prompts and replies are well under this limit, the model seems to loose context, as in, the replies have no connection to the prompt;
- hallucinations: while this has been a general issue with LLM's and even taking into account a reduction in the context window, GPT-4 seems to hallucinate a lot more now;
- not answering: prompts for which I have received answers before get a "my cut-off date ..." answer now.
I used GPT-4 mostly via the API and I saved the threads (history) of all prompts I've been sending, so it's fairly easy for me to "test" the before/after changes.
Some tests that are easy to run:
- the "apple" test: write 10 sentences that end with the word "apple" - I used to get 8/10 most of the time, sometimes even 10/10. Now it's more like 6-7/10.
- a friend suggested the "jwst" prompt: when was the jwst launched? Depending on the answer (sometimes it says it is scheduled to launch, sometimes it answers correctly), subsequently ask "what hour and minute"
- use a 1-2k prompt and analyse the answer - this shouldn't be an issue with a 4k limit, but sometimes it will hallucinate or respond with something totally unrelated
edit: I've noticed that speed has been improved, but that doesn't really help when the quality suffers; also, data point of 1, but when using it from Germany (Europe?), the quality suffers, when prompting via vpn from US, it seems to be slower but the quality is better (but again, take that with a big grain of salt)
I like what Sam Altman suggested. OpenAI should open source all aspects of ChatGPT immediately and be reimbursed via tax revenue in the form of totalCostsThusFar * 7 so that each employee gets 7 years of their compensation paid for, so they can take a sabbatical and then contribute to building plugins.
Run the model itself and the plugin store as distributed simple software that runs on the common internet backbone.
Make it as simple as Google search used to be, but with the resources of the government behind it. Privacy and security guaranteed by the Constitution and the American military. Free chalice of knowledge for all.
Then roll out the same compensation structure for all who build plugins people want. We know what plugin and api iPhone moments look like.
Make training additional models 1% of American GDP each year moving forward and reinvest that into plugin development and general self development grants for technology companies that make a difference in making things better.
What makes you think total costs and total salaries are even close to the same? What makes you think the company getting that amount of money means the employees getting the same amount?
I mean the blind spots you have are just way too numerous, large and fundamental to even bother with your comment.
Those questions aren't blind spots at all. Those types of problems are very easy to solve, just think about it like a child would. "Keep paying people for 7 years"
Every employee at Open AI gets their current comp for the next seven years. The idea is to turn this into an open library or model in its entirety so that anyone can run it on their own harwdware and so that training costs go down more.
My opinion of him changed when he started fear mongering in an attempt to exaggerate the current abilities of machine learning and present OpenAI as more impactful than it actually is.
Speaking as a dev building products on the OpenAI APIs, I thought it was a super helpful article when I first read it, so I was sad to see that it was removed.
I suspect they have to release more products to climb up the value chains where their technology is currently being used to generate value for customers.
Maybe removed because of this "...said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs..."
Probably got a phone call with Area Code 206 after that...
The skeletons will slowly coming out of the closet now. People's motivations should never be taken at face value and neither should any response to why it was redacted. Everyone should be given the benefit of the doubt but you also don't hand over 50% or more of a company to a large corporate after decades in startups and literally running an incubator unless you have alterior motives to the altruistic one put forth. Elon Musk is a tyrranical actor in the system but he was also flagging something pretty clearly with the profit model shift. Everyone has a number and it always comes down to power, control and influence. Having "impact" is about being the one influencing the decisions and direction of not just technology but politics and policy too. So much more could be said on the topic and again we have to give everyone the benefit of the doubt but people who have alterior motives are always very obvious because they say and do exactly what you think they should to be the most positive influence aka SBF.
For those of you gathering Altman pronouncements to train your AltmanGPT model that spits out 100% disingenuous and self-serving quotage, here is what he said that is sooooo sensitive:
OpenAI’s plans according to Sam Altman
Last week I had the privilege to sit down with Sam Altman and 20 other developers to discuss OpenAI’s APIs and their product plans. Sam was remarkably open. The discussion touched on practical developer issues as well as bigger-picture questions related to OpenAI’s mission and the societal impact of AI. Here are the key takeaways:
1. OpenAI is heavily GPU limited at present
A common theme that came up throughout the discussion was that currently OpenAI is extremely GPU-limited and this is delaying a lot of their short-term plans. The biggest customer complaint was about the reliability and speed of the API. Sam acknowledged their concern and explained that most of the issue was a result of GPU shortages.
The longer 32k context can’t yet be rolled out to more people. OpenAI haven’t overcome the O(n^2) scaling of attention and so whilst it seemed plausible they would have 100k - 1M token context windows soon (this year) anything bigger would require a research breakthrough.
The finetuning API is also currently bottlenecked by GPU availability. They don’t yet use efficient finetuning methods like Adapters or LoRa and so finetuning is very compute-intensive to run and manage. Better support for finetuning will come in the future. They may even host a marketplace of community contributed models.
Dedicated capacity offering is limited by GPU availability. OpenAI also offers dedicated capacity, which provides customers with a private copy of the model. To access this service, customers must be willing to commit to a $100k spend upfront.
2. OpenAI’s near-term roadmap
Sam shared what he saw as OpenAI’s provisional near-term roadmap for the API.
2023:
Cheaper and faster GPT-4 — This is their top priority. In general, OpenAI’s aim is to drive “the cost of intelligence” down as far as possible and so they will work hard to continue to reduce the cost of the APIs over time.
Longer context windows — Context windows as high as 1 million tokens are plausible in the near future.
Finetuning API — The finetuning API will be extended to the latest models but the exact form for this will be shaped by what developers indicate they really want.
A stateful API — When you call the chat API today, you have to repeatedly pass through the same conversation history and pay for the same tokens again and again. In the future there will be a version of the API that remembers the conversation history.
2024:
Multimodality — This was demoed as part of the GPT-4 release but can’t be extended to everyone until after more GPUs come online.
3. Plugins “don’t have PMF” and are probably not coming to the API anytime soon
A lot of developers are interested in getting access to ChatGPT plugins via the API but Sam said he didn’t think they’d be released any time soon. The usage of plugins, other than browsing, suggests that they don’t have PMF yet. He suggested that a lot of people thought they wanted their apps to be inside ChatGPT but what they really wanted was ChatGPT in their apps.
4. OpenAI will avoid competing with their customers — other than with ChatGPT
Quite a few developers said they were nervous about building with the OpenAI APIs when OpenAI might end up releasing products that are competitive to them. Sam said that OpenAI would not release more products beyond ChatGPT. He said there was a history of great platform companies having a killer app and that ChatGPT would allow them to make the APIs better by being customers of their own product. The vision for ChatGPT is to be a super smart assistant for work but there will be a lot of other GPT use-cases that OpenAI won’t touch.
5. Regulation is needed but so is open source
While Sam is calling for regulation of future models, he didn’t think existing models were dangerous and thought it would be a big mistake to regulate or ban them. He reiterated his belief in the importance of open source and said that OpenAI was considering open-sourcing GPT-3. Part of the reason they hadn’t open-sourced yet was that he was skeptical of how many individuals and companies would have the capability to host and serve large LLMs.
6. The scaling laws still hold
Recently many articles have claimed that “the age of giant AI Models is already over”. This wasn’t an accurate representation of what was meant.
OpenAI’s internal data suggests the scaling laws for model performance continue to hold and making models larger will continue to yield performance. The rate of scaling can’t be maintained because OpenAI had made models millions of times bigger in just a few years and doing that going forward won’t be sustainable. That doesn’t mean that OpenAI won't continue to try to make the models bigger, it just means they will likely double or triple in size each year rather than increasing by many orders of magnitude.
The fact that scaling continues to work has significant implications for the timelines of AGI development. The scaling hypothesis is the idea that we may have most of the pieces in place needed to build AGI and that most of the remaining work will be taking existing methods and scaling them up to larger models and bigger datasets. If the era of scaling was over then we should probably expect AGI to be much further away. The fact the scaling laws continue to hold is strongly suggestive of shorter timelines.
I wish he would make a blog post about his plans then.
Biggest question for me is, when is the next version after gpt-4-0314 coming out?
Also it would be good for people to know the real situation with GPUs or AI accelerators that can handle their LLMs.
Such as, did they order them and are now literally waiting for them to be manufactured, or are they installing them in a datacenter? Any kind of peek into those operations might help people understand the challenges so they complain less.
Also this is tangential but is there a factory and design center in North America that can create and manufacture the necessary level of AI hardware? Or does OpenAI's ability to take on new customers down the line depend on Taiwan and holding off WWIII?
I guess a lot of things kind of depend on that. But I don't see any good plans on how to avoid it.
Pretty sure he said a few months ago that scaling further than GPT 4 isn't going to help enough, and that other breakthroughs are needed or something about building on top of GPT 4. Since then the only foundational breakthrough was much bigger context windows.
Obviously to even run and train GPT 4 now is too compute intensive, though yeah.
Were you limited by GPU capacity, would you intentionally downplay the potential benefit to further scaling the model so as to discourage competitors (e.g., Google) who would be less GPU-limited from pursuing such a direction themselves?
I don't necessarily think this is the case, but it occurs to me that were there any practical impediment (however emphemeral) to them commercially exploiting a larger scale model, they'd likely adopt a posture of dismissing further scaling until they were ready.
I heard that ChatGPT 4 runs in the red even with $20 subscription. Makes sense since only 25 messages every 4 hours. The nerf coincides with browsing taking more power up probably.
Google probably doesn't want people using GPT 6 tech instead of search entirely, which is a real possibility since Google is often filled with SEO spam nowadays. I think they just want LLM + integrated with ads. Google had better results like 15 years ago for general information.
Of course their nerfs are all intentional. When compute and techniques improve/scale they can unnerf. If they catch wind of a competing product they can also unnerf. They still have stuff developed they can't affordably deploy like multimodal, it can already take images.
I think the issue is that both the alignment stuff and performance in terms of reasoning is largely the result of the reinforcement learning that happened after that model was completed.
I don't see any announcement. You mean they just silently replaced gpt-4-0314 somehow? I think that if they changed something its actually not the model, but maybe something that affects the performance of the model.
Altman's dealing with being the face of a civilization-changing (or ending, who knows) technology and probably a horribly uncomfortable amount of sudden fame, I think people forget this when they're trying to pick apart everything he does and says. I'm not going to guess why he (or Microsoft, or whoever) wanted that taken down but the site agreed to do it.
