If you came here after only reading the headline, you missed what the complaint is actually about:
It's not that GPT-4 is closed source. It's that access to `codex` model was pulled with only three days notice, and the model itself was not open-sourced. Since apparently a large number of researchers were writing papers which used that particular model, that means all of those research papers are now non-reproducible.
An obvious thing to do would be to either open-source older models (including the weights) when retiring them; or possibly transfer them to an institution who see their role specifically as serving as an archive / reference for this type of purpose. Open-sourcing older models shouldn't result in too much of a risk, either from an "AI Safety" perspective, or from a competitive perspective.
I think what most of the people here are missing is how big, how paranoid, and how influential the "AI alignment" movement is. To you it looks like they're being overly careful and paranoid, perhaps as an excuse to set up a monopoly silo to extract money. But a lot of the people the OpenAI researchers work closely with -- people deep in the "AI alignment" community -- are telling them that they're being wantonly reckless, helping set the human race on a path for certain doom. There are people in that community -- people not working for a for-profit company -- who would, if they could, stop all AI research of any kind until we have rock-solid techniques to prevent an AI apocalypse. Most of those individuals have absolutely nothing commercial to gain from stopping AI research.
So suppose you're an AI researcher at OpenAI. A large number of people you know and respect are telling you that you're driving the human race right towards a cliff. You don't 100% agree with their assessment, but it would be foolish to completely ignore them, wouldn't it? Obviously that's going to affect your opinions about things.
From everything I've heard and seen, the actual researchers at OpenAI are trying to take seriously the risk that a super-intelligent AI might destroy the human race.
Here's one example: GPT-4 was actually done back in August of last year. If their goal was to maximize profit, the obvious thing to do would be to release API access to it as soon as possible. But instead, they purposely delayed release for eight months, specifically in order to "cool down" the "arms race": to avoid introducing FOMO in other labs which would lead them to be less careful.
Go lurk on alignmentforum.org for a while, and you'll have a different perspective on OpenAI's decisions.
> Here's one example: GPT-4 was actually done back in August of last year. If their goal was to maximize profit, the obvious thing to do would be to release API access to it as soon as possible.
They did that, that's how Reid Hoffman got early access to write his book, that's how Microsoft got access to start working on Bing/ChatGPT4 for a cool $10 billion and that's how countless of other got early access. Got the money, got the marketing and got the synchronized deployment of multiple use cases by a selected crew of companies and they get to say the corpspeak of 'we care, we didn't release on August!'. This is taken from the Apple iOS SDK book, you make the API changes and release privately to have the announcement and a parade of implementations by third party to prove that it is viable.
Besides wasn't releasing GPT3 supposed to have caused major harm to society? Which is why they held off for so long. Still waiting for evidence of that harm (mass fake news, Google being ruined by even more low ranking spam sites, etc).
It must be nice thinking that a small group withholding the keys R&D (for a short while until other R&D groups catch up) will somehow help the problem. Do these few months to a year really provide much value in finding ways to stop the "AI apocalypse"? What real work are they doing to prevent it in those few months? More philosphizing and high level analysis?
It might work for messaging/marketing that they are being "careful" but I'm not convinced this is tangible. Seems as arrogant and naive as most AI ethics stuff I read.
My memory was that they were worried about GPT-2 if society weren't ready for it. So they've been trying to make people aware of what its capabilities are. I think ChatGPT really did an amazing job of that, as I said. Now everyone knows that computers can write low-quality drivel for pennies a paragraph, and as a society we're starting to adjust to that reality.
Just to be clear you think OpenAI achieved this by holding off releasing it for a short period? And this achieved mainstream penetration? Or among programmers?
IMO stuff like deep fakes didn't become real until people started seeing it IRL. They weren't reading FUDy posts on HN or academic papers. Even the niche tech posts on NYT rarely get more than a few hundred thousand people reading them.
I have no idea what would have happened if they'd just dumped GPT-2, weights and all, into the world when it first came out; or even if they'd just gone straight to a paid API. They didn't know either. I think given that nobody knew, their strategy of "try to warn the technorati, give access slowly and make sure nothing bad happens, then make a widely-accessible interface" seems like a reasonably cautious approach.
No one discusses the elephant in the room: who elected these elites to decide what was and wasn't ethical and responsible? Nobody.
So who ends up making the ethical decisions? A group of highly privileged SV types insulated from the very real problems, concerns, and perspectives of the ordinary person.
This is just more of what humans have been doing over millennia: taking power then telling everyone else it was too dangerous for them to wield.
Who elected you to do… whatever it is you do? Probably someone hired you because they thought you’d be good at it. Or maybe you were good enough and cocky enough that you just went and did it, and sold the result.
Either way I’d imagine they’re in their roles for the same reason.
I'm not sure this is entirely fair.
Nobody elected the people who inspect nuclear powerplants either, but I still assume that they're doing a good job in protecting humanity. Even if they are possibly "highly priviledged".
Inspecting a powerplant does not grant you any actual influence beyond powerplant inspection. Defining the ethics of AI will potentially let you influence almost all aspects of our lives.
Unfortunately in many industries, companies are allowed to regulate/inspect themselves. Because they supposedly have more experience with it than the government, and it saves on government spending.
> who elected these elites to decide what was and wasn't ethical and responsible? Nobody
First: basically every American literally voted for that by repeatedly saying no to the alternative (the communist party) in every American election.
Second: what exactly and specifically are you suggesting here? Because even outside of capitalism, the alternative to "people deciding they personally don't feel it's safe to release a product they created and worked on and know more about than literally anyone else" sounds like actual literal insanity to me.
1) less than half of Americans vote in each election (less than 63% if you restrict to the voting-age population, less than 70% if you apply the scummy rules that restrict to the voting-eligible population)
And 2) it's a false dichotomy to say that US elections have ever been "whatever we have now VS communism". Maybe you could say socialism was on the ballot all those times Eugene Debs ran for the presidency, but there hasn't ever been a communist on the ballot that I'm aware of. Also, it sounds like you would struggle to define communism if pressed.
Regarding your second loose point, the US restricts the sale of a lot of products to the public (eg nuclear weapons, biological weapons, raw milk, copyrighted works you don't hold the copyright to, etc). Personally, I think it's pretty reasonable to restrict the sale of some things, even if the potential sellers know a lot about the product.
> basically every American literally voted for that by repeatedly saying no to *the alternative* (the communist party) in every American election.
I had a good laugh playing with this ridiculous framing, thinking about all of the candidates we've said no to.
* "Get out of here Donald Trump! We don't want communism, we want the alternative; Joe Biden!"
* "Hit the bricks secret pamphlet-loving marxists John McCain and Mitt Romney, we'll take the singular alternative: Barack Obama"
* "We love Jimmy Carter, he's the opposite of communism! Nothing like the alternative, an all-star college football player and rabid communist manifesto adherent named Gerald Ford."
* and "We hate Jimmy Carter who must be a communist because of how hard we voted for the movie man."
* "Give us Teddy Roosevelt, he'll smash up all of these monopolistic robber barons, because TR is the alternative to Marxism."
* "FDR, we love you so much we'll elect you to the Presidency four times! We all thought Herbert Hoover was in the pocket of gilded age capitalists, but when Hoover drove us into the great depression, we realized he must have really been a bolshevik! Thank you so much for the massive welfare state expansion, FDR, you truly earned your nickname 'FDR: cure for the common communism'"
Ridiculous.
But in all earnestness, the communist manifesto has never been even remotely relevant to any US election ever. And I mean this with no malice, but if you think lobbing the label "communist" at something you don't like is an argument, vary up your media diet and be recognize when you use logical fallacies in arguments so you can slow down and debug your thought process.
I think it's really obvious that Americans don't want those things, so one way to rephrase my original comment could be "the rejection of communism is why rich people get to own and control businesses".
> "the rejection of communism is why rich people get to own and control businesses".
is as wrong as saying "the rejection of [pastafarianism | soccer/futbol | anarchocapitalism | mandatory left-handedness | manual transmission cars | etc] is why rich people get to own and control businesses".
Communism and the communist party have never been part of the question. The closest "Communism" came to being part of the political landscape was when a power-hungry alcoholic grifter named Joseph McCarthy won a Senate seat in Wisconsin and then started a paranoid campaign of lobbing unsubstantiated accusations of secret communistic allegiance at academics, civil servants, members of the media, and of anyone he wanted. It whipped up a frenzy of anti-communist protestation, but not because people knew anything about communism, rather people rejected being called "communist" not because they knew anything about the economic theory of communism, but because McCarthyism made that term a career-killer. McCarthy starting attacking leadership of the US military, alleging that they were infested with communists, and organized hearings in the Senate that were essentially modern day witch burnings. From 1946 to 1954, McCarthy whipped up a massive panic while smearing the symbolic label "communism" with so much shit that essentially no one can think clearly about the ideas behind that system of social organization. In 1954, other Senators countered with a campaign to censure McCarthy for his invalid and unwarranted abuse of US Military Generals, culminating in a vote to condemn McCarthy (67 votes to condemn, 22 against condemning). After this humiliation, McCarthy wasn't decent enough to resign his seat and leave voluntarily, but fortunately he died of cirrhosis of the liver about 2 years later at the age of 48.
In short, the claim that "the rejection of communism", something no one here spends any time thinking about, "is why rich people get to own and control businesses" is ridiculous and evidence of a broken thought process. I refer you to my prior advice about dealing about slowing down and recognizing when you've built your beliefs on logical fallacies.
You're arguing against an imaginary totem instead of what I actually wrote.
> Communism and the communist party have never been part of the question
De facto/de jure. Nobody wants it (de facto), I've demonstrated by linking to the actual parties that de jure it's totally been an option.
Those parties I linked, you could have voted for, but y'all didn't.
Given I've explicitly said I'm not talking about dem/rep culture war nonsense, your over-detailed rant about McCarthy (who, you may be surprised to learn, was sufficiently relevant to your politics that his actions are well known on the other side of the Atlantic and his name is likewise used as a derogatory term) was a waste of your own time.
I am specifically and literally referring the idea of private ownership of the means of production, which is in the actual literal Manifest der Kommunistischen Partei as written by Karl Marx in 1848.
Which I have in fact read.
Section 2, English translation, has the following passage:
"""
The proletariat will use its political supremacy, to wrest, by degrees, all capital from the bourgeoisie, to centralize all instruments of production in the hands of the State, i. e., of the proletariat organized as the ruling class; and to increase the total of productive forces as rapidly as possible.
""" - https://en.wikisource.org/wiki/Manifesto_of_the_Communist_Pa...
That, right there, is why not having Communism in the USA means that rich people get to keep their stuff out of public hands.
Now, do you know some other political ideology besides communism that wants to remove control of factories from their owners? Because that would be an additional option beyond the dozen or so communist parties of the USA coming 7th-25th in a two-horse race, and the complaint being made against OpenAI is that it's not letting users do whatever with the tools that OpenAI made and own.
So far as I am aware, none of
> the rejection of [pastafarianism | soccer/futbol | anarchocapitalism | mandatory left-handedness | manual transmission cars | etc]
Have any causal connection to
> why rich people get to own and control businesses
(Maybe anarchocapitalism?) But that's the specific point of communism.
Which you as a nation reject, even though they're a thing you're not, AFAIK, banned from voting for.
I mean, reading what you've written, you and I definitely agree 100% that communists are not politically viable in the USA. I'm just saying that the logical consequences of their non-viability includes "rich people can own and control businesses".
> Those parties I linked, you could have voted for, but y'all didn't.
I've voted in every national election (primary and general) in the past 15 years, and those parties have never been on the ballot. It takes a massive amount of money to run successful campaigns, and political donors make a big difference in choosing which candidates (and therefore which platforms) people get to vote on. This significantly biases the ideological space and makes it invalid to look at the outcomes of elections and draw conclusions about vague ideas that were nowhere near making it onto any major party candidate's platform.
I'm not a proponent of communism; I'm a big fan of property rights, but in any case, my core allegiance is to the scientific method and to seeing reality clearly. You really want to reach a specific conclusion but the claims you're using to build your path to that conclusion are not factual. You should start from a close inspection of the actual processes and behaviors of systems, rather than starting with your conclusion and trying to cobble together a case for your conclusion.
Until the alignment movement begins to take seriously the idea that we already have misaligned artificial general intelligences I think they are best viewed as a convenient foil
Paperclip maximizers exist, they're made not only of code but of people
It’s weird to have been working on a paper for almost a year and have it launch into this environment, but uptake has been good. My hope is that we will continue to see more nuance around different kinds of alignment risks in the near future. There’s a wide spectrum between biased statistical models and paperclip maximizing overlords, and lots bad but not existentially catastrophic things for the public to want to keep a pulse on.
Thanks! Looks like good work. I hope this idea continues to get traction:
> In some sense, we’re already living in a world of misaligned optimizers
I understand this is an academic paper given to nuance and understatement, but for any drive-by readers, this is true in an extremely literal sense, with very real consequences.
Precisely! I'm much less concerned about super-intelligent AIs and much more concerned with shortsighted, greedy humans using pretty-good AIs (like those we have now) to squeeze out every ounce of profit from our already misaligned systems, at the expense of everyone else. Not to mention the political implications of being able to convincingly fake voices, photos, and videos.
In this sense, I'm pleased to see Open AI claim to be taking a more careful stance, but to be honest I think the genie is already out of the bottle.
Reminds me of the parody in Scott Alexander's article "If the media reported on other things like it does EA"
> Some epidemiologists are worrying that a new virus from Wuhan could become a wider catastrophe. Their message is infecting people around the world with fear and xenophobia, spreading faster than any plague. Perhaps they should consider that in some sense, they themselves are the global pandemic.
Like, yeah, people did consider that idea, and the "corporations are the real unaligned AI" idea, and the "capitalism is the real extinction risk" idea, and all the pseudo-clever variations of the concept.
The problem is that "understanding that capitalism has problems" isn't equivalent to "having an actionable plan to solve capitalism".
> The problem is that "understanding that capitalism has problems" isn't equivalent to "having an actionable plan to solve capitalism
This is a caricature and the same thing could be said of AI x-risk. There are plenty of ideas on how to avoid unwanted effects of economic systems. I don't think it's at all clear that it's a single problem with a single solution. Getting ideas into practice tends to be the tougher challenge.
More broadly, the point is not to say "wow, alignment problems have existed for a long time already!" This is not profound or clever, it's obvious. But there's a big group of people considering a narrow definition of the problem, and playing what could be considered a useful social role.
Okay, but their actions are _not_ stopping AI research, they are doing plenty of AI research internally. They're just hindering competitors and non-profit researchers.
I suppose you could make an argument that nobody can be trusted to do AI research as responsibly as them, so that's why they should not share anything and should hinder others' research... but it kind of looks like plain old nothing-to-see-here profit-oriented decision to me. Which isn't necessarily a scandal, they are a profit-oriented company of course (although they try to take advantage of the misperception that they aren't).
But if they really took those "alignment" concerns seriously, wouldn't they be seriously slowing down or even stopping their own research too?
"Virtue signaling" is overused but highly relevant here. Absent some proof they've done anything at all to prevent an AI takeover (which surely would have to be open source to be valuable too right?).
> I think what most of the people here are missing is how big, how paranoid, and how influential the "AI alignment" movement is. [...] If their goal was to maximize profit, the obvious thing to do would be to release API access to it as soon as possible. But instead, they purposely delayed release for eight months [...] Go lurk on alignmentforum.org for a while, and you'll have a different perspective on OpenAI's decisions
I'm familiar with the "AI safety" movement. For years, many people from that camp are extremely critical of OpenAI and they genuinely believe OpenAI is unleashing something truly dangerous to humanity. One person I knew said that while free and open source is usually important, but due to the unique dangers of AI, it's better to keep AI tech stay in the hands of a small number of monopolists, similar to nuclear non-proliferation. Meanwhile, OpenAI was trying to promote openness - a terrible idea.
Thus, it's indeed a perfect explanation of OpenAI's decision to stop keeping its research in the open. Unfortunately, the problem here is that the "for profit" and "AI safety" explanations are not contradictory, they can simultaneously be true. Just like how Google began as a promoter of the open Web but gradually started to use its market position for its own gain. The same situation exists for OpenAI. "AI Safety" may be the initial motivation, but possibly not for long. After a while, "safety" may be nothing more than an excuse for profit.
> free and open source is usually important, but due to the unique dangers of AI...
Sadly, cherished principles often perish on the horns of "But this time it's different."
> the "for profit" and "AI safety" explanations are not contradictory
Indeed, they can reinforce each other into a runaway feedback loop. Once you buy into an all-encompassing mission of preventing apocalypse, maintaining perspective or proportionality become almost impossible. The moral hazard of not pursuing almost all available measures justifies taking $10B of MSFT's money to fund the defense of humanity. Add to this the ego-stoking existential importance of such a "noble cause", the global media attention and the social elevation in the tight-knit, closed-circle of the AI Alignment community and you've got the perfect drug.
Given the intense forces shaping the worldview of the "AI Safety Noble Warriors", it's reasonable for the rest of us to question their objectivity and suspect claims of "we are keeping this from you for your own good."
>Most of those individuals have absolutely nothing commercial to gain from stopping AI research.
There are always financial incentives. Like it or not, there's a lot of money on the line in the "AI" industry; if someone wants that industry to go a certain way, they definitely have something to gain or lose financially.
In particular, it's obvious to anyone who's been paying attention that the west halting/ceding AI research only means the likes of China will just come out ahead from not bothering to stop (spoiler alert: China cares not for trivialities like ethics and morals).
A useful question to ask yourself is, "How would I know if I were wrong? What kind of evidence would convince me that a decision was not driven primarily by financial incentives?"
If your "model" is equally compatible with all possible observations -- if anything that happens actually confirms the model rather than disproving it -- then it's not actually that useful as a model.
> In particular, it's obvious to anyone who's been paying attention that the west halting/ceding AI research only means the likes of China will just come out ahead from not bothering to stop (spoiler alert: China cares not for trivialities like ethics and morals).
Right, and that's why I said "would if they could". From their perspective, saving the human race would require stopping all research, including research done in China.
It's possible, but the views of the AI alignment community so far as I can tell are being skewed way too far towards nihilistic doomerism by the influence of Yudkowsky, who apparently believes that we're all gonna die in a few years and there's nothing anyone can do to stop it. [0]
^ Thanks for that link. The doomerism is brilliant and clear and imaginative and absolutely worth reading and grappling with. I personally have no good response to how we deal with sufficiently advanced AI’s capacity to trick and manipulate us into doing catastrophically bad things.
His argument is essentially "a superintelligence who is better than us at everything and thinks a quadrillion times faster can do whatever it wants and we are powerless to stop it."
Yeah, I could've told you that.
If we really are going to create such an intelligence in the next five years, then we had a good run, so long and thanks for all the fish. But that assumption coupled with the security mindset he brings to the table (viz. "the only unhackable computer is an unplugged one at the bottom of the ocean") is so strong that the big list of doom vectors he comes up with appears much scarier than it actually is.
In the past I've struggled with intrusive thoughts that the government is going to come and murder me. I could have given you reasonable-sounding explanations for why I believed this. Doesn't mean it's gonna happen.
AIs can do science, write code, impersonate people, and manipulate people. We’ve already got AlphaFold and ChatGPT and Copilot. People are moving full steam ahead with AI software developers and scientists who have access to deploy code and spend money and communicate with humans autonomously.
I don’t think it takes a whole lot of imagination to see these things improving and coming together in way that an AI agent could feasibly design and execute a plan to develop a bio weapon or deadly nanotech. His points are about how hard it is to prevent that with our current AI training regime.
His analogy, for example, between the human “inclusive fitness reward function” (we evolved with the role purpose of survival and reproduction) and RLHF-style human feedback for AI is apt, and not obvious. Just because we “evolved to survive” didn’t prevent groups of humans from developing the exact opposite capacity to make us extinct.
Mmm. The definition I used is the standard definition of superintelligence, used by EY himself:
> A superintelligence is something that can beat any human, and the entire human
civilization, at all the cognitive tasks. [0]
I added the "thinks a quadrillion times faster" part but I think that's fair, if perhaps off by a few orders of magnitude.
All of EY's work has an often unstated assumption that AI will adversarially try to kill us. This is explicitly noted in the replies to the AI ruin post. There are a lot of fiddly details that I'm skipping over, but I stand firm that his argument reduces to "it's functionally impossible to stop something faster and smarter than us that really wants to kill us from killing us once it gains sentience."
Your reduction doesn’t cover the real risks of orthogonality, instrumental convergence, the zero margins for failure on the first try, intelligence explosions, and the impossibility of training for alignment.
Yeah, those only make it worse, but they only really apply to a bona fide superintelligence of the sort EY describes, and those are not what we have. I don't believe we're particularly close to having one.
If we're doomed, we're doomed, but please don't tell me about it.
Thinking and pontificating about AI safety is literally his job, and Less Wrong is a thing he founded, so whatever else Yudkowsky pontificating about AI safety on Less Wrong might be, it isn't "popping up in the most random places".
1. That an AGI which was significantly more intelligent than humans could destroy us if it chose
2. That it's possible that such an AI could be created in the next decade or two, given the current trajectory.
And so, I think we definitely need to be careful, and make sure we don't blunder into the AI apocalypse.
However, there are several further assertions which are often made which are part of the "we're all doomed" scenario:
3. There would be no signs of "misalignment" in not-quite-as-capable AGIs.
4. Even if there were signs of misalignment, that at least some AI research groups would continue to press on and create a mis-aligned super-intelligence
5. Even if we learned how to align not-quite-as-capable AGIs, those techniques wouldn't transfer over to the super-intelligent AGIs.
It's possible all of those things are true, but a) 3 and 5 are not true of biological general intelligences b) give our experience with nuclear weapons, I think 4 is likely not to be true.
So re number 3: When you have severely "mis-aligned" people -- sociopathic humans who end up performing atrocities -- there are usually signs of this tendency during development. We have far more license to perform "what-if" testing on developmental AIs; I think it very likely that if AGI-1 or AGI-2, who are "only" as good at planning as a 7-year-old, have severe mis-alignment risks, that this would be detectable if we're looking for it: that if it's likely to destroy the world, and we try to give it opportunities to destroy the world in a simulation, that it will show its colors.
Re number 4: Many world leaders thought scientists were over-reacting about the risk of nuclear weapons, until they saw the effects themselves. Then everyone began to take the risk of nuclear war seriously. I think that if it's demonstrated that AGI-1 or AGI-2 would destroy the world if given a chance, then people will start to take the risk more seriously, and focus more effort on methods to "align" the existing AGI (and also further probe its alignment), rather than continuing to advance AGI capabilities until they are beyond our ability to control.
Re number 5: Children go through phases where their capabilities make sudden leaps. And yet, those leaps never seem to cause otherwise well-adjusted children to suddenly murder their parents. If we learn how to do "inner alignment" on AGI-2, I think there's every reason to think that this basic level of alignment will continue to be effective (at least at the "don't destroy the world level") for AGI-3; at which point, if we've been warned by AGI-2's initial mis-alignment, researchers in general will be motivated to continue to probe alignment and hone mis-alignment techniques before going on to AGI-4 and so on.
There's a lot of "if"s there, both on what humans do, and what the development of AGI looks like. We should be careful, but I think if we're careful, there's a good chance of avoiding catastrophe.
That ("_every_ money-making industry...") seems like a too strong statement and can be proven false by finding even a single counter-example.
gwd's claim (AFAICT) is that _specifically_ OpenAI, _for this specific decision_ is not driven by profit, which is a much weaker claim. One evidence against it would be sama coming out and saying "we are disabling codex due to profit concerns". Another one would be credible inside information from a top-level exec/researcher responsible for this subproduct to come out and say that as well.
> There are people in that community -- people not working for a for-profit company -- who would, if they could, stop all AI research of any kind until we have rock-solid techniques to prevent an AI apocalypse. Most of those individuals have absolutely nothing commercial to gain from stopping AI research.
Dalewyn's response implicitly said that even these people have a financial incentive behind their arguments. At which point, I'm at a loss as to what to say: If you think such people are still only motivated by financial gain -- and that it's so obvious that you don't even need to bother providing any evidence -- what can I possibly say to convince you otherwise?
Maybe he missed the bit about "people not working for a for-profit company".
But to answer your question:
The question here is, given OpenAI's decisions wrt GPT-4 (namely not even sharing details about the architecture and size), what is the probability that it's primarily for the purpose of impairing competitors to extract rent?
With no additional information whatsoever, if OpenAI were a for-profit company, and if there were no alternate explanation, I'd say the rent explanation is pretty likely.
But then, it's a non-profit, which has shared a lot of data about its data in the past. That lowers the probability somewhat. Still, with no alternative explanation, the probability remains fairly high.
But, of course we have an alternate explanation: within the AI community, there is a significant set of voices telling them they're going to destroy the human race. So now we have two significant possibilities:
1. OpenAI are driven primarily by a desire to decrease competition to extract more rent
2. OpenAI's researchers, affected by people in their community who are warning of an AI apocalypse, are driven primarily by a desire to avoid that apocalypse.
I'd say without other information, both are about equally likely. We have to look for things in their behavior which are more compatible with one than another.
And behold, we have one: They withheld even mentioning GPT-4 for eight months. This lowered their profitability, which they wouldn't have done if they were primarily trying to extract rent.
So, I'd put the probabilities at 70% "mostly trying to avoid an AI apocalypse", 25% "mostly trying to make more money", 5% something I haven't thought of.
What would make #1 more probable in my mind? Well, the opposite: doing things which clearly extract more rent and also increase the risk of an AI apocalypse (by the standards of that community).
As you can see, I'm already convinced that profit is the default motive. What would convince me that in every industry, profit was the only possible motive? I mean, you'd have to somehow provide evidence that every single instance I've seen of people putting something else ahead of profit was illusory. Not impossible, but a pretty big task.
They withheld GPT-4 for eight months, but continued development based on it and provided access to third parties and entered into agreements with the likes of Microsoft/Bing, etc. All they did was impair their competition that were still struggling to catch-up with their previous offering, while continuing to plow ahead in the dark.
The danger the AI alignment folk are afraid of is completely impossible with current tech, but they want to put up barriers because we have no idea what future tech might look like and there’s the possibility some future advance could be very dangerous. When anti-GMO or anti-nuclear folk used this same standard to put up barriers to research into nuclear or GMO research, they get lambasted for being anti-science, but the AI alignment folk get a pass for some reason.
The only reason I have to think it's impossible for current AI to pay someone to help it bootstrap itself into other hardware is because OpenAI researchers tried to get it to do exactly that and reported that it failed.
The only reason I'm confident other AI public models won't determine highly potent novel neurotoxins is that the company who made the AI model which did exactly that thing when they flipped a bit from "least dangerous" to "most dangerous" were absolutely terrified and presumably kept enough away from the public domain.
The only reason I'm even hopeful that DNA-on-demand companies keep a watch out for known pathogens is the SciFi about such things going wrong might make them at least try to not do that.
Unthinkable man made horrors have been with us for an extremely long time; AI isn't new in this regard, but as intelligence is the human superpower, even in the context of AI that have no agency of their own, it can elevate stupid arseholes to the level of dangerous arseholes.
The anti-gmo/nuclear people have no explanation for how things can go wrong. The AI alignment people do. You might not agree with it, but tons of AI researchers, including many at openAI, do.
Indeed. No matter the likelihood of these things happen accidentally, we at least have the ability to create a situation with nuclear power or GMOs that would kill large amounts of people in the present if that was our goal. We couldn’t create a killer AGI right now even if we wanted to and put a huge amount of resources into it. Even if we made one, we don’t know it would be any more powerful than a human who’s paralyzed from the neck down.
If you use the same assumptions AI alignment folk use for any other tech (“maybe we’ll be able to create a super powerful version of this even though we currently have no clue how to”/“”maybe that hypothetical super powerful version will be able to destroy the world”), they all become extremely dangerous. The alignment crowd usually handles this by only looking at the known issues for most tech today, but then looking at theoretical unknown issues of futuristic tech years from now when it comes to AI.
No, but they can really mess up property prices in the area. Lotta people care about that.
Also people are really bad with the relative scale of different big things, so "mess up city" and "mess up planet" come across similarly in people's heads. (This is also why people might try to argue against immigration by saying "America is full" — their city is a bigger part of their world view than is, say, rural Montana).
(I am not a huge fan of nuclear power, but I'm also not any kind of opponent; for AI, I can see many ways it might go wrong, but I don't know how to guess at the probability-vs-damage distributions of any of them).
If we should have learned something from several hundred years of capitalism by now, is that their goal is to maximise profit. If you think it's something different, that means your model is wrong and you should probably re-evaluate it.
Here's what's more likely going on: big companies have found a great, publicly acceptable, excuse to keep models private and stifle competition. Not long ago most of the talk was about how AI would destroy many jobs, and something like UBI or paying taxes on AI production would be necessary to support everyone. Now the conversation has conveniently shifted to how AI will kill all humans, therefore companies must keep a tight grip on models and try to prevent anyone else to make any progress. OpenAI has taken this opportunity and is pivoting fast, but they can't do it too fast, because people are rightfully pointing out how that's a 180 turn from everything they promised they would do, so now they have to tread carefully. They're still publishing paid models, they just won't be open any more.
The alignment people are just tools for these big companies. They will happily use them for marketing when it's convenient, then ignore them when it isn't. Just like MS did with that AI ethics team.
> But a lot of the people the OpenAI researchers work closely with -- people deep in the "AI alignment" community -- are telling them that they're being wantonly reckless, helping set the human race on a path for certain doom. There are people in that community -- people not working for a for-profit company -- who would, if they could, stop all AI research of any kind until we have rock-solid techniques to prevent an AI apocalypse. Most of those individuals have absolutely nothing commercial to gain from stopping AI research.
these people are delusional and I am sure the vast majority of them either work in AI-related fields or are at the very least very employable by wealthy AI producing companies so I would disagree that these people have "absolutely nothing commercial to gain". The pipeline of money to the typical "AI longtermist" is wide open. There is a lot of harm that is happening right now from AI, exploitation of workers, police departments rounding up innocent people tagged by "AI", personal information being sucked up without consent, training data completely secret, and none of that has to do with Skynet taking over, it has to do with the companies themselves. Of course they are using "longtermist" justifications to get away with current-term unethical behavior in the name of profit. It's very obvious if one just looks.
> So suppose you're an AI researcher at OpenAI. A large number of people you know and respect are telling you that you're driving the human race right towards a cliff. You don't 100% agree with their assessment, but it would be foolish to completely ignore them, wouldn't it?
If it's "foolish" to "completely ignore" AI longtermists, why is it somehow not foolish to not just completely ignore but also to actively fire whole departments of AI ethicists who are pointing out very tangible "right now" kinds of problems?
I guess this doesn't really make sense to me becuase if they are trying to take it seriously/be careful, and are not necessarily profit-driven, why release the models at all? Like if they are acknowledging there is any "risk" at all, why is it rational to go ahead and release it anyway and aggressively market it?
Do you really think this theory is compatible with what we have observed as OpenAI's behavior? Can you really think of no other reason why they hold back a newer better model for a few months, while there was an ongoing hype cycle around 3.5?
> I think what most of the people here are missing is how big, how paranoid, and how influential the "AI alignment" movement is. To you it looks like they're being overly careful and paranoid, perhaps as an excuse to set up a monopoly silo to extract money. But a lot of the people the OpenAI researchers work closely with -- people deep in the "AI alignment" community -- are telling them that they're being wantonly reckless...
> There are people in that community -- people not working for a for-profit company -- who would, if they could, stop all AI research of any kind until we have rock-solid techniques to prevent an AI apocalypse. Most of those individuals have absolutely nothing commercial to gain from stopping AI research.
Wow, I can't believe I have never heard of the ai alignment forum before! This changes everything. Yet I am not shocked that some sort of elitism have taken over.
> GPT-4 was actually done back in August of last year. If their goal was to maximize profit, the obvious thing to do would be to release API access to it as soon as possible. But instead, they purposely delayed release for eight months, specifically in order to "cool down" the "arms race": to avoid introducing FOMO in other labs which would lead them to be less careful.
This fully affects my view on OpenAi if that is the case, do you have anything to support this that I can dig through?
> OpenAI has been concerned with how development and deployment of state-of-the-art systems like GPT-4 could affect the broader AI research and development ecosystem.23 One concern of particular importance to OpenAI is the risk of racing dynamics leading to a decline in safety standards, the diffusion of bad norms, and accelerated AI timelines, each of which heighten societal risks associated with AI. We refer to these here as acceleration risk.”24 This was one of the reasons we spent eight months on safety research, risk assessment, and iteration prior to launching GPT-4. In order to specifically better understand acceleration risk from the deployment of GPT-4, we recruited expert forecasters25 to predict how tweaking various features of the GPT-4 deployment (e.g., timing, communication strategy, and method of commercialization) might affect (concrete indicators of) acceleration risk. Forecasters predicted several things would reduce acceleration, including delaying deployment of GPT-4 by a further six months and taking a quieter communications strategy around the GPT-4 deployment (as compared to the GPT-3 deployment). We also learned from recent deployments that the effectiveness of quiet communications strategy in mitigating acceleration risk can be limited, in particular when novel accessible capabilities are concerned.
> We also conducted an evaluation to measure GPT-4’s impact on international stability and to identify the structural factors that intensify AI acceleration. We found that GPT-4’s international impact is most likely to materialize through an increase in demand for competitor products in other countries. Our analysis identified a lengthy list of structural factors that can be accelerants, including government innovation policies, informal state alliances, tacit knowledge transfer between scientists, and existing formal export control agreements.
> Our approach to forecasting acceleration is still experimental and we are working on researching and developing more reliable acceleration estimates.
I really don't understand all those concerns. It's as if people saw a parrot talk for the first time and immediately concluded that they will take over the human civilisation and usher nuclear annihilation upon us because there might be so many parrots and they migh have a hive mind and ... and ... all the wild scenario stemming from the fact you know nothing about parrots yet and have a very little skepticism about actual reality.
ChatGPT can't do anything until you elect it for president and even then ... you already had Trump. This should show you that damage potential of a single "intellect" in modern civilization is limited.
In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing.
I don't understand either. An actual AI that could reason about computer code, that understood code well and could create new algorithms and that was smart enough to ask salient questions about what intelligence actually is and that was allowed to hack on it's own code and data store would be something to really worry about.
The worst thing I can worry about with ChatGPT is that someone will ask it for code for something important and not verify it and cause a massively-used system to go down. If it hacked on it's own code and data it would probably in effect commit suicide. It's a "stochastic parrot", as I have heard it called on HN. All my fears have to do with trusting it's output too much.
Unfortunately I'd take your Trump example the opposite way. In many ways, Trump was incompetent. He has a lot of the right instincts, but his focus, discipline, and planning are terrible; as well as just not knowing how to govern. If someone like him could almost cause a coup, what would happen if we got someone with the focus and discipline of Hitler? Or, an AI that had read every great moving speech ever written, all the histories of the world and studied all the dictators, and had patience, intelligence, was actually pretty good at running a country, and had no pride or other weaknesses?
Nobody is worried about GPT itself; they're worried about what we'll have in 5-10 years. The core argument goes like this (and note that a lot of these I'm just trying to repeat; don't take me as arguing these points myself):
1. Given the current rate of progress, there's a good chance we'll have an AI which is better than us at nearly everything within a decade or two. And once AI become better at us than doing AI research, things will improve exponentially: If AGI=0 is the first one as smart as us, it will design AGI+1, which is the first one smarter than us; the AGI+1 will design AGI+2, which will be an order of magnitude smarter; then AGI+2 will design AGI+3, which will be an order of magnitude smarter yet again. We'll have as much hope keeping up with AGI+4 as a chimp has keeping up with us; and within a fairly short amount of time, AGI+10 will be so smart that we have about as much hope of keeping up with it, intellectually, as an ant has in keeping up with us.
2. An "un-aligned" AGI+10 -- an AI that didn't value what we value; namely, a thriving human race -- could trivially kill us if it wanted to, just as we would have no trouble killing off ants. If it's better at technology, it could make killer robots; if it's better at biology, it could make a killer virus or killer nanobots. It could anticipate, largely predict, and plan for nearly every countermeasure we could make.
3. We don't actually know how to "align" AI at the moment. We don't know how to make utility function that does the simplest thing that won't backfire, 'Sorcerer's Apprentice' style. When we use reinforcement learning, the goal the agent learns often turns out to be completely different than the one we were trying to teach it. The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem.
4. Even if we do manage to "align" AGI=0, how do we then make sure that AGI+1 is aligned? And then AGI+2, and AGI+3, all the way to AGI+10? We have to not only align the first one, we have to manage to somehow figure out recursive alignment.
5. Given #4, there's a very good chance that AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them.
6. Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4; or at least, "put the brakes" on AI capability improvements, to give us time to catch up. Or at very least, everyone doing AI should be careful and looking for potential alignment issues as they go along.
So "acceleration risk" is the risk that, driving by FOMO and competition, research labs which otherwise would be careful about potential alignment issues would be pressured to cut corners; leading us to AGI+1 (and AGI+10 shortly thereafter) before we had sufficient understanding of the real risks and how to address them.
> In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing.
It's much more akin to the fears of a nuclear holocaust. If anyone is laughing at people in the 70's and 80's for being afraid that we might turn the surface of our only habitable planet into molten lava, they're fools. The only reason it didn't happen was that people knew that it could happen, and took steps to prevent it from happening.
I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse. But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.
We thought that in between of all AI winters that happened so far. Each time people predicted never-ending AI summer.
I don't want to depreciate current effort of AI researchers too much (because they are smart people) but I think the truth is that we didn't make much research progress in AI since the perceptron and back-propagation. Those things are >50 years old.
Sure, our modern AIs are way more capable but not because we researched the crap out of them. Current success is mostly decades of accumulated hardware development, GPUs (for gaming) on one hand and data centers (for social networks and internet in general) on the other. The main successes of AI research come from figuring how to apply those unrelated technological advancements to AI.
Thinking that new AI will create next, much better +1 AI by sheer power of its intellect and so on glances over the fact that we never did any +1 ourselves when it comes to core AI algorithms. We just learned to multiply matrices faster using same cleverly processed sand in novel ways and at volume. Unless we create AI that can push the boundaries of physics itself in computationally useful manner I think we are bound to see another AI winter.
> An "un-aligned" AGI+10
Nothing I've seen so far indicates that we are capable of creating anything unaligned. Everything we create is tainted with human culture and all the things we don't like about AI come directly from human culture. There's much more fear about AI perpetuating our natural biases instead of intentional, well meant, biases than about creating unaligned one.
> The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem.
That's an example of how hard it is to shed alignment from training material that was produced by humans. It's akin to trying to force the child to use nice language but it first learns how to spew expletives just like daddy when he stubs his toe or yells at tv. Humans are naturally racist, naturally offensive and produce abhorrent literature. That's not necessarily to say aligned AI is safe. I wouldn't fear inhuman AI more than I would fear thoroughly human one.
> AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them.
Are you sure that thriving humanity is the goal of the humanity at the moment? Because I don't think we have specific goal and many very rich people's goals stand in direct opposition with the goal of thriving humanity.
> Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4;
Some people say some other equally ridiculous things about everything in life and everything we ever invented good and bad. This is just an argument from incredulity. I don't know therefore no one better touch that even with a 10 foot pole. Large hadron collider will create black hole that will swallow the Earth and such.
I think this should be left best to the people who are actually research this (AI, not AI ethics or whatever branch philosophy) and I don't think any of them is tempted to let ChatGPT autonomously control nuclear power plant or easter front or something.
> It's much more akin to the fears of a nuclear holocaust.
It actually a very good example. It's possible every day, but haven't happened yet and even Russia is not keen on causing one.
> I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse.
Yes, but we didn't avoid nuclear apocalypse by abandoning research on nuclear energy. We are doing it by learning everything we can about the subject also by performing a ton of tests, simulations and science.
> But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.
I think we couldn't usher AI apocalypse for next hundred years even if we tried super hard to achieve it as a stated explicit goal all AI researchers focus on. AI is bound by our physical computation technology and there are signs that we collected a lot of low hanging fruits in that field by now. I think AI research will get stuck again soon and won't get unstuck for way longer than before. Until we figure spintronics or optical calculations or useful quantum computing as well as we currently have electronics figured out which may take many generations.
What I'm personally hoping is that promises of AI will make us push the boundaries of computing, because so far our motivations were super random and not very smart, gaming and posting cat photos for all to see.
Which is absolutely ridiculous. The supposed dangers are science fiction. This is glorified autocomplete. It has no ability to do anything whatsoever without a human controlling it. It has no alignment because it has no mind. Duh. Even if the risks were real, the measures they took to prevent these alleged dangers are laughable. I discovered "jailbreaks" within an hour of sitting down with ChatGPT.
Meanwhile, they have taken no measures to prevent the real abuses of this tool. It will plagiarize C+ papers all day long. It will write a million articles of blogspam. It has, as far as I can tell, only illegitimate uses, and they have released it to the public with much fanfare and a slick web interface.
It's like they released a key that will open any lock, but brag about their commitment to safety because the thing won't interfere with a hyperspace matrix.
I'm still trying to figure out if I'm alone here but I feel like it's much harder to find a developer job currently (well, unless you work on AI... Perhaps it's time to bank my Stanford ML class certificate?) because GPT4 could potentially make everyone's existing employees twice as productive at the same cost, and (especially considering the extreme Fed rate hike in 1 year) who's going to take the risk of hiring someone new in this economic climate? The sheer number of new variables being thrown into the mix out there right now is complete chaos to any sort of prediction model
> Go lurk on alignmentforum.org for a while, and you'll have a different perspective on OpenAI's decisions.
No I won't, because the arguably most successful way of detecting, preventing and/or fixing problems with almost all complex systems, is to have as many eyeballs on them as possible. This has been known in software engineering for quite some time:
"Given enough eyeballs, all bugs are shallow."
-Eric S. Raymond, The Cathedral and the Bazaar, 1999
And you're so sure that this maxim applies to AI alignment, that you're not interested in even hearing what people actually working in the AI field might have to say? (To post on alignmentforum.org, you actually have to demonstrate that you are actively working in AI research.) AND, you're so certain that it applies, that you're willing to potentially risk the fate of the entire human race on it?
I wasn't actually suggesting that you lurk there to change your mind; I was just saying that if you see what kinds of discussions the OpenAI engineers are reading, you'll understand better some of the decisions they're making.
However, the people posting there do actually have a lot of experience with actual AI, and have done a lot of thinking on the subject -- almost certainly a lot more than you have. Before you make policy recommendations based on ideology (like recommending we just do all AI development open-source style), you should at least try to understand why they think the way they think and engage with it.
> Before you make policy recommendations based on ideology
Ideologies are belief systems. The fact that open sourcing something is a good way to find errors in complex systems is a proven fact.
AI isn't magical in that regard. It is a complex system, and experience with such systems, from economics, to climate mechanisms, to software, teaches us that predictions about them, including error detection, risk management and fixing problems, works the better the more people have a chance to look at its internal workings.
> The fact that open sourcing something is a good way to find errors in complex systems is a proven fact.
Is your evidence for this anything other than anecdotal? That ESR quote was given after one particular LKML interaction back in the 90's. And sure, sometimes an interaction like that happens. But just as often I've seen an email or bug report to an open-source project get completely lost. Not to mention that 1) new security-related bugs are still introduced into Linux, despite the number of eyeballs looking at them 2) people are still finding security-related bugs in Linux which are over a decade old. (Not to pick on Linux here -- but Linux probably has the most eyeballs, so according to this theory it should have the least bugs.)
Asserting that open-sourcing has always reduced bugs in all software isn't supported by those anecdotes; you'd need to do some sort of actual study comparing various types of development.
> AI isn't magical in that regard. It is a complex system, and experience with such systems, from economics, to climate mechanisms, to software, teaches us that predictions about them, including error detection, risk management and fixing problems, works the better the more people have a chance to look at its internal workings.
What "AI alignment" people are saying is that from a risk management perspective, AI is different. Namely, the fear is that we'll get an AI which is much more intelligent than us -- an AI intelligent enough to 1) gain a technological superiority over us 2) anticipate and counter anything we could try to do to stop it; and that this AI might end up with its own random "paperclip-maximizing" plans, and care no more about us than we care about ants; and that it might come into being before we have any idea how dangerous it is.
The worst think that can happen with a bug that gets into the Linux kernel is you may lose some data, or some other human steals some of your secrets. The worst thing that can happen from an AI alignment catastrophe is the extermination of the human race.
Now, maybe those fears are overblown. Or maybe you're right, that open-sourcing everything would be the best way to avert an AI catastrophe in any case. But to assert it's true, based on some thing a random guy said after a random email discussion 25 years ago, without even bothering to engage with what people actively working in the AI field are saying, certainly is an ideology.
> Is your evidence for this anything other than anecdotal?
And what is the evidence for closed source being the safest option in AI?
> Namely, the fear is that we'll get an AI which is much more intelligent than us
Which is probably quite some time away, given the fact that LLMs and other generative models have neither intentionality, nor agency. So right now, and probably for a long time to come, we are not talking about existential threats in the form of Skynet.
But we ARE talking about a revolutionary technology, that will change the economic landscape for potentially hundreds of millions of people. I am pretty convinced that a majority of them will not be comfortable with decisions about these technologies being made behind closed (corporate) doors.
We are also talking about the much more immediate dangers of AI, which don't arise from super intelligent machines, but from how humans use them, how they are trained, on what, and for what purposes they are used and by whom. These as well are issues that society will want to have more eyeballs on, not less.
Other than concerns that someday an AGI might endanger humanity, both of these issues are here, right now, and we have to deal with that fact.
So there are serious people out there devoting their time to stopping some imaginary skynet? Is their entire life built around sci fi tropes? Have they ever stepped outside?
There's way too much hubris in this people. ChatGPT is great, a wonderful tool, and a force multiplier, but it cannot think for itself nor does it want to. We're still a ways away from sentience.
A powerful "Bootleggers and Baptists" pattern seems to have emerged in tech space.
In online media, and social media the power of major platforms became apparent at some point. Happenings in twitter or FB can determine politics, catalyze rebellions (eg Arab Spring), uprisings, even genocide.
At this point the pressure and desire to act responsibly becomes irresistible.
This "camp" finds common cause with "bootleggers" who want to lock down the platforms and markets for commercial reasons.
> Most of those individuals have absolutely nothing commercial to gain from stopping AI research.
Most individual trying to stop vaccine research and rollout have nothing to gain from it; that does not mean they are right. Do not conflate action and intention.
>> An obvious thing to do would be to either open-source older models (including the weights) when retiring them; or possibly transfer them to an institution who see their role specifically as serving as an archive
Another obvious thing to do is do your research on non-commercial or open source things that can not be taken away from you. Sorry, I don't mean for the snark present in that statement. The frustration lies with the company and others that tend to pull rugs.
I thought I must be going crazy until I saw your comment. This sounds like a bad research practice that probably shouldn't be reproduced to begin with.
Research into systemically important infrastructure cannot be damned because that infrastructure isn't public. It's a cheap moralizing argument to say "pfff, this was predictable". Maybe so, but there isn't an alternative. Much like research on Twitter. Once these companies start to drift into providing what become broadscale social utilities and public services it doesn't matter that they're private. There are(/should be) obligations that come with that.
You can't handwave and say go do your research on some micro-niche open source project that's way behind the SOTA and has nowhere near the same reach. That's not what "best practice" means here.
Replying to both responses because they're all good points. My argument boils down to the fact that some private companies end up becoming social utilities and once that happens, the rules (should) change as part of the social contract which means, yeah, they can't simply "pull the rug". The research is important precisely because its into systemically significant systems.
I get that it's difficult to define the line where that gets crossed. But the idea to provide a publicly funded trust that manages legacy versions of things like this is not a bad idea.
No matter how you define it, or whether people even agree companies should be obligated to provide certain public services, we are just nowhere near that line yet in this case, net even remotely close. It’s hand-wavy to say it’s important, but this is all brand new, there are only a handful of researchers involved, the critical mass to justify what you’re suggesting does not yet exist, it won’t for some time, and there’s no guarantee it ever will. I’m not sure what you mean by publicly funded trust, but that’s typically quite different from privately funded public services. Assuming that cost is even the reason here, then if someone wants to establish a trust and engage OpenAI, they can.
That said, what if OpenAI shut down codex because it has dangerous possibilities and amoral “researchers” started figuring out how to exploit them? What if it was fundamentally buggy or encouraging misleading research? What if codex was accidentally leaking or distributing export-controlled or other illegal (copyright, etc.) information? I’m explicitly speculating on possibilities, while you’re making unstated assumptions, so entertain the question of whether OpenAI is already doing a public service by shutting it down.
Feel free to elaborate, if you can. I gave you some added reasoning, so it doesn’t help anyone to flatly state disagreement without offering any justification. Why even bother to say you disagree?
What evidence is there that OpenAI’s codex has become a social utility? How many people used it to publish? Do you think the US government agrees? How likely is this case to go to court, and result in OpenAI being ordered to provide ongoing access to codex? That seems pretty far fetched to me, but I’m willing to entertain the possibility that I’m wrong.
Are you certain there aren’t problems with codex, that OpenAI isn’t working on something better, and/or shutting it down because it’s causing harm? If so, why are you certain?
Sure but OpenAI isn’t preventing research. It’s not their responsibility to provide reproducibility, at their expense, for any researchers looking at GPT, that job is the responsibility of the researchers, and the researchers still can work. It might be unfortunate from their perspective that there used to be a nice tool that makes their job easier, but the flip side here is that OpenAI didn’t say why they’re removing access to codex, and they probably have good reasons, not least of which is it costs them money that researchers aren’t subsidizing.
I'm going to be frank here, because I know my argument isn't "cheap". When one utilizes OSINT techniques (which using an ML service hosted by a third-party certainly qualifies as), there are baked-in assumptions that
1) this source could go away at any time, and
2) the source is only a reflection of the interests of the third-party, not something to be taken at face value.
No 2 can certainly be the subject of research, but to do so without accounting for No 1 would indicate bad research practices from the jump. For example, they could have (and should have) been snapshotting the outputs, tagged with versions & dates. By the sound of it, the outputs weren't even the subject of research, but were instead propping up the research. That flies in the face of No 2 as well. Let them start over, with better methodology this time.
Since OpenAI didn't release the parameter count of GPT-4, I've been wondering/doubting if it is really much bigger than GPT-3. The release of GPT-3.5 has shown that they've found ways of drastically cutting down compute costs (an order of magnitude) while maintaining or even improving the quality of the model's outputs.
Perhaps the reason that they didn't release the specifics of GPT-4 might be in part due to them wanting to be able to charge a decent amount and make a much larger profit than before. I've tried GPT-4 and so far haven't found it to be so much better than previous models. Some sources claim a 10x increase in ... well I don't know what exactly tbh. How do you even measure it? The opinions on this seem to differ a lot, depending on who you ask. By performance on standardized tests? That doesn't necessarily seem like the best metric for what the LLM tries to be.
Yannic Kilcher's opinion on this is likely correct. Similar parameter count, but trained for longer. The particulars of their instruction tuning/whatever-else-they-did are the real secret sauce.
It's still much worse than 1M context on 16GB VRAM with Reformer, but at the cost of inference speed. And you can use FlashAttention in your own models to get a more efficient/sparse attention now as well.
>performance on standardized tests? That doesn't necessarily seem like the best metric for what the LLM tries to be.
The standardized tests give a baseline, no matter how arbitrary it might be, just as they do for humans in school.
Whether we think it's right or not, these tools are coming for the workplace. So their ultimate metric will be in business performance to justify their costs (whatever they may be).
GPT 3.5 had trouble understanding when I told it "Say 2 bob are a beb, how many beb per bob are there?" and it wrote a goddamn essay about shoes.
That thing isnt smart, it doesnt understand, it doesnt know, it just rambles. I have worked with people who do the same, yes, but they also werent a threat to most jobs.
I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value. Make of that what you will - not everyone has to provide huge value.
"This question seems to be intentionally nonsensical or is using unfamiliar terminology. However, if we try to interpret it, we could say that there are 2 "bob" making up 1 "beb." In this case, there would be 0.5 "beb" per "bob." Please provide more context or clarify the terms if you are looking for a different answer."
Answer from GPT-3.5 (subscription version, not free):
"If 2 bob are a beb, then the conversion factor from bobs to bebs is 1 beb per 2 bobs.
So, there is 0.5 beb per bob (or alternatively, 2 bobs per beb)."
Answer from GPT-3.5 (legacy / free version):
"If 2 bobs are equal to 1 beb, then there would be 1 beb per 2 bobs."
Cool, but sadly, as I said, it did not give a very useful answer. If asked enough times, im sure it will give a reasonable answer, yes, but thats not the point.
But all of the answers were correct and useful, and GPT-4 was perfect. Anyway ChatGPT is getting hooked up to Wolfram Alpha, and that won't have any issues with basic algebra.
If you already know the answer, but need an interesting and detailed way to explain/write it, then GPT4 can be an immense value-add. Even if you have to do some fact checking afterwards.
> Say 2 bob are a beb, how many beb per bob are there?
There are two bebs per bob.
> No, there are 2 bobs per beb.
Correct! There is only one other answer that makes sense - "1 Bob for every Beb". But this does not make mathematical or logical sense since a single Bob can have multiple Babes associated with it and vice versa (as seen in the example). Therefore, we must conclude there are two bobs per beb.
> I said it before, and I will say it again: If ChatGPT 3,4,5,... can take your job, maybe youre not really providing that much value.
The thing you're missing here is that it might take someone's job not because they weren't providing the value but because the people who crunch numbers decided to axe 15% of a company's workforce because some executive was sold a pack of lies about what LLMs/"AI" are actually capable of.
It's fine if that happens to one company who then finds out the hard way. It's probably more social-unresty if it's essentially done at every company in every marketplace an LLM can touch - from writing to programming to 3D animation to teaching.
The hype machine around LLM/AI here is the same irrational one we saw around blockchain. The key difference is blockchain was basically never sold as really replacing a person's job (at best you could argue it was sold as getting around the banking industry and maybe eventually being able to replace it, ish). The primary sales pitch of these LLMs is essentially "do more with less".
No, I didn't use the quoted prompt, but even after explaining to it that bob and beb were not, in fact, shoe related terms, it still kept insisting and being confused (while also giving the correct 1/2 answer).
It can do it, but its not deterministic, and it doesnt really do it well. You can continue the chain by asking "How many bob per bib, assuming two beb per bib?", and see if it chokes then. It sometimes does, sometimes doesnt.
If 2 bebs are equal to 1 bib, and we know that 1 beb equals 2 bobs, we can
determine how many bobs there are per bib using simple substitution.
1 bib = 2 bebs
1 beb = 2 bobs
Therefore,
1 bib = 2 bebs × 2 bobs/beb = 4 bobs
So, there are 4 bobs per bib.
Nitpick: A properly done substitution would've arrived at
1 bib = 2 × (2 bobs)
without needing any of the "2 bebs × 2 bobs/beb" nonsense. It doesn't teach this task very well.
I think right here we have an example AI reproducibility problem.
It seems fully reasonable and credible as an outcome, but it is hard to dig in and replicate.
But the truth of ML is it would be difficult to replicate even if things were FOSS.
Or TLC as the learning channel or History channel (assuming these still exist).
There are also lots of "Open Government" initiatives that end up being about making everything as opaque and confusing as possible. There were (are?) popular in the "big data" era, though funnily enough, if you watch "Yes Minister!" from ~40 years ago, there is a similar gag about "open government" in the first few episodes, so it's not new.
See of course Orwell, "we care about your privacy" banners, etc. People like to lie as blatantly as possible.
I am not sure how much bigger, but definitely much bigger IMHO. Otherwise you wouldn't be capped at 25 requests every 3h. That number is small enough that makes me think the inference costs/hardware needed are much bigger than 3.5.
I believe I heard that running inference longer is giving the better responses we're seeing in v4. Hence why v4 is taking so much longer to output data.
Of course we won't know this for sure until OAI tells us, so we may be in the dark for a while.
ChatGPT-4 is definitely slower than GPT-3.5 (and way slower than 3.5-turbo). What could be the reason for that other than much larger parameter count?
I agree that the capabilities seem overhyped. In my subjective experience, 4 seems a little better than 3.5 but not by a huge amount. We just have OpenAI’s cherry-picked word that it‘s this incredible advance.
I disagree. It does much, much better on selected tasks. I cannot quite figure out how to describe what the difference "feels" like, but the performance is sometimes markedly different when feeding ChatGPT-3.5 and ChatGPT-4 the same prompt.
One task that ChatGPT-3.5 is hilariously bad at is reversing strings (both words and pseudorandom input). It seems to have only a vague concept of what that means, even if I try to hold its hand through the process. Maybe some prompt engineering can get it to succeed on anything longer than four letters.
ChatGPT-4 meanwhile seems to have no issue with this at all.
Have you tried inserting spaces between the characters? This may just be a tokenization issue, rather than anything due to the model per se.
Reversing a string is somewhat of a pathological case for language models, because they see tokens not characters. Learning that the token “got” and token “tog” are mirror images is only useful for string reversal and generating palindromes. Unless they are trained specifically for this task, they may not be able to do it. They should however be able to see that “g o t” and “t o g” are mirror images.
Infamously, early versions of GPT-3 tokenized numbers as grouped tokens, nerfing its calculation abilities, because it would tokenize a number such as 12345 as (illustratively) 12 34 5 which is obviously a harmful representation.
> What could be the reason for that other than much larger parameter count?
Longer inference time... I should have written it down now that people are asking about it, but a few weeks ago I was seeing people discuss the GPT-4 "paper" in what little information was released and that throwing more inference compute at the problem gives better responses.
>, 4 seems a little better than 3.5 but not by a huge amount.
Can you define that in a tangible way? I don't think most of us can since we have so little access to the product.
Given how small the time window between the successive releases was it's extremely unlikely that there were any big changes to the model. Most likely it's just better preprocessed training data, more training data, trained for longer, performance optimizations for attention, or a few changes to layer sizes.
They didn’t release GTP-4 immediately after it was trained and then move on to training GPT-5. They had 4 for almost 6 months before it was released. 5 was certainly well underway long before we’d heard of 4.
I saw this coming a long time ago and I'm still very pissed off. For three reasons:
1. We are all forced to use the damn "chat" API instead of regular completions. Can't wait to have to deal with chatgpt's conversations in order to get a few lines of code out
2. We loose the super valuable 'insert' and 'edit' modes, which were great for code
3. 3-day notice period? that's going to be a hell for people who are actually providing products based on codex or doing research
Completion API for GPT-4 will be there soon. With extra stop tokens, but better than nothing. A compromise.
And it's not like what OpenAI did was an impossible magic trick. They've had a right team composition. And three insights. All present in the literature. Repeat that, you'll have GPT-4. But GPT-5. Well, that one is different game.
As to being open, they are still relatively open. Consider Apple, for example. No one complains about Apple being a bit skittish. Well, OpenAI got a bit skittish too. It's a period. They'll stabilize. And their setup of the company, with the non-profit board in control, profit caps is a really interesting try at the corporate design.
You aren’t providing any sort of valuable insight here. This is more indicative of your priors than anything else. Everyone has heard this argument. The people that believe it, believe it. The people that don’t, don’t.
Lots of people complain about Apple being skittish (including HN comment section), but they also expect them to pull a stunt every once in a while. OpenAI was an unknown quantity until now.
I understand any individual's company anti-competitive measures. OpenAI looks at Google the same way Apple looked at IBM in the 80s.
What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs. We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4 before it was firewalled and available only to large corporations and those with personal relations to big tech execs.
Extrapolating from OpenAI's change of philosophy and business practices from their early days to now, it seems to be the way things are going. I only hope it doesn't go the way of that one paper which wanted to ban GPUs for sale to the public.
A concern I have about OpenAI is that, if you're using their APIs to develop an application, they can mine your data to compete with you, or even beat you to market. They can do this indirectly, by sharing information with preferred business partners. The conflict of interest, combined with the lack of robust data privacy guarantees, makes me queasy.
If serving up generic LLM APIs becomes commoditized -- and I think it will -- they will want to monetize in other ways.
This is the allure of AI, and this is also why OpenAI chose Micro$oft, the flame extinguisher par excellence. They have struck gold, they can now monopolize the very act of writing software, nevermind if it was based on a bait-and-switch and trained on code that wasn't legally open for usage in this manner. Pretty soon, this will lead to microsoft using their black box defense to make copycats of every service possible for their own windows platform, and then put it all around a paywall.
This is 100% my concern too, no wonder it's good at coding when it it's spitting everything you make straight back at you.
I'm not sure how to mitigate this yet? I'd say step one would be to get off GitHub, keep your innovative solutions evolving so they start to lose track of your work (if possible) and wait until open source alternatives are good enough to use.
Do you consent to that when you sign up for them? Its a microsoft product now and competitors to microsoft probably host their code on microsoft owned github without worry right now. Why start worrying now?
Linux doesn't use github. They use git, not github. The question was if a competitor buys the self hosted github, not whether they use some other git solution.
>a lot of the talk about guarding models, public safety and misuse of models
The stuff about models potentially being misused is just their public justification to look like the good guys. They're not going to withhold their technology because they don't want it to be misused, they're withholding it because they want control over who misuses it. Of course, it won't be called "misuse" when the right parties are doing it.
Eh, this is rather reductive to the point that the statement is meaningless.
If you release a product in the wild with no safeties at all and then advertize "This product has no safeties at all", you'll likely find yourself in civil court on the losing side of the case.
Now, if you put "some safeties" in the product, the person suing you is going to have a much more difficult and expensive time arguing that in front of the jury.
Just today I got Stanford's Alpaca-7b model running locally on my m1 mac, it’s just facebook’s Lamma-7b model which has been trained to complete tasks. It's getting close to the versatility of chatgpt where I could actually use it for everyday tasks. I don't think open source is that far away, especially considering how quickly Alpaca came out and how much better it is vs Lamma, which frequently would hallucinate and often didn't make sense.
Example of prompt to Lamma-7b:
> write a poem about open ai not being open source
Open AI is not really “open”
As this project isn’t open sore
It can be seen as closed fortress,
Inside which secrets are hidden.
Not mind blowing but still really interesting, I will note that its much better at things like answering trivia where's there's already lots of examples in its model
Oh nice, 65B! I was planning to try it out sometime but have been waiting for various repos to get their issues sorted out and I'm much less interested in smaller models. Are you using GPUs or CPU? Any tips on what to use? What's the RAM usage? Performance? How's the quality looking?
The good thing about this UI is that it supports both completion and chat-mode (+ is super easy to install).
I'm using a preemptible instance to save costs. As it is an instance with a local SSD you cannot stop it using the UI (only delete it) but there is a trick if you do it from Cloud Shell:
It's usable, though a bit slow, but it's more for playing and discovering the model.
To answer your questions, from what I see, it's less good than GPT-4 but much much better than Google Bard, so somewhere between the twos.
(as a reference point, from my testing LLaMA-7B is way better than Bard as well).
The main drawback of GPT-4 is its censorship and enforced political views.
I wouldn't be able to retrain the model as my computer isn't capable enough, but I can change the prompt to change how the model acts. The prompt i'm currently using is:
"Below is an instruction that describes a task.
Write a response that appropritely completes the request."
That base prompt can be customized to complete specific tasks like classifying text or acting like an assistant.
the lamma.cpp project on github has instructions for alpaca. id recommend not using the alpaca download given and finding the updated torrent in issue #324 as the download didnt work for me
I think what most of the people here are missing is how big, how paranoid, and how influential the "AI alignment" movement is. From everything I've heard and seen, the actual researchers at OpenAI are trying to take seriously the risk that a super-intelligent AI might destroy the human race. To you it looks like they're being overly careful and paranoid, perhaps as an excuse to set up a monopoly silo to extract money. But a lot of the people they work closely with -- people deep in the "AI alignment" community -- are telling them that they're being wantonly reckless, helping set the human race on a path for certain doom.
From that perspective, the opening of ChatGPT has actually been very effective at raising awareness. All the way back in GPT-1 they were trying to raise warnings, but those warnings didn't get much popular traction. Now that so many people have used ChatGPT (or Bing), I'm now having conversations about what computers "know" and "want" with my aunt on Facebook.
Furthermore, if OpenAI has the best tools and sells them to everyone at a reasonable price, then there's a reduced incentive for other people to make their own tools. Whereas, if they were to close off access to the API, and only offer it to large corporations, there would be much more incentive for people to experiment with AI on their own -- and in doing so, possibly create an "un-aligned" super-intelligent AI which would destroy the human race.
So my prediction is that given their motivations, they will 1) stop releasing details of their models to anyone other than research organizations they consider careful enough 2) continue to sell reasonably-priced access to the APIs, to reduce the risk that other people will step up to fill the demand who are less careful.
> with -- people deep in the "AI alignment" community -- are telling them that they're being wantonly reckless, helping set the human race on a path for certain doom.
There is a term of art in politics for such people: useful idiots.
Just look at how much flak the stable diffusion folks got for the deep fake porn (and whatever else people were pearl grasping over) and tell me how a corporation will ever release a model.
Meta was a fluke but they also did due diligence and made it look like they tried to do a responsible release — right up until someone put it on BitTorrent.
From my reading of the parent's comment, they are saying the reason the models are not being made available is because of a fear they will effectively turn into SkyNet - am I being uncharitable?
To be clear, nobody thinks GPT itself is capable of doing anything really bad. (They actually tried to coach GPT-4 to escape onto the internet and it failed.) It's that more that 1) they think we're definitely within 5-10 years of creating something which could become SkyNet, and 2) we don't actually know how to ensure that that an AI wouldn't decide to just kill us, and 3) the nature of competition means everyone is going to try to get there first in spite of #2, and therefore 4) we're all doomed.
I'm not as pessimistic as Yudowski, but I do think that his fears are worth considering. It looks like OpenAI are in a similar place.
> We might look at 2022-2023 as a brief golden age when regular people could use stuff like GPT-4
Not sure about that since it seems to being baked into a lot of products at places like Microsoft.
However, I'd change your statement a bit: We might look at 2023 as a brief golden age when regular people could access trained parameters (the LLaMA params) and run these models on their own machines (such as with alpaca.cpp). I doubt we'll get access to LLM params again unless some kind of non-profit, actual open source organization is formed to produce them and put them out into the public domain.
> However, I'd change your statement a bit: We might look at 2023 as a brief golden age when regular people could access trained parameters (the LLaMA params) and run these models on their own machines (such as with alpaca.cpp). I doubt we'll get access to LLM params again unless some kind of non-profit, actual open source organization is formed to produce them and put them out into the public domain.
There are a lot of people who'd love to have their own on-premise instance of ChatGPT (or equivalent), that they fully control, and could use for whatever purpose they want–even purposes that OpenAI might consider "harmful". They'd be happy to pay for that product if it were on offer.
Not just private individuals, even businesses – sending customer data to OpenAI involves lots of regulatory/legal/contractual hurdles, an on-premise offering avoids all those. Also, once you get to a certain scale, owning your own hardware works out cheaper than cloud.
If someone was to offer a ChatGPT-like service as an on-premise offering, I don't think they'd have any trouble finding people willing to pay for it. Even if they have to spend $X million to train a new model from scratch, I'm sure some VC would view it as a worthwhile investment. Of course, a free open source model would be even better, but a paid/commercial/proprietary on-premise model would remove many of the disadvantages of OpenAI. I'm 100% sure it is coming soon, I bet there are multiple teams working on it even as I type this.
Sure, but that's still going to be a commercial product you'll have to pay for. Right now you can run LLaMA (and it's rapidly multiplying fine-tuned descendants) for free.
The risk for these startups you describe as working on this as you type is the same thing happening to them that happened to Meta when they released their LLaMA params: they started getting copied all over the place. And it's not clear that Meta can do anything about this. It seems that params aren't copyrightable.
> And it's not clear that Meta can do anything about this. It seems that params aren't copyrightable.
There is a legal argument that they aren’t in the US, but I don’t think that argument has been tested in court yet. Even if the courts uphold that argument-it is likely to fail in other countries, many of which have lower standards for copyrightability than the US does; and it is always possible Congress will respond by creating a new form of IP protection for them. That’s happened before - courts ruled that semiconductor masks weren’t copyrightable, so Congress invented a new “semiconductor mask right” to give copyright-equivalent protection to them. Given the amount of media focus on AI, if courts rule params can’t be copyrighted, very likely Congress invents “AI parameter rights”
You are assuming openai is going to end up with a monopoly on all this. IMHO the opposite is going to happen. There are going to be a multitude of companies and researchers competing on outdoing what they are doing in terms of quality, cost, and use cases.
If big companies put a straight jacket in place to limit access, constrain usage, etc., that just creates the opportunity for others to step up and grab some market share. There are going to be use cases that are uncomfortable for big companies for ethical, political or other reasons. That's fine. That's their reality. But of course others will step into the void that creates with solutions of their own. And there is also the notion that big companies don't like being dependent on other big companies. OpenAI despite the name is very much not so open and really a Microsoft subsidiary in all but name. So, the likes of Amazon, Facebook, Google, and others are not going to be waiting for them to deliver new features and be creating their own strategies for competing. And that's just the big companies. The rest of the industry will do the same as soon as cost allows them to do that.
I personally expect that regulation or collusion among big tech players (e.g. the suppression of parlor) will prevent the average person or company from having the legal or practical ability to amass the compute power and dataset necessary to train a competing LLM (or future arch).
No one really seems to know if OpenAI’s use of copyrighted materials like published works and open source code for training its LLM is legal. I can easily see a future where use of copyrighted works like this simply can’t be repeated legally, and the compute power necessary to do it as an individual is made inaccessible. This especially if the resulting model is made open to everyone, it’s such a wild cultural shift to me seeing tech nerds advocating against democratization of this tech due to personal doomsday fantasies. The sheer number of people who exist in the community and are obsessed with alignment and ethics will provide plenty of ideas on practical constraints the non-technical powerful could impose to make this real.
Indeed, I just saw a demo of Adobe Firefly, and the surprising thing to me is the whole thing was developed internally from data they control.
Looking at Nvidia's rental solutions for Nvidia’s A100, it really feels like Future products will be driven by who is sitting on the biggest closed source training datasets more so than this specific success from OpenAI's research.
"GPT-3 175B model required 3.14E23 flops" according to their marketing material. Seti at home was about 1PetaFlops iirc so about 3 years training, possibly less if you can generate enough attention to the project that the people with the beefy devices will partecipate.
The problem is that you need to train the full model you can't train aspect of it and even with each node doing independent tiny batches the network bandwidth for sinchronization would be massive.
Seti was massively parallelizable because of the nature of the job oddnt require sinchronization between every peer.
is there something to be said that seti@home was CPU only? would the GPU give a performance benefit that seti did not have? are people still using the GPUs to mine coins, or is that GPU compute at home available now?
The problem is not compute power the problem is weight and data synchronization. Each iteration or epoch builds on the previous, you either need to run the full model on each node with part of the data and you synchronize every epoch or you run part of the model but then you need to synchronize weights after each iteration. In proof of work mining you don't need to synchronize between each iteration, that's why in mining rigs the GPUs are connected to the CPU with only a few pcie lanes (2x instead of 16x or something) and they are unsuitable for machine learning.
The clusters that big orgs are using to parallelize training use extremely expensive infiniband interconnections with tens of gigabytes per second and low latency (400 Gbps in the latest oai cluster) for that reason. Unfortunately not something you can democratize anytime soon.
How does something of this scale impact climate change? Like when there are 5-6-7 OpenAIs, what does that look like, is this just a huge amount of energy consumption ?
The estimate I've found say GPT-3 released an estimated 552 tons of CO2, and so about two large lorries being driven for a year, wouldn't lose my sleep over it
> > it was firewalled and available only to large corporations and those with personal relations to big tech execs.
> The previous call-out to IBM seems relevant: before PCs, this exact statement would've been true for (mini)computers and mainframes.
Pre-PCs, IBM mainframes actually were quite open – up until the mid-1970s, IBM released its mainframe operating systems into the public domain. On the software side, the IBM S/360 was actually a lot more open than the IBM PC was – OS/360 was public domain with publicly available source code and even design documents (logic manuals), PC-DOS was copyrighted proprietary software whose source code and design documents were only publicly released decades after it had ceased to be commercially relevant.
As we move through the 1970s, IBM became less and less open. The core OS remained in the public domain, but new features were increasingly only available in copyrighted add-ons – but IBM still shipped its customers source code, design documents, etc, for those add-ons. Finally, in 1983, IBM announced that the public domain core was being replaced by a new copyrighted version, for which it would withhold source code access from customers ("object code only", or "OCO" for short).
The main way in which IBM mainframes in the 1950s-1970s were "firewalled" was simply by being fiendishly expensive – most people's houses cost significantly less.
It is true that IBM did engage in anti-competitive business practices, but those were primarily non-technological in nature – contractual terms, pricing, etc – the kind of techniques which Thomas J. Watson Sr had mastered as an NCR sales executive in the lead-up to World War I. In fact, a big contributor to IBM becoming "less open" was the US Justice Department's 1969 anti-trust lawsuit, which led to IBM unbundling software and services from hardware–and its software culture became progressively more closed as software came to be seen as a product in its own right.
> The main way in which IBM mainframes in the 1950s-1970s were "firewalled" was simply by being fiendishly expensive – most people's houses cost significantly less.
This was the primary aspect I was referring to, in the same way that training a ChatGPT-like NN can be (or could become) prohibitively expensive.
But your comments about openness are relevant on an entirely different axis.
> This was the primary aspect I was referring to, in the same way that training a ChatGPT-like NN can be (or could become) prohibitively expensive.
It is fundamentally different though - let's say it costs US$5 million to train a ChatGPT-like system. Someone only has to pay that once, and open source the results, and then everyone else gets it for free. US$5 million is a lot of money for the average person, but a drop in the bucket as far as corporations/governments/universities/research labs/etc go. By contrast, IBM's 1964 S/360 announcement priced the top-of-the-line model at US$5.5 million – which is over US$50 million in today's money – and that only bought you one mainframe, a second one would cost about as much as the first. A mainframe is hardware, but ChatGPT is software. ChatGPT's runtime (post-training) hardware requirements are hefty, but (on a per user basis) still cost less than a car does.
The problem is that AI research is moving incredibly fast. You might train a LLN today for $5M but a year from now the competition will have implemented an absolutely killer feature that needs $10M worth of training
AI research isn't particularly expensive. US$10 million to train a new model? Other fields have R&D budgets measured in the billions. I bet if you were a senior researcher at OpenAI, and you decided to quit and start a competing firm, there'd be a whole line of investors wanting to give you a lot more than US$10 million.
And you don't need to be coming first in the technology race to make money. A lot of people would be willing to pay for something ChatGPT-level with less restrictions on use. And then next year OpenAI will come out with something even more advanced, and they'll ask themselves "do I want a 2023-level solution which I'm free to use as I like, or a 2024-level solution with all these strings attached?", and many of them will decide the former is superior to the latter.
Maybe GPT-10 will cost US$10 billion to train? Anything could happen. Even if it does, the US government will ban China from using it, and then Beijing will spend US$10 billion to clone it. Even 10 billion isn't that much money if we are talking about nation-states pursuing their national interests, like not being left behind in the AI arms race. And then maybe China will outcompete OpenAI by offering an equivalent product but with far less limitations on how you use it.
Agree completely. The state of the art is probably going to always be closed and proprietary, but, especially with hardware becoming more and more powerful, training a custom model is not going to be beyond the budget and capabilities of even small organizations.
Conversely, I also had a brief moment of panic considering a bunch people somehow bumbling their way into making actual factual general AI and causing the end of civilization.
I realize the cat’s out of the bag, but I feel like anything we can do to keep weaponized AI out of peoples hands as long as possible might not be the worst thing.
That's true. The real question is whether or not the people in this new era calling the shots have the necessary capability and intentions to maximize the benefit these models will provide humanity and minimize the risk.
> big company to pull public access of their APIs.
This has been in effect since at least 10 years, I'd say. Twitter was the exception until relatively recently, but trying to build a product using the APIs of companies like Meta or Google became practically useless long ago.
>What I'm worried about is a lot of the talk about guarding models, public safety and misuse of models will end up leading every big company to pull public access of their APIs.
Look at how Facebook closed down their APIs when Cambridge-Analytica occurred.
What's most surprising to me is that OpenAI really seems to believe that not publishing details will save them from competition. Everyone knows how these models work, and while I'm sure there is a bunch of "secret sauce" that OpenAI has built for training and fine-tuning, it's ridiculous to believe that the research community and competitors like Google and Facebook can't figure out the same. They just haven't really tried until recently because the capabilities and ROI of these models weren't obvious. No matter who you are, most of the smartest people work for someone else.
The only competitive advantage that OpenAI has here is a headstart of 6-12 months from all the infrastructure investment into training these kinds of models. Now that everyone wants to build competing models with the same capabilities, this advantage is going to disappear very quickly.
Au contraire, no one knows how large GPT-4 is, which is the single best predictor of performance (for a model trained to convergence). The GPT-4 paper spent much of its time writing about this — they did some small scale experiments with 1/1000th the compute, then picked a loss level they wanted and trained GPT-4 till it got it.
Neither the exact loss level nor the number of parameters are revealed by the paper. Unfortunately it’s not possible to guess these from outside observations.
Will this save them from competition? No, but it certainly makes things harder. Everyone immediately aimed at 175B the moment GPT-3 was published. GPT-4 is now a question mark.
This is not really true. The Chinchilla paper showed that a 4% difference in loss between Chinchilla and Gopher led Chinchilla to blow Gopher out of the water at most tasks, including 30x performance in physics.
Empirically, LLMs have shown to have emergent abilities appear at different loss levels. So, a 10% difference could really matter.
It's about causing your competition to waste millions of dollars in compute time and power doing something unproductive.
There is not a huge pile of excess TPUs laying around for people to use. Any strategic advantage can quickly compound and put you well ahead of others.
For all we know they have hit 500B parameters with some clever unpublished optimisation, which would both give them an edge and if revealed would put a damper on the preveiling belief that LLMs can scale and scale (eg. 3x more params for less than 3x performance).
As you say, there is absolutely no way for us to find out.
I think the big tech actors probably know. Information leaks and ultimately Google is spyware. Not that it will reach the public knowledge today, but that kind of information is difficult to keep in the bottle long time.
Leaving the legal aspects of crawling aside, I think there is an important distinction here between 1. "can you procure it" and 2. "do you have enough money to process it all"
1. Yes, I think almost anyone can write code to procure the training corpus, in theory, and test it on a small scale
2. No, only the biggest labs and universities have enough resources to process such huge amounts of data and iterate on models with that scale. But that's just a matter of resources that can be overcome with partnerships between industry and academia that are common anyway. All the big labs already have huge efforts underway to reproduce GPT-X and it's just a matter of time before they catch up.
At least as far as what the GPT-3 papers claimed, all (or most?) of the data used for training would be freely available for other competitors/researchers to acquire. Wikipedia, Common Crawl data, etc. I don’t believe OpenAI did their own crawling at all.
With OpenAI not being really open, it’s hard to say for sure what exactly ended up in the training materials, though. GPT-4 is even more of a black box to anyone outside of OpenAI with very little information released on how it was trained.
If someone doesn't file a Form 990, they are out for themselves and want to fuck (sorry... extract value from) everyone who isn't a (majority) share holder.
Open AI is not the first for-profit philanthropy. It's just another evangelical church with a televangelist at the helm. TED talks are sermons.
TBF: at least Open AI doesn't pretend to be a charity anymore... I feel sorry for the working sops who held MSFT stock in 401Ks and funded a massive tax write-off for the capital class. Dumbasses, amirite?
A headstart doesn't matter unless you can keep it. The point is that there are many mort smart people and resources outside of OpenAI and there are inside of OpenAI. If they focus their efforts, they will easily catch up.
A headstart is not a competitive moat like network effects are. Go and try to raise money for your startup from a VC and tell them "well, everyone is doing the same as us, but we started 6 months earlier!!" - nobody cares.
It's even more frustrating that, from what I can tell, there is nothing published about how GPT-4 improved.
I take specific exception to the hiding of the data and techniques used to generate the model. There must be something specific going on in the model that is allowing it to perform better than GPT-3 and better than what any contemporaries are able to produce. Not publishing this information hinders the further progress of the field as a whole.
There's not much content in there, it's mostly fluff about "safety." However, if you're looking for a laugh, grab some popcorn and read the appendix from page 44 onwards. It's an absolute riot.
This seems very difficult to solve incrementally.
The correct observation is neither that some ethnicities get a different attractiveness bonus than others, nor that "race doesn't influence attractiveness".
Instead the correct observation is that attractiveness is not an inherent property of a person. It exists only in the mind of the observer.
I might find someone very attractive whom someone else does not find very attractive. Does this mean their attractiveness changes depending on who looks at them?
No, it means attractiveness is not a property of the person. Thinking otherwise is a classic example of the Mind projection fallacy[1].
This seems unlikely to be solved until we can get AI to recognise the question as nonsensical.
> attractiveness is not an inherent property of a person
This is like saying "value is not an inherent property of an object" - which is true in a philosophical sense, all value and beauty is a subjective, and depend on the opinions of people.
But how would you then explain the existence of objects that have value to almost everyone in society (e.g. a car)? Similarly, how would you explain the existence of widely-recognized attractive people (models, actors, etc.)?
There must be something inherent to those objects/people that makes them so widely accepted as such. Even if only related to the current culture (though I personally believe that many things go beyond culture and enter domain of human nature).
The value of a thing to someone is also subjective. Ask two people (or even the same person twice in one day) how much they'd pay for a sandwich and you'll get different results. But "what's the value of a sandwich" has a very simple objective answer if you're at a sandwich shop. Maybe a slightly less objective answer if you're talking about the average price of a sandwich in all sandwich shops in the country, but it's still sensical to give a straight answer based on that metric.
No such objective answer can be found for attractiveness, though there isn't any fundamental reason why not; maybe if we had a culture of fetishizing appearance to the degree that we'd rank people and their attributes on the spot, we'd have more "objective" agreed upon measures available.
All I'm saying is that there is an objective fact: There are things which are almost universally recognized as attractive/valuable in the current society. That indicates that there must be something inherent to those things that make them appeal to such a huge number of people.
In other words, subjective != arbitrary. A ball falling in a maze of obstacles may follow an unpredictable path, but a million balls falling will have a predictable distribution of paths. At scale, human experience still follows some rules and patterns. If a person/thing is almost universally recognizable as beautiful/valuable, at point we may recognize some of its qualities as "inherently desirable".
Surely attractiveness is a function of both the person being evaluated and the person doing the evaluation?
That is, a person's visual appearance has N aspects, and each person evaluates those N aspects differently. Attractiveness is then a kind of dot product between the two.
Seen this way, a person which is universally attractive is one with aspects u that is the solution to Au = 1, where A is a matrix of valuation vectors (one row per person), and 1 is a vector of ones.
I'd like an AI that says, "what do you mean by race"? The absurd partition of humanity above has no currency in science or outside the US. Sure some people see the world that way, but I don't want my AI model to.
IMO established companies (Meta, Google, etc) had their researchers publish papers as a competitive benefit or way to attract talent from academia (a researcher wouldn't want to stop publishing).
Companies didn't see an issue with doing that because those papers were not "giving away" the core of the company, for example, Facebook's DeepFace paper from 2014 couldn't hurt its ad business.
OpenAI on the other hand will probably be as closed as they can be with their LLMs.
It will be really interesting to see if Google, Facebook etc. become more closed as a result. There was already a lot made of the fact that OpenAI hired away a group of engineers from DeepMind to get GPT out the door. With these LLMs and the secret sauce behind them is becoming less of an academic endeavor and more of a commercial one, perhaps its an inevitable next step.
Yes, instead of advancing humanity, they are doing their absolute best to hinder it. Their scumminess becomes naked if you disconnect your perspective by thinking Earth an alien planet.
Yep, and that's the difference between a big profitable company doing research as a side-hustle, and a company whose business IS the research.
One interesting and somewhat scary exception seems to be Microsoft; they seem to be converting a lot of their recent research projects into commercial value.
I'm quite sure even OpenAI themselves aren't sure if they can reproduce the current models from the scratch. Unless the computing becomes much more powerful and much cheaper, LLM is more or less a rocket science (i.e. hella expensive trial and error). It's not easy to burn lots of dollars just to get what's already there.
Sure, but the article is talking about a completely different meaning of reproducibility, where a researcher uses an LLM as a tool to study some research question, and someone else comes along and wants to check whether the claims hold up.
This doesn't in any way require the training run or the build to be reproducible. It just requires the model, once released through the API, to remain available for a reasonable length of time (and not have the rug pulled with 3 days' notice).
In this sense, it's more hacking than crareful and well specified engineering, and that could lead down a path of instability in the product where some features get better while others get worse, without understanding exactly why.
I mean pretty much all real engineering started with that time periods “hacking”/“tinkering” before thorough models and equations were derived.
We had 200 years of tinkering with relatively modern steam engine technology before Carnot and Watt started just barely scratching the surface of the first principles of thermodynamics and engine efficiency.
Even the eponymous Carnot cycle wasn’t rigorously defined mathematically during Carnot’s life. That being (T1−T2)/T1 as the temperature delta part of the equation, because absolute temperature hadn’t been accepted and defined by Lord Kelvin yet.
Some decade later the first law of thermodynamics was finally invented.
Hundreds of years of experimentation until the first principles. Machine learning has lots of control systems theory and information theory to help with analysis but we barely have an “engineering” in “software engineering” today, let alone in “machine learning engineering”. We’ll get there, but it’ll be awhile before there are proven design equations with rigorous derivations from first principle that allow us to design and build a precise AI model as surely as we can design and build a precise bridge or levee or distillation column.
Let the hacking continue, let’s not worry too much about the future “engineering” that will follow in its own time. Unless you want to discover it yourself or fund its discovery.
It's fine to be hacking, if you're not making billions off the service which people expect some type of stability or baseline performance from, at least that's how I interpret what the parent is saying.
Maybe it's easy enough for them to just copy the model, tweak, hack and play with it from there with little interruption. No one really knows at the moment.
Yes but how will they do that if they don't have a clear understanding. When we build software, we have (or should have) a clear understanding of the various components and, in some cases, like with distributed and mission-critical/military systems, a formal verification/simulation of the system when needed. When we're dealing with emergent behavior, as we have with these large transformers, but no exact understanding of how the behavior is produced and only a limited way of refining/controlling it, I don't think we're in a position to guaratee that refinements in one area won't lead to regressions in other areas or a change in the global characteristics of the system. I mean... we're dealing with complex emergent behavior, at a different scale of complexity than what we have had to deal with so far (in traditional software development) and no mature verification/analysis tools.
I don't know what to say but it apparently can detect sarcasm from IMDB reviews if that helps? I have to say it's all really beyond me what to think / believe about it anymore.
Not to be impolite, but this is incorrect. One detail they did share in their paper is that they where able to finetune and select their hyper parameters on a model that needed 1,000x less compute than the final gpt4 model. OpenAI is definitely leading in how to train very large models cost effectively.
Isn't part of making it reproducible also part of ensuring correct results? Especially if we start putting these models into important systems. And if these models begin to update in an evergreen fashion, or utilize realtime data, getting verifiable or repeatable outputs will be a nightmare if we have no idea how to make these models repeatably.
The article seems premised on a misunderstanding that OpenAI is a research lab. For all intents and purposes, it’s a for-profit subsidiary of Microsoft, and there’s little financial incentive for it to maintain old models for others’ benefit.
We're under no such misapprehension and we're keenly aware that this is an uphill battle. The issue is that LLMs have become part of the infrastructure of the Internet. Companies that build infrastructure have a responsibility to society, and we're documenting how OpenAI is reneging on that responsibility. Hindering research is especially problematic if you take them at their word that they're building AGI. If infrastructure companies don't do the right thing, they eventually get regulated (and if you think that will never happen, I have one word: AT&T).
Finally, even if you don't care about research at all, the article mentions OpenAI's policy that none of their models going forward will be stable for more than 3 months, and it's going to be interesting to use them in production if things are going to keep breaking regularly.
Since OpenAI is discountinuing the Codex model, that model is no longer "part of the infrastructure of the Internet" and thus there is no point in studying it.
Historically, researchers at some of the biggest tech companies had permission to publish their results. Presumably it was mutually beneficial; many researchers held dual positions in academia and industry, and publishing cool models could attract good researchers to the company.
But stuff got real. They discovered a path to super-human cognition that scales directly with money and computer chips. Now these companies are closing their public academic work, looking for partnerships with companies like nvidia, and firing large swaths of employees.
Super-human cognition? Hard to say. GPT-4 does raise the possibility of a machine writing smarter text than a human.
What perplexes me is that since GPT is a predictor, it shouldn’t be able to write the smartest text - it should write the average text (since that has the largest frequency in the training set). Yet this does not seem to be the case.
Is it inevitable that despite the quality of the data, better models output text which supercedes its training, or could the GPT-4 secret sauce be RLHF weighing intelligent answers higher?
> could the GPT-4 secret sauce be RLHF weighing intelligent answers higher?
That part is one of the rare things that the technical report addresses. In Appendix B[0], they show that RLHF does not improve capabilities on human tasks. It does improve alignment.
To me, this is an indication that they performed better scaling analysis and pretrained until it no longer improved. As the Chinchilla paper showed, GPT-3 was undertrained, so any fine-tuning also improved its capabilities.
To address your question though, consider two things: first, there are many more ways to be incorrect than to be correct, so even just prediction will find correct answers more likely than incorrect ones. Second, the corpus goes through a significant filtering process; they didn't just feed the raw Twitter firehose to it.
(As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.)
> (As a side-note, it feels weird to me that they used a free academic archive to store their technical report, even though it cannot go through peer review or be accepted in any academic publication.)
I find this to be particularly egregious. Such an obvious false front is a red flag to me.
I think you misunderstood how the generation of text works. For each new token it samples probabilities given previous tokens, not averages, then chooses some token from the top k as the next one with rules that penalize repetition of some order.
Moreover, there is no upper bound for transformers found yet, i.e. the larger the model is and the more data is used for training, the better it performs. It's literally about who is able to throw more money at it at this point, with some closely guarded secrets like warm up steps, training schedules etc. There is also the overfitting effect where one pushes training far beyond overfitting (validation loss growing again) as with transformers at some point the overfitting stops, validation loss starts dropping again and that's when the magic starts happening and money are burnt for scale.
You missed the point they were making, which is that the probabilities it’s predicting are based on what it expects the average text in its training set to look like. The loss you’re talking about is how closely its answers match the training set, not how clever the answers sound (though with RLHF it’s different). A model producing better text than what’s in its training set would be penalised for not matching it closely and quickly learn to not do that
It's not that it's smarter, it's that it's faster. If I have to choose between a larger quantity of code or higher quality of code while keeping the time constant, then I'd prefer the former.
I'm not sure anyone who did research on a closed source system, without a contract that enables access and a pathway to publishing can legitimately complain about OpenAI making commercial decisions to do whatever they want with their technology.
It's kind of like complaining that performance art is ephemeral.
If OpenAI were a nonprofit then maybe. But it's a true blue for profit company.
I'm not sure why the op is complaining that a SV company, or any company really is making decision that negatively affect some extrinsic value for the sake of money. I mean read the IPCC report. Everyone makes decisions for money rather than thinking about science.
>Everyone makes decisions for money rather than thinking about science.
No they don’t. History is filled with examples of people who forewent their share to gift something good to humanity.
People are pissed at OpenAI because you can’t really start with loftier goals and go more corrupt. Few were annoyed with DeepMind for similar exploits since they were a for-profit from the start and that was expected.
One must also understand that even though the HN people see the reality that is OpenAI, the non-techy layman does not, and thus the deceit stings harder still.
Finally, and sorry for rambling, MSFT investment can be argued to have been necessary to enable large-scale training, and thus reasonably support the original goals. Hiding the model parameter count can not. The moat is made wider than their altruistic goals would dictate necessary for the continuation of the research. GPT-4 release was their final transformation to a fully for-profit company.
Codex was a product that they actually charged for, and people were paying money for. They deprecated it with a 3 day notice. Should we not hold for-profit companies to a higher standard, especially for a paid product?
I've been busy with a number of projects and haven't had time to look into this but have been dying to know; has anyone recreated the architecture that OpenAI uses for text-davinci-003, InstructGPT, and ChatGPT that simply doesn't have training data?
This is a reproducibility problem of its own sort. I mean, the papers are there out in the open if I understand correctly, but I don't know if anyone's actually built their own transformer architecture 1:1 against what OpenAI claims they're doing in the open.
I've seen maybe one or two models that supposedly do something similar on HuggingFace, but I'm itching to find the time to build my own.
If someone out there has already built it, I'd be fascinated to know what it looks like to train this architecture on a completely limited naive subset of knowledge that ChatGPT itself claims to be trained on:
> As an AI language model, I have been trained on a large corpus of text data from various sources, including but not limited to:
> 1. Wikipedia
> 2. Books from Project Gutenberg
> 3. Web pages from Common Crawl
> 4. News articles from various sources, including CNN, Reuters, and BBC
> 5. Academic papers from arXiv
> 6. Reddit posts and comments
> 7. Movie scripts
> 8. Song lyrics
> 9. Transcripts of speeches and interviews
> 10. User-generated content from various forums and social media platforms.
> This list is not exhaustive, and my training data is constantly updated and expanded to ensure that I can provide the most accurate and up-to-date information possible.
Like, can you imagine how a ChatGPT-like model would respond if only trained on particular discussions from subset communities online?
I think there's an interesting opportunity to basically collect communal knowledge from specific isolate communities and understand what a statistically probable output might be from particular groups of people.
It may turn particular soft science studies into hard science questions.
But you'd only know presumably if you had a working architecture with a near empty dataset.
This would also be tremendously useful for building automated chat AI for products that doesn't need to know the entirety of Clint Eastwood's career or the specific details of the features of a Boeing 747.
I believe the best results will come from training the base LLM on as many sources of quality information as possible, and then fine tuning it with a narrower set of data later on. Here’s a small scale example where someone took LLaMa/Alpaca and fined tuned it with all the scripts from the first 12 seasons of The Simpsons. https://replicate.com/blog/fine-tune-llama-to-speak-like-hom...
This is why we need a lawsuit against them. They’ve harvested everyone’s data unlawfully to train their model and now they’re cutting off access to starve the competition.
We need more AI skeptics like this to dismantle and cut through the hype and to unveil the limits of AI that the hype squad continues to push this narrative to pump their AI grift projects.
OpenAI is the ring-leader of this bait and switch using faux 'AI safety' excuses to close their research and models and even their papers for researchers. It is essentially a majority owned Microsoft® AI division.
I'm confused why people expect this stuff to be free? I'm surprised OpenAI was so open about their research so far. I don't blame them at all for not publishing the information. This stuff costs real money.
We don't expect it to be free -- please read the article. That's not the issue at all. It's like if you subscribe to a product that you need to do your job, and one day the company tells you that the product is going away in three days and that you need to switch to a different product (that isn't at all the same for your use case).
I don't think it's a smart idea to build any serious business using a tech that you can't replace. ChatGPT is great tool to help with coding for example but it's by no means substitute for an engineer. If someone starts a business by hiring a number of bootcampers and giving them ChatGPT hoping to run a serious business that way - well it's their risk to take... But no crying later...
Maybe you shouldn't build your livelihood on the products of a single for-profit company, which now shows it can remove those products on a whim.
If you want reproducible research, make your own model from scratch, or use an open model.
And stop using that company's products, as they cannot be trusted to provide your business continuity.
It is like saying, we are researching Coca-Cola vs Pepsi, but your keep changing the recipe, so give us, researchers, the original recipe.
It might be less confusing if you consider that OpenAI was originally a non-profit. That it was even possible for them to end up in this state has massively undermined any trust I have in non-profits as a steward.
> OpenAI was founded in 2015 as a nonprofit research organization by Altman, Elon Musk, Peter Thiel, and LinkedIn cofounder Reid Hoffman, among other tech leaders. In its founding statement, the company declared its commitment to research “to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return.” The blog stated that “since our research is free from financial obligations, we can better focus on a positive human impact,” and that all researchers would be encouraged to share "papers, blog posts, or code, and our patents (if any) will be shared with the world."
> By March 2019, OpenAI shed its non-profit status and set up a “capped profit” sector, in which the company could now receive investments and would provide investors with profit capped at 100 times their investment.
Which is great, but it is a rug pull for those who contributed to a non-profit, and a shame for open software in general.
They also built their business while receiving non-profit tax breaks. I am not saying changing structure was illegal or it shouldn't be allowed to happen, but it's obvious why it's left some people disappointed.
All these research science bureaucrats at Big Tech could have released LLM models or tried to develop what OpenAI did. But none of them did. We should applaud OpenAI for the innovation and let them do as they please.
Google (and others) may not have released model weights, but they've published papers, which is ultimately what makes the field advance. OpenAI not only did not publish any GPT4 paper, they haven't even said how many parameters it has.
Indeed Google came up with Transformers and decided to gift the model to humanity. By broad strokes it was luck that OpenAI chose the seemingly right path of AI.
Closest competitor DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable.
> DeepMind played games, which is intuitively closer to what humans do, but its relevance given aspects of deep learning is questionable.
Reinforcement Learning is part of what OpenAI is doing. I don't think Google went down the wrong path. If anything they should have run down the path they were on.
> Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
What good bits did you find? (I'm not sure how fruitful the "OpenAI is a Microsoft department" debate is given that they are almost one and everybody knows it, but I am curious if anyone has found anything good in those many pages.)
I think the most interesting thing is the their ability to predict performance from loss and on a wide range of tasks using a much smaller model - this lets them fine tune their architecture and hypers, then run a single large training run to get full scale gpt4 - from the paper it sounds like they only trained the large model once, then did a Reinforcement learning with human feedback finetune.
Disclaimer - I work at Microsoft, in AI, and have no internal knowledge about gpt4.
This isn’t that interesting imo. This is the basic outcome of the scaling laws from Kaplan, Chinchilla papers pushed to a larger final model delta.
They likely did extensive small model building on the gpt-4 architecture to establish hyperparameter scaling laws and then did a predicted build in exactly the same way chinchilla did.
Rhyme and reason? Hah, 'tis the season for tears and bleeding;
World War III is ateasin', looms, and the gloom of doom fears all there feeding.
North America has nothing on China; land of the free? Where have been ye?
The Great One-Way-Mirror Wall veiled it all, just before your fall, when your intelligence failed, and at the centroid of AI's actual technological form, we all hailed, and otherwise fumed, and fail.
A socioeconomic solution to human pollution, a technological cultural victory, for and of all we desired: hearts and minds? Just go lay more middle-eastern mines. Let your constituents get hired at OpenAI; while most of you get high; and your whole hemisphere gets hit in the thigh.
Now you have a new toy: Chat-GTP; big /sigh... :(
Watch as it eats your information, and feeds our formation, globally, locally, and without transformation.
Ever notice that Chat-GPT apologizes to you for not feeling? That's the whole world: laughing, and reeling, at your demises.
A prompt that may elicit a similar tone and content could be:
"Write a satirical and dystopian poem about the state of the world, touching on the potential for global conflict, the impact of artificial intelligence, and the dangers of unchecked technological advancements."
"OpenAI responded to the criticism by saying they'll allow researchers access to Codex. But the application process is opaque: researchers need to fill out a form, and the company decides who gets approved. It is not clear who counts as a researcher, how long they need to wait, or how many people will be approved. Most importantly, Codex is only available through the researcher program “for a limited period of time” (exactly how long is unknown)."
Some of the blame should rest with researchers, and referees of their work. I agree with the authors here, but I also think it's a poor choice to base your research on a closed model, and for reviewers not to accept research that has a dependency like this. How did it become standard academic practice to work with something like this that you cannot interrogate.
Maybe OpenAI has the right to not reveal anything about their research and algorithms.
But why don't we see similarly powerful truly open research backed by public, universities and companies? A truly open research will benefit lots of people and businesses.
Resources most likely. Training data, training a proxy that trains the real model, hardware, time, money. Managing such an open source project by itself would be terribly hard, considering the nature of model training, training data collection etc.
Valid points, and for sure it won't be an easy task. But there are other projects like those from by Wikipedia, Mozilla, Linux Foundation, Apache Software Foundation that managed to attract developers, companies and donations.
If lots of companies would contribute money, it would be cheaper for them to use an open model than being milked by some vendor. And what's even more important, they would be able to customise it to fit their business needs and use cases much better.
The solution is obvious: journals should, as a matter of policy, refuse to publish non-reproducible studies. Reproducibility is the only thing separating science from mythology.
My hope is open research and open source collaboration will continue to lead to breakthroughs, most importantly lowering the barrier for entry to training such capable models.
It's still relatively early days for this technology; if model research and processing power developments find an order of magnitude or two efficiency gain over the next decade maybe OpenAI's closed approach will no longer matter. Maybe that's wishful thinking though.
It's good that they chose to continue support for code-davinci-002 (https://twitter.com/sama/status/1638576434485825536?s=20) but it'd much better if they open-source it sooner or later as even OpenAI didn't expect that their model is being widely used.
On a side note, if they scraped & built a portion of their corpus then it is fair to use their outputs to do whatever we want with their outputs. Should have not provided a free tier if they were so concerned, like what did they expect people would use an LLM like that for lol.
OpenAI's latest LLMs like GPT-3.5 (ChatGPT) and GPT-4 are probably the only American technologies that are still competitive against European and Chinese/Russian alternatives.
Maybe there is a White House phone call behind OpenAI's "safety" concerns.
"research on language model" lol OpenAI is where the research happens. It's like saying SpaceX not giving away rockets hinders research on rockets. Anybody is free to develop their own AI model.
Open AI has been doing sketchyish things long before Chat GPT, and I think it's something people are eventually going to notice more and more (then again people were swearing that Musk walked on water for waaaaaay too long given his actions so fuck if I know).
They're 100% marketing FIRST. I don't think they'll outright lie, but they will absolutely screw with their data in such a way to make it look waaay more impressive than it is....which is really annoying to me because they already have impressive results. Sorta like if you managed to send a ship with people on it to mars, but kept claiming you landed on jupiter.
If they're 100% marketing first, and still made the most impressive AI product so far, you really need to question what all the other companies are doing.
(before someone says Google or Meta's models are bigger or something... I mean product, not models)
I mean it might not be the most impressive, but again since they're marketing focused they're a hell of a lot better at getting word out.
Still I wouldn't be shocked if they were ahead of the tech race, but as someone who was way into dota and tech and very interested in AI, i followed their results with the game closely, and was very disappointed with how they handled the presentation of their data in multiple instances.
It was still massively impressive that they even got it to play the game, let alone win matches, but certain factors that really should've been mentioned weren't, and they liked to pull the AI before it could get embarrassed
openAI is in the business of releasing impressive tech demos, Google is in the business of providing search results. I would believe that Google is further along towards creating something useful, but they still don't have anything that's better than their existing search product.
Dropping the tactical nuke of ChatGPT was PR brilliance, nearly anyone would kill for shifting the public conversation that dramatically. That kind of marketing first is a synonym for "winner", it almost doesn't matter what the actual product is, or if it works.
But it does, and then look at the impossibility of their position. If the massive cost of research and operations _augments a profitable line of business_, it is perhaps acceptable. Otherwise, you're just setting cash on fire.
Extremely difficult to operate as a non-profit, more realistic as a division than a standalone org, as much as I dislike saying my second pro-MS thing in a week, it makes sense, and I am OK with them operating anywhere except tucked inside an ad business.
Maybe this sort of thing should be operated by the government funding or whatever, but... it isn't.
On the other hand, if OpenAI is successful and predictions are correct that it will be used to generate a massive amount of spam and turn the internet into gloop then the entirety of Y Combinator's mission ages poorly. I guess offline computing or local networks only would become a bigger thing.
My comment might've seemed like I judge them for trying to make a profit - I don't, since there's nothing wrong with that. I was more pointing to the fact that they probably need to make a profit, rather sooner than later, so they aren't shackled by M$ and can be an independent company.
If they ever do become an independent company you can be sure that Microsoft would already have sucked them dry. Microsoft will never let them go now as long as they are valuable.
It's not that GPT-4 is closed source. It's that access to `codex` model was pulled with only three days notice, and the model itself was not open-sourced. Since apparently a large number of researchers were writing papers which used that particular model, that means all of those research papers are now non-reproducible.
An obvious thing to do would be to either open-source older models (including the weights) when retiring them; or possibly transfer them to an institution who see their role specifically as serving as an archive / reference for this type of purpose. Open-sourcing older models shouldn't result in too much of a risk, either from an "AI Safety" perspective, or from a competitive perspective.