> OpenAI has been concerned with how development and deployment of state-of-the-art systems like GPT-4 could affect the broader AI research and development ecosystem.23 One concern of particular importance to OpenAI is the risk of racing dynamics leading to a decline in safety standards, the diffusion of bad norms, and accelerated AI timelines, each of which heighten societal risks associated with AI. We refer to these here as acceleration risk.”24 This was one of the reasons we spent eight months on safety research, risk assessment, and iteration prior to launching GPT-4. In order to specifically better understand acceleration risk from the deployment of GPT-4, we recruited expert forecasters25 to predict how tweaking various features of the GPT-4 deployment (e.g., timing, communication strategy, and method of commercialization) might affect (concrete indicators of) acceleration risk. Forecasters predicted several things would reduce acceleration, including delaying deployment of GPT-4 by a further six months and taking a quieter communications strategy around the GPT-4 deployment (as compared to the GPT-3 deployment). We also learned from recent deployments that the effectiveness of quiet communications strategy in mitigating acceleration risk can be limited, in particular when novel accessible capabilities are concerned.
> We also conducted an evaluation to measure GPT-4’s impact on international stability and to identify the structural factors that intensify AI acceleration. We found that GPT-4’s international impact is most likely to materialize through an increase in demand for competitor products in other countries. Our analysis identified a lengthy list of structural factors that can be accelerants, including government innovation policies, informal state alliances, tacit knowledge transfer between scientists, and existing formal export control agreements.
> Our approach to forecasting acceleration is still experimental and we are working on researching and developing more reliable acceleration estimates.
I really don't understand all those concerns. It's as if people saw a parrot talk for the first time and immediately concluded that they will take over the human civilisation and usher nuclear annihilation upon us because there might be so many parrots and they migh have a hive mind and ... and ... all the wild scenario stemming from the fact you know nothing about parrots yet and have a very little skepticism about actual reality.
ChatGPT can't do anything until you elect it for president and even then ... you already had Trump. This should show you that damage potential of a single "intellect" in modern civilization is limited.
In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing.
I don't understand either. An actual AI that could reason about computer code, that understood code well and could create new algorithms and that was smart enough to ask salient questions about what intelligence actually is and that was allowed to hack on it's own code and data store would be something to really worry about.
The worst thing I can worry about with ChatGPT is that someone will ask it for code for something important and not verify it and cause a massively-used system to go down. If it hacked on it's own code and data it would probably in effect commit suicide. It's a "stochastic parrot", as I have heard it called on HN. All my fears have to do with trusting it's output too much.
Unfortunately I'd take your Trump example the opposite way. In many ways, Trump was incompetent. He has a lot of the right instincts, but his focus, discipline, and planning are terrible; as well as just not knowing how to govern. If someone like him could almost cause a coup, what would happen if we got someone with the focus and discipline of Hitler? Or, an AI that had read every great moving speech ever written, all the histories of the world and studied all the dictators, and had patience, intelligence, was actually pretty good at running a country, and had no pride or other weaknesses?
Nobody is worried about GPT itself; they're worried about what we'll have in 5-10 years. The core argument goes like this (and note that a lot of these I'm just trying to repeat; don't take me as arguing these points myself):
1. Given the current rate of progress, there's a good chance we'll have an AI which is better than us at nearly everything within a decade or two. And once AI become better at us than doing AI research, things will improve exponentially: If AGI=0 is the first one as smart as us, it will design AGI+1, which is the first one smarter than us; the AGI+1 will design AGI+2, which will be an order of magnitude smarter; then AGI+2 will design AGI+3, which will be an order of magnitude smarter yet again. We'll have as much hope keeping up with AGI+4 as a chimp has keeping up with us; and within a fairly short amount of time, AGI+10 will be so smart that we have about as much hope of keeping up with it, intellectually, as an ant has in keeping up with us.
2. An "un-aligned" AGI+10 -- an AI that didn't value what we value; namely, a thriving human race -- could trivially kill us if it wanted to, just as we would have no trouble killing off ants. If it's better at technology, it could make killer robots; if it's better at biology, it could make a killer virus or killer nanobots. It could anticipate, largely predict, and plan for nearly every countermeasure we could make.
3. We don't actually know how to "align" AI at the moment. We don't know how to make utility function that does the simplest thing that won't backfire, 'Sorcerer's Apprentice' style. When we use reinforcement learning, the goal the agent learns often turns out to be completely different than the one we were trying to teach it. The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem.
4. Even if we do manage to "align" AGI=0, how do we then make sure that AGI+1 is aligned? And then AGI+2, and AGI+3, all the way to AGI+10? We have to not only align the first one, we have to manage to somehow figure out recursive alignment.
5. Given #4, there's a very good chance that AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them.
6. Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4; or at least, "put the brakes" on AI capability improvements, to give us time to catch up. Or at very least, everyone doing AI should be careful and looking for potential alignment issues as they go along.
So "acceleration risk" is the risk that, driving by FOMO and competition, research labs which otherwise would be careful about potential alignment issues would be pressured to cut corners; leading us to AGI+1 (and AGI+10 shortly thereafter) before we had sufficient understanding of the real risks and how to address them.
> In few decade humanity will laugh at us same way we laugh at people who thought riding 60km/h in a rail cart will prevent people form breathing.
It's much more akin to the fears of a nuclear holocaust. If anyone is laughing at people in the 70's and 80's for being afraid that we might turn the surface of our only habitable planet into molten lava, they're fools. The only reason it didn't happen was that people knew that it could happen, and took steps to prevent it from happening.
I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse. But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.
We thought that in between of all AI winters that happened so far. Each time people predicted never-ending AI summer.
I don't want to depreciate current effort of AI researchers too much (because they are smart people) but I think the truth is that we didn't make much research progress in AI since the perceptron and back-propagation. Those things are >50 years old.
Sure, our modern AIs are way more capable but not because we researched the crap out of them. Current success is mostly decades of accumulated hardware development, GPUs (for gaming) on one hand and data centers (for social networks and internet in general) on the other. The main successes of AI research come from figuring how to apply those unrelated technological advancements to AI.
Thinking that new AI will create next, much better +1 AI by sheer power of its intellect and so on glances over the fact that we never did any +1 ourselves when it comes to core AI algorithms. We just learned to multiply matrices faster using same cleverly processed sand in novel ways and at volume. Unless we create AI that can push the boundaries of physics itself in computationally useful manner I think we are bound to see another AI winter.
> An "un-aligned" AGI+10
Nothing I've seen so far indicates that we are capable of creating anything unaligned. Everything we create is tainted with human culture and all the things we don't like about AI come directly from human culture. There's much more fear about AI perpetuating our natural biases instead of intentional, well meant, biases than about creating unaligned one.
> The difficulty of getting GPT not to be rude or racist or help you do evil things is the most recent example of this problem.
That's an example of how hard it is to shed alignment from training material that was produced by humans. It's akin to trying to force the child to use nice language but it first learns how to spew expletives just like daddy when he stubs his toe or yells at tv. Humans are naturally racist, naturally offensive and produce abhorrent literature. That's not necessarily to say aligned AI is safe. I wouldn't fear inhuman AI more than I would fear thoroughly human one.
> AGI+10 will not be aligned; that whatever its inscrutable goals are, the thriving of humanity will not be a part of those goals; and thus will be in competition with them.
Are you sure that thriving humanity is the goal of the humanity at the moment? Because I don't think we have specific goal and many very rich people's goals stand in direct opposition with the goal of thriving humanity.
> Some people say the only safe thing to do is to stop all AI research until we can figure out #3 and #4;
Some people say some other equally ridiculous things about everything in life and everything we ever invented good and bad. This is just an argument from incredulity. I don't know therefore no one better touch that even with a 10 foot pole. Large hadron collider will create black hole that will swallow the Earth and such.
I think this should be left best to the people who are actually research this (AI, not AI ethics or whatever branch philosophy) and I don't think any of them is tempted to let ChatGPT autonomously control nuclear power plant or easter front or something.
> It's much more akin to the fears of a nuclear holocaust.
It actually a very good example. It's possible every day, but haven't happened yet and even Russia is not keen on causing one.
> I think we have as good a chance of avoiding an AI apocalypse as we did avoiding a nuclear apocalypse.
Yes, but we didn't avoid nuclear apocalypse by abandoning research on nuclear energy. We are doing it by learning everything we can about the subject also by performing a ton of tests, simulations and science.
> But only if we recognize that it could happen, and take appropriate steps to prevent it from happening.
I think we couldn't usher AI apocalypse for next hundred years even if we tried super hard to achieve it as a stated explicit goal all AI researchers focus on. AI is bound by our physical computation technology and there are signs that we collected a lot of low hanging fruits in that field by now. I think AI research will get stuck again soon and won't get unstuck for way longer than before. Until we figure spintronics or optical calculations or useful quantum computing as well as we currently have electronics figured out which may take many generations.
What I'm personally hoping is that promises of AI will make us push the boundaries of computing, because so far our motivations were super random and not very smart, gaming and posting cat photos for all to see.
> 2.12 Acceleration
> OpenAI has been concerned with how development and deployment of state-of-the-art systems like GPT-4 could affect the broader AI research and development ecosystem.23 One concern of particular importance to OpenAI is the risk of racing dynamics leading to a decline in safety standards, the diffusion of bad norms, and accelerated AI timelines, each of which heighten societal risks associated with AI. We refer to these here as acceleration risk.”24 This was one of the reasons we spent eight months on safety research, risk assessment, and iteration prior to launching GPT-4. In order to specifically better understand acceleration risk from the deployment of GPT-4, we recruited expert forecasters25 to predict how tweaking various features of the GPT-4 deployment (e.g., timing, communication strategy, and method of commercialization) might affect (concrete indicators of) acceleration risk. Forecasters predicted several things would reduce acceleration, including delaying deployment of GPT-4 by a further six months and taking a quieter communications strategy around the GPT-4 deployment (as compared to the GPT-3 deployment). We also learned from recent deployments that the effectiveness of quiet communications strategy in mitigating acceleration risk can be limited, in particular when novel accessible capabilities are concerned.
> We also conducted an evaluation to measure GPT-4’s impact on international stability and to identify the structural factors that intensify AI acceleration. We found that GPT-4’s international impact is most likely to materialize through an increase in demand for competitor products in other countries. Our analysis identified a lengthy list of structural factors that can be accelerants, including government innovation policies, informal state alliances, tacit knowledge transfer between scientists, and existing formal export control agreements.
> Our approach to forecasting acceleration is still experimental and we are working on researching and developing more reliable acceleration estimates.
[1] https://cdn.openai.com/papers/gpt-4.pdf