The model seems to be "build something fast, get users, engagement, and venture capital, hope you can grow fast enough to still be around after the Great AI cull".
> offers over 0.6 million different pretrained open models.
One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models does not become analogous to cryptocoin speculation, where resources are forever burned only in a quest to attract the greater fool.
Those startups and researchers would better invest in smarter algorithms and approaches instead of trying to outpolute OpenAI, Meta and Microsoft.
Flights from the Western USA to Hawaii are ~2 million tons a year, at least in 2017, wouldn’t be surprised if that number doubled.
500t to train a model at least seems like a more productive use of carbon than spending a few days on the beach. So I don’t think the carbon use of training models is that extreme.
GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase. So we are talking about thousands of tons of carbon per model, repeated every year or two or however fast they become obsolete. To that we need to add embedded carbon in the entire hardware stack and datacenter, it quickly adds up.
If it's just a handfull of companies doing it, fine, it's negligible versus benefits. If it starts to chase the marginal cost of the resources in requires, so that every mid to large company feels that a few million $ or so spent training their a model on their own dataset makes them more in competitive advantages, then it quickly spirals out of control hence the cryptocoin analogy. That's exactly what many AI startups are proposing.
AI models don’t care if the electricity comes from renewable sources. Renewables are cheaper than fossil fuels at this point and getting cheaper still. I feel a lot better about a world where we consume 10x the energy but it comes from renewables than one where we only consume 2x but the lack of demand limits investment in renewables.
This is a complete fantasy as the depreciation rate on the hardware is higher than the prices of electricity.
Again, look at bitcoin mining, the miners will happily pay any carbon tax to work 24/7, it's better to run the farm to cover electricity prices and then make some pennies then to keep it off and incur depreciation costs.
This is a dangerous fantasy. Everything we know about the de-carbonation of the grid suggests that conservation is a key strategy for the next decades. There is no credible scenario towards 100% renewables. Storage is insanely expensive and green load smoothing capacity such as hydro and biomass is naturally limited. So a substantial part of the production when renewables drop will be handled by natural gas, which seem to have equivalent emissions similar to coal when you factor in the lost methane, fracked methane in particular.
In addition, even 100% renewable would be attainable, that would still require massive infrastructure investment, resource use and associated emissions, since most of the corresponding industries, such as concrete and steel production, aluminum and copper ore mining and refining etc. are very far from net zero and will stay that way for decades.
To throw into this planet-sized bonfire a large uninterruptible consumer, whose standby capital depreciation on things like state of the art datacenters far exceeds the cost most industries are willing to pay for energy, all predicated on the idea that "demand spurs renewable investments", is frankly idiotic.
Sure there is, Nuclear is zero emission and renewable and powers 70% of the French electric grid. Uranium in the oceans is thought to regenerate itself but even if that turned out to not be true, should be good for at least a few hundred thousand years. It would require a massive infrastructure investment so now would be a good time to get started.
The average American family is responsible for something like 50 tons per year. The carbon of one family for a decade is nothing compared to the benefits. The carbon of 1000 families for a decade is also approximately nothing compared to the benefits. It's just not relevant in the scheme of our economy.
There aren't that many base models, and finetunes take very little energy to perform.
> GPT3 was a 175 bln parameters model. All the big boys are now doing trillions of parameters without a substantial chip efficiency increase.
It's likely not the model size that's bigger, but the training corpus (see 15T for llama3). I doubt anyone has a model with “trillions” of parameters right now, one trillion maybe as rumored for GPT-4, but even for GPT-4 I'm skeptical about the rumors given the inference cost for super large models and the fact that the biggest lesson we got since llama is that training corpus size alone is enough for performance increase, at a reduce inference cost.
Edit: that doesn't change your underlying argument though: no matter if it's the parameter count that increases while staying at “Chinchilla optimal” level of training, or the training time that increases, there's still a massive increase in training power spent.
I wonder what is greater, the CO2 produced by training AI models, the CO2 produced by researchers flying around to talk about AI models, or the CO2 produced by private jets funded by AI investments.
That's the amount that would be released by burning 50,000 gallons of gas, which is about that ten typical cars will burn throughout their entire lifespan.
Done once, I agree, that's very little.
But if each of those 600,000 other models used that much (or even a tenth that much), then that now becomes impactful.
Releasing 500 tons of CO2 600,000 times over would amount to about 1% of all human global annual emissions.
500 tons is like a few flights between SF and NYC dude.
And those 600k models are mostly fine-tunes. If running your 4090 at home is too much then we're going to have to get rid of the gamers.
This CO2 objection is such an innumerate objection. Just making 100 cars already is more than making one of these LLMs from scratch. A finetune is so cheap in comparison.
In fact, I bet if you asked most LLM companies they'd gladly support a universal carbon tax with even dividend based on emissions and then you'd see who's actually emitting.
One sees the high impact of the large model, and the growth of model training, and is concerned with how much that could increase in coming years.
The other group assumes the first group is complaining about right now, and thinks they're being ridiculous.
This whole thing reminds me of ten years ago when people were pointing out energy waste as a downside of bitcoin. "It's so little! Electricity prices will prevent it from ever becoming significant!" was the response that it was met with, just like people are saying in this thread.
In 2023, crypto mining accounted for about 0.5% of humanity's electricity consumption. If AI model training follows a similar curve, then it's reasonable to be concerned.
> The other group assumes the first group is complaining about right now, and thinks they're being ridiculous.
Except this is obviously not the case, as "the other group" is aware that many of these large training companies, such as Microsoft, have committed to being net negative on carbon by 2030, and are actively making progress with this whereas the other group seems to be motivated by flailing for anything they can use to point at AI and call it bad.
How many carbon-equivalent tons does training an AI in a net negative datacenter produce? Once the datacenters run on sunlight what is the new objection which will be found?
The rest of the world does not remain static with only the AI investments increasing.
> many of these large training companies, such as Microsoft, have committed to being net negative on carbon by 2030
Are you claiming that by 2030, the majority of AI will be trained in a carbon-neutral-or-better environment?
If not, then my point stands.
If so, I think that's an unrealistic claim. I'm willing to put my money where my mouth is. I'll bet you $1000 that by the year 2030, fewer than half of (major, trailed-from-scratch) models are trained in a carbon-neutral-or-better environment. Money goes to charity of the winner's choice.
I'm willing to take this bet, if we can figure out what the heck "major" trained-from-scratch models are and if we can figure out some objective source for tracking. Right now I believe I am on the path to easily win given that both the major upcoming models, (GPT-5 and Claude 4?) are training in large companies actively working on reducing their carbon output (Microsoft and Amazon data centers)
Mistral appears to be using the Leonardo supercomputer, which doesn't seem to have direct numbers available, but I did find this quote upon its launch in 2022:
> One of the most powerful supercomputers in the world – and definitely Europe’s largest – was recently unveiled in Bologna, Italy. Powerful machine Leonardo (which aptly means “lion-hearted”, and is also the name of the famous Italian artist, engineer and scientist Leonardo da Vinci) is a €120 million system that promises to utilise artificial intelligence to undertake “unprecedented research”, according to the European Commission. Plus, the system is sustainably-focused, and equipped with tools to enable a dynamical adjustment of power consumption. It also uses a water-cooling system for increased energy efficiency.
You might have a greater chance to win the bet if we think about all models trained in 2030, not just flagship/cutting-edge models, as it's likely that all the GPUs which are frantically being purchased now will be depreciated and sold to hackers by the truckload here in 4-5 years, the same way some of us collect old servers from 2018ish now. But even that is a hard calculation to make--do we count old H100s running at home but on solar power as sustainable? Will the new hardware running in sustainable datacenters continue to vastly outpace the old depreciated?
For cutting-edge models which almost by definition require huge compute infrastructure, a majority of them will be carbon neutral by 2030.
A better way to frame this bet might be to consider it in percentages of total energy generation? It might be easier to actually get that number in 2030. Like Dirty AI takes 3% of total generation and clean AI 3.5%?
Something else to consider is the algorithmic improvements between now and 2030. From Yann LeCunn: Training LLaMA 13B emits 24 times less greenhouse gases than training GPT-3 175B yet performs better on benchmarks.
I haven't done longbets before, but I think that's what we're supposed to use for stuff like this? :) My email is in my profile.
One more thing to consider before we commit is that the current global share of renewable energy is something close to 29%. You should probably factor in overall renewable growth by 2030, if >50% of energy is renewable by then, I win by default but that doesn't exactly seem sporting.
> if we can figure out what the heck "major" trained-from-scratch models are and if we can figure out some objective source for tracking
Hmm. Yeah, we'll need to hammer out a solid definition. Further complicating things are models that may not be publicly available and are internally used by companies, though those may not be trained from scratch.
I would be fine with your suggestion to frame it in terms of percent power generation, though it might be hard to disentangle training costs from usage costs from that number. I would argue that including usage energy cleanliness is in the "spirit" of the bet but I'm happy to try to disentangle it as I originally proposed training-only.
> Something else to consider is the algorithmic improvements between now and 2030. From Yann LeCunn: Training LLaMA 13B emits 24 times less greenhouse gases than training GPT-3 175B yet performs better on benchmarks.
This is an excellent point, and definitely works in your favor. Really I'm on the good side of this bet. I win either way :) either my charity makes money, or I'm pleasantly surprised by climate impacts.
I've also not used longbets before. I would think we want to hammer out exact terms here before we set something up there?
> If AI model training follows a similar curve, then it's reasonable to be concerned.
Yes, but one can at least still imagine scenarios where AI training being 0.5% of electricity use could still be a net win.
(I hope we're more efficient than that; but if we're training models that end up helping a little with humanity's great problems, using 1/200th of our electricity for it could be worth it).
The current crop of generative AIs seems well-poised to take over a significant amount of low-skill human labor.
It does not seem well-poised to yield novel advancements in unrelated-to-AI fields, yet. Possibly genetics. But things like solving global warming, there is not any sort of path towards that for anything we're currently creating.
It's not clear to me that spending 0.5% of electricity generation to put a solid chunk of the lower-middle-class out of work is worth it.
There was an important "if" there in what I said. That's why I didn't say that it was the case. Though, no matter what, LLMs are doing more useful work than looking for hash collisions.
Can LLMs help us save energy? It doesn't seem to be such a ridiculous idea to me.
And can they be an effort multiplier for others working on harder problems? Likely-- I am a high-skill worker and I routinely have lower-skill tasks that I can delegate to LLMs more easily than I could either do myself or delegate to other humans. (And, now and then, they're helpful for brainstorming in my chosen fields).
I had a big manual to write communicating how to use something I've built. Giving GPT-4 some bulleted lists and a sample of my writing got about 2/3rds of it done. (I had to throw a fraction away, and make some small correctness edits). It took much less of my time than working with a doc writer usually does and probably yielded a better result. In turn, I'm back to my high-value tasks sooner.
That is, LLMs may help attacking the great problems directly, or they may help us dedicate more effort to the great problems. (Or they may do nothing or may screw us all up in other ways).
I fully agree that any way you cut it, LLMs are more useful than looking for hash collisions.
The trouble I have is, what determines whether AI grow to 0.5% (or whatever %) of our electricity usage is not whether the AI is a net good for humanity even considering power use. It's going to be determined by whether the AI is a net benefit for the bank account of the people with the means to make AI.
We can just as easily have a situation where AI grows to 0.5% electricity usage, is economically viable for those in control of it, while having a net negative impact for the rest of society.
As a parent said, a carbon tax would address a lot of this and would be great for a lot of reasons.
I've seen estimates that training gpt3 consumed 10GWh, while inference by its millions of users consumes 1GWh per day, so inference Co2 costs dwarf training costs.
> offers over 0.6 million different pretrained open models.
One estimate I saw was that training GPT3 released 500 tons of CO2 back in 2020. Out of those 600k models, at least hundreds are of a comparable complexity. I can only hope building large models does not become analogous to cryptocoin speculation, where resources are forever burned only in a quest to attract the greater fool.
Those startups and researchers would better invest in smarter algorithms and approaches instead of trying to outpolute OpenAI, Meta and Microsoft.