I don't understand why Musk/Tesla is so insistent on having cars do self-driving purely on cameras.
Both from the technical and the marketing point of view having a car that can do better than any human ever could, even in the most optimal of conditions would be a great thing.
It's cost - LIDARs aren't cheap. Tesla is set up as a consumer car company first and a robotaxi company second, costs matter even more. They could jack up the price of the final product, but presumably it wasn't worth it (at least in his mind?). From a safety perspective more sensors in general are better, if integrated properly. With LIDAR prices are coming down I wonder if he will someday swallow his pride and reintegrate them.
It's almost certainly because he dug the hole so deep in advertising. By saying "we need LIDAR", means that his promise "your car that you own right now will someday be self driving" was a lie. I can't imagine the lawsuits.
I cannot imagine how a musk-fan (at this point, I would classify anyone, who believes his claims as a fan) thinks, but it seems so weird to me to actually cling on that hope, that it should make the promise easy to break. It has been a while he made this promise, and it's nowhere near to be delivered. What does it even matter? How long are you comfortable owning the same car (the one that has more electronics in it than iPhone)? 10 years? 20 years? If he doesn't deliver FSD in that time, it doesn't really matter if it will be the same old cars that will achieve FSD. And if you can achieve FSD with better technology on a newer car, it's a win-win, it doesn't really break the promise that your 20 y.o. Tesla will achieve FSD. Maybe in 30 more years it will.
Both the Valeo SCALA and Luminar LIDAR sensors are in the $500 ballpark (at volume). Even a few times over, this seems like a petty cost to forgo on a $50k car.
Anyway, Tesla have supposedly recently bought a massive pile of Luminar sensors, so I expect them to be in one of these vehicles soon.
I think the robotaxi might be the only actual viable approach to developing fsd.
You control the hardware and software, you drive in real world conditions, you can deploy regionally and control the hours and conditions you operate in letting you gradually increase when you operate as you judge it safe.
And if the robotaxi makes a mistake, well it's still in testing phase/beta.
If Tesla comes out with FSD then it has to be damned near "perfect" in all conditions, which just is never going to happen out the gate?
You can even design some type of system where the taxis can only physically go on a routed road, and then you won't have to worry about real world conditions. And then you can just integrate some sensors into the routed roads to detect the vehicles' location and then you can guarantee they'll never crash.
Oh! Oh! And then for efficiency maybe you can chain a ton of robotaxis together, so you only need one drivetrain... wait a second...
Why not have at least radar? Plenty of cars have radar for adaptive cruise control and blind spot monitoring, it can't be that expensive. Two sensor kinds would still be an improvement.
I think if you have two sensor systems with one being significantly less capable than the other, the gain from having both quickly diminishes. I believe Tesla dropped the radar because it was simply outclassed by the camera in 99% of circumstances. So the radar really just becomes a hacky failsafe for bery rare circumstances, and Musk would like to avoid such a situation, he wants a "clean" solution.
Sensor fusion could cover that though, cameras aren't good if they are obstructed by dirt, particles (fog, snow, heavy rain), don't capture enough light, and probably many other cases. Fusing camera input with a radar to cover shortcomings of the camera sounds sensible enough.
What I'm saying is that it might be the case that "capability c" (somehow measures as a positive scalar) of your fused sensor system is not the sum c = c_1 + ... + c_n of the capabilties of each sensor c_i, but more like c = (c_1^p + ... + c_n^p)^(1/p) where p is a large power. The case p = 1 is where every sensor contributes to the whole system with perfect efficiency. The case p = infinity, i.e. c = max (c_1, ..., c_n) is where really only the best sensor does all the work and the weaker sensors can not add anything useful. The case of p > 1 somewhat large represents that all sensors contribute but the real heavy work is being done by the best sensor and other sensors only have significant impact if they have roughly equal capability. So I believe that the team at Tesla has the intuition that p is too big for radar to be worth it.
People drive all the time in snowstorms around here in Sweden and Norway, lower speeds but outside of a major blizzard with zero-visibility people still drive and reasonably safely.
I used to drive through torrential storms in Brazil, where the highest level of the wipers was barely enough to keep water out of the windshield, never had an issue when keeping safe speeds either (even though Brazilian traffic is murderous).
Driving in a torrential rain can be terrifying. Literally can't see practically anything in any direction.
Haven't seen an equivalent snowstorm so far. Even in the worst case there's still at least 30 ft / 10 m visible range. Which is not great, but at least something.
In both cases, if at all possible, better to stay put!
But they now charge $5000 per year subscription fee, and they can demand the car be bought in for some onboarding. People are ready to pay for it and do whatever it takes to make their Tesla dance, so cost is not an excuse really.
I've seen this argument brought up a lot, but I've never seen someone attempt to answer the counterfactual. If LIDAR was added to Tesla's now, how much would FSD improve? I've seen people here who have used it say that with the current software FSD is already pretty good, but my guess is that even with LIDAR they'd still be far from L4. So what kind of difference would we expect to see if, say, tomorrow Tesla announced that they're going to start using LIDAR?
The worst thing about my Tesla are the phantom brakes and "emergency lane assist" function (which should be called "steer into opposing traffic for no reason". If I forget to turn those off before starting, my ride will be pure horror.
LIDAR probably would help to make Phantom Brakes happen less often, simply because there would be another info source for "is there REALLY an object that is dangerous to me?".
Those who say that FSD is "pretty good" are living in a fantasy world. There is hard data on miles between critical disengagements (which really should be called "if the driver doesn't respond within a fraction of a second, people will die"), and depending on region, model, weather etc it's between 13 and 115 miles right now.
Over here in Germany there are statistics that a human driver will have the equivalent every 155,000 miles.
"Pretty good" just doesn't cut it when it's about the risk of killing people.
My Model 3 right now detects about 60-70% of school children crossing the road (keep in mind roads in Germany are narrow, and humans including kids are using the roads, too). 30%-40% of those I would kill every morning on my way to the office.
And the thing is: 70% isn't enough for this, 80% isn't, 90% isn't, 99% isn't, and 99,999% isn't.
Side note: People constantly claim that Waymo is autonomous. It's as autonomous as a tram. They only work because it the cities they operate every single road they use have been mapped by hand and is constantly updated. Send a Waymo into my city over here, and will also kill a couple of kids per day. Years ahead of Tesla? Yes. Good enough? Hell no.
Looking around at the percentage of drivers around me on the road with their face looking at a phone, while moving, is probably in excess of 5%. These people won't see the kids crossing the road, either.
Adding an extra sensor onto something not designed for it may not do much. So if that specific question is important to answer, the answer is probably 'not a whole lot', especially since the AI has been training on images only, AFAIK.
The problem isn't that the sensors need to be added now, the problem is that without the sensors the cars are half blind and are more likely to make bad decisions and refusing to add them at all shows bad judgement.
Rain, snow, mist, fog, darkness -- these are things cameras are very bad at seeing through. Is that thing in front of us solid? Without extensive training on that particular type of object, the AI with the camera has to guess. I'm sure these aren't the best examples -- I am not an expert in LiDAR. But I do have experience with computer vision with camera systems, and they are woefully insufficient for life and death decision making systems when they haven't been trained on the exact specific scenarios they will encounter.
My guess is that it's not the lack of sensors that causes the problem with autonomous driving, rather it is the processing of the data. What should the car do if there is a flood covering a dip in the road?
> Both from the technical and the marketing point of view having a car that can do better than any human ever could, even in the most optimal of conditions would be a great thing.
Given that the leading causes for road accidents are speeding, DUI and distractions [1], a car doesn't need to be better than any human reasonably can. It just needs to be on the same level as a well-rested, sober human without a phone in their hand.
The decision LIDAR vs cameras is the pareto principle in action: what matters is the 80% - the mass that causes the most accidents.
The history of technology is pretty much directly opposed to your wishes.
The prosperity of technology basically solves the problem with the most terrible hardware possible.
For example, if you have a really good microprocessor that works, what you should do is shrink it until it barely works or speed it up until it barely works to get a similar version for cheaper or faster version for the same price.
Same goes with cars. If you have a car that works with ultrasonic and lidar and cameras, eliminate the expensive sensors and reduce the cost.
Reducing the cost will let more people afford the cars, and sell more cars.
To Musk that is basically his plan all along - to replace ICE cars with EVs. Just wish we could keep turn signal/wiper/headlight stalks.
It does in certain strictly determined conditions + human assistance.
But the point Tesla didn't go through "FSD with ultrasonic + lidar + cameras", and then start to reduce the sensors. They went from "no FSD" to "broken non-working FSD, but now only with cameras". That is not how you strip down a working product to reduce costs.
Waymo has sensors but they don't sense purely in real-time. Meaning the roads are mapped out in detail ahead of time. This is fine and effective, but it makes the comparison not really fair. A human driver will do splendidly in an area they've never been before, whereas a Waymo taxi won't.
I have an idea, replace the existing cameras with early 2000s era webcams. Since worse technology solves problems, it would work better.
If Musk had gone all-in on every sensor possible and put them all over the cars like strings of Christmas lights, I assume you would be defending that decision with just as much fervor.
Perhaps the worst hardware that can solve FSD is hundreds of sensors of all types plus a fleet of attached drones for other angles. The worst thing that could work is different from the worst thing.
They shrink their chips, they up their clock speeds and keep the costs the same.
They also remove hardware - they've relentlessly removed ports, intermediate interfaces (like removable batteries, memory and processor sockets) and changed screws into glue.
> you have a really good microprocessor that works,
and applying this analogy to the current situation: do we?
normally you have to sell a product that works first, take the profits and then invest that into improving the components to make more profits. I think Elon has cracked the code: sell the product you don't have to people who would love to have it, and you don't even need to produce a version 1 that works - just move on to solving the problems that you want to. the market will pay you either way.
Musk gets obsessed with big bets when people tell him something isn't possible. That, plus the hardware costs are much saner for Tesla if they don't need a bunch of lidars.
That said, this particular big bet looks pretty bad for Musk. He's been confidently promising self-driving either "this year" or "next year" for the last several years, and so far Tesla appears nowhere close to getting L4/L5 driving down.
I mean, Volvo is already delivering the EX90 with a LIDAR integrated from factory for similar money to the Model X - and I can't imagine Volvo is losing money on them, so it can be done. Tesla just chose not to.
And yes, I know the EX90 is currently an absolute mess in terms of software and their LIDAR isn't even enabled for driving assists yet, but the hardware is there so it can be built on scale.
I presume there is a bigger game here. Androids, eg humanform robots, will require the capacity to navigate on camera alone.
Specifically, loads of object detection with smaller objects, imagine an android cleaning off a table after dinner. Visual spectrum, camera identification is vital here.
So it's all one ball of wax. Tesla is not juet a car company, likely in 5 years the number of androids they sell will dwarf car sales.
> Androids, eg humanform robots, will require the capacity to navigate on camera alone.
But those cameras don't have to be simple visual spectrum cameras, do they? They could have, for example, in each eye, LIDAR, infrared, several focal lengths of visual, etc. Not to mention being able to augment those sensors at places humans just don't get sensor data (e.g: back of head; outward facing from shoulders, hips, etc.; constant presence monitoring via ultrasound / IR / 2.4GHz, etc.)
But those cameras don't have to be simple visual spectrum cameras, do they?
They do, if they want to do comparative analysis on the trillions upon trillions of existing photographs, and even motion picture frames.
Not to mention the entire AI industry is working full out for perfect image recognition, again in our visual spectrum. Something apparent to a person working with openai 5 years ago.
Tesla will likely never accomplish anything with robots because they can’t even build a car to typical market standards of quality and robotics is even more greenfield and unrefined in its mass production processes. I guess maybe it’s possible if someone like Shotwell takes over and Musk is relegated to a similar figurehead position.
Because he needs to maintain market value. Musk is part inventor and part salesman. He needs high valuations to keep risky endeavors afloat. What's better than sell the future.
This is a good practice to reduce at least some amount of dependency to outside US. Also this is the definition of innovation; making high tech out of low tech.
one answer is due to the fact that humans also do this with just 2 pretty bad cameras and a lot of offloading to the cortex.
It also simplifies the stack a lot to have a single set of sensors, so the software becomes mostly: getting good training data (iterative loops from failing production cases) and an efficient training algorithm.
This scales to more than just AD and also can leverage new breakthroughs from academia
> one answer is due to the fact that humans also do this with just 2 pretty bad cameras and a lot of offloading to the cortex.
No, humans do significant sensor fusion.
There's binaural audio: useful to detect and have a rough position of emergency services and/or high speeds cars(it's true that its a relatively unencumbered channel, but that makes it all the more valuables for emergencies)
And there's a working if imperfect IMU (performance can be altered if the power supply is set to an alternate mode) to sense all kinds of acceleration: for fine course correction on acceleration and bearing, for getting the road condition and adjust the driving profile accordingly, etc..
Humans don't just have two eyes though (and our eyes are pretty damn good as far as organic light sensors go), we have 3 mirrors giving all-around sight, a whole body full of nerves providing feedback, and excellent 3D reckoning to keep track of other vehicles.
Our eyes (+ brain visual processing) are way better than the vast majority of cameras you can buy, judged on things we care about in driving. Typically only high-end film cameras approach the dynamic range of our eyes, for example. The only major thing that even cheap cameras easily beat us at is zoom, but that is mostly irrelevant for this use case.
Humans have millions of years of evolution in vision processing, and yet we still regularly have 100+ deaths a day in the US, plus many more injuries and fender benders.
I don't doubt that it's possible with machine vision alone, but it makes the challenge substantially harder.
Another way of saying this is that we do this with a system comprised of human eyes and a human brain. We’re very good at making machine eyes, but the brain part is proving extremely difficult for us to reproduce with machines.
Yup - if there was a camera today that could consistently (and without any fiddling) reproduce things the way my eyes see them (especially in high dynamic range or low light situations), I would buy it immediately!
Humans drive with only 2 cameras pointed in a single direction. A Tesla has 8 cameras pointing in every direction. So by default it would already have superhuman awareness. But the difficulty is not perception, it is planning. Turns out to drive among other humans on public roads, you need to correctly understand and predict human behaviour first, which is like half the way to AGI.
Yes - but humans possess other senses, and crucially, our eyes have a much greater dynamic range than even the best cameras (and they are not fitting the absolute best cameras, because that would be even more expensive than LiDAR).
It also does away with one of the crucial aspects of "better than human safety" - driving in conditions where a camera is uselss - think moderate to dense fog, heavy rain etc. etc.
We don't have any senses that aren't trivially replicable in a car. Microphones and accelerometers were already plentiful before. And HDR is not really an issue anymore since all carmakers dynamically vary exposure times in their cameras. Other carmakers have tried to put LiDAR and stuff into cars for many years, but it didn't magically give them self-driving either. The key issue to solve is planning - not perception.
In case you really have two eyes pointing in a single direction, you should see an oculist urgently. Mine are doing a perfect 3D picture. Also, unlike the eyes of my Tesla which becomes blind when it rains, or if there is any light reflection, I can still see things.
Also, should your eye cameras be in fixed position, you even more urgently should see a doctor. Mine can be turned 180° on the X-axis, and 120° on the Y-axis.
And finally: My eye cameras have working high-speed auto-focus, HDR, are protected against rain or snow, can operate at very low ISO, and have retina resolution.
Human eyes have much higher resolution than Tesla's cameras, better dynamic range, no issues with rain or dirt on the lens, can take measures to avoid most glare from the sun, and are connected to a brain that is vastly better at driving than AI.
It makes no sense to say "humans can do it with only X, so machines should have to as well". Do cars run? Do planes flap?
I do agree that planning should be the hardest part of self-driving, but go and look at any Tesla FSD video on YouTube and it's clear they haven't solved the perception part yet either. Cars wobbling all over the place, morphing into vans, etc.
Both from the technical and the marketing point of view having a car that can do better than any human ever could, even in the most optimal of conditions would be a great thing.