>But those also happen quite infrequently, so a vehicle that doesn't handle road debris (or deer or rare obstructions) can work with supervision

Rare obstructions are exactly where "supervision" fails because the reaction is needed faster than the supervisor can take over.