You've just restated my point. If the deadlines are met, jitter doesn't matter. Ergo, you can't meet deadlines if your jitter is too large. Otherwise, it doesn't matter.
Wouldn't the deadline be now+zero for real time audio applications? If I'm building a guitar pedal (random example) ideally I want no delay from the input to the output. Any digital delay makes things strictly worse and so any jitter matters. That said, the difference between zero and very close to zero does become a moot point given small enough values for any practical purpose.
There are some digital audio systems that do sample-by-sample processing. Old school digidesign, for example.
But very little digital audio gear works that way these days. The buffer sizes may be small (e.g 8 or 16 samples), but most hardware uses block structured (buffer by buffer) processing.
