That's the thing - if the balls don't start clustered, then you "lose" the time direction again; now it doesn't matter which way you replay the trajectories, they are equally plausible in either direction. So if that's what you ultimately base the definition of time on, this implies that it is an emergent property.