Have you compared with something else than Chrome? Otherwise it might be that Chrome is just very power hungry compared to Safari, but maybe Firefox is more efficient by now? Chrome has slowly turned into a monster on it's own, not unlike what they competed against initially when Chrome first arrived.
>It's even more obvious when watching video where safari will be 5 to 10 points lower than Firefox.
Safari uses macOS for video so the points will be on macOS. Firefox uses it own internal video decoder. That is why image and video codec support on Safari is dependent on macOS upgrade not Safari.