Hacker Newsnew | past | comments | ask | show | jobs | submitlogin
Nvidia announces 4-bit training with NVFP4 (nvidia.com)
2 points by opcode84 3 months ago | hide | past | favorite | 1 comment


A version of the 12B Hybrid Mamba-Transformer model was initially trained with 8-bit precision—FP8, which has been shown in previous studies to closely match 16-bit precision, and hence served as our baseline for comparison. We then successfully trained this same 12B model from scratch using NVFP4, demonstrating that this new low-precision format can support full pretraining at trillion-token scale. The NVFP4 run exhibited stable convergence without the training instabilities or divergence issues that typically plague ultra-low precision training.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: