By truly open, we mean our releases use an OSI-recognized license (Apache-2) and we go beyond just model weights. Here are the things that we are open-sourcing:
i) Open-Sourced Model Weights
ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/
iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly.
v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.
It would be awesome if things weren’t rushed such that you have to say “we will” so often, rather than “here is the link”.
It’s awesome the work you all have done. But not sure if I’ll return and remember the “we will” stuff, meaning that I’m not likely every look at it or start using it.
i) Open-Sourced Model Weights
ii) Open-Sourced Fine-Tuning Pipeline. This is essentially the training code if you want to adapt this model to your use cases. This along with an associated cookbook will be released soon, so keep an eye on our repo for updates: https://github.com/Snowflake-Labs/snowflake-arctic/
iii) Open-Sourced Data Information: We trained on publicly available datasets, and we will share information on what these datasets are, how we processed and filtered them, composition of our datasets etc. They will be published as part of the cookbook series here: https://www.snowflake.com/en/data-cloud/arctic/cookbook/, shortly.
iv) Open-Sourced Research: We will share all of our findings from our architecture studies, performance analysis etc. Again these will be published as part of the cookbook series. You can already see a few blogs covering MoE Architecture and Training Systems here: https://medium.com/snowflake/snowflake-arctic-cookbook-serie..., https://medium.com/snowflake/snowflake-arctic-cookbook-serie...
v) Pre-Training System information: We actually used the already open-sourced libraries DeepSpeed and Megatron-DeepSpeed for training optimizations and the model implementation for training the model. We have already upstreamed several improvements and fixes to these libraries and will continue to do so. Our cookbooks provide the necessary information on the architecture and system configurations.