Practitioners should consider using already-optimized codebases, especially in the pre-training phase, to ensure effective use of computational resources, capital, power, and effort. Existing open-source codebases targeted at foundation model pretraining can make pretraining significantly more accessible to new practitioners and help accumulate techniques for efficiency in model training.
![Pretraining Repositories for Foundation Model Training](/foundation-model-resources/model-training-pretraining-repositories/model-training-pretraining-repositories_hu39c5528b56b9aa99ea0ebd9bef51a16b_80945_736x0_resize_q90_h2_lanczos_3.webp)