Welcome to our curated collection of resources for responsible foundation model development! Here, you’ll find a diverse array of tools, artifacts, and insightful papers aimed at guiding developers in navigating the complexities of model development. Our selection criteria emphasize the usefulness, documentation quality, and community awareness of each resource.
Understand the importance of pretraining data for foundation models. Careful data selection impacts model behavior and capabilities.
Discover the breadth of finetuning data sources available for foundation models. From HuggingFace Datasets to specialized catalogs, find resources with strong documentation and diverse data sets.
Learn how to explore and analyze training datasets effectively for foundation models. Understand the nuances of data distributions, topics, and formats to better train your model.
Master data cleaning, filtering, and mixing techniques for foundation model datasets.
Learn about data deduplication, a crucial preprocessing step for foundation model datasets. Discover how removing duplicates enhances model training efficiency and reduces the risk of memorizing undesirable information.
Explore data decontamination techniques for foundation model training datasets. Learn how to protect test data integrity and ensure reliable model evaluation with canaries and proactive decontamination methods.
Discover the importance of auditing datasets in foundation model development. Learn how systematic studies and exploration tools can ensure dataset integrity and effectiveness.
Understand the significance of data documentation for foundation model datasets. Thorough documentation ensures users understand data usage, legal restrictions, and privacy concerns, despite potential errors in crowdsourced documentation.
Explore data governance practices for foundation model datasets. Learn about data curation, access control, and enabling data subjects to request removal from hosted datasets to ensure compliance with privacy and legal requirements.
Discover pretraining repositories for foundation model development. Explore existing open-source codebases tailored for pretraining to optimize computational resources and enhance accessibility for new practitioners.
Explore finetuning repositories for foundation model development. Access resources for adapting foundation models after pretraining to ensure greater ecosystem compatibility and reduce barriers to experimentation.
Learn about efficiency and resource allocation in foundation model training. Explore resources and best practices for optimizing resource usage, reducing training costs, and maximizing the environmental impact of model training.
Discover educational resources for foundation model training. Access materials to learn about the considerations and best practices for successfully training or fine-tuning foundation models.
Explore evaluation capabilities for foundation models. Understand the challenges in evaluating open-ended use cases and discover benchmarks and methodologies for assessing model performance in diverse tasks and applications.
Discover taxonomies for evaluating risks and harms in foundation models. Learn about categorizing and understanding risks and hazards associated with AI systems, including issues related to hate speech, cybersecurity, and misuse of AI capabilities.
Explore evaluations of risks and harms in foundation models. Understand the importance of assessing risks and harms, and discover methodologies and taxonomies for evaluating potential risks, mitigations, and decision-making in model development and deployment.
Learn about model documentation for foundation models. Discover standards and tools for effectively documenting models, including specifications for model usage, recommended use cases, potential risks, and decisions made during training.
Understand the importance of reproducibility in foundation model development. Learn about the challenges of replicating evaluation results and discover best practices for ensuring scientific reproducibility through clear code, documentation, and setup.
Explore license selection considerations for foundation models. Learn about different types of licenses and their implications for model distribution, use, and adaptation. Discover resources and examples to help guide developers in selecting appropriate licenses for their models.
Discover resources for usage monitoring in foundation model development. Explore techniques for monitoring model usage, including watermarking, access control, and reporting adverse events. Learn about challenges and considerations in implementing usage monitoring strategies.
The foundation model cheatsheet is a curated collection of tools, datasets, code examples, and papers that guide the development of foundation models (large language models, image generators, etc.). It’s designed for newer developers who want to build and release these models responsibly.
This cheatsheet was created to: 1) Help developers navigate the complex landscape of responsible foundation model development. 2) Provide guidance on mitigating potential misuses or harms of these models. 3) Highlight helpful resources that might not be widely known.
The cheatsheet includes: data catalogs (especially for less common languages), tools for searching and analyzing data, repositories for evaluating models, and papers that summarize key development decisions.
The cheatsheet is a living document! You can contribute new resources by following the instructions given on the website. Your contributions will be reviewed for relevance and quality.
It’s primarily aimed at newer foundation model developers. Larger organizations that build commercial AI products have additional factors to consider.
Currently, it’s focused on text, vision, and speech models. The creators acknowledge that this is just a starting point.
