The Cauldron is part of the Idefics2 release. It is a massive collection of 50 vision-language datasets (training sets only) that were used for the fine-tuning of the vision-language model Idefics2. To load the dataset, install the library datasets with pip install datasets. Then, to download and load the config ai2d for example. An example of a sample looks as follows: In images, there is a list of images, to be placed before the text. In texts, there is a conversation between a user and an assistant about the images that is represented by a list of turns. The Cauldron contains only the train split of each sub-datasets. On top of that, we removed the few examples containing an image also…
Publicly accessible
This dataset was created using LeRobot.
Publicly accessible
apache-2.0
Publicly accessible
CommonsenseQA is a new multiple-choice question answering dataset that requires different types of commonsense knowledge to predict the correct answers. It contains 12,102 questions with one correct answer and four distractor answers. The dataset is provided in two major training/validation/testing set splits: "Random split" which is the main evaluation split, and "Question token split", see paper for details. The dataset is in English (en). An example of 'train' looks as follows: The data fields are the same among all splits. - id (str): Unique ID. - question: a string feature. - questionconcept (str): ConceptNet concept associated to the question. - choices: a dictionary feature…
Publicly accessible
mit
1K<n<10K
Publicly accessible
This dataset contains the pre-training phase data used to train all Ettin encoder and decoder models. The data is provided in MDS format ready for use with Composer and the ModernBERT training repository. For pre-training, see the ModernBERT repo: https://github.com/AnswerDotAI/ModernBERT Each folder contains one data source in MDS (Mosaic Data Shard) format: - arxiv/ - Academic papers from ArXiv - books/ - Literature and reference books - cchead/ - High-quality Common Crawl documents - ccnews/ - News articles from Common Crawl - dclm/ - DataComp-LM filtered web data - openwebmath/ - Mathematical web content - algebraicstackexchange/ - Math Q&A from StackExchange - pes2o/ - Scientific…
Publicly accessible
mit
Publicly accessible
apache-2.0
Publicly accessible
ABC-130k ABC-130k is the largest open-source robot teleoperation dataset. It contains bimanual manipulation trajectories collected on two-arm YAM stations. Episodes are distributed as MCAP files, with subtask annotations kept as separate artifacts so they can be revised or extended independently of the underlying episode data. For details on the accompanying paper, see abc.bot. Please see the GitHub repo here for code to train and deploy with this dataset. Dataset
Access requested at publisher
apache-2.0
n>1T
Publicly accessible
SWE-bench Verified is a subset of 500 samples from the SWE-bench test set, which have been human-validated for quality. SWE-bench is a dataset that tests systems’ ability to solve GitHub issues automatically. See this post for more details on the human-validation process. The dataset collects 500 test Issue-Pull Request pairs from popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. The original SWE-bench dataset was released as part of SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Want to run inference now? This dataset only contains the problemstatement (i.e. issue text) and the basecommit which…
Publicly accessible
Prophet's Mosque Library Overview Prophet’s Mosque Library is one of the primary resources for Islamic books. It hosts more than 48,000 PDF books across over 70 categories. In this dataset, we processed the original PDF files using Google Document AI APIs and extracted their contents into two additional formats: TXT and DOCX. Dataset Contents The dataset includes 70,884 PDF files (spanning 23,494,042 pages) representing 48,717 Islamic books. Each book is
Publicly accessible
mit
10K<n<100K
Studying scalable methods for visual generative modeling requires large, accessible, and stable datasets. We introduce GPIC, a Giant Permissive Image Corpus of approximately 28 trillion pixels. GPIC comprises diverse internet images captioned by a state-of-the-art vision-language model, including 100M training, 200K validation, and 1M test examples. Moreover, all GPIC images are permissively licensed for both research and commercial use. GPIC is safety-filtered, deduplicated, and centrally hosted on Hugging Face. We provide a benchmarking protocol for generative modeling on GPIC. Finally, we provide a reference baseline for pixel-space flow matching on GPIC. Our dataset, benchmark, and…
Access requested at publisher
mit
FineWeb-HQ is a high-quality, model-filtered pretraining dataset derived as a subset of FineWeb. FineWeb-HQ was created by selecting the top 10% of FineWeb documents based on a deep learning classifier trained to identify structured and knowledge-rich samples. This classifier uses XLM-RoBERTa embeddings to score documents. To validate our approach, we pretrained 1B-parameter LLM models with a Llama-like architecture across multiple languages and scripts. The results showed improvements on standard English benchmarks, with our dataset outperforming its English counterparts DCLM and FineWeb-Edu. For its multilingual version, FineWeb2-HQ, evaluations on CMMLU (Chinese), MMLU (German), and MMLU…
Publicly accessible
odc-by
n>1T
Publicly accessible
Access requested at publisher
cc-by-4.0
The HumanEval dataset released by OpenAI includes 164 programming problems with a function sig- nature, docstring, body, and several unit tests. They were handwritten to ensure not to be included in the training set of code generation models. The programming problems are written in Python and contain English natural text in comments and docstrings. An example of a dataset instance: - taskid: identifier for the data sample - prompt: input for the model containing function header and docstrings - canonicalsolution: solution for the problem in the prompt - test: contains function to test generated code for correctness - entrypoint: entry point for test The dataset only consists of a test split…
Publicly accessible
mit
n<1K
FastUMI-100K is a large-scale, high-quality UMI-style dataset designed for data-driven robotic manipulation learning. Featuring over 100K+ demonstration trajectories across 54 diverse tasks and hundreds of object types, the dataset provides multi-view wrist-mounted fisheye images and high-frequency end-effector states. To facilitate seamless integration with modern robot learning frameworks, all data has been standardized into the LeRobot v2.1 format. To use the FastUMI-100K dataset, you need to install the official Hugging Face LeRobot library. (For advanced usage or compiling from source, please refer to the official LeRobot repository.) The dataset is recorded at 20 FPS. All states and…
Publicly accessible
Publicly accessible
100M<n<1B
Publicly accessible
Publicly accessible
Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable. SQuAD 1.1 contains 100,000+ question-answer pairs on 500+ articles. Question Answering. English (en). An example of 'train' looks as follows. The data fields are the same among all splits. - id: a string feature. - title: a string feature. - context: a string feature. - question: a string feature. - answers: a dictionary feature containing: - text: a string feature. - answerstart: a int32…
Publicly accessible
cc-by-sa-4.0
10K<n<100K
Wikipedia dataset containing cleaned articles of all languages. The dataset is built from the Wikipedia dumps (https://dumps.wikimedia.org/) with one subset per language, each containing a single train split. Each example contains the content of one full Wikipedia article with cleaning to strip markdown and unwanted sections (references, etc.). All language subsets have already been processed for recent dump, and you can load them per date and language this way: Click the Nomic Atlas map below to visualize the 6.4 million samples in the 20231101.en split. The dataset is generally used for Language Modeling. You can find the list of languages here…
Publicly accessible
cc-by-sa-3.0
n<1K
Publicly accessible
