The data construction workflow can be summarized as follows: 1. Deduplicate: The FineWeb dataset is deduplicated using exact deduplication and MinHash techniques to remove redundant data. 2. URL Labeling: Root URLs from FineWeb are counted, and the top 1 million URLs are labeled using GPT-4. This step generates DoI (Domain-of-Interest) Coarse-Grained URLs and DoNI (Domain-of-Non-Interest) Coarse-Grained URLs as seed data sources. 3. Coarse Recall: a. Based on the labeled root URLs, data is sampled for each domain. b. The sampled data is labeled using Qwen2-7B-Instruct, producing 500K DoI Positive Data and 500K DoI Negative Data (note that for N>1 iterations, each 500K samples are composed…
Publicly accessible
apache-2.0
n>1T
これに伴い、2009年11月から運用されてきた旧システムは提供終了となり（事実上のサービス終了）、torne や BRAVIA などの家電への対応が軒並み終了する中、当時の生の声が詰まった約11年分の過去ログも同時に失われることとなってしまいました。 そこで 5ch の DTV 板の住民が中心となり、旧ニコニコ実況が終了するまでに11年分の全チャンネルの過去ログをアーカイブする計画が立ち上がりました。紆余曲折あり Nekopanda 氏が約11年分のラジオや BS も含めた全チャンネルの過去ログを完璧に取得してくださったおかげで、11年分の過去ログが電子の海に消えていく事態は回避できました。 しかし、旧 API が廃止されてしまったため過去ログを API 経由で取得することができなくなり、またアーカイブされた過去ログから見たい範囲のログを探す場合も、アーカイブのサイズが合計約 150GB もあることから、とても以前のように手軽に過去ログに触れることはできなくなってしまいました。 このデータセットでは、ニコニコ実況のすべての過去ログを後世に残すべく、Nekopanda 氏が配布されていた旧ニコニコ実況の 2020/12/15 までのすべての過去ログに加え、コミュニティでの実況番組も含めた新ニコニコ実況、さらに 2024/06/10 からは実況用代替コメントサーバーである NX-Jikkyo の当日分の過去ログを5分に1回収集し、随時反映しています。
Publicly accessible
mit
GLUE, the General Language Understanding Evaluation benchmark (https://gluebenchmark.com/) is a collection of resources for training, evaluating, and analyzing natural language understanding systems. The leaderboard for the GLUE benchmark can be found at this address. It comprises the following tasks: A manually-curated evaluation dataset for fine-grained analysis of system performance on a broad range of linguistic phenomena. This dataset evaluates sentence understanding through Natural Language Inference (NLI) problems. Use a model trained on MulitNLI to produce predictions for this dataset. The Corpus of Linguistic Acceptability consists of English acceptability judgments drawn from…
Publicly accessible
other
10K<n<100K
SuperGLUE (https://super.gluebenchmark.com/) is a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, improved resources, and a new public leaderboard. An example of 'test' looks as follows. An example of 'test' looks as follows. An example of 'train' looks as follows. An example of 'train' looks as follows. An example of 'train' looks as follows. The data fields are the same among all splits. - sentence1: a string feature. - sentence2: a string feature. - idx: a int32 feature. - label: a classification label, with possible values including entailment (0), notentailment (1). - premise: a string feature. - hypothesis: a string feature. - idx: a…
Publicly accessible
other
10K<n<100K
This dataset contains 108,463 human-labeled and 656k noisily labeled pairs that feature the importance of modeling structure, context, and word order information for the problem of paraphrase identification. The dataset has two subsets, one based on Wikipedia and the other one based on the Quora Question Pairs (QQP) dataset. For further details, see the accompanying paper: PAWS: Paraphrase Adversaries from Word Scrambling (https://arxiv.org/abs/1904.01130) PAWS-QQP is not available due to license of QQP. It must be reconstructed by downloading the original data and then running our scripts to produce the data and attach the labels. The text in the dataset is in English. Below are two…
Publicly accessible
other
100K<n<1M
Dataset · Text classification
Google
BoolQ is a question answering dataset for yes/no questions containing 15942 examples. These questions are naturally occurring ---they are generated in unprompted and unconstrained settings. Each example is a triplet of (question, passage, answer), with the title of the page as optional additional context. The text-pair classification setup is similar to existing natural language inference tasks. An example of 'validation' looks as follows. The data fields are the same among all splits. - question: a string feature. - answer: a bool feature. - passage: a string feature. BoolQ is released under the Creative Commons Share-Alike 3.0 license. Thanks to @lewtun, @lhoestq, @thomwolf…
Publicly accessible
cc-by-sa-3.0
10K<n<100K
Large Movie Review Dataset. This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well. An example of 'train' looks as follows. The data fields are the same among all splits. - text: a string feature. - label: a classification label, with possible values including neg (0), pos (1). Thanks to @ghazi-f, @patrickvonplaten, @lhoestq, @thomwolf for adding this dataset.
Publicly accessible
other
10K<n<100K
The Adversarial Natural Language Inference (ANLI) is a new large-scale NLI benchmark dataset, The dataset is collected via an iterative, adversarial human-and-model-in-the-loop procedure. ANLI is much more difficult than its predecessors including SNLI and MNLI. It contains three rounds. Each round has train/dev/test splits. English An example of 'trainr2' looks as follows. The data fields are the same among all splits. - uid: a string feature. - premise: a string feature. - hypothesis: a string feature. - label: a classification label, with possible values including entailment (0), neutral (1), contradiction (2). - reason: a string feature. Thanks to @thomwolf, @easonnie, @lhoestq…
Publicly accessible
cc-by-nc-4.0
100K<n<1M
Dataset · Text classification
Ceval
C-Eval is a comprehensive Chinese evaluation suite for foundation models. It consists of 13948 multi-choice questions spanning 52 diverse disciplines and four difficulty levels. Please visit our website and GitHub or check our paper for more details. Each subject consists of three splits: dev, val, and test. The dev set per subject consists of five exemplars with explanations for few-shot evaluation. The val set is intended to be used for hyperparameter tuning. And the test set is for model evaluation. More details on loading and using the data are at our github page. Please cite our paper if you use our dataset.
Publicly accessible
cc-by-nc-sa-4.0
10K<n<100K
BLiMP is a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each containing 1000 minimal pairs isolating specific contrasts in syntax, morphology, or semantics. The data is automatically generated according to expert-crafted grammars. An example of 'train' looks as follows. An example of 'train' looks as follows. An example of 'train' looks as follows. An example of 'train' looks as follows. An example of 'train' looks as follows. The data fields are the same among all splits. - sentencegood: a string feature. - sentencebad: a string feature. - field: a string feature. - linguisticsterm: a string…
Publicly accessible
cc-by-4.0
10K<n<100K
Dataset · Text classification
Ai2
Given a partial description like "she opened the hood of the car," humans can reason about the situation and anticipate what might come next ("then, she examined the engine"). SWAG (Situations With Adversarial Generations) is a large-scale dataset for this task of grounded commonsense inference, unifying natural language inference and physically grounded reasoning. The dataset consists of 113k multiple choice questions about grounded situations (73k training, 20k validation, 20k test). Each question is a video caption from LSMDC or ActivityNet Captions, with four answer choices about what might happen next in the scene. The correct answer is the (real) video caption for the next event in…
Publicly accessible
unknown
100K<n<1M
The SNLI corpus (version 1.0) is a collection of 570k human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral, supporting the task of natural language inference (NLI), also known as recognizing textual entailment (RTE). Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), is the task of determining the inference relation between two (short, ordered) texts: entailment, contradiction, or neutral (MacCartney and Manning 2008). See the corpus webpage for a list of published results. The language in the dataset is English as spoken by users of the website Flickr and as spoken by…
Publicly accessible
cc-by-sa-4.0
100K<n<1M
A continuously updated, de-identified corpus of Ukrainian Telegram news and the discussion around it, published by the Ukrainian non-profit Aisberg (ГО «АЙЗБЕРГ»). It comes in two layers. The first is the raw monthly stream: every post from a fixed set of public news channels, with its reactions and its comment thread. The second is the analysis behind every report Aisberg publishes: posts from different channels grouped into one event, the manipulation techniques found in the coverage, who published first and by how many minutes, view and reaction curves over the first hour, comment sentiment, and a fact-check with a ten-label verdict and the full evidence chain behind it. Free to use…
Publicly accessible
cc-by-4.0
100K<n<1M
Continuous knowledge datasets from the IDA Dataset Factory. - {name}.csv · {name}.jsonl · {name}.parquet (when available) · README.md See repository LICENSE. Public knowledge from trusted open sources.
Publicly accessible
other
1K<n<10K
Dataset · Text classification
Nyaamoe
NICOLOG アニメコメントアーカイブ commeonやNCOverlayでコメント付きのアニメを楽しもう！
Publicly accessible
mit
