MegaPairs-Standard

MegaPairs-Standard (Standardized Version)

Dataset Summary

This is a standardized, high-efficiency version of the JUNJIE99/MegaPairs dataset.

Why use this version? The original dataset is distributed as a massive Tar archive containing millions of images, accompanied by a separate JSONL annotation file. * The Problem: Using the original format requires extracting terabytes of small files (which can exhaust disk inodes) or writing complex logic to read from archives. It also requires manually mapping JSONL metadata to image paths. * The Solution (This Dataset): We have processed the data into Native Arrow format. Images are decoded and embedded directly alongside their text metadata.

Key Features: *Zero-Extraction:No need to unzip or untar anything. You can start training immediately after downloading. *Fastest Loading: Data is stored in the raw Arrow format (generated by datasets.save_to_disk). This allows for zero-copy memory mapping, offering the fastest possible local loading speed. *Self-Contained:Metadata (texts) and Images (PIL Objects) are merged into a single row. *Optimized Sharding: Data is saved in ~1GB shards for optimal network transfer and parallel processing.

Note on Preview: Since this dataset uses the native Arrow directory structure for performance, the Hugging Face "Dataset Viewer" on the website might not render the images directly. This is expected. Please follow the usage instructions below to load the data.

Dataset Structure

Each row in the dataset represents a Universal Retrieval Pair (Query -> Target).

Data Fields

Data Statistics

• Total Pairs: ~15.2M
• Original Source: JUNJIE99/MegaPairs

Usage

You can load this dataset directly using the datasets library.

Method 1: Using load_dataset (Recommended)

This is the easiest way. The library handles the Arrow files automatically.

```python from datasets import load_dataset

Load the dataset (this will download the files to your local cache)

dataset = load_dataset("86Cao/MegaPairs-Standard", split="train")

print(f"Total samples: {len(dataset)}")

Accessing data

sample = dataset[0] print(f"Text: {sample['query_texts'][0]}") sample['query_image'].show() # Displays the query image