PaperVault

PaperVault Dataset · 论文元数据库

项目简介 · Overview

PaperVault 是一份持续自动更新的统一论文元数据库，覆盖自然语言处理、计算机视觉、机器学习、数据挖掘、数据库、语音、系统、网络、安全、理论计算机科学、人机交互、计算机图形学与多媒体等方向的顶级会议与期刊。PaperVault is a continuously-updated, unified metadata database of papers from top-tier conferences and journals across NLP, Computer Vision, Machine Learning, Data Mining, Databases, Speech, Systems, Networking, Security, Theory, HCI, Graphics, and Multimedia.

源仓库 / Source:github.com/youngfish42/PaperVault — Web UI、REST API、采集流水线、Issues / PRs 全部在那里 · Web UI, REST API, crawling pipelines and issues/PRs all live there.

在线搜索 / Live search:papervault.top — 直接使用 Web 检索界面，无需下载数据 · Use the web search UI directly without downloading the dataset.

最近更新 · Recent Update

• 最近更新 · Last updated: 2026-10-18 (Asia/Shanghai)
• 数据库规模 · Database size: 714,769 篇论文 / 129 个刊物系列 / 625,152 篇含摘要 / 112,086 篇含开源代码（714,769 papers / 129 venue series / 625,152 with abstract / 112,086 with code）

数据看板 · Statistics at a Glance

下列 4 张统计图与 cache.jsonl.gz 同源同步，反映本数据集的最新状态。The four charts below are generated from the same cache.jsonl.gz and always reflect the latest state of this dataset.

数据集内容 · What's in this dataset

Hugging Face 会自动为 cache.jsonl.gz 生成 Parquet 视图，也可直接用 datasets.load_dataset(...) 读取，无需手动解压。Hugging Face also exposes an auto-generated Parquet view, so datasets.load_dataset(...) works out of the box.

Dataset Viewer 与datasets.load_dataset("youngfish42/PaperVault") 默认展示/加载的都是 papers 子集（即 cache/cache.jsonl.gz）。如需查看回填进度，请在 Viewer 顶部下拉框切换到 abstract_backfill_progress，或调用 load_dataset("youngfish42/PaperVault", name="abstract_backfill_progress")。The Dataset Viewer and datasets.load_dataset("youngfish42/PaperVault") both default to the papers subset (cache/cache.jsonl.gz). To inspect backfill progress, switch the Viewer's subset dropdown to abstract_backfill_progress or call load_dataset("youngfish42/PaperVault", name="abstract_backfill_progress").

字段 Schema

缺失字段请按空字符串处理。Treat missing fields as empty strings.

⬇ 获取方式 · How to download

下面三种方式任选其一即可，无需克隆 GitHub 仓库。Pick any one of the three options below — no GitHub clone is required.

方式 A · Option A — huggingface_hub（推荐 / recommended）

from huggingface_hub import hf_hub_download
import gzip, json

path = hf_hub_download(
    repo_id="youngfish42/PaperVault",
    filename="cache/cache.jsonl.gz",
    repo_type="dataset",
)

with gzip.open(path, "rt", encoding="utf-8") as f:
    for line in f:
        record = json.loads(line)
        # 在这里处理一条记录 · do something with the record

方式 B · Option B — datasets

from datasets import load_dataset

ds = load_dataset("youngfish42/PaperVault")
print(ds[next(iter(ds))][0])

数据集只有一个默认 split（非 ML 训练集），不要传 split="train"。Single default split — do not pass split="train".

方式 C · Option C — huggingface-cli / 直接 HTTPS · Plain HTTPS

huggingface-cli download youngfish42/PaperVault \
    cache/cache.jsonl.gz --repo-type dataset --local-dir ./data

文件压缩后约 120 MB（会随数据持续增长），解压后是 GB 级 JSONL 流，请按行流式读取，不要整体载入内存。The file is ~120 MB compressed (and growing) and decompresses to a multi-GB JSONL stream. Stream it line-by-line rather than loading the whole thing into memory.

更新节奏 · Update cadence

数据集由三个 GitHub Actions 工作流负责重建并推送到本 Hub 仓库 / The dataset is rebuilt and pushed to this Hub repo by three GitHub Actions workflows：

每次推送都使用 Hugging Face 的 parent_commit 乐观锁机制，避免并发覆盖。Each push uses Hugging Face's parent_commit optimistic-lock mechanism to avoid silently overwriting concurrent updates.

线上服务 · Live Service

无需下载数据集即可直接体验 Web 检索： No need to download the dataset — try the web search directly:

papervault.top

关联仓库 · Related repository

如果你需要完整的搜索 Web UI（智能搜索 + Web of Science 风格的高级查询 DSL）、REST API（/api/v1/*）、抓取 / 合并 / 摘要回填流水线源码、收录会议范围、统计仪表盘、项目截图或贡献指南，请前往 GitHub 项目仓库。If you are looking for the full search Web UI (smart search + Web-of-Science-style advanced query DSL), the REST API (/api/v1/*), the crawling / merging / abstract-backfill pipelines source code, conference coverage, statistics dashboards, screenshots or contribution guidelines, please visit the GitHub repository.

github.com/youngfish42/PaperVault

许可证 · License

代码以 GPL-3.0 发布；每条论文记录的著作权仍属于原作者 / 出版方，本数据集仅重新分发公开可获取的元数据与链接。Code is released under GPL-3.0; individual paper records remain the IP of their authors/publishers — this dataset only redistributes publicly available bibliographic metadata and links.