This is a smaller version of the datacomp1b dataset. Filtering was done by taking all rows that had self similarity (inner product) above 0.32. This resulted in 213009083 (213 million) rows. The results of the datacomp paper suggest that filtering by CLIP score is better than random sampling. Included in this repo are search indices created using autofaiss, over the text and image embeddings. There are two ways to access metadata, either in.parquet files in the./metadata directory, or the./index/metadata.hdf5 hdf5 file. I would suggest using embedding-reader to load the text and image embeddings.