Further cleaning done. Please look through the dataset and ensure that I didn't miss anything.
Dataset Card
By Z., published under apache-2.0, revision 192ab2185289.
Further cleaning done. Please look through the dataset and ensure that I didn't miss anything.
Update: Confirmed working method for training the model: https://huggingface.co/AlekseyKorshuk/vicuna-7b/discussions/4#64346c08ef6d5abefe42c12c Two choices:
- Removes instances of "I'm sorry, but": https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/blob/main/ShareGPT_V3_unfiltered_cleaned_split_no_imsorry.json
- Has instances of "I'm sorry, but": https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/blob/main/ShareGPT_V3_unfiltered_cleaned_split.json
The choice is yours. The first dataset may go to far and remove valuable data. The second is better for when the AI asks for clarification, but it also may refuse to do stuff like browse the internet, which it actually may be able to do with certain langchain implementations. These are important things to think about before training.
Details
- Repository
- anon8231489123/ShareGPT_Vicuna_unfiltered
- Publisher
- Z.
- Task category
- Not stated by the source
- Tags
- Not stated by the source
- Size category
- Not stated by the source
- Languages
- en
- Revision
- 192ab2185289094fc556ec8ce5ce1e8e587154ca
- Last updated
- 2023-04-12
SAVRN's Notes on ShareGPT_Vicuna_unfiltered
ShareGPT_Vicuna_unfiltered is a piece of open-model history: the ShareGPT V3 conversations behind the early Vicuna-style fine-tunes, 4.3 GB across 14 files, uploaded by an anonymous account ('Z.') and frozen since — the record still shows its 2023-04-12 update date. It pulled 427.6k downloads this month, mostly from people reproducing 2023-era instruction tuning, and the card links the original vicuna-7b training discussion.
The card is unusually candid about its processing: two JSON variants are offered, one scrubbing 'I'm sorry, but' refusals and one keeping them, with the author warning the scrubber 'may go to far' (sic) and remove valuable data. The page lists Apache-2.0, but that is the uploader's tag on ChatGPT transcripts, not a grant from the conversations' authors — treat commercial training as a legal judgment call, and expect no GPT-4-era data in a corpus frozen in 2023.
Files
14 files, 4.3 GB in total.
Every file
| File | Type | Size | SHA-256 |
|---|---|---|---|
| HTML_cleaned_raw_dataset/sg_90k_part1.json | Data | 921.6 MB | ed0e3af68ed9 |
| HTML_cleaned_raw_dataset/sg_90k_part1_html_cleaned.json | Data | 551.4 MB | 183c744455c6 |
| HTML_cleaned_raw_dataset/sg_90k_part2.json | Data | 932.3 MB | d74dfddfedc6 |
| HTML_cleaned_raw_dataset/sg_90k_part2_html_cleaned.json | Data | 548.2 MB | 32c3e772d044 |
| ShareGPT_V3_unfiltered_cleaned_split.json | Data | 672.8 MB | 35f0e213ce09 |
| ShareGPT_V3_unfiltered_cleaned_split_no_imsorry.json | Data | 670.5 MB | 014bcc3352fd |
| README.md | Documentation | 4.4 KB | — |
| Vicuna_unfiltered_train.ipynb | Other | 4.5 KB | — |
| clean_sharegpt.py | Other | 4.6 KB | — |
| flash_attn-0.2.8-cp39-cp39-linux_x86_64.whl | Other | 49.9 MB | ea4c106e475c |
| optional_clean.py | Other | 4.5 KB | — |
| pretty_json.py | Other | 475 B | — |
| split_long_conversation.py | Other | 3.4 KB | — |
| .gitattributes | Repository | 3.0 KB | — |
License and Download
- License
- apache-2.0
- Access
- No access gate
Released by Z. through its official repository on Hugging Face. Read the license.