MetaMathQA

View the project page: https://meta-math.github.io/ see our paper at https://arxiv.org/abs/2309.12284

Note

All MetaMathQA data are augmented from the training sets of GSM8K and MATH. None of the augmented data is from the testing set.

You can check the original_question in meta-math/MetaMathQA, each item is from the GSM8K or MATH train set.

Model Details

MetaMath-Mistral-7B is fully fine-tuned on the MetaMathQA datasets and based on the powerful Mistral-7B model. It is glad to see using MetaMathQA datasets and changing the base model from llama-2-7B to Mistral-7b can boost the GSM8K performance from 66.5 to 77.7.

To fine-tune Mistral-7B, I would suggest using a smaller learning rate (usually 1/5 to 1/10 of the lr for LlaMa-2-7B) and staying other training args unchanged. More training details and scripts can be seen at https://github.com/meta-math/MetaMath.

Installation