https://meta-math.github.io/ see our paper at https://arxiv.org/abs/2309.12284 All MetaMathQA data are augmented from the training sets of GSM8K and MATH. You can check the originalquestion in meta-math/MetaMathQA, each item is from the GSM8K or MATH train set. MetaMath-Mistral-7B is fully fine-tuned on the MetaMathQA datasets and based on the powerful Mistral-7B model. It is glad to see using MetaMathQA datasets and changing the base model from llama-2-7B to Mistral-7b can boost the GSM8K performance from 66.5 to 77.7. To fine-tune Mistral-7B, I would suggest using a smaller learning rate (usually 1/5 to 1/10 of the lr for LlaMa-2-7B) and staying other training args unchanged. More…