Calc-mawps: Dataset Card

Dataset Card for Calc-MAWPS

Summary

The dataset is a collection of simple math word problems focused on arithmetics. It is derived from https://huggingface.co/datasets/omarxadel/MaWPS-ar.

The main addition in this dataset variant is the chain column. It was created by converting the solution to a simple html-like language that can be easily parsed (e.g. by BeautifulSoup). The data contains 3 types of tags:

• gadget: A tag whose content is intended to be evaluated by calling an external tool (sympy-based calculator in this case)
• output: An output of the external tool
• result: The final answer to the mathematical problem (a number)

Supported Tasks

This variant of the dataset is intended for training Chain-of-Thought reasoning models able to use external tools to enhance the factuality of their responses. This dataset presents in-context scenarios where models can outsource the computations in the reasoning chain to a calculator.

Data splits

We provide 2 variants of the dataset. In the first one, the data splits correspond to the original one and can be loaded using:

datasets.load_dataset("MU-NLPC/calc-mawps", "original-splits")

The second one is filtered to prevent data leaks (overly similar examples in train and test/val splits) in between and across datasets in Calc-X collection. Specifically, we filtered out around 2,500 near-duplicates from the train set that were similar to some instances in the MAWPS val and test splits and ASDiv-A test split. You can load this variant via:

datasets.load_dataset("MU-NLPC/calc-mawps")

Attributes:

• id: id of the example
• question: problem description in English
• question_arabic: problem description in Arabic
• chain: series of simple operations (derived from expression) that lead to the solution
• result: the solution for x as a number or fraction (string)
• result_float: same as result but converted to a float
• equation: an equation that needs to be solved for x to obtain the result. Usually in the form of "x = ..." but not always.
• expression: arithmetic expression derived from equation that solves it for x

Attributes id, question, chain, and result are present in all datasets in Calc-X collection.

Related work

This dataset was created as a part of a larger effort in training models capable of using a calculator during inference, which we call Calcformers.

• Calc-X collection - datasets for training Calcformers
• Calcformers collection - calculator-using models we trained and published on HF
• Calc-X and Calcformers paper
• Calc-X and Calcformers repo

Here are links to the original dataset:

• original MAWPS dataset
• MAWPS dataset variant in Arabic
• original MAWPS paper
• original MAWPS repo

Licence

MIT, consistent with the original source dataset linked above.

Cite

If you use this version of the dataset in research, please cite the original MAWPS paper, and Calc-X paper as follows:

@inproceedings{kadlcik-etal-2023-soft,
    title = "Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems",
    author = "Marek Kadlčík and Michal Štefánik and Ondřej Sotolář and Vlastimil Martinek",
    booktitle = "Proceedings of the The 2023 Conference on Empirical Methods in Natural Language Processing: Main track",
    month = dec,
    year = "2023",
    address = "Singapore, Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/2305.15017",
}