CompMix

CompMix A Benchmark for Heterogeneous Question Answering CompMix 2022 long paper on Conversational Question Answering on Heterogeneous Sources. We provide an extended video as an introduction to the work (a shorter video is also available here ). --> --> Download Training Set (4966 questions) Download Dev Set (1680 questions) Download Test Set (2764 questions) The CompMix benchmark is licensed under a Creative Commons Attribution 4.0 International License . The CompMix dataset is also available at Hugging Face : https://huggingface.co/datasets/pchristm/CompMix Description CompMix collates the completed versions of the conversational questions in ConvMix, that are provided directly by crowdworkers from Amazon Mechanical Turk (AMT). Questions in CompMix exhibit complex phenomena like the presence of multiple entities, relations, temporal conditions, comparisons, aggregations, and more. It is aimed at evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes). The dataset has 9,410 questions , split into train (4,966 questions), dev (1,680), and test (2,764) sets. All answers provided in the CompMix dataset are grounded to the KB (except for dates which are normalized, and other literals like names). Further details will be provided in a dedicated write-up soon. How was CompMix created? CompMix collates the completed versions of the conversational questions in ConvMix, and are provided directly by the crowdworkers. The ConvMix benchmark, on which CompMix is based, was created by real humans. We tried to ensure that the collected data is as natural as possible. Master crowdworkers on Amazon Mechanical Turk (AMT) selected an entity of interest in a specific domain, and then started issuing conversational questions on this entity, potentially drifting to other topics of interest throughout the course of the conversation. By letting users choose the entities themselves, we aimed to ensure that they are more interested...