A Common Evaluation Setting for Just.Ask, Open Ephyra and Aranea QA systems

Abstract

Question Answering (QA) is not a new research field in Natural Language Processing (NLP). However in recent years, QA has been a subject of growing study. Nowadays, most of the QA systems have a similar pipelined architecture and each system use a set of unique techniques to accomplish its state of the art results. However, many things are not clear in the QA processing. It is not clear the extend of the impact of tasks performed in earlier stages in following stages of the pipelining process. It is not clear, if techniques used in a QA system can be used in another QA system to improve its results. And finally, it is not clear in what setting should be these systems tested in order to properly analyze their results.