We present results on using explanations as auxiliary features to improve stacked ensembles for Visual Question Answering (VQA). VQA is a challenging task that requires systems to jointly reason about natural language and vision. We present results applying a recent ensembling approach to VQA, Stacking with Auxiliary Features (SWAF), which learns to combine the results of multiple systems. We propose using features based on explanations to improve SWAF. Using explanations we are able to improve ensembling of three recent VQA systems.

PDF

In Proceedings of the IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI), pp. 43-47, Melbourne, Australia, August 2017.

@inproceedings{rajani:xai17, title={Using Explanations to Improve Ensembling of Visual Question Answering Systems}, author={Nazneen Fatema Rajani and Raymond J. Mooney}, booktitle={Proceedings of the IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)}, month={August}, address={Melbourne, Australia}, pages={43-47}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=127647", year={2017} }

Poster