Visual word representation in the brain

The human visual system is thought to use features of intermediate complexity for scene representation. How the brain computationally represents intermediate features is unclear, however. To study this, we tested the Bag of Words (BoW) model in computer vision against human brain activity. This computational model uses visual word histograms, candidate features of intermediate complexity, to represent visual scenes, and has proven effective in automatic object and scene recognition. We analyzed where in the brain and to what extent human fMRI responses to natural scenes can be accounted for by BoW representations. Voxel-wise application of a distance-based variation partitioning method reveals that BoW representations explain brain activity in visual areas V1, V2 and in particular V4. Area V4 is known to be tuned for features of intermediate complexity, suggesting that the BoW model captures intermediate-level scene representations in the human brain.