Convolutional neural networks align early in training with neural representations

Task-optimized deep convolutional neural networks (DCNNs) achieve human-level performance in object recognition and are leading in explaining neural activity across various brain measurement modalities. DCNNs are trained over numerous iterations to improve performance on a task, typically object recognition, whereby the underlying assumption is that optimizing network performance translates to better explanatory power for brain activity. Contrary to this assumption, our analysis of two published datasets (fMRI, EEG) reveals that the optimal alignment between brain activity and DCNNs already occurs after the first or one of the earliest iterations, and that changes in the brain-alignment are unrelated to changes in task-performance. This implies that extensive training on one task does not result in optimal brain alignment with visual cortex. It further suggests that much could be gained by aligning the training over epochs of a DCNN with learning in biological organisms.