Several theories have been proposed to explain the evolution of species differences in brain size, but no concensus has emerged. One unresolved question is whether brain size differences are a result of neural specializations or of biological constraints affecting the whole brain. Here I show that, among primates, brain size variation is associated with visual specialization. Primates with large brains for their body size have relatively expanded visual brain areas, including the primary visual cortex and lateral geniculate nucleus. Within the visual system, it is, in particular, one functionally specialized pathway upon which selection has acted: evolutionary changes in the number of neurons in parvocellular, but not magnocellular, layers of the lateral geniculate nucleus are correlated with changes in both brain size and ecological variables (diet and social group size). Given the known functions of the parvocellular pathway, these results suggest that the relatively large brains of frugivorous species are products of selection on the ability to perceive and select fruits using specific visual cues such as colour. The separate correlation between group size and visual brain evolution, on the other hand, may indicate the visual basis of social information processing in the primate brain.