Recent theory has suggested that sympatric speciation can occur quite easily when individuals that are ecologically similar mate assortatively. Although many of these models have assumed that individuals have equal mating success, in nature rare phenotypes may often suffer decreased mating success. Consequently, assortative mating may often generate stabilizing sexual selection. We show that this effect can substantially impede sympatric speciation. Our results emphasize the need for data on the strength of the stabilizing component of selection generated by mating in natural populations.