Recently, scholars started to focus on the degree of perceived polarization among the mass public in the US. Perceptions are important drivers for political behavior, so the question whether individuals overstate the polarization of party positions is an important one. Based on a theory of the individual perception of political scales that combines insights from psychophysics, political psychology and scaling, we derive a statistical model that allows us to simultaneously estimate perceptual biases and latent positions. We apply the method to study party perception in the U.S. public. The results show that partisans hold diametrically opposing beliefs about the ideological space. Political knowledge reduces bias in the perception of the supported party but increases bias in perception of the opposing party. Further, distortions in the perceptions of the political space of both partisans and independents have increased substantially since the year 2000, reaching an all-time high in 2012.