A National Association of Realtors spokesman tells CNBC that homes valued at $750,000 fell by 47% in the year ended in November. Homes valued at $400,000 or less fell by only 3% during the same period.

The driving force behind that has been the decline in the stock market. High net worth people are more effected by that than lower- and middle-income workers. The average household with a net worth of over $1 million has lost 30% of its net worth recently -- and that's curbing their demand for luxury homes.

Tightening in the mortgage market for jumbo loans is also a factor driving the decline in that market.

It's especially tough for homeowners who are desperate to sell or are carrying two mortgages because rental rates on luxury homes are rarely anywhere near high enough to cover a mortgage and property taxes.

But this data does suggest that the cries about plunging property values are effecting a smaller chunk of the population than the media would have you think. High-end homes account for 2% of the property market and a much larger percentage of the downward momentum.