Climbing interest rates are already tugging on home building companies, pulling shares of these firms down as mortgages become more expensive. According to a report from CNBC, some home building stocks, like iShares Home Construction and SPDR S&P Homebuilder ETF, are down more than 20 percent since January of this year.

The interest rate for a 30-year fixed rate home loan hit five percent, and “the 10-year Treasury yield” hit a seven-year high on Tuesday, according to a report from CNBC. Just a year ago, the same mortgage would have come with an interest rate of slightly less than four percent, and two years ago the rate would have been around 3.5 percent.

“Five percent is definitely an emotional level inasmuch as it scares prospective buyers about how high rates may continue to go,” said MND CEO, Matthew Graham, while speaking to CNBC.

Higher interest rates mean it costs the average home buyer more to get a loan, and some experts believe that high-end home sales are likely to suffer the most. Real estate markets that have historically performed well, such as California and the Pacific Northwest, may see a decline in sales, as well.

“It already is affecting housing. Every basis point rate rise already is affecting housing. The run-up in rates over the past year is now weighing increasingly heavily on single-family housing demand, particularly since housing prices have risen so much,” said Moody’s Analytics chief economist, Mark Zandi, while speaking to CNBC. “Affordability is now an issue for many potential home owners. Home sales have gone sideways since the beginning of the year and house price growth is now slowing in many markets so it’s already doing damage. Some of the weakening is due to the tax effect in markets where the deduction was critical.”

However, despite the poor performance of home building stocks and higher interest rates, many economists say that they don’t expect the housing market to slow.

“There are other factors, in addition to rates that affect housing — how well income is doing, whether people have jobs,” said Macroeconomic Advisers’ economist Ben Herzon during an interview with CNBC. “Rates is just one of the factors. In our forecast, the outlook is for continued improvement in the housing market, at least for the next couple of years,” he added.

Although higher interest rates may be frightening investors into dumping home building companies, many consumers believe that now is still the right time to buy. According to a study published by Fannie Mae, more consumers said in September that now was a good time to buy compared to August. The same consumers also agreed that interest rates are likely to continue their rise.