Mortgage companies could face penalties if they don’t take steps to prevent a deluge of foreclosures that threatens to hit the housing market later this year, a U.S. regulator said Thursday.

The Consumer Financial Protection Bureau warning is tied to forbearance relief that’s allowed million of borrowers to delay their mortgage payments due to the pandemic. To avoid what the bureau called “avoidable foreclosures” when the relief lapses, mortgage servicers should start reaching out to affected homeowners now to advise them on ways they can modify their loans.

“There is a tidal wave of distressed homeowners who will need help,” Dave Uejio, the CFPB’s acting director, said in a statement. “Servicers who put struggling families first have nothing to fear from our oversight, but we will hold accountable those who cause harm to homeowners and families.”

In a separate compliance bulletin released Thursday, the CFPB said that companies “that are unable to adequately manage loss mitigation can expect the bureau to take enforcement or supervisory action.”

More than 2 million borrowers as of January had either postponed their payments or failed to make them for at least three months, the bureau said. Once government-authorized forbearance plans begin to end in September, hundreds of thousands of people may need assistance getting back on track.