What happens next is unclear, but Mrs Merkel is due to meet President Frank-Walter Steinmeier, who has the power to call snap elections.
Her bloc won September's poll, but many voters deserted the mainstream parties.
After winning its first parliamentary seats, the far-right nationalist AfD (Alternative for Germany) vowed to fight "an invasion of foreigners" into the country.
Mrs Merkel said she regretted the collapse of the talks, adding she would meet the German president later on Monday to formally tell him negotiations had failed.
"It is a day of deep reflection on how to go forward in Germany," she said. "As chancellor, I will do everything to ensure that this country is well managed in the difficult weeks to come."
Aside from early elections, Mrs Merkel could also form a minority government with the Greens, who are yet to comment.
German daily Frankfurter Allgemeine Zeitung called the development the worst crisis of Mrs Merkel's 12 years in office.
"Today there was no progress but rather there were setbacks because targeted compromises were questioned," Mr Lindner said.
"It is better not to rule than to rule falsely. Goodbye!" he added.
The parties involved in the talks are reported to be deeply divided over tax, asylum and environmental policies.
The most bitter arguments have been over whether Syrian refugees should be allowed to bring family members to join them in Germany, the BBC's Jenny Hill reports.
Mrs Merkel's conservatives, mindful of the electoral success of the AfD, are cautious and want to extend a moratorium on so-called family reunions, our correspondent adds.
The second-largest party in parliament, the Social Democrats, have ruled out returning to a coalition government with Mrs Merkel.