If someone votes for a party without a manifesto, then whatever that party does they will claim to do in the names of those who voted for them. And those people will not be able to deny it however unwelcome, even abhorrent, that party's actions.
(Of course, they could say they didn't mean to...