In a speech at his party’s spring conference, the Lib Dem leader said too many older people who voted Leave longed for a world where “faces were white” and were “driven by nostalgia”.
Tory MPs have labelled the comments as “wrong” and “unwise”.
But Sir Vince called it a “simple truth of the matter” that the majority of the older generation voted Leave while younger people favoured Remain.
In his speech on Sunday, Sir Vince said a “nostalgia for a world where passports were blue, faces were white and the map was coloured imperial pink” had driven some older voters to Brexit.
“And it was their votes on one wet day in June which crushed the hopes and aspirations of young people for years to come,” he added.
Asked on BBC Radio 4’s Today programme whether he was suggesting older Brexit voters were racist, Sir Vince said: “I didn’t suggest that at all.”
But he repeated his claim that “nostalgia for that world” was a factor in how people had voted.
“Why else has so much fuss been made about the change in the colour of the passport?” he added.
Conservative MPs have criticised the Lib Dem leader over his speech.
Housing, Communities and Local Government Secretary Sajid Javid said: “Sir Vince should be trying to bring country together, not seeking to tear it apart.”
Tory party chair Brandon Lewis said his comments were “rude” and “offensive”.
Conservative MEP Daniel Hannan said “calling 17.4 million people racists is unfair and unwise” and the deputy chairman of the Conservatives, James Cleverly, said “not liking brown faces” was not the reason he voted Brexit.
Sir Vince Cable should apologise for his remarks, he has just insulted the very people who gave us the nation we have today, a free and prosperous society, a country with a proud history as well as an exciting future.
— Andrew Rosindell MP (@AndrewRosindell) March 11, 2018
The Lib Dems are campaigning for a second referendum on the final Brexit deal, which the government opposes.
Sir Vince told his audience in Southport: “I’ve myself been on a journey. I confess that my own initial reaction to the referendum was to think maybe there was little choice but to pursue Brexit.
“I thought, you know, the public had voted to be poorer – well, that was their right.
“What changed my mind was the evidence that Brexit had overwhelmingly been the choice of the older generation.
“75% of under 25s voted to remain. But 70% of over 65s voted for Brexit,” he said.
Sir Vince also took a swipe at his own party’s lack of diversity – it has had a lower proportion of non-white MPs and candidates than Labour or the Conservatives in recent years.
“Looking around the auditorium, we are very, very white,” he told the party faithful.
“We must prioritise making our party more ethnically diverse.”