Lawyer Harjap Singh Bhangal has insisted David Cameron "should not be going around making promises he can't deliver".
This follows reports from former aide Steve Hilton, who claims civil servants told the PM years ago his immigration target was "impossible" while Britain is a member of the EU.
Despite this, Cameron repeatedly stated his pledge to bring migration down to below 100,000 people. The target has not been reached.
Bhangal, who specialises in immigration law, insists this isn't a secret.
"Every year a report comes out which tells us the Home Office is not fit for its purpose," he told Julia Hartley-Brewer. "We know something is broken.
"It's like having a crack in a wall we just paint over, when the wall needs to be knocked down and rebuilt to support the structure."
Citizens of EU member states have freedom of movement within the union. Immigrants who come from EU countries are legally allowed to live and work in the UK while it is still a member.
"It's a real tricky situation," Bhangal explained. "The only way we can restrict immigration in the EU is if we introduce a visa system.
"But this works both ways, and therefore wouldn't benefit us.
"We haven't got the structure to deal with immigration at the moment. The Prime Minister should not be going around making promises he can't deliver on."