Boris Johnson refused to bow to pressure from the Prime Minister and the Conservative Party leadership to apologise for his "offensive" comments about burqas.
The former foreign secretary has caused outrage with a newspaper article comparing women in the face-covering veils to “bank robbers”, with Conservative Party chairman Brandon Lewis ordering him to apologise for the remarks.
But the standoff continued even after Theresa May added her voice to calls for him to apologise.
With no apology forthcoming, founder and president of the Conservative Muslim Forum Lord Sheikh told the BBC the party should take "severe action" against Mr Johnson.
He said: "Take the whip from him. Why not? He's not a super human being, he's a member of the party.
"The party chairman, the prime minister has the right to take the whip... that's the thing I'd like to see."
While former Conservative parliamentary candidate Shazia Awan-Scully accused Mr Johnson of "pandering to the extreme right", comparing his comments to Enoch Powell's divisive Rivers of Blood speech, and said he should be sacked.
"If you cast your mind back 50 years to when Enoch Powell gave his Rivers of Blood speech, he was sacked from the shadow cabinet immediately," she told BBC Newsnight.
"He was ostracised from his party immediately. Boris Johnson can say anything and get away with it.
"These comments are Islamophobic. They are against Muslims and I would urge any Muslims to not vote for the Conservative party because of their stance towards our community."
But The Conservative Woman co-editor Laura Perrins, appearing alongside Ms Awan-Scully, claimed Mr Johnson's remarks were taken out of context.
She said: "This idea the man should be run out of town is exactly the kind of authoritarian nonsense we must resist. It is not responsible."
Conservative peer Lady Warsi accused Mr Johnson of adopting the "dog-whistle" tactics of former Donald Trump aide Steve Bannon in the hope of attracting support from right-wing Tories for an eventual leadership bid.
Before Mrs May's intervention, a source close to Mr Johnson made clear he was not retracting his comments.
"It is ridiculous that these views are being attacked - we must not fall into the trap of shutting down the debate on difficult issues," said the source.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."