New Yorker editor David Remnick said in a statement to staff that he had changed his mind about interviewing Mr Bannon on stage.
The announcement that Mr Bannon would headline the festival caused a number of other guests to drop out in protest.
Those dropping out included US actor and producer John Mulaney, actor Jim Carrey and lead guitarist of indie band Fun, Jack Antonoff.
This year's guests include Emily Blunt, Zadie Smith and Sally Yates, who Mr Trump fired as deputy attorney general after she refused to back his initial ban on travellers from Muslim countries.
‘There is a better way to do this’
Mr Remnick said in his statement: “I've thought this through and talked to colleagues - and I've re-considered.
"There is a better way to do this. Our writers have interviewed Steve Bannon for The New Yorker before, and if the opportunity presents itself I'll interview him in a more traditionally journalistic setting as we first discussed, and not on stage."
Filmmaker Judd Apatow was another guest to withdraw from the festival after Mr Bannon’s interview was announced.
He wrote on twitter: “If Steve Bannon is at the New Yorker festival I am out.
“I will not take part in an event that normalises hate.”
He added: “Maybe they should read their own reporting about his ideology.”
Mr Remnick wrote that Mr Bannon was well aware of their political difference.
He said: "It's obvious that no matter how tough the questioning, Bannon is not going to burst into tears and change his view of the world. He believes he is right and that his ideological opponents are mere 'snowflakes'. The question is whether an interview has value in terms of fact, argument, or even exposure, whether it has value to a reader or an audience."