Despite his underwhelming performances against Croatia and Scotland, Southgate has backed Kane and confirmed that the 27-year-old will start against the Czech Republic on Tuesday.
Speaking to Sky Sports News, Neville was critical of the other members in the England squad: "I said last week I felt he was undroppable and I still maintain that.
"I think that if Harry Kane had someone behind him who was even close to him that could go into the team, then you'd say maybe you could look at a change. But you wouldn't change Kane quickly, or easily for that matter.
"I think he'll break all England's records. You don't just drop your captain and your main goalscorer after two matches. But I think the energy, urgency and intensity in England's performance has got to improve for Kane to do well in this tournament.
"In the first game, we felt that was a controlled performance and everyone was really happy with that. But against Scotland, it was a massive drop off. I don't care who you are - as a centre-forward, you would have struggled in that game."
Neville reflected on England's disappointing draw against Scotland and added: "The only thing that concerns me now is the physical levels, and I'm not sure why that occurred.
"It's one of two things: one, the players are very tired, and if they are then we've got a real problem. Or two, they were a little immature and mentally built the game up so much that they drained themselves.
"For me, going into the Czech Republic game - understanding that I have played with England teams in tournaments that looked fatigued - that is the only thing that I'm looking at in the first five, 10 minutes. Is the sharpness back, and was it just a mental thing going into that Scotland game?"
England need to beat the Czech Republic on Tuesday to finish as winners of Group D, as they currently sit second behind their final opponents on goal difference.