Referee Darren Drysdale has been removed from taking charge of the League Two fixture between Southend and Bolton on Saturday after his confrontation with Ipswich's Alan Judge, the EFL has confirmed.
Drysdale was charged with improper conduct by the Football Association on Wednesday over the Portman Road incident.
The 50-year-old Lincolnshire official appeared to square up to Judge and lean his head into the Republic of Ireland international after the midfielder's appeal for a penalty had been turned down during Ipswich's goalless draw with Northampton on Tuesday evening.
Drysdale, who apologised for his actions, has been removed from this weekend's EFL appointments list and Declan Bourne will now referee the Southend v Bolton game at Roots Hall.
Ipswich manager Paul Lambert had called Drysdale's apology "soft" and asked for the matter to be investigated.
But Judge told the Ipswich website: "Referees have a tough job to do and it was heat of the moment stuff that happens in football.
"The photo makes it look worse than it was and to be honest, the matter was finished with from my end as soon as I walked off the pitch.
"I wasn't looking for the referee to apologise; I wasn't looking for him to be charged. There was never going to be a complaint from me and I made that clear.
"Like I said, for me – it was finished with straight away."
Drysdale, who has until March 4 to provide a response to the FA, said in a statement released by the Professional Game Match Officials Limited before he was charged: "I fully understand that it is important for us as referees to maintain our composure throughout the game and always engage with players in a professional manner.
"I'm sorry that I did not do that last night and I can only apologise to Alan and Ipswich Town."
The referee has received support from a number of players on social media, including Aston Villa's England internationals Jack Grealish and Tyrone Mings.
Grealish wrote: "Ah come on?! No need to apologise at all! I think it's brilliant."
And his team-mate Mings believes more needs to be done to ensure referees receive respect from players.
"Refs and officials put up with a lot from players. I'm sure he isn't proud of it but in the world we're living in, I think you can allow for some compassion when people don't always act as they should," Mings replied to a tweet.
"I agree that they should be impartial. But methodical? They are human and like I said, some of the stuff said to them isn't nice.
"It probably highlights how professional they are and how well of a job they do that more of them haven't reacted like that."