Former Premier League referee Keith Hackett believes VAR specialists should operate the technology from Stockley Park, not current match officials.
The ex-head of Professional Game Match Officials Limited insisted video assistant referees were being used too much after another weekend where contentious decisions were aplenty in the top flight.
An 83rd penalty in this season's Premier League was awarded on Sunday, the same amount for the whole of the 2014-15 campaign, with technology required during the majority of games across the weekend.
"VAR is only supposed to get involved when it is a clear and obvious error," Hackett told the PA news agency.
"At the moment VAR is trying to get involved in everything. I think there needs to be a rethink of how VAR operates.
"I don't think it is great you are a referee one day and that same referee who had a good game or whatever game he had, then has to drive to Stockley Park to act as a VAR operator."
Michael Oliver took charge of Crystal Palace's home game with Burnley on Saturday and two days later was at Stockley Park to operate VAR for West Ham versus Sheffield United, while Martin Atkinson was asked to operate VAR in two fixtures this weekend instead of being out in the middle.
Hackett feels "specialists" being required for VAR would improve the way the technology is implemented in this country.
"If you are Michael Oliver or Anthony Taylor, you want them refereeing. You don't want them doing VAR or the fourth official role," he added.
"A referee is a specialist, an assistant referee is a specialist and they train. We have them running with flags, off the field of play, away from the game, practising and understanding the laws of the game and everything that goes with it.
"Each one is a different specialist role and this is how if I think we are to move forward, we need to operate VAR in the same way.
"The panel of VAR specialists, as I would call them, could be made up of former referees and that is their sole job.
"If you have people like Mike Dean and Andre Marriner, people who are on the horizon of retiring, train them up to be VAR specialists.
"The answer has to be this specialist panel who are trained, better trained than the current people, and understand the technology and they can train and analyse the film clips in front of them more quickly."
Hackett, who was referee for the 1981 FA Cup Final, pointed to the tricky situation that may arise if a younger referee told a more senior colleague to review an incident and vice versa.
With more penalty kicks set to be awarded in this season's Premier League than during the last decade, the once FIFA listed match official questioned if some had taken their eye off the ball.
"I think referees are becoming lazy," Hackett said.
"There is an analogy that Nigel Owens uses regarding TMO when explaining his career. He makes the reference that if you have no safety net, you're as sharp as a pin and really focused on the game and in the game.
"But when you have a safety net like TMO or VAR, then you become lazy. I think that is what is happening.
"They need to be as sharp as a pin, up with play and to operate successfully. Success will be VAR has not interfered with the game at all."
The PA news agency have approached PGMOL for a comment.
