They were written off as a waste of time and have learned to dodge cups of tea as well as punches when their antics have got too much for their long-suffering trainer.
But the McCormack twins, Pat and Luke, will highlight another remarkable chapter in the story of Birtley Boxing Club next week when it contributes a third of the Great Britain team for the European Games in Minsk.
The twins will travel alongside Calum French and 19-year-old Mark Dickinson to the Games, which will also double as the sport’s notoriously tough European Championships.
Rutherford, their no-nonsense club trainer, is credited by all four as the reason why they turned around relatively underwhelming starts to their careers to emerge as genuine Olympic prospects for Tokyo 2020.
Pat McCormack, who was the only one of the twins to go to Rio where he lost a controversial decision to Cuba’s Yasniel Toledo in his second bout, has responded by embarking on a two-year unbeaten streak.
“I like the way Graeme doesn’t take any crap off people,” said McCormack. “If he wants you to do something, you’d better do it. I think that – and what happened in Rio – has helped me grow up a lot more.”
For French, who gave up boxing in his early teens before being rescued by Rutherford, last year was a challenging one.
Arriving at the 2018 European Championships on a 30-fight winning streak, the 23-year-old saw it come to an end in the semi-finals and went on to suffer a handful of other painful losses, most notably at the Gold Coast Commonwealth Games.
“When I first got on the squad I went on a 35-fight unbeaten run and I knew there would have to be a bit of a trough,” said French.
“Graeme has been a massive influence on helping me turn things back around. He’s always there to pick me up and make me realise how much I want it. I’ve got the trough out of the way before Tokyo.”
Aged 19, Minsk will be Dickinson’s first major senior international but having started at the club at the age of six, he has grown used to travelling with French and the McCormacks through the junior ranks.
The production line is far from finished: Cyrus Pattinson is also a member of the elite GB squad, Thomas Hodgson is on the second-level podium potential, while Hannah Robinson and Georgia O’Connor are both in the women’s development programme.
But it is arguably on Luke McCormack, who had to endure the “sickening” sight of his twin brother heading off to Rio without him, that the influence of a firm hand such as Rutherford’s has been most keenly felt.
Failure to win a gold medal he expected on the Gold Coast has refocused a fighter who is doubly determined not to let another Olympic chance slip by – and be part of an ever-increasing Birtley band.
“The Commonwealth Games got to my head a bit,” admitted McCormack. “I was getting in the papers and enjoying myself too much. They say you learn from your mistakes and I’ve made every mistake going.
“I’m totally different now. I’ve improved a lot in and out of the ring. Graeme’s thrown plenty of cups of tea at me – though I’ve dodged most of them. He’s a bit of a hot-head but he needs to be with kids like us.
“Team Birtley is taking over Team GB. We’re the best club in Britain. The gym’s absolutely flying and everyone is just bringing everyone on. It just spurs you on to keep wanting to go away and win together.”