The Body Snatcher believes Mark Lyson was to blame for his defeat against Fury at Wembley Stadium.
Whyte started the fight in a southpaw position - something he had never done before - but then went back to his normal orthodox stance for the other five rounds.
Fury didn't get out of second gear throughout although the referee had his work cut out after both fighters clashed with illegal forearms and headbutts.
But things unravelled for Whyte in the sixth as he got knocked down with a solid right hand uppercut by Fury. The Brixton heavyweight got up but was unable to compose himself and the official waved the bout off. Replays showed Whyte's tooth got knocked out by the punch.
Whyte spoke for the first time since his defeat and questioned the performance of Lyson.
"When the uppercut landed, I was buzzed but obviously I was trying to regather my senses. He proper full on pushed me and I fell over and hit my head on the canvas which is illegal," Whyte told Sky Sports.
"This isn't wrestling, this is boxing but they let Fury do whatever he wants. I should've been allowed extra time to recover and carry on fighting. And then he pushed me and told the referee 'don't let the fight carry on'. It was like the referee wasn't doing his job.
"I got caught no doubt about that by a good shot, I went for a left uppercut and he went for the right uppercut he's a bit taller than me so obviously he landed his at range and I was hurt, I didn't go straight down, I was trying to get my senses together and then he fully full on two-handed pushed me and I fell over and hit my head. It was a terrible job by the referee but it is what it is.
"He said he'll retire but I hope not because I want another go," added Whyte when speaking about whether Fury will retire or not.
Whyte may want redemption but he will now have to rebuild himself and get back into title contention.
A tune-up fight could be next for Whyte before entering another blockbuster bout towards the end of the year or the start of 2023.
"I'm not one of those guys who wants to go out on a loss," said Whyte. "I'm still young enough, got a lot left in me and feel strong and am getting better. I wasn't outclassed or outboxed, he was just a bit taller than me and the range was a bit tricky.
"It's not a long hard road back, I showed the level I am and we sold 94,000 tickets together it wasn't just him. And I had value going in before, I've had loads of PPV fights before and I'm still good enough. One fight and I'm back."