The 32-year-old undisputed lightweight world champion triumphed by a contentious split decision when the pair clashed at the SSE Hydro in Glasgow last February.
The Scot was due to take on the 29-year-old Englishman again in Glasgow on March 4, but the bout has been postponed.
Taylor told BBC Scotland: "People saying I'm scared and this and that. It's absolute nonsense.
"If I was scared, I wouldn't have been undisputed world champion. I'm scared to fight no-one.
"I've made a lot of sacrifices to make this fight happen. I didn't have to do this fight. I could have gone on to much bigger fights and got offered much bigger fights, a lot more lucrative opportunities.
"I turned them down to do this to prove that it was just a bad performance and I am still the best fighter on the planet in my weight class."
Taylor tore the plantar fascia tendon in his heel while sparring which will means he can do nothing for "at least five to six weeks" and it could be late May or early June before he returns to the ring.
"I don't think the fight is going to go anywhere," he said. "I wouldn't expect Jack to sit around and wait for me to get better.
"He's got a career to pursue as well. I would expect him to go and try and get another fight for the time being.
"I think the interest in that rematch will always be there as long as we're both still winning and he doesn't get beaten by anyone.
"I think, if he boxes anyone in the top 10, the guys I've boxed previously, he gets beaten off them.
"Even then, I still think that rematch is still there at some point because there will be interest in it."