Robinson won stages of the Tour in both 1958 and 1959, and was also the first British rider to complete cycling's most famous race in 1955.
A pioneer for Britons racing on the continent, he also won the prestigious Criterium du Dauphine stage race in 1961 and was the first Briton to stand on the podium of one of cycling's Monuments, Milan-Sanremo, finishing third in 1957.
The Yorkshireman's death was announced by his grandson and fellow cyclist Jake Womersley, who wrote on Twitter: "It's with great sadness the family of Brian Robinson have to announce his passing yesterday."
Born in Mirfield, West Yorkshire, Robinson joined his local cycling club as a teenager and later took up racing in between working for the family building company.
He competed for Britain at the 1952 Olympics before turning professional, riding the Tour de France for the first time in 1955 when he finished 29th overall.
In 1958 he took Britain's first stage win, on stage seven from Saint-Brieuc to Brest, though he did not get to raise his arms in victory that day, only learning he had won later in the evening when news came through that Italian Arigo Padovan, first across the line, had been disqualified.
He made up for that a year later when he won stage 20 of the 1959 Tour, from Annecy to Chalon-sur-Saone, by a full 20 minutes.
At a time when many professional cyclists lived hand to mouth, Robinson at one point sold his car and lived in a caravan on the Cote d'Azur to pursue his dreams.
"It didn't seem so bad because I wanted to do it," he later said.
It was, however, something he did not want to do once he had a family to care for, and he retired at the age of 33 to return to the UK, still riding his bike in relative anonymity until he became an ambassador for the Grand Depart of the Tour de France in Yorkshire in 2014.
On Twitter, British Cycling said: "Pioneer. Trailblazer. Legend. Today we join the cycling community in sending our heartfelt condolences to the family of Brian Robinson - the first British rider to complete and win a stage at @LeTour, and member of the British Cycling Hall of Fame."
The Tour de France added: "A pioneer and trailblazer of British cycling passed away today.
"First British rider to win a stage on the Tour de France, in 1958, and ambassador of the Grand Départ in Yorkshire back in 2014, Brian Robinson was 91. We send our condolences to his family."
The Ineos Grenadiers said: "The man who blazed a trail for British cyclists at the Tour de France.
"We'd like to join the cycling world in sending condolences and love to the family and friends of Brian Robinson, who passed away yesterday, aged 91.
"A true legend of our sport."