Ujah tested positive for the prohibited substances ostarine and S-23 after the race on August 6, and the decision has now means Team GB's Tokyo medal total drops to 64, one less than the tally the team achieved at London 2012.
A statement issued by the Anti-Doping Division of the Court of Arbitration for Sport (CAS ADD) on Friday said Ujah did not challenge the anti-doping rule violation, but argued he had "not knowingly or intentionally doped, suggesting that the source of the prohibited substances could have been the ingestion of a contaminated supplement".
The statement added: "The CAS ADD sole arbitrator issued the present decision in which she determined, to her comfortable satisfaction, that an anti-doping rule violation had been committed."
The CAS ADD statement said that World Athletics is requested to consider any further action within its jurisdiction, including a decision on whether to impose a ban on Ujah.
The 27-year-old athlete said in a statement: "I accept the decision issued by the Court of Arbitration for Sport today with sadness.
"I would like to make it clear that I unknowingly consumed a contaminated supplement and this was the reason why an anti-doping rule violation occurred at the Tokyo Olympic Games.
"I sincerely regret that this has inadvertently led to the forfeiture of the men's 4x100m relay team's Olympic silver medals at the Tokyo Olympic Games.
"I would like to apologise to my team-mates, their families and support teams for the impact which this has had on them.
"I'm sorry that this situation has cost my team-mates the medals they worked so hard and so long for, and which they richly deserved. That is something I will regret for the rest of my life.
"I would also like to apologise to both British Athletics and Team GB. British Athletics has supported the relay athletes for years and this has been difficult for everyone involved in the programme.
"Representing my country at a second Olympic Games surpassed my childhood sporting ambitions and I will forever be devastated that this situation has marred the success achieved by the men's 4x100m relay team in Tokyo.
"Now that the IOC proceedings have concluded before CAS, my focus is on the forthcoming proceedings before World Athletics and I will therefore not be making any further comment until those separate proceedings have concluded."
The relay team also featured Zharnel Hughes, Nethaneel Mitchell-Blake and Richard Kilty.
UK Athletics said it was in communication with the athletes concerned but would not comment until the conclusion of the process set to follow involving World Athletics and the Athletics Integrity Unit.