Terry Downes was born in Paddington, London. His father Richard worked as a mechanic, and his mother Hilda in a department store. Downes boxed as a junior for the Fisher ABC. He moved with his parents to the United States in 1952, while still a teenager, to live with his trapeze Artist sister Sylvia, who had lost an arm in a traffic accident, going on to serve in the US Marine Corps from 1954-56, being recruited after boxing against them for the YMCA. In the marines he won several amateur trophies, including the all-services championship and the Amateur Golden Gloves. He missed out on selection for the US Olympic team, being ruled ineligible on residence grounds, and after his term of Service, he returned to London and turned professional.

Downes and his wife Barbara (nee Clarke) were married from 1958 until his death in 2017. They had four children and eight grandchildren.

Downes lost his first World Title shot to Paul Pender in Boston in January 1961. The following July, however, Downes fought Pender again, this time in London, and defeated the American in front of a raucous Wembley crowd, with Pender retiring at the end of the ninth round with cuts over both eyes. Downes was recognised as world champion by The Ring magazine in August 1961, and was named Sports Writers' Association Sportsman of the Year later that year. Pender won the title back the following year, defeating Downes in Boston once more, this time on points.

Downes responded to the loss of his title by winning his next seven bouts, including a win over Sugar Ray Robinson in September 1962. Robinson was, however, 41 at the time, and when asked after the fight how it felt to beat a boxer of such esteem, Downes famously replied, "I didn't beat Sugar Ray, I beat his ghost." Downes moved up to light heavyweight in 1963, winning his first three fights at the weight before facing Willie Pastrano for the world title in Manchester on 30 November 1964. Downes was knocked down twice in the 11th round, while reportedly well ahead on points, and Pastrano retained his title when Referee Andrew Smyth controversially waved it off – it was to be Downes' last fight.

Post-boxing, Downes acted occasionally between 1965 and 1990, usually appearing a thug, villain or bodyguard. One of his more prominent roles was in Roman Polanski's 1967 film The Fearless Vampire Killers, in which he played "Koukol", a hunchbacked servant. His other film credits included appearances in A Study in Terror (1965), Five Ashore in Singapore (1967), The Golden Lady (1979), If You Go Down in the Woods Today (1981), and the Derek Jarman film Caravaggio (1986).

During his boxing career, Downes had set up a chain of betting shops with Burns. By the early 1970s they had 90 shops, and these were taken over by william Hill, with Burns acting as managing Director into the 1980s. Downes also owned a car dealership and a nightclub, and worked as a boxing manager, working with British title challenger Colin Lake in the late 1960s.

Downes was awarded the British Empire Medal in 2012, in recognition of his sporting achievements and charity work.