Diffring was born as Alfred Pollack in Koblenz. His father Solomon Pollack was a Jewish shop-owner who managed to avoid internment by the Nazi authorities and survived the war. His mother Bertha Diffring was Christian. He studied acting in Berlin and Vienna but there is conjecture about when he left Germany prior to World War II. The audio commentary for the Doctor Who series Silver Nemesis mentions that he left Germany in 1936, to escape persecution due to his homosexuality. Other accounts point to him leaving Germany in 1939 and heading for Canada where he was interned in 1940, which is unlikely as he appears in the 1940 Ealing Studios film Convoy released in July as the officer of the U-37, although uncredited. His sister Jacqueline Diffring moved to Britain and became a famous sculptress. Although he made two fleeting uncredited appearances in films in 1940, it was not until 1950 that his acting career began to take off.

With numerous British war films being produced in the 1950s, Diffring's blond hair, blue eyes and his chiselled features saw him often cast as villainous German officers, in Albert R.N. (1953) and The Colditz Story (1955). Some of his more notable roles as German characters were in The Heroes of Telemark (1965), The Blue Max (1966), Where Eagles Dare (1968), Operation Daybreak (1975) (as SS officer Reinhard Heydrich) and the match commentator in Escape to Victory (1981), though he also played a Polish parachutist in The Red Beret (1953). In 1983 he played Hitler's foreign minister Joachim von Ribbentrop in the American mini-series The Winds of War. In the Italian war movie Uccidete Rommel, shot in an Egyptian desert in 1969, he played the role of a British officer of the SAS.

Diffring starred in a number of horror films, such as The Man Who Could Cheat Death (1959) and Circus of Horrors (1960) and played the lead in the 1958 television pilot Tales of Frankenstein. He also appeared in quite a number of international films, such as Fahrenheit 451 (1966) directed by François Truffaut.

His final performance was again as a Nazi for the BBC in the 1988 Doctor Who serial Silver Nemesis in which he agreed to appear because the recording coincided with the Wimbledon Championships, which he wanted to watch. He died in his home at Châteauneuf-Grasse in the South of France in 1989.