Jaffe was born to Heida (Ada) and Barnett Jaffe, a Russian Jewish family in New York City, New York. His mother, Ada Jaffe, was a Yiddish Actress in Odessa, Ukraine, prior to moving to the United States; his father was a jeweler. He was the youngest of four children; his siblings were Abraham, Sophie, and Annie. As a child, he appeared in Yiddish theater productions with his mother, who after moving to the United States became a prominent Actress and vaudeville star. He graduated from Townsend Harris High School and studied engineering at City College of New York, graduating in 1912. He later attended Columbia University for graduate studies. He also worked for several years as a Teacher, and then dean, of mathematics at the Bronx Cultural Institute, a college preparatory school, before returning to acting in 1915.

As a young man, he lived in Greenwich Village in the same apartment building as a young John Huston. The two men became good friends and remained so for life. Jaffe was later to star in two of Huston's films: The Asphalt Jungle and The Barbarian and the Geisha. Jaffe's closest friends included Zero Mostel, Edward G. Robinson, Ray Bradbury, and Igor Stravinsky. He began to work in film in 1934, rising to prominence with his very first role as the mad Tsar Peter III in The Scarlet Empress. In 1938, Jaffe was forty-seven years old when he played the title role of water "boy" Gunga Din.

Jaffe was blacklisted by the Hollywood movie studio bosses during the 1950s, supposedly for being a communist sympathizer. Despite this, he was hired first by Robert Wise for The Day the Earth Stood Still and then by Director william Wyler for his role in the 1959 Academy Award-winning version of Ben-Hur.

Jaffe co-starred in the ABC television series, Ben Casey as Dr. David Zorba from 1961 to 1965 alongside Vince Edwards. He also made many guest-starring roles on other series, including Batman as Mr. Zoltan Zorba, and the Western Alias Smith and Jones. In 1975, he co-starred as a retired Doctor who is murdered by Janet Leigh in the Columbo episode "Forgotten Lady". He also appeared with an all-star cast in the TV pilot film of Rod Serling's Night Gallery and as Emperor Norton in one episode of Bonanza.

Jaffe was married to American operatic Soprano and musical comedy star Lillian Taiz from 1926 until her death from cancer in 1941. In 1956, he married Actress Bettye Ackerman, with whom he later co-starred in Ben Casey. She died on November 20, 2006. He had no children from either marriage.

Sam Jaffe died of cancer in Beverly Hills, California 14 days after his 93rd birthday. His interment was in Eden Memorial Park Cemetery.