At age four, Mankiewicz moved with his family to New York City, graduating in 1924 from Stuyvesant High School. In 1928, he obtained a bachelor's degree from Columbia University. At 19, he was sent by his college professor father to Berlin where he was to study German drama at the University of Berlin. Instead, Mankiewicz got work at the UFA film studio translating film intertitles from German to English.

During his long career in Hollywood, Mankiewicz wrote forty-eight screenplays. He also produced more than twenty films including The Philadelphia Story which was nominated for the Academy Award for Best Picture in 1941. However, he is best known for the films he directed, twice winning the Academy Award for Best Director. In 1944, he produced The Keys of the Kingdom, which starred Gregory Peck, and featured Mankiewicz's then-wife, Rose Stradner, in a supporting role as a nun.

Comfortable in a variety of genres and able to elicit career performances from actors and actresses alike, Joseph L. Mankiewicz combined ironic, sophisticated scripts with a precise, sometimes stylized mise en scène. Mankiewicz worked for seventeen years as a Screenwriter for Paramount Pictures and as a Producer for Metro-Goldwyn-Mayer before getting a chance to direct at Twentieth Century-Fox. Over six years he made 11 films for Fox, reaching a peak in 1950 and 1951 when he won consecutive Academy Awards for Screenplay and Direction for both A Letter to Three Wives and All About Eve, which was nominated for 14 Academy Awards and won six.

In 1951 Mankiewicz left Fox and moved to New York, intending to write for the Broadway stage. Although this dream never materialized, he continued to make films (both for his own production company Figaro and as a director-for-hire) that explored his favorite themes – the clash of aristocrat with commoner, life as performance and the clash between people's urge to control their fate and the contingencies of real life.

In 1953 he directed Julius Caesar for MGM, an adaptation of Shakespeare's play. It received widely favorable reviews, and David Shipman, in The Story of Cinema, described it as a "film of quiet excellence, faltering only in the later moments when budget restrictions hampered the handling of the battle sequences". The film serves as the only record of Marlon Brando in a Shakespearean role; he played Mark Antony, and received an Oscar nomination for his performance. Mankiewicz was to direct Brando again two years later in the film version of the musical Guys and Dolls.

In 1958 Mankiewicz directed The Quiet American, an adaptation of Graham Greene's 1955 novel about the seed of American military involvement in what would become the Vietnam War. Mankiewicz, under career pressure from the climate of anti-Communism and the Hollywood blacklist, distorted the message of Greene's book, changing major parts of the story to appeal to a nationalistic audience. A cautionary tale about America's blind support for "anti-Communists" was turned into, according to Greene, a "propaganda film for America".

He was the younger brother of Herman J. Mankiewicz. His sons are Eric Reynal (from his first marriage), the late writer/director Tom Mankiewicz, and Producer Christopher Mankiewicz. He also has a daughter, Alex Mankiewicz. His great-nephew is radio and television personality Ben Mankiewicz, who currently can be seen on TCM. He also was the uncle of Frank Mankiewicz, a well-known political campaign manager who officially announced the death of the assassinated presidential candidate, Robert F. Kennedy, in 1968. He was not related to the similar-sounding British Screenwriter, Wolf Mankowitz.

Cleopatra consumed two years of Mankiewicz's life and ended up both derailing his career and causing severe financial losses for the studio, Twentieth Century-Fox, which were not fully recovered until Rodgers and Hammerstein's popular and acclaimed The Sound of Music was released two years later. Mankiewicz made more films, however, garnering an Oscar nomination for Best Direction in 1972 for Sleuth, his final directing effort, starring Laurence Olivier and Michael Caine. In 1983, he was a member of the jury at the 33rd Berlin International Film Festival.

Mankiewicz died of a heart attack on February 5, 1993, six days before his 84th birthday. He was interred in Saint Matthew's Episcopal Churchyard cemetery in Bedford, New York.