McMartin was born in Warsaw, Indiana on August 21, 1929 and raised in Minnesota. After graduating from high school McMartin joined the United States Army and became a paratrooper in the 101st Airborne Division He attended Columbia College Chicago, but did not graduate and he later attended college in New York. He made his off-Broadway debut in Little Mary Sunshine in 1959, opposite Eileen Brennan and Elmarie Wendel. He won a Theatre World Award for his role as Corporal Billy Jester, and married one of the show's producers, Cynthia Baer, in 1960; they divorced in 1971.

His first Broadway appearance was as Forrest Noble in The Conquering Hero in 1961, which was followed by Blood, Sweat and Stanley Poole. He created the role of Oscar in Sweet Charity in 1966, opposite Gwen Verdon, garnering a Tony nomination, and played the role again in the 1969 film opposite Shirley MacLaine. He was reportedly cast in Stephen Sondheim's A Funny Thing Happened on the Way to the Forum in 1962, but his role was cut before the show opened.

McMartin was a leading member of the New Phoenix Repertory Company during their three Broadway seasons in the early 1970s, appearing onstage in, among other productions, Eugene O'Neill's The Great God Brown (opposite Katherine Helmond), Molière's Dom Juan, and Luigi Pirandello's The Rules of the Game.

He later starred in the original Broadway production of Sondheim's Follies opposite Alexis Smith in 1971 as Benjamin Stone, introducing the ballad "The Road You Didn't Take". His association with Sondheim has continued, appearing in A Little Night Music as Frederick at the Ahmanson Theatre, Los Angeles, in 1991. The reviewer for the Orange County Register (California) wrote that the actor was "aggressively deadpan as her rediscovered old flame ..." He appeared in the Broadway revival of Into the Woods in 2002, in the dual role of the Narrator/Mysterious Man.

On film his roles included the foreign Editor in All the President's Men (1976), a Senator in Brubaker (1980), a political advisor in Blow Out (1981) and a significant part in Legal Eagles (1986) as millionaire Mr. Forrester.

Other Broadway roles include the Narrator in Happy New Year, Ben in A Little Family Business (adapted by Jay Presson Allen, 1982), Donner in Tom Stoppard's Artist Descending a Staircase, Cap'n Andy in Kern and Hammerstein's Show Boat (1994), and Uncle Willie in Cole Porter's High Society (1998). He also had a role, as the American Revolutionary naval hero John Paul Jones, in the unsuccessful Loesser/Spewack musical, Pleasures and Palaces, which closed in Detroit. In regional theater, he originated the role of Benteen in the Folger Theater Group's 1979 production of Custer at the Kennedy Center.

On television, he appeared on the soap opera As the World Turns as John Rice, whose mother's death had been blamed on Dr. Doug Cassen. He later was on the CBS drama Eastside/Westside and in the first two seasons of Beauty and the Beast (1987) as Charles Chandler, father of Catherine (Linda Hamilton). He also appeared in The Golden Girls (Season 2) as Frank Leahy who, unbeknownst to Dorothy (Bea Arthur) who is romantically attracted to him, is a priest. He appeared on "The Bob Newhart Show" (Season 2) as the Rev. Dr. Dan Bradford in "Somebody Down Here Likes Me." He appeared on Cheers (Season 7) in "The Visiting Lecher". He appeared as radio personality Fletcher Grey on Frasier (Season 1). He also appeared in four episodes of Murder, She Wrote. He also appeared as Shirley Jones' love interest in The Partridge Family episode titled "When Mother Gets Married".

He played "Anton Schell" opposite Chita Rivera in Kander and Ebb's musical The Visit (based on the play by Friedrich Dürrenmatt) at the Goodman Theatre. He created the roles of "J.V. 'Major' Bouvier" and Norman Vincent Peale in Grey Gardens, opposite Mary Louise Wilson and Christine Ebersole. He played Thomas Jefferson in the original cast of John Guare's A Free Man of Color at Lincoln Center (2010–2011), and next played "Elisha Whitney" in the 2011 Broadway revival of Anything Goes, opposite Jessica Walter.

McMartin died of cancer in Manhattan on July 6, 2016, aged 86. He is survived by his two daughters from his marriage, and his long-time partner, Actress Charlotte Moore, the artistic Director of the Irish Repertory Theatre.