Klein was born in the Bronx, the son of Frieda (née Moskowitz) and Benjamin Klein, and was raised in a "prototypical 1950s Bronx Jewish" environment. After graduating from DeWitt Clinton High School, Klein had planned to study medicine; however, he changed his mind during his studies at Alfred University, After graduating, he studied at the Yale Drama School when he learned about an opportunity to audition for The Second City. In a piece he wrote for the improvisational troupe's book, Klein recalled sitting in a room full of other hopefuls, including Fred Willard. Klein's audition consisted of an improvisation set with Willard about two guys in a nightclub, which was successful enough to get Klein and Willard hired by Second City. In the spring of 1965, Klein was chosen as a member of Second City. When he returned to New York City a year later, he was cast by Mike Nichols in the Broadway musical The Apple Tree.

Klein has appeared in such movies as The Owl and the Pussycat, Hooper, Primary Colors, Radioland Murders, Ira and Abby, One Fine Day, Two Weeks Notice, How to Lose a Guy in 10 Days, The Last Unicorn, and The Safety of Objects. He had a recurring role in the TV drama series Sisters. In the 1970s, he hosted Saturday Night Live twice. He also appeared as a guest star in the animated series Duckman and on the CBS sitcom The King of Queens. In 2014 he played the Canadian Ambassador on Madam Secretary.

In A Child of the Fifties (1973, Brut/Buddah Records), shown as Child of the 50's (on the cover), Klein talks about his life as a child in the 1950s: about air raid drills, Johnny Mathis music, showing off condoms while at the high school dance, the high school lunch ladies, Senator Joseph McCarthy, Governor W. Averell Harriman (of New York), meeting Yankee stars, the Yankees losing the World Series, and much more. He also goes into other things that he has observed in his life, such as substitute teaching, 1970s FM radio disc jockeys, late-night delis, and annoying commercials (e.g., Geritol). He also performed two songs that he wrote himself: "Fabulous '50s" and "Middle Class, Educated Blues."

His next album, Mind Over Matter (1974), included extensive discussion of the Watergate scandal and another song—the title track—about a kid who turned to humor to become popular.

His follow-up album, New Teeth (1975, Epic/CBS Records), featured the comedian's on-stage work on tracks such as "Mother Isn't Always Right" and his transposition of George Carlin's "Seven Words You Can't Say On Television", titled "Six Clean Words You Can Say Anywhere," with studio-recorded material such as "Continental Steel" and "On the Bayou."

In 1986, Klein had his own late-night talk show, Robert Klein Time, which ran on the USA Network until 1988.

He responded to the end of the sexual revolution with his 1990 album, Let's Not Make Love, which contained many of the same routines as his 1984 HBO special Child of the '50s, Man of the '80s and his 1986 special Robert Klein on Broadway.

Klein hosted Monty Python Live at Aspen, a reunion and tribute show for the five surviving members of the British comedy troupe, in a special that appeared on HBO in 1998.

Klein is the author of The Amorous Busboy of Decatur Avenue: A Child of the Fifties Looks Back, an autobiography published in 2006.

In March 2007, he re-collaborated with Adam Sandler (after appearing in Mixed Nuts) in Reign Over Me, and in September 2007 released a new DVD compilation of his eight live HBO specials, titled Robert Klein: The HBO Specials 1975–2005.

During the finale of The Tonight Show with Jay Leno in February 2014, Billy Crystal reminded Jay Leno that when Leno was an aspiring Comedian, the only decoration in Leno's apartment was a poster of Klein. Both implied that they were influenced by the Child of the 50's recording.

On January 29, 2018, it was announced that Klein would be playing Grace Adler's father Martin in the NBC revival of Will and Grace replacing Alan Arkin.