While there, he was a member of the Lambda Chi Alpha fraternity. Richard S. Huffman was one of his instructors and directed him in a lead role in Aristophanes' Lysistrata as well as Molière's Tricks of Scapin. After graduating in 1970, Rebhorn moved to New York City, where he earned a Master of Fine Arts in acting from Columbia University's School of the Arts, and joined the metropolitan theater scene.

An early performance was in Butterflies are Free at the Peterborough Players in New Hampshire in 1974. Rebhorn played Peter Latham in Forty Carats at the GasLight Dinner Theatre in Salt Lake City in the 1970s. He was known both for portraying WASP stereotypes, lawyers, politicians, doctors, and military men, as well as portraying individuals with Criminal behavior.

Rebhorn played several roles on television, including an abusive stepfather, Bradley Raines, on the soap opera Guiding Light from 1983 to 1985, and an abusive father, Henry Lange, on sister soap As the World Turns from 1988 to 1991. An earlier daytime role was as John Brady in Texas from 1981 to 1982.

Also in 1991, he played the role of 'Ezra' in the television movie/pilot Plymouth. Another performance was a cameo in Madonna's video for her single "Bad Girl" along with Christopher Walken (1993). In 1994, he appeared in an episode of the Nickelodeon series The Adventures of Pete & Pete, titled "Farewell, My Little Viking" as Mr. McFlemp.

He has delivered equally notable performances in a variety of other roles, including that of a brutal serial killer on NBC's Law & Order (he would later return to the show in the recurring role of defense attorney Charles Garnett), Ellard Muscatine in Lorenzo's Oil (1992), Fred Waters in Blank Check (1994), Clyde Frost, the father of famed Bull Rider Lane Frost, in 8 Seconds (1994), Lt. Tyler in White Squall (1996), and a shipping magnate in The Talented Mr. Ripley (1999).

One of his best known performances came in the popular 1996 film Independence Day, where he played Secretary of Defense Albert Nimzicki. He acted in Scent of a Woman (1992), and also played an expert witness in My Cousin Vinny (1992). He appeared in Carlito's Way the following year. Rebhorn also played an FBI Agent in the 1994 film Guarding Tess.

In 1998, he played the District Attorney in the two part series finale of Seinfeld. He also appeared in supporting roles in Regarding Henry, The Game, The Talented Mr. Ripley and Meet the Parents. In 2004, he appeared in the television miniseries Reversible Errors. His role in the short lived and controversial NBC drama The Book of Daniel cast him as the father of the title character. More recently, he appeared in the Showtime series Homeland as Carrie Mathison's father.

In 2004 he appeared on Broadway as Juror 4 in the Roundabout Theatre Company production of "Twelve Angry Men" at the American Airlines Theatre.

Rebhorn also appeared as a judge in Baby Mama. In the movie The Box, Rebhorn portrayed a NASA scientist. He had recurring roles on the series White Collar as Special Agent Reese Hughes, and also as Frank Mathison, the father of the protagonist Carrie Mathison, on Homeland. Rebhorn co starred in the Comedy Central sitcom Big Lake in 2010. He played Max Kenton's uncle in the movie Real Steel in 2011. He starred as Oren in the miniseries Coma. Rebhorn starred as Gary Pandamiglio in the 2012 Mike Birbiglia comedy Sleepwalk with Me.

In 2012, he also played the role of Archbishop McGovern in "Leap of Faith" the 13th episode of the 2nd season of the CBS police procedural drama Blue Bloods. He co starred in the romantic comedy The Perfect Wedding in 2013. His stage career included seven Broadway productions, as well as numerous appearances with New York City's Roundabout Theatre Company.

Rebhorn lived in South Orange, New Jersey. On March 21, 2014, he died at his home as a result of melanoma at the age of 65 (he had been receiving hospice care at his home for the melanoma, which was diagnosed in 1992). The season 4 finale of Homeland was dedicated to him. Rebhorn penned his own obituary.