His early television credits include appearing twice as Deputy Virgil Bramley in the NBC western series The Road West in the 1966–1967 season, which co-starred Barry Sullivan, Andrew Prine, and Glenn Corbett.

Gammon may be best-remembered for his characters Lou Brown, manager of the Cleveland Indians in the Major League films, and Nick Bridges, the father of Don Johnson's title character in the television series Nash Bridges. He appeared in the films Cool Hand Luke (1967), Urban Cowboy (1980), Silverado (1985), Noon Wine (1985), The Milagro Beanfield War (1988), Major League (1989), The Adventures of Huck Finn (1993), Major League II (1994), Wyatt Earp (1994), Wild Bill (1995), Truman (1995), Cold Mountain (2003), and more recently Appaloosa (2008). He also had an uncredited role in Natural Born Killers (1994).

In the 1970s, he helped found the Met Theatre in Los Angeles. While performing there, a rep from The Public Theater saw him and had him cast as Weston in Sam Shepard's Curse of the Starving Class in 1978. The two became friends afterward. He made his sole Broadway appearance as "Dodge" in a revival of Sam Shepard's Buried Child. He was nominated for a Tony Award for his performance. He also appeared on stage in Shepard's San Francisco debut of The Late Henry Moss along with Nick Nolte, Sean Penn, Cheech Marin and Woody Harrelson in 2000.

His first marriage ended in divorce. He has a brother, Philip, and a sister, Sandra (Glaudell). He was married to Nancy Jane Kapusta from 1972 until his death. He has two daughters, Allison Mann and Amy Gammon.

Gammon provided the voices of the animated characters Marv Loach and Floyd Turbeaux in the 1999 Warner Bros. feature film The Iron Giant.

Gammon portrayed a Korean War veteran on the hit ABC series Grey's Anatomy. He played Charles Goodnight in Streets of Laredo. In 2006, he played the stern grandfather, Sam, brother of notorious outlaw Butch Cassidy, in the film Outlaw Trail: The Treasure of Butch Cassidy. Gammon also plays a supporting role in Appaloosa (2008).

His friend, Sam Shepard, said this of Gammon: "This was a guy who could act circles around most other actors, and he never pretended to be other than a working kind of actor." He would go on to say about the star-studded cast of The Late Henry Moss, "I mean, a bunch of notoriously famous guys, and every single one of them would come up to me, alone, and say, ‘Who’s that Jim Gammon guy? Where did he come from?’" Shepard also would talk about why he cast Gammon, saying: "You’re probably aware of the notorious father figures in my plays, alcoholic Midwesterners who leave their families and get lost in the Southwestern desert. Jimmy had that familiarity about him with the way I grew up, the guys with the voice and the face and the whiskey. He definitely rang a bell with me."