Baker was born in Groesbeck, Texas, the son of Edna (née McDonald) and Doyle Charles Baker. He attended the University of North Texas, and graduated with a Business degree in 1958. Baker served a tour of duty in the United States Army. In 1964 he appeared on stage in Marathon '33 at the ANTA Theatre in New York City. His career had its roots in television, though he did appear in several movies, including a part in the 1967 film Cool Hand Luke. He appeared in many television series, graduating to featured guest roles in such series as The Big Valley (in which he played a Harvard-educated Native American with a penchant for fighting) and Mod Squad (where he appeared as an illiterate vending machine robber). He was the title character in the 1971 TV movie Mongo's Back in Town, starring Telly Savalas.

Physically impressive at nearly 6'3" tall (189 cm), he was a natural for Westerns, both on the big and little screens. While working regularly on television on top shows like Bonanza and Gunsmoke, he appeared in supporting roles in such films Guns of the Magnificent Seven (1969) and Blake Edwards' Wild Rovers, but his film career didn't start to quicken until he scored the role of Steve McQueen's younger brother in Sam Peckinpah's Junior Bonner, a film about a contemporary rodeo Cowboy, which was released in late 1972.

Later that year, his work in Charley Varrick helped solidify Baker's reputation. He was praised for a courageous and offbeat portrait of the sadistic hitman Molly. He also co-starred with Robert Duvall in the 1973 crime film The Outfit and starred in the 1974 adventure film Golden Needles.

Baker has given many performances in a career spanning four decades. In 1980, he became the first actor to receive $1 million to star in a television series, the short-lived Eischeid (known as "Chief Of Detectives" in the UK).

He was "The Whammer," a baseball player modeled after Babe Ruth, in the 1984 baseball drama The Natural that starred Robert Redford. In a scene, the Whammer takes three swings at pitches from the young Roy Hobbs to try to impress a mysterious woman they have met on a train.

In 1985, he portrayed the corrupt Chief Jerry Karlin in Fletch. In the UK, he played CIA agent Darius Jedburgh in the BBC Television drama serial Edge of Darkness. He was nominated for "Best Actor" by the British Academy Television Awards, losing to his co-star Bob Peck.

In 1987, Baker played the villainous arms dealer Brad Whitaker in the Bond film The Living Daylights, starring Timothy Dalton as 007. In 1995 and 1997, Baker returned to the series, this time playing a different character, CIA agent Jack Wade, in GoldenEye and Tomorrow Never Dies, with Pierce Brosnan as Bond. Baker is one of four actors to appear as both a Bond ally and a villain, the others being Charles Gray, who appeared as Henderson in You Only Live Twice and Ernst Stavro Blofeld in Diamonds Are Forever; Walter Gotell, who appeared as Morzeny, the SPECTRE Island trainer, in From Russia with Love and as General Gogol, the head of the KGB, in six films between 1977 and 1987; and Richard Kiel as Jaws, in The Spy Who Loved Me and Moonraker, assisting 007 in the latter film.

Martin Scorsese directed him as a private detective in 1991's Cape Fear, hired by a man (Nick Nolte) whose family is being threatened by a psychopathic ex-convict (Robert De Niro).

Baker received the Distinguished Alumni Award from the University of North Texas in 1994.

In 2009, Baker delivered another performance in The Cleaner on A&E, playing an alcoholic military veteran attempting to help a friend cope with the loss of his son. He hires william Banks (played by Benjamin Bratt) to help him start back down the road to sobriety.

Baker played "King" in 2012's Mud with Matthew McConaughey, Reese Witherspoon and Sam Shepard.