Armstrong's first film appearance was in the 1954 film Garden of Eden; however, it was television where he first earned a name for himself. He guest-starred in virtually every television western series produced in the 1950s and 1960s, including Have Gun - Will Travel, The Californians, Jefferson Drum, The Tall Man, Riverboat, The Rifleman, Zane Grey Theater, Wanted: Dead or Alive, The Westerner, The Big Valley, Bonanza, Maverick (as Louise Fletcher's character's father in the episode which drew the series' largest single viewership, "The Saga of Waco Williams"), Gunsmoke, Rawhide, and Wagon Train.

Armstrong appeared on The Twilight Zone, in the episode "Nothing in the Dark" along with Robert Redford. He appeared in three episodes of Perry Mason, twice in the role of the defendant. First in 1959 he played title character Harry Bright in "The Case of the Petulant Partner," then in 1962 he played John Gregory in "The Case of the Stand-in Sister." Armstrong also appeared on Alfred Hitchcock Presents, The Everglades, The Andy Griffith Show, The Fugitive, Daniel Boone, T.H.E. Cat, Hawaii Five-O, Starsky and Hutch, The Dukes of Hazzard, Dynasty, and in the miniseries War and Remembrance. Armstrong had a recurring role in the second season of Millennium as a reclusive visionary known only as the Old Man. In the late 1980s, he played the demonic "Uncle Lewis Vendredi" in the Canadian horror series Friday the 13th: The Series.

While working on The Westerner in 1960, Armstrong met the up-and-coming writer/director Sam Peckinpah. The two men immediately struck up a friendship. Peckinpah recognized Armstrong's inner turmoil regarding the religious beliefs of his family and utilized that to brilliant effect in his films. Armstrong would almost always play a slightly unhinged fundamentalist Christian in Peckinpah's films, usually wielding a Bible in one hand and a shotgun in the other. This character archetype appeared in Ride the High Country (1962), Major Dundee (1965), and perhaps most memorably in Pat Garrett and Billy the Kid (1973). However, Armstrong also appeared in The Ballad of Cable Hogue (1970), playing a more likeable character.

Early in 1961, Armstrong was cast in the title guest-starring role of Nathanael Grimm in "The Return of Mr. Grimm" of the ABC/WB western series Cheyenne, starring Clint Walker. In the storyline, the wealthy Grimm seeks the hanging of Sheriff Cheyenne Bodie for the justifiable homicide of Grimm's wayward son, who was fleeing from a posse. Grimm closes the businesses he controls in town, and the threatened townsmen demand that Bodie stand trial though no crime has been committed. Anita Sands appears as Grimm's secret daughter-in-law, Grace Evans, who unknown to him is carrying his grandson.

Even outside of Peckinpah's work, Armstrong became a tier-one character actor in his own right, appearing in dozens of films over his career, playing both villains and sympathetic characters. Some of his more memorable roles outside of Peckinpah's films include a sympathetic rancher in El Dorado (1966), Cap'n Dan in The Great White Hope (1970), outlaw Clell Miller in The Great Northfield Minnesota Raid (1972), a bumbling outlaw in My Name is Nobody (1973), Race with the Devil (1975), The Car (1977), as well as Children of the Corn (1984), Red Headed Stranger (1986) with Willie Nelson, and as General Phillips in Predator (1987). He appeared in several of Warren Beatty's films, including Heaven Can Wait (1978), Reds (1981), and as the character Pruneface in Dick Tracy (1990).

Despite being typecast as gruff and violent characters throughout his career, Armstrong is well known for having had a warm and affable personality offscreen. He semi-retired from films and television in the late 1990s, but he continued to be active in off-Broadway theater in New York and Los Angeles, until finally retiring from acting in 2005 because of near-blindness due to cataracts.

in 1991, Armstrong gained critical praise for his portrayal of the title character in the music video for Enter Sandman from famous heavy metal band Metallica.

Amstrong was married three times: his first wife was Ann Neale, with whom he had four children; he was then married to Susan Guthrie until 1976; he was married to his third wife, Mary Craven, until her death in 2004. Armstrong died of natural causes at the age of 95 on July 27, 2012 at his home in Studio City, California. He is survived by his four children from the first marriage.