Begley began his career as a Broadway and radio actor while in his teens. He appeared in the hit musical Going Up on Broadway in 1917 and in London the next year. He later acted in roles as Sgt. O'Hara in the radio show The Fat Man. His radio work included Stroke of Fate and a period as Charlie Chan among other roles. He also starred in the 1950s radio program Richard Diamond, Private Detective, playing Lieutenant Walter Levinson, head of homicide at the 5th Precinct, Manhattan. He was elected a member of The Lambs in 1943. In the late 1940s, he began appearing regularly in supporting film roles.

In the 1952–1953 television season, Begley co-starred with Eddie Albert in the CBS sitcom Leave It to Larry. Begley, though only five years older than Albert, played the father-in-law and employer of Albert's character, Larry Tucker, a shoe salesman, who with his young family lives with Begley. In 1954 Begley starred in the NBC Television show, Robert Montgomery Presents, in an episode sponsored by Lucky Strike called Big Boy as Joe Grant, an Engineer for the Union Pacific Railroad living in Cheyenne, Wyoming, who worked on the famous Union Pacific Big Boy steam locomotives. The show is about how Begley's character copes with the transition from steam locomotives to diesel locomotives in the 1950s.

In 1956, he appeared in the Broadway production of Inherit the Wind, in the role of Matthew Harrison Brady. For this performance, he won the Tony Award for Best Featured Actor in a Play.In 1968 he appeared with Clint Eastwood in the classic western Hang 'Em High.

He won the Academy Award for Best Supporting Actor for his role in Sweet Bird of Youth (1962). Some of his other notable films include Deadline – U.S.A. (1952), 12 Angry Men (1957) as juror #10, The Unsinkable Molly Brown (1964), and Wild in the Streets (1968). One notable role Begley played both on television (twice in 1955) and in the theatrical film (1956) is william (Bill) Briggs, one of the three primary characters in Rod Serling's Patterns.