O'Connell was born on March 29, 1908 in Manhattan, New York. He made his legitimate stage debut in the middle 1930s, at which time he fell within the orbit of Orson Welles' Mercury Theatre. Welles cast O'Connell in the tiny role of a reporter in the closing scenes of Citizen Kane (1941), a film often referred to as O'Connell's film debut, though in fact he had already appeared in Freshman Year (1938) and had costarred in two Leon Errol short subjects as Leon's conniving brother-in-law.

After numerous small movie parts, O'Connell returned to Broadway, where he appeared as the erstwhile middle-aged swain of a spinsterish schoolteacher in Picnic - a role he would recreate in the 1956 film version, earning an Oscar nomination in the process. Later the jaded looking O'Connell was frequently cast as fortyish losers and alcoholics; in the latter capacity he appeared as James Stewart's boozy attorney mentor in Anatomy of a Murder (1959), and the result was another Oscar nomination.

In 1959, O'Connell also played the part of Chief Petty Officer Sam Tostin, engine room chief of the fictional World War II submarine USS Sea Tiger, opposite Cary Grant and Tony Curtis in Operation Petticoat. In 1961, O'Connell played the role of Grandpa Clarence Beebe in the children's film classic Misty, the screen adaptation of Marguerite Henry's story of Misty of Chincoteague. In 1962, he portrayed the father of Elvis Presley's character in the motion picture Follow That Dream, and in 1964 in the Presley-picture Kissin' Cousins. In that same year O'Connell also portrayed the idealist-turned-antagonist Clint Stark, in The 7 Faces of Dr. Lao, which has become a cult classic, and in which O'Connell's is the only character other than star Tony Randall to appear as one of the "7 faces."

O'Connell continued appearing in choice character parts on both television and films during the 1960s, but avoided a regular television series, holding out until he could be assured top billing. He appeared as Matt Dexter, an aging Irish drifter in the episode "Songs My Mother Told Me" (February 21, 1961) on ABC's Stagecoach West series. In the story line, Dexter witnesses a shooting and is sought as a material witness to a crime. Two Criminals, one of whom is played by Richard Devon, also seek Dexter's whereabouts to make sure that he never testifies in court. Young Davey Kane, played by Richard Eyer, sneaks food and clothing to Dexter, who kills a rattlesnake that had threatened Davey and his dog. Dexter also teaches Davy new songs on his harmonica; hence the title of the episode.

In 1962, O'Connell married Ann Hall Dunlop (née Ann Byrd Hall; 1917–2000) of Washington, D.C., widow of william Laird Dunlop III (1909–1960). Arthur O'Connell and Ann Hall Dunlop divorced in December 1972 in Los Angeles.

In 1964, O'Connell played Joseph Baylor in the episode "A Little Anger Is a Good Thing" on the ABC medical drama about psychiatry, Breaking Point, starring Paul Richards. In 1966, he guest-starred as a scientist who regretfully realized that he has created an all-powerful android in the Voyage to the Bottom of the Sea episode titled "The Mechanical Man." In the February 1967 episode "Never Look Back" of the TV series Lassie, he played Luther Jennings, an elderly ranger manning the survey tower at Strawberry Peak, who takes it hard when he finds he'll lose his job when the tower is slated for destruction.

O'Connell accepted the part of a man who discovers that his 99-year-old father has been frozen in an iceberg on the 1967 sitcom The Second Hundred Years, having assumed that he would be billed first per the producers' agreement. Instead, top billing went to newcomer Monte Markham in the dual role of O'Connell's father and his son. O'Connell accepted the demotion to second billing as well as could be expected, but he never again trusted the word of any Hollywood executive.

Ill health forced O'Connell to reduce his acting appearances in the middle 1970s, but the actor stayed busy as a commercial spokesman, a friendly pharmacist who was a spokesperson for Crest.

At the time of his death from Alzheimer's disease in California in May 1981, O'Connell was appearing by his own choice solely in these commercials. O'Connell is interred at Calvary Cemetery, Queens, New York.