Gleason was born in New York City, the son of Mina (née Crolius; 1858-1931) and william L. Gleason (1850-1909). Coming from theatrical stock, as a schoolboy he made stage appearances while on holiday. He began earning his living at the age of thirteen, being a messenger boy, printer's devil, assistant in an electrical store and a lift boy. He enlisted in the United States Army at age 16 and served three years in the Philippines.

His film debut was in Polly of the Follies (1922), starring Constance Talmadge. Balding and slender with a craggy voice and a master of the double-take, Gleason portrayed tough but warm-hearted characters, usually with a New York background. He co-wrote The Broadway Melody, the second film to win the Academy Award for Best Picture, and had a small uncredited role in it. He also co-wrote and briefly appeared as a hot dog vendor in the 1934 Janet Gaynor vehicle Change of Heart. He performed in a number of films with his wife Lucile. In The Clock (1945), he played a milk cart driver who gives lessons in marriage to the characters played by Judy Garland and Robert Walker, while Lucile played his wife. The same year, he played the bartender in the film adaptation of A Tree Grows in Brooklyn. In the Frank Capra classic Meet John Doe, he played the cynical, "hard boiled" Editor brought in to pump up the newspaper that runs with the "John Doe" story. Gleason starred in two movie series, playing police inspector Oscar Piper in six Hildegarde Withers mystery films during the 1930s, starting with The Penguin Pool Murder, and Joe Higgins in the first seven of nine films about the Higgins Family, in which his wife Lucile and son Russell played Lil and Sydney Higgins. Gleason was nominated for an Academy Award for Best Supporting Actor for his performance as boxing manager Max "Pop" Corkle in the 1941 film Here Comes Mr. Jordan.

Gleason also performed in other media. In 1931, he co-starred with Robert Armstrong in the radio sitcom Gleason and Armstrong. His television credits include several episodes of Alfred Hitchcock Presents, the Reed Hadley legal drama The Public Defender and ABC's The Real McCoys. In "The Child", the Christmas 1957 episode of John Payne's The Restless Gun on NBC, Gleason and Anthony Caruso played Roman Catholic Priests who run an orphanage. Dan Blocker, just launching his acting career, also guest starred in the episode.

James and Lucile Gleason had a son, actor Russell Gleason. On December 26, 1945, the younger Gleason was in New York City awaiting deployment to Europe with his regiment, when he fell out of a fourth story window in the Hotel Sutton, which the army had commandeered to house the troops, resulting in his death. Reports varied, some saying the fall was accidental, while others stating it was a suicide. Russell's most prominent role had been as Muller in the Academy Award-winning version of All Quiet on the Western Front (1930). Russell Gleason was married to Cynthia Lindsay, a former Busby Berkeley chorus girl who later wrote a biography of family friend Boris Karloff.