Haley was born in Boston, Massachusetts, to Canadian-born parents John Joseph Haley Sr. and Ellen Curley Haley. His father was a Sailor by trade and died in a ship wreck off the coast of Nova Scotia on February 1, 1898, when Jack was only six months old. He had one older brother, Bill, who died of pneumonia in 1915 at the age of 20 after contracting tuberculosis.

Haley was raised Roman Catholic. He was a member of the Good Shepherd Parish and the Catholic Motion Picture Guild in Beverly Hills, California. He married Florence McFadden (1902–1996), a native of Wilkes Barre, Pennsylvania on February 25, 1921; they remained married until his death. Flo Haley opened a successful beauty shop and had many film personalities among her clients. The couple had a son, Jack Haley Jr. (1933–2001), who became a successful film Producer, and a daughter, Gloria (1923–2010). In 1974, the younger Haley married entertainer Liza Minnelli, the daughter of his father's Oz co-star Judy Garland. The marriage ended in divorce in 1979. Jack Haley Jr. died on April 21, 2001. Gloria Haley-Parnassus died on May 1, 2010. His nephew Bob Dornan served as a Republican congressman from California.

Haley headlined in vaudeville as a song-and-dance Comedian. One of his closest friends was Fred Allen, who would frequently mention "Mr. Jacob Haley of Newton Highlands, Massachusetts" on the air. In the early 1930s, Haley starred in comedy shorts for Vitaphone in Brooklyn, New York. His wide-eyed, good-natured expression gained him supporting roles in musical feature films, including Poor Little Rich Girl with Shirley Temple, Higher and Higher with Frank Sinatra and the Irving Berlin musical Alexander's Ragtime Band. Both Poor Little Rich Girl and Alexander's Ragtime Band were released by Twentieth Century-Fox. Haley was under contract to them and appeared in the Fox films Rebecca of Sunnybrook Farm and Pigskin Parade, marking his first appearance with Judy Garland.

Haley did not remember the makeup or the costume fondly. Interviewed about the film years later by Tom Snyder, he related that many fans assumed making the film was a fun experience. Haley said, "Like hell it was. It was work!" For his role as the Tin Woodman, Haley spoke in the same soft tone he used when reading bedtime stories to his children. Oz was one of only two films Haley made for MGM. The other was Pick a Star, a 1937 Hal Roach production distributed by Metro-Goldwyn-Mayer.

Haley returned to musical comedies in the 1940s. Most of his '40s work was for RKO Radio Pictures. He left the studio in 1947 when he refused to appear in a remake of RKO's Seven Keys to Baldpate. Phillip Terry took the role. He subsequently went into real estate, taking guest roles in television series over the next couple of decades.

On June 6, 1979, Haley died of a heart attack at the UCLA Medical Center in Los Angeles, California at the age of 81. He is buried in Holy Cross Cemetery, Culver City, California.

Haley's autobiography, Heart of the Tin Man, was published in 2000.