Caan was born on March 26, 1940, in the Bronx, New York, the son of Sophie (née Falkenstein; June 24, 1915 – January 18, 2016) and Arthur Caan, Jewish immigrants from Germany. His father was a meat dealer and butcher. One of three siblings, Caan grew up in Sunnyside, Queens, New York City. He was educated in New York City, and later attended Michigan State University. He later transferred to Hofstra University in Hempstead, New York, but did not graduate. His classmates at Hofstra included Francis Ford Coppola and Lainie Kazan.

In 1964, he starred as Jewish athlete Jeff Brubaker in the episode "My Son, the All-American" of Channing, a drama about college life. His first substantial film role was as a punk hoodlum in the 1964 thriller Lady in a Cage, which starred Olivia de Havilland. In 1965, he landed his first starring role, in Howard Hawks' auto-racing drama Red Line 7000.

In 1966, Caan appeared as Alan Bourdillion Traherne, a.k.a. Mississippi, in Hawks' El Dorado, with John Wayne and Robert Mitchum. He had a starring role in Robert Altman's second feature film, Countdown, in 1968. In 1969, he had an uncredited role as "Rupert of Rathskeller" on the spy sitcom Get Smart. That same year he won praise for his role as a brain-damaged football player in The Rain People (1969) directed by Francis Ford Coppola.

From 1971-82, Caan appeared in many films, playing a wide variety of roles. His films included T.R. Baskin, Cinderella Liberty, Freebie and the Bean, The Godfather Part II, Rollerball, a musical turn in Funny Lady, Harry and Walter Go to New York, A Bridge Too Far, Comes A Horseman, and Neil Simon's autobiographical Chapter Two.

Caan has been married four times. In 1961, he married Dee Jay Mathis; they divorced in 1966. They had a daughter, Tara (born 1964). Caan's second marriage to Sheila Marie Ryan (a former girlfriend of Elvis Presley's) in 1976 was short-lived; they divorced the following year. Their son, Scott Caan, who also is an actor, was born August 23, 1976.

In 1977, Caan rated several of his movies out of ten – The Godfather (10), Freebie and the Bean (4), Cinderella Liberty (8), The Gambler (8), Funny Lady (9), Rollerball (8), The Killer Elite (5), Harry and Walter Go to New York (0), Slither (4), A Bridge Too Far (7), Another Man Another Chance (10) and Kiss Me Goodbye (0). He also liked his performances in The Rain People and Thief.

For his contributions to the film industry, Caan was inducted into the Hollywood Walk of Fame in 1978 with a motion pictures star located at 6648 Hollywood Boulevard.

In 1980, Caan directed Hide in Plain Sight, a film about a father searching for his children, who were lost in the Witness Protection Program. Despite critical praise, the film was not a hit with the public.

Caan's subsequent notable performances include roles in Thief (1981), Misery (1990), For the Boys (1991), Eraser (1996), Bottle Rocket (1996) and Elf (2003), as well as the role of "Big Ed" Deline in the television series Las Vegas (2003–08). He also prominently lent his voice to Cloudy with a Chance of Meatballs (2009) and Cloudy with a Chance of Meatballs 2 (2013) as Tim Lockwood, father of Bill Hader's protagonist Flint Lockwood.

From 1982 to 1987, Caan suffered from depression over his sister's death from leukemia, a growing Problem with cocaine, and what he described as "Hollywood burnout," and did not act in any films. In a 1991 interview, Caan claimed that making the 1982 film Kiss Me Goodbye was another factor in this self-imposed exile. Caan called it one of the worst experiences of his life and professed that Director Robert Mulligan was the most incompetent filmmaker he had ever worked with. He walked off the set of The Holcroft Covenant and was replaced by Michael Caine. Caan devoted much of his time during these years to coaching children's Sports.

He returned to acting in 1987, when Coppola cast him as an army platoon sergeant for the 3rd US Infantry Regiment ("The Old Guard") in Gardens of Stone, a movie that dealt with the effect of the Vietnam War on the United States homefront.

In 1988 and 1990, Caan starred in the films Alien Nation, Dick Tracy, and Misery, a hit film that marked a comeback for Caan. Since the script for Misery called for Caan's character, Paul Sheldon, to spend most of his time lying in bed, the role was turned down by many of Hollywood's leading actors before Caan accepted.

In 1992, Caan appeared in Honeymoon in Vegas, and in 1993, he played Coach Winters in The Program, alongside Halle Berry. In 1996, he appeared in Bottle Rocket, and with Arnold Schwarzenegger in Eraser, and later starred as kingpin Frank Colton in Bulletproof with Adam Sandler and Damon Wayans. In 1998, Caan portrayed Philip Marlowe in the HBO film Poodle Springs.

In 1993, a 25-year-old West Hollywood man apparently lost his footing and tumbled to his death outside a Westwood apartment where Caan was staying. Caan said in an interview that he slept through the incident.

In 1994 he was arrested after being accused by a Los Angeles rap Artist of pulling a gun on him.

Caan was married to Ingrid Hajek from September 1990 to March 1994; they had a son, Alexander James Caan, born 1991. He married Linda Stokes on October 7, 1995, they have two sons, James Arthur Caan (born 1995) and Jacob Nicholas Caan (born 1998). Caan filed for divorce on November 20, 2009, citing irreconcilable differences.

Some of his more recent appearances have been in Mickey Blue Eyes (1999), The Way of the Gun (2000), The Yards (2000), City of Ghosts (2002), Night at the Golden Eagle (2002), Dogville (2003), and Elf (2003).

In 2002, he portrayed "Jimmy the Con" in the film "This Thing of Ours," whose associate Producer was Sonny Franzese, longtime mobster and underboss of the Colombo crime family, one of New York's Five Families, and is the oldest living member of the American Mafia.

In 2003, Caan auditioned for and won the role of Montecito Hotel/Casino President "Big Ed" Deline in Las Vegas.

On February 27, 2007, Caan announced that he would not return to the show for its fifth season to return to film work; he was replaced by Tom Selleck.

Caan played the President of the United States in the 2008 film Get Smart, and had a part in the movie Cloudy with a Chance of Meatballs as the voice of the father of the lead character, Flint.

In 2012, Caan was a guest-star on the re-imagined Hawaii Five-0, playing opposite his son, Scott Caan who plays Danny "Danno" Williams. As of 2010 Caan is the chairman of an Internet company, Openfilm, intended to help upcoming filmmakers.

In 2013, Caan portrayed Chicago mob kingpin Sy Berman in the Starz TV drama Magic City. The series was not renewed for a third season, and Caan's character was apparently killed by "the Butcher" Ben Diamond, his erstwhile protege, portrayed by Danny Huston.

Crackle premiered The Throwaways on January 30, 2015. Caan plays Lt. Col. Christopher Holden, who leads a team fighting a cyberterrorist.

On March 13, 2016, Caan appeared with his co-stars at the South by Southwest premiere of The Waiting (2016); the film was picked up for theatrical distribution by Vertical Entertainment in May 2016.