Coppola played the flute. He studied at Juilliard, later at the Manhattan School of Music and privately with Joseph Schillinger. During the 1940s, Coppola worked under Arturo Toscanini with the NBC Symphony Orchestra. Then in 1951, Coppola left the Orchestra to pursue his dream of composing music. During that time he mostly worked as an orchestra Conductor on Broadway and elsewhere, working with his son, filmmaker Francis Ford Coppola, on additional music for his Finian's Rainbow.

Carmine contributed to the music performed in the wedding scene in The Godfather (1972). Later, his son called on him to compose additional music for the score of The Godfather Part II (1974), in which he and his father received an in-movie tribute with the characters Agostino and Carmine Coppola, who appear in a deleted scene from the young Vito Corleone flashback segments. Principal score Composer Nino Rota and Carmine together won Oscars for Best Score for the film. He also composed most of the score for The Godfather Part III (1990). He made cameo appearances in all three Godfather films as a Conductor.

Carmine and Francis together scored Apocalypse Now (1979), for which they won a Golden Globe Award for best original score. He also composed a three-and-a-half-hour score for Francis' 1981 reconstruction of Abel Gance's 1927 epic Napoléon. Carmine composed the music for The Black Stallion (1979), on which Francis was executive Producer, and four other films directed by his son in the 1980s. In his audio commentary on The Godfather Part III DVD, Francis said that his father missed a cue during the shooting of that film's opening wedding reception—something he never did in his prime. At that point, Francis realized that his father had little time left. As it turned out, Carmine died less than four months after Part III premiered.

His wife, Italia, died in 2004 in Los Angeles. Coppola died in Northridge, California at the age of 80. Both Coppola and his wife are buried at San Fernando Mission Cemetery. Upon Coppola's death, his grandson Robert Schwartzman changed his last name to 'Carmine' in his grandfather's honor.