Cannonball moved to Tallahassee, when his parents obtained teaching positions at Florida A&M University. Both Cannonball and brother Nat played with Ray Charles when Charles lived in Tallahassee during the early 1940s. Adderley moved to Broward County, Florida, in 1948 after finishing his music studies at Florida A&M and became the band Director at Dillard High School in Fort Lauderdale, a position which he held until 1950. Cannonball was a local legend in Southeast Florida until he moved to New York City in 1955.

One of his known addresses in New York was in the neighborhood of Corona, Queens. He left Florida originally to seek graduate studies at New York conservatories, but one night in 1955 he brought his saxophone with him to the Café Bohemia. Asked to sit in with Oscar Pettiford in place of his band's regular Saxophonist, who was late for the gig, the "buzz" on the New York jazz scene after Adderley's performance announced him as the heir to the mantle of Charlie Parker.

Adderley formed his own group with his brother Nat after signing onto the Savoy jazz label in 1957. He was noticed by Miles Davis, and it was because of his blues-rooted alto saxophone that Davis asked him to play with his group. He joined the Davis band in October 1957, three months prior to the return of John Coltrane to the group. Some of Davis's finest trumpet work can be found on Adderley's first solo album Somethin' Else (also featuring Art Blakey and Hank Jones), which was recorded shortly after the two giants met. Adderley then played on the seminal Davis records Milestones and Kind of Blue. This period also overlapped with Pianist Bill Evans' time with the sextet, an association that led to recording Portrait of Cannonball and Know What I Mean?.

By the end of the 1960s, Adderley's playing began to reflect the influence of the electric jazz. From this period, he released albums such as Accent on Africa (1968) and The Price You Got to Pay to Be Free (1970). In that same year, his quintet appeared at the Monterey Jazz Festival in California, and a brief scene of that performance was featured in the 1971 psychological thriller Play Misty for Me, starring Clint Eastwood. In 1975 he also appeared in an acting role alongside José Feliciano and David Carradine in the episode "Battle Hymn" in the third season of the TV series Kung Fu.

His interest as an educator carried over to his recordings. In 1961, Cannonball narrated The Child's Introduction to Jazz, released on Riverside Records.

Adderley is remembered for his 1966 soul jazz single "Mercy, Mercy, Mercy", a crossover hit on the pop charts (it was also covered by The Buckinghams). He worked with trumpeter Miles Davis, on his own 1958 Somethin' Else album, and on the seminal Davis records Milestones (1958) and Kind of Blue (1959). He was the older brother of jazz trumpeter Nat Adderley, a longtime member of his band.

Later that year he was inducted into the Down Beat Jazz Hall of Fame. Joe Zawinul's composition "Cannon Ball" on Black Market is a tribute to his former leader. Pepper Adams and George Mraz dedicated the composition "Julian" on the 1975 Pepper Adams album of the same name days after Cannonball's death.