Born in Kansas City, Missouri, Gadson played with the first line-up of Charles Wright's Watts 103rd Street Rhythm Band, and recorded three albums with them between 1968 and 1970. Along with other members of Wright's band he went on to appear on many hit records, including with Dyke & the Blazers. Gadson started to become well known as a Drummer following the release of the album Still Bill by Bill Withers, released by Sussex Records in 1972. He played on The Temptations album 1990, released on the Motown label in 1973. In 1975 he played with Freddie King on Larger Than Life and went on to record with Martha Reeves, Randy Crawford, Quincy Jones, Herbie Hancock, BB King, Albert King, Rose Royce, Elkie Brooks and many more artists. In 1975 he anchored the Motown classic double platinum album City Of Angels, recorded by Billy Griffin & The Miracles. Gadson was also the Drummer on Marvin Gaye's "I Want You" in 1976, and appeared on two tracks, "At The Mercy" and "Riding To Vanity Fair", on the 2005 Paul McCartney album Chaos and Creation in the Backyard.

In June 2009, Gadson joined Beck, Wilco, Feist and Jamie Lidell covering Skip Spence's Oar as part of Beck's Record Club series, with videos appearing on Beck's website beginning November 2009. He has drummed on Beck's albums Sea Change, The Information and Morning Phase, as well as Jamie Lidell's 2010 album Compass. Gadson played drums, as well as hambone (slapping his legs), on the D'Angelo song "Sugah Daddy", on the Black Messiah album (2014).