He was born in Jammu to the singer Uma Dutt Sharma and his mother tongue is Dogri. His father started teaching him vocals and tabla when he was just five. Shivkumar started learning santoor at the age of thirteen. He gave his first public performance in Bombay in 1955.

Shivkumar is credited with making the santoor a popular classical instrument. He composed the background music for one of the scenes in Shantaram's Jhanak Jhanak Payal Baje in 1956. He recorded his first solo album in 1960.

In 1967, he teamed up with flautist Hariprasad Chaurasia and Guitarist Brij Bhushan Kabra to produce a concept album, Call of the Valley (1967), which turned out to be one of Indian classical music's greatest hits. He has composed music for many Hindi films in collaboration with Hariprasad Chaurasia starting with Silsila (1980). They came to be known as the Shiv-Hari music duo. Some of the movies they composed music for were big musical hits, such as Faasle (1985), Chandni (1989), Lamhe (1991), and Darr (1993).

Shivkumar is the recipient of national and international awards, including an honorary citizenship of the city of Baltimore, USA, in 1985, the Sangeet Natak Akademi Award in 1986, the Padma Shri in 1991, and the Padma Vibhushan in 2001.

Sharma married Manorama and has two sons. His son, Rahul, is also a santoor player and they have performed together since 1996. In a 1999 interview to rediff.com, Shivkumar stated that he chose Rahul as his shishya because he thought he had the "gift of God".