Khan was born on 21 March 1916 into a family of traditional Muslim Musicians in Bhirung Raut Ki Gali, Dumraon, in what is now the eastern Indian state of Bihar, as the second son of Paigambar Baksh Khan and Mitthan. While named Qamruddin at birth, his grandfather Rasool Baksh Khan, also a shehnai player, exclaimed "Bismillah" upon seeing the new born. Thereafter he came to be known by this name. His father was a court musician employed by the Raja of Dumraon. His great grandfather Ustad Salar Hussain Khan and grandfather Rasool Baksh Khan were also Musicians in the Dumraon palace.

Khan was perhaps single-handedly responsible for making the shehnai a famous classical instrument. He brought the Shehnai to the center stage of Indian music with his concert in the Calcutta, now known as Kolkata in All India Music Conference in the year 1937. He was credited with having the almost monopoly over the instrument as he and the shehnai are almost synonyms.

Khan had the rare honour of performing at Delhi's Red Fort on the eve of India's Independence in 1947. He also performed raga Kafi from the Red Fort on the eve of India's first Republic Day ceremony, on 26 January 1950. His recital had become a cultural part of India's Independence Day celebrations, telecast on Doordarshan every year on 15 August. After the prime minister's speech from Lal Qila (the Red Fort) in Old Delhi, Doordarshan would broadcast a live performance by the shehnai Maestro. This tradition dated from the days of Jawaharlal Nehru.

Khan had a brief association with movies. He played the shehnai for super star Rajkumar's role of Appanna in the Kannada movie Sanaadi Appanna which became a blockbuster. He acted in Jalsaghar, a movie by Satyajit Ray and provided sound of shehnai in Goonj Uthi Shehnai (1959). Noted Director Goutam Ghose directed Sange Meel Se Mulaqat, a documentary about the life of Khan. In the 1967 film The Graduate, there is a poster advertising "Bismillah Khan and the seven musicians" on a busy street of Berkeley, California. The most recent use of his shehnai music in a Hindi film was in Rockstar (2011).

He was awarded India's highest civilian honour, the Bharat Ratna, in 2001, becoming the third classical musician after M. S. Subbulakshmi and Ravi Shankar to be accorded this distinction. On his 102nd birthday, Google honored Bismillah Khan with a Google doodle.

On 17 August 2006, Bismillah Khan's health deteriorated and he was admitted to the Heritage Hospital, Varanasi for treatment. Ustad's last wish - to perform at India Gate, could not be fulfilled. He wanted to pay tributes to the martyrs. He waited in vain till his last rites He died after four days on 21 August 2006 because of a cardiac arrest. He is survived by five daughters, three sons and a large number of grandchildren and great-grandchildren, and his adopted daughter Soma Ghosh (a Hindustani Shastriya Sangeet exponent).

Sangeet Natak Akademi, New Delhi, instituted the Ustad Bismillah Khan Yuva Puraskar in 2007, in his honour. It is given to young artists in the field of music, theatre and dance.

Bismillah Khan was commemorated on his 102nd birth anniversary by Search Engine Google which showed a special doodle on its Indian home page for him on 21 March 2018.