Mohammed Rafi was the second eldest of six brothers born to Haji Ali Mohammad. The family originally belonged to Kotla Sultan Singh, a village near present-day Majitha in Amritsar district of Punjab, India. Rafi, whose nickname was Pheeko, began singing by imitating the chants of a fakir in the streets of his native village Kotla Sultan Singh where his family lived. Rafi's father moved to Lahore in 1935, where he ran a men's salon in Noor Mohalla in Bhati Gate. Rafi learnt classical music from Ustad Abdul Wahid Khan, Pandit Jiwan Lal Mattoo and Firoze Nizami. His first public performance came at the age of 13, when he sang in Lahore featuring K. L. Saigal. In 1941, Rafi, under Shyam Sundar, made his debut in Lahore as a playback singer in the duet "Soniye Nee, Heeriye Nee" with Zeenat Begum in the Punjabi film Gul Baloch (released in 1944) under music Director Shyam Sunder. In that same year, Rafi was invited by All India Radio Lahore station to sing for them.

As per Naushad, Rafi came to him with a letter of recommendation from Naushad's father. Rafi's first song for Naushad Ali was "Hindustan Ke Hum Hain" ("We belong to Hindustan") for the film Pehle Aap in 1944. The first song for the duo was the Soundtrack of the movie Anmol Ghadi (1946). Before Rafi, Naushad's favourite singer was Talat Mahmood. Once Naushad found Talat smoking during a recording. He was annoyed, and hired Rafi to sing all the songs of the movie Baiju Bawra.

Rafi appeared in two movies. In 1945, he appeared on the screen for the song "Tera Jalwa Jis Ne Dekha" in the film Laila Majnu. He sang a number of songs for Naushad as part of the chorus, including "Mere Sapnon Ki Rani, Roohi Roohi" with K. L. Saigal, from the film Shahjahan (1946). Rafi sang "Tera Khilona Toota Balak" from Mehboob Khan's Anmol Ghadi (1946) and a duet with Noor Jehan in the 1947 film Jugnu, "Yahan Badla Wafa Ka". After partition, Rafi decided to stay back in India and had the rest of his family flown to Mumbai. Noor Jehan migrated to Pakistan and made a pair with playback singer Ahmed Rushdi.

In 1948, after the assassination of Mahatma Gandhi, the team of Husanlal Bhagatram-Rajendra Krishan-Rafi had overnight created the song "Suno Suno Ae Duniyawalon, Bapuji Ki Amar Kahani". He was invited by the Indian Prime Minister, Jawaharlal Nehru, to sing at his house. In 1948, Rafi received a silver medal from Jawaharlal Nehru on Indian Independence Day.

In 1949, Rafi was given solo songs by music Directors such as Naushad (Chandni Raat, Dillagi and Dulari) Shyam Sunder (Bazaar) and Husnalal Bhagatram (Meena Bazaar).

Rafi and O. P. Nayyar (OP) created music in the 1950s and 1960s. O. P. Nayyar was once quoted as saying "If there had been no Mohd. Rafi, there would have been no O. P. Nayyar". He and Rafi created many songs together including "Yeh Hai Bombay Meri Jaan". He got Rafi to sing for singer-actor Kishore Kumar – "Man Mora Baawara" for the movie Raagini. Later, Rafi sang for Kishore Kumar in movies such as Baaghi, Shehzaada and Shararat. O. P. Nayyar used Rafi and Asha Bhosle for most of his songs. The team created many songs in the early 1950s and 1960s for movies such as Naya Daur (1957), Tumsa Nahin Dekha (1957), and Kashmir Ki Kali (1964). Rafi sang a total of 197 numbers (56 solo) for Nayyar. The songs "Jawaaniyan yeh mast mast" and the title song "Yun to humne lakh hansee dekhe hain, tumsa nahin dekha" of the film Tumsa Nahin Dekha were hits. They were followed by songs like "Taareef karoon kya uski jisne tumhe banaya" from Kashmir ki Kali.

Rafi's association with Naushad helped the former establish himself as one of the most prominent playback Singers in Hindi cinema. Songs from Baiju Bawra (1952) like "O duniya ke rakhwale" and "Man tarpat hari darshan ko aaj" furthered Rafi's credentials. Rafi ended up singing a total of 149 songs (81 of them solo) for Naushad.

S.D. Burman used Rafi as a singing voice of Dev Anand and Guru Dutt. Rafi worked with Burman in 37 films, including Pyaasa (1957), Kaagaz Ke Phool (1959), Kala Bazar, Nau Do Gyaran, Kala Pani , Tere Ghar Ke Samne (1963), Guide (1965), Aradhana (1969), and Abhimaan (1973).

Kalyanji Anandji composed arond 170 songs in the voice of Rafi. Kalyanji's relationship with Rafi started with the 1958 film, Samrat Chandragupta, his debut film as a solo Composer. Kalyani-Anandji and Rafi went on to work together for the music of the Shashi Kapoor-starrer Haseena Maan Jayegi (1968), which featured songs like "Bekhudi Mein Sanam" and "Chale The Saath Milke".

Rafi got his first Filmfare Award for the title song of Chaudhvin Ka Chand (1960), composed by Ravi. He received the National Award for the song "Babul Ki Duaen Leti Ja" from the film Neel Kamal (1968), also composed by Ravi. Rafi wept during the recording of this song. He admitted this in his interview to the BBC in 1977.

Ravi and Rafi produced several other songs in the films China Town (1962), Kaajal (1965), and Do Badan (1966).

The Composer duo Laxmikant-Pyarelal (L-P) patronised Rafi as one of their Singers, right from their very first song by him from the film Parasmani (1963). Rafi and L-P won the Filmfare Awards for the song "Chahoonga Main Tujhe Saanjh Savere" from Dosti (1964). Rafi rendered the highest number of songs for the music Director duo Laxmikant-Pyarelal: 388 numbers for L-P.

Rafi's song from the film Gumnaam (1965), "Jaan Pehechan Ho", was used on the Soundtrack of Ghost World (2001). The film opens with the lead character dancing around in her bedroom to a video of Gumnaam. The song has also been used for Heineken's 2011 "The Date" commercial.

Rafi recorded Hindi songs in English on 7" release in 1968. He also sang a song in Creole while on his visit to Mauritius in the late 1960s. Rafi recorded two English albums as well. One of them is Pop Hits.

Rafi sang for many successful films in the late 1970s and the early 1980s many of whose hit songs were dominating the charts in the late 70s on radio programs such as Vividh Bharati, Binaca Geetmala and Radio Ceylon. Some of these include Bairaag (1976), Dharamveer (1977), Apnapan (1978), Ganga Ki Saugandh (1978), Suhaag (1979), Sargam (1979), Qurbani (1980), Dostana (1980), Karz (1980), The Burning Train (1980), Abdullah (1980), Shaan (1980), Aasha (1980), Aap To Aise Na The (1980), Naseeb (1981) and Zamane Ko Dikhana Hai (1982). In 1978, Rafi gave a performance at the Royal Albert Hall and in 1980 he performed at the Wembley conference centre. From 1970 until his death he toured around the world extensively giving concert performances to packed halls.

In 1976, Rafi sang all the songs for Rishi Kapoor in the hit film Laila Majnu. Rafi went on to sing many more songs for Rishi Kapoor in the subsequent hit films, including Hum Kisise Kum Naheen (1977), and Amar Akbar Anthony (1977). In 1977, he won both Filmfare Award and the National Award for the song "Kya Hua Tera Wada" from the movie Hum Kisise Kum Naheen, composed by R.D. Burman. He was nominated as the best singer at the Filmfare Awards for the qawwali "Pardah Hai Pardah" from Amar Akbar Anthony (1977).

During his last years, Rafi was involved in a controversy over Lata Mangeshkar's entry in the Guinness Book of World Records. In a letter dated 11 June 1977 to the Guinness Book of World Records, Rafi had challenged the claim that Lata Mangeshkar has recorded the highest number of songs ("not less than 25,000" according to Guinness). Rafi, according to his fans, would have sung more songs than Lata – he being the senior of the two. They estimated the number of songs sung by Rafi to be anything from 25,000 to 26,000. This prompted Rafi to write a letter, in protest, to Guinness. After receiving a reply from Guinness, in a letter dated 20 November 1979, he wrote, "I am disappointed that my request for a reassessment vis-a-vis Ms Mangeshkar's reported world record has gone unheeded." In an interview to BBC recorded in November 1977, Rafi claimed to have sung 25,000 to 26,000 songs till then.

In December 1979, Rafi recorded six songs for the Hindi remake of Dilip Sen's Bengali superhit Sorry Madam; the film was never completed due to a personal tragedy in Dilip Sen's life. These songs, written by Kafeel Aazar and composed by Chitragupta, were released digitally in December 2009 by the label Silk Road under the title "The Last Songs". The physical album was released only in India by Universal.

Mohammed Rafi died at 10:25 pm on 31 July 1980, following a massive heart attack, aged 55. His last song was for the movie Aas Paas, which he had recorded with Laxmikant-Pyarelal. The last song of Rafi was "Shaam Phir Kyun Udaas Hai Dost / Tu Kahin Aas Paas Hai Dost", recorded just hours before his death.

A song in the 1990 Hindi film Kroadh "Na Fankar Tujhsa" picturised on actor Amitabh Bachchan and sung by singer Mohammed Aziz was also dedicated to the memory of Rafi.

Rafi is one of the recording artists mentioned in the 1997 hit British alternative rock song "Brimful of Asha" by Cornershop.

His "Aaj Mausam Bada Beiman Hai" is featured in the 2001 film Monsoon Wedding. His "Kya Mil Gaya" (Sasural, 1961) has been used in The Guru (2002), where Ramu and Sharonna sing a version of the song. His song "Mera Man Tera Pyasa" (Gambler, 1970) has been used as one of the soundtracks in the Jim Carrey-Kate Winslet starrer Eternal Sunshine of the Spotless Mind (2004). This song is played in the background in Kate Winslet's character's home while the lead pair are having a drink (at approximately 00.11.14 runtime).

In 2001, Rafi was named as the "best singer of the millennium" by Hero Honda and Stardust magazine.

On 22 September 2007, a shrine to Rafi designed by Artist Tasawar Bashir was unveiled on Fazeley Street, Birmingham, UK. Bashir is hoping that Rafi will attain sainthood as a result. The Padma Shri Mohammed Rafi Chowk in the Bandra suburb of Mumbai and Pune (extending MG Road) is named after Rafi.

In the summer of 2008, the City of Birmingham Symphony Orchestra released a double CD titled Rafi Resurrected comprising 16 songs by Rafi. Bollywood playback singer Sonu Nigam provided the vocals for this project and toured with the CBSO in July 2008 at venues including the English National Opera in London, Manchester's Apollo Theatre and Symphony Hall, Birmingham.

Mohammed Rafi Academy was launched in Mumbai on 31 July 2010 on the 30th anniversary of the singer's death, started by his son Shahid Rafi to impart training in Indian classical and contemporary music.

In 1962–1963, the popular female playback singer Lata Mangeshkar raised the issue of playback singers' share in the royalties. Recognizing Rafi's position as the leading male playback singer, she wanted him to back her in demanding a half-share from the 5% song royalty that the film's Producer conceded to select composers. Rafi refused to side with her, stating that his claim on the film producer's money ended with his being paid his agreed fee for the song. Rafi argued that the Producer takes financial risk and the Composer creates the song, so the singer does not have any claim over the royalty money. Lata viewed his stand as a stumbling block on the royalty issue and stated that it is because of the singer's name also that the records get sold. This difference of opinion subsequently led to differences between the two. During the recording of "Tasveer Teri Dil Mein" (Maya, 1961), Lata argued with Rafi over a certain passage of the song. Rafi felt belittled, as music Director Salil Chowdhury sided with Lata. The situation worsened when Lata declared that she would no longer sing with Rafi. Rafi stated that he was only so keen to sing with Lata as she was with him. The music Director Jaikishan later negotiated a reconciliation between the two. In an interview given to The Times of India on 25 September 2012, Lata claimed to have received a written apology from Rafi. However, Shahid Rafi, Mohammad Rafi's son, rebuffs the claim, calling it an act to dishonour his father's reputation.

In a CNN-IBN survey in 2013, he was voted the greatest voice of Hindi Cinema.

Rafi was commemorated on his 93rd birth anniversary by Search Engine Google which showed a special doodle on its Indian home page for him on 24 December 2017.