His presence attracted the curiosity of the villagers and the religiously-inclined such as Mhalsapati, Appa Jogle and Kashinatha regularly visited him, while others such as the village children considered him mad and threw stones at him.After some time he left the village, and it is unknown where he stayed at that time or what happened to him. However, there are some indications that he met with many saints and fakirs, and worked as a weaver; he claimed to have fought with the army of Rani Lakshmibai of Jhansi during the Indian Rebellion of 1857.

Sai Baba returned to Shirdi in 1858. He appeared at the Khandoba Mandir in Shirdi. The temple priest, Mhalsapati, upon seeing him for the very first time, welcomed him by saying 'Aao, Sai!' ('Come Sai'). From then on, He was known by the name (Sai Baba).

During Sai Baba's lifetime, the Hindu saint Anandanath of Yewala declared Sai Baba to be a "spiritual Diamond." Another saint, Gangagir, also called him a "jewel." Sri Beedkar Maharaj greatly revered Sai Baba, and in 1873, when he met him he bestowed the title Jagad guru upon him. Sai Baba was also greatly respected by Vasudevananda Saraswati (known as Tembye Swami). He was also revered by a group of Shaivic yogis, known as the Nath-Panchayat. He is considered an avatar of the Supreme Reality (Brahman or God), a satguru, or saint, depending on individual proclivities. This is not uncommon in Hinduism where there is no central doctrine or cosmology, but a basis in individual faith and spirituality.

After 1910, Sai Baba's fame began to spread in Mumbai. Numerous people started visiting him, because they regarded him as a saint with the power of performing miracles or even as an avatar. They built his first temple at Bhivpuri, Karjat.

Meher Baba, who was born into a Zoroastrian family, met Sai Baba once, during World War I, in December 1915. This event is considered as the most significant in Meher Baba's life. Shri Sai Satcharita (Sai Baba's life story), makes no mention of Meher Baba but Lord Meher, the life story of Meher Baba, there are numerous references to Sai Baba.

Because of Sai Baba, Shirdi has become a place of importance and is counted among the major Hindu places of pilgrimage. The first Sai Baba temple is situated at Kudal, Sindhudurg. This temple was built in 1922. It is believed that Sai Baba gave one Rupee to Dada Madye ji with which he built the temple in Kudal.

Baba is known to have grown, nurtured and cultivated a garden called "Lendi Baug".  during his lifetime. The garden was watered daily by Baba himself. Lendi Baug got its name from a well known river which used to previously flow there. The garden was full of trees and flowering plants that included a rose garden where deer and rabbits roamed about. Baba went to Lendi Baug daily and every day he threw some silver coins in the 'Lendi' ('Well'). Baba was also known to test his devotees to see if they hankered after money and gold. Baba used to come here every morning and afternoon entering alone through the west and rest under a Neem tree. Sai Baba also dug a pit, 2 feet deep, under the Neem tree and kept an earthen lamp lit continuously. Lendi Baug also has a well which was dug by Baba and his devotees.

In 2008, India Post has issued a commemorative postage stamp of ₹5.00 to honour Sai Baba.

Sai Baba of Shirdi is especially revered and worshiped in the states of Maharashtra, Odisha, Andhra Pradesh, Telangana, Karnataka, Tamil Nadu and Gujarat. In August 2012, an unidentified devotee, for the first time, donated two expensive diamonds valuing ₹11.8 million at the Shirdi temple, as revealed by Saibaba trust officials.

Sai Baba interpreted the religious texts of both Islam and Hinduism. He explained the meaning of the Hindu scriptures in the spirit of Advaita Vedanta. His philosophy also had numerous elements of bhakti. The three main Hindu spiritual paths — Bhakti Yoga, Jnana Yoga, and Karma Yoga — influenced his teachings.

The Shirdi Sai Baba movement began in the 19th century, while he was living in Shirdi. A local Khandoba priest, Mhalsapati Nagre, is believed to have been his first devotee. In the 19th century, Sai Baba's followers were only a small group of inhabitants of Shirdi, and a few people from other parts of India.