Xu was born in Wutai County, Shanxi. He was the son of a wealthy landowner and scholar who had passed the Qing civil Service examination. He attended the Taiyuan Normal College and graduated in 1923. After graduation he had a short career as school Teacher, then despite his parents' objections, he joined and attended the first class of the Whampoa Military Academy in 1924. After his graduation from the Academy he held various officer ranks in the National Revolutionary Army between 1925 and 1927. In 1926 he took part in Chiang Kai-shek's Northern Expedition to recover East China from several Warlords. After the campaign was successful he moved to Wuchang, where he taught at a military academy. While teaching in Wuchang he joined the Communist Party of China.

After the end of the Nationalist-Communist alliance in 1927 Xu went underground. He did not participate in the failed Nanchang Uprising, but led the failed Guangzhou Uprising shortly after. He did not become an associate of Mao Zedong, instead becoming one of the main military commanders of a rival Communist leader, Zhang Guotao. The army Xu commanded under Zhao was called the "4th Front Army". He served as Zhang's principle commander with Ye Jianying as Chief of Staff. During this time, he helped Zhang to establish new communist bases and expanded the 4th Front Army of the Chinese Red Army, despite the fact that his wife was executed by Zhang Guotao in his political purges. While under suspicion and the surveillance of Zhang's political commissars, Xu Xianqian lead the 80,000 strong 4th Front Army of the Chinese Red Army in Sichuan to victory against local Warlords troops that numbered more than 300,000. Over 100,000 warlord troops were killed in conflicts with Xu's forces, and the remaining 200,000 deserted or retreated to other Nationalist-aligned areas.

In 1934 Chiang Kai-shek defeated the armies associated with Zhou Enlai and Mao Zedong, forcing them to undertake the Long March. Zhang Guotao considered attacking them, but Xu refused. Xu's refusal to attack Zhang's rivals may have contributed to Mao's acceptance of Xu under his own leadership later, after Zhang's 4th Front Army was eventually defeated by Chiang. Zhang was purged after returning to the areas around Yan'an controlled by Mao, but Xu was allowed to rejoin the Red Army under Mao's leadership after making an extensive self-criticism. His first position under Mao, as the deputy commander of the 129th division, was effectively a demotion.

During the Second Sino-Japanese War (1937-1945), Xu did not remain with the 129th division, but was transferred to several different positions during the war. He briefly spend time with Luo Ronghuan building bases in Communist-controlled areas of Shandong before being transferred to He Long's United Defense Army, in which he served as deputy commander. The Communist bases Xu helped to establish proved useful after World War II ended 1945 and the Chinese Civil War resumed. In the early stages of the war, when the Kuomintang forced the Communist headquarters in Shaanxi to evacuate, it was evacuated to the bases established by Xu.

After During the Chinese Civil War, Xu participated in several battles in North China. Contrary to the Common tactic of many Communist commanders during the Civil War, who favored attacking only after establishing forces of several times the defenders, Luo often engaged numerically equal or superior forces and emerged victorious. In 1948 and 1949 Luo engaged and defeated the forces of Yan Xishan, a Shanxi warlord who was aligned with the Kuomintang.

After the Communists won the civil War in 1949, Xu served as the General Chief of Staff of the People's Liberation Army. In 1954 he was named the vice chairman of the Central Military Commission, and he was named one of the "Ten Marshals" in 1955. He became one of China's vice premiers in 1965.

Xu suffered political persecution by Red Guards in 1967, when he was accused of opposing the leadership of Lin Biao and attempting to moderate some of the more radical effects of the Cultural Revolution. He survived politically, and later that year was allowed to join both the Politburo and the Cultural Revolution Group. In 1969 he joined the Central Committee.

Xu protected Deng Xiaoping when Deng was purged from the government in 1976. Later in 1976 he was one of the military supporters of Hua Guofeng's coup against the Gang of Four, which eventually brought Deng back to power and formally ended the Cultural Revolution.

In 1978, Xu was almost killed in an accident of Chinese HJ-73 ATGM demonstration when the missile suddenly malfunctioned and turned 180 degrees after traveling several hundred meters, flying in opposite direction toward the observation platform, where Xu and other top ranking Chinese officers were sitting, and landed right in front of the platform. It was fortunate for Xu and the others on the platform that the missile failed to explode, and they survived and remained there until the completion of the demonstration. Xu did not originally plan to attend the demonstration, but because both Ye Jianying and Nie Rongzhen, who originally planned to attend, were hospitalized at the time, Xu was invited instead.

Xu led the preparations for PLA operations in the Sino-Vietnam War in 1979.

After resigning as Defense Minister in 1981, Xu remained active in politics. He served in the Politburo and the Central Committee, and was the vice chairman of the Central Military Commission. He was forced to resign his positions, along with Nie Rongzhen Ye Jianying, in 1985.

Xu died in 1990. His official obituary stated that "his life was a glorious one... Xu was an outstanding Communist, a great proletarian revolutionary, a strategist, and one of the founders of the Chinese People's Liberation Army."