Gilmore was born in Ballygar, County Galway. He started his music career at age fifteen, and spent time in Canada with an English band. He settled in Boston, Massachusetts in 1848, becoming leader of the Suffolk, Boston Brigade, and Salem bands in swift succession. He also worked in the Boston music store of John P. Ordway and founded Ordway's Aeolians, a group of blackface minstrels. With the Salem Band, Gilmore performed at the 1857 inauguration of President James Buchanan.

In 1858 he founded "Gilmore's Band," and at the outset of war the band enlisted with the 24th Massachusetts Volunteers, accompanying General Burnside to North Carolina. After the temporary discharge of bands from the field, Governor Andrew of Massachusetts entrusted Gilmore with the task of re-organizing military music-making, and General Nathaniel P. Banks created him Bandmaster-general.

When peace resumed, Gilmore was asked to organize a celebration, which took place at New Orleans. That success emboldened him to undertake two major music festivals at Boston, the National Peace Jubilee in 1869 and the World's Peace Jubilee and International Musical Festival in 1872. These featured Monster orchestras of massed bands with the finest Singers and instrumentalists (including the only American appearance by "waltz king" Johann Strauss II) and cemented Gilmore's reputation as the leading musical figure of the age. Coliseums were erected for the occasions, holding 60- and 120,000 persons. Grateful Bostonians presented Gilmore with medals and cash, but in 1873 he moved to New York City, as bandmaster of the 22nd Regiment. Gilmore took this band on acclaimed tours of Europe.

In many ways Gilmore can be seen as the principal figure in 19th-century American music. He was a Composer, the "Famous 22nd Regiment March" from 1874 is just one Example. He held the first "Promenade Concert in America" in 1855, the forerunner to today's Boston Pops. He set up "Gilmore's Concert Garden", which became Madison Square Garden. He was the Musical Director of the Nation in effect, leading the festivities for the 1876 Centennial celebrations in Philadelphia and the dedication of the Statue of Liberty in 1886. In 1888 he started the tradition of seeing in the New Year in Times Square.

In 1891, he played for some of Thomas Edison's first commercial recordings. Musically, he was the first arranger to set brass instruments against the reeds, which remains the basis for big band orchestration. His arrangements of contemporary classics did a great deal to familiarize the American people with the work of the great European musical masters.

It was back on home soil, preparing an 1892 musical celebration of the quadricentennial anniversary of Christopher Columbus' voyage of discovery, that Gilmore collapsed and died in St. Louis. Patrick S. Gilmore was buried in Calvary Cemetery, Queens, New York, where his wife would later be interred.

Patrick Sarsfield Gilmore was inducted into the Songwriters Hall of Fame in 1970.