Smith was born in Manayunk, now a neighborhood in North West Philadelphia, in 1864.

Smith began his professional baseball career at age 18 playing for the Baltimore club in the National Association in 1883. In 1884, he played one game as pitcher for the Philadelphia Athletics and another for the Pittsburg Alleghenys. He compiled an 0-2 win-loss records and a 6.35 earned run average (ERA) in his first season in the major leagues.

John Smith was married twice. The first time he married Margaret Cowan in 1884. They had two daughters, Isabel Smith Warnick, and Elsie Smith MacFarland. Margaret Cowan was from Scotland but her parents were born in Ireland. John was married to Hannah M. Smith in approximately 1900. They had at least ten children: Anna (born c. 1902), George (born c. 1903), John (born c. 1905), Charles (born c. 1907), Walter (born 1909), Agnes (born c. 1911), Mary (born c. 1913), James (born c. 1916), Florence (born c. 1920), and Margaret (born c. 1922).

In his first and only game for Brooklyn, on June 17, 1885, Smith's teammates were reported to have intentionally committed 14 errors to punish Smith for his perceived brash and cocky demeanor. Brooklyn lost the game, 18-5. All 18 were initially scored as unearned‚ but record books indicate that 11 of the runs were earned. He reportedly claimed that he was so good that he did not need his teammates to win. The intentional misplays of his teammates caused club President Lynch to fine the guilty players $500 each‚ but he instead released Smith "to ensure team harmony."

In 1886, Smith appeared in 33 games as a pitcher for Newark. He threw 33 complete games and compiled a 22-10 record with a 0.74 ERA.

In his eight seasons in the major leagues, Smith appeared as a pitcher in 140 games and compiled a 54-74 (.422) win-loss record with a 3.89 earned run average (ERA) and 519 strikeouts. He saw his most extensive playing time with the Baltimore Orioles of the American Association, compiling a 41-50 win-loss record during the 1887 and 1888 seasons.

In early October 1888, the Philadelphia Athletics purchased Smith from the Baltimore Orioles for $1000 or $1500. He pitched three games for the Athletics at the end of the 1888 season, compiling a 2-1 record and a 2.86 ERA. In the first half of the 1889 season, he went 2-3 with a 4.40 ERA for the Athletics. In early June 1889, Smith was released by the Athletics. He finished the 1889 season playing for Hartford in the Atlantic Association.

Smith played in the National League in 1890 for the Philadelphia Phillies and Pittsburgh Alleghenys. He compiled a 9-15 record and a 4.06 ERA in 29 games as pitcher.

Although his major league career ended in July 1891, Smith continued to play and manage in the minor leagues for another 15 years. His minor league career included stints with at least 12 different clubs, including the Milwaukee Brewers (1891), Green Bay Bays (1892), Reading Actives (1893), Pottsville Colts (1894–1895), Millville (1895), Pawtucket Phenoms (1896–1897), Fall River Indians (1898), Newark Colts (1898), Hartford Cooperatives (1898), Portland Phenoms (1899), and Norfolk Phenoms (1900).

Smith was a player-manager for most of his minor league career, and he led Pottsville to the Pennsylvania League pennant in 1894 and Norfolk to a Virginia League pennant in 1900.

While working as a minor league player-manager, Smith has been credited with discovering Baseball Hall of Fame inductee, Christy Mathewson. Smith met and played against Matthewson in 1899 while Matthewson was still attending Bucknell and pitching for the Taunton Herrings. The 19-year-old Mathewson compiled a 2-13 record for Taunton in 1899, but Smith saw potential. Smith signed Matthewson to play for his Norfolk team for $90 a month, and under Smith's guidance, Matthewson compiled a 21-2 record and 0.74 ERA in 1900 and was promoted to the major leagues before the end of the season.

Smith spent the last five years of his baseball career as a player-manager for the Manchester, New Hampshire baseball club from 1901 to 1905. At age 36, he won the 1901 New England League batting title with a .363 batting average in 284 at bats. Smith remained in Manchester for the rest of his life.

After retiring from baseball, Smith became a member of the Manchester, New Hampshire police department. He served in that capacity from approximately 1904 to 1932. He also coached the basketball team at Saint Anselm College in Goffstown, New Hampshire, for seven seasons between 1909 and 1932. In 1952, he died at a hospital in Manchester at age of 87. He was buried at St. Joseph Cemetery in Bedford, New Hampshire.