John Davis Long was born in Buckfield, Maine on October 27, 1838, to Zadoc Long and Julia Temple (Davis) Long. He was named for Massachusetts Governor John Davis, a cousin of his maternal grandfather. He received his primary education at Hebron Academy until attending Harvard University where he graduated Phi Beta Kappa in 1857. At Harvard he was a member of the Delta Kappa Epsilon fraternity's Alpha chapter. Long wrote both prose and verse for a student magazine, and was chosen to write an ode for his class's graduation. He also began a private journal some time before his arrival at Harvard, which he maintained throughout his life.

In addition to Long's extensive journal, he wrote on a variety of other subjects. During his unsuccessful attempt to start a law practice in Buckfield he produced a paper on Congressional power and slavery. While in Boston in the early 1860s he had a play produced locally. In 1878 he produced a verse translation of Virgil's Aeneid. In 1903 he published The New American Navy, a history of the Spanish–American War and the development of the Navy during that time.

After two years as headmaster of Westford Academy in Westford, Massachusetts, Long began attending Harvard Law School, becoming a member of the Massachusetts bar in 1861. He practiced law, first without success in Buckfield, and then in Boston, and was active in the state militia during the American Civil War. He moved to Hingham, Massachusetts in 1869, and the following year married Mary Woodford Glover of Hingham. The couple had two daughters (and one stillborn birth) before her death in 1882.

Long began his involvement in politics at the local level in Hingham in 1870. Temperance was a major issue which dominated his political beliefs. His early politics was somewhat independent: he supported the reformist Republican Benjamin Butler for governor in 1871, but received an unsolicited Democratic nomination later that year for a seat in the Massachusetts House of Representatives. He did not campaign, and lost the election. Nominated by both Democrats and reformist Republicans in 1872, he lost again. He thereafter became more of a Republican stalwart, convinced that reform would be best accomplished from within the party organization.

In 1874 Long chaired the state Republican convention, and finally won election to the state legislature. He formed a close relationship with Speaker John E. Sanford, and in what Historian James Hess describes as a probable move of political calculation, supported the successful gubernatorial candidate in 1875, Alexander H. Rice, even though Rice supported liberal legislation on alcohol sales that Long opposed. He was able to parlay this support into his own election to the speakership in 1876. He widened his reform views to the national stage by supporting Benjamin Bristow in his unsuccessful bid for the Republican presidential nomination.

In 1878 Long unsuccessfully challenged the incumbent Rice for the gubernatorial nomination. When Rice announced his retirement the following year, Long again sought the nomination. It went to former Lieutenant Governor Thomas Talbot, but Long won the lieutenant governor nomination by acclamation. The Democratic opposition was divided by Benjamin Butler's return to that party, and the Republican ticket won the general election. Long capitalized on Talbot's avoidance of public ceremonies to maintain a high profile despite his post's relative unimportance. He was easily nominated for governor when Talbot announced he would not run for reelection, despite a lack of support from the party leadership. The election was highly divisive, pitting Long against Butler and the divided Democrats. Long was criticized for his lack of Civil War Service and attacked for his diversions from the party line, but won a comfortable victory. He was reelected by comfortable margins the two following years.

Long returned to Massachusetts, where he resumed his law practice and remained interested in party politics. He sat on a few corporate boards and served as President of the Puritan Trust Company. He continued to advocate for women's suffrage, and served on the boards of several private schools, include his alma mater, Hebron Academy. He regularly spent time in Maine (having in 1882 repurchased the family home in Buckfield), and fell ill there in August 1915. He returned home to Hingham, where he died on August 28.

Long's tenure in Congress was uneventful, since the Congress was under Democratic Party control for the six years he served. In addition to lobbying the administration for patronage appointments, he sat on a joint committee examining interests of shipbuilding and shipowners, as well as on conference committees dealing with pensions and Navy financing. In 1886 Long married again, to Agnes Pierce, a Teacher and daughter of a Universalist minister; they had one son, born in 1887.

Long decided in 1888 not to run for another term in Congress, and spent the next eight years in private practice. His clients were typically corporate interests, and he appeared on their behalf in court as well as in legislative committee hearings. He was sought after as a public speaker, something he engaged in for many years. He remained somewhat active in Republican Party circles, supporting Roger Wolcott's Young Men's Republican Club, which sought to bring new blood into the party. When offered the opportunity to challenge longtime Republican Senator George Frisbie Hoar, he refused. In 1889 he was appointed to the committee overseeing the enlargement of the Massachusetts State House, a post he held until 1897.

After returning to private practice he was offered a cabinet post by his friend, President william McKinley, in 1896. He chose to become Secretary of the Navy despite lacking detailed knowledge of naval matters. He clashed with his Under-Secretary, Theodore Roosevelt, over expansion of the Navy when the Spanish–American War broke out in 1898. He resigned the post after Roosevelt became President, and resumed his law practice. He died at his home in 1915; his publications include a lifelong journal, a history of the Spanish–American War, and a verse translation of Virgil's Aeneid.

As a Congressman, Long had become a close friend of william McKinley, who was elected President in 1896. McKinley offered Long his choice of several cabinet posts; he chose Secretary of the Navy. He was confirmed by the United States Senate on March 2, 1897. The appointment brought on a storm of criticism from Henry Cabot Lodge. Lodge had been elected to the Senate, dominated the Republican Party in Massachusetts, and had expected to have a say in choosing a cabinet nominee in return for his support of McKinley. One of Lodge's supporters complained that Long was in poor health, and that he would not give the administration "back-bone and vigor". (Long had back from his law practice after a nervous breakdown.)

The loss of the Maine highlighted to the administration the nation's shortage of modern warships, setting off a scramble for the acquisition of more ships. One significant order given by Long was to transfer the USS Oregon (one of the Navy's most powerful ships) from the west coast to the Caribbean; the ship made the journey around Cape Horn from San Francisco to Key West, Florida, in 66 days. War was declared in April 1898. Roosevelt resigned his post the next month, a move Long thought foolhardy but later acknowledged was significant in advancing Roosevelt's career.

Among Long's charitable works was funding the establishment of a public library in Buckfield in 1900, which is now known as the Zadoc Long Free Library. USS Long (DD-209) was named in his honor.

After McKinley was assassinated in September 1901, Long had a change of heart, and tendered his resignation to President Roosevelt on May 1, 1902. The exact reasons for this are unclear, but several factors probably contributed. First, Roosevelt had a close relationship with Long's political rival Lodge, was known to disagree with Long on naval matters, and was not welcoming of his presence at the White House. Second, an inquiry into the actions of Admiral Winfield Scott Schley around the July 1898 Battle of Santiago de Cuba had resulted in a significant amount of criticism of Long's role in the war. Third, one of his daughters died in October 1901, less than a month after McKinley's assassination. These matters drove Long into a depression, and the situation was further exacerbated when Roosevelt squabbled with him over the beginning of the war, and then made newsworthy overrides of some of his decisions. Historian Wendell Garrett notes that Roosevelt took a great personal interest in the Navy, and had difficulty working with subsequent Secretaries.