He was educated at Whitestown Seminary, and then at Hamilton College in Clinton New York. At Hamilton, he was noted for his skills in oratory and debate. After his 1878 graduation, he remained at Hamilton for a for a year to study law, then continued his studies at the Utica office of Beardsley, Cookingham and Burdick, which included his brother in law Henry J. Cookingham as a partner. He was admitted to the bar in 1880, and practiced with Cookingham in the firm of Cookingham & Martin. Sherman was also President of the Utica Trust & Deposit Co. and the New Hartford Canning Company. He became mayor of Utica at age twenty-nine.

In 1881, he married Carrie Babcock of East Orange, New Jersey, and they had three sons.

In 1886, Sherman was elected U.S. Representative from New York's 23rd congressional district as a Republican, and he served 20 years in the House (four years, followed by a two-year break and 16 more years).

During his House career, Sherman served as chairman of the Committee on Indian Affairs from the 54th through the 60th Congresses (1895 to 1909).

In 1908, Sherman was nominated as the Republican candidate for Vice President on the ticket with william Howard Taft. Although not an obvious front-runner, he balanced Taft's profile, by being both an Easterner and a conservative (it was said that the two wings of the GOP 'flapped together'), and the New York delegates pressed hard for his nomination. The Republicans won by a comfortable margin. At first, Sherman and Taft found themselves at odds over both tariff policy and the role of the Vice President. But Taft presently moved to the right, and the two of them worked together more harmoniously – a relationship eased further by the First Lady's enjoyment of the company of Sherman and his wife. The President declared that Sherman accomplished much on Capitol Hill by his "charm of speech and manner, and his spirit of conciliation and compromise", backed by a "stubborn adherence" to his principles.

From 1910, Taft had experienced several disagreements with ex-President Theodore Roosevelt, who presently walked out and formed his own Bull Moose party. This made re-election for the Republicans almost impossible, but they campaigned on the same ticket in the 1912 contest, with New Yorkers once again supporting Sherman's nomination – the first time that a sitting Vice President had been re-nominated since John C. Calhoun in 1828.

Although not a high-powered administrator, he made a natural congressional committee chairman, and his genial personality eased the workings of the House, so that he was known as 'Sunny Jim'. He was the first Vice President to fly in a plane (1911), and also the first to throw out the ceremonial first pitch at a baseball game.

But Sherman's health (he was diagnosed with Bright's disease in 1904) was failing by the time of the 1912 campaign. Less than a week before the election, he died at home in Utica, six days after his 57th birthday, and President Taft was left with no running mate, although Nicholas Murray Butler was designated to receive the electoral votes that Sherman would have received. Taft and Butler came in third place in the election, carrying only eight electoral votes from Utah and Vermont. Democratic candidate Woodrow Wilson and his running mate Thomas R. Marshall won the election while Progressive candidate Theodore Roosevelt and his running mate Hiram Johnson came in second place. The office of Vice-President remained vacant until Marshall's inauguration, on March 4, 1913.

As Sherman had never held a party leadership post or been a chairman of a major committee such as Ways and Means or Appropriations, he was considered sufficiently neutral to frequently be appointed chairman of the "Committee of the Whole" — a crucial device for speeding up the passage of bills by suspending certain rules at the discretion of the chairman. Henry Cabot Lodge recognized this job as a major test of integrity and judgment, and declared that Sherman was supremely fitted for it. Through Sherman's efforts in 1900, the Sherman Indian High School in Riverside, California was built and named after him.