In his political career, Smith built on his working-class beginnings, identifying himself with immigrants and campaigning as a man of the people. Although indebted to the Tammany Hall political machine, particularly to its boss, "Silent" Charlie Murphy, he remained untarnished by corruption and worked for the passage of progressive legislation. It was during his early unofficial jobs with Tammany Hall that he gained renown as an excellent speaker. Smith's first political job was in 1895 as an investigator in the office of the Commissioner of Jurors as appointed by Tammany Hall.

His father Alfred owned a small trucking firm, but died when the boy was 13. At 14 Smith had to drop out of St. James parochial school to help support the family, and worked at a fish market for seven years. Prior to dropping out of school, he served as an altar boy, and was strongly influenced by the Catholic Priests he worked with. He never attended high school or college, and claimed he learned about people by studying them at the Fulton Fish Market, where he worked for $12 per week. His acting skills made him a success on the amateur theater circuit. He became widely known, and developed the smooth oratorical style that characterized his political career. On May 6, 1900, Al Smith married Catherine Ann Dunn, with whom he had five children.

Smith was first elected to the New York State Assembly (New York Co., 2nd D.) in 1904, and repeatedly elected to office, serving through 1915. After being approached by Frances Perkins, an Activist to improve labor practices, Smith sought to improve the conditions of factory workers. He served as vice chairman of the state commission appointed to investigate factory conditions after 146 workers died in the 1911 Triangle Shirtwaist Factory fire. Meeting the families of the deceased Triangle factory workers left a strong impression on him. Together with Perkins, Smith crusaded against dangerous and unhealthy workplace conditions and championed corrective legislation.

In 1911, the Democrats obtained a majority of seats in the State Assembly; and Smith became Majority Leader and Chairman of the Committee on Ways and Means. In 1912, following the loss of the majority, he became the Minority Leader. When the Democrats reclaimed the majority after the next election, he was elected Speaker for the 1913 session. He became Minority Leader again in 1914 and 1915. In November 1915, he was elected Sheriff of New York County, New York. By now he was a leader of the Progressive movement in New York City and state. His campaign manager and top aide was Belle Moskowitz, a daughter of Jewish immigrants.

After serving in the patronage-rich job of Sheriff of New York County, Smith was elected President of the Board of Aldermen of the City of New York in 1917. Smith was elected Governor of New York at the New York state election, 1918 with the help of Murphy and James A. Farley, who brought Smith the upstate vote.

In 1919, Smith gave the famous speech, "A man as low and mean as I can picture", making a drastic break with william Randolph Hearst. Publisher Hearst, known for his notoriously sensationalist and largely left-wing position in the state Democratic Party, was the leader of its populist wing in the city. Hearst had combined with Tammany Hall in electing the local administration. Hearst had attacked Smith for starving children by not reducing the cost of milk.

Smith was an articulate proponent of good government and efficiency, as was Hoover. Smith swept the entire Catholic vote, which had been split in 1920 and 1924 between the parties; he attracted millions of Catholics, generally ethnic whites, to the polls for the first time, especially women, who were first allowed to vote in 1920. He lost important Democratic constituencies in the rural North and in southern cities and suburbs. He did succeed in the Deep South, thanks in part to the appeal of his running mate, Senator Joseph Robinson from Arkansas, but he lost five southern states to Hoover. Smith carried the ten most populous cities in the United States, an indication of the rising power of the urban areas and their new demographics. In addition to the issues noted above, Smith was not a very good campaigner. His campaign theme song, "The Sidewalks of New York", had little appeal for rural folks, and they found his 'city' accent, when heard on the "raddio," seemed slightly foreign. Smith narrowly lost New York state, whose electors were biased to rural upstate and largely Protestant districts. But in 1928 his fellow Democrat Roosevelt (a Protestant of Dutch old-line stock) was elected to replace him as governor of New York. James A. Farley left Smith's camp to run Franklin D. Roosevelt's successful campaign for Governor, and later Roosevelt's successful campaigns for the Presidency in 1932 and 1936.

At the 1924 Democratic National Convention, Smith unsuccessfully sought the Democratic nomination for President, advancing the cause of civil liberty by decrying lynching and racial violence. Roosevelt made the nominating speech in which he saluted Smith as "the Happy Warrior of the political battlefield." Smith represented the urban, east coast wing of the party as an anti-prohibition "wet" candidate while his main rival for the nomination, California Senator william Gibbs McAdoo, stood for the more rural tradition and prohibition "dry" candidacy. The party was hopelessly split between the two. An increasingly chaotic convention balloted 100 times before both men accepted that neither would be able to win the two-thirds majority required to win, and so each withdrew. The exhausted party nominated the little-known John W. Davis of West Virginia. Davis lost the election by a landslide to the Republican Calvin Coolidge, who won in part because of the prosperous times.

Some political Scientists believe that the 1928 election started a voter realignment that helped develop the New Deal coalition of Franklin D. Roosevelt. As one political scientist explains, "...not until 1928, with the nomination of Al Smith, a northeastern reformer, did Democrats make gains among the urban, blue-collar and Catholic voters who were later to become core components of the New Deal coalition and break the pattern of minimal class polarization that had characterized the Fourth Party System." However, Allan Lichtman's quantitative analysis suggests that the 1928 results were based largely on religion and are not a useful barometer of the voting patterns of the New Deal era.

In 1929 Smith was elected as President of the Board of Trustees of the New York State College of Forestry at Syracuse University.

After the 1928 election, Smith became the President of Empire State, Inc., the corporation that built and operated the Empire State Building. Construction for the building began symbolically on March 17, 1930, St. Patrick's Day, per Smith's instructions. Smith's grandchildren cut the ribbon when the world's tallest skyscraper opened on May 1, 1931, which was May Day, an international labor celebration. It had been completed in a record 13 months for such a large project. As with the Brooklyn Bridge, which Smith had seen being built from his Lower East Side boyhood home, the Empire State Building was a vision and an achievement constructed by combining the interests of all, rather than being divided by interests of a few.

Smith felt slighted by Roosevelt during the latter's governorship. They became rivals for the 1932 Democratic presidential nomination. At the convention, Smith's animosity toward Roosevelt was so great that he put aside longstanding rivalries and managed to work with william McAdoo and william Randolph Hearst to try to block FDR's nomination for several ballots. This unlikely coalition fell apart when Smith refused to work on finding a compromise candidate; instead he maneuvered to become the nominee. After losing the nomination, Smith eventually campaigned for Roosevelt in 1932, giving a particularly important speech on behalf of the Democratic nominee at Boston on October 27 in which he "pulled out all the stops."

Smith was an early and vocal critic of the Nazi regime in Germany. He supported the Anti-Nazi boycott of 1933 and addressed a mass-meeting at Madison Square Garden against Nazism that March. His speech was included in the 1934 anthology Nazism: An Assault on Civilization. In 1938, Smith took to the airwaves to denounce Nazi brutality in the wake of the Kristallnacht.

Smith became highly critical of Roosevelt's New Deal policies, which he deemed a betrayal of good-government progressive ideals and ran counter to the goal of close cooperation with Business. Smith joined the American Liberty League, an organization founded by Conservative Democrats who disapproved of Roosevelt's New Deal measures and tried to rally public opinion against Roosevelt's New Deal. In 1934, Smith joined forces with wealthy Business executives, who provided most of the League's funds. The League published pamphlets and sponsored radio programs, arguing that the New Deal was destroying personal liberty. However, the League failed to gain support in the 1934 and 1936 elections, and rapidly declined in influence. It was officially dissolved in 1940. Smith's antipathy to Roosevelt and his policies was so great that he supported Republican presidential candidates Alfred M. Landon (in the 1936 election) and Wendell Willkie (in the 1940 election).

In 1939 Smith was appointed a Papal Chamberlain of the Sword and Cape, one of the highest honors the Papacy bestowed on a layman. In the early 21st century, this honor is styled a Gentleman of His Holiness.

Smith died at the Rockefeller Institute Hospital on October 4, 1944, of a heart attack, at the age of 70. He had been broken-hearted over the death of his wife from cancer five months earlier, on May 4, 1944. He is interred at Calvary Cemetery.

Although personal resentment was one factor in Smith's break with Roosevelt and the New Deal, Christopher Finan (2003) argues that Smith was consistent in his beliefs and politics—suggesting that Smith always believed in social mobility, economic opportunity, religious tolerance, and individualism. Despite the break between the men, Smith and Eleanor Roosevelt remained close. In 1936, while Smith was in Washington making a vehement radio attack on the President, she invited him to stay at the White House. To avoid embarrassing the Roosevelts, he declined. Historian Robert Slayton notes that Smith and Franklin Roosevelt did not reconcile until a brief meeting in June, 1941, and suggests that during the early 1940s the antipathy Smith held toward his former ally had waned. Upon the death of Smith's wife Katie in May, 1944, FDR sent Smith a note of personal condolences; Smith's grandchildren later recalled that Smith was "greatly touched by it."

Smith was born at 174 South Street, and raised in the Fourth Ward on the Lower East Side of Manhattan; he resided here for his entire life. His mother, Catherine (Mulvihill), was the daughter of Maria Marsh and Thomas Mulvihill, who were immigrants from County Westmeath, Ireland. His father, Alfred Emanuele Ferraro, took the anglicized name Alfred E. Smith ('ferraro' means 'blacksmith' or 'smith' in Italian). The elder Alfred was the son of Italian and German immigrants. He served with the 11th New York Fire Zouaves in the opening months of the Civil War.