Mellon was born in Pittsburgh, Pennsylvania on March 24, 1855. His name is listed on the 1860 Census as "William A. Mellon." His Father was Thomas Mellon, a banker and judge who was a Scots-Irish immigrant from County Tyrone, Ireland; his mother was Sarah Jane Negley Mellon. He had three older brothers, Thomas A., James R. and Samuel, and he also had a younger brother named Richard B. Mellon. He was educated at the Western University of Pennsylvania (now the University of Pittsburgh) and left before graduating.

Mellon demonstrated financial ability early. In 1872 his Father set him up in a lumber and coal Business, which he soon turned into a profitable enterprise. He joined his father's banking firm, T. Mellon & Sons, in 1880 and two years later had ownership of the bank transferred to him. In 1889, Mellon helped organize the Union Trust Company and Union Savings Bank of Pittsburgh. He also branched into industrial activities: oil, steel, shipbuilding, and construction.

Mellon was a member of the South Fork Fishing and Hunting Club (whose earthen dam failed in May, 1889, causing the Johnstown Flood), and he belonged to the Duquesne Club in Pittsburgh. Along with his closest friends Henry Clay Frick and Philander Knox (also South Fork Fishing and Hunting Club members), Mellon served as a Director of the Pittsburgh National Bank of Commerce.

In 1900, Mellon, then 45 years old, married Nora Mary McMullen (1879–1973), a 20-year-old Englishwoman who was the daughter of Alexander P. McMullen, a major shareholder of the Guinness Brewing Co. They had two children, Ailsa, born in 1901, and Paul, born in 1907. Their marriage ended in a bitter divorce in 1912, which was granted on grounds of Nora Mellon's desertion and her adultery with Capt. George Alfred Curphey, an English soldier, and other men. Mellon did not remarry. In 1923 his former wife married Harvey Arthur Lee, a British-born antiques dealer 14 years her junior. Two years after the Lees' divorce in 1928, Nora Lee resumed the surname Mellon, at the request of her son, Paul.

Mellon also was one-third owner with Frick and Charles W. Mauck in the Old Overholt whiskey distillery in West Overton, Pennsylvania. At the time, Old Overholt was one of the largest and most respected whiskey producers in the country. In 1907, as prohibition became more popular across the country, Frick and Mellon removed their names from the distilling license, although they retained ownership in the company. It is believed that Mellon's connections in the Treasury Department are what allowed the company to secure a medicinal permit during Prohibition. This permit allowed Overholt to sell existing whiskey stocks to druggists for medicinal use.

In 1913, along with his brother, Richard B. Mellon, he established a memorial for his Father, the Mellon Institute of Industrial Research, as a department of the University of Pittsburgh. In 1965, the institute merged with the Carnegie Institute of Technology to form Carnegie Mellon University. Mellon also served as an alumni President and trustee of the University of Pittsburgh, and made several major donations to the school, including the land on which the Cathedral of Learning and Heinz Chapel were constructed. In total it is estimated that Mellon donated over $43 million to the University of Pittsburgh. During World War I he participated in many fundraising activities such as the American Red Cross, the National War Council of the Y.M.C.A., the Executive Committee of the Pennsylvania State Council of National Defense, and the National Research Council of Washington.

Mellon's policies helped reduce the overall public debt (the national debt skyrocketed from $1.5 billion in 1916 to $24 billion in 1919 because of World War I obligations) from $33 billion in 1919 to about $16 billion in 1929, but then the Depression caused it to rise again because of reduced revenue and increasing spending. The top tax rate went to 80% by 1935 and the federal government increased excise taxes in an attempt to make up for the lost revenue.

When Frick died in December 1919, he left his share to Mellon. In 1925, under pressure from prohibitionists, Mellon sold his share of the company to a New York grocer.

The administration of President Franklin D. Roosevelt subjected Mellon to intense investigation of his personal income tax returns. The US Justice Department empaneled a grand jury, which declined to issue an indictment. Roosevelt hated Mellon, as the embodiment of everything he thought was bad about the 1920s; Mellon vehemently denied the charges. A two-year civil action beginning in 1935, dubbed the "Mellon Tax Trial", eventually exonerated Mellon, albeit several months after his death.

Mellon believed that the income tax should remain progressive but with lower rates than those enacted during World War I. He thought that the top income earners would willingly pay their taxes only if rates were 25% or lower. Mellon proposed tax rate cuts, which Congress enacted in the Revenue Acts of 1921, 1924, and 1926. The top marginal tax rate was cut from 73% to 58% in 1922, 50% in 1923, 46% in 1924, 25% in 1925, and 24% in 1929. Rates in lower brackets were also cut substantially, relieving burdens on the middle-class, working-class, and poor households.

Mellon also championed preferential treatment for "earned" income relative to "unearned" income. As he argued in his 1924 book, Taxation: The People's Business

In the alternate history/time travel story "A Slip in Time" by S. M. Stirling, featuring a history in which the First World War was avoided and the Austro-Hungarian Empire survived, Andrew Mellon was the President of the United States in 1926.

In 1929–31, Mellon spent much of the time overseas, negotiating for repayment of European war debts from World War I. In 1928 Mellon issued a denial to allegations made by a speaker for the Democratic Party in North Carolina that he held an interest in a liquor distillery, and that he was the largest distiller in the world prior to Prohibition. In a letter written to the Republican executive committee of Mecklenburg County, North Carolina, Mellon admits having stock in a distillery, but that he had disposed of all interest in the company prior to becoming Secretary of the Treasury.

Mellon was a key negotiator on Germany's war debt, even traveling to Paris in the summer of 1931 to conduct talks on it. He responded to fears that Germany might strike out against its debt plight with: "everything will work out all right".

In January 1932, Rep. Wright Patman (D-TX) (later an F.D.R. New Deal supporter) and others introduced articles of impeachment against Mellon, with hearings before the House Judiciary Committee at the end of that month. After the hearings were over, but before the scheduled vote on whether to report the articles to the full House, Mellon accepted an appointment to the post of Ambassador to the Court of St. James's and resigned as Treasury secretary in February. He served for one year as ambassador and then retired to private life. Representative Louis Thomas McFadden (R-PA) invoked Mellon's appointment while an impeachment was pending in his subsequent attempt to impeach President Hoover.

Mellon died on August 26, 1937, in Southampton, Long Island, New York, and was buried at Trinity Episcopal Church Cemetery, Upperville, Virginia.

The Andrew W. Mellon Foundation, the product of the merger of the Avalon Foundation and the Old Dominion Foundation (set up separately by his children), is named in his honor, as is the 378-foot US Coast Guard Cutter Mellon (WHEC-717). Washington, D.C.'s Andrew W. Mellon Auditorium and Andrew Mellon Building were both named in Mellon's honor as was the National Gallery of Art during its early years. The Andrew W. Mellon Memorial Fountain in Washington D.C., located just north of the National Gallery of Art, at Pennsylvania Avenue, Constitution Avenue, and 6th Street NW was created by Sidney Waugh and dedicated on May 9, 1952, by Harry S. Truman as a memorial to Andrew Mellon.