Weill, shortly after graduating from Cornell University, got his first job on Wall Street in 1955 – as a Runner for Bear Stearns. In 1956, he became a licensed broker at Bear Stearns. Rather than making phone calls or personal visits to solicit clients, Weill found he was far more comfortable sitting at his desk, poring through companies' financial statements and disclosures made to the U.S. Securities and Exchange Commission. For weeks his only client was his mother, Etta, until Joan persuaded an ex-boyfriend to open a brokerage account.

While working at Bear Stearns, Weill was a neighbor of Arthur L. Carter who was working at Lehman Brothers. Together with Roger Berlind and Peter Potoma, they formed Carter, Berlind, Potoma & Weill in May 1960. In 1962 the firm became Carter, Berlind & Weill after the New York Stock Exchange brought disciplinary proceedings against Potoma.

In 1968, with the departure of Arthur Carter, the firm was renamed Cogan, Berlind, Weill & Levitt (Marshall Cogan, Arthur Levitt), or CBWL jokingly referred to on Wall Street as "Corned Beef With Lettuce". Weill served as the firm's Chairman from 1965 to 1984, a period in which it completed over 15 acquisitions to become the country’s second largest securities brokerage firm. The company became CBWL-Hayden, Stone, Inc. in 1970; Hayden Stone, Inc. in 1972; Shearson Hayden Stone in 1974, when it merged with Shearson Hammill & Co.; and Shearson Loeb Rhoades in 1979, when it merged with Loeb, Rhoades, Hornblower & Co.

Long a proponent of education, Weill instituted a joint program with the New York City Board of Education in 1980 that created the Academy of Finance, which trains high school students for careers in financial services. He serves as Founder and Chairman of the National Academy Foundation (NAF), which oversees more than 60,000 students in 500 career-themed academies of Finance, hospitality and tourism, information Technology and engineering, in 41 states, as well as the District of Columbia. Ninety-seven percent of NAF’s students graduate, with eighty percent going on to post-secondary education – often as the first in their families to attend college. New York Governor Andrew Cuomo appointed Weill as a member of his New York Education Reform Commission. Weill has received honorary degrees from Howard University, Hofstra University, University of New Haven, The New School, and Sonoma State University.

In 1981, Weill sold Shearson Loeb Rhoades to American Express for about $930 million in stock. (Sources differ on the precise figure.) In 1982, he founded the National Academy Foundation with the Academy of Finance to educate high school students. Weill began serving as President of American Express Co. in 1983 and as chairman and CEO of American Express's insurance subsidiary, Fireman's Fund Insurance Company, in 1984. Weill was succeeded by his protégé, Peter A. Cohen, who became the youngest head of a Wall Street firm. While at American Express, Weill began grooming his newest protégé, Jamie Dimon, the Future CEO of J P Morgan Chase.

Weill resigned from American Express in August 1985 at age 52. After an attempt to become the CEO of BankAmerica Corp. (and "take over" Merrill Lynch, according to a Jamie Dimon interview in 2002), he persuaded Minneapolis-based Control Data Corporation to spin off a troubled subsidiary, Commercial Credit, a consumer Finance company. In 1986, with $7 million of his own money invested in the company, Weill took over as CEO of Commercial Credit. After a period of layoffs and reorganization, the company completed a successful IPO.

He is also the Chairman of the Board of Carnegie Hall and is an avid champion of classical music in the United States. Since 1986, one of the three performance halls in Carnegie Hall has been named after Weill and his wife, Joan and Sanford I. Weill Recital Hall. The 1997 recipient of the New York State Governor’s Art Award, Weill has been Chairman of the Board of Trustees of Carnegie Hall since 1991. For Weill's 70th birthday, Carnegie Hall raised a record $60 million in one evening through a generous $30 million match by Weill and his wife for the Weill Music Institute, which established broad-reaching music education programs. Weill is also chairman of the Green Music Center Board of Advisors at Sonoma State University and a Director of the Lang Lang International Music Foundation.

In 1987, he acquired Gulf Insurance. The next year, he paid $1.5 billion for Primerica, the parent company of Smith Barney and the A. L. Williams insurance company. In 1989 he acquired Drexel Burnham Lambert's Retail brokerage outlets. In 1992, he paid $722 million to buy a 27 percent share of Travelers Insurance, which had gotten into trouble because of bad real estate Investments.

In 1993 he reacquired his old Shearson brokerage (now Shearson Lehman) from American Express for $1.2 billion. By the end of the year, he had completely taken over Travelers Corp in a $4 billion stock deal and officially began calling his corporation Travelers Group Inc. In 1996 he added to his holdings, at a cost of $4 billion, the property and casualty operations of Aetna Life & Casualty. In September 1997 Weill acquired Salomon Inc., the parent company of Salomon Brothers Inc. for over $9 billion in stock.

In 1998, Weill was the recipient of FinancialWorld Magazine's CEO of the Year Award, and received the same honor from ChiefExecutive Magazine in 2002.

Joan and Sanford Weill have been co-chairs, of the annual "Louis Marshall Award Dinner", for most of the past decade (2000–2010).

In 2001, Weill became a Class A Director of the Federal Reserve Bank of New York. Class A Directors are those elected by Federal Reserve member banks.

In 2002, the Joan Weill Adirondack Library and Joan Weill Student Center were dedicated at Paul Smith's College.

In May 2003, he received the Baruch Medal for Business and Civic Leadership, presented by Baruch College for his work in public education and his accomplishments in Business.

In 2004, Citigroup agreed to pay $2.65 billion to settle a class-action suit brought by Investors over its role in the WorldCom scandal. Citigroup's Salomon Smith Barney had issued optimistic research reports on WorldCom and, at the same time, helped it raise money by selling its securities.

In September 2006, Joan and Sanford Weill Hall was dedicated at the University of Michigan. The building is home to the Gerald R. Ford School of Public Policy. Weill donated $5 million towards the construction of the building and an additional $3 million to endow the position of the dean of the school.

The Weills are recipients of the 2009 Carnegie Medal of Philanthropy Award. Sanford I. Weill was the 2015 recipient of the Carnegie Hall Medal of Excellence.

In 2010, the Weills bought a 362-acre estate in Sonoma County, California. In March 2011, the Weills announced a $12 million gift to Sonoma State University, providing the funds to complete the Donald and Maureen Green Music Center concert hall for a fall 2012 opening. The facility, inspired by Seiji Ozawa Hall at Tanglewood, has been named the Joan and Sanford I. Weill Hall. "We love to be involved in the communities where we spend time," Sandy Weill commented to an interviewer.

In 2011, Rambam Medical Center in Haifa, Israel and the American Friends of the Rambam Medical Center announced that Joan and Weill and the Weill Family Foundation made a donation of $10 million. In addition, the money was intended to support the Israeli-Palestinian Friendship Center and enable the hospital to better serve patients from Gaza and the West Bank by making residential hostel facilities available to their families while providing advanced medical training to Palestinian residents, fellows, and nursing staff.

In 2012, Weill was elected a member of the American Academy of Arts and Sciences.

In September 2013, Weill and his wife wrote an op-ed for CNBC stating that philanthropy goes beyond just money. "For us, philanthropy is much more than just writing a check. It's donating your time, Energy, experience, and intellect to the causes and organizations you are passionate about."

In 2015, Joan Weill offered an additional $20 million to Paul Smith's College, but only if it changed its name to Joan Weill-Paul Smith's College, a change that would have violated the terms of the devise of the school's real property, which required that the school be "forever known" as Paul Smith's College of Arts and Sciences. Paul Smith's applied to the New York Supreme Court for a release from the naming clause of the donor's will, arguing that its continued financial survival depended on receipt of Mrs. Weill's $20 million gift. Notwithstanding that argument, there was considerable opposition to the requested name change from alumni and others. The college was originally funded by the will of Paul Smith's son, Phelps Smith, who specified that the institution should be “forever known” by his father’s name. In light of the potential donation, the college petitioned to be released from the will’s conditions, but their appeal was denied by Judge Ellis.

In 2016 Sandy and Joan Weill announced a $185 million contribution to the University of California, San Francisco (UCSF) for a new neuroscience institute. The gift is the largest donation in the school's history. The Weill Institute for Neurosciences will be housed in a $316 million facility to be built at UCSF's Mission Bay campus. The Weills hope the institute will develop more effective treatments for such diseases as Alzheimer's, Parkinson's, multiple sclerosis, sleep disorders, autism, and other brain-related ailments.