Kohlberg joined Bear Stearns in 1955 where he would go on to manage the corporate Finance department. Working for Bear Stearns in the late 1960s and early 1970s, Kohlberg, alongside Bear Stearns executives began advising a series of what they described as "bootstrap" Investments. Their acquisition of Orkin Exterminating Company in 1964 is considered to have been among the first significant leveraged buyout transactions. In the following years the three Bear Stearns Bankers would complete a series of buyouts including Stern Metals (1965), Incom (a division of Rockwood International, 1971), Cobblers Industries (1971), and Boren Clay (1973) as well as Thompson Wire, Eagle Motors and Barrows through their investment in Stern Metals. Although they had a number of highly successful Investments, the $27 million investment in Cobblers ended in bankruptcy. Kravis and his associates created a series of limited partnerships to acquire these various corporations, ones they judged were performing well below their sales and profit potential or where there were untapped financial assets that could be monetized. In most cases, Kohlberg Kravis Roberts & Co put up ten percent of the acquisition price from its own funds and borrowed the rest from Investors by issuing high-yield bonds.

By 1976 tensions had built up between Bear Stearns and the trio of Kohlberg, Kravis and Roberts leading to their departure and the formation of Kohlberg Kravis Roberts in that year. Most notably, Bear Stearns executive Cy Lewis had rejected repeated proposals to form a dedicated investment fund within Bear Stearns and Lewis took exception to the amount of time spent on outside activities. Early Investors in KKR included the Hillman Family Group of Henry Hillman and the Hillman Company. By 1978, with the revision of the ERISA regulations, the nascent KKR was successful in raising its first institutional fund with approximately $30 million of investor commitments.

Kohlberg was raised in a Jewish family graduating from New Rochelle High School in New Rochelle, New York. Kohlberg served in the United States Navy during World War II and went on to college and graduate school on the GI Bill. He earned an undergraduate degree from Swarthmore College. He later received MBA degree from Harvard Business School and Bachelor of Laws in Columbia Law School. In 1986, he founded the Philip Evans Scholarship Foundation at Swarthmore.

In 1987 Kohlberg resigned from KKR over differences in strategy and Henry Kravis and George Roberts assumed full leadership of the firm. Kohlberg did not favor the large buyouts (which would likely have included the 1989 takeover of RJR Nabisco) or hostile takeovers. Instead, Kohlberg chose to return to his roots, acquiring smaller, middle-market companies and, in 1987, founded a new private equity firm Kohlberg & Company. As of the end of 2007 Kohlberg & Company had raised six private equity funds since its inception, with approximately $3.7 billion of investor commitments. Additionally, Kohlberg also operated a series of debt investment funds under the banner of Katonah Debt Advisors, as well as a publicly traded investment vehicle Kohlberg Capital (NASDAQ:KCAP). Kohlberg retired from Kohlberg & Company in 1994.

He was listed on the 2008 Forbes List of billionaires as the 785th richest person in the world with an estimated net worth of $1.5 billion.

Kohlberg died of cancer on July 30, 2015 on Martha’s Vineyard in Massachusetts, twenty days after his 90th birthday.