In 1951, Diller married Helen Samuels (born 1928), the daughter of Jewish immigrants from Poland, and fellow University of California, Berkeley graduate who he met in college. Her Father was a clothing salesman and shop owner. They have three children: Brad Diller, Ron Diller, and Jackie Safier, President of Prometheus. The Dillers live in Woodside, California.

The son of Jewish immigrants from Austria who had settled in San Francisco, Diller graduated from the University of California, Berkeley and then graduated with a J.D. from the Hastings College of the Law. After law school, Diller began acquiring and developing real estate in the mid-1960s. In 1965, he founded the Prometheus Real Estate Group. Prometheus grew to become the San Francisco Bay area's largest closely held owner of apartments with over 11,000 owned units and is valued at $2.3 billion. Prometheus is owned by Diller and his wife through a revocable trust.

In 2002, the Helen Diller Family Program in Jewish Studies donated $5 million to establish an endowment for a visiting Israeli scholar at the Center for Middle Eastern Studies at UC Berkeley. In 2003, the Dillers donated $35 million to fund The Helen Diller Family Comprehensive Care Center at the UC San Francisco complex on the biomedical campus of Mission Bay. Since 1999, the Dillers have donated $200 million via their family foundation named after his wife, The Helen Diller Family Foundation. Other major gifts have funded Jewish studies programs at UC Berkeley and UC Santa Cruz, the renovation of the Julius Kahn playground at the Presidio, and a new preschool in the San Francisco Jewish Community Center. Since 2007, the Diller Teen Tikkun Olam Awards have recognized Californian Jewish teens who are outstanding role Models in their communities.