Kroeber was born in Hoboken, New Jersey, to upper middle-class parents: Florence Kroeber, who immigrated at the age of 10 to the United States with his parents and family from Germany, and Johanna Muller, who was of German descent. His family moved into New York when Alfred was quite young, and he was tutored and attended private schools there. He had three younger siblings and all had scholarly interests. The family was bilingual, speaking German at home, and Kroeber also began to study Latin and Greek in school, beginning a lifelong interest in languages. He attended Columbia College at the age of 16, joining the Philolexian Society and earning an A.B. in English in 1896 and an M.A. in Romantic drama in 1897. Changing fields to the new one of anthropology, he received his Ph.D. under Franz Boas at Columbia University in 1901, basing his 28-page dissertation on decorative symbolism on his field work among the Arapaho. It was the first doctorate in anthropology awarded by Columbia.

Kroeber married Henrietta Rothschild in 1906. She contracted tuberculosis (TB) and died in 1913, after several years of illness.

Kroeber's textbook, Anthropology (1923, 1948), was widely used for many years. In the late 1940s, it was one of ten books required as reading for all students during their first year at Columbia University. His book, Configurations of Cultural Growth (1944), had a lasting impact on social scientific research on genius and greatness; Kroeber believed that genius arose out of culture at particular times, rather than holding to "the great man" theory.

Kroeber and his students did important work collecting cultural data on western tribes of Native Americans. The work done in preserving information about California tribes appeared in Handbook of the Indians of California (1925). In that book, Kroeber first described a pattern in California groups where a social unit was smaller and less hierarchically organized than a tribe, which was elaborated upon in The Patwin and their Neighbors in which Kroeber first coined the term "tribelet" to describe this level of organization. Kroeber is credited with developing the concepts of culture area, cultural configuration (Cultural and Natural Areas of Native North America, 1939), and cultural fatigue (Anthropology, 1963).

In 1926 he married again, to Theodora Kracaw Brown, a widow whom he met as a student in one of his graduate seminars. They had two children: Karl Kroeber, a literary critic, and the science fiction Writer Ursula Kroeber Le Guin. In addition, Alfred adopted Theodora's sons by her first marriage, Ted and Clifton Brown, who both took his surname.

Kroeber spent most of his career in California, primarily at the University of California, Berkeley. He was both a Professor of Anthropology and the Director of what was then the University of California Museum of Anthropology (now the Phoebe A. Hearst Museum of Anthropology). The anthropology department's headquarters building at the University of California is named Kroeber Hall in his honor. He was associated with Berkeley until his retirement in 1946.

He is noted for working with Ishi, who was claimed to be the last California Yahi Indian. (Ishi may have been of mixed ethnic heritage, with a Father from the Wintu, Maidu or Nomlaki tribes.) His second wife, Theodora Kracaw Kroeber, wrote a well-known biography of Ishi, Ishi in Two Worlds. Kroeber's relationship with Ishi was the subject of a film, The Last of His Tribe (1992), starring Jon Voight as Kroeber and Graham Greene as Ishi.

In 2003, Clifton and Karl Kroeber published a book of essays on Ishi's story, which they co-edited, called, Ishi in Three Centuries. This is the first scholarly book on Ishi to contain essays by Native American Writers and academics.