Farmer was a child prodigy; as a freshman in 1934 at the age of 14, he enrolled at Wiley College, a historically black college where his Father was teaching in Marshall, Texas. He was selected as part of the debate team. Melvin B. Tolson, a professor of English, became his mentor.

Farmer earned a Bachelor of Science at Wiley College in 1938, and a Bachelor of Divinity from Howard University School of Religion in 1941. At Wiley, Farmer became anguished over segregation, recalling particular occasions of racism he had witnessed or suffered in his younger days. During the Second World War, Farmer had official status as a conscientious objector.

In 1942, Farmer co-founded the Committee of Racial Equality in Chicago with George Houser and Bernice Fisher. It was later called the Congress of Racial Equality (CORE), and was dedicated to ending racial segregation in the United States through nonviolence. Farmer served as the national chairman from 1942 to 1944. He was an honorary vice chairman in the Democratic Socialists of America.

Farmer married Winnie Christie in 1945. Winnie became pregnant soon after they were married. Then she found a note from a girl in one of Farmer's coat pockets, an event that served as a catalyst for the end of their marriage. She miscarried, and the couple divorced not long afterwards.

During the 1950s, Farmer served as national secretary of the Student League for Industrial Democracy (SLID), the youth branch of the socialist League for Industrial Democracy. SLID later became Students for a Democratic Society.

A few years later, Farmer married Lula A. Peterson. She had been diagnosed with Hodgkins disease, so the two were told not to have children, because at that time pregnancy was thought to exacerbate cancer. Years later, they sought a second opinion. At that time, Lula was encouraged to try to have children. She had a miscarriage but then successfully had a daughter, Tami Lynn Farmer, born on February 14, 1959. A second daughter, Abbey Farmer, was born in 1962.

By the 1960s, Farmer was known as "one of the Big Four civil rights Leaders in the 1960s, together with King, NAACP chief Roy Wilkins and Urban League head Whitney Young."

They planned for a mixed race and GENDER group to test segregation on interstate buses. The group would be trained intensively in Washington D.C. and embark on May 4, 1961: half by each of the two major carriers, Greyhound Bus Company and Trailways. They would ride through Virginia, the Carolinas, Georgia, Alabama, Mississippi, and finish in New Orleans on May 17. They planned to challenge segregated seating in bus stations and lunch rooms as well. For overnight stops they planned rallies and support from the black community, and scheduled talks at local churches or colleges.

As the Director of CORE, Farmer was considered one of the "Big Six" of the Civil Rights Movement who helped organize the March on Washington for Jobs and Freedom in 1963. (The press also used the term "Big Four", ignoring John Lewis and Dorothy Height.) Growing disenchanted with emerging militancy and black nationalist sentiments in CORE, Farmer resigned as Director in 1966. By that time Congress had passed the Civil Rights Act of 1964, ending segregation, and the Voting Rights Act of 1965, authorizing federal enforcement of registration and elections.

In an interview with Robert Penn Warren in 1964 for the book Who Speaks for the Negro?, Farmer described the founding principles of CORE as follows:

Farmer took a teaching position at Lincoln University, a historically black college (HBCU) near Philadelphia, Pennsylvania. He also lectured around the country. In 1968 Farmer ran for U.S. Congress as a Liberal Party candidate backed by the Republican Party, but lost to Shirley Chisholm.

In 1969 the newly elected Republican President Richard Nixon offered Farmer the position of Assistant Secretary of the Department of Health, Education, and Welfare (now Health and Human Services). The next year, frustrated by the Washington bureaucracy, Farmer resigned from the position.

Farmer retired from politics in 1971 but remained active, lecturing and serving on various boards and committees. He was one of the signers of the Humanist Manifesto II in 1973. In 1975 he co-founded Fund for an Open Society. Its vision is a nation in which people live in stably integrated communities, where political and civic power is shared by people of different races and ethnicities. He led this organization until 1999.

He published his autobiography Lay Bare the Heart in 1985. In 1984, Farmer began teaching at Mary Washington College (now The University of Mary Washington) in Fredericksburg, Virginia.

The following year, civil rights groups, supplemented by hundreds of college students from the North, worked with local Activists in Mississippi on voter registration and education. James Chaney, Andrew Goodman and Michael Schwerner, all of whom Farmer had helped recruit for CORE, disappeared during the Mississippi Freedom Summer. A full-scale FBI investigation aided by other law enforcement, found their murdered corpses buried in an earthen dam. The murders inspired the 1988 feature movie, Mississippi Burning. Years later, recalling the event, Farmer said, "Anyone who said he wasn't afraid during the civil rights movement was either a liar or without imagination. I think we were all scared. I was scared all the time. My hand didn't shake but inside I was shaking."

Several issues of Fellowship magazine of the Fellowship of Reconciliation in 1992 (Spring, Summer and Winter issues) contained discussions by Farmer and George Houser about the founding of CORE. A conference at Bluffton College in Bluffton, Ohio, on October 22, 1992, Erasing the Color Line in the North, explored CORE and its origins. Both Houser and Farmer attended. Academics and the participants unanimously agreed that the founders of CORE were Jim Farmer, George Houser and Bernice Fisher. The conference has been preserved on videotape available from Bluffton College.

Farmer retired from his teaching position in 1998. He died on July 9, 1999 of complications from diabetes in Fredericksburg, VA.