Born into slavery in Louisville, Kentucky in 1842, Allensworth was the youngest of thirteen children of Phyllis (c. 1782 - 1878) and Levi Allensworth. Over the years, their family was scattered: his sister Lila escaped with her intended husband to Canada by the Underground Railroad; and the older boys william, George, Frank, Levi and Major were sold downriver to plantations in the Deep South, which continued to buy enslaved workers from the Upper South to develop the cotton industry. Mary Jane was his only sibling who grew up in Kentucky and married there; she purchased her freedom in 1849, gaining stability.

His mother was held by A.P. and Bett Starbird. The mistress assigned Allen as a young slave to her son Thomas. When the Starbird boy started school, Allen began to learn from him, although it was illegal. After his father died when Allen was young, his mother chose to be sold as a cook to a neighbor, the attorney Nat Wolfe. When the Starbirds found Allen was learning to read, they separated him from their son and placed him with another family, the Talbots. Mrs. Talbot, a Quaker, was kind to Allen and continued to teach him to read and write; she also took him to a Sunday school for slave children. When Bett Starbird discovered this, she took Allen back. In 1854 she made arrangements with her husband's partner John Smith to send the boy to his brother Pat's plantation down the Mississippi River in Henderson, Kentucky, to put an end to his learning. On the steamboat, the boy was placed in the care of a slave steward rather than being chained with other slaves below deck. They were being transported for sale to downriver markets.

Hebe Smith, Allen's new mistress, assigned him to be a houseboy; she prohibited him from continuing his studies, and whipped him for trying to do so. Also working in the household was a white orphan boy Eddie; the two boys became friends and helped each other. Suffering on the farm from a cruel overseer, in 1855 at age 13, Allen planned to escape to Canada. He spent two weeks hiding at a neighboring farm before returning to the Smiths for punishment. Later he ran away again. The Smiths and Starbirds agreed to sell him on the auction block in Henderson.

In early 1861 the Civil War loomed, but horse racing continued. Scruggs took Allen and his horses upriver for the fall meet in Louisville. Allensworth hoped to see his mother Phyllis again, as he had learned that her last master, a Rev. Bayliss, had freed her after she cared for his dying wife. He found that she had recently gone to New Orleans with a Union man to look for her sons. (She found Major in prison.) Waiting for her return, Allensworth was reunited with his sister Mary Jane, who had married and had a son. She had purchased her freedom in 1849. When Phyllis Starbird returned to Louisville, she and Allen were reunited.

While working nearby on a farm where Scruggs' deputy had placed him, Allensworth met Soldiers from the 44th Illinois Volunteer Infantry Regiment, a Union unit encamped near Louisville. When he told them of wanting freedom, they invited him to join the Hospital Corps. In disguise, he marched with the unit past his old master through Louisville and off to war. After serving as a civilian nursing aide for some time, he was invited to accompany Dr. A. J. Gordon, one of the Surgeons, to his home in Georgetown, Ohio. There Allensworth dined with Gordon's family, was given a room of his own, and felt he first walked as a free man. With the war continuing, in 1863 Allensworth enlisted in the US Navy, where he earned his first pay as a free man. He was soon promoted to Captain's steward and clerk, and served on the gunboats Queen City and Tawah for two years.

Allensworth first returned to Kentucky to work and study. In 1868 he joined his brother william in St. Louis, where they operated two restaurants. Within a short time, they received a favorable offer and sold them out; Allensworth returned to Louisville. He worked while putting himself through the Ely Normal School, one of several new schools in the South established by the American Missionary Association. During Reconstruction, Allensworth taught at schools for freedmen and their children operated by the Freedmen's Bureau. Inspired by his own teaching, he began attending courses at the Nashville Institute, later known as the Roger Williams University, but did not graduate. The school later gave him an honorary Master of Arts.

Allensworth became involved with the Baptist Church in Louisville and attended the Fifth Street Baptist Church led by Henry Adams. He was ordained in 1871 by the Baptists as a preacher. In the 1870s, Allensworth went to Tennessee to study theology. During this time he also served as a preacher in Franklin, Tennessee, south of Nashville.

In 1875, Allensworth started working as a Teacher in Georgetown, Kentucky. He also served as the financial agent of the General Association of the Colored Baptists in Kentucky. They had joined together to support the founding of a religious school for black teachers and Preachers. Allensworth was among the founders of The State University, helped guarantee the salary of the President in the early years, and served on the Board of Trustees.

In 1877 he married Josephine Leavell (1855–1938), also born in Kentucky; they had met while studying at Roger Williams University in Nashville, Tennessee. She was an accomplished Pianist, organist and music Teacher. They had two daughters together, Eva and Nella.

The year of his marriage, Allensworth invited his mother to live with him and Josephine. They had several months together before she died in 1878 at the age of 96.

With his leadership positions and public speaking, Allensworth became increasingly interested in politics. In 1880 and 1884, he was selected as Kentucky's only black delegate to the Republican National Conventions.

In 1886, when he was 44, Allensworth gained support by both southern and northern politicians for appointment as a chaplain in the US Army; his appointment was confirmed by the Senate, as necessary at the time, and approved by the President. He was one of the few black chaplains in the US Army and was assigned to the 24th Infantry Regiment, known as the Buffalo Soldiers. His family accompanied him on assignments in the West, ranging from Fort Bayard, New Mexico Territory to Fort Supply, Indian Territory, and Fort Harrison, near Helena, Montana. His wife played organ in the fort chapels.

By the time of his retirement in 1906, Allensworth had been promoted to the rank of lieutenant colonel, the first African American to gain that rank.

In 1908, he founded Allensworth in Tulare county, about thirty miles north of Bakersfield, in the heart of the San Joaquin Valley. The black settlers of Allensworth built homes, laid out streets, and put up public buildings. They established a church, and organized an orchestra, a glee club, and a brass band.

Allen Allensworth died at the age of 72, on September 14, 1914. He was killed by a motorcyclist in Monrovia, California.