Armistead, an enslaved man, was owned by william Armistead of Virginia. Most sources indicate that James Armistead was born in 1748 in New Kent County, Virginia, though others put his birth around 1760 in Elizabeth City, Virginia.

After Arnold departed north in the spring of 1781, James went to the camp of Lord Charles Cornwallis and continued his work. He moved frequently between British camps where the officers would speak openly about their strategies in front of him. Armistead documented this information in written reports, which he then delivered to other American spies. In this way, he relayed much information about the British's plans for troop deployment and their arms. The intelligence reports from his espionage were instrumental in helping defeat the British during the Battle of Yorktown.

Although Virginia passed a manumission act in 1782 allowing for the freedom of any slave by his or her owner, James Armistead remained the property of william Armistead. This was because a 1783 law specifically only freed slaves whose owners had used them as substitutes for army Service in exchange for their liberty. This was not the case for Armistead due to being a spy and not a soldier. However, in 1786, with the support of william Armistead – then a member of the House of Delegates – and a 1784 testimonial of his Service from the Marquis de Lafayette, James petitioned the Virginia Assembly for his freedom. On January 9, 1787, the Assembly granted the petition. At that time, he chose to add "Lafayette" to his name to honor the general.

Armistead continued to live in New Kent County with his new wife, one son and several other children and became a rich farmer . In 1818, he applied to the state legislature for financial aid; he was granted $60 for present relief and $40 annual pension for his services in the Revolutionary War.

In 1824, the Marquis de Lafayette returned to the United States at the invitation of President James Monroe. He made a tour of all 24 states, in which he was met by huge crowds and everywhere feted as a hero. While in Virginia, where he visited Washington's grave and gave a speech to the House of Delegates, he abruptly had his carriage stopped when he saw Armistead in the crowd and rushed to embrace him. At around this time, he also wrote a testimonial on Armistead's behalf.