Anne Hill Carter, often called Nancy within her family, was born on March 26, 1773, at Shirley Plantation on the James River in Charles City County, Virginia. She was a daughter of Charles Carter and his second wife, Anne Butler Moore Carter. Charles Carter, heir to one of Virginia’s most powerful planter families, fathered twenty-three children across two marriages; Anne was the tenth of those children and the second child of his second wife. Her ancestry connected her to Robert “King” Carter and the intermarried network of Virginia’s colonial elite.

Shirley was a large plantation household whose wealth depended upon the labor and sale of enslaved people. Anne grew up amid numerous siblings and half-siblings, household servants, plantation workers, visiting relatives, and the elaborate obligations of elite hospitality. Little direct evidence survives about her childhood, and the National Park Service explicitly notes the scarcity of details. Later family descriptions portray her as musically accomplished, a strong chess player, and possessed of a reserved humor, suggesting the domestic education in music, needlework, reading, religion, household management, and social deportment customary for a planter’s daughter.

Her youth coincided with the Revolution. Although Shirley lay away from the principal northern battlefields, war disrupted Chesapeake trade, brought military movements into Virginia, and shaped the reputations of the men around her. Anne’s family retained immense property and social standing through the conflict.

On June 18, 1793, at twenty, Anne married the widowed Revolutionary cavalry commander Henry “Light-Horse Harry” Lee in Shirley’s great house. Lee was then governor of Virginia and already had children by his first wife. Charles Carter placed Anne’s inheritance in trust so that Lee and his creditors could not control it, a precaution that proved important given Lee’s financial instability.