Fitzhugh was a friend and colleague of George Washington, whose family's farm was just down the Rappahannock River from Chatham. Washington's diaries note that he was a frequent guest at Chatham. He and Fitzhugh had served together in the House of Burgessesbefore the American Revolution and shared a love of farming and horses. Fitzhugh's daughter, Molly, married the first president's step-grandson, George Washington Parke Custis, and became a leading abolitionist together with her friend Ann Randolph Meade Page. Their daughter Mary Anna, born at Ann Page's estate, later wed the future Confederate General Robert E. Lee, who freed the Custis slaves as the executor after his in-laws' deaths.

The 1,280-acre plantation included an orchard, mill, and a race track where Fitzhugh's horses vied with those of other planters for prize money. Fitzhugh named the mansion after the British parliamentarian William Pitt, 1st Earl of Chatham, who championed many of the opinions held by American colonists before the Revolutionary War.[2] Flanking the main house were dozens of supporting structures: slave quarters, a dairy, ice house, barns, and stables, plus fish traps installed on the river.

Fitzhugh sold the Chatham plantation to Major Churchill Jones, who had served under Col. William Washington and Gen. "Light Horse" Harry Lee. The elderly Fitzhugh then moved to a city house in Alexandria, Virginia.