In the early 1760s, Madison Sr. built a new house half a mile away, which structure forms the heart of the main house at Montpelier today. Built around 1764, it has two stories of brick laid in a Flemish bond pattern, and a low, hipped roof with chimney stacks at both ends. Phase 2 (1797–1800) of construction began in 1797, after son James returned to Montpelier with his new wife Dolley Madison. He was then 39 and she was a young widow with a child. At this time Madison added a thirty-foot extension and a Tuscan portico to the house. Madison's widowed mother, Nelly, still resided in the house following the death of her husband, James Sr., in 1801.

In the last period of construction, Phase 3 (1809–1812), Madison had a large drawing room added, as well as one-story wings at each end of the house; these provided space for the separate household of the newlyweds James and Dolley Madison. After his second term as president, in 1817 Madison retired there full-time with his wife Dolley. James Madison died in 1836 and is buried in the family cemetery at Montpelier. His widow Dolley Madison moved back to Washington, D.C., in 1837 after his death. In 1844 she sold the plantation to Henry W. Moncure. After Dolley Madison died in 1849, she was buried in Washington, D.C., and later re-interred at Montpelier near her husband James.

This was also a working plantation whose appearance and economy depended on the labour of enslaved men, women and children. Domestic work, farming, construction and skilled trades supported the family’s wealth and Madison’s public career. Their stories cannot be reduced to the architectural phases of the main house. Archaeological investigation and descendant partnerships have recovered evidence of households and lives that older house-centred interpretation often neglected.

The property passed through later ownership and land-use changes after the Madisons. Modern restoration used documentary, architectural and archaeological evidence to interpret the house and grounds, while exhibitions address the contradiction between Madison’s constitutional ideals and slavery. Different visitor experiences cover the home, gardens, trails and the enslaved community. A visit benefits from understanding the estate as a cultural landscape rather than a single preserved presidential room.

Montpelier’s conservation is an ongoing process. Historic buildings require careful repair, but the setting also holds archaeological traces and burial places whose significance may not be visible from a tour path. The site’s public interpretation has expanded through research and collaboration with descendants. Visitors should distinguish the documented eighteenth- and nineteenth-century house from later reconstructions and exhibits that help explain the people who lived and worked here.