In the early 1760s, Madison Sr. built a new house half a mile away, which structure forms the heart of the main house at Montpelier today. Built around 1764, it has two stories of brick laid in a Flemish bond pattern, and a low, hipped roof with chimney stacks at both ends.

In the early 1760s, Madison Sr. built a new house half a mile away, which structure forms the heart of the main house at Montpelier today. Built around 1764, it has two stories of brick laid in a Flemish bond pattern, and a low, hipped roof with chimney stacks at both ends. Phase 2 (1797–1800) of construction began in 1797, after son James returned to Montpelier with his new wife Dolley Madison. He was then 39 and she was a young widow with a child. At this time Madison added a thirty-foot extension and a Tuscan portico to the house. Madison's widowed mother, Nelly, still resided in the house following the death of her husband, James Sr., in 1801.

In the last period of construction, Phase 3 (1809–1812), Madison had a large drawing room added, as well as one-story wings at each end of the house; these provided space for the separate household of the newlyweds James and Dolley Madison. After his second term as president, in 1817 Madison retired there full-time with his wife Dolley. James Madison died in 1836 and is buried in the family cemetery at Montpelier. His widow Dolley Madison moved back to Washington, D.C., in 1837 after his death. In 1844 she sold the plantation to Henry W. Moncure. After Dolley Madison died in 1849, she was buried in Washington, D.C., and later re-interred at Montpelier near her husband James.