Continental Army general William Heath and his officers occupied the house as early as September 5, 1776, holding it for their commander in chief, George Washington. Washington used the mansion as a headquarters for a month after British troops forced his army to retreat to Upper Manhattan. He entered the house on the night of September 14–15, 1776; the exact date and time of his arrival is unclear.

Washington stayed at the mansion for a month with his military secretary and several aides, strategizing for the Battle of Harlem Heights while headquartered there. About 8,000 troops stayed in nearby camps, while some troops set up wooden huts along modern-day Sylvan Terrace. He reportedly observed the Great Fire of 1776 from the mansion's second-floor balcony. The Continental Army remained in "undisturbed possession of their camps" until about October 18, when the Battle of Pell's Point began. Washington retreated around October 21–22 to flee advancing British troops, and Continental Army colonel Robert Magaw was left to guard the house. On November 16, 1776, during the Battle of Fort Washington, Washington's troops tried to reenter the house but were beaten back by British troops.

The British occupied the house from 1776 until the evacuation of New York in 1783. The mansion became the headquarters of British lieutenant-general Henry Clinton until 1777 and Hessian commander Baron Wilhelm von Knyphausen during 1778.