Casa Blanca was begun in 1521 when Spanish colonists transferred Puerto Rico's principal settlement from Caparra to the defensible islet now called Old San Juan. It was intended for Juan Ponce de León, the island's first Spanish governor, but he died from wounds received during an expedition to Florida before occupying it. The first structure was probably timber and soon gave way to masonry built under his relative García Troche. Overlooking the harbor, the house served simultaneously as residence, storehouse, administrative center, and early fortification in an unsettled colonial landscape.

The Ponce de León family lived at Casa Blanca for generations, linking domestic authority to conquest and municipal government. Their household depended on servants, laborers, artisans, and enslaved people whose lives appear unevenly in surviving records. Spanish settlement dispossessed Taíno communities through warfare, forced labor, disease, missionization, and imported legal systems. The fortified character of the house reflected fear of Indigenous resistance and foreign attack, but it also represented the new regime imposed on the island. Later stone walls and courtyards accumulated around an irregular plan rather than following one preserved sixteenth-century design.

As larger defenses arose at La Fortaleza, San Felipe del Morro, and San Cristóbal, Casa Blanca lost its role as the governor's principal residence. The Ponce de León descendants nevertheless remained connected to the property until the late eighteenth century. In 1779 it passed to the Spanish Crown and housed military engineers responsible for strengthening San Juan's fortifications. Rooms became quarters, offices, storage spaces, and training areas. Gardens and terraces continued to exploit the steep site and harbor view, while repairs reflected changing military needs and Caribbean building practices.

The United States took Puerto Rico in 1898 during the Spanish-American War, and Casa Blanca transferred to the U.S. Army. Commanders and their families occupied the old compound, adding utilities, partitions, balconies, landscaping, and other alterations suited to modern residence. This use preserved the property from some forms of redevelopment but also obscured earlier fabric. Works Progress Administration funding supported repairs in the late 1930s, part of a broader American program that combined employment, heritage documentation, military administration, and a selective interest in Spanish colonial architecture.

After the Second World War the site's future became uncertain. Puerto Rican preservation leader Ricardo Alegría opposed proposals to return Casa Blanca to official residential use and argued for a museum interpreting early colonial history. The property passed to the government of Puerto Rico in the 1960s and was restored by the Institute of Puerto Rican Culture. The museum opened in 1974 with period objects, maps, and reconstructed rooms. Gardens with channels, fountains, and shaded terraces became as important to visitors as the heavily modified house itself.

Casa Blanca is now valued as one of San Juan's oldest architectural complexes, although little can be assigned confidently to a single year. Its restored appearance joins sixteenth-century origins, later Spanish military use, American occupation, and twentieth-century heritage choices. Responsible interpretation must move beyond the unrealized home of a famous conquistador to address Taíno dispossession, coerced labor, enslaved households, engineers, soldiers, servants, and preservationists. The house's importance lies precisely in this accumulation: five centuries of rulers repeatedly adapted a strategic residence while Puerto Rico's sovereignty and public memory changed around it.