Castle Stalker stands on a tidal islet in Loch Laich, where its name recalls the Gaelic word Stalcaire, meaning hunter or falconer. A small fortified building associated with the MacDougall lords of Lorn may have occupied the site by about 1320. The MacDougalls had lost much of their power after opposing Robert the Bruce, and the lordship later passed to the Stewarts. The present tower house was probably built around the 1440s by Sir John Stewart of Lorn, who used the island position to command movement along the coast and to display restored family authority.

The castle became closely connected with the Stewarts of Appin. In 1463 Sir John was attacked outside Dunstaffnage church while preparing to marry the mother of his son Dugald. He survived long enough for the marriage to legitimize Dugald, who became the first Stewart chief of Appin. Five years later, Stewart and MacLaren forces defeated their MacDougall rivals at the Battle of Stalc opposite the castle. The tower thus stood within a landscape of kinship, succession, and violent competition rather than serving as a large conventional fortress.

Royal and clan politics continued to shape the site. James IV is traditionally said to have used Castle Stalker as a base for hunting and hawking, and improvements may have been made during his reign. The Stewarts supported the crown at Flodden in 1513 and remained entangled in feuds with neighboring families. Around 1620, according to a long-preserved family account, the seventh Stewart chief lost or transferred the castle to the Campbells of Airds through a drunken wager involving an eight-oared boat. Whatever the precise circumstances, Campbell possession marked a major change in local power.

The Stewarts briefly recovered the castle during the Jacobite rising of 1689, but surrendered it after a Campbell siege in 1690. In 1745, Castle Stalker held a government garrison while the Appin Stewarts supported Charles Edward Stuart. Jacobite cannon were too light to breach the walls, and the tower served government shipping between Inveraray and Fort William. After Culloden it became a place where local clansmen surrendered weapons and prisoners were temporarily held. The Campbells occupied it until about 1800, after which it was used as a storehouse; the roof was gone by the mid-nineteenth century.

Charles Stewart of Achara bought the ruin in 1908 and carried out basic preservation. A more ambitious rescue began after Lieutenant Colonel D. R. Stewart Allward acquired it in 1965. Working with family and friends over roughly a decade, he rebuilt the roof and floors, repaired masonry, and made the tower habitable again. The privately owned castle later gained a worldwide popular association through its appearance in Monty Python and the Holy Grail. Its survival reflects both medieval island strategy and modern hands-on restoration, with the compact tower still dominating the seascape that shaped its history.