James Sullivan established a legal practice in the Maine district of Massachusetts and entered revolutionary politics through the Provincial Congress in 1774. He served in the Massachusetts legislature and became a justice of the state's highest court in 1776, remaining until 1782. He also participated in the constitutional convention of 1779. Although elected to the Continental Congress in the 1780s, he did not attend.

Sullivan later served as a Suffolk County probate judge and, from 1790 to 1807, as Massachusetts attorney general. Alongside legal work he promoted commercial improvements, including the Charles River bridge, the Middlesex Canal and a water supply drawn from Jamaica Pond. He became the first president of the Massachusetts Historical Society in 1791, helping establish its organization and collecting work. His publications included The History of the District of Maine in 1795 and a later study of land titles, subjects connected to regional legal disputes.

After several unsuccessful campaigns, Sullivan won election as governor in 1807 and was reelected in 1808. A Democratic Republican governing a state with a powerful Federalist opposition, he faced bitter disputes over national policy, including Jefferson's embargo and its effects on Massachusetts trade. Illness limited his activity during his second term, and he died in office in December 1808. His public work extended across the courts, state administration, historical writing and transportation projects.