The present structure was erected subsequent to the extinction of the Jacobite rebellion.
Lord Stormont's family was Jacobite in its politics, and his second son James (c. 1690-1728), being apparently mixed up in some of the plots of the time, joined the court of the exiled Stuarts and in 1721 was created earl of Dunbar by James Edward, the Old Pretender.
He was educated at Loretto, Eton and Oriel College, Oxford, and in 1869 was restored by Act of Parliament to the barony of Balfour of Burleigh, to which he was entitled by his descent from the 5th baron, who was attainted after the Jacobite rebellion of 1715.
He was made acting-lieutenant in the West Indies in the same year, and the rank was confirmed in 1744 During the Jacobite rising of 1745 he commanded the "Baltimore" sloop in the North Sea, and was dangerously wounded in the head while co-operating with a frigate in an engagement with two strong French privateers.
The church of St James, belonging to a small community of Jacobite Christians, and a few pillars and blocks of masonry are the only remains of the former greatness of the town.