The 1st earl of Hertford destroyed it in 1544, and after it was rebuilt the Reformers demolished it again, some of its stones being used in erecting the tolbooth.
Henry Seymour Conway's elder brother, Francis, 2nd Baron Conway, was created marquess of Hertford in 1793; his mother was a sister of Sir Robert Walpole's wife, and he was therefore first cousin to Horace Walpole, with whom he was on terms of intimate friendship throughout his life.
He graduated in 1877, with a first class in classics, having won the Hertford, Craven, Eldon and Derby scholarships, and was elected to a fellowship of New College.
It was sacked and burnt by the English under the earl of Hertford in 1544, and again in 1547.
The Wallace Art Collection, Hertford House, Manchester Square, was bequeathed by Sir Richard Wallace to the nation on the death of his wife in 1897.