Somersets own brother, Thomas Seymour, jealous of the protector, intrigued against the government; he sought to secure the hand of Elizabeth, the favor of Edward VI.
The remedial measures which he favored failed; and the rising of Ket in Norfolk and others less important in nearly all the counties of England, made Somersets position impossible.
The castle passed with the lordship or seigniory of Gower, of which it was the caput, into the hands of the De Braose family in 1203 (by grant from King John) and eventually it came by marriage to the Somersets and is still held by the dukes of Beaufort, whose title of barons de Gower dates from 1506.
Cavendishes and Bentincks were murmuring; Somersets and Wyndhams were hastening to kiss hands.
Afterwards it came into the possession of Edmund Beaufort, duke of Somerset; from the Somersets it passed to Sir George Rodney, and in 1639 came to the Maynard family.