The Leslies were created Earls of Rothes in 1457. William Leslie the 3rd Earl of Rothes was killed at Flodden in 1513. George, the 4th Earl was one of the commissioners for the marriage of Queen Mary to the Dauphin, he was poisoned and died at Dieppe. Queen Mary herself visited Ballinbreich in 1565.
John Leslie, 6th Earl of Rothes led the Covenanters in 1638 and John the 7th Earl was captured at the Battle of Worcester in 1651 and remained in prison until 1658. After the Restoration in 1663 he was made Treasurer of Scotland, then four years later Chancellor and his career culminated in the Dukedom of Rothes in 1680. This title became extinct on his death, although the earldom continued through his daughter.
The Barony of Ballinbreich was sold by John Leslie, 10th Earl of Rothes to Sir Lawrence Dundas for £20 000. Sir Lawrence recouped the cost of his investment by felling the trees surrounding the castle for sale as timber. The Barony of Ballinbreich was later sold to pay for the reconstruction of Leslie House which had been destroyed by a fire in 1763. So the Leslie family lost both their great houses.