Mary, Queen of Scots visited the castle with her third husband, the earl of Bothwell and in 1605 the Drummonds were created Earls of Perth, by her son James VI. James third Earl of Drummond ought with James Graham, Marquis of Montrose and was captured after the Battle of Philiphaugh in 1645, this led the keep to be destroyed by Cromwell's forces in 1650.
Recovery came with James Drummond, 4th Earl of Perth who succeeded in 1675, he built the mansion house beyond the keep forming the inner courtyard. Following James VII flight into exile, he was imprisoned for four years in Stirling Castle. In 1693 he was freed and followed the king into exile. For his loyalty he was created Duke of Perth (in the Jacobite Peerage) His son, the 2nd Duke of Perth joined the Earl of Mar in the Rising of 1715. The castle was occupied by government forces who refortified the keep.
The 3rd Duke after completing his education in France joined his mother at Drummond Castle. However he remained close to the Jacobite cause and joined Prince Charles, leading the left wing at the Battle of Culloden in 1746. He was wounded during the fighting but managed to escape on board a ship bound for France where he died during the passage. His brother John, the 4th Duke managed to escape to France where he died in 1747.
The Drummond Estates were attainted in 1746 and managed over the next 39 years by the Crown's Commissioners. A special Act of Parliament in 1785 restored the Perth Estates to Capt. James Drummond who was created Baron Perth. His daughter Clementina Drummond succeeded her father Lord Perth in 1800. In 1807 she married Peter Robert Burrell who succeeded his mother as Baron Willoughby de Eresby in 1827. Although Sir Charles Barry drew up pland for an extensive rebuilding of the castle but in the end the alterations were more modest. The keep was restored in 1822 and made good for Queen Victoria's visit in 1842. In 1878, Clementina's daughter, Baroness Willoughby de Eresby commissioned local architect G.T.Ewing to remodel the exterior of the castle in the Scots Baronial style.
The current chatelaine of Drummond Castle is Jane Heathcote-Drummond-Willoughby, 28th Baroness Willoughby de Eresby who succeeded her father the last earl of Ancaster in 1983. In addition to Drummond Castle she also inherited Grimsthorpe Castle in Lincolnshire and a total of75,000 acres in England and Scotland. Her Scottish estate, the Drummond Estates comprises 60, 939 of those acres..