William Duff gave the work to build a grand new seat for his dynasty to William Adam (1698-1748), the leading Scottish architect of his day. A commission to build on a fresh sight, rather than try and impose a symmetrical house onto an old tower must have been particularly exciting. Building started in 1735.
The extraordinary quality of the work can be explained by the fact that the stone was carved at Adam's quarry near South Queensferry and carved under the architect's supervision and shipped north when it was completed. Adam himself made an annual visit to Duff to monitor progress.The wood used in the building came from the Mar Estate, and it is possible that Duff had bought the estate for this purpose. The timber was floated down the Dee, and the floorboards were proudly stated to be made from wood from the forests of Mar.
The entire facade is embraced by giant Corinthian order pilasters with richly carved capitals supporting a pediment in the centre. The plan of the house reflected traditional Scottish arrangements with the lowest floor for service rooms, the first floor was for the family while the piano nobile on the second floor comprised an enfilade of the grandest rooms on a processional route approached by a grand staircase. This arrangement gave the house a towering aspect, further emphasized by the corner closet towers rising to domes supporting octagonal chimneys. It is a magnificent sight.In the end Duff and Adam fell out in a dispute over costs and the intended wings were never built.
In 1889 the 6th Earl of Fife married Princess Louise, later the Princess Royal, oldest daughter of the Prince of Wales who became Edward VII, and he was created Duke of Fife. Duff House was not considered suitable for a royal princess, instead Mar Lodge was rebuilt after a fire in 1895and became their residence.Financial problems caused the Duke and Duchess to sell Duff House to the towns of Banff and Macduff in 1906.