The designed landscape was first laid out in the early 17th century; the policies and woodland were planted in the late 18th century and altered in the mid-19th century. The Flower Garden was replanted in the 19th century and again in the mid-20th century.
In 1454 William, the 6th Thane of Cawdor, was granted a licence to fortify Cawdor Castle. The story goes that, because of a dream, the site was originally chosen by a donkey laden with gold which lay down under a tree. This tree was then enclosed by the new tower. In the vaults there is the preserved stem of a holly tree which has recently been carbon-dated to 1372. In the early 1600s Sir John Campbell, the 12th Thane, began to improve the Castle. He added to the family fortunes by acquiring the Island of Islay and marrying his wealthy cousin from Taymouth. The first mention of a garden and an orchard was in 1635.
The 12th Thane died bankrupt and his son was mad. His grandson, Sir Hugh Campbell, the 15th Thane, restored the family fortunes and transformed the Castle. Educated in Europe, he returned following the Restoration and between 1663 - 1668 built new wings and renovated the old tower. In about 1720, the walled Flower Garden was laid out by the then Thane's brother Sir Archibald Campbell. Sir Alexander Campbell, the 16th Thane, having been stormbound in Milford Haven then married a local heiress, Elizabeth Lort of Stackpole. Although sympathizing with the Jacobite cause, the Campbells of Cawdor spent most of the 18th century residing on their Welsh estates. Cawdor was managed by factors who were younger brothers or relations and it was one of these factors who laid out most of the policies and woodlands towards the end of the century. In 1789 John, the 19th Thane, married the daughter of the Earl of Carlisle of Castle Howard.
Throughout the 19th century, the family generally stayed at Cawdor in the late summer months and the Gardens were developed to provide colourful displays bet ween August and October. Lady Cawdor redesigned the Flower Garden in 1850. John, 5th Earl and 24th Thane, was the first member of the family, for over 200 years, to have lived permanently at Cawdor. With his wife, he replanted the Flower Garden and created a small wild garden on the banks of the rocky Cawdor Burn. His son Hugh, the 6th Earl, succeeded in 1970 and in 1981 with his wife laid out a holly maze in the Lower Garden. More work is planned.