The designed landscape was first laid out in the early part of the 17th century and improved during the latter part of that century. The estate was looked after throughout the 18th century. In the early part of the 19th century the park was extended and the large walled garden laid out. The garden has been maintained without a break ever since.
In about 1490, the 1st Lord Drummond built a tower castle on land held by his family for more than a hundred years. Early in the 17th century, his descendant, James Drummond, was sent to Spain as Ambassador by James VI (I) and on his return was created the 1st Earl of Perth. He altered the castle. His son John, the 2nd Earl, was captured by Cromwell in 1646, five years after the army had besieged and sacked the castle. In about 1630, before he became entangled with the Civil War, John had laid out the terrace garden, similar to the ones being built at the same time at Culzean and Drumlanrig.
James, the 4th Earl and 1st (titular) Duke of Perth, was a close advisor to James II and became Lord High Chancellor of Scotland. In about 1689, he built the core of the present house which is separate from the castle at the eastern end of the courtyard. He was recognised by his contemporaries as a great 'improver'. After four years in prison, the Duke and his family followed James II to France where they stayed until the Duke's death in 1716. His eldest son, James, the 5th Earl, known as the Marquess of Drummond had returned to Scotland to look after the estate. During the 1745 uprising, James followed the Jacobite cause and was forced to flee to France where he died two years later. His wife returned to Drummond Castle where she managed the property for her son James, the 3rd Duke, until 1750, when all the Drummond estates were forfeited following the Duke's involvement with the 1745 disturbances. The estate was managed by Commissioners appointed by the Crown who carried out quite a number of improvements.
In 1784, the property was returned to a member of the family called Captain James Drummond, a descendant of the 3rd Duke's younger brother. He made considerable changes and on his death in 1800 the property was inherited by his daughter. She managed Drummond for the next 65 years and with the help of her husband, Lord Gwydyr, laid out the park and created the gardens on the terraces. These were soon recognised for their beauty and were visited by Queen Victoria and her family in 1842, only 10 years after they were laid out. Various alterations and improvements were made for the Queen's visit.
Drummond Castle was inherited by Lady Gwydyr's eldest daughter, Clementina, who in 1870, became Barones Willoughby de Eresby in her own right following the death of her brother. She married Sir Gilbert Heathcote and, in 1878, they renovated the house to designs by G.T. Ewing. Ten years later, their son inherited the property and was created Earl of Ancaster in 1892. The 2nd Earl of Ancaster succeeded his father in 1910 and on his death in 1951 his son, the 3rd Earl, inherited. In 1933, the 3rd Earl married the Hon Nancy Astor and in about 1956, they replanned and replanted the garden. The gardens and part of the policies are now managed by a charitable trust company.