The designed landscape was laid out in the mid and late 18th century and the first half of the 19th century. An improvement plan was prepared by Thomas White in 1794. Reference to the 1846 survey of the policies indicates that its suggestions for the siting of the stable-block, walled garden, the east drive and the serpentine lake were carried out but the planting proposals appear to have been ignored. The lake disappeared between 1846 and the survey for the 1st edition OS map of c.1860. Further improvements were made within the 19th century layout between 1921 and the 1970s.
The family of Fraser of Castle Fraser are descended from the Frasers of Cornton (Stirlingshire). In 1454, on the break up of the estates of the Earldom of Mar, the lands of Muchall (later Castle Fraser) were created into a barony which James II granted to Thomas Fraser of Kilmundy. It is unclear whether a house existed on the site at the time when the charter was granted or if this first laird built a new house.
Michael Fraser, the 6th Laird, succeeded his father in 1565 and, within five years, had begun an extensive programme of alterations and additions to the Castle. His son, Andrew, continued the work after his father's death c.1588 and was largely responsible for the formation of the Castle as it remains today, employing master mason, John Bell, to carry out much of the work. In 1633, three years before his death, Andrew was created Lord Fraser of Muchalls. His son, the 2nd Lord Fraser, was an active Covenanter and it was during his time as laird that the policies were sacked by the forces of Lord Aboyne although the Castle itself was defended. Charles, 4th Lord Fraser, succeeded in 1674 to an estate which was somewhat impoverished due to the actions of his father. He was 'out' in the 1715 uprising and met an untimely death in 1716 when the title of Lord Fraser was also lost. His successor was Charles Fraser of Inverallochy whose exact relationship to the 4th Lord is subject to confusion. He was created titular Lord Fraser by the Old Pretender in 1723. His son, William, succeeded in 1787 but died unmarried in 1792 when the estate of Castle Fraser passed to his unmarried sister, Elyza Fraser, then aged 56.
Miss Elyza, 1734-1814, was responsible for the extension and improvement of the policies to their present structure. She commissioned Thomas White to prepare an improvement plan of the estate in 1794. Her friend, Miss Bristow, is thought to have influenced many of Miss Elyza's decisions, in particular the woodland in the south of the policies which, in addition to an estate cottage, was named after her. Another great friend, James Byres of Tonley, designed Miss Elyza's mausoleum.
Colonel Charles MacKenzie Fraser inherited the estate from Miss Elyza, his great- aunt. He made several extensions to the house and remodelled the earlier 17th & 18th century interiors. He was, however, a great agricultural 'improver' and continued the work on the policies which his predecessor had begun. In later years it is thought that James Giles may have advised him on the design of the landscape. Records were kept of almost all activities in the Castle and policies which have proved an invaluable source of information as to their way of life. Colonel Charles Mackenzie Fraser was succeeded by his only surviving son, Colonel Frederick, in 1871 and, on his death in 1897, the estate was left in the hands of trustees.
The male heir was Thomas Fraser Croft Fraser who showed little interest in the estate, his work being incompatible with that of a country landowner; he ultimately became Privy Chamberlain to the Pope and Master of Ceremonies at St Peter's. Thus the widow of Colonel Frederick sold the estate to Lord Cowdray in August 1921 for his 2nd son, the Hon Clive Pearson. He commissioned Sir Robert Lorimer to survey the Castle and, whilst he made many proposals for alterations to the Castle between 1922- 27, none of his ideas were acted on. Dr William Kelly was ultimately commissioned to restore the Castle and work was carried out until World War II. The theft of lead in 1950 from the roofs of the 19th century additions prompted their demolition and restoration of the courtyard into its original form. In 1947 the estate was passed to Mrs Lavinia Smiley, daughter of the Hon Clive Pearson, and, with her husband, Major Michael Smiley, she continued the restoration begun by her father. The work was completed in the 1960s, the stable-block having already been converted into the family home. Except for two wings the Castle had not been lived in since 1921 and, in 1976, it was given to the NTS, together with some 26 acres of surrounding land.