The lands and Barony of 'Aultdownie' (NAS) were originally a property of Mackintosh of Kyllachy (1625) in Nairnside. In 1776 Aldourie passed by the marriage of Alexander Tytler (1747-1813), later Lord Woodhouselee, with Ann Fraser of Balnain into the Fraser-Tytler family.
The 17th century mansion house consisted of a rectangular main block, one room deep, with a round tower at the south-west corner. In 1839 it was extended to the west, with a two-storey wing. William Fraser-Tytler, a Lieutenant in the Bengal Army and Sheriff-Depute of Inverness-shire for 42 years, was in the 1860s, responsible for transforming Aldourie from this relatively modest mansion house into a picturesque castle with landscaped grounds. He commissioned Mackenzie & Matthews to extend the house 'in all directions, parading the full repertoire of early 17th century Baronialism, including a balustraded round tower cribbed from Castle Fraser, Grampian, oriel windows, scroll-sided steeply pedimented dormers, candle-snuffered turrets, corbelling, rope- moulded stringcourses and gunloops' (Gifford, 1992, p.148). A fire-proof cement floor was laid between the ground and first floor. The parkland was probably laid out around this time, involving land-drainage and canalising and ornamenting the Dobhrag Burn with small cascades and footbridges. North-east of the Castle, considerable earth-modelling, taking advantage of the incised topography, formed a series of enclosed dells and a long terrace set with a formal walk overlooking the Dobhrag.
William Fraser-Tytler married Mary Grant and had 11 children. A number of them entered military service or the East India Company, including their son Charles who married Etheldred St Barbe. After Etheldred's death in 1851, their daughters were brought up at Aldourie, moving in 1861, on Charles return from India, to Sanquhar. Mary, the youngest, married the eminent painter George Frederick Watts (1817-1904) in 1886, becoming Mary Seton Watts (1849-1938). She was thirty-six years his junior and devoted her life to his care and, following his death, his reputation. Shortly after their marriage Mary designed a triptych for the entrance to the family Burial Ground. This included bronzes, modelled on Watts drawings 'Love and Death' 'The Messenger' and 'Death and the Angel Crowning Innocence'. The latter was drawn as a memorial for Mary's nephew who had died six weeks before their marriage (Gould, 1998, p.30). Her watercolours of the parkland at Aldourie survive (Gould, 1998, p.29).
In 1891 the Watts moved to Limnerslease, Compton, Guildford. Mary Watts gave up painting to develop her skills in modelling and terracotta work. In 1893 she set up a school of decorative terracotta work, which produced external decorations for the Compton Memorial Chapel and funerary monuments. She set up a professional Pottery Arts Guild at Compton followed, in 1900, by one at Aldourie. She trained Louis Deuchars to model terracotta and assist in the classes held at Compton before he went to Aldourie to run the workshop. Terracotta memorials, garden ornaments and sundials produced to her designs were exhibited internationally, won awards, including medals by the Royal Botanical and Royal Horticultural Societies, and were marketed by Liberty.
In 1893, the Lovat Scouts reputedly were founded at Aldourie by Lord Lovat and Edward Fraser-Tytler (Gould, 1998, p.62-3).