In general, the designed landscape at Mount Stuart has retained a similar structure from its inception in 1717 to the present date. The main changes have been the moving of the public road to the west of the house, and the gradual loss of the twists and small paths within the formal garden layout. A new bowling green was laid out by the 1st Edition map of 1860 on the former parterre area in front of the house, and a tennis court was later constructed on the former site of the bowling green. The Wee Garden was added to the south of the policies in 1823 and the Mawson Rock Garden was added to the north of the house in 1893-8.
The Stuarts are descended from John Stuart, the natural son of Robert II who was made Hereditary Sheriff of Bute in 1385. The Stuarts were also hereditary keepers of Rothesay Castle and lived there until 1685 when it was destroyed by fire and assault. The family then lived in the mansion house which had been built in 1680 in the Rothesay High Street. In 1703 Sir James Stuart, the 3rd Baron was created Earl of Bute, Viscount Kingarth, Lord Mountstuart, Cumra and Inchmernock. He was succeeded by his son in 1710 and in 1718 the foundations were laid for a new house at Mount Stuart designed by Alexander McGill. The 2nd Earl married Anne Campbell, daughter of the Duke of Argyll, and much of the grounds at Mount Stuart were planted out from 1717, from which time accounts exist for young trees and seed. The 2nd Earl died in 1723 and was succeeded by his son aged 9 years. Although building work continued, the 3rd Earl did not visit Mount Stuart until 1734. In 1736 he married Mary Wortley Montague and they spent more time at Mount Stuart. He was a keen gardener and in 1737 started a Journal of Planting from the time of the first laying out of the gardens by his father.
A plan of the grounds drawn up for the 3rd Earl in 1759 by John Fowles shows a parterre in front of the house (ie. to the east), a bowling green to the south and a kitchen garden immediately to the north of the parterre. These were laid out as three of many compartments of a larger, formal design which extended from the house to the clifftop, and contained designed vistas, clearings and avenues. The Lime Tree Avenue and the Forty Five Avenue are marked as is the main drive. The Deer Park is shown to the north-west of the house, the Wilderness to the north-east. A folly on the top of Mount Montague provides a focal point to the west, and a Druid Temple is marked in the Deer Park. John Fowles's survey recorded 305 acres of gardens and paddocks, 'of which there are 135 acres under gardens and plantations, 120 acres in the Deer Park and the remaining 50 acres are divided into eight small enclosures and paddocks'. He also refers to 'the forest trees of very great size and beauty' and states that half of the Deer Park is arable.
The 3rd Earl became gradually more involved with political life in London. He was made a Knight of the Garter in 1762 and was Prime Minister from 1762-3. He was a patron of the Arts and also an arts collector, and he commissioned Robert Adam to build his new house at Luton Hoo. He was influential in the foundation of Kew Gardens and published privately his own botanical tables. He died in 1792 and was buried in Rothesay. The 4th Earl was married first in 1766 to the daughter of Viscount Windsor, and later to Frances the daughter of Thomas Coutts. He lived for part of the time at Mount Stuart and in 1805 employed Buchanan and Menzies to report on agricultural conditions and improvements needed on the estate. For his diplomatic service he was created Viscount Mountjoy, Earl of Windsor and Marquess of Bute in 1796. He died in 1814 and was succeeded by his grandson who was already 7th Earl of Dumfries in succession to his maternal grandfather.
The 2nd Marquess and his wife, Maria, the daughter of the 3rd Earl of Guilford, were frequently in residence at Mount Stuart and made many improvements to the estate and gardens, adding the English style estate village of Kerrycroy to the north of the estate. In 1823 Lady Bute designed a new small garden to the south of the formal compartments, south of the bowling green. William Burn was commissioned to design a new north entrance lodge and the ruins of Rothesay Castle were partly conserved at about this period.
After the death of his first wife, the 2nd Marquess married Lady Sophie Hastings, daughter of the Marchioness of Hastings, in 1845. Their son John was born in 1847 and succeeded as 3rd Marquess six months later. After his marriage in 1872, he had extensive improvements made to Mount Stuart but a fire in 1877 destroyed the upper floors of the house. Rowand Anderson was commissioned to submit plans for its rebuilding, most of which was completed in 1885. A new garden to the west of the house was designed for the 3rd Marquess in 1898 by Thomas Mawson. The 3rd Marquess died in 1900 and his son inherited his interest in restoring old buildings, becoming Chairman of the Scottish Historic Buildings Record Council and President of the Scottish History Society, an interest which is carried on today by the 6th Marquess. Mount Stuart was used as a hospital during World War I.