Improvements were carried out to the estate in 1751, 1786 and 1872 but there are no estate or design plans left at the Castle and historical evidence relies on the 1st & 2nd edition OS maps. In 1900 Andrew Carnegie commissioned Thomas Mawson to carry out some work on the Terraced Garden.
A fortress was probably first constructed here as long ago as the 9th or 10th century, probably by the Vikings, Skibo being a Norse name. The earliest records are from 1186 when Hugh Freskyn granted it to his kinsman Gilbert, Archdeacon of Moray, who became Bishop of Caithness and Sutherland in 1223. There are historical references to the grounds of Skibo first being laid out by the monks from the Bishop's Palace in Dornoch. There have been several changes of ownership over the years since then, with a long period of stability between 1560 and 1744 when the lands of Skibo were feued from the Sutherlands by the Gray family. In 1744 the Estate was bought by Sir Patrick Dowall, whose nephew the Hon George Mackay succeeded in 1751. He was the son of the 3rd Lord Reay and is recorded as greatly improving the estates by planting. He sold the estate a few years after his succession when it was purchased by William Gray, a descendent of the Grays of Skibo, who had made a fortune in Jamaica. He rebuilt and renovated the Castle but died in 1760. In 1771, Thomas Pennant referred to Skibo's beautiful situation and referred to the modern house as being 'habitable'.
In 1786, George Dempster, MP, a great agricultural restorer, bought the estates and set to work improving the conditions of the tenants, implementing drainage schemes and trying to improve local industry. He was a great friend of Sir Adam Ferguson of Kilkerran, and an associate of Boswell, Hume and other Edinburgh literary figures of the time. He extended the estate by buying Pulrossie and Overskibo. His heirs predeceased him and the estate was broken up after his death in 1818. Skibo went to his great-nephew, George Soper-Dempster, who sold it in 1866, when it was bought by an Australian, Mr Chirnside, who kept it for a few years before selling it again.
In 1872 the estate of 20,000 acres was purchased by Mr Evan Charles Sutherland- Walker who made further extensive additions and alterations to the Castle, with new farm buildings and stables. He redesigned the grounds and put in a new avenue to the east lodge, but spent so much that his loans were foreclosed and the estate was managed by Trustees for the next few years.
It was purchased in 1898 by Andrew Carnegie, the multi-millionaire philanthropist, who greatly extended the estate and made extensive alterations to the Castle. He began his improvements around 1900, building new tenants' houses, the dairy, the coach-house, the swimming-pool, the conservatories, and the West Lodge. Improvements were also made to the grounds, the lochs were constructed and water and electricity supplies installed. Thomas Mawson was commissioned to improve the Terraced Garden, and an avenue was laid to the west lodge. The Carnegies spent part of the summer and autumn every year at Skibo, returning to America for the winter, up until the outbreak of World War I. Mrs Carnegie continued to visit Skibo each year up until 1934 and was succeeded in 1946 by her daughter, Mrs Millar, who continued to visit until 1980. The estate was again put up for sale in 1982 when it was purchased by Mr Holt who has carried out further improvements to the Castle and the grounds.