Balnagown was built as a defensive stronghold in the 14th century and the lairds of Ross belonged to a warfaring clan who did not settle or lay out the landscape around the Castle until the late 17th century. The grounds were completely cleared and redesigned in the mid-19th century in the Picturesque style.
Balnagown is the historic home of the Clan Ross chieftains. The building of the castle was begun in the early 14th century by Hugh, the 5th Earl of Ross, who had married the sister of King Robert the Bruce. The Earl was killed in battle in 1333 and his son William offended the King so much that his title and lands were forfeited. It was left to his younger step-brother to become the 1st laird of Balnagown Castle in 1375. The 3rd laird, Walter, acquired an extensive tract of land across to the western shores of Ross by marriage with the daughter of Paul MacTyr. The next four generations lived in warfaring times and in 1585 the 9th laird, Alexander, was outlawed for his crimes and plundering. His son George followed his example, and in 1615 he too died an outlaw, having bankrupted the estate. His son, David, lived in quieter times and was created a Baronet. His son, Sir David, raised a royalist army in support of Charles II but was captured and died in the Tower of London. His son, also David, married Lady Anne Stewart who initiated the first rebuilding of the Castle and laid out a formal landscape around the Castle, with walks and driveways. However, these improvements left the estate encumbered with debts when David died in 1711 without an heir. In the ensuing legal struggle, the estate was won by the Rosses of Halkhead, while the chiefship passed to the Pitcairnie Rosses. By 1754 all of the Halkhead Rosses had died without issue and the estate passed to a distant connection, Admiral Sir John Lockhart Ross of the Lockhart Rosses.
The Admiral enjoyed a successful naval career and at that time was entitled to half the booty captured from enemy vessels. This he used for the improvement of his estate, bringing with him the latest French textbooks on agriculture. He became the most efficient and enterprising Highland estate manager of his day, enclosing fields, draining marshes, planting forests and, despite local scepticism, introducing fine sheep to the moorlands. There are references to John Adam working on the garden at Balnagown in c.1762 (A.A. Tait). The Admiral died in 1790 and his son, General Sir Charles Ross, married Lady Mary Fitzgerald who initiated the next major phase of improvements to the Castle and pleasure grounds. There is reference to James Gillespie Graham being consulted on the improvements to the Castle which were made in the Gothic Revival style, as introduced at the Brighton Pavilion, incorporating Indian style features such as the Porch. The defensive slit windows at the rear of the Castle were opened up to exploit the views along the river and, in 1847, the Head Gardener from Holyrood Palace was employed to lay out the Italian Gardens. The policies were cleared and replanted at this time. Sir Charles and Lady Mary's son was somewhat eccentric and was known as 'The Jackdaw' from his habit of sitting in an oak tree in the grounds and having his food brought out to him. Attempts to have him committed failed.
His son Charles succeeded in c.1882 and was also rather eccentric, becoming an inventor, a big-game hunter, a philanderer, a gun manufacturer, and finally carrying on the family tradition by being made an outlaw. He was, early on, a keen agricultural improver, introducing the first British silo to Balnagown, together with an early combine harvester. He first imprisoned, and then set his mother's hair on fire to get her to leave the Castle. He invented the Ross Rifle and in 1910 he set up the Ross Rifle Company in America, manufacturing rifles for the Canadian Army. The British Inland Revenue threatened to confiscate Balnagown Estate as their share of his profits and so he had the Estate made a ward of the Delaware Court or, in effect, American territory. From then on he could only return to Britain on pain of imprisonment. During the war, British troops were billeted at Balnagown until the government was informed that they were on American soil, when they were withdrawn. Sir Charles died in 1942, when his last wife eventually made a Trust to keep the grounds managed and used for sporting parties. The Castle was not lived in between 1942-1972, and in 1972 the Al Fayed family purchased the estate and began another phase of development to the Castle and grounds.