The original Ardtornish House stood approximately 5km to the south of the present house site near Ardtornish Castle at Ardtornish Point. It was built by the Duke of Argyll for Donald Campbell, his factor on the Morvern Estates. In time, the house passed to Campbell's successor, Angus Gregorson, whose son George purchased the surrounding estate in 1819. Patrick Sellar from Sutherland acquired the Estate with other ground further inland in Morvern in 1844. His holdings were separated by the Achranich Estate which was purchased by Octavius Smith from London in 1845. He began the construction of a new house on his estate between 1856-60. By 1860, the estates were amalgamated after Octavius Smith had bought Acharn and Ardtornish from Sellar's heirs. Smith's new house became known as Ardtornish Towers. During this century the name has been changed to Ardtornish House or, more usually, Ardtornish. He carried out many improvements throughout the estate. These improvements were later continued by his son, Valentine Smith, a distiller in London, who inherited on his father's death in 1871.
Of the Smiths' improvements, the most outstanding is the generally high standard of construction achieved in the new estate buildings which is considered remarkable for this period in Scottish history. Samuel Barham, the Estate Master of Works, was responsible for much of the design and his use of concrete in the structures is thought to be amongst the earliest in the United Kingdom. Following the completion of the new house, the two properties continued to be inhabited by the family. In the early years of 1880 when it had only been completed for fourteen years, it was discovered that extensive repairs were required to the Octavius Smith house. Valentine Smith demolished it, retaining the old clocktower, and commissioned Alexander Ross to construct the present house which was built between 1884-91. Following its completion, Valentine Smith moved permanently to the new house. The present gardens were begun by him. His sister, Gertrude, who had married one of Partrick Sellar's sons, returned to Morvern in 1906 with her son. She inherited her brother's estates. The old Ardtornish House was found to be in a state of dereliction and it was demolished. Gertrude Sellar and her son continued to develop the gardens round the present house in the Edwardian style.
In 1930, Owen and Emmeline Hugh Smith purchased the property from the executors of the Sellar family. The gardens contained an interesting range of plants to which the new owners enthusiastically began to make additions. Sir John Stirling Maxwell was a family friend and he donated plant material from his home at Pollok, Glasgow.
During World War II the garden inevitably became overgrown through reduction in maintenance resources; however, Mr & Mrs Hugh Smith continued to plant. Since then the scrub invasion has gradually been cleared. Within the last seventeen years, their daughter, Mrs Faith Raven, has cleared, improved and diversified the garden. It was a feature in the book by her husband, Mr John Raven, 'A Botanist's Garden', published by Collins in 1970, which described many of the plants grown at Ardtornish and in their other garden at Docwra's Manor, Shepreth, Cambridgeshire.