While the earliest house, a medieval Castle belonging to the MacDuffs, Thanes of Fife, was built to take advantage of the defensive, coastal site, 'Castlehill,' a small 18th century mansion house, was built further east to exploit the site's picturesque qualities (Beveridge 1888, p.197-8). The Blaw family owned the estate from the 16th century until 1830, when it was sold to Lady Margaret Keith of Tulliallan (q.v. An Inventory of Gardens and Designed Landscapes, Volume 4, pp. 415-19). Shortly afterwards the small mansion was incorporated into a castellated villa (1839-45) for Mrs Magdalene Sharpe Erskine (1787-1872), sister and heir to Sir John Drummond Erskine of Torrie (1776-1836) a cadet branch of the Earls of Buchan. Her marriage to Admiral Kilpatrick Sharpe, when in her 50s, lasted three days after which a permanent separation was arranged. She turned her attentions to her estate, changing its name to Dunimarle, and transformed the property, commissioning the architects Robert and Richard Dickson, whose 'general concept' for their design derived 'from Nash's East Cowes Castle' (Gifford 1992, p.196-8). This composition relied on an asymmetrical and irregular outline to the Castle and careful consideration of its orientation in relation to its seaside, cliff-top setting. A three-storey wing with mullioned and transomed windows and crenellated parapet was added to the east. An orangery, a castellated gateway with monumental iron gates, a castellated garden gate, drum tower and screen wall, and an extensive raised terrace enclosed the forecourt. To the south a long terrace overlooking the Forth and backed by a crenellated garden wall extended the picturesque composition.
An account of the gardens in 1841, by the gardener Jasper Wallace, described old pear trees, claiming them to be 240 years old, and planted in 1600. He notes some of them with a girth of 6 feet. He described the planting of new trees and the need for soil improvement (The Gardener's Magazine 1841). A guidebook to Dunimarle and its collection passed through various editions (1873-84) and provides a good description of the gardens at this period. Of particular mention is the terrace ' a perfect paradise of beauty…beautiful flowers, the most lovely views of wood, mountain, and sea with all the accessories of artistic decoration, which the most refined taste could devise'. There were shrubberies, large fruit gardens and a spectacular Vinery. This contrasted with the Dell, a woodland garden with winding coastal walks, which led to the neighbouring Balgownie House policies.
Mrs Sharpe Erskine, who inherited a collection of pictures from her brother, was an avid fine and decorative arts collector. She endowed Dunimarle as a museum of art (c 1853) to display her collection of 850 works. These remained at the museum, which opened to the public in 1872, until the 1950s. In 1870 she commissioned the architect Robert Rowand Anderson to design St Serf's Chapel. Mrs Sharpe-Erskine died in 1872 and the estate was put into the hands of trustees. In 1872 a flash-flood swept across the site, toppling the lower terrace wall and walk (below the Castle). The walk was re-instated and the slopes modified. Mrs Sharpe Erskine's collection finally left Dunimarle Castle in 1995, when the National Galleries of Scotland transferred it to Duff House to form the basis of the collection there. The Castle and grounds are administered and managed by the Sharp Erskine Trust.