Brough Lodge was built in c 1820 for Arthur Nicolson, who enclosed the Fetlar lands for sheep and evicted the tenants. In 1825 Nicolson took the title Baronet of Nova Scotia, a family title which had lapsed on the death of the 6th Baronet in 1743. The Nicolson family, long-established in Shetland, owned large areas of land, including Papa Stour. However, they only acquired the Fetlar lands in 1805, which were received in payment of a debt owed to Arthur Nicolson by Andrew Bruce of Urie (d.1803).
Initially, Nicolson lived in the Haa at Urie before building Brough Lodge. The main entrance to this castellated, Gothic house lay on the south side. The roof was crenellated all around with bartizans at each corner. Nicolson built The Tower, c 1840 on the site of an Iron Age broch, 'situate on the top of a small hill; the walls entirely in ruins; without any outworks as its situation did not require them' (Low, 1879). A stair-tower adjoined the tower. Nicolson also built another folly, the Round House at Gruting, which seems originally intended as a summer house but came to be used as the estate rent office. Local tradition recounts that it was built out of stones from evicted tenants' houses. Sir Arthur only spent one night there and heard strange noises during the night. Thereafter he was dissuaded from staying at Gruting. The noises were rumoured to be the curses of tenants cleared from the land.
When Arthur Nicolson died in 1863, the title passed to his cousin Arthur Bolt Nicolson, who spent much of his life in Australia. He became Commissioner of Goldfields and died in Australia in 1879. Despite returning briefly to Britain, it is unlikely that he ever visited Fetlar. The estate itself was settled on Lady Eliza Jane Nicolson (d.1891), Arthur Nicolson's widow, who lived in Cheltenham. On her death the estate passed to Arthur T.B.R. Nicolson (1842-1917), Arthur Bolt Nicolson's son, who had inherited the title. In 1891 he was living with his family at Fetlar, which was in a poor condition, having been neglected by absentee ownership. Arthur T.B.R. Nicolson, born in Australia and educated at Melbourne College, was a Justice of the Peace for Victoria and, later, Shetland. He married Annie Rutherford (d.1930), who maintained a detailed diary of life at Brough and on Fetlar recording how the Brough landscape accommodated golf, tennis, fishing and shooting (Brough Lodge Trust).
When Nicolson died in 1917, the estate continued in the Nicholson family. Brough Lodge has been unoccupied since 1988 and the outbuildings used for agricultural storage