The monument consists of an early castle adapted for occupation and defence over a number of different periods until its abandonment as a residence during the 20th century.
Skelbo Castle was a seat of the de Moravia or Moray family, who acquired lands in the area at some date before 1211. The castle closely echoes both the form and the development of Duffus Castle near Elgin, the seat of the main branch of the family, who indeed owned Skelbo 1529-1787.
The site is complex, with evidence for a number of different building phases, and covers an extensive area. Initially, it seems to have been an earthwork structure, with the natural motte at the north end of the site probably supporting a wooden tower. Over time, the defences were rebuilt in stone, and the fragmentary hall/keep now standing on the motte probably dates to the 14th century. It appears to have had a first-floor hall with a wooden floor supported by a central row of posts. Stone curtain walls, probably on the lines of earlier wooden defences, form a roughly triangular courtyard with the motte at its northern angle. There are traces of buildings at several points around the perimeter and a building at the SE angle may have been a gatehouse. Traces of an outer ditch also remain.
Against the W wall of the enclosure stands a range dating from c.1600, measuring 20.6m x 6.8m. It contains a series of vaulted basements beneath a separately-entered hall and chamber, with a garret above. This form is suggested as a typological link between the medieval hall house and the bastle house common in early 17th-century Scotland. The SE corner of this range has collapsed, but the N end of the slated roof remains. The house was occupied by the Factor during the 18th century, and traces of several phases of alteration and refenestration are apparent. A garden lay to the W of this range, overlying part of the castle ditch.
The area to be scheduled is roughly triangular in shape, measuring a maximum of 110m N-S by a maximum of 135m E-W, as defined in red on the enclosed map. It extends 20m beyond the existing fence and wall lines to the N and W, and is defined by the N edge of the track to the SE, to cover the castle and outer ditch together with part of the 18th-century garden. The scheduling does not include the gate leading from the track to the field W of the castle.