The monument consists of a late-fifteenth-century tower-house of L-shaped plan and of fine ashlar construction. It stands at the north-eastern corner of a courtyard of buildings. The tower was modified in 1635, when larger windows were inserted to light the hall, when that date was inscribed on one of the lintels.The castle was the principal residence of the Stewarts of Rosyth, to whom the barony was confirmed in 1428; the family became extinct in the late seventeenth-century.
The courtyard is of mainly sixteenth- and seventeenth-century date, and was originally entirely enclosed by ranges of buildings. Of the courtyard ranges parts of those on the north and west sides still stand to a considerable height, while the lower walls of the east and south ranges have been located by excavation. The entrance to the courtyard was through a small salient gatehouse in the north range, immediately adjacent to the tower-house; associated with the heraldic panels above the gateway is the date 1561.
The castle originally stood on a rocky tidal promontory off the north shore of the Firth of Forth, and it is likely there were extensive ancillary buildings on the mainland, of which the dovecot is the only relic. The castle is now surrounded by the reclaimed land of the naval dockyard.
The area to be scheduled includes the quadrilateral area that has been in the care of Historic Scotland and its predecessors since 1928, defined (in the imperial units at use in the time) as extending 51 feet out from the north side of the castle and 49 feet out from the east. The boundary on the west side is defined by the pavement flanking the road on that side, which is 21 feet from the north-west corner and 11 feet from the south-west corner. The area is marked in red on the accompanying map.