The castle has the reputation for being one of the most haunted houses in Wales, these include: the "Grey Lady" which is said to be the most commonly seen; the ghost of a monk said to have been trapped in a tunnel leading from the secret room; and that of Sir John Wynn. Apparently, many local people will testify to having seen ghosts in the castle.

The hall is reputedly haunted by the ghost of Sir Thomas Kyffin, as punishment in life for allowing his violent temper causing the death of a boy. It is a Grade 1 listed building located on a rise approximately 200m east of the A470, 1.2km northwest of Llandogged village; accessed via a long drive leading off the main road.

Hafod Ucha lay somewhere between Llanrwst and Conwy, and once had trouble pre-1870 with the spirit of Abel Jones. Eventually, his spirit was captured in a bottle and buried in a “gors”.