The ghost of Charlotte, an Aunt of the Portman family, supposedly haunted the old building on the site. A family legend stated that if the white and coloured peacocks left Bryanston, then the Portman's would soon follow. Curiously in 1923, the third Viscount sold the birds and died not longer afterwards. The building was then sold to the school. A ghostly coach and horses has been seen driving out of the gates (pictured above).