In the garden there are twin mounds and a dry moat. This is the site of King Edwin's castle, who married Queen Ethelburga, who was a Christian. In 1557 William Darley bought the manor of Buttercrambe from Earl of Westminster. An early estate map dated 1633 shows a Tudor house situated 25 yards east of the present one. The present house was built in 1726 by John Brewster Darley, probably to his own design, and with the help of the many talented craftsmen that abounded in Yorkshire at that time.

In 1743 John Brewster Darley died and his son Henry succeeded him. He continued to work on the house and employed Thomas Knowlton to remodel the park and garden. Knowlton at this time was Lord Burlington's gardener at Londesbrough. Knowlton's bill of 1746 shows that he was responsible for laying out the terraces that stretch down the steep bank to the Derwent, for the walk along the top of the bank, and the paths which wind round the Saxon Mound. He also produced a plan for landscaping the deer park. Neglect by the Government during WWII when the house was occupied by the Army, made the house uninhabitable and it was only in the mid-1960's that it was re-occupied by the present owners and the immense task of clearing and re-planting the garden begun - with the help of just one gardener!