Robert Adam’s cliff-top masterpiece rises above a world of woods, beaches, secret follies and play parks