I used to live about 10 metres from the Castle. There has been trouble there, but it's not like it kicks off every day. I think in 5 years, I saw 2 big fights spill out onto the road. Of course, like all pubs you get people standing outside drinking, smoking, talking on the phone... but presumably you've already figured that out, and it only affects you if you're very close! Le Moulin actually do some quite nice food... although some of the dishes are distinctly unappetising - e.g. pork in pineapple sauce, the kind of thing which may have seemed adventurous at a dinner party in 1978. But not everything is like that. Definitely worth a try. The Plough is a decent enough pub (you should have seen it a couple of years ago!) with quite passable food. And of course you have the library and plenty of local shops. Dulwich park is a short walk away, and is a very nice place to while away the summer afternoons. Dulwich Village is close too. You can walk to East Dulwich station in 15 mins if you go along the full length of Melbourne Grove.