Cats choose their humans. As much as we would like an exclusive relationship with them, they don't always agree.
Also, not everyone reads posts on this forum. you may find posting through letterboxes or posters on lamp posts more effective in asking for him not to be fed.
You can't ask people to lock their cat flaps to stop him coming in - and they may not be home to ask him to leave. In fact, he may be just eating anything other people leave out for their cats.
I'm sorry - but cats like to do their own thing.
It's strange that he leaves when he's hungry - have you tried changing his food? it sounds like he might prefer a different restaurant.