I've only had missionaries bother me a few times. The only time it was an issue was with a couple of Mormon guys in an access-restricted apartment building. I think they knew they shouldn't be there because they wouldn't answer how they got in. I told them they had 30 seconds to leave before I called the police; they were gone in 10.