I always wondered about the crack heads/squatters/hobos when I was starting out. I've seen plenty of evidence of them...even stepped in some stinky evidence, if you catch my drift, but I've never actually bumped into them. They're very shy creatures indeed and probably 9 times out of 10 don't pose a threat.
Honestly the most common safety issue with me, being a gammy legged giraffe, is slipping. I've slipped on poop, oil, wet patches, rubble, algae, mud, etc more times than I can remember and just narrowly avoided injury. The major dangerous thing is heights - losing your footing, falling through a rotten floor, tripping down stairs, etc. For that reason, I don't do high stuff and I don't do sketchy entries/exits. I've also had 1 live electricity mains on a wet floor, a few hypodermic needles and seen my share of disturbed asbestos.