ingwe-
It seems very likely the refugee camp you saw in 1999 was the camp at Osire, so you were within 3-4 miles of the castle.
We drove past the camp four times, twice before dawn and twice after sunset, and I couldn't really see it in the dark. Google finds a fair amount of information about the camp on the web. Its population reportedly peaked at 20,000 about the time of your visit; it's now about a quarter of that.
I was told that the refugees from Osire have a history of poaching game and medicinal plants from ranches and farms near the camp. Here's a note about that:
http://afrol.com/News2002/nam001_osire_camp.htm--Bob
