The sentence you've provided implies that the speaker is referring to someone they dislike, and they want to avoid places where that person might be found. However, the use of the term "toe-rag" is extremely disrespectful and makes the sentence sound crude and unpleasant.
It's important to remember that using derogatory language is never acceptable, even if it's meant to be humorous or casual. There are always better ways to express your feelings and opinions.