Can irrelevant backlinks reverse the effects of your SEO?

Google will not penalize your activity until your content isn't restricted and illegal and people don't report spam.