There was a recent related discussion on Hacker News and the top comment discusses why this sort of solution is not likely to be the best fit for smaller organizations. In short, doing it well requires time and effort from someone technically sophisticated, who must do more than the bare minimum for good results, as you just learned.
Even then, it’s likely to be less reliable than solutions hosted by big corporations and when there’s a problem, it’s your problem. I don’t want to discourage you, but understand what you’re committing to and make sure you have adequate buy-in in your organization.
It seems to me software designed to facilitate discussion shouldn’t have a downvote buttton. There should be a UI for marking comments as inappropriate, but it should require a second step saying why. Perhaps one of the reasons should even be “I disagree”, but that option should have no effect.
It’s not impossible to abuse of course, but it nudges people in the right direction. Those UI nudges can be pretty effective.