Not all users can downvote things, I'm not sure if it's directly tied to how much karma you have but it's a measure to give more weight to the opinions of the more respected members of the community instead of anyone who has an account. Hopefully this will mean that Hacker News never turns into Reddit.