I don’t get why why this would matter. Even if he said it, it would be true, so what’s the big deal?

Because they’re all a bunch of bigoted assholes and him saying he’s black means that he was really born in Kenya and Sarah Palin is now President.