You can talk to him instead of about him.

I’m not sure it would be more polite, but I think it might be less rude.