And then there was Secretary of State James Baker’s infamous “fuck the Jews” remark. In a private conversation with a colleague about Israel, Baker reportedly uttered the vulgarity, noting that Jews “didn’t vote for us anyway.” This was more or less true—Bush got 27 percent of the Jewish vote, compared with 73 percent for Dukakis, in 1988. And thanks in part to Baker, it was even truer in 1992, when Bill Clinton got 78 percent of the Jewish vote and Bush got only 15 percent—the poorest showing by a Republican candidate since Barry Goldwater in 1964.

I’ve looked at several articles about it and none of them says who he was supposed to have said this to. Not saying he didn’t, but..
I can’t seem to find anything from snopes on it.