I’m going from vague memory here, but JFK was the first Catholic to win the presidency. So this speech was much like Romney’s speech a while back about Mormonism and the presidency. Essentially he was saying we shouldn’t exclude Americans based on their religion. (lots of bigotry towards Catholics at the time out of fear that any Catholic would be beholden to the pope, thus making the US subject to papal law, gosh doesn’t that all sound like the Sharia nonsense of late?)
So really, if you reject this speech you are rejecting any religion not your own.