Supposedly it’s offensive because in Old English wealh just meant “foreigner”. That’s probably the least offensive thing one ethnic group ever called another. What about the Slavs calling the Germans Nemets: “silent”, because they didn’t speak their language?