Found the book I was thinking about:
Terror in the Name of God: Why Religious Militants Kill by Jessica Stern

It’s not just about domestic terrorism but it does cover that too.