I think that some of Brexit is the byproduct of the harshly changing parliamentary style of government.
In a parliament, there is no Constitution, Presidency or Court system that can temper a Congress. When one party takes power, it rewrites half the statute book, and undoes much of what its rivals just did. From what I understand, the British tax code has swung wildly from the 1970's onward depending on who is in power.
When there is this sort of continuous political upheaval, a Brexit probably does not strike Britons as that big of a deal. They are used to running bums out of office and shaking things up when they are dissatisfied.
As far as xenophobia goes, I think that Britain can have a legitimate concern over controlling and securing its borders from other countries. Every other country besides the EU members does this. The US only lets aliens with a clean criminal record, a good work record remain in the United States (unless they play baseball or have valuable scientific knowledge). The EU freedom of travel, as I understand it, allows a person to travel from country to country. That means that there can be a depression in Greece, Latvia, Czech Republic, Lithuania, Romania and the rest of Eastern Europe, and an entire population can move to the next country over. Allowing a people the traditional control over their borders does not make them xenophobes.
Also, footing the bill for Eastern Europe is probably not cheap either.
Would like to hear from a UK person about this.