The world of banks is not just FED and JP Morgan. It is all central banks, and the mother of them all is the central bank of central banks, aka. The Bank for International Settlements. Shortly said, the players that control this bank are all other central banks. If central banks echoes the same "worry" for Bitcoin, it is because they all have the same mother, The Bank for International Settlements.
Trillions upon trillions are handled through this behemoth of a bank. This is where the real power lies, worldwide, beyond politics and beyond local interest. Bitcoin is a threat to pretty much any elite establishment on the planet, and not until this "bank" bows, will Bitcoin take over.
Edit: Some info here:
http://en.wikipedia.org/wiki/Bank_for_International_Settlements and here of course:
http://www.bis.org/