If Yubikey works anything like RSA SecureID does, then no, they couldn't break in without the key itself. (At least not unless the whole Yubikey infrastructure had been compromised, which as best anyone knows, it has not.)
HOWEVER.... You should *never* click a link in any email sent by any business to access their web site. Unless you're technically sophisticated enough to check the source of an email (most people who don't run their own mail servers are not), you won't always be able to tell a phish from the real thing. Instead, go to their home page from the URL you saved in your bookmarks, or type the web site URL into your browser's address box.
You should also not discuss your financial information with somebody who calls you on the phone, even if that person claims to be from a bank, financial institution, or business you use unless you know the caller personally and can recognize their voice on the phone. Instead, get their name, hang up, call 411 or look up the main phone number to that bank, call it, and ask for them. When you call them, you know you're talking with somebody at the business and not some scammer who stole a database and got your private information. :/