And I don't understand something about Armory. They give you a passphrase that can restore your wallet. So can anyone just sit at home and attempt to "restore" other people's wallets? What am I missing?
It's a private key, not a passphrase. It's a tad too long to be used as a passphrase unless you have a very good memory.
But yes, you could
sit at home and keep entering private keys until you get one that someone's already using (note that the use of private keys is fundamental to how Bitcoin works, and is not limited to Armory or any other specific software). What you're missing is that it would require more time and energy than the universe contains
to do so.