how that hash do knows this is the correct key for the hash Alice provided?
Hashing goes in one direction. You can take some data, and calculate a hash of it. But you cannot take a hash, and find a matching data easily. If you can, then a given hash function is broken.
Some example:
https://emn178.github.io/online-tools/sha256.htmlSHA-256("Hello World!")=7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069And then, by requiring a message, which will hash exactly to 0x7f83b1657ff1fc53b92dc18148a1d65dfc2d4b1fa3d677284addd200126d9069, you can be quite sure, that it will be "Hello World!". If you could go in an opposite direction, and find a different message, which would hash to this value, or have any software, which will find you a valid message for any hash, then this hash function would be broken.
Alice could have entered the wrong key
If you use a wrong key, or a wrong hash, then coins will be unspendable. Sending coins to random public keys or hashes is possible. And then, if nobody knows the key, and the message, then they will stay on a given address, potentially forever, or until someone will break a given public key or hash. Also, there are invalid public keys and hashes, and then coins are just provably unspendable.
More technical details:
https://en.bitcoin.it/wiki/Atomic_swap