Ok, so apparently this is in fact Kleopatra's way of saying the file is fine, it returns this error because I've not signed Noah's key or some issue like that. It seemed strange that there should be a state between pass and fail for this test.
See this post for more info on issues with Kleopatra's verification states.Does anyone know of any other tools for checking this stuff? I find GPG Win and Kleopatra a bit clunky to say the least, plus they keep crashing on basic operations so I'm always having to restart them.
The wallet should be able to do that, perhaps you can find a light one for this purpose.
Edit: I use keybase and did not expect they to let you verify messages of something else than xlm, thanks for pointing out.