As I envision it, each Bitcoin address could issue claims on its value simply by signing a claim with its private key.
Yes, you can always do that. You can use Bitcoin Message, or you can sign things in the signet way, by signing a transaction that is invalid on the main network, using
BIP-322.
Claims could have restrictions on who could claim it, under what conditions, etc. (Much of this machinery already exists in Bitcoin's script language, if I understand correctly.)
Exactly. For that reason, Bitcoin Message is not enough, and you should have an ability to sign any scripts. You can do that by signing transactions, that would be valid only in your network.
Of course, this is essentially what the Lightning Network does.
Yes, you could get a sidechain-like system, just by collecting Lightning Network transactions, and putting them on your own chain. Then, some problems are solved, for example then you don't need any watchtower, and you can batch transactions, by observing the network, and by signing cut-through transactions, simplifying A->B->C on-chain chains into A->C transactions, when all parties agree, and when all penalty transactions are prepared (and automatically unlocked, if any old state will be broadcasted, so all honest nodes will act as a one huge watchtower, active 24/7).
If there are insufficient funds, the transfer does not occur, and the failed claim remains on the blockchain as a blackmark on the address's record.
Yes, there are many options. You can release a penalty transaction, or you can just blacklist some output in a publicly known way, by sending some signed transaction to everyone as a proof that something bad happened. It is up to you, how you will handle cheating in your network.