Well I agree that in the "two-phase" blackmail scenario, the argument for Alice to comply is less clear since there's nothing forcing Bob to release the initial transaction once he receives payment from Alice. I would just point out, however, that this type of "two-phase" blackmail is attempted in real life, and the blackmailed party often complies, even if they have no guarantee the blackmailer won't follow through with their threat, or that no future blackmail will occur. In other words, in the face of large potential losses, actors do not necessarily behave rationally. If you concede that this is the case, it may be worth it for Bob to at least attempt a large number of blackmail attacks, hoping he'll find a target that complies with his request. This could result in a loss for all of Bob's targets, including the ones that don't comply. Whether this attack makes sense depends on how costly it is for Bob to initiate an attack, the potential reward of a successful attack, and the distribution of compliant victims.

After thinking about this some more, I agree with this.

I think this is an interesting observation, but this may still be vulnerable to attack. Bob can still hold the transaction hostage, promising only to release it if Alice transfers some amount of BTC to Bob in another, unrelated transaction. Again, Alice is probably motivated to comply to minimize her loss.

If you believe this then your protocol has no purpose. Alice can simply transfer some amount of money to Bob's bank account, and then Bob can transmit an equivalent amount of BTC to Alice, without the need for all this multisig business.

I don't have a complete analysis, but I think that this scheme is vulnerable to blackmail in a way which is impossible to mitigate. Consider the case in which Alice wants to buy X BTC from Bob. The scheme depends on Alice and Bob both depositing BTC into a shared, multisig address, where Alice deposits cA, some collateral, and Bob deposits cB, his collateral, plus X, the amount of BTC to be purchased. Neither Alice nor Bob can withdraw from the multisig address unilaterally, so it is their best interest to come to an agreement on how to disperse the BTC in the transaction.

An attacker, in this case Alice, who had no intention of actually buying BTC, could at this point suspend the process by telling Bob she has no intention of transferring any money. The value of the transaction is X+cA+cB, and Alice proposes dividing the contents of the transaction evenly between Alice and Bob, and publishes her half of a transaction to this effect. At this point Bob has two options, he can either reject the new agreement and suffer a loss of X+cB, or he can accept the new agreement, complete the transaction, and suffer a loss of only 0.5X + 0.5cB - 0.5cA. It is therefore in Bob's best interest to accept the new agreement. The strategy of simply not agreeing to the blackmail is economically irrational.

The primary reason this attack can succeed is because Bob has committed more resources to the shared transaction than Alice has. You might then suggest that the solution is to raise the value of Alice's collateral cA such that it is equal to Bob's total commitment, cA = X+cB. If cA > X+cB, then Alice has no motivation to proceed with the attack because she will only lose BTC in the process.

However, if cA > X+cB, then Bob can become the attacker. After a sincere Alice has initiated the exchange and transferred the money from her bank account to Bob's, we are in the same situation as before. There is a multisig transaction of value X+cA+cB, and Bob can at this point suspend the the process, and demand that the BTC in the transaction be distributed evenly between Alice and Bob. Alice has more BTC committed to the transaction, and must agree to minimize her loss.

The point of this is that there are no values of cA and cB you can select such that one party isn't motivated to blackmail the other party.