Jameson Lopp's multisig hardware wallet signing test

[Pmalek]

Jameson Lopp carried out an interesting test to check how different hardware wallets performed when signing multisig transactions of various complexities. Each hardware wallet has its own limitations, so the question is, which ones are the best for multisig transactions involving 10 and 100 inputs?

First, here is a list of the hardware wallets he tested:

• Blockstream Jade
• Cobo Vault
• Coinkite Coldcard Mk4
• Coinkite Coldcard Q
• Foundation Passport Founder’s Edition
• Keystone Pro
• Keystone 3 Pro
• Ledger Nano S
• Ledger Nano S Plus
• SeedSigner
• BitBox02
• Specter DIY
• Trezor Model One
• Trezor Model T
• Trezor Safe 3

*More information on firmware and software versions in the source.

The tests were done using Sparrow Wallet 1.8.2 and Electrum 4.5.3 (Ledger only) on Ubuntu 22.04.
Jameson created native segwit multisig wallets on the hardware devices he tested. He funded each wallet with 100 transactions to create 100 UTXOs. Everything was done on Testnet.


Testing Results

2-of-3 multisig:

The first test was for 2-of-3 multisig. He made two transactions: the first one spent 10 UTXOs, while the second one spent all 100 UTXOs.

Here are the results:



We can see that Seedsigner is the absolute winner, with the best results. Coldcard Mk4, Cobo Vault, and Keystone Pro also performed decently.
Ledger Nano S and Ledger Nano S Plus performed the worst using the HW1 library, with the Nano S granddad requiring almost two hours to sign the 100-input transaction. The results were better using Electrum.
The Passport Founder's Edition didn't have enough memory to complete the second test but did well for the first transaction. Jameson noted he used outdated firmware and will perform a second test with the newest version.


3-of-5 multisig

The second test was for a 3-of-5 multisig. Again, he made two transactions: one spending 10 UTXOs, while the second spent all 100 UTXOs.

Here are the results:



Seedsigner was again ahead of the pack, with Cobo Vault and Keystone Pro being the runners-up.
Just like with the first test, Ledger Nano S and Nano S Plus had terrible results using the HW1 library. It's a different picture when using Electrum, but it's still not good enough.
The Passport Founder's Edition again failed to sign the 100 inputs multisig transaction.


Scaling performance test:

The last test compares each device against each other. The aim is to show if the hardware wallet scales linearly. That means that the 100-input transaction should take 10 times as long as the 10-input transaction. If it scales poorly, it will take longer. If it scales well, it will take less time.

Here are the results:



The overall winners in this category are Bitbox 02, Keystone Pro, and Trezor Safe 3.
The Ledger Nano S/Nano S Plus didn't perform bad on Electum, but their results using HWI are the worst of all tested models.
The Passport Founder's Edition couldn't be assessed for the previously mentioned reasons.


I recommend you read the rest of Jameson's report in his blog post. He discusses a major downside of the Seedsigner, despite the excellent performance during his tests. He also offers additional notes on Ledger's performance and the reasons for the bad results.


Source:
https://blog.casa.io/bitcoin-multisig-hardware-signing-performance-2024/

[goldkingcoiner]

Wow I really did not expect Nano to perform so badly...

[satscraper]

The Passport Founder's Edition didn't have enough memory to complete the second test but did well for the first transaction. Jameson noted he used outdated firmware and will perform a second test with the newest version.

Founder's Edition is  their obsolete device  which has  been superseded by Passport batch 2.  I think the latter has advanced hardware which would perform better at the tests. Once or twice, I  had  signed with Passport 2 the 2 of 2 multisig transaction with around  of twenty  inputs and wallet has coped with the task.

[Charles-Tim]

The Passport Founder's Edition didn't have enough memory to complete the second test but did well for the first transaction. Jameson noted he used outdated firmware and will perform a second test with the newest version.

I am interested to see him carry out another test on Passport hardware wallet. Not using the old wallet which has been replaced and also using the newest firmware. Everything about Trezor 3 are very new which provided good results. I still only prefer airgapped hardware wallet which will make me feel secure that my hardware wallet is completely offline with signing of PSBT with the use of QR code.

[satscraper]

and the reasons for the bad results.

His research is not obvious , I think, as it lacks  two important details i.e. 1) the signing mass of each UTXO serving as input in transactions for every wallet he tested 2) the attitude of each HW to inputs it received for signing. i.e. whether PSBT file  must specify    the full previous transaction or some wallets are content to  just the previous output rather than full relevant transaction.

Results would be different.

More on the signing mass  is here.

[Pmalek]

Founder's Edition is  their obsolete device  which has  been superseded by Passport batch 2.

Obsolete or not, it's a genuine device which was on-sale and it wasn't cheap, and some people purchased it. The Nano S is also obsolete but still found its way into this multisig test.

Everything about Trezor 3 are very new which provided good results.

Actually, if you remove the Ledgers from the equation which were the worst performers, you will see that Trezor's devices take the last place as the second-slowest for signing multisig transactions.

His research is not obvious , I think, as it lacks  two important details i.e. 1) the signing mass of each UTXO serving as input in transactions for every wallet he tested...

This is the first time I hear about the term "signing mass." I did a quick search on Ninjastic Space and didn't find anything useful there. A search using "site:bitcointalk.org signing mass" also finds nothing relevant. I wouldn't consider it an important factor or a topic that's been researched enough.

[SFR10]

The Passport Founder's Edition didn't have enough memory to complete the second test but did well for the first transaction. Jameson noted he used outdated firmware and will perform a second test with the newest version.

Founder's Edition is  their obsolete device  which has  been superseded by Passport batch 2.  I think the latter has advanced hardware which would perform better at the tests.

I have no idea what random access memory the above devices have ^{[assuming that's what Jameson Lopp is referring to]}, but I remember Foundation released a fix to improve its memory allocation for signing large PSBTs and even though at that time it appeared to be for batch 2, I'm seeing now they've merged the release notes for both of the devices ^{[e.g. in "release notes of batch 2", there are also binary files for the founder's edition]} so I expect the Founder's edition to at least not fail the second test ^{[with the latest firmware]}.

[satscraper]

His research is not obvious , I think, as it lacks  two important details i.e. 1) the signing mass of each UTXO serving as input in transactions for every wallet he tested...

This is the first time I hear about the term "signing mass." I did a quick search on Ninjastic Space and didn't find anything useful there. A search using "site:bitcointalk.org signing mass" also finds nothing relevant. I wouldn't consider it an important factor or a topic that's been researched enough.

You could be abreast after reading this stuff.

All I was saying is that the wallets under discussion would be at nonequivalent conditions during their testing should some of them encountered transactions with enormous signing mass which depends on the complexity of the the previous transactions that provide UTXOs.

[Pmalek]

All I was saying is that the wallets under discussion would be at nonequivalent conditions during their testing should some of them encountered transactions with enormous signing mass which depends on the complexity of the the previous transactions that provide UTXOs.

Perhaps, but I can't comment on that. Regarding that signing mass blog post by Tom Honzik, do you know any critiques of his work? Do experts agree or disagree with him about the phenomena of signing mass? Is anyone else talking about it?
Also, these multisig tests that Jameson made were all carried out in a testnet environment. So, who knows what kind of affect an UTXO's prior history has in those conditions. 

[satscraper]

Do experts agree or disagree with him about the phenomena of signing mass?

You don't have to travel far afield and could    see it for yourself by signing two different transactions with single input  and output but differing in the ways  which deliver their   UTXOs in previous transactions. Let's say one UTXO is secured by 100 inputs while the other - by a couple of them. The difference would be drastic if you have done this.