Large-scale Coldcard compromise (1360.23 BTC stolen so far)

[vapourminer]

Won't be easy for the thieves to spend the stolen coins. Will take a lot of mixing.

Spending the coins may not be their goal.

Hurting BTC may be the goal.

Just think If ledger and their recovery system is going to be cracked in a few weeks by the same guy.

if they had a serious shot at ledger would not they of been the better target? is their customer base larger? ie you only get the surprise factor once, why waste it for "paltry" 1000+ btc with coldcards when a ledger recovery have could of had 10 times that as a damage payoff?

but of course its "all hackers on deck" now.

im on team dice as far as entropy is concerned but ngl ima gonna start getting nervous if this keeps up.

[Meuserna]

Have you seen this news yet? https://blockonomi.com/coldcard-bitcoin-theft-hits-88-6m-as-third-attack-wave-emerges/

The third wave of theft from Coldcard devices has occurred. Now, a hacker (or group of hackers) has gained access to wallets with smaller balances (approximately 0.1BTC) and withdrew (by combining wallets into one transaction) approximately ~$13 million \ near 208 BTC (a considerable sum from wallets with small balances) to different wallets (change of tactics?).

I'm surprised that after the first wave of thefts, Coldcard wallets owners didn't move their cryptoassets to other HW devices or wallets generated with the new seed phrase (after Coldcard firmware update). That's why they're now in trouble. They either weren't aware of what was happening (the first wave and the vulnerability) or ignored the possibility of a repeat theft.

The hacker turned out to be quite cunning and completely cleared out BTC-wallets, gradually, in several stages (like a hungry fox that found itself in a henhouse).

It's different hackers now.

Once the exploit became known, hackers worldwide surely started swooping in, going after ColdCard wallets since the pool of possible seed phrases is relatively small and easy to duplicate.

if they had a serious shot at ledger would not they of been the better target? is their customer base larger? ie you only get the surprise factor once, why waste it for "paltry" 1000+ btc with coldcards when a ledger recovery have could of had 10 times that as a damage payoff?

We have to assume hackers are targeting every wallet, looking for any kind of exploit.

Eventually, I expect Ledger's key extraction API to get hacked, and just like this ColdCard situation, we'll probably be shocked by what the specific vulnerability ends up being.

[cAPSLOCK]

It may be worth noting if you have a coldcard and you have updated the firmware to the latest patched version, there have been reports that the device is being bricked by errors in the new firmware.

I have seen one that I own, turn itself off in the middle of various menu navigation twice. I'm leaving it off for now.

https://x.com/i/status/2083633778572787862

Whee.

Anyone that has the wallet but not affected should transfer all his coin from that wallet to another wallet entirely and the person should not use the wallet again.

Those that were affected can keep the wallet in case the hacker is later known and arrested. It can make their legal coin recovery faster. But for new coins they should also stop using the wallet.

There are open sorice wallets that can be used.

I actually only know one person who uses a cold card device.  And I've already helped him secure his funds. Otherwise, I'd be very busy right now. Lol.

Won't be easy for the thieves to spend the stolen coins. Will take a lot of mixing.

Spending the coins may not be their goal.

Hurting BTC may be the goal.

Just think If ledger and their recovery system is going to be cracked in a few weeks by the same guy.

if they had a serious shot at ledger would not they of been the better target? is their customer base larger? ie you only get the surprise factor once, why waste it for "paltry" 1000+ btc with coldcards when a ledger recovery have could of had 10 times that as a damage payoff?

but of course its "all hackers on deck" now.

im on team dice as far as entropy is concerned but ngl ima gonna start getting nervous if this keeps up.

I think it is sure to keep up until it becomes significantly too expensive to be worth doing anymore.

It would be interesting to run numbers on the theoretical costs for the attack. The low-hanging fruit is mostly gone at this point. I imagine some GPU farms that are for rent have noticed a lot of traffic. I imagine prices on that kind of thing arising all of a sudden.

[Ambatman]

Spending the coins may not be their goal.

Hurting BTC may be the goal.

Something similar happened last year but I guess the amount wasn't alarming nor the popularity.
This doesn't necessarily just hurt Bitcoin it pushes to believe that self custody isn't as safe as an ETF would.

Just think If ledger and their recovery system is going to be cracked in a few weeks by the same guy.

I believe even with their lackluster actions
Everybody would be on their tiptoe in trying to check for loose ends
And on the other side many would be looking for bugs to exploit
It's definitely going to be a bumpy ride.

[Somegory]

Spending the coins may not be their goal.

Hurting BTC may be the goal.

Something similar happened last year but I guess the amount wasn't alarming nor the popularity.
This doesn't necessarily just hurt Bitcoin it pushes to believe that self custody isn't as safe as an ETF would.

Just think If ledger and their recovery system is going to be cracked in a few weeks by the same guy.

I believe even with their lackluster actions
Everybody would be on their tiptoe in trying to check for loose ends
And on the other side many would be looking for bugs to exploit
It's definitely going to be a bumpy ride.

I'm not encouraging this hacker but this might not even be what we all thought.
I could pretend to be another hacker trying to make it look exactly like what you all thinking.

On the other part I am a FBI agent trying to lay my trap so that the hacker doesn't get away one way or the other.
Crazy that I am the only thinking that it could actually be a trap in motion to go after the original hacker.

[shahzadafzal]

Is there an actual or exact failure in what has happened? To correctly do something about it, there should be something definitive to understand the root cause of it.

The coldcard source code attempts to disable the hardware RNG support in micropython because they provide their own implementation, but the handling of the the disable flag is inconsistent.

To disable it they #define MICROPY_HW_ENABLE_RNG (0)   and this successfully disables it.

But to enable their replacement they check #ifndef MICROPY_HW_ENABLE_RNG   ---   and this is ineffective because the if(n)def directive checks only if the flag MICROPY_HW_ENABLE_RNG is defined, its value "(0)" is irrelevant and counts as defined just as any other value would.  Probably any sufficiently long-expirenced C programmer has encountered a form of this #if vs #ifdef confusion.

This inconsistency makes the coldcard firmware not use its own HWRNG support code but instead call the micropython random function.  But the micropython function has its hardware support disabled and when it is disabled it replaces it with a placebo insecure PRNG function which is seeded with part of the devices hardware id and a timer.  "Micropython: We've replaced this users randomness with Folgers Crystals, lets see if they notice".

The replacement function *looks* like good random numbers... but they're not random at all, and provide essentially no security.  However, if you were to try to judge their quality with tools like diehard you would probably get a result that they were high quality -- this the reason RNGs are such a risky part: the quality of a random stream depends on how it was created and can't be detected purely from the random values itself.

So for mk3 the attacker just needs to search the likely hwids and timer values for wallets.

For mk4+ the not-random random value from above gets xored with another instance of the same placebo function (more Folgers Crystals), initialized with a 32-bit presumably secure random value that ultimately comes from another chip inside the wallet.  The 32-bits of extra security is not enough to make it secure, but it may delay attacks by speak-and-spell wielding toddlers.

The use of "fallback" and no-security providing PRNG "whiteners" is a practice that some people have previously identified as risky due to the risk of accidentally hiding more serious bugs.  Unfortunately both the authors of micropython and coldcard itself engaged in this particular sin, and fixing either one alone wouldn't have made the error immediately obvious.  I wouldn't say that belt-and-suspenders like these are unconditionally bad, but incredible care must be taken to be sure that they aren't covering up other issues.

Take my analysis with a grain of salt: it's purely a product of personally reading the source code.  I do understand that expert AI users also independently reached the same conclusion including with validation against a disassembled firmware image, which is why I feel confident enough sharing it.

Edit: This post is also consistent with the new coldcard blogpost: https://blog.coinkite.com/entropy-technical-backgrounder/

Well explained by gmaxwell. 🙂

I had been wondering why a developer would disable the hardware RNG with:

#define MICROPY_HW_ENABLE_RNG (0) ...and then later check for it using #ifndef MICROPY_HW_ENABLE_RNG

As gmaxwell explained, that's the root of the inconsistency. Since #ifndef only checks whether the macro is defined—not its value—setting it to 0 still counts as "defined." As a result, the replacement RNG code was never compiled.

The tweet below mentions another interesting theory from Core Lightning developer ddustin. According to his analysis, the developer may have originally set MICROPY_HW_ENABLE_RNG to 0 as a workaround for a compiler/build issue. A tiny preprocessor mistake then silently routed wallet generation to a much weaker fallback PRNG.

And to make it even more ironic, the commit message was simply: runs

COLDCARD BUG MAY HAVE BEEN CAUSED BY A COMPILER WORKAROUND

A new technical analysis by Core-Lightning dev ddustin suggests the 2021 COLDCARD vulnerability may have started while a developer was trying to connect three layers of the firmware: the wallet’s Python code, MicroPython’s C code, and the STM32 hardware random number generator.

The custom code appears to have conflicted with MicroPython’s existing implementation, likely triggering a compiler error.

The evidence suggests the developer then disabled the hardware RNG by setting MICROPY_HW_ENABLE_RNG to 0, allowing the firmware to compile.

That change had an unintended consequence. When users created new wallets, the firmware no longer used the hardware random number generator. Instead, it fell back to MicroPython’s much weaker Yasmarang software random number generator.

The commit message left by the dev was just the word “runs.”

The analysis argues this is a reminder that developers should never ship security-critical code they don’t fully understand, especially when it protects billions of dollars in Bitcoin.

[bitmover]

if they had a serious shot at ledger would not they of been the better target? is their customer base larger? ie you only get the surprise factor once, why waste it for "paltry" 1000+ btc with coldcards when a ledger recovery have could of had 10 times that as a damage payoff?

We have to assume hackers are targeting every wallet, looking for any kind of exploit.

Eventually, I expect Ledger's key extraction API to get hacked, and just like this ColdCard situation, we'll probably be shocked by what the specific vulnerability ends up being.

I agree that Ledger key extraction feature can be hacked .

But the difference is that the user need to physically authorize the extraction in his device first in order to work.
The user basically accepted that his funds are not only in his control anymore .

I doubt that most people really use that feature..

[PrivacyG]

This is probably the moment we have to realize that MORE hardware wallets, MORE software wallets, MORE platforms to use et cetera are not always a good thing but in fact may make things worse for many.

I was thinking today of how many software wallets are out there and how good some of their design looks while still being used by a low number of people.  And I get it.  Different wallets serve different purposes and for some people ease of use may be important.  But I think we are kind of foolish after all as we nowadays rely on gigantic hardware wallet producers instead of doing things on our own the TRUSTLESS way.

People download the newest wallet out there and then they find out Electrum and Bitcoin Core are much more secure, and before you know they import their Seed in to Electrum continuing to use it for many years with out even considering the risk that the newest wallet they initially had that Seed in has been generated using the wrong, WAY more weak Random Number Generator.  Which years down the line may create even larger disasters than this Coldcard one is!

Now that it may be faster for some people to use 'Artificial Intelligence', even for people who do not understand any code at all, things will speed up a LOT vulnerability wise.  It is going to be a competition of who can exploit and steal the first!  After all.  All that is required is that someone writes enough prompts to make the 'AI' understand what to look after and it may expose a vulnerability no other developer has found, published or exploited yet.

This should be a wake up call for all of us to start thinking twice before generating a Seed using a coded generator instead of generating our own manually, before using even a hardware wallet instead of building our own Airgapped Computer which in my opinion STILL proves to be the best way to hold Bitcoin stashes et cetera.  Think how you could do things with LESS trust every time!  Satoshi did not have a hardware wallet and the Bitcoin that is thought to be his is still intact.  Hardware wallets are only as secure as the developers and manufacturers make them, and situations like these should show that even the best, most well known of them may have catastrophic vulnerabilities.

And before every thing, this should be a gigantic wake up call for anyone who has not seriously considered securing their Bitcoin the right way and is still using hot wallets for their main chunk!

[Wq59wy8]

I heard about people receiving some BTC DUST BTC earlier this year...  might have been part of the planning strategy for this?  I could see that. 

[Forsyth Jones]

And to think that the flaw in the wallet generation code only exists because Novak changed the code's license when the Passport Foundation copied his code to base it on theirs. Passport claims to have reviewed and improved the code, after which, in 100 commits, Coinkite closed the code by adding something called "libNgU," thus introducing the flaw.

Yes, the flaw practically only exists because of Novak's pure and simple temper tantrum due to the emergence of a new competitor who based their code directly on theirs. Novak didn't accept that other people were redistributing his code commercially, completely ignoring the opensource philosophy. With the change in the license, fewer people were motivated to review the code, so the flaw remained "invisible" for 5 years.



This tweet touches on this a lot: https://x.com/KuptoKosmos/status/2083916236002336780

Literally out of pure and simple temper tantrum, the flaw exists. Coldcard's reputation is already tarnished, if they want, they could revert the GPL license as an initial apology.

Those who used strong passphrases and BIP-85 were safe the entire time. Coldcard's audience is advanced, but many didn't even use passphrases.

This is a grotesque failure, the premise of a hardware wallet is to be a vault, and they failed at the basics: entropy generation.

I heard about people receiving some BTC DUST BTC earlier this year...  might have been part of the planning strategy for this?  I could see that.  

That is an interesting theory and it makes perfect sense, I saw it in this tweet.

[decodx]

Just saw this popping up on X... wild prediction from @JWWeatherman all the way back in 2020.



Here is the link to the original tweet: https://x.com/JWWeatherman_/status/1329613163973668865?s=20

[stompix]

I heard about people receiving some BTC DUST BTC earlier this year...  might have been part of the planning strategy for this?  I could see that.  

You can randomly pick drained addresses from the last batch to first and see it's not a thing

https://mempool.space/address/bc1qn9llv9pux95afeyl2dxsdxlwpxdvsts8as8c0u
https://mempool.space/address/bc1qst04y9gy52jls76yhqcz6ueyd6s7djg267780j
https://mempool.space/address/bc1q3yr0m5r9ud3rh9lds7875pmjpchfvj28h29htx
https://mempool.space/address/bc1qqcvspmz2gc3m3zjs9vx3t2ydl57q4ssn54kzva
https://mempool.space/address/bc1qqfda33wwnej3pj8u09s96etk0ad7cmu9n2rrwg
https://mempool.space/address/bc1qxkh5jgtj443pgnr0zt597a4a4tn0ayd5ae2q49

Most of them have only funding and drain tx, only two in total.
Besides, what wold even be the point of that?

Just saw this popping up on X... wild prediction from @JWWeatherman all the way back in 2020.

So their plan to exit scam was to leave a bug in the open for 3 years just to scam $1000 btc...

Eventually, I expect Ledger's key extraction API to get hacked, and just like this ColdCard situation, we'll probably be shocked by what the specific vulnerability ends up being.

People need to take a chill pill! Right now!
It's one thing discussing the bug, feeling sorry for the victims, advising on how to do things right and it's a different thing to act like a paranoid headless chicken.

[decodx]

Just saw this popping up on X... wild prediction from @JWWeatherman all the way back in 2020.

So their plan to exit scam was to leave a bug in the open for 3 years just to scam $1000 btc...

Coinkite said the flawed firmware range begins with version 4.0.1, released in March 2021, so the bug has been out in the open for over 5 years, not 3. As for lost bitcoins, the current counter is at over $86 million worth of BTC at the current rate, not $1000. Anyway, that wasn't the point of my post.

[stompix]

Just saw this popping up on X... wild prediction from @JWWeatherman all the way back in 2020.

So their plan to exit scam was to leave a bug in the open for 3 years just to scam $1000 btc...

Coinkite said the flawed firmware range begins with version 4.0.1, released in March 2021, so the bug has been out in the open for over 5 years, not 3. As for lost bitcoins, the current counter is at over $86 million worth of BTC at the current rate, not $1000. Anyway, that wasn't the point of my post.

Fixed the $ since I left just the BTC amount when I wanted to put the number in $, my mistake.

Now, what was the point of your post?
That a guy on X predicted 6 years ago that in 5 years we would have a hardware wallet company exit scam?
And they would blame it for a bug?

Seriously, if that wasn't your point, which one was it?
Are we playing the Simpsons predicted that here?

[PrivacyG]

I heard about people receiving some BTC DUST BTC earlier this year...  might have been part of the planning strategy for this?  I could see that. 

Could you explain your theory on how that would even work in this particular situation?

-----

like a paranoid headless chicken.

Reading this felt like I was mentioned.  My paranoia has been regurgitated on about half of these pages!

[Meuserna]

It appears the next wave of attackers are churning through passphrases to find wallets, which is only possible because the pool of ColdCard seed phrases is known due to the ColdCard firmware exploit.

IMPORTANT UPDATE: We just had our first confirmed loss of a Mk3 + 2 Word Passphrase.

Drained at 2pm Aug 2nd Australia Time - Roughly 17hrs ago.

-- @BTCsessions

https://x.com/BTCsessions/status/2084024733511921691

I believe ColdCard 2-of-3 multisig wallets will be next. The pool of seeds is known.

I hate the feeling of knowing that hodlers are being robbed, and there's nothing we can do to stop it. It's one thing if people leave their coins on an exchange that goes bust. We've warned them not to do that. But this is happening to people who took self custody seriously enough to use what was considered to be a reputable hardware wallet. It's heartbreaking.

[julerz12]

we lost some users. We can't blame those people. You would probably have the same viewpoint if you lost your entire life-savings held in Bitcoin.

8 years of stacking, gone. I think it's time to move on.
~snip~
To everyone who has lost their BTC, I wish you the best and good health. I hope you find the strength to start again.

This is really sad. I see lots of similar posts like this on r/Bitcoin. These guys have been slowly accumulating BTC for years, 8 to 10 years, or even more! WTF.

https://www.reddit.com/r/Bitcoin/s/BwUY4IfvgN


https://www.reddit.com/r/Bitcoin/s/lnyuXqjxTp


https://www.reddit.com/r/Bitcoin/s/7NGJ5ch0Ml

[philipma1957]

The debate over Coldcard's use of entropy illustrates how cryptographic integrity can be compromised not because an algorithm is insecure, but because there's a hiccup in the implementation. Even though the algorithm itself follows all standards, one mistake at compile-time logic or a tiny bit of conditional compilation could mean the overall security has been weakened unintentionally.

Something I find compelling here is how it is basically impossible to tell that random data isn't random simply by looking at it. The sequence might look good statistically and yet still be guessable if its origins are sufficiently poor. Developers shouldn't trust random number tests exclusively but must understand how a piece of entropy is used and how its sources are treated over the life-span of their firmware.

Another conclusion is to approach with care how we deal with fallbacks. Should secure pieces fail and the code falls back to a lesser one without a notice, it could simply hide the problem as opposed to exposing it. Explicit errors might be a safer option than opaque failure for such firmwares.

This also showcases the importance of code reviews. Those are necessary to enable independent scrutiny and find those edge cases where there are real risks that others may expose only later. As a result, the whole infrastructure and ecosystem become stronger.

Edit: The latest article from Coldcard appears to support the conclusions outlined in this post: https://blog.coinkite.com/entropy-technical-backgrounder/

It was random but from a smaller set of seeds than 256 bits

40 bits is tiny compared to 256 bits.

[gmaxwell]

Yes, but is this a credible error? Why would the developer think "let me use #ifndef rather than #if"?

Yes, it's a common error particularly when working across different codebases with different styles.  In the case of coldcard there were three relevant codebases: micropython, libngu, and the coldcard codebase itself.

Likelyhood with be greater with less experience with C but this is the sort of error that even the most experienced C developers can make:

Both styles-- value and defineness checking-- are commonly used for configuration and you can mix them up.  *Usually* the result of doing so is something that is obviously wrong and gets caught right away.  But because of the PRNG fallback in micropython the bad behavior was difficult to detect because it would look and act like real randomness and pass most tests you might throw at it.

The testing technique that would best detect this would be fault injection (e.g. intentionally break the TRNG and confirm that its broken) but they apparently didn't do that and for MK4+ even that wouldn't work because the additional PRNG would have further blocked detecting it.  -- this isn't just hindsight speaking: it's a criticism I made from just someone mentioning the behavior in marketing copy:

https://www.reddit.com/r/Bitcoin/comments/15lu8ps/comment/jvnn1p4/

but we also maintain a PRNG which is mixed (by XOR) into the TRNG output

FWIW, 'whitening the TRNG' with a trivial LFSR or LCG is a practice that I've seen in some microcontroller libraries and seems really dubious to me for anything where security matters. Often it ends up concealing total misbehavior of the 'hardware' RNG, which can be flaky (e.g. prone to producing useless output due to analog defects). E.g. the hardware flakes out, and all the rng is really putting out is some LCG output, but you don't notice because the LCG output 'looks' random.

(and FWIW, this criticism applies not just to coldcard's code but the underlying 'TRNG' in the STM32 microprocessor,  as per the datasheet it does that kind of potentially failure hiding whitening under the hood).


But tests like fault injection or binary analysis which would be effective are not common.  I don't know if there was ever a formal third party security audit.

[m2017]

Sorry to say but ColdCard just sucks big time.

This situation also demonstrates the need to have a second hadeware wallet, a different model (or, better yet, from a different manufacturer). This would allow you to quickly transfer your assets to another secure (?) wallet and avoid a similar U5=SE2 fakup, where your funds are effectively locked away from the owner (but not the attacker). Of course, you can use the seed phrase (but what if the seed phrase is stored far from the device owner?) and restore access to the wallet not on the hardware wallet, but on, say, electrum, but this also creates certain risks if your PC \ laptop is infected with malware. In short, it's worth remembering the old adage that you should have prepared for such incidents in advance (planned actions in case something like this occurs and prepared).

Now, Coldcard owners are scrambling to save their assets, but as often happens, things are only getting slower due to unforeseen circumstances (like bricking a hardware wallet).

Users were completely unprepared for this kind of attack, as the security of HW devices was taken for granted. It's time to reconsider.