My Interview with Famous Hardware Hacker Joe Grand aka Kingpin

[Pmalek]

I was asking about OneKey wallet because they use forked Trezor code, but I guess he earned good money from his previous gigs like this.

Yeah, I am sure he has been doing this longer than we can imagine. He probably has some regular or semi-regular work going on. I know he travels a lot and is a guest speaker and lecturer on conferences and security workshops. Plus there is the new company that focuses on hardware wallet hacking.

I just checked his youtube channel and I saw he was working on his thinnest boombox project... he is a bit weird, but I like his style.

Yeah, that one came out a few months ago. I laughed during the scene when he met those kids in the street and cursed.

[Pmalek]

Hardware hacker Joe Grand is back with another hacking video:
I hacked time to recover $3 million from a Bitcoin software wallet

This time, he managed to hack a software wallet that had been inaccessible since 2013. His client created a password using the password generator RoboForm (closed-source). It was a 20-character pass with (what the client believed were) upper and lower-case letters, numbers, and special characters.

He used the generated password as a passphrase for his wallet and also saved it in an encrypted partition using TrueCrypt on his computer. This partition later got corrupted, so he couldn't access the data.

The value of the 43.6 BTC at the time was a couple of thousand euros. With the current price, that's around $3 million.

With the help of a software hacker named Bruno, the pair decided to reverse-engineer RoboForm instead of trying to bruteforce a 20-character password.
The following is a quick summary of the video with spoilers. Don't read it if you want to watch the video first!


The experiment proved that RoboForm wasn't generating random passwords in the past (maybe it still isn't).
By going through the changelog of the software, they discovered that RoboForm developers increased the randomness of generated passwords in 2015. That was a hint that previous releases weren't random enough.

Joe and Bruno's target became the part of the code responsible for password generation. They discovered that the password generated in the GUI is stored in memory. With the help of Ghidra, they disassembled the code to find where the password generation happens.

While looking through the code, they noticed a reference to system time. The function was used in combination with the random number generator to generate passwords. They figured out that by changing the time, RoboForm would generate predictable passwords.   

Using a debugger, they found out that by changing the time value in the code, they could trick the software into generating passwords from the past.

They created a piece of code to change the system time and save each generated password. The client gave them a timeframe when he believed he had created his password, and they started retrieving old passwords. It ended up being millions of passwords.

After some additional work and tweaking, they discovered the correct password and recovered the lost bitcoin.


Link to the video:
I hacked time to recover $3 million from a Bitcoin software wallet

[dkbit98]

Hardware hacker Joe Grand is back with another hacking video:
I hacked time to recover $3 million from a Bitcoin software wallet

Another self-commercial for his, but it was really interesting video to watch, especially for people who think that something is random just because some software claims it's random.
In this case guy who owned bitcoin was happy to recover access to his coins, but I can only imagine how many people are using even weaker passwords.
Another good reason why everyone should also have offline physical backup is corruption of hard drives, this happens more often than people think.

[Pmalek]

<Snip>

I can't shake the feeling that the software was configured to generate predictive passwords by accident. That "bug" was also an excellent opportunity for someone to keep records of all the generated passwords, just in case they needed them for something. The change log says they improved their password generation systems, but I wonder if they still have a way to know who generated what and when!?

[Pmalek]

Joe Grand has released a two-part video series where he is hacking hardware wallets, attempting to recover lost coins for his clients. I haven't seen the whole thing yet, but it seems that his sole focus in this video is on Trezor One's, a device he has had success with in the past. Multiple people with similar stories: they lost PINs or seeds.

You can watch it here if you are interested:
https://www.youtube.com/watch?v=MhJoJRqJ0Wc

[Dump3er]

Joe Grand has released a two-part video series where he is hacking hardware wallets, attempting to recover lost coins for his clients. I haven't seen the whole thing yet, but it seems that his sole focus in this video is on Trezor One's, a device he has had success with in the past. Multiple people with similar stories: they lost PINs or seeds.

You can watch it here if you are interested:
https://www.youtube.com/watch?v=MhJoJRqJ0Wc

Thanks for sharing this video, this man is a legend!

I didn't know about him before I found out about this thanks to your post and it is unbelievable to see what he is able to do to solve the problems. Or not problems as he said.

He said "ok this is not a problem, it's a puzzle, let's figure this out, I have the tools, I have the skills.". What I found inconceivable is how he scanned over that chip with his scanning machine to find weak spots that allow him to get into it. I still don't understand it technically, but this is how he described it. Those chips have weaker spots and stronger spots.

When you see a video like this, you would often times think that it must be staged to some degree and some parts are fake, but at 8:30 you can see that the woman is not an actor. Her eyes are so serious and the trouble is real!  The same for all the other guys, the one almost collapsing when he found out his friend has stolen it all. It's heart-breaking and I think everyone who has lost a valuable thing in life knows what it must mean to a person losing a fortune when they are not ultra rich. 

And I like the guy at 15:00 when he was asked whether he would have sold earlier if his coins weren't believed to be lost. He immediately answered yes, he would have sold at a lower price and now he is happy that he couldn't, but could recover the coins.

Joe Grand says a lot of true things about his job in regards to emotional pressure because he can change lives with his skills, but can as well find out that there is a soul falling apart on the other side when they find out it can't either be done or the wallet has been emptied by someone else apparently. Or when he accidentally destroys a wallet. The scene where the couple that lost their newborn got the money back, it must feel overwhelming if you are the one to have recovered that wallet with the best news possible. Everything is there and appreciated in price.

The amount of work that Grand puts into preparation, do test wallet after test wallet to make 99.9% sure that it is not him who messes up a real wallet is crazy. He is one of those guys I believe deserve to be rich and what I liked so much about him during the video is that you can tell from his body language that he would never scam anyone. A good portion of his motivation is empathy and altruism and this is why he says there is serious mental pressure for him. I enjoyed the video because of his mood and passion and authenticity. 

Thanks Pmalek and it is a video I will have in the back of my head from now on because there are people who mess up their hardware wallets.

For all the real crypto guys around here, I think this video is a must watch!

[Pmalek]

<Snip>

I am glad you enjoyed it. Just for the info, there is a second part to that recovery video. It's actually a two-parter. You can watch the conclusion here.
Joe seems to have become an expert in physically manipulating the Trezor One. He knows his way around its chip very well at this point.

I hope we will see more of his great work in the future.

[Dump3er]

<Snip>

I am glad you enjoyed it. Just for the info, there is a second part to that recovery video. It's actually a two-parter. You can watch the conclusion here.
Joe seems to have become an expert in physically manipulating the Trezor One. He knows his way around its chip very well at this point.

I hope we will see more of his great work in the future.

Yes I will watch the second one shortly, but if there is anyone knowledgeable around here, or maybe you know it, how does scanning the chip mini fraction by mini fraction and then find a weak spot? I can't get my head around that why a weak spot gives him some type of access or connection? I think he said is filtering the chip structure in regards to material and that left me puzzled how this could be an identifiable backdoor. I thought it is all about algorithms and encoding, but can the quality or consistency of the material offer a vector for attacks?

[Pmalek]

Yes I will watch the second one shortly, but if there is anyone knowledgeable around here, or maybe you know it, how does scanning the chip mini fraction by mini fraction and then find a weak spot? I can't get my head around that why a weak spot gives him some type of access or connection? I think he said is filtering the chip structure in regards to material and that left me puzzled how this could be an identifiable backdoor. I thought it is all about algorithms and encoding, but can the quality or consistency of the material offer a vector for attacks?

I think the technique he uses is called Fault Injection. He uses instruments in his lab to manipulate the power on the chip to make it do something it's not supposed to. To instigate a "fault." Hence the name, Fault Injection. The goal is to make the chip leak the data it holds. He wants it to leak the PIN to unlock the device and the recovery seed. If he gets the recovery phrase, he has what is needed to recover a client's wallet, assuming there isn't also a passphrase in play or an unusual derivation path that traditional scanning can't identify.

Because the old Trezor One doesn't have a secure element, Joe has learned how to extract the needed data from the memory using the attacks he performs. I am guessing that the same type of manipulations wouldn't be possible on newer hardware wallets that are equipped with secure element chips.

[Dump3er]

-

I think the technique he uses is called Fault Injection. He uses instruments in his lab to manipulate the power on the chip to make it do something it's not supposed to. To instigate a "fault." Hence the name, Fault Injection. The goal is to make the chip leak the data it holds. He wants it to leak the PIN to unlock the device and the recovery seed. If he gets the recovery phrase, he has what is needed to recover a client's wallet, assuming there isn't also a passphrase in play or an unusual derivation path that traditional scanning can't identify.

Because the old Trezor One doesn't have a secure element, Joe has learned how to extract the needed data from the memory using the attacks he performs. I am guessing that the same type of manipulations wouldn't be possible on newer hardware wallets that are equipped with secure element chips.

Hi Pmalek,

I watched the first video again to try and get a better understanding what this guy is doing and now I think I have a rough idea. There is an industry that revolves electromagnetic current to test material for damages and abnormalities. The induced magnetic field changes when spots are discovered that are different from the norm of a whole piece. Say if you have a quadratic piece of steel and you induce a magnetic field at every pixel of that field, the magnetic field won't always be the same and where it changes, you can interpret a weakness in the material even though not visible, or a dent, or some other inconsistency. I think this is what he is doing when he tries to detect a weak spot.

When he discovered a weak spot, in the next step he lets the chip perform a security task like validating a PIN and then injects the fault that can come in many forms, but one of them is to adjust the electrical current and that causes the chip to glitch during the security check for a split of a second and his software setup catches it. One glitch can be that it reveals the real PIN.

It's crazy and hard to understand why the chip reveals information or skips steps, but one explanation, which I think makes a lot of sense, is that chips are made to execute consistently in voltage, clock timing and execution of instruction orders. If that is interrupted, it means some transistors get out of order and that leads to orders being skipped or accidentally granting access to encryption keys. Incredibly interesting.

It took me some time to get down to the second video, but I got goose bumps at 8:00. This young guy lost a fortune because his friends lived in his place and threw away the seed phrase accidentally. It was so bad that the friend had suicidal thoughts and the young guy fell into depression. Joe cracked the Trezor and recovered that money. I am still jealous and I wish I would have his skills, he is blessed. He is such an impressive and humble person, wearing these lumberjack shirts to meet people at the airport and transfer them their cryptocurrencies in person.

Will watch the rest of the second video now.

[Pmalek]

<Snip>

I think Joe has real passion for the work he does. It's not only about the money and his share of the spoils. We hear people say all the time how they love to help others and bla, bla, bla. But it sounds so insincere and fake. I think Joe really means it when he says it. He also helps people change the way they view hackers. The average person probably sees them as bad and evil characters who sit in their dark basements, looking to exploit anything so they can and steal and hurt people. But we see a different story here. A story of someone who uses their hacking skills to give back to those who have suffered losses in the past. It's nice to see.

[Dump3er]

<Snip>

I think Joe has real passion for the work he does. It's not only about the money and his share of the spoils. We hear people say all the time how they love to help others and bla, bla, bla. But it sounds so insincere and fake. I think Joe really means it when he says it. He also helps people change the way they view hackers. The average person probably sees them as bad and evil characters who sit in their dark basements, looking to exploit anything so they can and steal and hurt people. But we see a different story here. A story of someone who uses their hacking skills to give back to those who have suffered losses in the past. It's nice to see.

I couldn't agree more and frankly, I might have said it myself when eventually I had to admit at some point I couldn't do much about another person's situation. Motivation was there, goodwill was there, results were not.  I think some people don't say it with bad intentions, but maybe overestimate their ability to really resolve an issue for someone else (or even their own issues and don't get help...).

In Joe's case it is so unbelievably tangible and you need a profound skillset to solve the problems he solves. It's not like mowing the grass in the neighbors yard. Joe said himself that he feels like a surgeon every now and then in regards to the mental burden he carries and how he perceives the people approaching him. They are often like patients, mentally broken to a point where dark thoughts flood their minds and in that respect, I fully agree with Joe. The young guy I mentioned in my post above is the best example. Just listen to his voice again, he was done and he said despite having the money back now, he is still living through this dark emotional world of the past.

Since I have been there although not on the same financial level, I know what they are talking about, Joe and his "patients" It is tough when you lose anything of life-changing quality. Can be money, can be a loved one, can be a piece of art if you will. And Joe is able to bring back lost things that were thought to not be recoverable.

The last part of the second video is like a nail biting thriller, when it is about those &44 million. Securities there with guns and tasers, Joe doing the work while clients are present, everyone of them is blurred. What I like about Joe's videos is that they are authentic. He could as well choose only the successful cases, but he presents the audience whatever is most important for them to get to know. This last guy has been either scammed or the friend accidentally wiped the Trezor. Both is tough if you lose $66 million.

Alright, thanks for sharing again and if one day a Trezor coincidentally falls into my hands and it is a model Joe can hack, I know where to go.

[Pmalek]

Alright, thanks for sharing again and if one day a Trezor coincidentally falls into my hands and it is a model Joe can hack, I know where to go.

I wouldn't wish on anyone that they have to go and seek out Joe, unless it's a private and friendly visit. If it isn't and you are going to meet him for business purposes, then that means that your or someone close to you committed serious mistakes and you are close to losing your money. It's best to not be in that position. Still, it's some comfort knowing there may be a way out.   

[Dump3er]

I have checked his Youtube channel from time to time whether any new video was uploaded, but he isn't uploading very often.

Do you think he is onto the next device trying to find exploits? As great as it is that he was able to find that exploit and therefore recover funds, all the relevant hardware manufacturers probably have seen his video or got their attention drawn to it. That gives them insights and the possibility to fix that stuff.

Now I am wondering what is the chance that he will find exploits on updates hardware devices and hence be able to recover funds? With every video explaining how he does things, the chances go down for future losses to be recovered I would assume.

[Pmalek]

I have checked his Youtube channel from time to time whether any new video was uploaded, but he isn't uploading very often.

Do you think he is onto the next device trying to find exploits?

When I spoke with Joe a few years ago, I got the impression that he was working on hacking the devices of a different and popular hardware wallet manufacturer. So not just Trezor, but another brand. There were speculations, someone wrote about it, I asked him, but he didn't want to confirm nor deny anything. We have yet to see another hardware wallet hacking video that doesn't involve a Trezor. Don't forget that the majority of hardware wallets nowadays have secure element chips. Their purpose is to prevent someone from accessing the private data in the way Joe does it on the Trezor One.

Making these educational videos isn't Joe's main occupation, so it's normal that the period between two uploads can be quite long. I am sure that his company helps people from the shadows without the spotlights always on them and that's more important.

[Dump3er]

I have checked his Youtube channel from time to time whether any new video was uploaded, but he isn't uploading very often.

Do you think he is onto the next device trying to find exploits?

When I spoke with Joe a few years ago, I got the impression that he was working on hacking the devices of a different and popular hardware wallet manufacturer. So not just Trezor, but another brand. There were speculations, someone wrote about it, I asked him, but he didn't want to confirm nor deny anything. We have yet to see another hardware wallet hacking video that doesn't involve a Trezor. Don't forget that the majority of hardware wallets nowadays have secure element chips. Their purpose is to prevent someone from accessing the private data in the way Joe does it on the Trezor One.

Making these educational videos isn't Joe's main occupation, so it's normal that the period between two uploads can be quite long. I am sure that his company helps people from the shadows without the spotlights always on them and that's more important.

It is not easy to come up with any idea how he could hack another hardware wallet once this fault injection method was published and then hopefully considered by hardware wallet manufacturers. But I think those videos on his channel show that he has a very special talent to come up with potential security vulnerabilities and if he does again, it would blow my mind. You have to have such a deep understanding of those mechanics because you can't just be wildly guessing. He is probably narrowing down the space for hardware or software production failures far more methodically than one would think. At first I thought he is most likely approaching this with trial and error, but his knowledge about all that stuff is incredible.

I am sure that his company helps people from the shadows without the spotlights always on them and that's more important.

I thought hacking these wallets is one of his major income sources, meaning that he takes a cut in case of a successful recovery of funds and he only put together a small number of cases in those videos. or is hacking those wallets more of a sidekick to him? Since he said in the interview that he wouldn't call himself a programmer, what is the main revenue source for his company?

[Pmalek]

I thought hacking these wallets is one of his major income sources, meaning that he takes a cut in case of a successful recovery of funds and he only put together a small number of cases in those videos. or is hacking those wallets more of a sidekick to him? Since he said in the interview that he wouldn't call himself a programmer, what is the main revenue source for his company?

If you are talking about Offspec, then that company's purpose is helping people with lost passwords and recovering data and private keys for both software and hardware wallets. That's what that team does. Offspec takes a cut of the recovered funds if the work they do is successful. So they do hack wallets and recover crypto but don't always make videos about it. Joe is also involved with other products around cybersecurity, engineering, and development.

[Pmalek]

Reverse Engineering a Ledger Nano X Hardware Implant

A crypto user from Thailand had a Ledger Nano X hardware wallet, but the internal battery wasn't working properly. He decided to buy another Nano X, the cheapest one he could find, in order to take its battery and replace the one in his device. He bought one from Lazada. Lazada is an official Ledger reseller for Thailand, but it turns out that the service had (maybe it still has) fake listings as well.

When he got his device, he opened it, noticed that the battery was smaller, and that the circuitry and wires looked different from his device. He posted about it on Reddit and went to the police in Thailand, but nothing was done about it. So, he found Joe and sent him the newly bought wallet for inspection.

Here is an image of what the user found after opening the bought and modified Nano X:


Jos also purchased multiple modified Ledger wallets from Lazada to inspect them.

Here is a close-up of one such device:


And a comparison between a legit Nano X at the top and a modified one at the bottom:


This is the back side of the implant:


Joe started taking them apart to figure out how they were working. He noticed there was an antenna, which signals potential wireless capabilities. He found out that the implant sends something over a cellular network. Joe accessed and dumped the flash memory of the microchip on the implant.

He also discovered that the implant had an eSIM on it. Joe extracted the data from the eSIM, hoping to find information about the person/group who created the implant.  He learned that it was a data-only eSIM that was originally registered in the Netherlands through Vodafone.

Here is another picture of the entire implant with information about the individual components:


Through reverse engineering, Joe found out that the implant has a storage area where it stores recovery phrases. The implant monitors the Nano X’s screen. When the user starts configuring their wallet and generates the seed, the implant logs the data, moves it to flash, and sends it over a cellular network to the hacker.

Joe contacted Ledger and informed them about his findings. They exchanged some information and told him they have had a similar experience with such implants in Ledger devices as well. They also told him that their future devices will have better mitigations against such attacks.

[jahead]

[...]
A crypto user from Thailand had a Ledger Nano X hardware wallet, but the internal battery wasn't working properly. He decided to buy another Nano X, the cheapest one he could find, in order to take its battery and replace the one in his device. He bought one from Lazada. Lazada is an official Ledger reseller for Thailand, but it turns out that the service had (maybe it still has) fake listings as well.
[...]

It would be really interesting to follow the lead regarding sales on "Lazada" again.
After a quick search, I got a better idea of what this platform is. Lazada is one of the biggest online shopping platforms in Southeast Asia and can best be described as the "Amazon of Asia."  
This isnt a huge surprise, but somehow the distribution channels must have been compromised, or the Ledger was sold by a third-party seller on Lazada. Either way, its definitely worth looking into, especially considering how much effort went into tampering with and sabotaging the fake Ledger. Crazy story somehow.

[Pmalek]

It would be really interesting to follow the lead regarding sales on "Lazada" again.
...
Either way, its definitely worth looking into, especially considering how much effort went into tampering with and sabotaging the fake Ledger. Crazy story somehow.

That's easy to do. I visited Lazada for both Thailand and Philippines. If you search for "Ledger/Trezor hardware wallet" on either of the two websites, you will get thousands of results. Of course, not everything is related to hardware wallets and those two brands, but the best matches are. There are many listings. Some are from official resellers for sure but I am also certain there are plenty of non-affiliated third party deals and probably manipulated devices as well. Pretty dangerous online shopping location.