Bitcoin Forum
April 27, 2024, 11:12:51 PM *
News: Latest Bitcoin Core release: 27.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Recovered .db files using Testdisk  (Read 207 times)
bobkee (OP)
Newbie
*
Offline Offline

Activity: 3
Merit: 0


View Profile
February 23, 2022, 01:55:53 PM
 #1

Hi,

I ran testdisk (photorec) to recovery wallet files from a formatted usb. Photorec was able to recovery .db files using berkeley signature which is great but now I don't know how to extract just the wallet file out of the .db files and convert it to .dat or make it readable for bitcoin core.
Any suggestions would be greatly appreciated?
1714259571
Hero Member
*
Offline Offline

Posts: 1714259571

View Profile Personal Message (Offline)

Ignore
1714259571
Reply with quote  #2

1714259571
Report to moderator
1714259571
Hero Member
*
Offline Offline

Posts: 1714259571

View Profile Personal Message (Offline)

Ignore
1714259571
Reply with quote  #2

1714259571
Report to moderator
There are several different types of Bitcoin clients. The most secure are full nodes like Bitcoin Core, but full nodes are more resource-heavy, and they must do a lengthy initial syncing process. As a result, lightweight clients with somewhat less security are commonly used.
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
1714259571
Hero Member
*
Offline Offline

Posts: 1714259571

View Profile Personal Message (Offline)

Ignore
1714259571
Reply with quote  #2

1714259571
Report to moderator
1714259571
Hero Member
*
Offline Offline

Posts: 1714259571

View Profile Personal Message (Offline)

Ignore
1714259571
Reply with quote  #2

1714259571
Report to moderator
NotATether
Legendary
*
Offline Offline

Activity: 1582
Merit: 6695


bitcoincleanup.com / bitmixlist.org


View Profile WWW
February 23, 2022, 02:30:03 PM
Merited by NeuroticFish (4), ABCbits (2)
 #2

If you have a Unix (that's Mac or Linux) box, open a terminal and run the file command against the file to check if it's in a format that is used by one of the well-known wallets:

Code:
file full_path_to_file

It will print

"Berkeley DB (Btree, version 9, native byte-order)"

If it's a Bitcoin Core wallet,

"ASCII text, with very long lines"

If it's an Electrum wallet. Any other output means it's either not a wallet file or it was made by obscure wallet software (in particular, file just prints Armory ".wallet" and ".lmdb" files as "data").

.
.BLACKJACK ♠ FUN.
█████████
██████████████
████████████
█████████████████
████████████████▄▄
░█████████████▀░▀▀
██████████████████
░██████████████
████████████████
░██████████████
████████████
███████████████░██
██████████
CRYPTO CASINO &
SPORTS BETTING
▄▄███████▄▄
▄███████████████▄
███████████████████
█████████████████████
███████████████████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████████████
█████████████████████
███████████████████
▀███████████████▀
█████████
.
bobkee (OP)
Newbie
*
Offline Offline

Activity: 3
Merit: 0


View Profile
February 24, 2022, 05:17:11 AM
 #3

Thanks for your reply.

I ran file and it did print

"Berkeley DB (Btree, version 9, native byte-order)"

Then I ran hexdump -C

https://ibb.co/zSgSh2X

So Testdisk (photorec) found the files based on the photorec.sig for -
"Berkeley DB (Btree, version 9, native byte-order)"

And the file command confirmed that is it Berkley DB

So now how do I turn these files into wallet.dat or what can I do testdisk recovered a few GB worth of files in .db format

What I have noticed is the first file will be few KB and out put "b1 main" and the next file will be 741 MB of heaps of data keys pool ect and then again the next file is few KB "b1 main" and so on...if that makes any sense
NotATether
Legendary
*
Offline Offline

Activity: 1582
Merit: 6695


bitcoincleanup.com / bitmixlist.org


View Profile WWW
February 25, 2022, 03:56:11 AM
Merited by ABCbits (1)
 #4

testdisk recovered a few GB worth of files in .db format

Size of wallet generated by Bitcoin Core shouldn't be that big, unless it contain LOTS (could be few million) of address/transaction. If your USB storage is really old (2013 or earlier), it's possible you recovered Berkeley DB files which contain transaction/block on Bitcoin blockchain. Bitcoin Core (used to be called Bitcoin Qt) use Berkeley DB (rather than LevelDB) before Bitcoin Qt 0.8.1 to store transaction/block on Bitcoin blockchain.

That's probably because photorec could not determine the end of the Bitcoin Core wallet file, so it just appended whatever data it thought was part of it until it became that size.

I'm pretty sure there are some magic bytes at the end of wallet.dat files that OP can find to trim (by copying the bytes out to a new file) the wallet size and make it readable by programs, but I have to go manually inspect some wallet.dat's I have first, for this data.

.
.BLACKJACK ♠ FUN.
█████████
██████████████
████████████
█████████████████
████████████████▄▄
░█████████████▀░▀▀
██████████████████
░██████████████
████████████████
░██████████████
████████████
███████████████░██
██████████
CRYPTO CASINO &
SPORTS BETTING
▄▄███████▄▄
▄███████████████▄
███████████████████
█████████████████████
███████████████████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████████████
█████████████████████
███████████████████
▀███████████████▀
█████████
.
bobkee (OP)
Newbie
*
Offline Offline

Activity: 3
Merit: 0


View Profile
February 25, 2022, 06:00:33 PM
 #5


 
I thought magic numbers were only the first bit of a file
But my knowledge is minimal..

Can I confirm what OP is please? And by copying bytes to a new file will it change to a .dat like it originally was or will it stay at a .db?

You are brilliant thank you so much I have been stuck on this for some time now

Appreciate your time so much!
litecoin_messiah
Sr. Member
****
Offline Offline

Activity: 355
Merit: 268



View Profile WWW
February 25, 2022, 08:02:04 PM
 #6

Databases(Berkeley) have more than one database in it, I noticed with photo wreck usually calves them separately if so like they’ll be a small and large Output. The database wouldn’t also load properly and would need manual cleaning.

If you can find the correct headers and config feel free to share I think it’s on Google somewhere


Also dont share any screenshots if they include data from your wallet, especially ones with 01 01 04 20 hex str
NotATether
Legendary
*
Offline Offline

Activity: 1582
Merit: 6695


bitcoincleanup.com / bitmixlist.org


View Profile WWW
February 26, 2022, 11:43:44 AM
 #7


 
I thought magic numbers were only the first bit of a file
But my knowledge is minimal..

Can I confirm what OP is please? And by copying bytes to a new file will it change to a .dat like it originally was or will it stay at a .db?

You are brilliant thank you so much I have been stuck on this for some time now

Appreciate your time so much!

OP is forum-speak for Original Poster.

Copying the bytes, if the correct number of bytes is copied, will produce a valid .db file, otherwise, it'll be still be a .db file but it cannot be opened by applications because of the junk at the end of the file.

A bitcoin core wallet.dat starts with the following bytes:

Code:
000000  \0  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0   b   1 005  \0
000010  \t  \0  \0  \0  \0 020  \0  \0  \0  \t  \0  \0  \0  \0  \0  \0

(Generates using od -Ax -v -c on the wallet.dat file)

And doesn't appear to end with a predictable sequence of bytes, but it's always aligned to a 4096-byte boundary. That means the size will be a multiple of 4KB. Perhaps if I look at some Berkeley DB source code I will find what kind of stuff is written at the end of a file.

.
.BLACKJACK ♠ FUN.
█████████
██████████████
████████████
█████████████████
████████████████▄▄
░█████████████▀░▀▀
██████████████████
░██████████████
████████████████
░██████████████
████████████
███████████████░██
██████████
CRYPTO CASINO &
SPORTS BETTING
▄▄███████▄▄
▄███████████████▄
███████████████████
█████████████████████
███████████████████████
█████████████████████████
█████████████████████████
█████████████████████████
███████████████████████
█████████████████████
███████████████████
▀███████████████▀
█████████
.
kirya
Newbie
*
Offline Offline

Activity: 28
Merit: 1


View Profile
July 10, 2023, 10:45:35 PM
 #8

Are there other good data recovery tools I can try? Just deleted some important files by mistake and now looking for a solution.
litecoin_messiah
Sr. Member
****
Offline Offline

Activity: 355
Merit: 268



View Profile WWW
July 12, 2023, 02:45:43 AM
 #9

found this in my old notes,

Code:
<?xml version="1.0" encoding="UTF-8"?>
<FileTypeList version="2.0">
    <FileType id="50001" group="Other files" description="bitcoin wallets" features="" extension="dat">
        <Begin combine="OR">
            <Signature>\x61\x15\x06\x00</Signature>
            <Signature>\x00\x06\x15\x61</Signature>
            <Signature>\x62\x31\x05\x00</Signature>
            <Signature>\x00\x05\x31\x62</Signature>
            <Signature offset="12">\x61\x15\x06\x00</Signature>
            <Signature offset="12">\x00\x06\x15\x61</Signature>
            <Signature offset="12">\x62\x31\x05\x00</Signature>
            <Signature offset="12">\x00\x05\x31\x62</Signature>
            <Signature offset="12">\x53\x22\x04\x00</Signature>
            <Signature offset="12">\x00\x04\x22\x53</Signature>
            <Signature offset="12">\x88\x09\x04\x00</Signature>
            <Signature offset="12">\x00\x04\x09\x88</Signature>
        </Begin>
        <End combine="OR"/>
    </FileType>
</FileTypeList>


i think it's supposed to be used in https://www.google.com/search?q=R-Studio


I can't vouch for it, but like i said before if you recover using photorec anything that resembles a wallet database (bdb) it will have a .db extension and be missing the first 8kb or less, that will probably be seperared, the wallet wont read correctly but you can fix it








edit: this post: https://bitcointalk.org/index.php?topic=2637884.0 is also very useful but i have yet to test properly, had to take a break from trying to recover my old coins.


Two years ago I formatted my harddisk and installed Windows 10 on it. Before this I did a backup, but unfortunately the backup was broken. So I lost my wallet.dat, with a few Bitcoins in it. I could restore some files with RStudio, and I had older backups for the rest, but seemed to be that the latest wallet.dat was already overwritten, and I frequently add new addresses. So I gave up, not a big deal, maybe $200 lost. But I didn't use the harddisk and bought a new one.

Fast forward to December 2017: Now a few Bitcoins is some serious money, so I decided to give it another try. I tried any option I could find in RStudio, checking the dozens of filesystems it reported after scanning it for hours (only a few where valid from previous installations), but I couldn't restore it. Ok, this needed some more work.

My assumption was, that the file headers were broken, so I wrote a small C program myself, which scanned the whole harddisk for the wallet.dat signature (testing for the first 16 bytes). The filesystem was NTFS, which has 4k sector sizes and a file starts always at sector start, if I understand it correctly, which makes things easier. Also usually if there is enough space, contiguous sectors are used to save a file. My hope was that somewhere I could find old version of the wallet.dat, but not too old that the new keys were missing.

This is the very simple and straightforward scan program I hacked together:

Code:
#include <stdio.h>
#include <stdint.h>
#include <inttypes.h>
#include <string.h>

uint8_t buf[4096];
char filename[1000];

uint8_t search[] = {
    0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x62, 0x31, 0x05, 0x00
};

int main(int argc, char** argv)
{
    uint64_t pos = 0;
    FILE* f = fopen("/dev/sdd", "rb");
    FILE* w = NULL;
    int walletNumber = 0;
    int walletIndex = 0;
    uint64_t max = 1000000000000ULL;
    while (1) {
        int c = fread(buf, 1, 4096, f);
        if (c != 4096) break;
        if (!w) {
            if (memcmp(search, buf, 16) == 0) {
                sprintf(filename, "wallet%i.dat", walletNumber++);
                walletIndex = 0;
                w = fopen(filename, "wb");
                printf("found: %" PRIu64 "\n", pos);
            }
        }
        if (w) {
            fwrite(buf, 1, 4096, w);
            walletIndex++;
            if (walletIndex == 256) {
                fclose(w);
                w = NULL;
            }
        }
        pos += 4096;
    }
    fclose(f);
    return 0;
}

I used it on Linux as my host system and the old harddisk was visible as /dev/sdd (you can see this with dmesg). I compiled it with "gcc -O3 scan.cpp -o scan" and started it with "sudo ./scan", and a few hours later (it was a 1 TB harddisk) I got a wallet0.dat to wallet9.dat, each 1 MB in size (it doesn't matter if there is crap after the wallet data). This was a nice start Grin

Then I tried to copy it to a wallet.dat of a current Bitcoin installation, but most of the time it said the wallet was corrupt, once it even crashed at start and when it said it could salvage some information, no keys were in it.

My rescue was https://github.com/joric/pywallet This program could decode all files and output it in JSON format. It needs the wallet.dat in a bitcoin-qt installation in the .bitcoin directory. I knew one of my old addresses, so I wrote a script which did test all files (actual key changed) :

Code:
for i in $( ls wallet*.dat ); do
    echo item: $i
    cp $i .bitcoin/wallet.dat
    ./pywallet.py --dumpwallet --datadir=.bitcoin | grep -i 12QDRXssT63Pv5KTGBN2kyAvfLW3s7jxBs
done

The output looked like this:

Code:
item: wallet0.dat
item: wallet10.dat
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.
item: wallet11.dat
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.
item: wallet1.dat
WARNING:root:encrypted wallet, specify password to decrypt
item: wallet2.dat
item: wallet3.dat
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.
item: wallet4.dat
Traceback (most recent call last):
  File "./pywallet.py", line 1706, in <module>
    main()
  File "./pywallet.py", line 1683, in main
    read_wallet(json_db, db_env, True, True, "")
  File "./pywallet.py", line 1556, in read_wallet
    parse_wallet(db, item_callback)
  File "./pywallet.py", line 1287, in parse_wallet
    for (key, value) in db.items():
bsddb.db.DBPageNotFoundError: (-30986, 'BDB0075 DB_PAGE_NOTFOUND: Requested page not found')
item: wallet5.dat
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.
item: wallet6.dat
WARNING:root:encrypted wallet, specify password to decrypt
            "addr": "12QDRXssT63Pv5KTGBN2kyAvfLW3s7jxBs",
item: wallet7.dat
WARNING:root:encrypted wallet, specify password to decrypt
item: wallet8.dat
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.
item: wallet9.dat
ERROR:root:Couldn't open wallet.dat/main. Try quitting Bitcoin and running this again.

So the address was in wallet6.dat, success! I then used "/pywallet.py --dumpwallet --datadir=.bitcoin --password=mysecrectpassword > keys.txt" and I got all my keys back. In the bitcoin client I could import it with importprivkey (don't forget the "false" parameter as the last parameter, to avoid rescanning after each import, if you import multiple keys) and after the final rescan, I got my Bitcoins back. One day work for like 2 Bitcoins, which I already sold, that's a nice hourly rate Cool

Maybe this will help some other people as well. In case you rescue a lot of Bitcoins, I would really love it if you would send me some to 1ieKggPzp2DfroFBNie4ib77kHKNbJMkw.

Marilyn wishes you a merry Christmas, a merry Christmas, And a happy New Year!
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!