[BOUNTY claimed] Help test version 0.8, blockchain upgrade

[Gavin Andresen]

There are three 0.5 BTC bounties available for people who will help us test the upgrade-to-release-0.8 code.

I'm looking for:

CLAIMED: Somebody running (or willing to install and sync-up) bitcoind/bitcoin-qt version 0.3.24
(need to test upgrading from a very old version of Bitcoin)

CLAIMED: Somebody running a version of Windows with their bitcoin data directory on a FAT32 filesystem
(need to test upgrading on a very old filesystem)

CLAIMED: Somebody running Windows XP with an NTFS filesystem
(need to test upgrading on a very old operating system)


Detailed instructions on how to test and then claim a bounty are here:
  https://github.com/gavinandresen/QA/blob/master/HardLinksUpgrade.md

[1714986062]

1714986062

[1714986062]

1714986062

[1714986062]

1714986062

[burger]

Somebody running (or willing to install and sync-up) bitcoind/bitcoin-qt version 0.3.24
(need to test upgrading from a very old version of Bitcoin)

In progress...

Btw, don't 0.3.24 support timestamps in the logfile?

[Gavin Andresen]

Btw, don't 0.3.24 support timestamps in the logfile?

I think you're right, 0.3.24 does not support -logtimestamps. If I recall correctly, it does log the time at startup; for this particular test, I'm mostly interested in how long the upgrade process takes, so the old version not supporting timestamps isn't a problem.

[burger]

Btw, don't 0.3.24 support timestamps in the logfile?

I think you're right, 0.3.24 does not support -logtimestamps. If I recall correctly, it does log the time at startup; for this particular test, I'm mostly interested in how long the upgrade process takes, so the old version not supporting timestamps isn't a problem.

ok, good.
Btw I'm still downloading the blockchain with 8 connections...

[deepceleron]

There are three 0.5 BTC bounties available for people who will help us test the upgrade-to-release-0.8 code.

I'm looking for:

Somebody running (or willing to install and sync-up) bitcoind/bitcoin-qt version 0.3.24
(need to test upgrading from a very old version of Bitcoin)

Somebody running a version of Windows with their bitcoin data directory on a FAT32 filesystem
(need to test upgrading on a very old filesystem)

Somebody running Windows XP with an NTFS filesystem
(need to test upgrading on a very old operating system)

Detailed instructions on how to test and then claim a bounty are here:
  https://github.com/gavinandresen/QA/blob/master/HardLinksUpgrade.md

I have a virtual xp machine with 0.3.24 wallet backups, provided I can remember the password I can take #1 and #2 on wallets not touched since Nov 2011. Any preference for what blockchain height at "migration" would reveal zee boogs. New sync or loadblocks the old blockchain?

Edit: doing full 0.3.24 sync on XP/FAT32 datadir to current height, then upgrade (per instructions). Local/SSD, so shouldn't take tooo long.

[deepceleron]

I thought I'd try a really old upgrade, but it looks like Bitcoin never worked on Win95...

[burger]

I'm still downloading the blockchain... =/ it's very slow now.

[deepceleron]

Completed both 0.3.24 and FAT32 at the same time....
Links to debug logs, screenshots, etc in "code" below.

Conclusions:
1. on FAT32, full blockchain re-download as expected (6 hours), for a 12GB datadir
2. Reversion to 0.3.24 does not work, fails on unacceptable/unsupported log file version errors,
3. 0.8.0 salvaged a questionable wallet automatically, but continued to nag with wallet warnings,
4. 0.8.0-qt immediately closes when asked to close, but leaves a termination error in the logs.

I also tried to follow the "forking" instructions but don't understand what is being asked.

Code:

0.8 blockchain data 0.3.24 upgrade test report + FAT32
--------
A. OS environment:

Windows XP SP3 x86, all updates, Virtualbox 4.2.6 64 bit

Drive structure: C: NTFS (previous Bitcoin dir & datadir renamed) (rotating HDD)
E: FAT32 (blank for use as datadir) (physical drive SSD)
command line: bitcoin.exe -datadir=E:\datadir

bitcoin.exe 0.3.24 Windows installer (bitcoin-0.3.24-win32-setup)
local network --connect to Bitcoin 0.7.2


B. Initial environment setup notes:

1. crashed my virtual machine while downloading from scratch, creating a corrupted wallet.dat
with some bitcoins that were sent to it. Could not get bitcoin started again
with that wallet without db.log errors and crashing. Tada, a corrupted wallet for later testing.

2. Wiped datadir, seeded datadir with 206000 height detached blockchain dat/idx 
(created with 0.7.1, from imported mkbootstrap blockchain)

3. Downloaded blocks 206001-215964 - 1:50-7:36 hours 

4. Shutdown of Bitcoin 0.3.24 takes ~3 minutes with no indication of execution
::rename of debug.log: "debug.log.0.3.24"; blank db.log

6. Replaced blank wallet with "corrupted" wallet, now appears to run fine.
Launched bitcoin with rescan to correct wallet balance.
::rename of debug.log: "debug.log.0.3.24.newwallet.rescan"

C. Upgrade to 0.8.0 

1. Copied test bitcoin-qt.exe to program files alongside bitcoin.exe install

2. First instantiation, Bitcoin-qt finds the problems with the wallet:

Warning: wallet.dat corrupt, data salvaged! Original wallet saved as .bak
http://we.lovebitco.in/img/xp1-wallet.dat-corrupt.png

3. Bitcoin displays an error with the wallet.dat that will never go away,
occurs after every subsequent launch of bitcoin-qt:

Warning: error reading wallet.dat! All keys read correctly, but transaction
data or address book entries might be missing or incorrect

http://we.lovebitco.in/img/xp2-walleterror.png

4. analysis of db.log shows:

Program version 4.8 doesn't match environment version 4.7
Page 14: unreferenced page
Page 14: totally zeroed page (...repeated until page 33...)
wallet.dat: DB_VERIFY_BAD: Database verification failed

5. Block re-download

Expected linking error for FAT32:

2013-01-10T09:32:07 Error hardlinking blk0001.dat : boost::filesystem::create_hard_link: Incorrect function: "E:\datadir\blk0001.dat", "E:\datadir\blocks\blk00000.dat"

(why can't we attempt to import those blocks instead of hitting the network)

Blockchain download from local client: 9:30-15:05 hours

6. Sent coins out of wallet successfully; transaction added to wallet, although
wallet.dat warning continues after a close and restart of Bitcoin.
::rename of debug.log: "debug.log.after0.8.0up";
::rename of db.log   : "db.log.after0.8.0.up"


D. Downgrade to 0.3.24: Complete failure

Closing bitcoin-qt.exe 0.8.0 (and verifying task complete), then
launching 0.3.24 gives a series of dire obfuscated errors followed by program
termination:
EXCEPTION: 22DbRunRecoveryException; DbEnv::open: DB_RUNRECOVERY: Fatal error, ...

http://we.lovebitco.in/img/xp3-afterdowngrade.png
http://we.lovebitco.in/img/xp4-afterdowngrade.png
http://we.lovebitco.in/img/xp5-afterdowngrade.png

::rename of debug.log: "debug.log.0.3.24.downgrade";
::rename of db.log   : "db.log.0.3.24.downgrade"


E. Relaunch of 0.8.0: still working 
1. corrupt wallet warnings remain (perhaps it should quit warning if it salvaged wallet)
2. replacing wallet.dat with blank wallet from 0.3.24: no errors.

========
DB log files; original (renamed) corrupted wallet.bak and after salvage; config files etc:
http://we.lovebitco.in/testplan-datadir.zip


========
Directory listing after 0.8.0 upgrade:

 Volume in drive E is FAT32
 Volume Serial Number is 7852-6B09

 Directory of E:\

01/09/2013  03:55 PM    <DIR>          datadir
01/10/2013  12:18 PM                 0 dirafter080up.txt
               1 File(s)              0 bytes

 Directory of E:\datadir

01/09/2013  03:55 PM    <DIR>          .
01/09/2013  03:55 PM    <DIR>          ..
01/10/2013  01:26 AM             4,101 bitcoin.conf
01/10/2013  01:32 AM    <DIR>          blocks
01/10/2013  01:32 AM    <DIR>          blktree
01/10/2013  01:32 AM    <DIR>          coins
01/10/2013  01:22 AM           278,528 wallet.1357810229.bak
01/10/2013  01:23 AM           266,563 debug.log.rescan
01/10/2013  12:13 PM        73,291,561 debug.log
01/10/2013  12:13 PM           122,880 wallet.dat
01/10/2013  01:30 AM        11,891,978 Copy of debug.log
01/10/2013  12:16 PM    <DIR>          wallet-blank-0.3.24
01/10/2013  12:13 PM             1,106 peers.dat
01/10/2013  12:25 AM         7,519,728 debug.log.0.3.24
01/10/2013  12:13 PM             1,276 db.log
01/09/2013  05:39 PM    <DIR>          15btcbadwallet
01/09/2013  08:22 PM     2,097,295,438 blk0002.dat
01/10/2013  01:23 AM     1,533,804,544 blkindex.dat
11/05/2012  03:23 PM     2,097,361,271 blk0001.dat
01/09/2013  05:50 PM                 0 .lock
01/09/2013  05:50 PM    <DIR>          database
01/10/2013  01:22 AM           933,888 addr.dat
01/10/2013  01:11 AM       633,945,002 blk0003.dat
              15 File(s)  6,456,717,864 bytes

 Directory of E:\datadir\blocks

01/10/2013  01:32 AM    <DIR>          .
01/10/2013  01:32 AM    <DIR>          ..
01/10/2013  01:56 AM       134,217,728 blk00000.dat
01/10/2013  01:56 AM        15,728,640 rev00000.dat
01/10/2013  02:00 AM       134,217,728 blk00001.dat
01/10/2013  02:00 AM        15,728,640 rev00001.dat
01/10/2013  02:03 AM       134,217,728 blk00002.dat
01/10/2013  02:03 AM        15,728,640 rev00002.dat
01/10/2013  02:38 AM       134,217,728 blk00003.dat
01/10/2013  02:38 AM        15,728,640 rev00003.dat
01/10/2013  02:41 AM       134,217,728 blk00004.dat
01/10/2013  02:41 AM        15,728,640 rev00004.dat
01/10/2013  02:45 AM       134,217,728 blk00005.dat
01/10/2013  02:45 AM        16,777,216 rev00005.dat
01/10/2013  02:49 AM       134,217,728 blk00006.dat
01/10/2013  02:49 AM        17,825,792 rev00006.dat
01/10/2013  02:52 AM       134,217,728 blk00007.dat
01/10/2013  02:52 AM        16,777,216 rev00007.dat
01/10/2013  02:56 AM       134,217,728 blk00008.dat
01/10/2013  02:56 AM        17,825,792 rev00008.dat
01/10/2013  02:59 AM       134,217,728 blk00009.dat
01/10/2013  02:59 AM        17,825,792 rev00009.dat
01/10/2013  03:02 AM       134,217,728 blk00010.dat
01/10/2013  03:02 AM        17,825,792 rev00010.dat
01/10/2013  03:04 AM       134,217,728 blk00011.dat
01/10/2013  03:04 AM        17,825,792 rev00011.dat
01/10/2013  03:07 AM       134,217,728 blk00012.dat
01/10/2013  03:07 AM        17,825,792 rev00012.dat
01/10/2013  03:11 AM       134,217,728 blk00013.dat
01/10/2013  03:11 AM        17,825,792 rev00013.dat
01/10/2013  03:19 AM       134,217,728 blk00014.dat
01/10/2013  03:19 AM        17,825,792 rev00014.dat
01/10/2013  03:22 AM       134,217,728 blk00015.dat
01/10/2013  03:22 AM        17,825,792 rev00015.dat
01/10/2013  03:25 AM       134,217,728 blk00016.dat
01/10/2013  03:25 AM        17,825,792 rev00016.dat
01/10/2013  03:28 AM       134,217,728 blk00017.dat
01/10/2013  03:28 AM        17,825,792 rev00017.dat
01/10/2013  03:31 AM       134,217,728 blk00018.dat
01/10/2013  03:31 AM        17,825,792 rev00018.dat
01/10/2013  03:34 AM       134,217,728 blk00019.dat
01/10/2013  03:34 AM        17,825,792 rev00019.dat
01/10/2013  03:37 AM       134,217,728 blk00020.dat
01/10/2013  03:37 AM        17,825,792 rev00020.dat
01/10/2013  03:40 AM       134,217,728 blk00021.dat
01/10/2013  03:40 AM        17,825,792 rev00021.dat
01/10/2013  03:43 AM       134,217,728 blk00022.dat
01/10/2013  03:43 AM        17,825,792 rev00022.dat
01/10/2013  03:47 AM       134,217,728 blk00023.dat
01/10/2013  03:47 AM        17,825,792 rev00023.dat
01/10/2013  03:50 AM       134,217,728 blk00024.dat
01/10/2013  03:50 AM        17,825,792 rev00024.dat
01/10/2013  03:54 AM       134,217,728 blk00025.dat
01/10/2013  03:54 AM        17,825,792 rev00025.dat
01/10/2013  03:57 AM       134,217,728 blk00026.dat
01/10/2013  03:57 AM        17,825,792 rev00026.dat
01/10/2013  04:01 AM       134,217,728 blk00027.dat
01/10/2013  04:01 AM        17,825,792 rev00027.dat
01/10/2013  04:04 AM       134,217,728 blk00028.dat
01/10/2013  04:04 AM        17,825,792 rev00028.dat
01/10/2013  04:07 AM       134,217,728 blk00029.dat
01/10/2013  04:07 AM        17,825,792 rev00029.dat
01/10/2013  04:22 AM       134,217,728 blk00030.dat
01/10/2013  04:22 AM        17,825,792 rev00030.dat
01/10/2013  05:01 AM       134,217,728 blk00031.dat
01/10/2013  05:01 AM        17,825,792 rev00031.dat
01/10/2013  05:25 AM       134,217,728 blk00032.dat
01/10/2013  05:25 AM        17,825,792 rev00032.dat
01/10/2013  05:49 AM       134,217,728 blk00033.dat
01/10/2013  05:49 AM        17,825,792 rev00033.dat
01/10/2013  06:12 AM       134,217,728 blk00034.dat
01/10/2013  06:12 AM        16,777,216 rev00034.dat
01/10/2013  06:36 AM       134,217,728 blk00035.dat
01/10/2013  06:36 AM        17,825,792 rev00035.dat
01/10/2013  12:11 PM        16,777,216 blk00036.dat
01/10/2013  12:11 PM         2,097,152 rev00036.dat
              74 File(s)  5,478,809,600 bytes

 Directory of E:\datadir\blktree

01/10/2013  01:32 AM    <DIR>          .
01/10/2013  01:32 AM    <DIR>          ..
01/10/2013  12:13 PM            67,828 LOG
01/10/2013  01:32 AM                 0 LOCK
01/10/2013  02:46 AM         1,135,122 000356.sst
01/10/2013  06:26 AM         2,141,443 000518.sst
01/10/2013  12:13 PM           192,381 000516.log
01/10/2013  03:58 AM         2,139,957 000490.sst
01/10/2013  12:13 PM            32,066 MANIFEST-000002
01/10/2013  03:58 AM         2,139,993 000491.sst
01/10/2013  03:58 AM         2,139,961 000492.sst
01/10/2013  01:32 AM                16 CURRENT
01/10/2013  03:58 AM         2,140,325 000493.sst
01/10/2013  03:58 AM         2,140,463 000494.sst
01/10/2013  03:58 AM         2,139,792 000495.sst
01/10/2013  03:58 AM         2,112,292 000496.sst
01/10/2013  06:26 AM         2,141,851 000519.sst
01/10/2013  06:26 AM         2,142,378 000520.sst
01/10/2013  06:26 AM         2,142,054 000521.sst
01/10/2013  06:26 AM         1,552,554 000522.sst
01/10/2013  01:48 AM         2,142,759 000181.sst
01/10/2013  01:48 AM         2,142,566 000182.sst
              20 File(s)     30,785,801 bytes

 Directory of E:\datadir\coins

01/10/2013  01:32 AM    <DIR>          .
01/10/2013  01:32 AM    <DIR>          ..
01/10/2013  12:13 PM         3,415,970 LOG
01/10/2013  01:32 AM                 0 LOCK
01/10/2013  11:29 AM         2,141,346 022853.sst
01/10/2013  11:29 AM         2,148,394 022893.sst
01/10/2013  11:29 AM         2,142,926 022863.sst
01/10/2013  11:30 AM         2,141,620 022903.sst
01/10/2013  11:29 AM         2,142,339 022894.sst
01/10/2013  11:29 AM         2,141,589 022854.sst
01/10/2013  12:13 PM         2,299,346 MANIFEST-000002
01/10/2013  11:30 AM         2,140,329 022904.sst
01/10/2013  11:29 AM         2,143,821 022895.sst
01/10/2013  01:32 AM                16 CURRENT
01/10/2013  11:29 AM         2,141,418 022864.sst
01/10/2013  11:29 AM         2,141,880 022896.sst
01/10/2013  11:29 AM         2,146,395 022855.sst
01/10/2013  11:29 AM         2,140,980 022897.sst
01/10/2013  11:29 AM         2,143,390 022856.sst
01/10/2013  11:29 AM         2,144,515 022865.sst
01/10/2013  11:29 AM         2,142,836 022898.sst
01/10/2013  11:30 AM         2,139,970 022905.sst
01/10/2013  11:29 AM         2,144,642 022866.sst
01/10/2013  12:13 PM           941,714 022833.log
01/10/2013  11:29 AM         2,144,449 022857.sst
01/10/2013  11:29 AM         2,140,654 022873.sst
01/10/2013  11:29 AM         2,142,285 022858.sst
01/10/2013  11:29 AM         2,144,776 022874.sst
01/10/2013  11:29 AM         2,141,920 022883.sst
01/10/2013  11:29 AM         2,140,182 022867.sst
01/10/2013  11:29 AM         2,142,189 022875.sst
01/10/2013  11:29 AM         2,144,485 022859.sst
01/10/2013  11:29 AM         2,142,796 022899.sst
01/10/2013  11:29 AM         2,144,571 022860.sst
01/10/2013  11:29 AM         2,143,936 022861.sst
01/10/2013  11:29 AM         2,146,623 022900.sst
01/10/2013  11:29 AM         2,138,697 022862.sst
01/10/2013  11:29 AM         2,142,711 022868.sst
01/10/2013  11:29 AM         2,141,946 022869.sst
01/10/2013  11:29 AM         2,142,726 022901.sst
01/10/2013  11:29 AM         2,144,766 022870.sst
01/10/2013  11:29 AM         2,140,667 022871.sst
01/10/2013  11:29 AM         2,144,364 022876.sst
01/10/2013  11:29 AM         2,142,254 022872.sst
01/10/2013  11:29 AM         2,143,502 022877.sst
01/10/2013  11:29 AM         2,122,204 022902.sst
01/10/2013  11:29 AM         2,143,388 022878.sst
01/10/2013  11:30 AM         2,142,910 022906.sst
01/10/2013  11:29 AM         2,144,328 022879.sst
01/10/2013  11:30 AM         2,145,738 022907.sst
01/10/2013  11:29 AM         2,149,651 022880.sst
01/10/2013  11:30 AM         2,145,016 022908.sst
01/10/2013  11:30 AM         2,142,383 022909.sst
01/10/2013  11:30 AM         2,144,736 022910.sst
01/10/2013  11:30 AM         2,142,973 022911.sst
01/10/2013  11:30 AM         2,144,603 022912.sst
01/10/2013  11:29 AM         2,144,723 022842.sst
01/10/2013  11:30 AM             5,014 022913.sst
01/10/2013  11:29 AM         2,143,362 022881.sst
01/10/2013  11:29 AM         2,140,430 022882.sst
01/10/2013  11:29 AM         2,141,461 022884.sst
01/10/2013  11:29 AM         2,146,151 022885.sst
01/10/2013  11:29 AM         2,143,497 022886.sst
01/10/2013  11:29 AM         2,145,678 022887.sst
01/10/2013  10:08 AM             5,031 022811.sst
01/10/2013  11:29 AM         2,146,403 022888.sst
01/10/2013  11:29 AM         2,143,341 022889.sst
01/10/2013  11:29 AM         2,143,117 022890.sst
01/10/2013  11:29 AM         2,144,120 022891.sst
01/10/2013  11:29 AM         1,605,629 022892.sst
01/10/2013  11:29 AM         2,141,967 022843.sst
01/10/2013  11:29 AM         2,145,195 022844.sst
01/10/2013  11:29 AM         2,140,304 022845.sst
01/10/2013  11:29 AM         2,142,718 022846.sst
01/10/2013  11:29 AM         2,146,672 022847.sst
01/10/2013  11:29 AM         2,144,873 022848.sst
01/10/2013  11:29 AM         2,141,580 022849.sst
01/10/2013  11:29 AM         2,143,872 022850.sst
01/10/2013  11:29 AM         2,142,659 022851.sst
01/10/2013  11:29 AM            48,503 022852.sst
              78 File(s)    156,190,165 bytes

 Directory of E:\datadir\wallet-blank-0.3.24

01/10/2013  12:16 PM    <DIR>          .
01/10/2013  12:16 PM    <DIR>          ..
01/10/2013  12:24 AM           106,496 wallet.dat
               1 File(s)        106,496 bytes

 Directory of E:\datadir\15btcbadwallet

01/09/2013  05:39 PM    <DIR>          .
01/09/2013  05:39 PM    <DIR>          ..
01/09/2013  04:20 PM           278,528 wallet.dat.badmaybe
01/10/2013  12:13 PM           122,880 wallet.dat
               2 File(s)        401,408 bytes

 Directory of E:\datadir\database

01/09/2013  05:50 PM    <DIR>          .
01/09/2013  05:50 PM    <DIR>          ..
01/10/2013  12:13 PM        10,485,760 log.0000000265
               1 File(s)     10,485,760 bytes

 Directory of E:\Recycled

01/09/2013  04:47 PM    <DIR>          .
01/09/2013  04:47 PM    <DIR>          ..
01/10/2013  12:56 AM           106,496 De105.dat
01/09/2013  04:20 PM           278,528 De106.dat
01/10/2013  12:14 PM    <DIR>          De107
01/10/2013  12:14 PM    <DIR>          De108
01/10/2013  01:22 AM           278,528 De109.dat
01/10/2013  12:16 PM    <DIR>          De110
               3 File(s)        663,552 bytes

 Directory of E:\Recycled\De107

01/10/2013  12:14 PM    <DIR>          .
01/10/2013  12:14 PM    <DIR>          ..
               0 File(s)              0 bytes

 Directory of E:\Recycled\De108

01/10/2013  12:14 PM    <DIR>          .
01/10/2013  12:14 PM    <DIR>          ..
               0 File(s)              0 bytes

 Directory of E:\Recycled\De110

01/10/2013  12:16 PM    <DIR>          .
01/10/2013  12:16 PM    <DIR>          ..
               0 File(s)              0 bytes

     Total Files Listed:
             195 File(s) 12,134,160,646 bytes
              32 Dir(s)   4,460,224,512 bytes free

[burger]

I hope you got all the data you needed from the tests.

[burger]

https://github.com/gavinandresen/QA/blob/master/HardLinksUpgrade.md isn't updated with claimed as the top post has here.

[Gavin Andresen]

Oops, sorry, update the github page, all the bounties are claimed.

Thanks to everybody who tested, you were extremely helpful.

What did you think about the bounty amounts:  too much?  too little?  just right?

[zvs]

whatever that is that's on the git right now is an absolute mess

block 216948 is too big, and thus corrupt, apparently

[deepceleron]

I filed a gitbug about the repeated wallet warnings after salvaging based on the testing above.

Was reversion to an older version such as 0.3.24 expected to work? I can spend some time testing if a 0.8.0 branch (unencrypted) wallet can be used with 0.3.24, with or without walletupgrade, if the datadir is otherwise cleaned, find out which log file 0.3.24 complains about, etc.

Is there a later 0.8.0 target Win32 pre-build? The bluematt repository seems down.