Ultraprune merged in mainline

[mezzomix]

Just tried to compile HEAD on FreeBSD:

Code:

localhost|/tmp/bitcoin-bitcoin-bb790aa/src> gmake -f makefile.unix 
/bin/sh ../share/genbuild.sh obj/build.h
Building LevelDB ...
"Makefile", line 18: Need an operator
"Makefile", line 63: Missing dependency operator
"Makefile", line 65: Missing dependency operator
"Makefile", line 70: Need an operator
"Makefile", line 82: Need an operator
"Makefile", line 87: Need an operator
"Makefile", line 172: Missing dependency operator
"Makefile", line 194: Need an operator
"Makefile", line 200: Need an operator
make: fatal errors encountered -- cannot continue
g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/tmp/bitcoin-bitcoin-bb790aa/src -I/tmp/bitcoin-bitcoin-bb790aa/src/obj -I/usr/local/include -I/usr/local/include/db48 -DUSE_IPV6=1 -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/include -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/leveldb.d -o obj/leveldb.o leveldb.cpp
In file included from leveldb.cpp:5:
leveldb.h:144:2: warning: no newline at end of file
g++ -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/tmp/bitcoin-bitcoin-bb790aa/src -I/tmp/bitcoin-bitcoin-bb790aa/src/obj -I/usr/local/include -I/usr/local/include/db48 -DUSE_IPV6=1 -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/include -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -o bitcoind obj/alert.o obj/version.o obj/checkpoints.o obj/netbase.o obj/addrman.o obj/crypter.o obj/key.o obj/db.o obj/init.o obj/irc.o obj/keystore.o obj/main.o obj/net.o obj/protocol.o obj/bitcoinrpc.o obj/rpcdump.o obj/rpcnet.o obj/rpcmining.o obj/rpcwallet.o obj/rpcblockchain.o obj/rpcrawtransaction.o obj/script.o obj/sync.o obj/util.o obj/wallet.o obj/walletdb.o obj/noui.o obj/leveldb.o obj/txdb.o -Wl,-z,relro -Wl,-z,now  -L/usr/local/lib -L/usr/local/lib/db48 -Wl,-Bdynamic -l boost_system -l boost_filesystem -l boost_program_options -l boost_thread -l db_cxx -l ssl -l crypto -Wl,-Bdynamic -l z -l pthread /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libleveldb.a /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libmemenv.a
g++: /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libleveldb.a: No such file or directory
g++: /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libmemenv.a: No such file or directory
gmake: *** [bitcoind] Error 1

[jgarzik]

Pieter,

I am trying to make my blockparser utility work with your changes ...

In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).

They should be exactly the same as <= 0.7.1:  4-byte pchMessageStart, 4-byte size, data.

If that is not the case, there is a bug that should be fixed.

[Diapolo]

Sounds resonable that there are some unused bytes at the end of these files. Afaik they contain reserved space and if the conditions to start a new blockfile are met that space is left in the files, could that be the answer?

Dia

[kjj]

Pieter,

I am trying to make my blockparser utility work with your changes ...

In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).

They should be exactly the same as <= 0.7.1:  4-byte pchMessageStart, 4-byte size, data.

If that is not the case, there is a bug that should be fixed.

They block magic is not always pchMessageStart.

It's very often zero, but looking at things closer, it looks like it's always close
to the end of the blkXXXX.dat, and therefore, I suspect it's an EOF mark.

The older blkXXX.dat did end exactly on the last byte of the last block.

The parser in the reference client is extremely tolerant of extra cruft in the block files.  Of course, it is only used to load extra block files and bootstrap files, but still.  Since the real block files are never parsed sequentially in normal operation, they are allowed to be a little odd.

Take a look at the reference implementation for a solid, robust block file parser.

[jgarzik]

Pieter,

I am trying to make my blockparser utility work with your changes ...

In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).

They should be exactly the same as <= 0.7.1:  4-byte pchMessageStart, 4-byte size, data.

If that is not the case, there is a bug that should be fixed.

They block magic is not always pchMessageStart.

It's very often zero, but looking at things closer, it looks like it's always close
to the end of the blkXXXX.dat, and therefore, I suspect it's an EOF mark.

The older blkXXX.dat did end exactly on the last byte of the last block.

Any chance you can do a hexdump, or otherwise narrow down what is being seen?

Just re-reviewed the code.  Each record is written as described (4/4/block), with no EOF marks or anything else.

Perhaps your bitcoind crashed during a write, or a similar cause?  Try downloading again, and see if you still have a corrupted blk*.dat.

[Pieter Wuille]

You are probably seeing the pre-allocation. The files are now allocated in blocks of 16 MiB, and only afterward filled in with data, to prevent fragmentation.

At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).

[kjj]

Robustness is not what my block parser is about, speed is.
As a matter of fact, robustness is the very least of my concerns.

Read the reference loadblocks function anyway.  Robust vs. Fast isn't always a trade-off.  Whoever did this code did it right and got both.  From my read of the function, it can find valid blocks buried under anything, even if an attacker managed to feed you a bogus file (for some strange reason), the worst it would do is waste a little of your time, and the parser is optimized to waste as little as possible.

At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).

Agreed.  I couldn't find even a single byte out of place when I was parsing my block files, which were started with version 0.3.something and living on a computer that seems to crash every month or two.  But writing the parser well didn't seem any harder than writing it poorly, and no slower in operation.

[Pieter Wuille]

While I have your attention ... what's in the revXXXX.dat files ?

They contain undo data for blocks. If you see blocks are authenticated patches that modify the state of unspent transaction outputs forward in time, the rev* files contain data to go backward in time. These are needed when doing reorganisations.

Is the dataset in blkXXXX.dat a complete enough representation of the ledger
or must the info in revXXXX.dat somehow be taken into account ?

Yes, the rev* files are constructed from the normal block files when connecting blocks, so the information in them is redundant in practice.

[finway]

@sipa:

running pre0.8 edtion,
shutdown for 3 days,
now it's stuck at 209586, shows:
Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.

I guess it's stuck because of reorg ?

[Pieter Wuille]

Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.

Can you put your debug.log online somewhere, or send it to me?

[finway]

Can you put your debug.log online somewhere, or send it to me?

sent to pieter.wuille@gmail.com

btw: other 2 pre0.8 nodes are ok.

[Pieter Wuille]

Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.

This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?

[finway]

Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.

This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?

I think you are right, i had an unclean shutdown (Window 7, power off).
Never mind, i'll reinstall. Thanks for your time.

[Diapolo]

@sipa Can you think of a cause, that leads to a massive spam of ORPHAN BLOCKS with current master during an initial chain sync in testnet?

Could that be caused by a BitCoinJ:0.6-SNAPSHOT/BC vend server:1.0 node somehow?

Dia