Bitcoin Forum
December 14, 2017, 01:37:30 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 [4]  All
  Print  
Author Topic: Ultraprune merged in mainline  (Read 25219 times)
mezzomix
Legendary
*
Offline Offline

Activity: 1932


View Profile
October 27, 2012, 10:32:19 AM
 #61

Just tried to compile HEAD on FreeBSD:

Code:
localhost|/tmp/bitcoin-bitcoin-bb790aa/src> gmake -f makefile.unix
/bin/sh ../share/genbuild.sh obj/build.h
Building LevelDB ...
"Makefile", line 18: Need an operator
"Makefile", line 63: Missing dependency operator
"Makefile", line 65: Missing dependency operator
"Makefile", line 70: Need an operator
"Makefile", line 82: Need an operator
"Makefile", line 87: Need an operator
"Makefile", line 172: Missing dependency operator
"Makefile", line 194: Need an operator
"Makefile", line 200: Need an operator
make: fatal errors encountered -- cannot continue
g++ -c -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/tmp/bitcoin-bitcoin-bb790aa/src -I/tmp/bitcoin-bitcoin-bb790aa/src/obj -I/usr/local/include -I/usr/local/include/db48 -DUSE_IPV6=1 -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/include -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -MMD -MF obj/leveldb.d -o obj/leveldb.o leveldb.cpp
In file included from leveldb.cpp:5:
leveldb.h:144:2: warning: no newline at end of file
g++ -O2 -pthread -Wall -Wextra -Wformat -Wformat-security -Wno-unused-parameter -g -DBOOST_SPIRIT_THREADSAFE -I/tmp/bitcoin-bitcoin-bb790aa/src -I/tmp/bitcoin-bitcoin-bb790aa/src/obj -I/usr/local/include -I/usr/local/include/db48 -DUSE_IPV6=1 -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/include -I/tmp/bitcoin-bitcoin-bb790aa/src/leveldb/helpers -DHAVE_BUILD_INFO -fno-stack-protector -fstack-protector-all -Wstack-protector -D_FORTIFY_SOURCE=2  -o bitcoind obj/alert.o obj/version.o obj/checkpoints.o obj/netbase.o obj/addrman.o obj/crypter.o obj/key.o obj/db.o obj/init.o obj/irc.o obj/keystore.o obj/main.o obj/net.o obj/protocol.o obj/bitcoinrpc.o obj/rpcdump.o obj/rpcnet.o obj/rpcmining.o obj/rpcwallet.o obj/rpcblockchain.o obj/rpcrawtransaction.o obj/script.o obj/sync.o obj/util.o obj/wallet.o obj/walletdb.o obj/noui.o obj/leveldb.o obj/txdb.o -Wl,-z,relro -Wl,-z,now  -L/usr/local/lib -L/usr/local/lib/db48 -Wl,-Bdynamic -l boost_system -l boost_filesystem -l boost_program_options -l boost_thread -l db_cxx -l ssl -l crypto -Wl,-Bdynamic -l z -l pthread /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libleveldb.a /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libmemenv.a
g++: /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libleveldb.a: No such file or directory
g++: /tmp/bitcoin-bitcoin-bb790aa/src/leveldb/libmemenv.a: No such file or directory
gmake: *** [bitcoind] Error 1
1513215450
Hero Member
*
Offline Offline

Posts: 1513215450

View Profile Personal Message (Offline)

Ignore
1513215450
Reply with quote  #2

1513215450
Report to moderator
1513215450
Hero Member
*
Offline Offline

Posts: 1513215450

View Profile Personal Message (Offline)

Ignore
1513215450
Reply with quote  #2

1513215450
Report to moderator
1513215450
Hero Member
*
Offline Offline

Posts: 1513215450

View Profile Personal Message (Offline)

Ignore
1513215450
Reply with quote  #2

1513215450
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1513215450
Hero Member
*
Offline Offline

Posts: 1513215450

View Profile Personal Message (Offline)

Ignore
1513215450
Reply with quote  #2

1513215450
Report to moderator
1513215450
Hero Member
*
Offline Offline

Posts: 1513215450

View Profile Personal Message (Offline)

Ignore
1513215450
Reply with quote  #2

1513215450
Report to moderator
1513215450
Hero Member
*
Offline Offline

Posts: 1513215450

View Profile Personal Message (Offline)

Ignore
1513215450
Reply with quote  #2

1513215450
Report to moderator
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
November 20, 2012, 08:22:18 PM
 #62

Pieter,

I am trying to make my blockparser utility work with your changes ...

In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).

They should be exactly the same as <= 0.7.1:  4-byte pchMessageStart, 4-byte size, data.

If that is not the case, there is a bug that should be fixed.


Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Diapolo
Hero Member
*****
Offline Offline

Activity: 676



View Profile WWW
November 20, 2012, 09:24:59 PM
 #63

Sounds resonable that there are some unused bytes at the end of these files. Afaik they contain reserved space and if the conditions to start a new blockfile are met that space is left in the files, could that be the answer?

Dia

Liked my former work for Bitcoin Core? Drop me a donation via:
1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x
bitcoin:1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x?label=Diapolo
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
November 20, 2012, 09:25:38 PM
 #64

Pieter,

I am trying to make my blockparser utility work with your changes ...

In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).

They should be exactly the same as <= 0.7.1:  4-byte pchMessageStart, 4-byte size, data.

If that is not the case, there is a bug that should be fixed.



They block magic is not always pchMessageStart.

It's very often zero, but looking at things closer, it looks like it's always close
to the end of the blkXXXX.dat, and therefore, I suspect it's an EOF mark.

The older blkXXX.dat did end exactly on the last byte of the last block.

The parser in the reference client is extremely tolerant of extra cruft in the block files.  Of course, it is only used to load extra block files and bootstrap files, but still.  Since the real block files are never parsed sequentially in normal operation, they are allowed to be a little odd.

Take a look at the reference implementation for a solid, robust block file parser.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
jgarzik
Legendary
*
qt
Offline Offline

Activity: 1484


View Profile
November 20, 2012, 11:14:10 PM
 #65

Pieter,

I am trying to make my blockparser utility work with your changes ...

In particular, it looks like blkXXXX.dat files do not obey the same structure as the
original blkXXX.dat files of the previous version (e..g sometimes the block magic
is not what I expect it to be).

They should be exactly the same as <= 0.7.1:  4-byte pchMessageStart, 4-byte size, data.

If that is not the case, there is a bug that should be fixed.



They block magic is not always pchMessageStart.

It's very often zero, but looking at things closer, it looks like it's always close
to the end of the blkXXXX.dat, and therefore, I suspect it's an EOF mark.

The older blkXXX.dat did end exactly on the last byte of the last block.

Any chance you can do a hexdump, or otherwise narrow down what is being seen?

Just re-reviewed the code.  Each record is written as described (4/4/block), with no EOF marks or anything else.

Perhaps your bitcoind crashed during a write, or a similar cause?  Try downloading again, and see if you still have a corrupted blk*.dat.

Jeff Garzik, Bloq CEO, former bitcoin core dev team; opinions are my own.
Visit bloq.com / metronome.io
Donations / tip jar: 1BrufViLKnSWtuWGkryPsKsxonV2NQ7Tcj
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
November 21, 2012, 12:10:40 AM
 #66

You are probably seeing the pre-allocation. The files are now allocated in blocks of 16 MiB, and only afterward filled in with data, to prevent fragmentation.

At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
kjj
Legendary
*
Offline Offline

Activity: 1302



View Profile
November 21, 2012, 01:35:05 AM
 #67

Robustness is not what my block parser is about, speed is.
As a matter of fact, robustness is the very least of my concerns.

Read the reference loadblocks function anyway.  Robust vs. Fast isn't always a trade-off.  Whoever did this code did it right and got both.  From my read of the function, it can find valid blocks buried under anything, even if an attacker managed to feed you a bogus file (for some strange reason), the worst it would do is waste a little of your time, and the parser is optimized to waste as little as possible.

At the end of the file you should see some zeroes, or other cruft, but there shouldn't be any in between the actual blocks. In fact, contrary to the old code, even when there's a crash in the middle of writing a block, there shouldn't ever be cruft except at the end of file (it remembers hoz much useful data is in each file, and starts appending/overwriting at that point).

Agreed.  I couldn't find even a single byte out of place when I was parsing my block files, which were started with version 0.3.something and living on a computer that seems to crash every month or two.  But writing the parser well didn't seem any harder than writing it poorly, and no slower in operation.

p2pcoin: a USB/CD/PXE p2pool miner - 1N8ZXx2cuMzqBYSK72X4DAy1UdDbZQNPLf - todo
I routinely ignore posters with paid advertising in their sigs.  You should too.
Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
November 21, 2012, 02:02:39 PM
 #68

While I have your attention ... what's in the revXXXX.dat files ?

They contain undo data for blocks. If you see blocks are authenticated patches that modify the state of unspent transaction outputs forward in time, the rev* files contain data to go backward in time. These are needed when doing reorganisations.

Quote
Is the dataset in blkXXXX.dat a complete enough representation of the ledger
or must the info in revXXXX.dat somehow be taken into account ?

Yes, the rev* files are constructed from the normal block files when connecting blocks, so the information in them is redundant in practice.

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
finway
Hero Member
*****
Offline Offline

Activity: 714


View Profile
November 28, 2012, 02:06:16 PM
 #69

@sipa:

running pre0.8 edtion,
shutdown for 3 days,
now it's stuck at 209586, shows:
Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.

I guess it's stuck because of reorg ?

Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
November 28, 2012, 02:08:15 PM
 #70

Warning: Displayed transactions may not be correct! You may need to upgrade, or othernodes may need to upgrade.

Can you put your debug.log online somewhere, or send it to me?

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
finway
Hero Member
*****
Offline Offline

Activity: 714


View Profile
November 28, 2012, 02:13:24 PM
 #71

Can you put your debug.log online somewhere, or send it to me?
sent to pieter.wuille@gmail.com

btw: other 2 pre0.8 nodes are ok.

Pieter Wuille
Legendary
*
qt
Offline Offline

Activity: 1050


View Profile WWW
November 28, 2012, 02:21:39 PM
 #72

Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.

This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?

aka sipa, core dev team

Tips and donations: 1KwDYMJMS4xq3ZEWYfdBRwYG2fHwhZsipa
finway
Hero Member
*****
Offline Offline

Activity: 714


View Profile
November 28, 2012, 02:35:17 PM
 #73

Apparently you had an unclean shutdown, and lost a data file. The debug.log is full of errors from LevelDB missing files.

This is interesting. First, it should complain much more loudly (like trying to recover, or exit with a fatal error). Secondly, this shouldn't happen... as far as I can see, there wasn't even an attempt to write to the blockdb. Did you copy the datadir from one system to another, or did your OS crash?
I think you are right, i had an unclean shutdown (Window 7, power off).
Never mind, i'll reinstall. Thanks for your time.

Diapolo
Hero Member
*****
Offline Offline

Activity: 676



View Profile WWW
November 28, 2012, 02:52:45 PM
 #74

@sipa Can you think of a cause, that leads to a massive spam of ORPHAN BLOCKS with current master during an initial chain sync in testnet?

Could that be caused by a BitCoinJ:0.6-SNAPSHOT/BC vend server:1.0 node somehow?

Dia

Liked my former work for Bitcoin Core? Drop me a donation via:
1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x
bitcoin:1PwnvixzVAKnAqp8LCV8iuv7ohzX2pbn5x?label=Diapolo
Pages: « 1 2 3 [4]  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!