[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[John Tobey]

Is there any more information you need? Is there someway I can tell abe to connect to bitcoind with RPC over a specific port?

Sorry for the delayed reply.  Please post the last few lines of output before "catch_up_rpc: abort" or "Exception: RPC load failed."  Have you run Bitcoin with -reindex since adding txindex to bitcoin.conf?

RPC is slower for the initial load, I think over twice as slow as blockfiles.  Once caught up, it usually inserts blocks faster than the blockfile method, since RPC has already loaded most transactions by the time the block comes (assuming that you arrange for it to catch up frequently).

Abe normally does not "read the blockfiles over and over again": it stores the current position in the database.

[1713568687]

1713568687

[1713568687]

1713568687

[1713568687]

1713568687

[katriel]

Abe is really terrific! I used it to create a transaction network and play around with that.

However, on my machine I left it running (writing to a SQLite database) for I think about two weeks and only got about 18 months into the blockchain (2.4 million transactions). The readme says it typically takes several days to get through the chain--is there something I'm probably doing wrong?

[John Tobey]

Abe is really terrific! I used it to create a transaction network and play around with that.

Good to hear!

However, on my machine I left it running (writing to a SQLite database) for I think about two weeks and only got about 18 months into the blockchain (2.4 million transactions). The readme says it typically takes several days to get through the chain--is there something I'm probably doing wrong?

I don't use SQLite for the full block chain, so I don't really know.  One user, K1773R, reported a fast load with the database file in tmpfs (RAM).  (https://bitcointalk.org/index.php?topic=22785.msg2639709#msg2639709)  I think K1773R uses SQLite.  If your system has enough memory, you could try that.  Abe takes a lot of space, though.  I estimate 4-5 times the size of all block files combined.  (~/.bitcoin/blocks/blk00???.dat in Unix)  You can reduce this somewhat by setting keep-scriptsig=false or by using the no-statistics branch and default-trim-depth.  See the comments in abe.conf for what those options entail.

K1773R mentioned possibly distributing the SQLite file via torrent: https://github.com/jtobey/bitcoin-abe/issues/15

[johntawaki]

nice one, but I heard it's not working for some new coins, such as Diamond (DMD).

[John Tobey]

nice one, but I heard it's not working for some new coins, such as Diamond (DMD).

Yes, several new coins trace their ancestry to PPCoin and need the "ppcoin" branch in Git.  See: https://github.com/jtobey/bitcoin-abe/issues/19

Another one, CopperLark, has its own format, which the "copperlark" branch handles.

Ideally, Abe would automatically detect among known variants and handle them without code changes.

[internetbitcoins]

Hi,

I run

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks.

Is there a way to recover them without restart all the process of reading all the blocks on the blockchain??
For example, any way to launch the

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

only for missing ids?

Thanks in advances and for Abe!

[John Tobey]

I run

Code:

python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

but 2 or 3 times I had to restart the server, so now in the block table, I check on block_id and there are some missing blocks.

block_id is not the same as block number (a.k.a. block_height).  The gaps in block_id are not a problem.

[unick]

Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?

thanks

[John Tobey]

Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?

thanks

Sure, just post how far you got and any detailed error message.  If the transactions have a new format, we may need to find how the coin's source code serializes CTransaction.  Bitcoin example: src/core.h line 194.

[unick]

Hey John Tobey, I am trying to set up the explorer for the DMD coin, I've read the issue about PPcoin branches, could you give any help on setting this up for DMD?

thanks

Sure, just post how far you got and any detailed error message.  If the transactions have a new format, we may need to find how the coin's source code serializes CTransaction.  Bitcoin example: src/core.h line 194.

This is the error message I get when trying to read the block chain

Code:

root@blockchains:/var/www/blockchain/bitcoin-abe/Abe# python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve
failed to load /root/.Diamond/bitcoin.conf: [Errno 2] No such file or directory: u'/root/.Diamond/bitcoin.conf'
catch_up_rpc: abort
Opened /root/.Diamond/blk0001.dat
Exception at 72056513853653130
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.Diamond', 'id': 24L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2637, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2895, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3022, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 3053, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 88, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "/usr/local/lib/python2.7/dist-packages/Abe/deserialize.py", line 46, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32 (self): return self._read_num('<I')
  File "/usr/local/lib/python2.7/dist-packages/Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

I am not sure if this is what you are looking for

https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59

[John Tobey]

I am not sure if this is what you are looking for

https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59

Close enough, it's here, and the new field is strTxComment: https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/main.h#L452-462

I've downloaded some of the DMD chain and got Abe to load it, but only over RPC, since this coin uses strange block hashing rules.  Loading from the blockfile resulted in only Block 0 being displayed, since Abe could not link any block to its parent.  Also, I had to disable verification and make Abe trust the RPC server's block hashes.  (If you have info about the differences between DMD and Bitcoin block hashing, I may be able to fix this.)

Please pull branch "diamond", copy Diamond.conf as bitcoin.conf, add "rpcport=17772" to that file if not already present, and use a config such as this:

Code:

datadir [{"dirname":"/root/.Diamond", "chain":"Diamond", "code3":"DMD", "address_version":"\u005a"}]
default-loader rpc

[tacoman71]

You should make a newbie friendly .exe version of this that they can just double click to install.

[John Tobey]

You should make a newbie friendly .exe version of this that they can just double click to install.

I have too little time.  I would happily show another developer around, should one wish to package it nicely.

[b!z]

Very cool. This looks like a good way to view the blockchain without relying on an external site.

[unick]

I am not sure if this is what you are looking for

https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/serialize.h#L59

Close enough, it's here, and the new field is strTxComment: https://github.com/diamondcoinproject/diamondcoin/blob/dd388c28bf89abeaaad761982bb4d04f18995c27/src/main.h#L452-462

I've downloaded some of the DMD chain and got Abe to load it, but only over RPC, since this coin uses strange block hashing rules.  Loading from the blockfile resulted in only Block 0 being displayed, since Abe could not link any block to its parent.  Also, I had to disable verification and make Abe trust the RPC server's block hashes.  (If you have info about the differences between DMD and Bitcoin block hashing, I may be able to fix this.)

Please pull branch "diamond", copy Diamond.conf as bitcoin.conf, add "rpcport=17772" to that file if not already present, and use a config such as this:

Code:

datadir [{"dirname":"/root/.Diamond", "chain":"Diamond", "code3":"DMD", "address_version":"\u005a"}]
default-loader rpc

I am having the following error

Code:

RPC data not understood: block hash mismatch
Failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.Diamond', 'id': 36L}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2635, in catch_up
    raise Exception("RPC load failed")
Exception: RPC load failed
Abe initialized.

I have pulled the diamond branch from the repository
I have cp Diamond.conf. to bitcoin.conf (making sure the rpc port was on 17772)
change abe-my.conf accordingly

while the server listens... I can't seem to have the RPC loaded... did I miss something ?

[John Tobey]

Code:

RPC data not understood: block hash mismatch

Are you sure you are using the latest from the diamond branch?  These changes should be reflected in your Python files:

https://github.com/jtobey/bitcoin-abe/commit/829890b0c3bd2599e66108c24d90d16a5a1b0ea8

In particular, the part about "block hash mismatch" should appear commented out.

[unick]

yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.

any thing else?

[John Tobey]

yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.

any thing else?

Perhaps you have an older version installed on your system?  Otherwise, I don't see how Abe could produce that message.  What does this show:

Code:

python -c 'import Abe.abe; print Abe.abe;'

[unick]

yep, Git tells me I'm up to date and I have manually checked for those changes and everything is there.

any thing else?

Perhaps you have an older version installed on your system?  Otherwise, I don't see how Abe could produce that message.  What does this show:

Code:

python -c 'import Abe.abe; print Abe.abe;'

This is the output

Code:

<module 'Abe.abe' from '/usr/local/lib/python2.7/dist-packages/Abe/abe.pyc'>

[John Tobey]

This is the output

Code:

<module 'Abe.abe' from '/usr/local/lib/python2.7/dist-packages/Abe/abe.pyc'>

This means you are getting the previously installed version.  You want the one from from 'Abe/abe.py'.  Either install the new version (as root, python setup.py install) or set PYTHONPATH to the git clone directory (bitcoin-abe, not the Abe subdirectory).  Normally, sys.path starts with an empty string, so if you "cd" to the correct directory, Python should find the local modules.