[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[madmax_ger]

Code:

ImportError: dlopen(/Users/thisuser/.python-eggs/MySQL_python-1.2.4-py2.7-macosx-10.7-intel.egg-tmp/_mysql.so, 2): no suitable image found.  Did find:
	/Users/thisuser/.python-eggs/MySQL_python-1.2.4-py2.7-macosx-10.7-intel.egg-tmp/_mysql.so: mach-o, but wrong architecture

Sorry, this is an issue with Python modules on Mac, and I have no relevant experience.  Put this in a file and run it with "python FILE.py".  It will have to complete without error before you can use Abe:

Code:

import MySQLdb

Thank you John, also for your respond on my PN.
Indeed, this helps me as a non abe-experienced to locate the mac-based problem.

In case of news I'll post.

[madmax_ger]

I got the libs running now. I had to reinstall everything in 64bit mode, I wasn't aware that 32bit won't work.

Now I've got another issue, it seems to be the last one.

Code:

$ python -m Abe.abe --config /usr/local/mysql/my-abe.cnf --commit-bytes 100000 --no-serve
Skipping datadir /Users/thisuser/Library/Application Support/Bitcoin/blocks: [Errno 2] No such file or directory: '/Users/thisuser/Library/Application Support/Bitcoin/blocks/blk0001.dat'

Path is correct and working if I try to cd in it.

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin/blocks

Any idea?

Block files are located there for sure.

Code:

$ ls
blk00000.dat	blk00002.dat	index		rev00001.dat	rev00003.dat
blk00001.dat	blk00003.dat	rev00000.dat	rev00002.dat

[John Tobey]

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin/blocks

Leave /blocks off.  This should be:

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin

[madmax_ger]

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin/blocks

Leave /blocks off.  This should be:

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin

If I leave it off, the error message is "../Application Support/Bitcoin/blck0001.dat not found".
In case I use /blocks, the error message points correctly to "../Application Support/Bitcoin/blocks/blck0001.dat".
Now I am even more confused.

While ABE searches after blck0001.dat, blck00001.dat is existent, and would be perhaps the correct file?
Is this a misconfiguration?

The blockchain loading is still at the very beginning, I just ran bitcoin-qt once, as listed in the read me.

[robanswe]

Now after about 14 days of importing it's finally done Anyway everything seems to be working just fine I only got one small problem. It's only importing new blocks when I load a new webpage in the Abe interface. Should it really be like that? So if I for example stop using the web interface for about 1 hour then when I request one page load it will star importing the 6 missing blocks. Why doesn't Abe just import the block as soon my bitcoin-qt instance accepts the new block?



My config:

Code:

dbtype MySQLdb
connect-args {"user":"****","db":"abe","port":3306,"host":"127.0.0.1"}
#datadir =C:\****
default-loader =rpc
datadir [{"dirname":"C:\\*****","chain":"Bitcoin","loader":"rpc"}]
upgrade
port 2750

My os:
Windows 7 64bit

Also note that I edited the database manually to get rpc to work. Like someone else explained in this thread.








One more question is there a way to check so the information in the mysql database is intact? (That does not take 14 days)

[John Tobey]

Now after about 14 days of importing it's finally done Anyway everything seems to be working just fine I only got one small problem. It's only importing new blocks when I load a new webpage in the Abe interface. Should it really be like that? So if I for example stop using the web interface for about 1 hour then when I request one page load it will star importing the 6 missing blocks. Why doesn't Abe just import the block as soon my bitcoin-qt instance accepts the new block?

Yup, this is an underdocumented quirk.  There are many ways to keep Abe up to date, an no one fits every need.

You can use a script like abe_loader.  Actually, abe_loader is more complex than needed.  It watches bitcoind's debug.log for hints that a block has arrived.  Now that bitcoind supports blocknotify, we could just add something like this to bitcoin.conf:

Code:

blocknotify=python -m Abe.abe --config /etc/abe/abe.conf --no-serve

Another strategy is to run in a loop, like this:

Code:

while true; do python -m Abe.abe --config abe.conf --no-serve; sleep 10; done

I used to have a cron job that just loaded the homepage, serving as both a system monitor and a blockchain updater:

Code:

* * * * * /usr/bin/wget -q -O /dev/null http://john-edwin-tobey.org:2750/

One more question is there a way to check so the information in the mysql database is intact? (That does not take 14 days)

A complete check of all invariants does not exist.  It would be a nice feature.  Abe/verify.py does a small fraction of the work, calculating Merkle roots from stored transaction hashes.  I haven't tested it lately.

[TreasureSeeker]

Hello John, I'm trying to set up an Abe block explorer for ElephantCoin, but when trying to do the initial load of:

Code:

    python -m Abe.abe --config abe-my.conf --commit-bytes 100000 --no-serve

It fails with the error

Code:

error:
failed to catch up {'blkfile_offset': 0, 'blkfile_number': 1, 'chain_id': Decimal('9'), 'loader': None, 'dirname': u'/root/.elephantcoin', 'id': 5L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2596, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2854, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 2982, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "Abe/DataStore.py", line 1750, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=13405c6ebfa325bf97847d3b998341b4f3db5805c55bd413543cccb74e262258

I've tried commenting out in datastore.py:

Code:

 # Verify Merkle root.
        #if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
        #raise MerkleRootMismatch(b['hash'], tx_hash_array)

 
This allows the data to start loading but it stops after a short while

Code:

Skipped 1 bytes at block end
block_tx 641 644
Skipped 1 bytes at block end
block_tx 642 645
Skipped 1 bytes at block end
block_tx 643 646
Skipped 1 bytes at block end
block_tx 644 647
block_tx 644 648
Skipped 46 bytes at block end
Exception at 13428709007330245116
Failed to catch up {'blkfile_offset': 104283, 'blkfile_number': 1, 'chain_id': 8L, 'loader': None, 'dirname': u'/root/.elephantcoin', 'id': 1L}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2637, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2895, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3022, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "Abe/DataStore.py", line 3053, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "Abe/deserialize.py", line 90, in parse_Transaction
    d['txOut'].append(parse_TxOut(vds))
  File "Abe/deserialize.py", line 65, in parse_TxOut
    d['value'] = vds.read_int64()
  File "Abe/BCDataStream.py", line 72, in read_int64
    def read_int64  (self): return self._read_num('<q')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

The resulting block explorer (or part of one up to block 643 only, which is where it stops) can be seen on my test page at http://exploretheblocks.com:27700/chain/ElephantCoin?count=500.  (I've changed everything to port 27700 for testing purposes).  This part-block-explorer seems to have a number of very odd-looking blocks, for example block 513, 515 and 517 and the "final" block 643

My abe-my.conf:

Code:

    dbtype MySQLdb
    connect-args {"user":"abe","db":"abetest","passwd":"################################"}
    upgrade
    port 27700
    host exploretheblocks.com 


datadir += [{
  "dirname" : "/root/.elephantcoin",
  "chain": "ElephantCoin",
  "code3": "ELP",
  "address_version": "\u005c"
}]

ignore-bit8-chains = ["Bitcoin", "Testnet", "ElephantCoin"]

If you can offer any suggestions I'd be really grateful as I really like the way Abe displays its output, and would really like to get an Abe Explorer set up for the Elephants!

[diatonic]

If I remember correctly ElephantCoin incorporates transaction messages like Florincoin ... which results in some blocks which confuse Abe.

[TreasureSeeker]

Yes, you're right, Elephantcoin does have transaction messages.  I had a feeling that might be something to do with the problems I've been having.  Is there any fix that might allow an Abe Explorer to be set up?

[John Tobey]

Hi, I've been away the last few days and will take a while to catch up.

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin/blocks

Leave /blocks off.  This should be:

Code:

datadir = /Users/thisuser/Library/Application Support/Bitcoin

If I leave it off, the error message is "../Application Support/Bitcoin/blck0001.dat not found".
In case I use /blocks, the error message points correctly to "../Application Support/Bitcoin/blocks/blck0001.dat".
Now I am even more confused.

While ABE searches after blck0001.dat, blck00001.dat is existent, and would be perhaps the correct file?
Is this a misconfiguration?

Old versions of Bitcoin start with blk0001.dat, and new (0.8.x) ones start with blocks/blk00000.dat.  To accommodate all known versions, including users who upgrade, Abe first searches in datadir for the blk*.dat series, and then in the blocks subdirectory for blk00000.dat.  Since you use a new bitcoin-qt, it should not find blk0001.dat but should continue and load blocks/blk00000.dat.

The blockchain loading is still at the very beginning, I just ran bitcoin-qt once, as listed in the read me.

How many blk*.dat files are in  "../Application Support/Bitcoin/blocks" ?  Does Abe exit after the error message about blck0001.dat?  Is that a typo, and did you mean blk0001.dat?  Please paste the last several lines of output if Abe exits before it finishes loading.

[madmax_ger]

Hi John,

good to hear from you

Ok, I see. But I dont see any problem then:

Copy'n Paste:

$ sudo python -m Abe.abe --config /usr/local/mysql/my-abe.cnf --commit-bytes 100000 --no-serve
Password:
Skipping datadir /Users/btcinvest/Desktop/bitcoin-datadir: [Errno 2] No such file or directory: u'/Users/btcinvest/Desktop/bitcoin-datadir/blk0001.dat'

Btw. whats that 'u' before the path begins? Typo in your source I guess?

Copy'n Paste:

blocks btcinvest$ ls
blk00000.dat   blk00013.dat   blk00026.dat   blk00039.dat   blk00052.dat   blk00065.dat   rev00005.dat   rev00018.dat   rev00031.dat   rev00044.dat   rev00057.dat   rev00070.dat
blk00001.dat   blk00014.dat   blk00027.dat   blk00040.dat   blk00053.dat   blk00066.dat   rev00006.dat   rev00019.dat   rev00032.dat   rev00045.dat   rev00058.dat   rev00071.dat
blk00002.dat   blk00015.dat   blk00028.dat   blk00041.dat   blk00054.dat   blk00067.dat   rev00007.dat   rev00020.dat   rev00033.dat   rev00046.dat   rev00059.dat
blk00003.dat   blk00016.dat   blk00029.dat   blk00042.dat   blk00055.dat   blk00068.dat   rev00008.dat   rev00021.dat   rev00034.dat   rev00047.dat   rev00060.dat
blk00004.dat   blk00017.dat   blk00030.dat   blk00043.dat   blk00056.dat   blk00069.dat   rev00009.dat   rev00022.dat   rev00035.dat   rev00048.dat   rev00061.dat
blk00005.dat   blk00018.dat   blk00031.dat   blk00044.dat   blk00057.dat   blk00070.dat   rev00010.dat   rev00023.dat   rev00036.dat   rev00049.dat   rev00062.dat
blk00006.dat   blk00019.dat   blk00032.dat   blk00045.dat   blk00058.dat   blk00071.dat   rev00011.dat   rev00024.dat   rev00037.dat   rev00050.dat   rev00063.dat
blk00007.dat   blk00020.dat   blk00033.dat   blk00046.dat   blk00059.dat   index      rev00012.dat   rev00025.dat   rev00038.dat   rev00051.dat   rev00064.dat
blk00008.dat   blk00021.dat   blk00034.dat   blk00047.dat   blk00060.dat   rev00000.dat   rev00013.dat   rev00026.dat   rev00039.dat   rev00052.dat   rev00065.dat
blk00009.dat   blk00022.dat   blk00035.dat   blk00048.dat   blk00061.dat   rev00001.dat   rev00014.dat   rev00027.dat   rev00040.dat   rev00053.dat   rev00066.dat
blk00010.dat   blk00023.dat   blk00036.dat   blk00049.dat   blk00062.dat   rev00002.dat   rev00015.dat   rev00028.dat   rev00041.dat   rev00054.dat   rev00067.dat
blk00011.dat   blk00024.dat   blk00037.dat   blk00050.dat   blk00063.dat   rev00003.dat   rev00016.dat   rev00029.dat   rev00042.dat   rev00055.dat   rev00068.dat
blk00012.dat   blk00025.dat   blk00038.dat   blk00051.dat   blk00064.dat   rev00004.dat   rev00017.dat   rev00030.dat   rev00043.dat   rev00056.dat   rev00069.dat

As you can see my conf-file is in the mysql folder, but I guess this doesnt matter.
Also the bitcoin-datadir was customized by me to get rid of the blank space in the path, who knows.

EDIT: Abe is installed in /Library/Python/2.7/site-packages/Abe

[MobGod]

Hello John w/b just need to ask you where would I get the api for the explorer
http://blockexplorer.coinworld.us/

[John Tobey]

Hello John w/b just need to ask you where would I get the api for the explorer
http://blockexplorer.coinworld.us/

http://blockexplorer.coinworld.us/chain/AlphaCoin/q
etc. replacing AlphaCoin with coin name.

[MobGod]

Hello John w/b just need to ask you where would I get the api for the explorer
http://blockexplorer.coinworld.us/

http://blockexplorer.coinworld.us/chain/AlphaCoin/q
etc. replacing AlphaCoin with coin name.

Thanks John I hope that Is all they need

[diatonic]

Hey John, I scanned a coin in with the wrong address version, and now addresses appear incorrectly. I would reload te whole database, but there are 36 coins and it would take over 24 hours. Is there a way to drop a single chain and reload it?

[John Tobey]

Hey John, I scanned a coin in with the wrong address version, and now addresses appear incorrectly. I would reload te whole database, but there are 36 coins and it would take over 24 hours. Is there a way to drop a single chain and reload it?

If you have not enabled firstbits, use SQL to update chain.chain_address_version with the correct value.  That should magically fix all the addresses.  To find the correct value, you could start loading the chain in a new database, identically configured except for that address version.  Then:

SQL> SELECT chain_name, chain_address_version FROM chain;

SQL> UPDATE chain SET chain_address_version='value' WHERE chain_name='chain';

This may require conversion to and from a printable format such as hex, if you use binary_type=binary, which is now the default.  In MySQL, replace chain_address_version with HEX(chain_address_version) in the select statement, and replace 'value' with UNHEX('hex value') in the update.

Ideally, we would have a command-line and web interface to manage chains.

[diatonic]

Thanks John, that fixed my issue. Now I just need to figure out why Megacoin & Bytecoin aren't updating. They both use the new 0.8 /blocks/ format.

When I manually run it and it hits Megacoin I get

Code:

Opened /home/diatonic/.megacoin/blocks/blk00000.dat
Exception at 15709119118651544644
Failed to catch up {'blkfile_offset': 25339280, 'blkfile_number': 100000, 'chain_id': 26, 'loader': None, 'dirname': '/home/diatonic/.megacoin', 'id': Decimal('26')}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2632, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2890, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "Abe/DataStore.py", line 3015, in import_blkdat
    b = store.parse_block(ds, chain_id, magic, length)
  File "Abe/DataStore.py", line 3046, in parse_block
    d['transactions'].append(deserialize.parse_Transaction(ds))
  File "Abe/deserialize.py", line 90, in parse_Transaction
    d['txOut'].append(parse_TxOut(vds))
  File "Abe/deserialize.py", line 65, in parse_TxOut
    d['value'] = vds.read_int64()
  File "Abe/BCDataStream.py", line 72, in read_int64
    def read_int64  (self): return self._read_num('<q')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

With Bytecoin it appears to add the new blocks but it's not showing them.

[John Tobey]

Thanks John, that fixed my issue. Now I just need to figure out why Megacoin & Bytecoin aren't updating. They both use the new 0.8 /blocks/ format.

When I manually run it and it hits Megacoin I get

Code:

Opened /home/diatonic/.megacoin/blocks/blk00000.dat
Exception at 15709119118651544644
[...]
OverflowError: Python int too large to convert to C long

This is typical of ppcoin-derived currencies.  Have you tried the "ppcoin" branch?  See: https://github.com/jtobey/bitcoin-abe/issues/4

With Bytecoin it appears to add the new blocks but it's not showing them.

This can result from a block being skipped or incorrectly loaded.  Have you tried rescanning the blockfiles?  Use --rescan or, if you have many coins in one database, you can rescan just the one by setting blkfile_number=1, blkfile_offset=0 in the appropriate row in the datadir table.

[MobGod]

Hello John again lol could you please look at this I'm going crazy on how to fix it mega
http://blockexplorer.coinworld.us/

[Mooshire]

This is cool, good project.