[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[John Tobey]

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install.

Code:

  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

I won't be surprised if SC2 has changed the Merkle tree hashing algorithm.  If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py:

Code:

        # Verify Merkle root.
        if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
            raise MerkleRootMismatch(b['hash'], tx_hash_array)

A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain.

[1714244002]

1714244002

[1714244002]

1714244002

[tiker]

Thanks.  Don't know if I trust it or not.. I'll send some SC from my windows client to this linux install and see if they show up.. guess that's the only way to tell.

Thanks.

[eurekafag]

Please run the script in this post: https://bitcointalk.org/index.php?topic=22785.msg707302#msg707302

I've committed a small change that I hope will stop these MySQL duplicate key errors.

Thanks, now it works. Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.

[John Tobey]

Just out of curiosity why don't you support MyISAM engine? I tried replacing innodb with myisam in the source but then abe complains about inability to create a transactional database. Isn't MyISAM faster than InnoDB? I don't know just read some articles comparing them so I'm asking you.

As I understand, MyISAM is non-transactional.  It does not support rollback.  See: http://en.wikipedia.org/wiki/ACID

Abe relies on the ability to roll back the current transaction and recover when interrupted or in the event of an error.  You may find MyISAM works for a while, but when something goes wrong, typically you will have to rebuild the database.  With transactional tables, it normally suffices to restart Abe.

[eurekafag]

Hmm, I didn't experience something like this but it could be crucial for abe. Thanks for your help and answers, keep up the good work!

[John Tobey]

Simple answer, Abe runs on many databases, all of them transactional.  This knowledge permits one to code with a certain set of assumptions that may not hold in a MyISAM environment.  Fearing inconsistencies in users' MyISAM tables, I put in the check that you discovered.

I think the changes needed to support MyISAM's semantics safely would be too invasive.  To use MyISAM unsafely is easy, just disable the check in Abe.DataStore._test_transaction, but don't expect support when you start to see wrong data.

[tiker]

I've seen somewhere an Abe install with the SolidCoin2 chain but I can't seem to get it going on my install.

Code:

  File "Abe/DataStore.py", line 1493, in import_block
    raise MerkleRootMismatch(b['hash'], tx_hash_array)
MerkleRootMismatch: Block header Merkle root does not match its transactions. block hash=725ca2009ba1b949a820d3a48511dc21c7f8d76d06d8afb6abb5988f5a911a8b

I won't be surprised if SC2 has changed the Merkle tree hashing algorithm.  If you trust your copy of the block chain, you may get around this by simply deleting these lines from Abe/DataStore.py:

Code:

        # Verify Merkle root.
        if b['hashMerkleRoot'] != util.merkle(tx_hash_array):
            raise MerkleRootMismatch(b['hash'], tx_hash_array)

A long-term solution would be to add support for different algorithms per chain, or at least an option to skip the verification in a given chain.

Just an FYI, after doing some checking it turns out the block chain for SC2 isn't downloading correctly and is getting stuck.  The windows client is stuck at block 17999 (like the other posts on this board) so I can't trust anything coming down for SC2 at the moment.  I'm leaving the above lines in the code.

Thanks.

[kstepyra]

Hi there,

I've started using abe(initial check right now ), had to quit app and rerun it and after using ctrl-c  i lack one record :

Code:

mysql> SELECT count(*) FROM `block` WHERE block_id=116245;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM `block` WHERE block_id=116246;
+----------+
| count(*) |
+----------+
|        0 |
+----------+
1 row in set (0.00 sec)

mysql> SELECT count(*) FROM `block` WHERE block_id=116247;
+----------+
| count(*) |
+----------+
|        1 |
+----------+
1 row in set (0.00 sec)

commit should prevent that, right? i am ussing InnoDB.

Is here any possibility to recheck for missing data? EDIT: commits are only for tx's so maybe just wrong id given while selecting last block in app after starting it? like 116247 instead of 116246 while having 116245 put in mysqlDb?

Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?

and most important: how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p

[John Tobey]

had to quit app and rerun it and after using ctrl-c  i lack one record :

More likely it seems to me, you have one unassigned block_id.  This is not the same as a missing block.  block_id is an internal identifier, not visible in any HTML or API output.  So it doesn't really matter what the block_id is.  It will vary from one Abe instance to another for the same block.  Abe allocated a block_id but hadn't used it when you stopped it, so it remains unused.  This is not a problem.

By the way, the visible "block number" is called "height" internally.  Often height and block_id are numerically close, depending on how you load the data, but they will diverge if, for example, your block file contains side chains or you load Testnet or alt chain data along with the main BTC block chain.

Is here any possibility to recheck for missing data?

It would be nice to have a validation tool that checks for missing data.  We don't have that.  Part of it is covered by the foreign key constraints, and there is a tool (Abe/verify.py) that verifies the transaction Merkle hashes, but it could do much more.

If you haven't done anything like delete table rows, and barring any bugs, if Abe shows a block number, it has all data from the genesis through that block.

Another thing - when i have bitcoin running - can i rerun app to check for new blocks, or i have to stop bitcoin?

I have never noticed a problem running both at once.

And btw - how is this tool different from Bitcointools ? i am new to this tools, was using crappy json-rpc methods given in bitcoind : p

Abe borrows some low-level code from Bitcointools and adds many features, notably SQL database support, speed improvements through indexing, and the HTML interface similar to Bitcoin Block Explorer.

[tiker]

Another question here...

http://blockexplorer.funkymonkey.org/block/00000000bc06d31f6fc31e99e3be7adfbc47c04e9bf56cddf1accda6cac69529

What does it mean when there are 2 "next block" links, one looking to be a valid block and 1 appearing to be part of a different chain?  I'm would take a guess and say 1 got orphaned and the other was accepted but the incorrect "to" address makes it look like it's part of another chain.


Edit:  Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.

[John Tobey]

Edit:  Now that I look at it a bit more.. I think it is an orphaned block but also a bug where the block version code isn't being applied to properly show the correct addresses.

Correct you are.

[tiker]

I've run into a new problem.  Trying to import the bitcoin testnet, was working fine then this came up...

Code:

Failed to catch up {'blkfile_number': 1, 'dirname': '/home/ed/.bitcoin/testnet', 'chain_id': 20, 'id': Decimal('95'), 'blkfile_offset': 30108327}
Traceback (most recent call last):
  File "Abe/DataStore.py", line 2146, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe/DataStore.py", line 2167, in catch_up_dir
    store.import_blkdat(dircfg, ds)
  File "Abe/DataStore.py", line 2279, in import_blkdat
    b = deserialize.parse_Block(ds)
  File "Abe/deserialize.py", line 176, in parse_Block
    d['auxpow'] = parse_AuxPow(vds)
  File "Abe/deserialize.py", line 153, in parse_AuxPow
    d = parse_MerkleTx(vds)
  File "Abe/deserialize.py", line 103, in parse_MerkleTx
    d = parse_Transaction(vds)
  File "Abe/deserialize.py", line 86, in parse_Transaction
    d['txIn'].append(parse_TxIn(vds))
  File "Abe/deserialize.py", line 45, in parse_TxIn
    d['sequence'] = vds.read_uint32()
  File "Abe/BCDataStream.py", line 71, in read_uint32
    def read_uint32(self): return self._read_num('<I')
  File "Abe/BCDataStream.py", line 110, in _read_num
    (i,) = struct.unpack_from(format, self.input, self.read_cursor)
OverflowError: Python int too large to convert to C long

Other chains continue to import fine so far from what I can tell.  It seems to be just this current spot in this chain.

[John Tobey]

The testnet contains a block with an unusual value in the version field.  Abe thinks it is a merged-mining block.  I mean to provide a workaround, especially since the main BTC chain is vulnerable to it, but haven't got to it yet.  Abe will need a new kind of chain metadata with flags for features such as merged mining, rather than rely on a universal interpretation of block version.

[tiker]

So I'm stuck at block 42156 on the testnet for now then?  It doesn't sound like a quick upgrade / change to Abe. 

[John Tobey]

So I'm stuck at block 42156 on the testnet for now then?  It doesn't sound like a quick upgrade / change to Abe.  

To be compatible with Testnet, but incompatible with Namecoin and chains that use its proof-of-work format, remove or comment out these lines in Abe/deserialize.py, lines 175-176 in the latest commit:

Code:

  if d['version'] & (1 << 8):
    d['auxpow'] = parse_AuxPow(vds)

To be compatible with both, until I have a better fix, I suggest you run this query:

Code:

SELECT DISTINCT c.chain_name, b.block_version
FROM block b
JOIN chain_candidate cc ON (b.block_id = cc.block_id)
JOIN chain c ON (cc.chain_id = c.chain_id)

You will see something like:

Code:

 chain_name | block_version 
------------+---------------
 NcTestnet  |         65537
 NcTestnet  |             1
 Namecoin   |             1
 Namecoin   |         65537
 Namecoin   |         65793
 Bitcoin    |             1
 NcTestnet  |         65793

Make a list of the values under block_version that belongs to a network with Namecoin-style merged mining.  For the above output, the list would be (1, 65537, 65793) corresponding to Namecoin and NcTestnet.  Then add " and d['version'] in (LIST)" after "d['version'] & (1 << " on line 175, e.g.:

Code:

  if (d['version'] & (1 << 8)) and d['version'] in (1, 65537, 65793):
    d['auxpow'] = parse_AuxPow(vds)  

Let us know how it goes.

[John Tobey]

A less fragile workaround would be to run a separate Abe process for loading Testnet using --no-serve and a copy of the software with the lines commented out.  Your server process would not be able to add new testnet blocks at every request the way it does for other chains, and you would have to schedule the Testnet process to run every minute or so.  But you might prefer this, since it should survive any new block versions introduced by the chains.

[tiker]

The output from the SQL query is as follows:

Code:

chain_name	block_version
Bitchip	1
Bitcoin	1
Bitcoin Testnet	1
Coiledcoin	1
Coiledcoin	1048577
Coiledcoin	1048833
Devcoin	1
Devcoin	196609
Devcoin	262145
Devcoin	262401
Fairbrix	1
GeistGeld	1
GeistGeld	257
i0coin	1
i0coin	131073
i0coin	131329
Ixcoin	1
Ixcoin	196609
Ixcoin	196865
Liquidcoin	1
Litecoin	1
Namecoin	1
Namecoin	65537
Namecoin	65793
Rucoin	1
Rucoin	196609
Rucoin	458753
Tenebrix	1

When you say namecoin like merged mining do you mean any chain that's compatible to be merged with namecoin?  In this case, it would be namecoin, ixcoin, i0coin, geistgeld, coiledcoin, devcoin and soon groupcoin (not imported yet).

That would be values  1, 1048577, 1048833, 196609, 262145, 262401, 257, 131073, 131329, 196609, 196865, 65537, 65793; and then whatever groupcoin reports and any other future chains.  Is this correct?


Would commenting out those lines to get the import running again, then un-commenting afterwords be an option or will every block from this point onwards have the problem?

[John Tobey]

The output from the SQL query is as follows:

(snipped)

Wow!  I guess everybody creates their own block version to go along with the magic number and address version.

When you say namecoin like merged mining do you mean any chain that's compatible to be merged with namecoin?  In this case, it would be namecoin, ixcoin, i0coin, geistgeld, coiledcoin, devcoin and soon groupcoin (not imported yet).

That list is probably right.  In practice, it's whoever started with a copy of Namecoin's merged-mining code and didn't change it too much.  Small chains try to be compatible with tools like Abe.  Abe tries to be compatible with big chains.

That would be values  1, 1048577, 1048833, 196609, 262145, 262401, 257, 131073, 131329, 196609, 196865, 65537, 65793; and then whatever groupcoin reports and any other future chains.  Is this correct?

That looks about right.

Would commenting out those lines to get the import running again, then un-commenting afterwords be an option or will every block from this point onwards have the problem?

I don't know.  It depends whether the Testnet block was a one-time test or that miner solved more blocks.  Strange things happen on Testnet.

[John Tobey]

So I'm stuck at block 42156 on the testnet for now then?  It doesn't sound like a quick upgrade / change to Abe. 

I've committed a fix.  By default, only Bitcoin and Testnet are allowed to have any block version number.  If there are more such chains, uncomment ignore-bit8-chains and add them to the list in abe.conf.

[tiker]

That worked. 

Uncommented the line, removed bitcoin so it shows "Bitcoin Testnet" (which is how i've named it here) and started.

First line reads:
Ignored bit8 in version 0x558d394b of chain_id 20

The import blocks are listed after that.

Thanks!