[update] ABE: AGPL Block Explorer on github

[John Tobey]

Edit: latest features & discussion: https://bitcointalk.org/index.php?topic=22785

Tired of waiting for theymos to release the original Block Explorer code, just before khal's Namecoin explorer announcement, I went and started my own based on Gavin's bitcointools.

Code: https://github.com/jtobey/bitcoin-abe
Service: http://abe.john-edwin-tobey.org/ and anywhere you host it

June 26 updates in orange
Features:

• free and open source (GNU Affero General Public License)
• block, transaction, and address history pages similar to http://blockexplorer.com/
• block/tx/address search box
• does not require patching bitcoind (uses slightly modified bitcointools, not getblock)
• low block delay: often shows blocks not yet on http://blockexplorer.com/
• chain page can list more than the 20 newest blocks
• can handle several chains in the same database (Bitcoin, Testnet, Namecoin, ...)
• real-time Bitcoin Days Destroyed and Average Coin Age data
• written in Python and portable SQL, tested with PostgreSQL and Sqlite

Coming later, maybe:

• speedups to match http://blockexplorer.com/, I hope
• stats pages/API like http://blockexplorer.com/q
• see the TODO.txt file
• support for indirect block hashes along these lines
• Whatever you implement

Enjoy!

[1714025379]

1714025379

[AntiVigilante]

Egg celery ant.

[just_someguy]

Very nice!
Do you plan on implementing the json return option for queries?

[John Tobey]

Very nice!

Thanks!

Do you plan on implementing the json return option for queries?

Do you mean the "raw" block and transaction pages?  It's on the to-do, but not near the top.  Patches are welcome. 

[just_someguy]

Yes, the raw stuff.

I ask because I believe writing against blockexplorer like implementations will be more common than writing against bitcoin itself... especially for ecommerce when you are generally only on the receiving side.

Not that theymos's isn't trustworthy but I wouldn't want to count on it being around if I was relying on it to run my software. Its also not fair to him.

Even for sending I think it will be a lost easier to have a small program decrypt an offline wallet, query your addresses, gather inputs for transaction creation, send, then shutdown.
It relies on having a trusted (and most likely private) db to query against so I think an open source version is very important.
Since it seems like blockexplorer won't be open sourced I'm glad to see someone is taking this on.

[John Tobey]

Announced under Project Development: http://forum.bitcoin.org/index.php?topic=22785.0

[ama]

If a chain changes the directory where it keeps the block chain file, is it possible to change the configuration of Abe to get it continuing reading from the new directory without needing to completely rebuild the database?  I've tried changing it both in the abe.conf file and directly on the datadir table in the database to no avail.

If not, is it at least possible to start again with such a chain, leaving the rest of the chains as they are in the database?  Thank you!

[John Tobey]

If a chain changes the directory where it keeps the block chain file, is it possible to change the configuration of Abe to get it continuing reading from the new directory without needing to completely rebuild the database?  I've tried changing it both in the abe.conf file and directly on the datadir table in the database to no avail.

You shouldn't need to rebuild the database or even the chain.  If blk0001.dat is the same, except for possibly being appended to (by bitcoind) it should have no problem when you update datadir.dirname and restart the process.  Even if blk0001.dat is different (bitcoind restarted from an empty dir or copied blk0001.dat from another host) the load should go 10-100x faster, because Abe assumes that it has loaded a block when it recognizes the header hash and skips ahead in the file.

Please provide details of what you did and what error you got.

[ama]

Please provide details of what you did and what error you got.

I changed the datadir both in abe.conf and in the datadir table in the database.  The chain block was downloaded new.  No new blocks were added to the database.  I didn't get any error, it simply didn't update anymore.  Then I tryed replacing the contents of the datadir by the old ones, restarted the chain daemon, which continued adding blocks to the chain, and Abe worked fine too.  So, finally it worked out, with the old blk0001.dat file, but not with a new blk0001.dat file.

[John Tobey]

I changed the datadir both in abe.conf and in the datadir table in the database.  The chain block was downloaded new.  No new blocks were added to the database.  I didn't get any error, it simply didn't update anymore.  Then I tryed replacing the contents of the datadir by the old ones, restarted the chain daemon, which continued adding blocks to the chain, and Abe worked fine too.  So, finally it worked out, with the old blk0001.dat file, but not with a new blk0001.dat file.

This needs documenting, but in the log perhaps it mentioned forcing a rescan with "UPDATE datadir SET blkfile_number=1, blkfile_offset=0 WHERE dirname='...'"

(Replace "..." with the dirname.)

This ought to let it use the new file, and it will scan it 10x faster (or more) than if it had to rebuild the data structures.  I hope one day we'll have an administrative web interface to perform this task and others, like configuring chains, etc.

[ama]

This needs documenting, but in the log perhaps it mentioned forcing a rescan with "UPDATE datadir SET blkfile_number=1, blkfile_offset=0 WHERE dirname='...'"

(Replace "..." with the dirname.)

This ought to let it use the new file, and it will scan it 10x faster (or more) than if it had to rebuild the data structures.  I hope one day we'll have an administrative web interface to perform this task and others, like configuring chains, etc.

Yes, I see.  In the README.txt file it says:

"If you rebuild your block file, you must rebuild Abe's database or at least set datadir.blkfile_offset = 0 to force a rescan.  Abe does not currently handle block file changes gracefully."

It wasn't very clear to me how or where to set it, I didn't interpret it as updating a field in the database, so I tried setting datadir.blkfile_offset = 0 in the conf file, which made the daemon to fail to launch, of course.   I shpuld have known better!   

[ama]

Is it possible to rebuild the data of a single chain or maybe delete a chain and add it again?

I'm having some problems with a duplicated key in a chain which prevents the new blocks to be added to the database.

[ama]

I have tried seting blkfile_offset to 0 in the database, delete all the data directory and download the block chain again, and it hasen't helped.  It keeps saying there is a duplicated key and refuses to save any more data into the database.

[ama]

According to http://www.ruby-forum.com/topic/64428, the problem is caused by a bug in PostgreSQL.  They also explain what the solution is, but I cannot understand how they know what to do with the information provided by the OP, so I cannot really apply it to my problem here.  Apparently simply reindexing a table should fix it, but I can't see what table needs reindexing from the error I'm getting:

Warning: failed to catch up /home/ama/.tenebrix/tenebrix_data:  duplicate key violates unique constraint «block_txin_pkey»
{'blkfile_number': 1, 'dirname': '/home/ama/.tenebrix/tenebrix_data', 'chain_id': 11, 'id': Decimal('77'), 'blkfile_offset': 2386329}


I've reindexed block_txin, to no avail.  Any idea, please?

[ama]

I've reindexed block_txin, to no avail.  Any idea, please?

I've asked in #PostgreSQL @ Freenode and they say there is nothing to fix in the database, because there is no index being used.  They say it's an application problem.  I guess I made a mistake while trying to run two instances of ABE (one for adding a chain and another one for the web service).

So, I'm back at the begining.  Can I somehow eliminate the chain to add it again later or do I need to destroy the database and build it again entirely (all chains)?

[ama]

I've tried to build a different database with this chain with the idea that if it worked fine, I could then add the rest of the chains and replace the original database with the new one, but I'm facing the same problem.  So, it doesn't seem as if I made a mistake the first time, but as if the chain has some blocks with duplicated transactions.  I don't think there is anything which can be done at the chain level.  Is there a way to cope with such problem in Abe, please?

[ama]

The chain is Tenebrix.  It seems that block 7177 and 7320 have dupe coinbases and the second transaction in block 9971 spends one of those.  It's while trying to insert this when it fails. 

[John Tobey]

The chain is Tenebrix.  It seems that block 7177 and 7320 have dupe coinbases and the second transaction in block 9971 spends one of those.  It's while trying to insert this when it fails. 

Good sleuthing!  Indeed, this is a bug.  While I think about the right fix, you might get past it with this:

Code:

diff --git a/Abe/DataStore.py b/Abe/DataStore.py
index a04333c..5eec013 100644
--- a/Abe/DataStore.py
+++ b/Abe/DataStore.py
@@ -1426,12 +1426,13 @@ store._ddl['txout_approx'],
 
         # Create rows in block_txin.
         for row in store.selectall("""
-            SELECT txin.txin_id, obt.block_id
+            SELECT txin.txin_id, MAX(obt.block_id)
               FROM block_tx bt
               JOIN txin ON (txin.tx_id = bt.tx_id)
               JOIN txout ON (txin.txout_id = txout.txout_id)
               JOIN block_tx obt ON (txout.tx_id = obt.tx_id)
-             WHERE bt.block_id = ?""", (block_id,)):
+             WHERE bt.block_id = ?
+             GROUP BY txin.txin_id""", (block_id,)):
             (txin_id, oblock_id) = row
             if store.is_descended_from(block_id, oblock_id):
                 store.sql("""

[ama]

Good sleuthing!  Indeed, this is a bug.  While I think about the right fix, you might get past it with this:

Thank you for the quick fix, John.  It worked fine.   

[Binford 6100]

https://bitcointalk.org/index.php?topic=22785.0
the other thread has longer 'proof of work' ; )