Print Page - [update] ABE: AGPL Block Explorer on github

Title: [update] ABE: AGPL Block Explorer on github
Post by: John Tobey on June 13, 2011, 04:32:03 AM

Edit: latest features & discussion: https://bitcointalk.org/index.php?topic=22785

Tired of waiting for theymos to release the original Block Explorer (http://blockexplorer.com/) code, just before khal's Namecoin explorer (http://explorer.dot-bit.org/) announcement, I went and started my own based on Gavin's bitcointools.

Code: https://github.com/jtobey/bitcoin-abe
Service: http://abe.john-edwin-tobey.org/ and anywhere you host it

June 26 updates in orange
Features:

free and open source (GNU Affero General Public License)
block, transaction, and address history pages similar to http://blockexplorer.com/
block/tx/address search box
does not require patching bitcoind (uses slightly modified bitcointools, not getblock)
low block delay: often shows blocks not yet on http://blockexplorer.com/
chain page can list more than the 20 newest blocks
can handle several chains in the same database (Bitcoin, Testnet, Namecoin, ...)
real-time Bitcoin Days Destroyed (https://en.bitcoin.it/wiki/Bitcoin_Days_Destroyed) and Average Coin Age data
written in Python and portable SQL, tested with PostgreSQL and Sqlite

Coming later, maybe:

speedups to match http://blockexplorer.com/, I hope
stats pages/API like http://blockexplorer.com/q
see the TODO.txt file (https://github.com/jtobey/bitcoin-abe/blob/master/TODO.txt)
support for indirect block hashes along these (https://en.bitcoin.it/wiki/Alternative_Chains) lines
Whatever you implement

Enjoy!

Title: Re: AGPL Block Explorer on github
Post by: AntiVigilante on June 13, 2011, 07:34:48 AM

Egg celery ant.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: just_someguy on June 23, 2011, 08:15:20 PM

Very nice!
Do you plan on implementing the json return option for queries?

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: John Tobey on June 23, 2011, 09:20:35 PM

Quote from: just_someguy on June 23, 2011, 08:15:20 PM

Very nice!

Thanks!

Quote

Do you plan on implementing the json return option for queries?

Do you mean the "raw" block and transaction pages? It's on the to-do, but not near the top. Patches are welcome. :)

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: just_someguy on June 23, 2011, 09:53:37 PM

Yes, the raw stuff.

I ask because I believe writing against blockexplorer like implementations will be more common than writing against bitcoin itself... especially for ecommerce when you are generally only on the receiving side.

Not that theymos's isn't trustworthy but I wouldn't want to count on it being around if I was relying on it to run my software. Its also not fair to him.

Even for sending I think it will be a lost easier to have a small program decrypt an offline wallet, query your addresses, gather inputs for transaction creation, send, then shutdown.
It relies on having a trusted (and most likely private) db to query against so I think an open source version is very important.
Since it seems like blockexplorer won't be open sourced I'm glad to see someone is taking this on.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: John Tobey on June 27, 2011, 07:26:13 PM

Announced under Project Development: http://forum.bitcoin.org/index.php?topic=22785.0

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on September 27, 2011, 07:00:17 PM

If a chain changes the directory where it keeps the block chain file, is it possible to change the configuration of Abe to get it continuing reading from the new directory without needing to completely rebuild the database? I've tried changing it both in the abe.conf file and directly on the datadir table in the database to no avail.

If not, is it at least possible to start again with such a chain, leaving the rest of the chains as they are in the database? Thank you!

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: John Tobey on September 27, 2011, 08:51:25 PM

Quote from: ama on September 27, 2011, 07:00:17 PM

You shouldn't need to rebuild the database or even the chain. If blk0001.dat is the same, except for possibly being appended to (by bitcoind) it should have no problem when you update datadir.dirname and restart the process. Even if blk0001.dat is different (bitcoind restarted from an empty dir or copied blk0001.dat from another host) the load should go 10-100x faster, because Abe assumes that it has loaded a block when it recognizes the header hash and skips ahead in the file.

Please provide details of what you did and what error you got.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on September 27, 2011, 11:31:57 PM

Quote from: John Tobey on September 27, 2011, 08:51:25 PM

Please provide details of what you did and what error you got.

I changed the datadir both in abe.conf and in the datadir table in the database. The chain block was downloaded new. No new blocks were added to the database. I didn't get any error, it simply didn't update anymore. Then I tryed replacing the contents of the datadir by the old ones, restarted the chain daemon, which continued adding blocks to the chain, and Abe worked fine too. So, finally it worked out, with the old blk0001.dat file, but not with a new blk0001.dat file.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: John Tobey on September 28, 2011, 01:02:51 AM

Quote from: ama on September 27, 2011, 11:31:57 PM

This needs documenting, but in the log perhaps it mentioned forcing a rescan with "UPDATE datadir SET blkfile_number=1, blkfile_offset=0 WHERE dirname='...'"

(Replace "..." with the dirname.)

This ought to let it use the new file, and it will scan it 10x faster (or more) than if it had to rebuild the data structures. I hope one day we'll have an administrative web interface to perform this task and others, like configuring chains, etc.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on September 28, 2011, 03:01:11 AM

Quote from: John Tobey on September 28, 2011, 01:02:51 AM

Yes, I see. In the README.txt file it says:

"If you rebuild your block file, you must rebuild Abe's database or at least set datadir.blkfile_offset = 0 to force a rescan. Abe does not currently handle block file changes gracefully."

It wasn't very clear to me how or where to set it, I didn't interpret it as updating a field in the database, so I tried setting datadir.blkfile_offset = 0 in the conf file, which made the daemon to fail to launch, of course. I shpuld have known better! ;D

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on September 30, 2011, 06:58:26 PM

Is it possible to rebuild the data of a single chain or maybe delete a chain and add it again?

I'm having some problems with a duplicated key in a chain which prevents the new blocks to be added to the database.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on September 30, 2011, 10:09:53 PM

I have tried seting blkfile_offset to 0 in the database, delete all the data directory and download the block chain again, and it hasen't helped. It keeps saying there is a duplicated key and refuses to save any more data into the database.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on October 01, 2011, 02:32:07 AM

According to http://www.ruby-forum.com/topic/64428, the problem is caused by a bug in PostgreSQL. They also explain what the solution is, but I cannot understand how they know what to do with the information provided by the OP, so I cannot really apply it to my problem here. Apparently simply reindexing a table should fix it, but I can't see what table needs reindexing from the error I'm getting:

Warning: failed to catch up /home/ama/.tenebrix/tenebrix_data: duplicate key violates unique constraint «block_txin_pkey»
{'blkfile_number': 1, 'dirname': '/home/ama/.tenebrix/tenebrix_data', 'chain_id': 11, 'id': Decimal('77'), 'blkfile_offset': 2386329}

I've reindexed block_txin, to no avail. Any idea, please?

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on October 01, 2011, 03:19:31 AM

Quote from: ama on October 01, 2011, 02:32:07 AM

I've reindexed block_txin, to no avail. Any idea, please?

I've asked in #PostgreSQL @ Freenode and they say there is nothing to fix in the database, because there is no index being used. They say it's an application problem. I guess I made a mistake while trying to run two instances of ABE (one for adding a chain and another one for the web service).

So, I'm back at the begining. Can I somehow eliminate the chain to add it again later or do I need to destroy the database and build it again entirely (all chains)?

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on October 01, 2011, 06:41:50 PM

I've tried to build a different database with this chain with the idea that if it worked fine, I could then add the rest of the chains and replace the original database with the new one, but I'm facing the same problem. So, it doesn't seem as if I made a mistake the first time, but as if the chain has some blocks with duplicated transactions. I don't think there is anything which can be done at the chain level. Is there a way to cope with such problem in Abe, please?

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on October 01, 2011, 07:54:50 PM

The chain is Tenebrix. It seems that block 7177 and 7320 have dupe coinbases and the second transaction in block 9971 spends one of those. It's while trying to insert this when it fails.

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: John Tobey on October 02, 2011, 01:28:05 AM

Quote from: ama on October 01, 2011, 07:54:50 PM

The chain is Tenebrix. It seems that block 7177 and 7320 have dupe coinbases and the second transaction in block 9971 spends one of those. It's while trying to insert this when it fails.

Good sleuthing! Indeed, this is a bug. While I think about the right fix, you might get past it with this:

Code:

diff --git a/Abe/DataStore.py b/Abe/DataStore.py
index a04333c..5eec013 100644
--- a/Abe/DataStore.py
+++ b/Abe/DataStore.py
@@ -1426,12 +1426,13 @@ store._ddl['txout_approx'],
 
         # Create rows in block_txin.
         for row in store.selectall("""
-            SELECT txin.txin_id, obt.block_id
+            SELECT txin.txin_id, MAX(obt.block_id)
               FROM block_tx bt
               JOIN txin ON (txin.tx_id = bt.tx_id)
               JOIN txout ON (txin.txout_id = txout.txout_id)
               JOIN block_tx obt ON (txout.tx_id = obt.tx_id)
-             WHERE bt.block_id = ?""", (block_id,)):
+             WHERE bt.block_id = ?
+             GROUP BY txin.txin_id""", (block_id,)):
             (txin_id, oblock_id) = row
             if store.is_descended_from(block_id, oblock_id):
                 store.sql("""

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: ama on October 02, 2011, 02:40:58 AM

Quote from: John Tobey on October 02, 2011, 01:28:05 AM

Good sleuthing! Indeed, this is a bug. While I think about the right fix, you might get past it with this:

Thank you for the quick fix, John. It worked fine. :)

Title: Re: [update] ABE: AGPL Block Explorer on github
Post by: Binford 6100 on October 02, 2011, 02:43:38 PM

https://bitcointalk.org/index.php?topic=22785.0
the other thread has longer 'proof of work' ; )

Bitcoin Forum

Bitcoin => Development & Technical Discussion => Topic started by: John Tobey on June 13, 2011, 04:32:03 AM