[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[John Tobey]

I was expecting 50*210000 + 25*(height-209999) as well, that was why I was wondering if it added fees or something.

chain/Bitcoin/q/totalbc/100000 : 5000049.98
chain/Bitcoin/q/totalbc/200000 : 10000037.06486171
chain/Bitcoin/q/totalbc/220998 : 10774961.89978955

The numbers are actually below what the formula gives.  After block 100000, there had been 100001 blocks (counting block 0), so the expected total was 5000050, 0.02 more than actual.  After block 200000, expected 10000050, 12.93513829 more than found.  After 220998, expected 10774975, 13.10021045 more than found.  It looks okay to me, although we could take off another 100 for at least two cases of duplicate 50BTC coinbase transactions, if I recall correctly.

[Jouke]

I was expecting 50*210000 + 25*(height-209999) as well, that was why I was wondering if it added fees or something.

chain/Bitcoin/q/totalbc/100000 : 5000049.98
chain/Bitcoin/q/totalbc/200000 : 10000037.06486171
chain/Bitcoin/q/totalbc/220998 : 10774961.89978955

The numbers are actually below what the formula gives.  After block 100000, there had been 100001 blocks (counting block 0), so the expected total was 5000050, 0.02 more than actual.  After block 200000, expected 10000050, 12.93513829 more than found.  After 220998, expected 10774975, 13.10021045 more than found.  It looks okay to me, although we could take off another 100 for at least two cases of duplicate 50BTC coinbase transactions, if I recall correctly.

Interesting to see those numbers

/tx/2c637592a4b4a95cf4b19260730c66de540d7d3b14d8d352de591c5ee6eac0fc indeed has bitcoins destroyed

[Jouke]

I've reimplemented /unspent so it should outperform /address. 

Let me know if this helps.

It works great

[Jouke]

Am I correct that firstbit is not yet available trough api-calls?

[John Tobey]

Am I correct that firstbit is not yet available trough api-calls?

No, there is /q/fb/ADDRESS and /q/addr/FIRSTBITS.  However, firstbits is disabled by default, since it requires a big table.  (Something like 99% of all addresses are single-use "change addresses" but must be remembered for firstbits.)  Also, please note that Abe's algorithm probably differs slightly from firstbits.com and blockchain.info's.  See elsewhere in this thread for details.

More info, including how to turn on Abe's firstbits support, is in README-FIRSTBITS.txt.

[Jouke]

Am I correct that firstbit is not yet available trough api-calls?

No, there is /q/fb/ADDRESS and /q/addr/FIRSTBITS.  However, firstbits is disabled by default, since it requires a big table.  (Something like 99% of all addresses are single-use "change addresses" but must be remembered for firstbits.)  Also, please note that Abe's algorithm probably differs slightly from firstbits.com and blockchain.info's.  See elsewhere in this thread for details.

More info, including how to turn on Abe's firstbits support, is in README-FIRSTBITS.txt.

Oh nice!

In readme.txt: "The Abe.reconfigure module turns firstbits on and off once you have upgraded Abe's schema."

How do I updrage Abe's schema?

[John Tobey]

Oh nice!

In readme.txt: "The Abe.reconfigure module turns firstbits on and off once you have upgraded Abe's schema."

How do I updrage Abe's schema?

You already have a schema supporting firstbits, so just ignore that step.  Sorry it's unclear, but upgrading is for databases created by earlier versions of Abe, before firstbits support.  Anyway, to upgrade, just use the new version of bitcoin-abe and include "--upgrade" after "python -m Abe.abe".

[John Tobey]

I've added a branch to the code repository: no-statistics.

no-statistics is an experimental branch of Abe that lacks certain features present in the master branch.  The features are Percent Coin-Days Destroyed and Chain Age.  Removing these features simplifies the code and reduces Abe's storage and processing requirements.

I have only lightly tested it, but I consider it the direction of future development.  I would like to reimplement the original statistics and others, but their inclusion should be a configurable option so as not to burden applications that do not need them.

There is not (yet) an option to convert a database in the original format to one that uses the no-statistics branch or vice versa.

[John Tobey]

I've added a branch to the code repository: no-statistics.

This branch can now TRIM SPENT OUTPUTS, transactions, and addresses from the database.  It is not polished code, but it basically seems to work.  Loading the blockchain may be a little slower than before, since it does all the same SQL work, plus some deleting.  The space reduction appears to be "only" around 50% as of Block 134000 (under PostgreSQL 8.4).  This may be due in part to all those early unspent coinbase outputs.

To use the new feature, pass --default-trim-depth=6 (EDIT: make that --default-trim-depth=40 thanks to the March 12 fork) when creating the database.  Replace "6" with a higher number for defence against corruption by long side chains, at the cost of slightly more space.  See "default-trim-depth" in abe.conf for details and caveats.

[molecular]

it's awesome to see you keep improving.

I don't have use for the newer features, but let me say I frequently use bitcoin-abe for various analyses on the blockchain.

A big thanks!

[Jouke]

Wow, that is seriously awesome! I am going to set up a public server so everyone is able to use this great piece of software.

[John Tobey]

I am not receiving email at John.Tobey@gmail.com since three days ago.  If you sent email there in the last 72 hours, it did not reach me, and GMail is not sending delivery failure messages.  I've filled out the forms and await a solution in the next couple of days, or I'll be an ex-GMail user.  I'll check my personal messages here.

Sorry for any inconvenience.

[K1773R]

i stoped supporting I0Coin due to its memhogging, how do i cleanly delete the I0Coin chain from the MySQL DB? (already uncommented it in abe.conf)

[John Tobey]

i stoped supporting I0Coin due to its memhogging, how do i cleanly delete the I0Coin chain from the MySQL DB? (already uncommented it in abe.conf)

Sorry, I haven't written this feature.  You can remove the currency from the main page by finding its chain_id (in the chain table) and deleting all matching rows from chain_candidate and chain, I think.  It's been a while since I did this.

Then reclaiming the block and tx space is a matter of identifying rows in a half dozen tables and deleting them in dependency order, being careful not to delete other chains' data.  Or you could disable keys, delete in any order, and enable keys again, but this is not advisable while other chains may be loading.  I seem to remember "clean" chain removal taking me 20-30 minutes.  In 1-2 hours I could develop a utility that does it all, but it's still a to-do.

* Admin interface to delete chains, etc.

[SgtSpike]

Is there a list somewhere of all of the publicly-available servers/websites running Abe?  When blockchain.info was down, I went looking for one, but couldn't find any that were still running.

[John Tobey]

Is there a list somewhere of all of the publicly-available servers/websites running Abe?  When blockchain.info was down, I went looking for one, but couldn't find any that were still running.

The wiki page links to http://abe.bitcoinstats.org:2750/, which appears current.

[notawake]

My Abe installation was stuck on Namecoin block 99501 (2013-03-10 13:15:28). Attempting to rescan the block chain yielded the following error:

Code:

block 326226 already in chain 3
commit
Exception at 463166665
Failed to catch up {'blkfile_number': 1, 'dirname': '/home/notawake/.namecoin', 'chain_id': None, 'id': Decimal('2'), 'blkfile_offset': 463114051}
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2428, in catch_up
    store.catch_up_dir(dircfg)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2493, in catch_up_dir
    store.import_blkdat(dircfg, ds, blkfile['name'])
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2626, in import_blkdat
    store.import_block(b, chain_ids = chain_ids)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 1662, in import_block
    tx['tx_id'] = store.import_and_commit_tx(tx, pos == 0)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2147, in import_and_commit_tx
    tx_id = store.import_tx(tx, is_coinbase)
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 2091, in import_tx
    store.binin(txout['scriptPubKey']), pubkey_id))
  File "/usr/local/lib/python2.7/dist-packages/Abe/DataStore.py", line 464, in sql
    store.cursor.execute(cached, params)
DataError: value too long for type character varying(20000)

The next Namecoin block, 99502, has a size of 52606 bytes. In order to allow 99502 (and subsequent blocks) to be imported into the Abe database, I dropped the txout_detail view (because PostgreSQL does not allow changing the column types in tables used by views), removed the length restriction from the column txout_scriptpubkey in the table txout, and recreated the txout_detail view.

I'm sure this solution is dangerous if someone manages to stuff much larger blocks into a block chain, but I do not know a reasonable limit for this column yet.

[John Tobey]

The next Namecoin block, 99502, has a size of 52606 bytes. In order to allow 99502 (and subsequent blocks) to be imported into the Abe database, I dropped the txout_detail view (because PostgreSQL does not allow changing the column types in tables used by views), removed the length restriction from the column txout_scriptpubkey in the table txout, and recreated the txout_detail view.

I'm sure this solution is dangerous if someone manages to stuff much larger blocks into a block chain, but I do not know a reasonable limit for this column yet.

Thanks for the report.  It's too bad that junk got into the Namecoin blockchain.  Do you have the actual script length?  http://explorer.dot-bit.org/tx/1474553 shows the transaction size as 32767, and the long script would be most of that, but that explorer may have truncated it.  You can verify this with something like:

Code:

SELECT MAX(LENGTH(txout_scriptPubKey)) FROM txout JOIN tx USING (tx_id) WHERE tx_hash = '0bb558f73a543f2631acbd8c5614d3ed2171eb710a586b4485b8303d4a4a0b61';

I'll consider simply increasing the column width, but that has the side effect of making MySQL create the column as type TEXT rather than VARCHAR (unless using binary-type=binary).  This may not be so bad.  The column does not take part in any joins, and the vanilla app never searches it.  Anyone?

[John Tobey]

FYI, http://abe.john-edwin-tobey.org is current with the BTC chain using the no-statistics branch and the HOMEPAGE template variable to make chain/Bitcoin the default view.  It uses the new abe_loader script (in tools/) to stay up-to-date, and the FCGI process has read-only SQL permissions.

Problems?  Please report them here or by email to jtobey@john-edwin-tobey.org.

[ShadesOfMarble]

Any news on multisig support? Are you working on it or waiting for donations?