[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[K1773R]

Code:

Traceback (most recent call last):
  File "/usr/lib/python2.6/wsgiref/handlers.py", line 93, in run
    self.result = application(self.environ, self.start_response)
  File "/usr/local/lib/python2.6/dist-packages/Abe/abe.py", line 205, in __call__
    handler(page)
  File "/usr/local/lib/python2.6/dist-packages/Abe/abe.py", line 459, in handle_chain
    ss = int(ss)
TypeError: int() argument must be a string or a number, not 'NoneType'

Thanks for the report.  I've replied on GitHub.  Unfortunately, I don't think it will be easy to debug, since it appears to depend on the order the blocks were loaded, and that in turn depends on network timings.  I will post when I have time to address this.  Meanwhile, I suggest using an older version of Abe, either v0.6 or the Sept. 15 commit (fc10484).

lets continue the chat @ github

[molecular]

I just produced a dataset called "difficulty and hashrate by day" using bitcoin-abe (thanks so much, John Tobey, I love bitcoin-abe): https://bitcointalk.org/index.php?topic=123238.msg1365248#msg1365248

When designing the query I wanted to see the resulting sql string. How can I set the loglevel of store.sqllogger without modifying the python code in Datastore.py?

[John Tobey]

When designing the query I wanted to see the resulting sql string. How can I set the loglevel of store.sqllogger without modifying the python code in Datastore.py?

--log-sql shows all queries' SQL and parameters.  This is currently just a boolean switch.  To set the level to something other than DEBUG, you would use log-sql and a custom logging config with (for example) "loggers":{"Abe.DataStore.sql": {"level": "INFO"}}.

[John Tobey]

Code:

Traceback (most recent call last):
  File "/usr/lib/python2.6/wsgiref/handlers.py", line 93, in run
    self.result = application(self.environ, self.start_response)
  File "/usr/local/lib/python2.6/dist-packages/Abe/abe.py", line 205, in __call__
    handler(page)
  File "/usr/local/lib/python2.6/dist-packages/Abe/abe.py", line 459, in handle_chain
    ss = int(ss)
TypeError: int() argument must be a string or a number, not 'NoneType'

Thanks for the report.  I've replied on GitHub.  Unfortunately, I don't think it will be easy to debug, since it appears to depend on the order the blocks were loaded, and that in turn depends on network timings.  I will post when I have time to address this.  Meanwhile, I suggest using an older version of Abe, either v0.6 or the Sept. 15 commit (fc10484).

I think this is fixed.  It turned out to depend not on block order, but on duplicate coinbase transactions.  If this works out, I'll call it 0.7.2.

[John Tobey]

A maintenance release to address "TypeError: int() argument must be a string or a number, not 'NoneType'": 0.7.2

* Fixed bug affecting chains containing duplicate coinbase transactions.

[crazy_rabbit]

This is a decidedly newbie question- but how can I safely turn off ABE? I have installed it to install Electrum, but once it's running, how do I turn it off? So as to not have any conflicts?

thanks!

[John Tobey]

This is a decidedly newbie question- but how can I safely turn off ABE? I have installed it to install Electrum, but once it's running, how do I turn it off? So as to not have any conflicts?

thanks!

Ctrl-C it from the command line or use "kill" in Linux.  Don't worry about the stack trace that it prints.  One of Abe's strongest (IMHO) features is its ability to recover automatically from interruption.

[BitAurum]

Hi folks, hello developers (i.e. John Tobey),
I am playing with the thought of implementing this on my server, since I am already running a full 24/7 BTC-node, LAMP and still have some HDD-space left. Also, since the main fallibility of P2P-Networks is the fact that the truth is determined in a democratic process, you can't have enough block explorers with different blockchain-sources out there.

Nevertheless, I have a problem with understanding the license this was published under: Let's say I want to change the look of the site by rewriting a bit of the CSS-sheet and adding a different logo, I basically have to publish the new sources, even for small derivations, right? But I am allowed to do so, even commercially (aka. linking the logo to my site, which is adversing and therefore commercial...)

Second question: Is anyone running this on MySQL? How big is the database when finished, and is there a somewhat linear relationship to the size of the blockchain (one might assume so...)? Had it loading for a few hours now and got to block 143000, it's around 2.9 GB so far.

Thanks for the code and all the hard work!

edit:
Just some arbitrary data points regarding database size (MySQL) and block number...
Block 143000     2.9 GB
Block 152000     3.7 GB
Block 170000     5.3 GB
Block 184200     8.2 GB (this is where the heavy lifting starts...)
Block 200000   14.9 GB
Block 216276   22.9 GB (full chain January 12th 2013)

Got bored and built LAMP on my gentoo home box. Had the database done within about 18 hours (some delay due to full hdd, note to future self: default Gentoo my.cnf has binary logging enabled) Speed benefits from giving innodb some 6GB ram. Should have done some multithreading, but most of the load still comes from read/write-operations, even with excessive buffering. So one 3.3 GHz Phenom II core might have gotten slightly warmer. Good news is: I will maybe be able to host an instance on my VPS for a few weeks before the size of the blockchain and the database eat my 40 GB HDD. Yay, I guess

[John Tobey]

Hi folks, hello developers (i.e. John Tobey),

Hi!

I am playing with the thought of implementing this on my server, since I am already running a full 24/7 BTC-node, LAMP and still have some HDD-space left. Also, since the main fallibility of P2P-Networks is the fact that the truth is determined in a democratic process, you can't have enough block explorers with different blockchain-sources out there.

Nevertheless, I have a problem with understanding the license this was published under: Let's say I want to change the look of the site by rewriting a bit of the CSS-sheet and adding a different logo, I basically have to publish the new sources, even for small derivations, right? But I am allowed to do so, even commercially (aka. linking the logo to my site, which is adversing and therefore commercial...)

Right.  One easy way to respect the license is to get an account on GitHub (or another public Git service like Gitorious) and maintain your version there.  They give you plenty of handy features.  If it is just a logo and CSS changes (branding as opposed to functionality) I don't care.  If this becomes a sticking point, let me know and I will try to fix it by adding exceptions to the license.

Second question: Is anyone running this on MySQL? How big is the database when finished, and is there a somewhat linear relationship to the size of the blockchain (one might assume so...)? Had it loading for a few hours now and got to block 143000, it's around 2.9 GB so far.

I think it is roughly 4-5 times the sum of the block file sizes.  You can check which block file it is on and how far along it is by reading the datadir table as it loads.  blkfile_number=1 means it has not finished reading blk0001.dat.  blkfile_offset is how far into the current file it has processed.  If blkfile_number is greater than 1, add the sizes of the lower blkNNNN.dat files on disk.

Code:

mysql> select * from datadir;
+------------+------------------------+----------------+----------------+----------+
| datadir_id | dirname                | blkfile_number | blkfile_offset | chain_id |
+------------+------------------------+----------------+----------------+----------+
|          1 | /home/bitcoin/.bitcoin |              1 |      699231568 |     NULL |
+------------+------------------------+----------------+----------------+----------+

Thanks for the code and all the hard work!

You're welcome and good luck!

[molecular]

some charts I produced with bitcoin-abe.

https://bitcointalk.org/index.php?topic=135982.msg1450346#msg1450346

Thanks so much again, John Tobey, for such an awesome product.

[Jouke]

Is there an easy way to get all unspent transactions out of a list of bitcoinaddresses?

And is there maybe a server on wich I can test stuff on?

[Jouke]

Nevermind, I am running my own server atm.

Code:

Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 85, in run
    self.result = application(self.environ, self.start_response)
  File "abe.py", line 205, in __call__
    handler(page)
  File "abe.py", line 909, in handle_rawtx
    abe.do_raw(page, abe.do_rawtx(page))
  File "abe.py", line 921, in do_rawtx
    return json.dumps(tx, sort_keys=True, indent=2)
  File "/usr/lib/python2.7/json/__init__.py", line 238, in dumps
    **kw).encode(obj)
  File "/usr/lib/python2.7/json/encoder.py", line 203, in encode
    chunks = list(chunks)
  File "/usr/lib/python2.7/json/encoder.py", line 428, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/lib/python2.7/json/encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 326, in _iterencode_list
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 402, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib/python2.7/json/encoder.py", line 436, in _iterencode
    o = _default(o)
  File "/usr/lib/python2.7/json/encoder.py", line 178, in default
    raise TypeError(repr(o) + " is not JSON serializable")
TypeError: Decimal('4294967295') is not JSON serializable
<ip> - - [26/Jan/2013 20:15:37] "GET /rawtx/38b46ec5330c445d0e6451d305a08e0fd91a660510ba5aa3ed3c9b1f43356905 HTTP/1.1" 500 59

And I am willing to put a bounty of 3btc for this:

Support for unspent:
============
http://blockchain.info/unspent?address=3JEFRC6tuXRjxG6YUDtYzEmYEphf5pb86C
(something in the direction on listunspent in https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list)

[John Tobey]

Code:

TypeError: Decimal('4294967295') is not JSON serializable
<ip> - - [26/Jan/2013 20:15:37] "GET /rawtx/38b46ec5330c445d0e6451d305a08e0fd91a660510ba5aa3ed3c9b1f43356905 HTTP/1.1" 500 59

Fixed, thanks.

And I am willing to put a bounty of 3btc for this:

Support for unspent:
============
http://blockchain.info/unspent?address=3JEFRC6tuXRjxG6YUDtYzEmYEphf5pb86C
(something in the direction on listunspent in https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list)

This is done, but with caveats: I do not have a full database to test with, it may require optimization, the output fields are slightly different, and it won't work with your example address because Abe does not yet parse script-to-hash transactions.

/unspent/ADDRESS1|ADDRESS2|...
http://john-edwin-tobey.org:2752/unspent/1CBXL5tBSqJZyyCTWVkrLiQsKm4KULBfPA|1CfoGajDKynwB1JpaWq2tML4DDsP8VnuPy

[Jouke]

Code:

TypeError: Decimal('4294967295') is not JSON serializable
<ip> - - [26/Jan/2013 20:15:37] "GET /rawtx/38b46ec5330c445d0e6451d305a08e0fd91a660510ba5aa3ed3c9b1f43356905 HTTP/1.1" 500 59

Fixed, thanks.

And I am willing to put a bounty of 3btc for this:

Support for unspent:
============
http://blockchain.info/unspent?address=3JEFRC6tuXRjxG6YUDtYzEmYEphf5pb86C
(something in the direction on listunspent in https://en.bitcoin.it/wiki/Original_Bitcoin_client/API_calls_list)

This is done, but with caveats: I do not have a full database to test with, it may require optimization, the output fields are slightly different, and it won't work with your example address because Abe does not yet parse script-to-hash transactions.

/unspent/ADDRESS1|ADDRESS2|...
http://john-edwin-tobey.org:2752/unspent/1CBXL5tBSqJZyyCTWVkrLiQsKm4KULBfPA|1CfoGajDKynwB1JpaWq2tML4DDsP8VnuPy

Thanks!

[Jouke]

Because I was impatient, I already had my own unspent transactions working by interpreting the pages already provided by bitcoin abe.

I would get the all the adresses (/bitcoin-abe/address/<address>). If it had any balance I would make a list of the transactions, sorting input and output transactions. I would then fetch all the input transactions to see if the transaction was already used in an output transaction and would then subtract that transaction from the output transaction list. The remaining transactions are the unspent transactions.

This way is a lot faster then your unspent api call, maybe you could one day look in to it.

(and I hope you still use your donation address, because that is where I sent the coins to)

[John Tobey]

@Jouke,

Thanks for the generous tip and the report of slow performance.  It should be simple to change /unspent to match the speed of repeated /address calls.  Please feel free to remind me if I haven't got to it in a week.

Yes, 1PWC7PNHL1SgvZaN7xEtygenKjWobWsCuf is still under my control.

[John Tobey]

I've reimplemented /unspent so it should outperform /address.  A couple of things to note.  The denial-of-service limit on history rows applies to the total across all addresses given to /unspent, so in some cases, /unspent will return "too many records" when iterated /address calls would work.  This limit applies to both spent and unspent coins, unlike in the previous implementation, so you may see "too many records" where my first /unspent worked.  In abe.conf, address-history-rows-max controls this limit.  For now, I've left the first implementation available by adding "?impl=1" to the URL.  It should yield the same results apart from "too many records" errors.

Let me know if this helps.

[Jouke]

I've reimplemented /unspent so it should outperform /address.  A couple of things to note.  The denial-of-service limit on history rows applies to the total across all addresses given to /unspent, so in some cases, /unspent will return "too many records" when iterated /address calls would work.  This limit applies to both spent and unspent coins, unlike in the previous implementation, so you may see "too many records" where my first /unspent worked.  In abe.conf, address-history-rows-max controls this limit.  For now, I've left the first implementation available by adding "?impl=1" to the URL.  It should yield the same results apart from "too many records" errors.

Let me know if this helps.

Excellent! I'll look into it this week.

Question regarding this: https://bitcointalk.org/index.php?topic=22785.msg741984#msg741984
An Abe instance that is started with --no-serve automatically keeps the blockchain up to date?


And what does /q/totalbc show? Bitcoins created, or bitcoins included in coinbase transaction? Because
Local Abe: 10774286.89978955
Blockexplorer: 10736300.00000000
Blockchain.info: 10761275.00000000

Not that it really matters, I don't use it, but just wondering

[John Tobey]

Question regarding this: https://bitcointalk.org/index.php?topic=22785.msg741984#msg741984
An Abe instance that is started with --no-serve automatically keeps the blockchain up to date?

A loop that runs it with --no-serve would keep it up to date.  It brings it up to date and then exits.

And what does /q/totalbc show? Bitcoins created, or bitcoins included in coinbase transaction? Because
Local Abe: 10774286.89978955
Blockexplorer: 10736300.00000000
Blockchain.info: 10761275.00000000

Not that it really matters, I don't use it, but just wondering

It gives a statistical estimate of amount in circulation.  I am curious which block number you got the Abe value from?  You can add the number to the URL like /q/totalbc/220962 to be sure.

The number is normally 50*210000 + 25*(height-209999) but is lower due to the following rare events:

• coinbase transactions outputting less than they are entitled to,
• coins sent to known dead addresses (Abe recognizes the hash consisting of all 0 bits),
• duplicate coinbase transactions (I don't think Abe accounts for them, but the others may)

and there may be bugs.

[Jouke]

Question regarding this: https://bitcointalk.org/index.php?topic=22785.msg741984#msg741984
An Abe instance that is started with --no-serve automatically keeps the blockchain up to date?

A loop that runs it with --no-serve would keep it up to date.  It brings it up to date and then exits.

And what does /q/totalbc show? Bitcoins created, or bitcoins included in coinbase transaction? Because
Local Abe: 10774286.89978955
Blockexplorer: 10736300.00000000
Blockchain.info: 10761275.00000000

Not that it really matters, I don't use it, but just wondering

It gives a statistical estimate of amount in circulation.  I am curious which block number you got the Abe value from?  You can add the number to the URL like /q/totalbc/220962 to be sure.

The number is normally 50*210000 + 25*(height-209999) but is lower due to the following rare events:

• coinbase transactions outputting less than they are entitled to,
• coins sent to known dead addresses (Abe recognizes the hash consisting of all 0 bits),
• duplicate coinbase transactions (I don't think Abe accounts for them, but the others may)

and there may be bugs.

I was expecting 50*210000 + 25*(height-209999) as well, that was why I was wondering if it added fees or something.

chain/Bitcoin/q/totalbc/100000 : 5000049.98
chain/Bitcoin/q/totalbc/200000 : 10000037.06486171
chain/Bitcoin/q/totalbc/220998 : 10774961.89978955