[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[Atheros]

I have a feature request. I think it is simple but important.

(I'm running the newest version of Abe on Windows with MySQL. I'm using the Abe web server.)

While random clicking and exploring, I clicked the address 1VayNert3x1KzbpzMGt2qdqrAThiRovi8 which is used by some kind of bot for transactions and it has hundreds of thousands of transactions. It took the SQL server many minutes to process the query and then Abe spent half an hour processing and sending the data to the browser. During this time, Abe would not respond to other requests and the CPU usage on the server was at 100% which killed other things. I didn't let it finish; I gave up and killed it. On your installation and someone else's I tried, the server spits out an internal server error after 45 seconds or so. Neither of these behaviors seems acceptable.

I would like to tell people I am running Abe and let them use it but I can't tell people about it right now because if they click certain addresses, it effectively kills Abe and my server's other tasks. Could Abe do one of the following:

* Only list the first/last couple (thousand?) transactions and then say that the rest are truncated. Perhaps have a next button to go to the next page.
* Say that there are too many records and that the page cannot be displayed. Blockexplorer.com does this.

This is the only problem I've come across. Thank you for all of your hard work. You have made a great piece of software!

EDIT: I started digging through source code. Doesn't it seem like the 'limit' in this query should be limiting the output? Or am I looking in the wrong place?

            ret += filter(None, map(process, abe.store.selectall(
                "SELECT pubkey_hash FROM pubkey WHERE pubkey_hash" +
                # XXX hardcoded limit.
                neg + " BETWEEN ? AND ? LIMIT 100", (bl, bh))))

[1714107442]

1714107442

[1714107442]

1714107442

[1714107442]

1714107442

[1714107442]

1714107442

[1714107442]

1714107442

[John Tobey]

@Atheros

Good question.  This patch is not the most polished piece of code, but you may try it, editing ADDRESS_HISTORY_ROWS_MAX in Abe/abe.py to your liking:

Code:

diff --git a/Abe/abe.py b/Abe/abe.py
index 1982434..4811e01 100755
--- a/Abe/abe.py
+++ b/Abe/abe.py
@@ -92,6 +92,8 @@ HEIGHT_RE = re.compile('(?:0|[1-9][0-9]*)\\Z')
 HASH_PREFIX_RE = re.compile('[0-9a-fA-F]{0,64}\\Z')
 HASH_PREFIX_MIN = 6
 
+ADDRESS_HISTORY_ROWS_MAX = 500  # XXX hardcoded limit
+
 NETHASH_HEADER = """\
 blockNumber:          height of last block in interval + 1
 time:                 block time in seconds since 0h00 1 Jan 1970 UTC
@@ -930,12 +932,13 @@ class Abe:
             count[txpoint['is_in']] += 1
 
         txpoints = []
-        rows = []
-        rows += abe.store.selectall("""
+        max_rows = ADDRESS_HISTORY_ROWS_MAX
+        in_rows = abe.store.selectall("""
             SELECT
                 b.block_nTime,
                 cc.chain_id,
                 b.block_height,
+                block_tx.tx_pos,
                 1,
                 b.block_hash,
                 tx.tx_hash,
@@ -949,13 +952,22 @@ class Abe:
               JOIN txout prevout ON (txin.txout_id = prevout.txout_id)
               JOIN pubkey ON (pubkey.pubkey_id = prevout.pubkey_id)
              WHERE pubkey.pubkey_hash = ?
-               AND cc.in_longest = 1""",
-                      (dbhash,))
-        rows += abe.store.selectall("""
+               AND cc.in_longest = 1
+             ORDER BY
+                   b.block_nTime,
+                   cc.chain_id,
+                   b.block_height,
+                   block_tx.tx_pos,
+                   txin.txin_pos
+             LIMIT ?""",
+                      (dbhash, max_rows + 1))
+        truncated = in_rows[max_rows] if len(in_rows) > max_rows else None
+        out_rows = abe.store.selectall("""
             SELECT
                 b.block_nTime,
                 cc.chain_id,
                 b.block_height,
+                block_tx.tx_pos,
                 0,
                 b.block_hash,
                 tx.tx_hash,
@@ -968,11 +980,41 @@ class Abe:
               JOIN txout ON (txout.tx_id = tx.tx_id)
               JOIN pubkey ON (pubkey.pubkey_id = txout.pubkey_id)
              WHERE pubkey.pubkey_hash = ?
-               AND cc.in_longest = 1""",
-                      (dbhash,))
+               AND cc.in_longest = 1""" + ("""
+               AND (b.block_nTime < ? OR
+                    (b.block_nTime = ? AND
+                     (cc.chain_id < ? OR
+                      (cc.chain_id = ? AND
+                       (b.block_height < ? OR
+                        (b.block_height = ? AND
+                         block_tx.tx_pos <= ?))))))"""
+                                           if truncated else "") + """
+             ORDER BY
+                   b.block_nTime,
+                   cc.chain_id,
+                   b.block_height,
+                   block_tx.tx_pos,
+                   txout.txout_pos
+             LIMIT ?""",
+                      (dbhash,
+                       truncated[0], truncated[0], truncated[1], truncated[1],
+                       truncated[2], truncated[2], truncated[3], max_rows + 1)
+                      if truncated else
+                      (dbhash, max_rows + 1))
+
+        if len(out_rows) > max_rows:
+            if truncated is None or truncated > out_rows[max_rows]:
+                truncated = out_rows[max_rows]
+        if truncated:
+            truncated = truncated[0:3]
+
+        rows = in_rows + out_rows
         rows.sort()
         for row in rows:
-            nTime, chain_id, height, is_in, blk_hash, tx_hash, pos, value = row
+            if truncated and row >= truncated:
+                break
+            (nTime, chain_id, height, tx_pos,
+             is_in, blk_hash, tx_hash, pos, value) = row
             txpoint = {
                     "nTime":    int(nTime),
                     "chain_id": int(chain_id),
@@ -1008,7 +1050,11 @@ class Abe:
 
         body += abe.short_link(page, 'a/' + address[:10])
 
-        body += ['<p>Balance: '] + format_amounts(balance, True)
+        body += ['<p>']
+        if truncated:
+            body += ['<strong>Results truncated</strong></p>']
+        else:
+            body += ['<p>Balance: '] + format_amounts(balance, True)
 
         for chain_id in chain_ids:
             balance[chain_id] = 0  # Reset for history traversal.

The maximum applies to sends and receives independently, so for example, an address with 600 receives and 100 spends will show the first 500 receives and whichever of the spends occurred before the 501st receive.  I replace the balance with "Results truncated", since it is probably inaccurate, and it may never have been the right balance (may even be negative) due to out-of-order blocks and transactions.

Switching to FastCGI should give you a bit better parallelism, too, though FCGI mode may need fixes for Windows.  You could try Python's multithreaded socket server with the built-in HTTP but I won't be surprised if you hit bugs.

Let us know how it goes.

[Atheros]

Thank you for your reply.
There was one bug in the code: It fails out on line 1012, in handle_address
rows.sort()
AttributeError: 'tuble' object has no attribute 'sort'

I couldn't quite figure out what to do to fix it so I took out the sort line. This obviously screwed up the way transactions for an address are displayed but the program didn't crash. Upon viewing the 1VayNert address, my SQL server has been working as fast as it can for the last ten minutes in a state of "Copying to tmp table". It appears to me that while Abe might do a better job of limiting the amount of information it tries to send to the browser thanks to this patch, the SQL server is still trying to pull it all.

I would like to get Abe working with FastCGI on my Windows machine but I'll have to wait until someone with more experience writes some documentation.

[John Tobey]

Atheros:

Thanks for the report.  I fixed the sort() bug, made the limit configurable and committed the change to github as 945ed46829.

I have only 63938 blocks loaded in my MySQL database, and the longest address history (1XPTgDRhN8...) has 3207 rows, apparently too few for a good test.  That address loaded much faster with a low limit than with none.  However, MySQL "explain" showed it still having to scan and sort the whole history, so as you observed, it did not solve the problem.  Adding an index on block.block_nTime did not help, so I went BlockExplorer's route and now display no history if the limit is exceeded.  To show partial history efficiently might require significant table and index overhead.

Let me know if the latest commit does not solve the load problem.

[Atheros]

Before I went to upgrade, I found that Abe had stopped working. My block file just exceeded 1GB in size a couple hours ago which is causing this error:

Failed to catch up {'blkfile_number': 1, 'dirname': 'C:\\Documents and Settings\
\Jon\\Application Data\\Bitcoin', 'chain_id': None, 'id': 1L, 'blkfile_offset':
0}
Traceback (most recent call last):
  File "Abe\DataStore.py", line 2169, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe\DataStore.py", line 2184, in catch_up_dir
    ds = open_blkfile()
  File "Abe\DataStore.py", line 2180, in open_blkfile
    ds.map_file(open(filename, "rb"), 0)
  File "Abe\BCDataStream.py", line 27, in map_file
    self.input = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ)
WindowsError: [Error 8] Not enough storage is available to process this command

It is because I'm running a 32-bit system. Apparently this issue will eventually affect Linux 32-bit systems as well because of the way mmap works; except that it will occur when the block file hits 2 or 3 GB in size. Is Abe particularly tied to mmap? If it is, it seems like it will soon be (or is now) for 64 bit systems only.

[molecular]

fyi: had this
  https://bitcointalk.org/index.php?topic=22785.msg716310#msg716310
problem again, which is explained here
  https://bitcointalk.org/index.php?topic=22785.msg718444#msg718444
and fixed it using this
  https://bitcointalk.org/index.php?topic=22785.msg718481#msg718481
solution.

[John Tobey]

fyi: had this
  https://bitcointalk.org/index.php?topic=22785.msg716310#msg716310
problem again, which is explained here
  https://bitcointalk.org/index.php?topic=22785.msg718444#msg718444
and fixed it using this
  https://bitcointalk.org/index.php?topic=22785.msg718481#msg718481
solution.

Strange.  I'll try to debug it if you give me steps to reproduce it.  (Steps could include "git clone" and building Litecoin.)

[John Tobey]

Before I went to upgrade, I found that Abe had stopped working. My block file just exceeded 1GB in size a couple hours ago which is causing this error:

Failed to catch up {'blkfile_number': 1, 'dirname': 'C:\\Documents and Settings\
\Jon\\Application Data\\Bitcoin', 'chain_id': None, 'id': 1L, 'blkfile_offset':
0}
Traceback (most recent call last):
  File "Abe\DataStore.py", line 2169, in catch_up
    store.catch_up_dir(dircfg)
  File "Abe\DataStore.py", line 2184, in catch_up_dir
    ds = open_blkfile()
  File "Abe\DataStore.py", line 2180, in open_blkfile
    ds.map_file(open(filename, "rb"), 0)
  File "Abe\BCDataStream.py", line 27, in map_file
    self.input = mmap.mmap(file.fileno(), 0, access=mmap.ACCESS_READ)
WindowsError: [Error 8] Not enough storage is available to process this command

It is because I'm running a 32-bit system. Apparently this issue will eventually affect Linux 32-bit systems as well because of the way mmap works; except that it will occur when the block file hits 2 or 3 GB in size. Is Abe particularly tied to mmap? If it is, it seems like it will soon be (or is now) for 64 bit systems only.

Interesting.  I don't think it will affect Linux, because bitcoin starts a new file (blk0002.dat) before 2GB:

Code:

        // FAT32 filesize max 4GB, fseek and ftell max 2GB, so we must stay under 2GB
        if (ftell(file) < 0x7F000000 - MAX_SIZE)
        {
            nFileRet = nCurrentBlockFile;
            return file;
        }

No mention of mmap, but I think a change from 0x7F000000 to 0x3F000000 is worth proposing, or at least making it a compile-time option:

Code:

diff --git a/src/main.cpp b/src/main.cpp
index 9951952..2d838b7 100644
--- a/src/main.cpp
+++ b/src/main.cpp
@@ -1773,6 +1773,10 @@ FILE* OpenBlockFile(unsigned int nFile, unsigned int nBlockPos, const char* pszM
 
 static unsigned int nCurrentBlockFile = 1;
 
+#ifndef BITCOIN_MAX_BLKFILE_SIZE
+#define BITCOIN_MAX_BLKFILE_SIZE 0x7F000000
+#endif
+
 FILE* AppendBlockFile(unsigned int& nFileRet)
 {
     nFileRet = 0;
@@ -1784,7 +1788,7 @@ FILE* AppendBlockFile(unsigned int& nFileRet)
         if (fseek(file, 0, SEEK_END) != 0)
             return NULL;
         // FAT32 filesize max 4GB, fseek and ftell max 2GB, so we must stay under 2GB
-        if (ftell(file) < 0x7F000000 - MAX_SIZE)
+        if (ftell(file) < BITCOIN_MAX_BLKFILE_SIZE - MAX_SIZE)
         {
             nFileRet = nCurrentBlockFile;
             return file;

Abe inherits the use of mmap() from Bitcointools, so if you can verify the bug exists there, we might get Gavin interested in fixing it.

[tiker]

I have a new question / problem for you.

I had to setup a new box to run alternate currencies because the existing box is getting too heavily loaded.

New box setup with same OS, loaded on Abe and a new alternate currency MMMCoin

Importing this new chain to the same MySQL db running on a 3rd box looks like it runs fine from the prompt but when looking at it from the web interface, it shows with 0 blocks.  Clicking on block 0 shows a block but no links to the rest...

Is this a problem with the the new chain or something with the new box I just setup?

Abe site - http://blockexplorer.funkymonkey.org

[John Tobey]

I had to setup a new box to run alternate currencies because the existing box is getting too heavily loaded.

Thanks for providing this!

Importing this new chain to the same MySQL db running on a 3rd box looks like it runs fine from the prompt but when looking at it from the web interface, it shows with 0 blocks.  Clicking on block 0 shows a block but no links to the rest...

By "runs fine" do you mean it logs a lot of block_tx records in the new chain?  Are you comfortable issuing SQL queries?  I'd like to know what is in the database.

If you have a block hash that should have been loaded, try finding its block_id.  You can do this with the partial hash shown in ~/.bitcoin/debug.log (or MMMCoin equivalent) as "best=...".  For example, if you see best=000000000661131faf72 and then bring Abe up to date, try:

Code:

SELECT block_id FROM block WHERE block_hash LIKE '000000000661131faf72%';

If this gives a block_id, for example 12165, see whether the block_id is attached to the right chain:

Code:

SELECT cc.*, c.chain_name FROM chain_candidate cc JOIN chain c USING (chain_id) WHERE block_id = 12165;

I may need the first few blocks (or say 10kB) of MMM's blk0001.dat to figure this out.  I don't know anything about the new chain, and it may do something unusual that would prevent Abe from loading it.

[tiker]

By runs fine I mean:

Code:

block_tx 1919268 4979880
commit
block_tx 1919269 4979881
commit
block_tx 1919270 4979882
commit
block_tx 1919271 4979883
commit

I didn't see any errors.  First line was about new chain and setting new chain id.

I hit CTRL C after about 100 blocks or so when I noticed the website wasn't updating properly.

I'll see what I can find for the hash block and will reply.  Give me a few.

Thanks.

[tiker]

The first occurance of the best= line (when I first launched the new client):

Code:

Loading block index...
Genesis block time: 1325369107
Genesis block bits: 1e0fffff
Genesis block nonce: 386206
Genesis block hash: 00000f2e9f8ffa3abc79a4dcbff01dcfd10f480e897ebcd50cb0c37afbe57d51
Genesis merkle root: 7dee73b5c259a6e2eb06482b289af5cb7f24476310fc80081667baf8f2f27c5b
SetBestChain: new best=00000f2e9f8ffa3abc79a4dcbff01dcfd10f480e897ebcd50cb0c37afbe57d51  height=0  work=1048577
 block index              83ms

The last (or current) is:

Code:

Flushed wallet.dat 24ms
askfor block 000003b4ef02c872071895b03082326f807ee202bdfc33619c3b3122620c2188   0
sending getdata: block 000003b4ef02c872071895b03082326f807ee202bdfc33619c3b3122620c2188
askfor block 000003b4ef02c872071895b03082326f807ee202bdfc33619c3b3122620c2188   1331918649000000
received block 000003b4ef02c872071895b03082326f807ee202bdfc33619c3b3122620c2188
CalculateChainWork() : New trusted block at height 93363. Last work: 9233139665732, new work: 253336966, combined: 9233393002698
SetBestChain: new best=000003b4ef02c872071895b03082326f807ee202bdfc33619c3b3122620c2188  height=93363  work=9233393002698
ProcessBlock: ACCEPTED
03/16/12 17:24:12 Flushing wallet.dat
Flushed wallet.dat 23ms

I don't understand the part about "and then bring abe up to date".  I'm assuming it means import everything now so I'm going ahead with that now.

[tiker]

Not sure if this helps or matters but from reading the forums through Google's translator it seems like changes were made recently.  The original client / network (if I read correctly) got hit with a 51% attack.  A new client was released a few days ago with fixes for that.  I don't know if it affects the block chain or not (beyond my knowledge of block chains).

Also, if helpful, the client was compiled from sources pulled from https://github.com/alexhz/MMMCoin_public

[John Tobey]

I don't understand the part about "and then bring abe up to date".  I'm assuming it means import everything now so I'm going ahead with that now.

Yes.  Just to be sure the hash does not refer to a block received since Abe last read the data.

Not sure if this helps or matters but from reading the forums through Google's translator it seems like changes were made recently.  The original client / network (if I read correctly) got hit with a 51% attack.  A new client was released a few days ago with fixes for that.  I don't know if it affects the block chain or not (beyond my knowledge of block chains).

Possible I guess.

Also, if helpful, the client was compiled from sources pulled from https://github.com/alexhz/MMMCoin_public

I don't know if I'll have time to build it, but thanks.

I am curious about Abe's console output.  Have you tried loading the currency to a fresh database with no other datadirs configured?  I'm curious what Abe prints.

[tiker]

I am curious about Abe's console output.  Have you tried loading the currency to a fresh database with no other datadirs configured?  I'm curious what Abe prints.

I haven't tried that yet but I can...  I'll get a second import running to a new db while the existing one continues.

[tiker]

Here's how a new import to a new db goes:

Code:

:~/apps/bitcoin-abe$ python -m Abe.abe --config imports/abe-mmcoin-test.conf
ddl_implicit_commit=true
create_table_epilogue=' ENGINE=InnoDB'
Abe/DataStore.py:418: Warning: Converting column 'a' from VARCHAR to TEXT
  store.cursor.execute(stmt)
Abe/DataStore.py:418: Warning: Converting column 'b' from VARCHAR to TEXT
  store.cursor.execute(stmt)
max_varchar=4294967295
clob_type=LONGTEXT
binary_type=hex
int_type=int
Created silly table abe_dual
sequence_type=mysql
limit_style=native
Assigned chain_id 8 to MMMCoin
block_tx 1 1
commit
block_tx 2 2
commit
block_tx 3 3
commit
block_tx 4 4
commit
block_tx 5 5
commit
block_tx 6 6
commit
block_tx 7 7
commit
block_tx 8 8
commit
block_tx 9 9
commit
block_tx 10 10

Config file:

Code:

:~/apps/bitcoin-abe$ cat imports/abe-mmcoin-test.conf
    dbtype MySQLdb
    connect-args {"host":"removedserver","user":"removedusername","db":"abe-test","passwd":"removedpassword"}
    no-serve
    commit-bytes = 0

datadir = [{
        "dirname":"/home/ed/.mmm",
        "chain":"MMMCoin",
        "code3":"MMM",
        "address_version":"\u0000"}]

And the same result when viewing through the web interface.
http://i147.photobucket.com/albums/r300/lethaltiker/F4D42AF6A0326EB5EB858F2D4510FB78.jpg

[tiker]

Current status:

I had to enable the "ignore-bit8-chains" flag for MMMCoin.  It failed with the same error I had with the bitcoin testnet.

Import resumed.

Code:

commit
block_tx 1982463 5051769
commit
Ignored bit8 in version 0x00000100 of chain_id 22
block_tx 1982464 5051770
block_tx 1982464 5051771
commit
block_tx 1982465 5051772
block_tx 1982465 5051773
commit
Ignored bit8 in version 0x00000100 of chain_id 22
block_tx 1982466 5051774
block_tx 1982466 5051775
commit
block_tx 1982467 5051776
block_tx 1982467 5051777
commit
Ignored bit8 in version 0x00000100 of chain_id 22
block_tx 1982468 5051778
block_tx 1982468 5051779
commit

[tiker]

The import completed.

SQL results:

Code:

SELECT block_id FROM block WHERE block_hash LIKE '000003b4ef02c872071895b0308232%';

gave me empty results...

[John Tobey]

The import completed.

SQL results:

Code:

SELECT block_id FROM block WHERE block_hash LIKE '000003b4ef02c872071895b0308232%';

gave me empty results...

Thanks.  It looks as if MMMCoin uses a scrypt-like hash function.  Abe relies on the block header containing the previous block's Bitcoin hash (double SHA256).  Other scrypt-using chains apparently still use SHA256 for the previous-block pointer, or otherwise Abe couldn't link their blocks together.  If this is right, I actually prefer MMM's design, although it breaks Abe.

To fix it, we will need a Python implementation of MMMCoin's hashing algorithm.  Probably one exists, we just have to find and test it, and I'll think about how to support it compatibly.

[tiker]

Might be a bit more than just a change in the header hashing..

During the import I noticed every 2nd block reported "Ignored bit8 in version 0x00000100 of chain_id 22".  Seemed a bit odd that every 2 blocks had a different header...

Reading the MMMCoin forum today through google's translator it turns out that the block chain links blocks differently. - http://mmmcoin.org/forum/viewtopic.php?f=8&t=84&start=20#p447

Every 2nd block put into the chain comes from the miners connected to the clients and the blocks in between are created by "trusted nodes".  If someone solves a block and it gets put into the chain, no one else can solve another block until one of the trusted nodes puts a block in.

miner block -> trusted node block -> miner block -> trusted node block  ... etc.

Seeing how every 2nd import by Abe reports ignoring bit8 I'd guess that there are 2 different block headers to process depending on the source of the block.