[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[molecular]

John,

have you implemented q/getreceivedbyaddress and q/getsentbyaddress yet?

I have:

    import decimal.Decimal

    def q_getreceivedbyaddress(abe, page, chain):
        """getreceivedbyaddress"""
        if chain is None:
            return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \
                '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS*\n'
        addr = wsgiref.util.shift_path_info(page['env'])

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            select
              ifnull(sum(txout.txout_value),0) from pubkey
            join txout on txout.pubkey_id=pubkey.pubkey_id
            join block_tx on block_tx.tx_id=txout.tx_id
            join block b on b.block_id=block_tx.block_id
            join chain_candidate cc on cc.block_id=b.block_id
            where
              pubkey_hash='%s' and
              cc.chain_id=%s and
              cc.in_longest=1
            ;""" % (binascii.hexlify(hash), chain['id'])
          #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging
          rows = abe.store.selectall(sql)
          ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string()
        else:
          ret = 'address invalid'

        return ret
       
    def q_getsentbyaddress(abe, page, chain):
        """getsentbyaddress"""
        if chain is None:
            return 'returns amount of money sent from given address\n' \
                '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'
        addr = wsgiref.util.shift_path_info(page['env'])

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            select
              ifnull(sum(txout.txout_value),0) from pubkey
            join txout txout on txout.pubkey_id=pubkey.pubkey_id
            join txin on txin.txout_id=txout.txout_id
            join block_tx on block_tx.tx_id=txout.tx_id
            join block b on b.block_id=block_tx.block_id
            join chain_candidate cc on cc.block_id=b.block_id
            where
              pubkey_hash='%s' and
              cc.chain_id=%s and
              cc.in_longest=1
            ;""" % (binascii.hexlify(hash), chain['id'])
          #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging
          rows = abe.store.selectall(sql)
          ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string()
        else:
          ret = 'address invalid'

        return ret

If you want to use it, please take a look at the following things especially:

• SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
• Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"

I didn't implement the "confirmations" filter yet.

[1713495556]

1713495556

[1713495556]

1713495556

[newminerr]

chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263

Though i am not running bitcoin at the moment.

[John Tobey]

chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263

Though i am not running bitcoin at the moment.

Hmm, running will not help with this error.  Have you redownloaded the block chain since the last abe.py run?  If so,

Code:

UPDATE datadir SET blkfile_number = 1, blkfile_offset = 0

will force a rescan.

If not, well, now would be a good time to implement scanning for the magic number sequence, but that is not trivial.  Maybe skipping past '\x00' would solve it here.  I am curious what this program prints if you run it in the bitcoin-abe directory:

Code:

#!/usr/bin/env python
import BCDataStream
ds = BCDataStream.BCDataStream()
ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"),
            169336263)
offset = ds.read_cursor
magic = ds.read_bytes(4)
print "at %d of %d: %s" % (offset, len(ds.input), repr(magic))
if magic[0] == chr(0):
    ds.read_cursor = offset
    while ds.read_cursor < len(ds.input):
        b = ds.read_bytes(1)[0]
        if b != chr(0):
            ds.read_cursor -= 1
            break
    if ds.read_cursor + 4 <= len(ds.input):
        magic = ds.read_bytes(4)
        print "at %d: %s" % (ds.read_cursor, repr(magic))
    else:
        print "%d near end %d" % (ds.read_cursor, len(ds.input))

[John Tobey]

• SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
• Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"

I didn't implement the "confirmations" filter yet.

The joining looks correct.  My "way" to format decimals is

Code:

format_satoshis(satoshis, chain)

.

Couple of requests.  Instead of IFNULL, use the portable COALESCE with the same arguments.  Instead of '%s' and the % operator, use an unquoted question mark (?) and pass a tuple of bind values as the second argument to selectrow.  (selectrow is the same as selectall but returns just the first row.)  (Abe's database abstraction layer converts the standard qmark parameter style to whatever the driver needs.)  Please capitalize SQL keywords like SELECT FROM JOIN ON WHERE AND for easier readability.  Please prefix error messages with "ERROR:" as per BBE convention.  And please show the usage message if address is None, even if chain is given.  This is my convention where address is required, as in q_translate_address.

Thanks!

[newminerr]

Code:

#!/usr/bin/env python
import BCDataStream
ds = BCDataStream.BCDataStream()
ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"),
            169336263)
offset = ds.read_cursor
magic = ds.read_bytes(4)
print "at %d of %d: %s" % (offset, len(ds.input), repr(magic))
if magic[0] == chr(0):
    ds.read_cursor = offset
    while ds.read_cursor < len(ds.input):
        b = ds.read_bytes(1)[0]
        if b != chr(0):
            ds.read_cursor -= 1
            break
    if ds.read_cursor + 4 <= len(ds.input):
        magic = ds.read_bytes(4)
        print "at %d: %s" % (ds.read_cursor, repr(magic))
    else:
        print "%d near end %d" % (ds.read_cursor, len(ds.input))

It returned this.

Code:

C:\abe>python ab.py
at 169336263 of 698256440: '\x00\x00\x00\x00'
at 169394422: '\xf9\xbe\xb4\xd9'

[molecular]

The joining looks correct.  My "way" to format decimals is

Code:

format_satoshis(satoshis, chain)

.

I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)

Code:

format_satoshis(decimal.Decimal('12.3456'), 1)

gives

Code:

1.23456E-7.012.3456

other than that, I made alle the changes you requested:

Code:

    def q_getreceivedbyaddress(abe, page, chain):
        """getreceivedbyaddress"""
        addr = wsgiref.util.shift_path_info(page['env'])
        if chain is None or addr is None:
            return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \
                '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS\n'

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            SELECT 
              CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey 
            JOIN txout on txout.pubkey_id=pubkey.pubkey_id 
            JOIN block_tx on block_tx.tx_id=txout.tx_id 
            JOIN block b on b.block_id=block_tx.block_id 
            JOIN chain_candidate cc on cc.block_id=b.block_id 
            WHERE 
              pubkey_hash=? AND
              cc.chain_id=? AND
              cc.in_longest=1
            ;"""
          row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id']))
          ret = format_satoshis(row[0], chain);
        else:
          ret = 'ERROR: address invalid'

        return ret
        
    def q_getsentbyaddress(abe, page, chain):
        """getsentbyaddress"""
        addr = wsgiref.util.shift_path_info(page['env'])
        if chain is None or addr is None:
            return 'returns amount of money sent from given address\n' \
                '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            SELECT 
              CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey 
            JOIN txout txout on txout.pubkey_id=pubkey.pubkey_id 
            JOIN txin on txin.txout_id=txout.txout_id
            JOIN block_tx on block_tx.tx_id=txout.tx_id 
            JOIN block b on b.block_id=block_tx.block_id 
            JOIN chain_candidate cc on cc.block_id=b.block_id 
            WHERE 
              pubkey_hash=? AND
              cc.chain_id=? AND
              cc.in_longest=1
            ;"""
          row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id']))
          ret = format_satoshis(row[0], chain);
        else:
          ret = 'ERROR: address invalid'

        return ret

Anything else?

EDIT: I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)

[John Tobey]

It returned this.

Code:

C:\abe>python ab.py
at 169336263 of 698256440: '\x00\x00\x00\x00'
at 169394422: '\xf9\xbe\xb4\xd9'

Thanks.  I've committed a fix that I think will get you past this error.

I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)

The problem was that format_satoshis expected the caller to cast its first argument to int.  I decided it would be better for the function to do the casting.  (txout_value is in 1e-8 BTC, so all values are integer.)

I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)

I like little variables like that, they help document the intent and aid future tinkerers.

Anything else?

Just some minor reformatting which I did.  Thanks!

[molecular]

Just some minor reformatting which I did.  Thanks!

You're welcome. Love to be able to give back a little code, too.

[John Tobey]

Thinking about also hosting a copycat of this... posting here to remind me to check up on it later. 

If you do, it could be abe.john-edwin-tobey.org, since the hosting arrangement there is ending.  I'm switching DNS back to my pathetic excuse for a server, but if it can't take the load, I'll punt to whoever wants to run it.

Of course, anyone with serious uptime requirements is welcome to run a private or public instance.

[John Tobey]

Version 0.6 highlights:

• Python packaging; abe.py moved; run as "python -m Abe.abe".
• Big speed improvements (c. 10x) for MySQL and SQLite.
• ODBC tested successfully.
• IBM DB2 tested successfully.
• HTTP API functions: getreceivedbyaddress getsentbyaddress.
• Verify transaction Merkle roots on block import.
• Show Namecoin-style network fees and name transaction outputs.
• Adjust coins outstanding and coin-days destroyed for Namecoin-style network fees.
• Native SolidCoin support.
• Suppress display of empty chains on home page.
• Show the search form on /chain/CHAIN pages.
• Many minor improvements; see the Git log.

Next up are a couple of bug fixes and the test suite.  When the test suite passes, it is 1.0.

[molecular]

Cool, a release!

• Native SolidCoin support.

What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?

[John Tobey]

• Native SolidCoin support.

What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?

Yup, just that.  No need to read SC source code to figure out magic number and address version byte, no need to touch Python code.  Just pass "--datadir ~/.solidcoin".

Btw, I've cut the stuff down a bit:

Code:

    {"chain":"SolidCoin",
     "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},

[molecular]

Yup, just that.  No need to read SC source code to figure out magic number and address version byte, no need to touch Python code.  Just pass "--datadir ~/.solidcoin".

Btw, I've cut the stuff down a bit:

Code:

    {"chain":"SolidCoin",
     "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},

Nice one!

Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.

Btw: should be possible to add caching headers and gzip compression, right?

[John Tobey]

Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.

Btw: should be possible to add caching headers and gzip compression, right?

Provided it does not complicate the core stuff.  Are you thinking of working on this?  You could do it at a high level of abstraction ("squid") or a very low level (store compressed pages in db, add "last modified" columns to tx, pubkey, and block records) or in between, perhaps some WSGI middleware.  If you are going for low level, some refactoring might be in order beforehand.

To get cache headers on static content, I simply use FastCGI and configure static-path to something Apache can serve directly as described in README-FASTCGI.txt.

[newminerr]

BTW it's working great now, keep up the good work.

[John Tobey]

BTW it's working great now, keep up the good work.

Great.  Let me know if you try the Windows installer how it goes: http://pypi.python.org/pypi/Abe/0.6

PGP key: https://raw.github.com/jtobey/bitcoin-abe/master/doc/jtobey.pubkey
Key fingerprint: 413D 2920 B776 08D1 A867  232A 3CB0 B139 6A71 5348

[John Tobey]

I've taken down abe.john-edwin-tobey.org due to heavy load.  Email me or post here if you have server space and want help setting up!

[molecular]

I've taken down abe.john-edwin-tobey.org due to heavy load.  Email me or post here if you have server space and want help setting up!

Just rented a VPS for solidcoin.kicks-ass.org. Using bitcoin-abe there already for solidcoin. Want to add bitcoin anyways.

I'm not sure if that thing (it's a virtual, of course) would be up to the task.

Could you provide some traffic stats from prior to taking your server down so we can have a rough estimate?

[John Tobey]

I've taken down abe.john-edwin-tobey.org due to heavy load.  Email me or post here if you have server space and want help setting up!

Backpedaling... log showed 90% of requests from Googlebot.  I'm attempting to solve this with robots.txt.  Site is up for now.

Just rented a VPS for solidcoin.kicks-ass.org. Using bitcoin-abe there already for solidcoin. Want to add bitcoin anyways.

I'm not sure if that thing (it's a virtual, of course) would be up to the task.

Could you provide some traffic stats from prior to taking your server down so we can have a rough estimate?

It wasn't getting a lot of hits, roughly 4 address or tx requests per minute.  The VPS's swap space was chronically all used, though, causing the kernel to kill processes roughly once per hour, despite my cron job that bounced Apache and bitcoind every hour.  So it was thrashing, and Postgres must have had to read from disk for most queries.

There's certainly room for optimization in the schema, especially if you have plenty of disk, which I don't.  But before I go that route, I'd like to know what's possible with a server with 1GB+ RAM.

By the way, if you want to hide the "empty" currencies from the homepage, you could delete the respective rows from the chain table or upgrade to the latest 0.7pre.

If you set up an instance with BTC, at least one alt chain, and better uptime than mine, and if you are willing, I will advertise it as the demo site.

[molecular]

There's certainly room for optimization in the schema, especially if you have plenty of disk, which I don't.  But before I go that route, I'd like to know what's possible with a server with 1GB+ RAM.

Well, it seems my server wouldn't cut it. Only 512MB memory. No swapspace (!!).

I'll probably setup bitcoin chain on it anyways, because I want to make my coin-age graphs live, so then I could do some manual benchmarking to see if it could make sense at all.