molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
August 26, 2011, 02:46:35 PM |
|
John, have you implemented q/getreceivedbyaddress and q/getsentbyaddress yet? I have: import decimal.Decimal
def q_getreceivedbyaddress(abe, page, chain): """getreceivedbyaddress""" if chain is None: return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \ '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS*\n' addr = wsgiref.util.shift_path_info(page['env'])
if ADDRESS_RE.match(addr): version, hash = decode_address(addr) sql = """ select ifnull(sum(txout.txout_value),0) from pubkey join txout on txout.pubkey_id=pubkey.pubkey_id join block_tx on block_tx.tx_id=txout.tx_id join block b on b.block_id=block_tx.block_id join chain_candidate cc on cc.block_id=b.block_id where pubkey_hash='%s' and cc.chain_id=%s and cc.in_longest=1 ;""" % (binascii.hexlify(hash), chain['id']) #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging rows = abe.store.selectall(sql) ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string() else: ret = 'address invalid'
return ret def q_getsentbyaddress(abe, page, chain): """getsentbyaddress""" if chain is None: return 'returns amount of money sent from given address\n' \ '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n' addr = wsgiref.util.shift_path_info(page['env'])
if ADDRESS_RE.match(addr): version, hash = decode_address(addr) sql = """ select ifnull(sum(txout.txout_value),0) from pubkey join txout txout on txout.pubkey_id=pubkey.pubkey_id join txin on txin.txout_id=txout.txout_id join block_tx on block_tx.tx_id=txout.tx_id join block b on b.block_id=block_tx.block_id join chain_candidate cc on cc.block_id=b.block_id where pubkey_hash='%s' and cc.chain_id=%s and cc.in_longest=1 ;""" % (binascii.hexlify(hash), chain['id']) #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging rows = abe.store.selectall(sql) ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string() else: ret = 'address invalid'
return ret
If you want to use it, please take a look at the following things especially: - SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
- Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"
I didn't implement the "confirmations" filter yet.
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
newminerr
Member
Offline
Activity: 147
Merit: 11
The day to rise has come.
|
|
August 26, 2011, 03:52:58 PM |
|
chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263 Though i am not running bitcoin at the moment.
|
|
|
|
John Tobey (OP)
|
|
August 26, 2011, 05:09:10 PM |
|
chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263 Though i am not running bitcoin at the moment. Hmm, running will not help with this error. Have you redownloaded the block chain since the last abe.py run? If so, UPDATE datadir SET blkfile_number = 1, blkfile_offset = 0 will force a rescan. If not, well, now would be a good time to implement scanning for the magic number sequence, but that is not trivial. Maybe skipping past '\x00' would solve it here. I am curious what this program prints if you run it in the bitcoin-abe directory: #!/usr/bin/env python import BCDataStream ds = BCDataStream.BCDataStream() ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"), 169336263) offset = ds.read_cursor magic = ds.read_bytes(4) print "at %d of %d: %s" % (offset, len(ds.input), repr(magic)) if magic[0] == chr(0): ds.read_cursor = offset while ds.read_cursor < len(ds.input): b = ds.read_bytes(1)[0] if b != chr(0): ds.read_cursor -= 1 break if ds.read_cursor + 4 <= len(ds.input): magic = ds.read_bytes(4) print "at %d: %s" % (ds.read_cursor, repr(magic)) else: print "%d near end %d" % (ds.read_cursor, len(ds.input))
|
|
|
|
John Tobey (OP)
|
|
August 26, 2011, 05:34:44 PM |
|
- SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
- Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"
I didn't implement the "confirmations" filter yet. The joining looks correct. My "way" to format decimals is format_satoshis(satoshis, chain) . Couple of requests. Instead of IFNULL, use the portable COALESCE with the same arguments. Instead of '%s' and the % operator, use an unquoted question mark (?) and pass a tuple of bind values as the second argument to selectrow. (selectrow is the same as selectall but returns just the first row.) (Abe's database abstraction layer converts the standard qmark parameter style to whatever the driver needs.) Please capitalize SQL keywords like SELECT FROM JOIN ON WHERE AND for easier readability. Please prefix error messages with "ERROR:" as per BBE convention. And please show the usage message if address is None, even if chain is given. This is my convention where address is required, as in q_translate_address. Thanks!
|
|
|
|
newminerr
Member
Offline
Activity: 147
Merit: 11
The day to rise has come.
|
|
August 26, 2011, 06:44:15 PM |
|
#!/usr/bin/env python import BCDataStream ds = BCDataStream.BCDataStream() ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"), 169336263) offset = ds.read_cursor magic = ds.read_bytes(4) print "at %d of %d: %s" % (offset, len(ds.input), repr(magic)) if magic[0] == chr(0): ds.read_cursor = offset while ds.read_cursor < len(ds.input): b = ds.read_bytes(1)[0] if b != chr(0): ds.read_cursor -= 1 break if ds.read_cursor + 4 <= len(ds.input): magic = ds.read_bytes(4) print "at %d: %s" % (ds.read_cursor, repr(magic)) else: print "%d near end %d" % (ds.read_cursor, len(ds.input))
It returned this. C:\abe>python ab.py at 169336263 of 698256440: '\x00\x00\x00\x00' at 169394422: '\xf9\xbe\xb4\xd9'
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
August 26, 2011, 07:05:12 PM |
|
The joining looks correct. My "way" to format decimals is format_satoshis(satoshis, chain) . I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query) format_satoshis(decimal.Decimal('12.3456'), 1) gives other than that, I made alle the changes you requested: def q_getreceivedbyaddress(abe, page, chain): """getreceivedbyaddress""" addr = wsgiref.util.shift_path_info(page['env']) if chain is None or addr is None: return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \ '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS\n'
if ADDRESS_RE.match(addr): version, hash = decode_address(addr) sql = """ SELECT CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey JOIN txout on txout.pubkey_id=pubkey.pubkey_id JOIN block_tx on block_tx.tx_id=txout.tx_id JOIN block b on b.block_id=block_tx.block_id JOIN chain_candidate cc on cc.block_id=b.block_id WHERE pubkey_hash=? AND cc.chain_id=? AND cc.in_longest=1 ;""" row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id'])) ret = format_satoshis(row[0], chain); else: ret = 'ERROR: address invalid'
return ret def q_getsentbyaddress(abe, page, chain): """getsentbyaddress""" addr = wsgiref.util.shift_path_info(page['env']) if chain is None or addr is None: return 'returns amount of money sent from given address\n' \ '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'
if ADDRESS_RE.match(addr): version, hash = decode_address(addr) sql = """ SELECT CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey JOIN txout txout on txout.pubkey_id=pubkey.pubkey_id JOIN txin on txin.txout_id=txout.txout_id JOIN block_tx on block_tx.tx_id=txout.tx_id JOIN block b on b.block_id=block_tx.block_id JOIN chain_candidate cc on cc.block_id=b.block_id WHERE pubkey_hash=? AND cc.chain_id=? AND cc.in_longest=1 ;""" row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id'])) ret = format_satoshis(row[0], chain); else: ret = 'ERROR: address invalid'
return ret
Anything else? EDIT: I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
John Tobey (OP)
|
|
August 26, 2011, 09:25:03 PM |
|
It returned this. C:\abe>python ab.py at 169336263 of 698256440: '\x00\x00\x00\x00' at 169394422: '\xf9\xbe\xb4\xd9'
Thanks. I've committed a fix that I think will get you past this error. I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)
The problem was that format_satoshis expected the caller to cast its first argument to int. I decided it would be better for the function to do the casting. (txout_value is in 1e-8 BTC, so all values are integer.) I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)
I like little variables like that, they help document the intent and aid future tinkerers. Anything else?
Just some minor reformatting which I did. Thanks!
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
August 26, 2011, 09:41:17 PM |
|
Just some minor reformatting which I did. Thanks!
You're welcome. Love to be able to give back a little code, too.
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
John Tobey (OP)
|
|
August 26, 2011, 11:54:38 PM |
|
Thinking about also hosting a copycat of this... posting here to remind me to check up on it later. If you do, it could be abe.john-edwin-tobey.org, since the hosting arrangement there is ending. I'm switching DNS back to my pathetic excuse for a server, but if it can't take the load, I'll punt to whoever wants to run it. Of course, anyone with serious uptime requirements is welcome to run a private or public instance.
|
|
|
|
John Tobey (OP)
|
|
August 31, 2011, 05:43:51 AM |
|
Version 0.6 highlights: - Python packaging; abe.py moved; run as "python -m Abe.abe".
- Big speed improvements (c. 10x) for MySQL and SQLite.
- ODBC tested successfully.
- IBM DB2 tested successfully.
- HTTP API functions: getreceivedbyaddress getsentbyaddress.
- Verify transaction Merkle roots on block import.
- Show Namecoin-style network fees and name transaction outputs.
- Adjust coins outstanding and coin-days destroyed for Namecoin-style network fees.
- Native SolidCoin support.
- Suppress display of empty chains on home page.
- Show the search form on /chain/CHAIN pages.
- Many minor improvements; see the Git log.
Next up are a couple of bug fixes and the test suite. When the test suite passes, it is 1.0.
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
August 31, 2011, 10:43:08 PM |
|
Cool, a release! - Native SolidCoin support.
What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
John Tobey (OP)
|
|
September 01, 2011, 02:18:54 AM |
|
- Native SolidCoin support.
What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That? Yup, just that. No need to read SC source code to figure out magic number and address version byte, no need to touch Python code. Just pass "--datadir ~/.solidcoin". Btw, I've cut the stuff down a bit: {"chain":"SolidCoin", "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
September 01, 2011, 07:02:51 AM |
|
Yup, just that. No need to read SC source code to figure out magic number and address version byte, no need to touch Python code. Just pass "--datadir ~/.solidcoin". Btw, I've cut the stuff down a bit: {"chain":"SolidCoin", "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},
Nice one! Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver. Btw: should be possible to add caching headers and gzip compression, right?
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
John Tobey (OP)
|
|
September 01, 2011, 11:21:33 PM |
|
Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.
Btw: should be possible to add caching headers and gzip compression, right?
Provided it does not complicate the core stuff. Are you thinking of working on this? You could do it at a high level of abstraction ("squid") or a very low level (store compressed pages in db, add "last modified" columns to tx, pubkey, and block records) or in between, perhaps some WSGI middleware. If you are going for low level, some refactoring might be in order beforehand. To get cache headers on static content, I simply use FastCGI and configure static-path to something Apache can serve directly as described in README-FASTCGI.txt.
|
|
|
|
newminerr
Member
Offline
Activity: 147
Merit: 11
The day to rise has come.
|
|
September 02, 2011, 06:11:07 AM |
|
BTW it's working great now, keep up the good work.
|
|
|
|
|
John Tobey (OP)
|
|
September 04, 2011, 12:18:28 AM |
|
I've taken down abe.john-edwin-tobey.org due to heavy load. Email me or post here if you have server space and want help setting up!
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
September 04, 2011, 11:52:17 AM |
|
I've taken down abe.john-edwin-tobey.org due to heavy load. Email me or post here if you have server space and want help setting up!
Just rented a VPS for solidcoin.kicks-ass.org. Using bitcoin-abe there already for solidcoin. Want to add bitcoin anyways. I'm not sure if that thing (it's a virtual, of course) would be up to the task. Could you provide some traffic stats from prior to taking your server down so we can have a rough estimate?
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
John Tobey (OP)
|
|
September 04, 2011, 07:34:58 PM |
|
I've taken down abe.john-edwin-tobey.org due to heavy load. Email me or post here if you have server space and want help setting up!
Backpedaling... log showed 90% of requests from Googlebot. I'm attempting to solve this with robots.txt. Site is up for now. Just rented a VPS for solidcoin.kicks-ass.org. Using bitcoin-abe there already for solidcoin. Want to add bitcoin anyways.
I'm not sure if that thing (it's a virtual, of course) would be up to the task.
Could you provide some traffic stats from prior to taking your server down so we can have a rough estimate?
It wasn't getting a lot of hits, roughly 4 address or tx requests per minute. The VPS's swap space was chronically all used, though, causing the kernel to kill processes roughly once per hour, despite my cron job that bounced Apache and bitcoind every hour. So it was thrashing, and Postgres must have had to read from disk for most queries. There's certainly room for optimization in the schema, especially if you have plenty of disk, which I don't. But before I go that route, I'd like to know what's possible with a server with 1GB+ RAM. By the way, if you want to hide the "empty" currencies from the homepage, you could delete the respective rows from the chain table or upgrade to the latest 0.7pre. If you set up an instance with BTC, at least one alt chain, and better uptime than mine, and if you are willing, I will advertise it as the demo site.
|
|
|
|
molecular
Donator
Legendary
Offline
Activity: 2772
Merit: 1019
|
|
September 04, 2011, 09:03:18 PM |
|
There's certainly room for optimization in the schema, especially if you have plenty of disk, which I don't. But before I go that route, I'd like to know what's possible with a server with 1GB+ RAM.
Well, it seems my server wouldn't cut it. Only 512MB memory. No swapspace (!!). I'll probably setup bitcoin chain on it anyways, because I want to make my coin-age graphs live, so then I could do some manual benchmarking to see if it could make sense at all.
|
PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0 3F39 FC49 2362 F9B7 0769
|
|
|
|