Bitcoin Forum
December 08, 2016, 04:27:20 PM *
News: Latest stable version of Bitcoin Core: 0.13.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 [6] 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 »
  Print  
Author Topic: [ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff  (Read 175759 times)
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
August 26, 2011, 02:46:35 PM
 #101

John,

have you implemented q/getreceivedbyaddress and q/getsentbyaddress yet?

I have:

    import decimal.Decimal

    def q_getreceivedbyaddress(abe, page, chain):
        """getreceivedbyaddress"""
        if chain is None:
            return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \
                '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS*\n'
        addr = wsgiref.util.shift_path_info(page['env'])

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            select
              ifnull(sum(txout.txout_value),0) from pubkey
            join txout on txout.pubkey_id=pubkey.pubkey_id
            join block_tx on block_tx.tx_id=txout.tx_id
            join block b on b.block_id=block_tx.block_id
            join chain_candidate cc on cc.block_id=b.block_id
            where
              pubkey_hash='%s' and
              cc.chain_id=%s and
              cc.in_longest=1
            ;""" % (binascii.hexlify(hash), chain['id'])
          #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging
          rows = abe.store.selectall(sql)
          ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string()
        else:
          ret = 'address invalid'

        return ret
       
    def q_getsentbyaddress(abe, page, chain):
        """getsentbyaddress"""
        if chain is None:
            return 'returns amount of money sent from given address\n' \
                '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'
        addr = wsgiref.util.shift_path_info(page['env'])

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            select
              ifnull(sum(txout.txout_value),0) from pubkey
            join txout txout on txout.pubkey_id=pubkey.pubkey_id
            join txin on txin.txout_id=txout.txout_id
            join block_tx on block_tx.tx_id=txout.tx_id
            join block b on b.block_id=block_tx.block_id
            join chain_candidate cc on cc.block_id=b.block_id
            where
              pubkey_hash='%s' and
              cc.chain_id=%s and
              cc.in_longest=1
            ;""" % (binascii.hexlify(hash), chain['id'])
          #ret = NETHASH_HEADER + '\n' + sql + '\n\n' # debugging
          rows = abe.store.selectall(sql)
          ret = (rows[0][0] / decimal.Decimal('1E8')).to_eng_string()
        else:
          ret = 'address invalid'

        return ret



If you want to use it, please take a look at the following things especially:

  • SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
  • Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"

I didn't implement the "confirmations" filter yet.



PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
1481214440
Hero Member
*
Offline Offline

Posts: 1481214440

View Profile Personal Message (Offline)

Ignore
1481214440
Reply with quote  #2

1481214440
Report to moderator
1481214440
Hero Member
*
Offline Offline

Posts: 1481214440

View Profile Personal Message (Offline)

Ignore
1481214440
Reply with quote  #2

1481214440
Report to moderator
1481214440
Hero Member
*
Offline Offline

Posts: 1481214440

View Profile Personal Message (Offline)

Ignore
1481214440
Reply with quote  #2

1481214440
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1481214440
Hero Member
*
Offline Offline

Posts: 1481214440

View Profile Personal Message (Offline)

Ignore
1481214440
Reply with quote  #2

1481214440
Report to moderator
1481214440
Hero Member
*
Offline Offline

Posts: 1481214440

View Profile Personal Message (Offline)

Ignore
1481214440
Reply with quote  #2

1481214440
Report to moderator
newminerr
Member
**
Offline Offline

Activity: 70

like no other.


View Profile
August 26, 2011, 03:52:58 PM
 #102

Quote
chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263
Though i am not running bitcoin at the moment.
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 26, 2011, 05:09:10 PM
 #103

Quote
chain not found for magic '\x00\x00\x00\x00' in block file C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat at offset 169336263
Though i am not running bitcoin at the moment.

Hmm, running will not help with this error.  Have you redownloaded the block chain since the last abe.py run?  If so,
Code:
UPDATE datadir SET blkfile_number = 1, blkfile_offset = 0
will force a rescan.

If not, well, now would be a good time to implement scanning for the magic number sequence, but that is not trivial.  Maybe skipping past '\x00' would solve it here.  I am curious what this program prints if you run it in the bitcoin-abe directory:
Code:
#!/usr/bin/env python
import BCDataStream
ds = BCDataStream.BCDataStream()
ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"),
            169336263)
offset = ds.read_cursor
magic = ds.read_bytes(4)
print "at %d of %d: %s" % (offset, len(ds.input), repr(magic))
if magic[0] == chr(0):
    ds.read_cursor = offset
    while ds.read_cursor < len(ds.input):
        b = ds.read_bytes(1)[0]
        if b != chr(0):
            ds.read_cursor -= 1
            break
    if ds.read_cursor + 4 <= len(ds.input):
        magic = ds.read_bytes(4)
        print "at %d: %s" % (ds.read_cursor, repr(magic))
    else:
        print "%d near end %d" % (ds.read_cursor, len(ds.input))

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 26, 2011, 05:34:44 PM
 #104

  • SQL Query correct? (longest chain only, correct joining of txout in getsentbyaddress?)
  • Handling of decimal values correct? Is it "your way"? Probably not since I needed to "import decimal"

I didn't implement the "confirmations" filter yet.

The joining looks correct.  My "way" to format decimals is
Code:
format_satoshis(satoshis, chain)
.

Couple of requests.  Instead of IFNULL, use the portable COALESCE with the same arguments.  Instead of '%s' and the % operator, use an unquoted question mark (?) and pass a tuple of bind values as the second argument to selectrow.  (selectrow is the same as selectall but returns just the first row.)  (Abe's database abstraction layer converts the standard qmark parameter style to whatever the driver needs.)  Please capitalize SQL keywords like SELECT FROM JOIN ON WHERE AND for easier readability.  Please prefix error messages with "ERROR:" as per BBE convention.  And please show the usage message if address is None, even if chain is given.  This is my convention where address is required, as in q_translate_address.

Thanks!

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
newminerr
Member
**
Offline Offline

Activity: 70

like no other.


View Profile
August 26, 2011, 06:44:15 PM
 #105

Code:
#!/usr/bin/env python
import BCDataStream
ds = BCDataStream.BCDataStream()
ds.map_file(open(r"C:\Users\user\AppData\Roaming\Bitcoin\blk0001.dat", "rb"),
            169336263)
offset = ds.read_cursor
magic = ds.read_bytes(4)
print "at %d of %d: %s" % (offset, len(ds.input), repr(magic))
if magic[0] == chr(0):
    ds.read_cursor = offset
    while ds.read_cursor < len(ds.input):
        b = ds.read_bytes(1)[0]
        if b != chr(0):
            ds.read_cursor -= 1
            break
    if ds.read_cursor + 4 <= len(ds.input):
        magic = ds.read_bytes(4)
        print "at %d: %s" % (ds.read_cursor, repr(magic))
    else:
        print "%d near end %d" % (ds.read_cursor, len(ds.input))
It returned this.
Code:
C:\abe>python ab.py
at 169336263 of 698256440: '\x00\x00\x00\x00'
at 169394422: '\xf9\xbe\xb4\xd9'
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
August 26, 2011, 07:05:12 PM
 #106

The joining looks correct.  My "way" to format decimals is
Code:
format_satoshis(satoshis, chain)
.

I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)

Code:
format_satoshis(decimal.Decimal('12.3456'), 1)
gives
Code:
1.23456E-7.012.3456

other than that, I made alle the changes you requested:

Code:

    def q_getreceivedbyaddress(abe, page, chain):
        """getreceivedbyaddress"""
        addr = wsgiref.util.shift_path_info(page['env'])
        if chain is None or addr is None:
            return 'returns amount of money received by given address (not balance, sends are not subtracted)\n' \
                '/chain/CHAIN/q/getreceivedbyaddress/ADDRESS\n'

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            SELECT
              CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey
            JOIN txout on txout.pubkey_id=pubkey.pubkey_id
            JOIN block_tx on block_tx.tx_id=txout.tx_id
            JOIN block b on b.block_id=block_tx.block_id
            JOIN chain_candidate cc on cc.block_id=b.block_id
            WHERE
              pubkey_hash=? AND
              cc.chain_id=? AND
              cc.in_longest=1
            ;"""
          row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id']))
          ret = format_satoshis(row[0], chain);
        else:
          ret = 'ERROR: address invalid'

        return ret
        
    def q_getsentbyaddress(abe, page, chain):
        """getsentbyaddress"""
        addr = wsgiref.util.shift_path_info(page['env'])
        if chain is None or addr is None:
            return 'returns amount of money sent from given address\n' \
                '/chain/CHAIN/q/getsentbyaddress/ADDRESS\n'

        if ADDRESS_RE.match(addr):
          version, hash = decode_address(addr)
          sql = """
            SELECT
              CONVERT(COALESCE(SUM(txout.txout_value),0), UNSIGNED) from pubkey
            JOIN txout txout on txout.pubkey_id=pubkey.pubkey_id
            JOIN txin on txin.txout_id=txout.txout_id
            JOIN block_tx on block_tx.tx_id=txout.tx_id
            JOIN block b on b.block_id=block_tx.block_id
            JOIN chain_candidate cc on cc.block_id=b.block_id
            WHERE
              pubkey_hash=? AND
              cc.chain_id=? AND
              cc.in_longest=1
            ;"""
          row = abe.store.selectrow(sql, (binascii.hexlify(hash), chain['id']))
          ret = format_satoshis(row[0], chain);
        else:
          ret = 'ERROR: address invalid'

        return ret

Anything else?

EDIT: I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 26, 2011, 09:25:03 PM
 #107

It returned this.
Code:
C:\abe>python ab.py
at 169336263 of 698256440: '\x00\x00\x00\x00'
at 169394422: '\xf9\xbe\xb4\xd9'
Thanks.  I've committed a fix that I think will get you past this error.

I've had some problem with format_satoshis (worked around them by casting to UNSIGNED in the sql query)
The problem was that format_satoshis expected the caller to cast its first argument to int.  I decided it would be better for the function to do the casting.  (txout_value is in 1e-8 BTC, so all values are integer.)

I could put the sql string directly into the call to selectrow(...). The sql variable was only used for debugging output, which doesn't make too much sense any more anyways since the values are to filled in any more (% -> ?)
I like little variables like that, they help document the intent and aid future tinkerers.

Anything else?
Just some minor reformatting which I did.  Thanks!

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
August 26, 2011, 09:41:17 PM
 #108

Just some minor reformatting which I did.  Thanks!

You're welcome. Love to be able to give back a little code, too.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 26, 2011, 11:54:38 PM
 #109

Thinking about also hosting a copycat of this... posting here to remind me to check up on it later.  Wink
If you do, it could be abe.john-edwin-tobey.org, since the hosting arrangement there is ending.  I'm switching DNS back to my pathetic excuse for a server, but if it can't take the load, I'll punt to whoever wants to run it.

Of course, anyone with serious uptime requirements is welcome to run a private or public instance.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
August 31, 2011, 05:43:51 AM
 #110

Version 0.6 highlights:
  • Python packaging; abe.py moved; run as "python -m Abe.abe".
  • Big speed improvements (c. 10x) for MySQL and SQLite.
  • ODBC tested successfully.
  • IBM DB2 tested successfully.
  • HTTP API functions: getreceivedbyaddress getsentbyaddress.
  • Verify transaction Merkle roots on block import.
  • Show Namecoin-style network fees and name transaction outputs.
  • Adjust coins outstanding and coin-days destroyed for Namecoin-style network fees.
  • Native SolidCoin support.
  • Suppress display of empty chains on home page.
  • Show the search form on /chain/CHAIN pages.
  • Many minor improvements; see the Git log.

Next up are a couple of bug fixes and the test suite.  When the test suite passes, it is 1.0.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
August 31, 2011, 10:43:08 PM
 #111

Cool, a release!

  • Native SolidCoin support.

What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 01, 2011, 02:18:54 AM
 #112

  • Native SolidCoin support.

What's that mean? I added SolidCoin before myself by adding all the magic and policy stuff in DataStore.py. That?

Yup, just that.  No need to read SC source code to figure out magic number and address version byte, no need to touch Python code.  Just pass "--datadir ~/.solidcoin".

Btw, I've cut the stuff down a bit:
Code:
    {"chain":"SolidCoin",
     "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
September 01, 2011, 07:02:51 AM
 #113

Yup, just that.  No need to read SC source code to figure out magic number and address version byte, no need to touch Python code.  Just pass "--datadir ~/.solidcoin".

Btw, I've cut the stuff down a bit:
Code:
    {"chain":"SolidCoin",
     "code3":"SCN", "address_version":"\x7d", "magic":"\xde\xad\xba\xbe"},

Nice one!

Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.

Btw: should be possible to add caching headers and gzip compression, right?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 01, 2011, 11:21:33 PM
 #114

Now I just have to figure out how I can rebase that with git, because I'm misusing abe as a webserver.

Btw: should be possible to add caching headers and gzip compression, right?
Provided it does not complicate the core stuff.  Are you thinking of working on this?  You could do it at a high level of abstraction ("squid") or a very low level (store compressed pages in db, add "last modified" columns to tx, pubkey, and block records) or in between, perhaps some WSGI middleware.  If you are going for low level, some refactoring might be in order beforehand.

To get cache headers on static content, I simply use FastCGI and configure static-path to something Apache can serve directly as described in README-FASTCGI.txt.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
newminerr
Member
**
Offline Offline

Activity: 70

like no other.


View Profile
September 02, 2011, 06:11:07 AM
 #115

BTW it's working great now, keep up the good work.
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 02, 2011, 01:27:41 PM
 #116

BTW it's working great now, keep up the good work.

Great.  Let me know if you try the Windows installer how it goes: http://pypi.python.org/pypi/Abe/0.6

PGP key: https://raw.github.com/jtobey/bitcoin-abe/master/doc/jtobey.pubkey
Key fingerprint: 413D 2920 B776 08D1 A867  232A 3CB0 B139 6A71 5348

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 04, 2011, 12:18:28 AM
 #117

I've taken down abe.john-edwin-tobey.org due to heavy load.  Email me or post here if you have server space and want help setting up!

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
September 04, 2011, 11:52:17 AM
 #118

I've taken down abe.john-edwin-tobey.org due to heavy load.  Email me or post here if you have server space and want help setting up!


Just rented a VPS for solidcoin.kicks-ass.org. Using bitcoin-abe there already for solidcoin. Want to add bitcoin anyways.

I'm not sure if that thing (it's a virtual, of course) would be up to the task.

Could you provide some traffic stats from prior to taking your server down so we can have a rough estimate?

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
John Tobey
Hero Member
*****
Offline Offline

Activity: 481



View Profile WWW
September 04, 2011, 07:34:58 PM
 #119

I've taken down abe.john-edwin-tobey.org due to heavy load.  Email me or post here if you have server space and want help setting up!

Backpedaling... log showed 90% of requests from Googlebot.  I'm attempting to solve this with robots.txt.  Site is up for now.

Just rented a VPS for solidcoin.kicks-ass.org. Using bitcoin-abe there already for solidcoin. Want to add bitcoin anyways.

I'm not sure if that thing (it's a virtual, of course) would be up to the task.

Could you provide some traffic stats from prior to taking your server down so we can have a rough estimate?

It wasn't getting a lot of hits, roughly 4 address or tx requests per minute.  The VPS's swap space was chronically all used, though, causing the kernel to kill processes roughly once per hour, despite my cron job that bounced Apache and bitcoind every hour.  So it was thrashing, and Postgres must have had to read from disk for most queries.

There's certainly room for optimization in the schema, especially if you have plenty of disk, which I don't.  But before I go that route, I'd like to know what's possible with a server with 1GB+ RAM.

By the way, if you want to hide the "empty" currencies from the homepage, you could delete the respective rows from the chain table or upgrade to the latest 0.7pre.

If you set up an instance with BTC, at least one alt chain, and better uptime than mine, and if you are willing, I will advertise it as the demo site.

Can a change to the best-chain criteria protect against 51% to 90+% attacks without a hard fork?
molecular
Donator
Legendary
*
Offline Offline

Activity: 2142



View Profile
September 04, 2011, 09:03:18 PM
 #120

There's certainly room for optimization in the schema, especially if you have plenty of disk, which I don't.  But before I go that route, I'd like to know what's possible with a server with 1GB+ RAM.

Well, it seems my server wouldn't cut it. Only 512MB memory. No swapspace (!!).

I'll probably setup bitcoin chain on it anyways, because I want to make my coin-age graphs live, so then I could do some manual benchmarking to see if it could make sense at all.

PGP key molecular F9B70769 fingerprint 9CDD C0D3 20F8 279F 6BE0  3F39 FC49 2362 F9B7 0769
Pages: « 1 2 3 4 5 [6] 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!