[ANNOUNCE] Abe 0.7: Open Source Block Explorer Knockoff

[John Tobey]

No luck?

When you have finished following the instructions, the file /home/user/cgi-bin/abe should exist and be executable.  Issue "ls -l /home/user/cgi-bin/abe" and post its output if unsure.

Then test the site and post any error message from your server log or browser.

[terrytibbs]

No luck?

When you have finished following the instructions, the file /home/user/cgi-bin/abe should exist and be executable.  Issue "ls -l /home/user/cgi-bin/abe" and post its output if unsure.

Then test the site and post any error message from your server log or browser.

Yeah, I got it up and running. What I've been doing is reverse-proxying the requests from nginx to the built in server Abe uses. That isn't feasible for a production environment, though.

My next big challenge is getting Abe to run in a spawned FastCGI process using nginx.

[terrytibbs]

I might also add that getting it run over Apache using FastCGI was easy as pie.

[btc_artist]

Following this discussion.  Looks like a very nice tool.

[John Tobey]

Yeah, I got it up and running. What I've been doing is reverse-proxying the requests from nginx to the built in server Abe uses. That isn't feasible for a production environment, though.

My next big challenge is getting Abe to run in a spawned FastCGI process using nginx.

I have no experience with nginx, but FastCGI should work the same on any server.  I might add that the "sudo" overhead is not essential, it's just how I set things up to keep any Abe exploits out of Apache and vice versa.  If your security policy doesn't need that separation, I suggest naming the script /usr/lib/cgi-bin/abe and removing --watch-pid="$1":

Code:

#! /bin/sh
PYTHONUNBUFFERED=1 exec python -m Abe.abe \
--config /home/USER/abe.conf --static-path static/

And set "Alias / /usr/lib/cgi-bin/abe/" or the nginx equivalent in the server configuration.

[notawake]

I used Abe to output transactions for a given address to CSV format: https://bitcointalk.org/index.php?topic=51121

[John Tobey]

I used Abe to output transactions for a given address to CSV format: https://bitcointalk.org/index.php?topic=51121

Thanks for sharing!

I wonder if your GetTransactions function would work as a VIEW.  That might translate easily to MySQL etc.

[notawake]

I used Abe to output transactions for a given address to CSV format: https://bitcointalk.org/index.php?topic=51121

Thanks for sharing!

I wonder if your GetTransactions function would work as a VIEW.  That might translate easily to MySQL etc.

I don't think views can take a dynamic parameter, which is why I went with a function.

However, one could make a view with all the transactions and then just use a select statement from the view for records with the given address. I don't know which method (function vs. view) would be faster for selecting transactions and updating the database with new transactions. I guess the view would need to either be persisted, meaning it gets updated on every update, or dynamically generated on every query, which I would guess does not have a significant performance difference from the current function method.

As for compatibility, I know contemporary versions of MySQL have function capabilities, but I'm not sure about some of the older versions. Actually, I think older versions of PostgreSQL may not allow for the type of function I'm using, i.e. returning a table.

[terrytibbs]

And set "Alias / /usr/lib/cgi-bin/abe/" or the nginx equivalent in the server configuration.

It's not quite that simple. nginx's lightweight nature does not automagically spawn FastCGI threads.

If anyone cares, you can use spawn-fcgi, 'fastcgi_pass'ing the requests from nginx to loopback or UNIX socket or whatever you decide to use. Remember to have nginx serve your static files, or you'll lose the entire benefit of using it in the first place.

[terrytibbs]

If I'd like to modify Abe without modifying the original code, what's the go-to way of hooking or wrapping into it so as to maintain compatibility with the original codebase and its updates?

[molecular]

If I'd like to modify Abe without modifying the original code, what's the go-to way of hooking or wrapping into it so as to maintain compatibility with the original codebase and its updates?

+1. I've been "hacking up" bitcoin-abe by adding functions "q_whatever(...)" to abe.py. Now I can't easily update.

[John Tobey]

If I'd like to modify Abe without modifying the original code, what's the go-to way of hooking or wrapping into it so as to maintain compatibility with the original codebase and its updates?

+1. I've been "hacking up" bitcoin-abe by adding functions "q_whatever(...)" to abe.py. Now I can't easily update.

Just uploaded an example (untested by me) of subclassing Abe.abe to add handle_* functions.  I suspect q_* would work too.

https://github.com/jtobey/bitcoin-abe/blob/master/contrib/ecdsa.py

If you come to rely on an undocumented interface, the best way to keep it around is to document it and ask me to commit the doc.

[terrytibbs]

I'm getting "Commands out of sync; you can't run this command now" MySQL errors. Could this tie in to Litecoin experiencing spamming in the form of hundreds if not thousands of 0.00000001 outputs per block?

Case in point: http://explorer.liteco.in/

[John Tobey]

I'm getting "Commands out of sync; you can't run this command now" MySQL errors.

Hmm, that's a new one to me.  Could you post a way to reliably produce the error in my own environment?  Or the next best thing would be to run with --log-sql and post a section of log including, say, 5 SQL commands leading up to the error.

[terrytibbs]

I'm getting "Commands out of sync; you can't run this command now" MySQL errors.

Hmm, that's a new one to me.  Could you post a way to reliably produce the error in my own environment?  Or the next best thing would be to run with --log-sql and post a section of log including, say, 5 SQL commands leading up to the error.

I've switched to Postgres, and it seems to be holding up fairly well so far, other than the FastCGI process dying every once in a while.

I will set a test environment up this weekend and work on reproducing it myself.

[molecular]

Hey John,

I ran into some weird and massive performance issues (litecoin data, older version of abe, mysql). Couldn't find out what exactly is wrong. mysqld just seems to be really slow. It's inserting the blocks almost as slowly as they are mined and mysqld cripples my system hogging i/o, I guess.

So I just upgraded to newest version from git (actually made a fork) and ran into a typo, I guess:

        if count % 1000 == 0:
            store.commit()
            stor.log.info("Updated %d blocks", count)

NameError: global name 'stor' is not defined

[John Tobey]

Hey John,

I ran into some weird and massive performance issues (litecoin data, older version of abe, mysql). Couldn't find out what exactly is wrong. mysqld just seems to be really slow. It's inserting the blocks almost as slowly as they are mined and mysqld cripples my system hogging i/o, I guess.

If it persists, I find an easy way to "profile" abe is to interrupt it (ctrl-C) and note the stack trace, then restart, let it get going, and repeat a few times.  If the interrupt usually happens in the same query or two, I know what to optimize.

NameError: global name 'stor' is not defined

Fixed, thanks.

[John Tobey]

I'm getting "Commands out of sync; you can't run this command now" MySQL errors.

Hmm, that's a new one to me.  Could you post a way to reliably produce the error in my own environment?  Or the next best thing would be to run with --log-sql and post a section of log including, say, 5 SQL commands leading up to the error.

Come to think of it, a plain old stack trace would be better than nothing if you have one around...

I've switched to Postgres, and it seems to be holding up fairly well so far, other than the FastCGI process dying every once in a while.

Any error message in the log or browser?

[molecular]

I ran into some weird and massive performance issues (litecoin data, older version of abe, mysql). Couldn't find out what exactly is wrong. mysqld just seems to be really slow. It's inserting the blocks almost as slowly as they are mined and mysqld cripples my system hogging i/o, I guess.

If it persists, I find an easy way to "profile" abe is to interrupt it (ctrl-C) and note the stack trace, then restart, let it get going, and repeat a few times.  If the interrupt usually happens in the same query or two, I know what to optimize.

It persists. Even accross db venors, meaning that postgres is - while faster than mysql - also having a hard time doing these inserts (manly tx, I guess)

You don't happen to have a part with O

I think

[John Tobey]

It persists. Even accross db venors, meaning that postgres is - while faster than mysql - also having a hard time doing these inserts (manly tx, I guess)

Please try with --commit-bytes=100000 and let me know.  This will prevent a commit after every tx insertion but may lead to errors when concurrent processes insert.  I recently changed this setting from being the default.

How is system load?  Is the database server busy with CPU?  IO?  Is there free memory?

When I have time, I will try some load testing.