CryptoCoin Explorer - Coin Block Explorers.

[Sunny King]

PPC explorer seems down to me.

[John Tobey]

After some research I found that Chrome has a tendency to create a rather large amount of parallel connections and then proceeds to communicate on the second connection it created.

Well, that's pretty perverse, but all right, we should be multithreaded.

I will start working on using a different server mechanism for the explorers, possibly Twisted or Fastcgi.

If anybody has suggestions or would like to help, it would be greatly appreciated. The source for the PPC and FRC explorers are on github.

If you have any trouble with README-FASTCGI.txt, I'd like to know.

Easier, though not as well tested: define a server class in abe.py:

Code:

from wsgiref.simple_server import WSGIServer
import SocketServer
class HttpServer(SocketServer.ThreadingMixIn, WSGIServer):
    pass

Then add the text in bold: httpd = make_server(args.host, port, abe, server_class=HttpServer)

There is also SocketServer.ForkingMixIn in case ThreadingMixIn gives weird errors.

[dreamwatcher]

After some research I found that Chrome has a tendency to create a rather large amount of parallel connections and then proceeds to communicate on the second connection it created.

Well, that's pretty perverse, but all right, we should be multithreaded.

I will start working on using a different server mechanism for the explorers, possibly Twisted or Fastcgi.

If anybody has suggestions or would like to help, it would be greatly appreciated. The source for the PPC and FRC explorers are on github.

If you have any trouble with README-FASTCGI.txt, I'd like to know.

Easier, though not as well tested: define a server class in abe.py:

Code:

from wsgiref.simple_server import WSGIServer
import SocketServer
class HttpServer(SocketServer.ThreadingMixIn, WSGIServer):
    pass

Then add the text in bold: httpd = make_server(args.host, port, abe, server_class=HttpServer)

There is also SocketServer.ForkingMixIn in case ThreadingMixIn gives weird errors.

Thank you.

Hopefully I can get to it this weekend.

I have also been looking at cherry.py.

[dreamwatcher]

I have seen a couple of posts and I have also had some issues with test nets on some of the alt-coins. I will put up some test net daemons on my sites as I see requests or as I see are needed.

The addnode command is flexible in the sense it will work for both live and test net. Addnode will attempt a connection and still try to find other nodes, unlike the "connection="  which is an explicit command to only connect to that address.


First up TRC:

I have started a test net daemon for TRC on the cryptocoinexplorer.com server.

add to your config file:

addnode=84.200.84.74

This will have your daemon automatically try the cryptocoin explorer server for a connection.I will keep the daemon up as long as it does not start to effect the explorer site itself.

You can also add:

addnode=84.200.84.76

That is the IP for my future alt-coin mining site (cryptocoinmine.com)I will run a TRC test net daemon as much as possible, at least until the mining site goes live.  

[dreamwatcher]

Update:

Some may have noticed the explorers going up and down today, that was myself working to solve the major issues.

Some things behind the scenes have changed.

1. The database refresh is now completely separate from the explorers web serving.  I have set one instance of ABE for each coin to serve without updating (using the datadir = [] trick I read about in the ABE github. Thanks John Tobey).

2. I am using the blocknotify feature of the daemons to run a no server config of ABE to update the database whenever there is a new block on the network. The no server config exits once the database is refreshed.

3. I am now testing on the TRC explorer a pull request by dermoth for ABE, that allows ABE to recover gracefully from broken pipes. I had tried a variation of this pull request when I first put the explorers on github. Those changes are what caused the problems with generating a new database that I had to revert the first day. I am sure it was some change I did to dermoths' commit, so after some more testing on the live server, I will try the modified TRC explorer with creating a new database.

I am hoping by limiting the public facing explorers to reading the database, it will solve some of the deadlocking problems. I also believe by precisely updating the database only when the blocks actually change and then stopping the refresh process until the next block, will reduce the thread load and help keep things running smooth.

After more testing I will issue a commit on github with the changes and documentation to make it all work.


If these changes work out, the last major bug to squash is the TRC difficulty (0 nBits) bug. It appears that the difficulty issue is only a symptom and the real issue is ABE is putting a null block on the latest database block entry.

[dreamwatcher]

Finally some good news...It looks like  dermoths' commit is working well. i just noticed the first bout of connection resets and broken pipes, and ABE gracefully handled them without getting stuck or crashing.

Code:

Traceback (most recent call last):
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 86, in run
    self.finish_response()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 127, in finish_response
    self.write(data)
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 210, in write
    self.send_headers()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 268, in send_headers
    self.send_preamble()
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 189, in send_preamble
    self._write('HTTP/%s %s\r\n' % (self.http_version,self.status))
  File "/usr/lib/python2.7/wsgiref/handlers.py", line 389, in _write
    self.stdout.write(data)
  File "/usr/lib/python2.7/socket.py", line 324, in write
    self.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 104] Connection reset by peer
----------------------------------------
Exception happened during processing of request from ('xxx.xxx.xxx.xxx', 52683)
Traceback (most recent call last):
  File "/usr/lib/python2.7/SocketServer.py", line 284, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 310, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 640, in __init__
    self.finish()
  File "/usr/lib/python2.7/SocketServer.py", line 693, in finish
    self.wfile.flush()
  File "/usr/lib/python2.7/socket.py", line 303, in flush
    self._sock.sendall(view[write_offset:write_offset+buffer_size])
error: [Errno 32] Broken pipe
----------------------------------------
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:37:55] "GET /logo32.png HTTP/1.1" 200 13245
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:37:55] "GET /abe.css HTTP/1.1" 200 222
xxx.xxx.xxx.xxx- - [01/Apr/2013 01:38:05] "GET /chain/Terracoin HTTP/1.1" 200 8023
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:05] "GET /chain/Terracoin/q/getblockcount HTTP/1.0" 200 5
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:05] "GET /chain/Terracoin/q/getdifficulty HTTP/1.0" 200 8
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:05] "GET /chain/Terracoin/q/getblockcount HTTP/1.0" 200 5
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:05] "GET /chain/Terracoin/q/getdifficulty HTTP/1.0" 200 8
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:05] "GET /abe.css HTTP/1.1" 200 222
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:05] "GET /logo32.png HTTP/1.1" 200 13245
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:06] "GET / HTTP/1.1" 200 2492
xxx.xxx.xxx.xxx- - [01/Apr/2013 01:38:07] "GET /abe.css HTTP/1.1" 200 222
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:07] "GET /logo32.png HTTP/1.1" 200 13245
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:07] "GET /favicon.ico HTTP/1.1" 200 20222
xxx.xxx.xxx.xxx - - [01/Apr/2013 01:38:07] "GET /favicon.ico HTTP/1.1" 200 20222

[Sunny King]

Nice!

[dreamwatcher]

All the explorers and webpages are down.

It appears the issue is on the hosting end as I cannot SSH into either the Cryptocoinexplorer server or the new pool server I am developing.


I will post any updates here.   

[dreamwatcher]

Everything is back up.

I will get the test net daemons up in a few.

[dreamwatcher]

Just thought I would give an update as to what is going on with the explorer and related projects.

1. dermoths' ABE pull request that I applied to the TRC explorer does much in the way of reducing explorer lock up. However, it does not completely solve the issue and the TRC explorer has locked a couple of times, but that is a big improvement. Hopefully this is no longer an issue.

2. I have decided the only way to correctly fix the issue is to run the explorers with a different server framework. I have tried to get it to run on the FastCGI framework, but I cannot get it to work correctly and it is probably  an issue on my part. Moved to FastCGI.

3. I have been working hard on the new pool. I have gained enough experience with  Pushpool, to use it for all the alt-coins I plan to support. I have been looking at Eloipool and I should be able to adapt it relatively quickly once the alt-coins move to a 0.8 BTC daemon base. Eloipool is written in python which I find much easier to code with then C. After experimenting with integrating front end components into Pushpool, I have found I really dislike C vs Python.

4. Through writing the front end of the pool from scratch, I have come to the decision to use Cherrypy as the server framework for the pool. I am currently working on the payout system, once that is finished I will most likely open the pool up with Bytecoin. This is not because of any kind of "coin loyalty" , but for the simple reason that it is probably the least mined of the current coins because of its newness. Besides, my "coin loyalty" lies mostly with PPC, as it was the first coin I started to do projects with.  

TRC and FRC will be the second pools as they really needs some pool choices. PPC will come last only because of its popularity, I need to shake the bugs out and make sure of server capacity before opening.


5. My experience with Cherrypy has also led me to the decision to use it for the server framework replacement in the explorers. Moved to FastCGI instead

6. I apologize for the slow speed at which improvements and other projects are coming out. I am doing these projects alone, along with a wife, school and keeping enough cash flow through the business to keep everything going. I cannot remember ever pulling as many all nighters in my life as I have recently  



I am a bit sad to see doublec shutdown his projects in the alt-coin world. I can see where he is coming from as the production and upkeep of projects in this realm can be much more work then most people realize.

I wish him the best of luck in his future endeavors.

I will try and get as much done as I can this weekend, although I have some "Honeydo" projects around the house to get done.  

[dreamwatcher]

This evening I finished switching the entire CryptoCoinExplorer site to use the FastCGI server framework.

Since the sudden up shoot of Alt-coins this weekend, I felt I did not have the luxury of time to completely switch over to a Cherrypy framework.

Now it is observe and tweak as needed......and chase down bothersome IP's that just make it slower for everybody...

[dreamwatcher]

So far so good, I have not had to unlock an explorer since the framework change. I only reset the web server once as I had forgotten to set the ppcexplore.org address in the server alias config.


I want to gauge the interest in seeing Novacoin and/or Bytecoin on CCE.

Both these coins already have at least one public explorer, so the decision to add them is going to be based mostly on the interest of CCE users through the poll.

[dreamwatcher]

Poll bump......

[Walter Rothbard]

Hey, dreamwatcher, have you updated to the new Terracoin emergency build from yesterday?  Block 101631 should be showing a difficulty drop (I think), and is not.

[dreamwatcher]

Yes I updated the daemon last night .....  Yet again!?!?!?...

Code:

101642	2013-04-09 16:29:09	1	20	1.0	2032890	30.0376	165.304	64.2496%
101641	2013-04-09 16:25:09	1	20	1.0	2032870	30.0351	165.301	64.2517%
101640	2013-04-09 16:25:09	1	20	1.0	2032850	30.0354	165.301	64.2517%
101639	2013-04-09 16:23:09	1	20	1.0	2032830	30.0343	165.3	64.2528%
101638	2013-04-09 16:22:00	1	20	2.113	2032810	30.0338	165.299	64.2534%
101637	2013-04-09 16:22:00	1	20	8.452	2032790	30.0341	165.299	64.2534%
101636	2013-04-09 16:22:00	4	5343.18432958	33.809	2032770	30.0344	165.299	64.2534%
101635	2013-04-09 16:22:00	1	20	135.239	2032750	30.0397	165.299	64.2474%
101634	2013-04-09 16:01:04	1	20	1352.383	2032730	30.0255	165.285	64.2585%
101633	2013-04-09 15:40:30	1	20	13523.835	2032710	30.0115	165.27	64.2695%
101632	2013-04-09 15:20:17	1	20	135238.105	2032690	29.9977	165.256	64.2802%
101631	2013-04-09 15:00:00	1	20	135238.105	2032670	29.9839	165.242	64.291%
101630	2013-04-09 15:00:00	1	20	135238.105	2032650	29.9842	165.242	64.291%

It appears the actual drop started on block   101633. Depending on the alog it may have taken a couple of blocks for the difficulty to actually drop, even though the version cutoff may have been on 101631.

For example noted in the code:

Code:

If the new block's timestamp is more than 20 minutes
    // have to be greater than the max accepted time delta ; 15mins.
    // this way, if one would artificially increase block nTime to its max value,
    // we'd still take the 5mins periods without block before allowing a one-shot
    // diff decrase, later keeping the block time used for ema computation.

[Walter Rothbard]

Thanks, dw

[Bitsinmyhead]

How do I get the terracoin difficulty in API?
When I use: http://www.cryptocoinexplorer.com:3750/q/getdifficulty

I only get:
Shows the difficulty of the last block in CHAIN.
/chain/CHAIN/q/getdifficulty

Someone please help.

[dreamwatcher]

How do I get the terracoin difficulty in API?
When I use: http://www.cryptocoinexplorer.com:3750/q/getdifficulty

I only get:
Shows the difficulty of the last block in CHAIN.
/chain/CHAIN/q/getdifficulty

Someone please help.

The API instructions read:

Code:

Shows the difficulty of the last block in CHAIN.
/chain/CHAIN/q/getdifficulty

So the API command would look like:

Code:

http://cryptocoinexplorer.com:3750/chain/Terracoin/q/getdifficulty

That is the basic structure of all the API commands. One appends the url with  "/chain/<NAME OF COIN>/q/<API Command>"

The name of the coin is case sensitive,  Terracoin and Freicoin both start with an upper case with the rest lower case.

PPC is a bit different as I should have coded it like "PPCoin", however I do not want to change it now as it might break an application that has it set the current way.

For PPC use "PPcoin"


For more information on using the API, click the API link in the various explorers.  

[dreamwatcher]

Just a small update:

1. I changed the way database update process is called so FastCGI does not try and use those temporary processes to process web requests. This would cause slow response times due to the process being set not to serve and shutdown after updating the database.

2. Changed the way error logs are handled, I hope this will suppress those page long error messages, mostly due to the issue addressed above.




POLL

Only 38 votes!!!!!!......The site has over 70,000 hits in the last ~45 days....It is hard for me to justify a decision based on such a small sample size of site users.  

Please take a second to vote.

Your input is very important to me.

[dreamwatcher]

I will close the poll Friday.

With only 41 votes, I am starting to think nobody cares 

Anyway, so far Novacoin is a shoo in,  Bytecoin fans get to voting!!!!