[UPDATES] CryptoCoin Explorer III Development Updates (CryptoCoinExplorer.com)

[dreamwatcher]

I may have solved the issue (1) with a combo of using a persistent DB wrapper and specifically forking ledger requests.

I have applied the changes to all the explorers.

Keeping my fingers crossed. 

[1715085431]

1715085431

[1715085431]

1715085431

[1715085431]

1715085431

[dreamwatcher]

UPDATE 10/02/2013:

WHEW!!! What a roller-coaster the last few days. All kinds of bugs and reversions popped up, so the test explorers have been kind of screwy the last few days.

Fixed:

I finally tracked down the exact cause of the thread delay and lockup during a large ledger requests. I had been looking at the wrong part of the server function. The problem was not the threads themselves, but the database connection(s). The persistent database connection appeared to solve the issue, but caused many other issues that made me lose more hair off my already bald head. 
It was causing major problems with the dbloader, and the whole of CCE3 went sideways for a bit.

I found the solution within Cherrypy itself (It is amazing how flexible this framework is!!). The web server now issues each thread its own database connection upon creation and closes the connection upon thread closing. No more sessions (cookies on the client end, the entire session tool is disabled) or persistent database connections. CCE3 is back to being the light, efficient application I wanted it to be.

The last major explorer issue is redirecting requests during a orphan database rebuild. I could take the easy way out and set a default error page that states the explorer is rebuilding its database and to try the request again in a few minutes, but that seems kind of a messy way to do it as it would go to that page regardless of the error. However, in the interest of getting the time table back on track, I may have to go that route and patch a more elegant solution after release.

API:

API solution is next on the table. The developers message box access is going to be included in the API as it seems the appropriate place for it.

The API is going to be a separate module from the web server. I feel this is the best way to do this for two reasons.

The API module can be streamlined into handling JSON request/response and Long Polling without all the HTML baggage.

When the need arises I can shut down web services and API services separately as needed. Since other sites depend on the API, if there is a problem with a web server, hopefully I can keep the API up while working on the web server issue. This of course works both ways, no need to take down the web server for an API issue.

Work on the API will begin tonight or tomorrow.

If things go smooth      I can see release being on time in the middle of this month.

[dreamwatcher]

BottleCaps has been added to the CCE3 test server:

http://cap.cryptocoinexplorer.com

[dreamwatcher]

I will be placing i0coin on cryptocoinexplorer.com as one of the coins testing CCE3. This means it is a permanent [art of CCE (as long as it stays active), but to be introduced under the CCE3 test program.

Address will follow the normal CCE address scheme:

i0c.cryptocoinexplorer.com

i0coin has a couple year old block chain, so it will take some time for for the initial database to be built, more than likely overnight and even into tomorrow.

I will probably turn on the web server before the database is complete, but I will post here when complete.

[dreamwatcher]

It looks like it will be least a day or two before before the entire i0coin chain (over 900K blocks!!) is fully analyzed and CCE3 has made a complete database.It might be quicker as it hits the range where the coin lost popularity and there is only coinbase transactions.

I will open up the web server later today, though I doubt the database will complete.

[dreamwatcher]

To the site-developers out there that would be using the API/Long poll system in CCE3:

How do you want the long poll to work?

As of now it looks like the long poll system will be completely separate from CCE3, it could even be used with the current ABE explorers.

How would you like to handle the variety of coins? Do you have a separate URL for each coin to poll too, or one address and have a JSON field giving the 3 letter code the coin that has polled?

Would just a simple poll with the coin code be enough, or would you like the poll system to push the block information also?

The issue I see here is it does take at least a few seconds for the database loader to process block and transactions information before it becomes available on the explorer and API.

If the Long Poll pushes the block, it will get the block information directly from the daemon and just pass on the RPC response from the daemon.

[dreamwatcher]

Looks cool. How about you add one for CGB?

Once CCE3 goes live, I will be adding the majority of coins to CCE.

However, I am slowly adding coins to the test server in an attempt to stress test and see how many coins a server can comfortably serve under cce3.

I will consider adding CGB as one of the test coins when I get ready to add another.

[ahmed_bodi]

To the site-developers out there that would be using the API/Long poll system in CCE3:

How do you want the long poll to work?

As of now it looks like the long poll system will be completely separate from CCE3, it could even be used with the current ABE explorers.

How would you like to handle the variety of coins? Do you have a separate URL for each coin to poll too, or one address and have a JSON field giving the 3 letter code the coin that has polled?

Would just a simple poll with the coin code be enough, or would you like the poll system to push the block information also?

The issue I see here is it does take at least a few seconds for the database loader to process block and transactions information before it becomes available on the explorer and API.

If the Long Poll pushes the block, it will get the block information directly from the daemon and just pass on the RPC response from the daemon.

thats a good idea and something i would use if u implement it. The Small delay would be negligable imo. Even the long polling by miners (when notify scripts arent used on a pool) has a small delay. IMO id prefer the method of having a seperate URL per coin.

[dreamwatcher]

Update on i0coin -  The database is still building, it is going much slower then any of the coins I have put on the test server so far. I am not quite sure why, unless the daemon is just a bit slower with RPC responses.

Developers:

What is your opinion on Ubuntu 13.10 server:

Ubuntu Server 13.10 is available from 17th October; first fully supported release of the new OpenStack Havana, with faster node installation and a new version of Juju that supports ultra-dense containerised application deployment.

http://insights.ubuntu.com/news/ubuntu-13-10-delivers-faster-cloud-setup-and-management-for-scale-out-environments/

[RoadTrain]

Good progress, just wanted to complain about colors and overall design.
Could you use something like twitter-bootstrap to build your layout on?

[dreamwatcher]

Good progress, just wanted to complain about colors and overall design.
Could you use something like twitter-bootstrap to build your layout on?

It has been awhile since I updated. So many fires on other projects 

There are going to be many changes in version 4.0:

1. I have found that orphans can be trickier then I thought. I have caught the explorers in endless loops of restore from what should be a orphan free backups.I am going to have to completely change the way the database is indexed, so individual blocks and all their related transactions can be updated individually without a complete database restore.I did not want to go to that level of complexity, but it appears it is going to be needed.

2. As long as I am making a major revision, I am going to change the web lay out to be completely HTML 5 compliant  and use a different kind of page template system. A new look is in store for version 4.

3 .I may write some C++ modules for certain functions to gain speed. I need to run some more tests to see if the bottlenecks in the system are due to the database or code,  but most likely I will start rewriting parts of the explorer in C++.

4. I am going to push the release estimate to Mid November. I want to put out a quality product, not just rush out a sub standard piece of software.

5. I will continue to add coins to the test project, as it is important to test along a variety of block chains and stress test the servers.

[dreamwatcher]

UPDATE!!!!!!

I am now currently installing test version 3.99 on the test server. The explorers will be coming back up as the new databases are created.

New in test version 3.99:

New database structure -  The new database structure is smaller and makes "hot-fixing" orphans possible.

Rewrite of much of the core logic - Core logic updated to use the new database  + addtional features listed below.

HOT_FIX OF ORPHAN BLOCKS - Every time the database loader is called, it will check the last number of blocks set in the configuration file. I have all the explorers set to 250 for now. If an orphan block is found, the loader will "Hot-Fix" the block in the database. No more dumping the database and reloading from backup!!! The web server will remain up during a hot-fix, as it only takes a second or two.

Automatic database backups every ~1000 blocks - Whenever the loader detects 1000+ blocks since the last backup, it will auto backup the database.

Major changes to the address ledger page - The ledger page now has two tables, one for credits and one for debits, this makes the ledger much more readable. Web page ledgers are limited to the last 1000 credit and 1000 debit transactions. Longer searches will be available through the API. Anything more then 1000 (Even that is a high number) is pretty much unusable while looking at a web page. Transactions are listed newest to oldest.

[mercSuey]

Looks cool. How about you add one for CGB?

Once CCE3 goes live, I will be adding the majority of coins to CCE.

However, I am slowly adding coins to the test server in an attempt to stress test and see how many coins a server can comfortably serve under cce3.

I will consider adding CGB as one of the test coins when I get ready to add another.

Thanks.

[dreamwatcher]

Update:

I think I have finally kicked the Db/user thread issue.

I have stopped using Cherrypy for the DB connection/user thread matching. It works well opening database connections as user threads start, but does not appear to have any way of closing the connection when the thread is closed. This was so unbelievable , it took me weeks to finally give in...

I am now using DButils persistent DB connections system and it appears to be working well. Database processes appear to be stable in number and well within limits.

This means I can move back to working on the API and the CCE3 project can move forward.

[zuper]

Update:

I think I have finally kicked the Db/user thread issue.

I have stopped using Cherrypy for the DB connection/user thread matching. It works well opening database connections as user threads start, but does not appear to have any way of closing the connection when the thread is closed. This was so unbelievable , it took me weeks to finally give in...

I am now using DButils persistent DB connections system and it appears to be working well. Database processes appear to be stable in number and well within limits.

This means I can move back to working on the API and the CCE3 project can move forward.

Great! Please consider to add DMD (Diamond Coin)

[Mad_Max]

How about add CHNCoin?
It actually already was on CCE but you threw it out this summer when the network almost died due to "Difficulty trap" (hight diff and slow retarget = miners exodus).
But ~1.5 month ago diff retarget algo was replaced (now PPC type - little diff retarget every block):
https://bitcointalk.org/index.php?topic=322488.0

And now (after update to new version) coin works good and active:
~ 500 MH/s active mining
traded on two exchanges at least and have descent market cap (~ 600 000 - 800 000 USD)

[ProfMac]

http://bte.cryptocoinexplorer.com/chain/Bytecoin is down.

[dreamwatcher]

http://bte.cryptocoinexplorer.com/chain/Bytecoin is down.

At this Time I have multiple VPS down. I will try to get everything back up ASAP.

[Byates]

Can you add VegasCoin?

[dreamwatcher]

Cryptocoinexplorer.com will be converting to CCE3 over the weekend.

Please see the thread at:

https://bitcointalk.org/index.php?topic=576209.0