[UPDATES] CryptoCoin Explorer III Development Updates (CryptoCoinExplorer.com)

[dreamwatcher]

CCE3 TEST VERSION 3.5  IS ONLINE FOR UFC, KED and GPL

Federation Credit (UFC)

http://ufc.cryptocoinexplorer.com

Klingon Darsek (KED)

http://ked.cryptocoinexplorer.com:

Gold Pressed Latinum (GPL)

http://gpl.cryptocoinexplorer.com

The server change is due to the need to test the new server environment in the near future.I cannot do this on the current live coin servers. The port addresses are used for now, but once the server environment is ready for testing the addresses will change to the permanent CCE address system (ufc.cryptocoinexplorer.com ... etc) I will post when that is ready.

[dreamwatcher]

CCE3 Test Version 3.5 is up and running with Goldcoin.

Gold Coin:
http://gold.cryptocoinexplorer.com

Primecoin
xpm.cryptocoinexplorer.com

or

www.xpm.cryptocoinexplorer.com

Federation Credits
http://ufc.cryptocoinexplorer.com

Klingon Darsek
http://ked.cryptocoinexplorer.com

Gold Pressed latinum
http://gpl.cryptocoinexplorer.com


All except Primecoin are running test version 3.51. Primecoin will be on 3.51 as soon as the 3.51 database for Primecoin is populated.


She isn't all that pretty right now, but this is a test of logic and function. The pretty interface and extra functions are the next test version release.

[dreamwatcher]

All the test explorers are running in version 3.51.

A few changes in 3.51.

The ledger balance now always matches in the total balance and the final balance at the end of the ledger.

The transaction date/time is displayed next to the transactions in the ledger.

[dreamwatcher]

The first of the visual updates is up on the Primecoin Explorer.

xpm.cryptocoinexplorer.com

or

www.xpm.cryptocoinexplorer.com



I have a question:

Do you prefer the ABE style block view, like what is on the test explorer now or would you prefer the list style used on Blockchain Info?

http://blockchain.info/block-index/415691/000000000000001f2225ba979095ddde6c3ea479608a34cbc2a4286dfbd23286

[dreamwatcher]

The block explorers are back up. The reason for the issues  continues to be orphan blocks. This issue has plagued me since starting the CCE3 project.

Now for a little rant:

I have asked over and over for developer input on how to handle orphans. Even something as simple as an opinion on how long the daemon takes to realize it has an orphan block, correct the information, and start reporting the correct information.

Know what I got:

Crickets. Not a single answer from ANY developer. In fact, I have not gotten ANY advice from any developer. Sometimes makes me wonder why I am bothering. I put all this effort, time and money into the project and seems like nobody gives a crap.

CCE has been around a little over a year now, so the excuse " I have not heard of it" is a complete garbage.

--End Rant--

[dreamwatcher]

The DB load module then queries the daemon for all the block and transaction information for blocks not in the explorers' database.

The issue arises when the daemon gives the db loader information about an orphan block because it has not realized it has an orphan block stored.

Two things...

First, before the Load Module requests to have the transactions fetched... run the query to fetch all non greater than zero output txids, and loop deletetransaction <txid> on the array... then restart daemon and upon restart and sync, then the Load Module can fetch the good blocks. I see the only issue here being how often this would run depending on if you have a ton of orphans... maybe a test to not run the array and deletion with restart if no zero outputs detected?

Second, given it is an orphan, that means you are seeing TWO blocks with the same block number being brought in by the Load Module? So, a line to test when it sees duplicate block numbers and drop the zero output one? Or maybe instead, a test to find zero output txids and have them in an array that the Load Module references as it fetches so it knows before hand that XXX txid is trash?

Come to think of it... would it not be just as easy to make a seperate module just for this and call it before the Load Module? Have this module test for orphans based on zero ouput, delete them and restart daemon if there is, then run Load module, or if there are none, go on its way?

A basic run down of the DB loader logic:

1. Initialization of DB connection and loading of variables needed for RPC communication with the daemon.
2. Query the daemon for the latest block count.
3. Query the database for the greatest block height stored.
4. Begin the loop for the difference
5. Load the getblock information for the current block in the loop into the database
6. Get the transactions listed in the block info and sort,index and store.This includes all the accounting and transaction index/address matching.
7. Repeat until loop is finished
8. Update the Utility table with the information you see on the homepage.
9. Commit and close database connections

The loader will not see two blocks, it only sees the block it queries the daemon for. The daemon will not give two blocks out, only the block it believes is the official block, which can be an orphan block until it learns otherwise.

I guess one of my questions would be: If the answer is watching for non-zero outputs, why is the daemon giving out orphan block information in the first place? Why does the daemon not see the non-zero outputs and query the network for the correct block information? If the daemon is not checking for this, I see this as a HUGE flaw in the coin daemon design.

The simpler answer is to add a delay between when the loader gets the block height information and when it begins processing the blocks. This gives the daemon a chance to catch the orphan block and correct the information. My question is..How long?

The other solution would be to run a check on the previous stored block in the database and see if the block hash in the database matches the block hash the daemon is reporting on that block. I certainly hope by the next block the daemon has correct information, otherwise it will reject future blocks. The previous hash is part of the header, so if the daemon thinks the previous orphan block is the official block, it will see all future blocks as invalid until it corrects the orphan block. I have some filled, but unused, fields in the database to support this function. However my problem with this is that incorrect block information can appear on the explorer until the next block is processed and the previous block corrected. There is also the issue of fast block times, where processing block information takes longer then the targeted block time. The loader then processes 2 or 3 blocks at a time.

A simpler version of the above , is simply to have the explorer permanently one block behind, though this may not work on the fast coins.

[dreamwatcher]

I have started to test the new server environment.

As a result The Primecoin explorer had Moved to its permanent home:

xpm.cryptocoinexplorer.com

or

www.xpm.cryptocoinexplorer.com

[daemonfox]

I am curious if instead of it triggering when it sees a solved block... it can instead be triggered when the next block has all the new data to begin taking hashes for solutions? Would that ensure you are fetching the previous valid block?

[daemonfox]

I must ask... is the wallet daemon you are using MINING? How do you see orphans in the block chain... I was under the impression orphans do not get broadcasted, local clients only learn they are orphans when the daemon finds the block stats do not match what is in the chain after that local client found what it though was a valid block.

I may be missing something here... so please feel free to fill me in, this whole topic has me interested in how this works now.

[dreamwatcher]

I must ask... is the wallet daemon you are using MINING? How do you see orphans in the block chain... I was under the impression orphans do not get broadcasted, local clients only learn they are orphans when the daemon finds the block stats do not match what is in the chain after that local client found what it though was a valid block.

I may be missing something here... so please feel free to fill me in, this whole topic has me interested in how this works now.

None of the daemons on any of the CCE servers are mining.

Orphans will be broadcast from the originating source, after all that daemon does not know the block it has produced is an orphan until it learns otherwise. Any of the daemons it is connected to that do not have the correct block with the earlier time stamp, will also assume the orphan block is correct until they learn otherwise and broadcast it to whatever nodes they are connected to.
I do not know how the qt-clients handle it. I always thought the qt-clients were just the daemon with a GUI. However, that is not relevant to CCE as I do not run qt-clients , only daemons.

ABE based explorers just store everything and will show the orphan blocks in the explorer. However, the ABE based explorers on CCE do not show orphans as I have modified the code a bit to block them showing on the web page. If you go to just about any other ABE based explorer, you will sometimes see extra hashes on the block page. These are the hashes of orphaned blocks it has encountered. The only orphans that an individual daemon will see will be the ones it created or were broadcast to it before it knew of the earlier produced block.

[daemonfox]

The other solution would be to run a check on the previous stored block in the database and see if the block hash in the database matches the block hash the daemon is reporting on that block. I certainly hope by the next block the daemon has correct information, otherwise it will reject future blocks.

I was thinking more on this today...

This is your answer... each time the module reaches out to get the new block height, it also needs to have an integrity check to confirm all existing blocks in the DB match the chain at that time, and if it doesnt, then the range from the current block height to the furthest should be adjusted to take in the mismatched block that is orphaned, after dropping the orphaned record rom the DB of course.

That seems to be the proper way to do it... audit the data to ensure integrity and clean it up before the call to import fresh blocks.

That should solve your issue once and for all.

So, in generic coding english...

Call the Load Module
Query the last block height in the DB and the last 5 blocks before it, giving you a sample of 6 blocks to test (redundant and can probably be just 2 blocks)
Have that loaded into an array with each DB record's hashes and data
Query the daemon for the data of the blocks in your array and test them
If the blocks pass, the current Load Module can proceed, if there is an orphan mismatch found, then trigger a deletion on that block record set
Now the Load Module can test for the new DB block height and proceed as usual.

Sounds simple enough? Hope I have helped you.

[dreamwatcher]

I agree with the basic premise.

It looks more and more like the best solution is to back check block hashes and rollback when a mismatch is found.

The thing I do not like about it, is when an orphan blocks appears, it will show up on the explorer until the back check catches it.

I am going to try one more thing first, a long delay between when the module gets the block range to fetch and when it fetches the block info. Right now I have it set at 13 seconds. I think I will try a 30-45 second delay.

daemonfox, I appreciate your interest in helping this project out. So far, you have been the only person to take a serious interest in helping me. It has been kinda of a tough couple of months when it comes to dealing with other developers. I tend to get tossed to the sidelines after helping with coin launches, pools and everything else I try to help with. Quite honestly, some of the behavior I have seen (unrelated to any of my projects) by some of the new developers coming on scene, is disappointing.

The plans for CCE3 are quite large, and there is a surprise or two coming with test version 5. As of now, at least one large scale project outside of CCE3, is going to be very involved in version 5 (API).

The ultimate goal is to have at least the vast majority of coins of CCE. Right now I believe the number of alt-coins hovers around 140.

[dreamwatcher]

Newest Update:

I have most likely solved the orphan issue with two updates.

1. I have increased the wait time between when the database loader gets the block height information and when it starts querying the daemon for block and transaction information to 30 seconds.

2. I have built a database integrity check module that will run on a cron tab. I am thinking every couple of hours at first.

    a. The module will check the block hash stored in the database with a current call to the daemon for the block hash. It will get the range of blocks from the last_check field in the utility table and the highest block    stored in the database.

    b. If all the hashes match it will store the last block it checked in the utility table and execute a complete database dump (backup)

    c. If there is a hash that does not match the module will exit and note the bad hash in the debug log.

In the future I plan to automate recovery. If a bad hash is found the module will shut down the coin daemon (to avoid a clash between the dbloader and the integrity check module). The module will then clear the database and reload the database from the last backup dump. Activate the coin daemon and close out.

I will be adding this module to the test server soon.

If this issue is solved, the testing versions should be coming faster, as this issue is what has slowed down progress so far.

[daemonfox]

Good work Dream... progress is fun.

curious though...  why an hour on the check... blockss that are orphaned should never be that far back... and the load module triggers way more often then that right?

I would think, as you said before, that by the NEXT block, an orphan is fully detected and gone from the blockchain. So, would that not make it pointless to wait an hour to run the check?

I had another thought on this as I was reading more about the MEC happenings this weekend... for this orphan issue, would it not solve EVERYTHING if you changed the code to always load the new blocks detected up to but NOT INCLUDING the most recent solved? As long as the chain is broadcasting timely enought, the premise of the second to last block NEVER being an orphan should always be true... or are you finding otherwise? It would be interesting to see the first few days worth of logs from your integrity module.

So, maybe always regarding the newest block as "immature" and only processing up to the block before that is a good model as well... but both that idea, and what you have put together are solutions.

Daemon

[sudo23]

Hello dreamwatcher,
could you add COL (CollossusCoin) to your CryptoCoin Explorer? I like your Website!

https://bitcointalk.org/index.php?topic=279601.0
Source: https://github.com/muddafudda/Infinitecoin-V2
Thank you!
Sudo

[dreamwatcher]

Good work Dream... progress is fun.

curious though...  why an hour on the check... blocks that are orphaned should never be that far back... and the load module triggers way more often then that right?

I would think, as you said before, that by the NEXT block, an orphan is fully detected and gone from the blockchain. So, would that not make it pointless to wait an hour to run the check?

I had another thought on this as I was reading more about the MEC happenings this weekend... for this orphan issue, would it not solve EVERYTHING if you changed the code to always load the new blocks detected up to but NOT INCLUDING the most recent solved? As long as the chain is broadcasting timely enought, the premise of the second to last block NEVER being an orphan should always be true... or are you finding otherwise? It would be interesting to see the first few days worth of logs from your integrity module.

So, maybe always regarding the newest block as "immature" and only processing up to the block before that is a good model as well... but both that idea, and what you have put together are solutions.

Daemon

The integrity check takes care of two things. Checking to make sure there are no orphan blocks in the database and creating known clean database backups. IF CCE3 goes according to plan, not only will the current coins on cryptocoinexplorer.com be converted to CCE3, but I will also be adding as many coins as possible to CCE. This count could reach hundreds, and I needed an automated database backup system in place because manually backing up that many databases would be almost impossible.

There is also the issue of fast block chains (under 1 minute). Even without the delay, the dbloader can take more time then the block target time to process a block or blocks (in the case of fast chains). One of the basic design principals is to do all the indexing and sorting on the back end. The web server only has to make very short shallow searches into the database to retrieve the information. Every address(account) has its' own tx index which records every transaction that address is involved in. There is also some basic accounting happening and being recorded. This only needs to be done once at the back end, whereas if the web server did more deeper searches and sorting, it would have do it every time information is requested. This would be repetitive work that would slow down the user experience.

I have observed that catching orphan blocks is not particularly effected by block target times, but is more a factor of how many hops away correct block info is from the daemon. So in the case of fast chains, it may be a few blocks before the orphan is discovered and the daemon reorders its' database.

Performing a previous block check would be easy and add nothing to the processing time. For example, the integrity check module can process the entire Primecoin chain in just a few minutes, whereas a complete database build from block 0  takes a few hours. The integrity check only checks from the height of its last check, so it only takes a few seconds to scan. Most of the time is spent in the database dump, which can take about a minute for the larger chains. So, I may incorporate a previous block check as an extra check. I also need to have the dbloader recognize when it catches a orphan block (which is pretty easy to do as the error is pretty specific) and trigger the integrity check to clear and reload the database from the last backup. I prefer this method of fixing a database with an orphan block as it guarantees the data is orphan free. Whereas a rollback becomes more complicated and prone to errors.

[dreamwatcher]

The CCE3 test has moved to test version 3.98!!

The Primecoin explorer is now running 3.98. The other explorers should be switched over soon.

New in this version:

Complete database integrity and automated recovery system
Complete removal of all per coin custom coding in the modules.
General clean-up and tighter code. Started basic code comment documentation.

***Database integrity and automated recovery system****

I believe I have finally devised a system to squash those orphan critters.

1. There is now a 30 second delay between block height acquisition and querying the daemon for block/transaction information.
So far it looks to have improved the situation greatly, all the test explorers have been running that way for a couple of days. The only orphan to muck up the system was in the Primecoin explorer this afternoon.
This is OK, because the Primecoin test explorer is now running with 3.98 and it can now detect and squash those dam orphans with extreme prejudice.  

2.At first I was going to add a third module to perform integrity checks, but have since incorporated it into the dbloader module. The integrity check will not be on a cron tab, but rather will run every so many blocks configurable in the cce3 config file.
The integrity check compares the block hashes in the database (from the last check/back-up to present database height) to what the daemon currently reports the hash to be. If a non matching hash is found, it then triggers the orphan killer function..err..automated recovery function.  
The recovery function drops and recreates the database with the last backup. The database backup is always clean as the integrity function only dumps a database backup if the database has all matching hashes to what the daemon currently reports.

The transaction function can also call the orphan killer..I mean recovery function, if a transaction input has a prev_out tx that does not exist (A transaction that is part of an orphan block).

The other advantage to this system is always having recent database backup, depending on how many blocks are set in the config file.


*Complete removal of all per coin custom coding in the modules.*

All custom code and coin parameters have been moved to the database. dbloader and the web server are now completely generic modules that can run any coin (In theory   ) without customization. The config file consists of four lines.

database name
database user
database password
Block count (Determines how often the integrity check is run)

Everything else is set in the utility table of the database.

Development should start becoming faster as the logic coding is pretty much finished and in late beta.

Next up:

One last major bug to squash. This is not a logic or coding bug, but rather a server set-up issue. If somebody requests a address ledger with a large amount of transactions (1000's) and then navigates away before the ledger shows up, the server gives a 107 error and basically needs to be killed by a level 9 kill. If not killed it will not serve any more connections and wait forever for the session to connect.
The ledger is very fast and only takes more then a second or two when dealing with extremely long ledgers. Even then it is rarely longer then 10 seconds or so.

Developers message box. The framework is in place and it should be ready with version 4. I will give access to this system as soon as it is ready to the developers whose coins are currently on the test server.

I still have to decide on how I want to handle market tickers. Each market has its own API and it looks like it could be a pain to accommodate. Perhaps I will need to get the information from some of the coin valuation sites that use CCE as an information source.

A few technical specifications on the project so far:

CCE3 consists of two python modules (dbloader and webserver), one config file and a database with 6 tables ranging from 6 columns to 33 columns (utility, which only consists of one row). There are also the normal web files: css, robots.txt and icon files.

The two python modules have a combined total of 865 lines of code(including white space). I set out to make CCE3 light and efficient and hope I have accomplished this goal.

Web serving is based on CherryPy and Apache2 using the mod_rewrite module. Apache serves the static files, while Cherrypy serves the dynamic content. More then likely this set-up will change, with CherryPy still being the prominent web serving system.


As I head into version 4 and look at version 5, I need site admins to start giving me thought on how you would the the API to work. Version 5 is all about the API.

The development cycle on Version 4 is expected to be short, so do not miss out on your chance to help shape CCE3 API.

[lukemarshall]

Great work Dreamwatcher...

It is a pleasure to have a repository of block information at our fingertips.

[dreamwatcher]

All test explorers are now running test version 3.98

[dreamwatcher]

Update 9/27/13:

I have run into a couple of snags and I have spent a week unsuccessfully trying to solve them.

1. If somebody requests a large ledger, two things happen.
    A. All other requests (threads) will be delayed until that request is fulfilled.The Cherrypy server is a multi-threaded server and I have upped  the default value of number of threads.
    B. IF the above user navigates away from the ledger page before the request is fulfilled, the server will wait forever for the request to be filled and not process any more requests(threads). It will not check to see if the socket is closed, only wait for a response.


2. The second issue is having the webserver module switch to a "Please try your request later" type page during a database reconstruction. The obvious methods to do this do not work. Lock files and system signals do not work as the Cherrypy server will ignore them after the first time it checks them during start up. The third solution, which I have not had time to try, is to simply set up a database with one table and two columns. One for the coin code and the other a flag for when database reconstruction is taking place on that coin. The issue I have with this solution is that it doubles the amount of database connections.

I have to decide whether or not to start from scratch with the server framework or continue to try and work this issue out. I may need to ask other developers to help out and make the code available to them. I will continue to work through the weekend to figure out these issues. The release may be delayed if i do not find solutions to these issues soon.