Bitcoin Forum

CryptoCoinExplorer (http://www.cryptocoinexplorer.com)


The CCE3 test has moved to test version 3.98!!

https://bitcointalk.org/index.php?topic=262539.msg3177164#msg3177164

All test explorers are now running test version 3.98

New in this version:

Complete database integrity and automated recovery system
Complete removal of all per coin custom coding in the modules.
General clean-up and tighter code. Started basic code comment documentation.

Primecoin
http://xpm.cryptocoinexplorer.com

Goldcoin
http://gold.cryptocoinexplorer.com

Federation Credits
http://ufc.cryptocoinexplorer.com

Klingon Darsek
http://ked.cryptocoinexplorer.com - Temporary Down until KED relauch

Gold Pressed latinum
http://gpl.cryptocoinexplorer.com

Cthulhu Offering
http://off.cryptocoinexplorer.com

BottleCaps

http://cap.cryptocoinexplorer.com

Cubits

http://qbt.cryptocoinexplorer.com




I believe a separate thread is needed for the development of CCE III, as it is a large project not only in the sense of CCE, but as a new development tool for other developers.

CryptoCoin Explorer III  Update:

History:

CCE started as the PPC explorer when PPC was released in August 2012.
Note: About three weeks until the PPC one year anniversary August 16!

Terracoin and Freicoin followed as they were released.

The CryptoCoin Explorer site came online November 14, 2012

Novacoin, BBQ, Feathercoin and Bytecoin became part of CCE.

Since this April: Bitbar, Digitalcoin, CHNCoin, JKC, Franko, Goldcoin, Worldcoin and Craftcoin have been added.

CCE is now spread across two decent sized VPS.

CCE has a Alexa global rank of  280,031. Although I do not have any detailed metrics installed on the site, I have seen estimates of 5000 – 10000 hits a day. The site sends out about 10GB of data a day over both servers.

A couple of months ago I updated the look and feel of CCE which I called CCE 2.

About a month ago I decided to write completely new explorer software from scratch.  There were a few additions to the API being worked on and the bigger issue of resources versus the number of coins coming on the market. I also wanted to provide a one stop site for coin information as it became harder to search and find information for all the new coins.

It became clear shortly after I put the first few explorers online, CCE became a trusted source for block chain information. I take that responsibility seriously,  and part of the reasons for the rewrite is to give the community an even larger source of information in both number of coins and quantity of information. I also want both new and experienced alt-coin users to have a place to go and get links to official sources of information.

CryptoCoin Explorer III:

Current goals:

1. Lighter and faster implementation of block explorer.

2. Coin homepages acting an information center for the coins. Each homepage will have:
   A. Network Hash Rate (Hourly, Daily, and Weekly where possible)
   B. Basic information on the last five blocks
   C. Top 5 account balances – In the new explorer, balances for accounts are updated every block.This means no more restrictions on how many transactions an account can have to be displayed
   D.  Possibility Market tickers (Volume, Top 5 prices with depth from 2 exchanges)
   E.   Basic coin info (Official website, forum, source, bitcointalk.org thread)
   F. Technical info (RPC and P2P ports, address version, network prefix, target times, etc.)

3. Time localization options. Explorers can be set to display times in a particular time zone.

4. Developers message box. This would allow developers to place announcements and messages on their coins' CCE home page.

5. Subscription based polled API support. Other site administrators would be able to subscribe for API information that will work much like LP works in mining. Possibility with some customization as to the information they want sent.

Open versus Closed source:

Eventually part of the new explorer will become open source. I am writing a tool system that can be used to create other applications that depend on block chain information. This system will gather data from the coin daemon, populate a database and provide a basic query system. It will be adaptable to any information one can get from the coin daemon and its configuration will almost be completely database driven. The configuration file will only need to point the system to the database it is to work with.
The ultimate goal will automatically sort the information keys it receives from the daemon based on column names and disregard keys that do not have columns. The system written so far does work that way, but code customization is still needed at this point for embedded objects in the JSON output of the daemon (Embedded lists for example. Python sees them as a list object inside a Dict. object.)

The final system will let anybody setup a customized block chain database by simply setting up tables and columns. A configuration table would tell it what commands to issue and in which tables to sort and store the results. I also believe this table will be where one defines the objects within objects stored in the key values it gets from the daemon.

One could build any alt-coin application from it, including of course a block explorer.  I will be keeping the CCE block explorer application of this tool private. Though I will probably provide a basic block explorer example in the source.


And so on.....

CCE is not run by a team of developers. I am the sole developer of CCE. Although CCE and this new project are quite large projects I have taken on, I believe it will benefit the alt-coin community greatly.


This also means, aside from donations, I am the sole financial support for CCE. I appreciate all the donations received and I am not writing this to beg for more donations (though they would be extremely helpful). Donations fall way short of costs in both servers and my time.
I have tried to acquire funding through the opening of pools (cryptocoinmine.com), but the pools have ended up losing money rather then gaining it. I am going to shut down CryptoCoin mine completely except for the Goldcoin pool as the Goldcoin community has shown outstanding support for the pool. I may open a Primecoin pool at a later date after the parameters of how such a pool would work best is established.

This means I am going to move to advertising on the new explorers. In the past I have been very much against it, but I see no other choice at this time. I will keep them them as small and unobtrusive as possible and my plan is only one advertisement banner at the bottom of each page. I have made no decisions on who I am going to use for advertising, but it is coming in CCE III.

I will setup a website mock-up of the look of CCE III sometime in the near future. Until then, back to the world of getting coin daemons, python applications and databases to play nice with each other.  ???  :D

these are really very good ideas you want to implement!! i like it!
please provide an good API (better than the one from abe) for getting data from each coin. i would definitely pay for such a service because i want to use some data on www.cryptocoincharts.info for my new version 2 ;)

Quick update:

The database propagation and update system is finished with the alpha stage and will move to beta as soon as the web server application is ready for beta.

I am working on the web server now, so it will not be long before a beta version is up and live on the internet.

The beta will be a PrimeCoin explorer. The first beta will be an explorer only with no API. I feel it is better to get the bugs out of the general explorer before adding the API to narrow down the troubleshooting as the API for the most part is a JSON extension of the explorer.

The web server is written around CherryPY. This means the Beta will be a port address as I do not want to deal with Apache/CGI wrappers. I would rather troubleshoot head on and not discover issues later when Apache is removed from the servers and CCE goes full CherryPy.

I am hoping to have it up before the end of the week, but I am hopeful it will be up within a couple of days. The web server development has hit its stride and is moving along smoothly.

http://cryptocoinexplorer2.com:5555/

I have put up a live alpha/beta of Explorer III.

This is a PrimeCoin Explorer.

-At this stage only the basic explorer is online, none of the extra features are active yet.

-Search function is not implemented yet, however the URL scheme is simple, and searching by hand quite easy.

-There are a couple of known bugs - for example the transactions on some blocks do not show correctly.

-Spent output field in the transaction page is not implemented yet

This is strictly a testing stage, so expect the explorer to be up and down as bugs are fixed and features are added.

Phase two of testing the new explorer design has begun.

http://cryptocoinexplorer2.com:5555

This version has all (I Hope  :D),the logic issues from test version one fixed and working for the basic explorer.

The address page is still limited to the running balance of the address. However, test version 3 is already well under way and it will have the full transaction history of the address available along with the "spent output' field on the transaction page. The search system should be in place (At most by test version 3.5), but manual searching can be done by simply observing the url addresses a bit and placing your own values in place of the explorer generated values in the url.

Test version 4 is slated to incorporate most of the 'frills' features on the Homepage of the explorer.

Test version 5 will be in the incorporation of the API.

I checked out your new explorer.  It's very smooth and responsive.   I didn't play with the API, I assume it's similar to Abe defaults?

The only thing from a user perspective that I didn't find great, is the design.  The colors really remind of a 90s or early 2000 website.  I would suggest trying something a little more engaging.  Some sites that come to mind our the layouts of blockchain.info or cryptonit.net -- the white with lighter blue seems to be the more modern look these days.  If you prefer darker colors, I think that cryptocoincharts.info looks amazing with it's Holo style.

Just some feedback, great site -- I heavily rely on your service and it's really appreciated!

The API is not active yet, we are still a few stages away from testing it.

You are right about the color scheme, I thought it needed some tweaking myself, so I have redone some of the theme with lighter colors and plan to add a Nav. bar across the top of the pages.





New testing announcement:

I will be parallel testing the CCE explorer rewrite with Goldcoin along with the current testing with Primecoin starting a bit later today. I am generating the database now and when ready I will upload everything to the CCE server. I will then provide the address for using(testing) the Goldcoin version of the new explorer code.

The current ABE based Goldcoin explorer will remain running unchanged until all the testing is complete and all of CCE gets converted.

Just remember, these are TEST versions and are subject to go down, crash or be inaccurate. Although inaccuracy is unlikely as the first two stages of testing have weeded out the few serious logic issues in the basic explorer. One always has the current ABE based explorer(GLD) or Block Crawlers(XPM) to compare to.  :D

The Goldcoin test explorer is up and running!  :D

http://cryptocoinexplorer2.com:7777

Remember both of these are test versions, once complete they will look complete, unlike now  ;D

I have a question for coin developers:

How long does it take the daemon to realize it has an orphan block in its chain and insert the correct block?

I have run into an issue where the daemon will trip a database update through block notify and the db update module will query the daemon for the block information and get an orphan block.

The solution I want to use is to simply back compare the block hash stored in the database to the daemon response to getblockhash a second time. If they are not the same update the database entry for that block and related transactions with the new accurate information.

I could simply back check every update, but that will leave false block information on the block explorer until the next block is solved.

Is the correction quicker? Could I back check right after entering the block information, or could the routine request a block hash, wait a couple of seconds, and request again before updating the database?

The only problem I see to the later is the super fast coins (10-25 second target times).

Quick update:

Many things have been going on behind the scenes. Mosty the boring kind of stuff nobody sees... :D

I have greatly optimized the web sever code, basically cutting the number of lines of code in half. At the same time, I also incorporated techniques to lock down against SQL injection. This paves the way to publishing search pages.
The optimized web server code is now up. I am not going to consider this a version update as no features have been added.

The database has also been improved to allow for the home page features.

I am still a bit on the fence about what to do about orphan blocks, hopefully I can finally sync up with some of the coin developers and get a discussion going. I sometimes keep some strange hours, so it can be tough to meet on or offline.

For now the plan is to check the previous block hash in the database with the daemons record of the previous hash every update cycle (every block).

The history of the issue involves "phantom" transactions that cause database updater to error out unable to find the orphaned transaction. Further debugging found that orphan blocks were causing it.

Test Version 3 is live for Primecoin.

Besides many optimizations and behind the scenes improvements, the SEARCH QUERIES ARE UP!  :)

Left to do before Version 3.5:

- Finish the ledger interface to back trace and display the transactions associated with the account. As of now it only shows the current balance.
- Finish the 'Spent Output' display in the transaction page.

I have decided to try and solve the orphan block issue by placing a 20 second delay between Block Notify and when the database management module queries the daemon for the current block/transaction information.
I am hoping that 20 seconds is enough time for the daemon to discover it has an orphan block and store/report the correct information.

CCEIII still needs many visual improvements. These will come with version 4 as the extra features are rolled out.


I will work on getting the Goldcoin CCEIII test server up to version 3, and back up for testing ASAP.

Just finished a little make over on the color and table schemes.

Both the test explorers will be down for a bit while they catch up from the last database backup.

This issue was strictly dumbassery on my part, and not a bug in the test explorers.  :-X

Both test explorers are up and running.

Primecoin:

http://cryptocoinexplorer2.com:5555 (http://cryptocoinexplorer2.com:5555)

Goldcoin:

http://cryptocoinexplorer2.com:7777 (http://cryptocoinexplorer2.com:7777)

www.cryptocoinsnews.com/2013/08/22/cryptocoin-explorer-iii-development-updates/

I have set up block explorers for both UFC and KED.

They are being hosted on the beta test CCEIII platform.

the ABE based explorer would have required some re-working to use these coins. I am certainly not going to be modifying ABE at this point with the CCE3 project testing.

These coins also give me an unique testing stress that I needed. Because they are so young with low difficulty, there are plenty of orphans.

Orphans are one of the major things I am trying to test CCE3 logic with.

UFC:

http://www.cryptocoinexplorer.com:9222/ (http://www.cryptocoinexplorer.com:9222/)

KED::

http://cryptocoinexplorer3.com:9222/ (http://cryptocoinexplorer3.com:9222/)

The port addresses are needed until testing is over and I fully convert CryptoCoin Explorer over to CCE3 (which is based on Cherrypy rather then Apache)

Remember these are test explorers, so they may go offline or have bugs. Please report bugs to me.

Test version 3.4 of the Primecoin Explorer is online.

Two things to note:

Some of the fields have changed in the Home page and Block list. This is in preparation to move to a Blockchain Info general type of theme.

Account Ledger is activated, but with a couple of restrictions. The ledger will only show the last 10 transactions, and because of this only the credit or debit amount is shown. The balance field would be too confusing without the full ledger.

The search algorithm for the ledger is rather complex and needs some major optimization to be practical. I have a few ideas on how to accomplish this and it is the next major hurdle to overcome.

The spent output feature will also be added next. It requires another field in the database. The code is ready, it is just a matter of rebuilding the database with the new field used.

 

The test explorers have been moved to their own server to facilitate a complete sever setup. A complete setup and test cannot be done on servers currently hosting live coins on CCE.

Test Version 3.5 is a completely functional base coin explorer. Full ledgers and searches are available.

New to 3.5:

Fully functional and quick account ledger system.
If you experience a delay when getting a ledger, give it a few seconds, you will be amazed at how long some of these account ledgers are!!! Literally 1000's of transactions long.
There can be some minor differences in the balance at the end of the ledger and balance in the "address" bar.This involves some minor rounding differences and the balance at the end of the ledger is more precise.

Various other tweaks and optimizations.

There will be a 3.51, but it will not bring any new features. The 3.51 release is the final database system that will be universal. Base chain, POS chains and prime chains will no longer have separate database structures or dbloader/webserver modules as they do now. This system will appear first in the SCI-FI coins that will be brought online later in the weekend.


That being said:

Primecoin test server 3.5 is online!

http://cryptocoinexplorer4.com:2001 (http://cryptocoinexplorer4.com:2001)


Please note that the address will eventually be changed once the testing phase of the new server system is started. The address should then change to the normal CryptoCoin Explorer address system.

CCE3 TEST VERSION 3.5  IS ONLINE FOR UFC, KED and GPL

Federation Credit (UFC)

http://ufc.cryptocoinexplorer.com

Klingon Darsek (KED)

http://ked.cryptocoinexplorer.com:

Gold Pressed Latinum (GPL)

http://gpl.cryptocoinexplorer.com

The server change is due to the need to test the new server environment in the near future.I cannot do this on the current live coin servers. The port addresses are used for now, but once the server environment is ready for testing the addresses will change to the permanent CCE address system (ufc.cryptocoinexplorer.com ... etc) I will post when that is ready.

CCE3 Test Version 3.5 is up and running with Goldcoin.

Gold Coin:
http://gold.cryptocoinexplorer.com

Primecoin
xpm.cryptocoinexplorer.com (http://xpm.cryptocoinexplorer.com)

or

www.xpm.cryptocoinexplorer.com

Federation Credits
http://ufc.cryptocoinexplorer.com

Klingon Darsek
http://ked.cryptocoinexplorer.com

Gold Pressed latinum
http://gpl.cryptocoinexplorer.com


All except Primecoin are running test version 3.51. Primecoin will be on 3.51 as soon as the 3.51 database for Primecoin is populated.


She isn't all that pretty right now, but this is a test of logic and function. The pretty interface and extra functions are the next test version release.

All the test explorers are running in version 3.51.

A few changes in 3.51.

The ledger balance now always matches in the total balance and the final balance at the end of the ledger.

The transaction date/time is displayed next to the transactions in the ledger.

The first of the visual updates is up on the Primecoin Explorer.

xpm.cryptocoinexplorer.com (http://xpm.cryptocoinexplorer.com)

or

www.xpm.cryptocoinexplorer.com



I have a question:

Do you prefer the ABE style block view, like what is on the test explorer now or would you prefer the list style used on Blockchain Info?

http://blockchain.info/block-index/415691/000000000000001f2225ba979095ddde6c3ea479608a34cbc2a4286dfbd23286

The block explorers are back up. The reason for the issues  continues to be orphan blocks. This issue has plagued me since starting the CCE3 project.

Now for a little rant:

I have asked over and over for developer input on how to handle orphans. Even something as simple as an opinion on how long the daemon takes to realize it has an orphan block, correct the information, and start reporting the correct information.

Know what I got:

Crickets. Not a single answer from ANY developer. In fact, I have not gotten ANY advice from any developer. Sometimes makes me wonder why I am bothering. I put all this effort, time and money into the project and seems like nobody gives a crap.

CCE has been around a little over a year now, so the excuse " I have not heard of it" is a complete garbage.

--End Rant--

A basic run down of the DB loader logic:

1. Initialization of DB connection and loading of variables needed for RPC communication with the daemon.
2. Query the daemon for the latest block count.
3. Query the database for the greatest block height stored.
4. Begin the loop for the difference
5. Load the getblock information for the current block in the loop into the database
6. Get the transactions listed in the block info and sort,index and store.This includes all the accounting and transaction index/address matching.
7. Repeat until loop is finished
8. Update the Utility table with the information you see on the homepage.
9. Commit and close database connections

The loader will not see two blocks, it only sees the block it queries the daemon for. The daemon will not give two blocks out, only the block it believes is the official block, which can be an orphan block until it learns otherwise.

I guess one of my questions would be: If the answer is watching for non-zero outputs, why is the daemon giving out orphan block information in the first place? Why does the daemon not see the non-zero outputs and query the network for the correct block information? If the daemon is not checking for this, I see this as a HUGE flaw in the coin daemon design.

The simpler answer is to add a delay between when the loader gets the block height information and when it begins processing the blocks. This gives the daemon a chance to catch the orphan block and correct the information. My question is..How long?

The other solution would be to run a check on the previous stored block in the database and see if the block hash in the database matches the block hash the daemon is reporting on that block. I certainly hope by the next block the daemon has correct information, otherwise it will reject future blocks. The previous hash is part of the header, so if the daemon thinks the previous orphan block is the official block, it will see all future blocks as invalid until it corrects the orphan block. I have some filled, but unused, fields in the database to support this function. However my problem with this is that incorrect block information can appear on the explorer until the next block is processed and the previous block corrected. There is also the issue of fast block times, where processing block information takes longer then the targeted block time. The loader then processes 2 or 3 blocks at a time.

A simpler version of the above , is simply to have the explorer permanently one block behind, though this may not work on the fast coins.

I have started to test the new server environment.

As a result The Primecoin explorer had Moved to its permanent home:

xpm.cryptocoinexplorer.com (http://xpm.cryptocoinexplorer.com)

or

www.xpm.cryptocoinexplorer.com

I am curious if instead of it triggering when it sees a solved block... it can instead be triggered when the next block has all the new data to begin taking hashes for solutions? Would that ensure you are fetching the previous valid block?

I must ask... is the wallet daemon you are using MINING? How do you see orphans in the block chain... I was under the impression orphans do not get broadcasted, local clients only learn they are orphans when the daemon finds the block stats do not match what is in the chain after that local client found what it though was a valid block.

I may be missing something here... so please feel free to fill me in, this whole topic has me interested in how this works now.

None of the daemons on any of the CCE servers are mining.

Orphans will be broadcast from the originating source, after all that daemon does not know the block it has produced is an orphan until it learns otherwise. Any of the daemons it is connected to that do not have the correct block with the earlier time stamp, will also assume the orphan block is correct until they learn otherwise and broadcast it to whatever nodes they are connected to.
I do not know how the qt-clients handle it. I always thought the qt-clients were just the daemon with a GUI. However, that is not relevant to CCE as I do not run qt-clients , only daemons.

ABE based explorers just store everything and will show the orphan blocks in the explorer. However, the ABE based explorers on CCE do not show orphans as I have modified the code a bit to block them showing on the web page. If you go to just about any other ABE based explorer, you will sometimes see extra hashes on the block page. These are the hashes of orphaned blocks it has encountered. The only orphans that an individual daemon will see will be the ones it created or were broadcast to it before it knew of the earlier produced block.

I was thinking more on this today...

This is your answer... each time the module reaches out to get the new block height, it also needs to have an integrity check to confirm all existing blocks in the DB match the chain at that time, and if it doesnt, then the range from the current block height to the furthest should be adjusted to take in the mismatched block that is orphaned, after dropping the orphaned record rom the DB of course.

That seems to be the proper way to do it... audit the data to ensure integrity and clean it up before the call to import fresh blocks.

That should solve your issue once and for all.

So, in generic coding english...

Call the Load Module
Query the last block height in the DB and the last 5 blocks before it, giving you a sample of 6 blocks to test (redundant and can probably be just 2 blocks)
Have that loaded into an array with each DB record's hashes and data
Query the daemon for the data of the blocks in your array and test them
If the blocks pass, the current Load Module can proceed, if there is an orphan mismatch found, then trigger a deletion on that block record set
Now the Load Module can test for the new DB block height and proceed as usual.

Sounds simple enough? Hope I have helped you.

I agree with the basic premise.

It looks more and more like the best solution is to back check block hashes and rollback when a mismatch is found.

The thing I do not like about it, is when an orphan blocks appears, it will show up on the explorer until the back check catches it.

I am going to try one more thing first, a long delay between when the module gets the block range to fetch and when it fetches the block info. Right now I have it set at 13 seconds. I think I will try a 30-45 second delay.

daemonfox, I appreciate your interest in helping this project out. So far, you have been the only person to take a serious interest in helping me. It has been kinda of a tough couple of months when it comes to dealing with other developers. I tend to get tossed to the sidelines after helping with coin launches, pools and everything else I try to help with. Quite honestly, some of the behavior I have seen (unrelated to any of my projects) by some of the new developers coming on scene, is disappointing.

The plans for CCE3 are quite large, and there is a surprise or two coming with test version 5. As of now, at least one large scale project outside of CCE3, is going to be very involved in version 5 (API).

The ultimate goal is to have at least the vast majority of coins of CCE. Right now I believe the number of alt-coins hovers around 140.

Newest Update:

I have most likely solved the orphan issue with two updates.

1. I have increased the wait time between when the database loader gets the block height information and when it starts querying the daemon for block and transaction information to 30 seconds.

2. I have built a database integrity check module that will run on a cron tab. I am thinking every couple of hours at first.

    a. The module will check the block hash stored in the database with a current call to the daemon for the block hash. It will get the range of blocks from the last_check field in the utility table and the highest block    stored in the database.

    b. If all the hashes match it will store the last block it checked in the utility table and execute a complete database dump (backup)

    c. If there is a hash that does not match the module will exit and note the bad hash in the debug log.

In the future I plan to automate recovery. If a bad hash is found the module will shut down the coin daemon (to avoid a clash between the dbloader and the integrity check module). The module will then clear the database and reload the database from the last backup dump. Activate the coin daemon and close out.

I will be adding this module to the test server soon.

If this issue is solved, the testing versions should be coming faster, as this issue is what has slowed down progress so far.

Good work Dream... progress is fun.

curious though...  why an hour on the check... blockss that are orphaned should never be that far back... and the load module triggers way more often then that right?

I would think, as you said before, that by the NEXT block, an orphan is fully detected and gone from the blockchain. So, would that not make it pointless to wait an hour to run the check?

I had another thought on this as I was reading more about the MEC happenings this weekend... for this orphan issue, would it not solve EVERYTHING if you changed the code to always load the new blocks detected up to but NOT INCLUDING the most recent solved? As long as the chain is broadcasting timely enought, the premise of the second to last block NEVER being an orphan should always be true... or are you finding otherwise? It would be interesting to see the first few days worth of logs from your integrity module.

So, maybe always regarding the newest block as "immature" and only processing up to the block before that is a good model as well... but both that idea, and what you have put together are solutions.

Daemon

Hello dreamwatcher,
could you add COL (CollossusCoin) to your CryptoCoin Explorer? I like your Website!
http://s22.postimg.org/y4pn9qv35/Colossus_Coin3.png
https://bitcointalk.org/index.php?topic=279601.0 (https://bitcointalk.org/index.php?topic=279601.0)
Source: https://github.com/muddafudda/Infinitecoin-V2 (https://github.com/muddafudda/Infinitecoin-V2)
Thank you!
Sudo

The integrity check takes care of two things. Checking to make sure there are no orphan blocks in the database and creating known clean database backups. IF CCE3 goes according to plan, not only will the current coins on cryptocoinexplorer.com be converted to CCE3, but I will also be adding as many coins as possible to CCE. This count could reach hundreds, and I needed an automated database backup system in place because manually backing up that many databases would be almost impossible.

There is also the issue of fast block chains (under 1 minute). Even without the delay, the dbloader can take more time then the block target time to process a block or blocks (in the case of fast chains). One of the basic design principals is to do all the indexing and sorting on the back end. The web server only has to make very short shallow searches into the database to retrieve the information. Every address(account) has its' own tx index which records every transaction that address is involved in. There is also some basic accounting happening and being recorded. This only needs to be done once at the back end, whereas if the web server did more deeper searches and sorting, it would have do it every time information is requested. This would be repetitive work that would slow down the user experience.

I have observed that catching orphan blocks is not particularly effected by block target times, but is more a factor of how many hops away correct block info is from the daemon. So in the case of fast chains, it may be a few blocks before the orphan is discovered and the daemon reorders its' database.

Performing a previous block check would be easy and add nothing to the processing time. For example, the integrity check module can process the entire Primecoin chain in just a few minutes, whereas a complete database build from block 0  takes a few hours. The integrity check only checks from the height of its last check, so it only takes a few seconds to scan. Most of the time is spent in the database dump, which can take about a minute for the larger chains. So, I may incorporate a previous block check as an extra check. I also need to have the dbloader recognize when it catches a orphan block (which is pretty easy to do as the error is pretty specific) and trigger the integrity check to clear and reload the database from the last backup. I prefer this method of fixing a database with an orphan block as it guarantees the data is orphan free. Whereas a rollback becomes more complicated and prone to errors.

The CCE3 test has moved to test version 3.98!!

The Primecoin explorer is now running 3.98. The other explorers should be switched over soon.

New in this version:

Complete database integrity and automated recovery system
Complete removal of all per coin custom coding in the modules.
General clean-up and tighter code. Started basic code comment documentation.

***Database integrity and automated recovery system****

I believe I have finally devised a system to squash those orphan critters.

1. There is now a 30 second delay between block height acquisition and querying the daemon for block/transaction information.
So far it looks to have improved the situation greatly, all the test explorers have been running that way for a couple of days. The only orphan to muck up the system was in the Primecoin explorer this afternoon.
This is OK, because the Primecoin test explorer is now running with 3.98 and it can now detect and squash those dam orphans with extreme prejudice.  :D

2.At first I was going to add a third module to perform integrity checks, but have since incorporated it into the dbloader module. The integrity check will not be on a cron tab, but rather will run every so many blocks configurable in the cce3 config file.
The integrity check compares the block hashes in the database (from the last check/back-up to present database height) to what the daemon currently reports the hash to be. If a non matching hash is found, it then triggers the orphan killer function..err..automated recovery function.  ;D
The recovery function drops and recreates the database with the last backup. The database backup is always clean as the integrity function only dumps a database backup if the database has all matching hashes to what the daemon currently reports.

The transaction function can also call the orphan killer..I mean recovery function, if a transaction input has a prev_out tx that does not exist (A transaction that is part of an orphan block).

The other advantage to this system is always having recent database backup, depending on how many blocks are set in the config file.


*Complete removal of all per coin custom coding in the modules.*

All custom code and coin parameters have been moved to the database. dbloader and the web server are now completely generic modules that can run any coin (In theory  :) ) without customization. The config file consists of four lines.

database name
database user
database password
Block count (Determines how often the integrity check is run)

Everything else is set in the utility table of the database.

Development should start becoming faster as the logic coding is pretty much finished and in late beta.

Next up:

One last major bug to squash. This is not a logic or coding bug, but rather a server set-up issue. If somebody requests a address ledger with a large amount of transactions (1000's) and then navigates away before the ledger shows up, the server gives a 107 error and basically needs to be killed by a level 9 kill. If not killed it will not serve any more connections and wait forever for the session to connect.
The ledger is very fast and only takes more then a second or two when dealing with extremely long ledgers. Even then it is rarely longer then 10 seconds or so.

Developers message box. The framework is in place and it should be ready with version 4. I will give access to this system as soon as it is ready to the developers whose coins are currently on the test server.

I still have to decide on how I want to handle market tickers. Each market has its own API and it looks like it could be a pain to accommodate. Perhaps I will need to get the information from some of the coin valuation sites that use CCE as an information source.

A few technical specifications on the project so far:

CCE3 consists of two python modules (dbloader and webserver), one config file and a database with 6 tables ranging from 6 columns to 33 columns (utility, which only consists of one row). There are also the normal web files: css, robots.txt and icon files.

The two python modules have a combined total of 865 lines of code(including white space). I set out to make CCE3 light and efficient and hope I have accomplished this goal.

Web serving is based on CherryPy and Apache2 using the mod_rewrite module. Apache serves the static files, while Cherrypy serves the dynamic content. More then likely this set-up will change, with CherryPy still being the prominent web serving system.


As I head into version 4 and look at version 5, I need site admins to start giving me thought on how you would the the API to work. Version 5 is all about the API.

The development cycle on Version 4 is expected to be short, so do not miss out on your chance to help shape CCE3 API.

Great work Dreamwatcher...

It is a pleasure to have a repository of block information at our fingertips.

All test explorers are now running test version 3.98

Update 9/27/13:

I have run into a couple of snags and I have spent a week unsuccessfully trying to solve them.

1. If somebody requests a large ledger, two things happen.
    A. All other requests (threads) will be delayed until that request is fulfilled.The Cherrypy server is a multi-threaded server and I have upped  the default value of number of threads.
    B. IF the above user navigates away from the ledger page before the request is fulfilled, the server will wait forever for the request to be filled and not process any more requests(threads). It will not check to see if the socket is closed, only wait for a response.


2. The second issue is having the webserver module switch to a "Please try your request later" type page during a database reconstruction. The obvious methods to do this do not work. Lock files and system signals do not work as the Cherrypy server will ignore them after the first time it checks them during start up. The third solution, which I have not had time to try, is to simply set up a database with one table and two columns. One for the coin code and the other a flag for when database reconstruction is taking place on that coin. The issue I have with this solution is that it doubles the amount of database connections.

I have to decide whether or not to start from scratch with the server framework or continue to try and work this issue out. I may need to ask other developers to help out and make the code available to them. I will continue to work through the weekend to figure out these issues. The release may be delayed if i do not find solutions to these issues soon.

I may have solved the issue (1) with a combo of using a persistent DB wrapper and specifically forking ledger requests.

I have applied the changes to all the explorers.

Keeping my fingers crossed.  ;)

UPDATE 10/02/2013:

WHEW!!! What a roller-coaster the last few days. All kinds of bugs and reversions popped up, so the test explorers have been kind of screwy the last few days.

Fixed:

I finally tracked down the exact cause of the thread delay and lockup during a large ledger requests. I had been looking at the wrong part of the server function. The problem was not the threads themselves, but the database connection(s). The persistent database connection appeared to solve the issue, but caused many other issues that made me lose more hair off my already bald head.  :D
It was causing major problems with the dbloader, and the whole of CCE3 went sideways for a bit.

I found the solution within Cherrypy itself (It is amazing how flexible this framework is!!). The web server now issues each thread its own database connection upon creation and closes the connection upon thread closing. No more sessions (cookies on the client end, the entire session tool is disabled) or persistent database connections. CCE3 is back to being the light, efficient application I wanted it to be.

The last major explorer issue is redirecting requests during a orphan database rebuild. I could take the easy way out and set a default error page that states the explorer is rebuilding its database and to try the request again in a few minutes, but that seems kind of a messy way to do it as it would go to that page regardless of the error. However, in the interest of getting the time table back on track, I may have to go that route and patch a more elegant solution after release.

API:

API solution is next on the table. The developers message box access is going to be included in the API as it seems the appropriate place for it.

The API is going to be a separate module from the web server. I feel this is the best way to do this for two reasons.

The API module can be streamlined into handling JSON request/response and Long Polling without all the HTML baggage.

When the need arises I can shut down web services and API services separately as needed. Since other sites depend on the API, if there is a problem with a web server, hopefully I can keep the API up while working on the web server issue. This of course works both ways, no need to take down the web server for an API issue.

Work on the API will begin tonight or tomorrow.

If things go smooth  ::)    I can see release being on time in the middle of this month.

BottleCaps has been added to the CCE3 test server:

http://cap.cryptocoinexplorer.com

I will be placing i0coin on cryptocoinexplorer.com as one of the coins testing CCE3. This means it is a permanent [art of CCE (as long as it stays active), but to be introduced under the CCE3 test program.

Address will follow the normal CCE address scheme:

i0c.cryptocoinexplorer.com (http://i0c.cryptocoinexplorer.com)

i0coin has a couple year old block chain, so it will take some time for for the initial database to be built, more than likely overnight and even into tomorrow.

I will probably turn on the web server before the database is complete, but I will post here when complete.

It looks like it will be least a day or two before before the entire i0coin chain (over 900K blocks!!) is fully analyzed and CCE3 has made a complete database.It might be quicker as it hits the range where the coin lost popularity and there is only coinbase transactions.

I will open up the web server later today, though I doubt the database will complete.

To the site-developers out there that would be using the API/Long poll system in CCE3:

How do you want the long poll to work?

As of now it looks like the long poll system will be completely separate from CCE3, it could even be used with the current ABE explorers.

How would you like to handle the variety of coins? Do you have a separate URL for each coin to poll too, or one address and have a JSON field giving the 3 letter code the coin that has polled?

Would just a simple poll with the coin code be enough, or would you like the poll system to push the block information also?

The issue I see here is it does take at least a few seconds for the database loader to process block and transactions information before it becomes available on the explorer and API.

If the Long Poll pushes the block, it will get the block information directly from the daemon and just pass on the RPC response from the daemon.

Once CCE3 goes live, I will be adding the majority of coins to CCE.

However, I am slowly adding coins to the test server in an attempt to stress test and see how many coins a server can comfortably serve under cce3.

I will consider adding CGB as one of the test coins when I get ready to add another.

thats a good idea and something i would use if u implement it. The Small delay would be negligable imo. Even the long polling by miners (when notify scripts arent used on a pool) has a small delay. IMO id prefer the method of having a seperate URL per coin.

Update on i0coin -  The database is still building, it is going much slower then any of the coins I have put on the test server so far. I am not quite sure why, unless the daemon is just a bit slower with RPC responses.

Developers:

What is your opinion on Ubuntu 13.10 server:


http://insights.ubuntu.com/news/ubuntu-13-10-delivers-faster-cloud-setup-and-management-for-scale-out-environments/

Good progress, just wanted to complain about colors and overall design. :)
Could you use something like twitter-bootstrap to build your layout on? :)

It has been awhile since I updated. So many fires on other projects  :D

There are going to be many changes in version 4.0:

1. I have found that orphans can be trickier then I thought. I have caught the explorers in endless loops of restore from what should be a orphan free backups.I am going to have to completely change the way the database is indexed, so individual blocks and all their related transactions can be updated individually without a complete database restore.I did not want to go to that level of complexity, but it appears it is going to be needed.

2. As long as I am making a major revision, I am going to change the web lay out to be completely HTML 5 compliant  and use a different kind of page template system. A new look is in store for version 4.

3 .I may write some C++ modules for certain functions to gain speed. I need to run some more tests to see if the bottlenecks in the system are due to the database or code,  but most likely I will start rewriting parts of the explorer in C++.

4. I am going to push the release estimate to Mid November. I want to put out a quality product, not just rush out a sub standard piece of software.

5. I will continue to add coins to the test project, as it is important to test along a variety of block chains and stress test the servers.

UPDATE!!!!!!

I am now currently installing test version 3.99 on the test server. The explorers will be coming back up as the new databases are created.

New in test version 3.99:

New database structure -  The new database structure is smaller and makes "hot-fixing" orphans possible.

Rewrite of much of the core logic - Core logic updated to use the new database  + addtional features listed below.

HOT_FIX OF ORPHAN BLOCKS - Every time the database loader is called, it will check the last number of blocks set in the configuration file. I have all the explorers set to 250 for now. If an orphan block is found, the loader will "Hot-Fix" the block in the database. No more dumping the database and reloading from backup!!! The web server will remain up during a hot-fix, as it only takes a second or two.

Automatic database backups every ~1000 blocks - Whenever the loader detects 1000+ blocks since the last backup, it will auto backup the database.

Major changes to the address ledger page - The ledger page now has two tables, one for credits and one for debits, this makes the ledger much more readable. Web page ledgers are limited to the last 1000 credit and 1000 debit transactions. Longer searches will be available through the API. Anything more then 1000 (Even that is a high number) is pretty much unusable while looking at a web page. Transactions are listed newest to oldest.

Thanks. :)

Update:

I think I have finally kicked the Db/user thread issue.

I have stopped using Cherrypy for the DB connection/user thread matching. It works well opening database connections as user threads start, but does not appear to have any way of closing the connection when the thread is closed. This was so unbelievable , it took me weeks to finally give in... :D

I am now using DButils persistent DB connections system and it appears to be working well. Database processes appear to be stable in number and well within limits.

This means I can move back to working on the API and the CCE3 project can move forward.

Great! Please consider to add DMD (Diamond Coin)

How about add CHNCoin?
It actually already was on CCE but you threw it out this summer when the network almost died due to "Difficulty trap" (hight diff and slow retarget = miners exodus).
But ~1.5 month ago diff retarget algo was replaced (now PPC type - little diff retarget every block):
https://bitcointalk.org/index.php?topic=322488.0

And now (after update to new version) coin works good and active:
~ 500 MH/s active mining
traded on two exchanges at least and have descent market cap (~ 600 000 - 800 000 USD)

http://bte.cryptocoinexplorer.com/chain/Bytecoin is down.

At this Time I have multiple VPS down. I will try to get everything back up ASAP.

Can you add VegasCoin?

Cryptocoinexplorer.com will be converting to CCE3 over the weekend.

Please see the thread at:

https://bitcointalk.org/index.php?topic=576209.0 (https://bitcointalk.org/index.php?topic=576209.0)