Open Source CCE 4.0 Block Explorer

[dreamwatcher]

Open Source CCE 4.0: https://github.com/cryptocoinexplorer/Open-Source-CCE-4.0


CryptoCoin explorer was started in late 2012 as a PPC explorer shortly after the PPC block chain went live.
The PPC explorer was based on ABE with some modifications to allow ABE to work with the then new POS block chain.
Eventually more explorers were added and the site became www.cryptocoinexplorer.com

After the first year a few things became apparent.

ABE (Created by John Tobey - https://github.com/bitcoin-abe/bitcoin-abe):

Before I go further, I want to give a big credit to John Tobey. At the time ABE was written, coin daemon RPC commands were not as advanced as they are today.
ABE was a brilliant piece of software and is still largely used as a base for explorers to this day.
In no way do I want to imply otherwise. John Tobey and his team have contributed greatly to the Cryptocoin community.

ABE quite often would need to have its code modified to work with new emerging block chains. At the time, ABE would get its information by directly reading the blk database files.
As new block chain technologies came out and fields were added or modified, ABE would be unable correctly read the files as it expected fields to be in certain byte positions from certain markers.
If the database structure was not Bitcoin or Litecoin, it would have a difficult time parsing the block chain. ABE also needed to be able to natively hash. At the time it could hash SHA-256 and Scrypt.

CCE (CryptoCoin Explorer):
With CCE expanding, it was becoming increasing difficult to keep altering the code to work with more and more block chains.
RPC commands were becoming more advanced and reached a point where all the information needed for an explorer could be retrieved from the coin daemon RPC.
I decided to write a new explorer from the ground up that would get all the information from the daemon RPC thereby negating the need to have the explorer
understand the block structure or hash algorithm. Using just configuration options, CCE can be made to work with almost all coins without the need to patch code.

CCE was always a full block chain explorer, storing data in a mysql database as new blocks were accepted into the block chain. The database loader is triggered by the
block notify daemon function and populates the database. The web server works with the database exclusively and does not communicate with the coin daemon.

NOTE: Later versions added a couple of functions where the web server does communicate with the daemon, but those functions are not in the open source version.

The first release of the new CCE was named CCE 3.0 and released on CCE in 2013. Later CCE 3.5 and eventually CCE 3.99 (A final patch version adding some of the core CCE 4.0 improvements) were released on CCE.
These were all closed source.

At the beginning of this year, I started working on CCE 4. This was to be a basic rewrite of CCE 3.5 with many features added and core code improvements.
I also decided to release an open source version of CCE 4. This is a VERY BASIC version of CCE 4 and meant to act as a foundation for building an explorer, though it can be used as is.
The closed source version being developed for CCE is based on the open source version, but is more robust and feature rich.

Open Source CCE 4.0: https://github.com/cryptocoinexplorer/Open-Source-CCE-4.0

[thrax]

That's a nice piece of work.

I like the top 100 rich list and the list of peers. I have read countless requests for nodes in the altcoin threads here.

All the main features I would want from a block explorer are there, like searches for balances in addresses, and searches for transactions. If I ever set one up I will use your latest block explorer.

[electron-coin]

Fantastic work!
I have been using the sponsored explorer for over a year and it works great. As mentioned, the open sourced explorer has lots of good features so definitely going to try it for some coins.

[dreamwatcher]

FYI:

The official BCT thread for the Cryptocoin Explorer website:

https://bitcointalk.org/index.php?topic=922515.0

If you are looking for a professionally hosted and maintained block explorer, please see the sponsorship page.

http://www.cryptocoinexplorer.com/sponsor

These explorers are CCE version 3.99 as the full version of CCE 4.0 is in development.

[dreamwatcher]

New commit pushed to the repository.

• Bug Fix: Replaced a line of code that was unintentionally removed. This line checked the stats Boolean in the configuration file.
• Documentation: Various corrections. Moved the database loader and web server documentation to separate folder "docs".

[cinnamon_carter]

looks great / love your work

[YarkoL]

Great work, thanks!

[mullick]

This is great news. I have followed your work on them for some time. Im a bit surprised it went open source but am happy you did.

Well I guess my weekend is booked now lol

[dreamwatcher]

This is great news. I have followed your work on them for some time. Im a bit surprised it went open source but am happy you did.

Well I guess my weekend is booked now lol

There are a few reasons I released an open source version of CCE.

• To contribute to the open source community. My knowledge of development has been expanded by the open source community. I felt it was appropriate to give something back.
• I wanted to show the cryptocoin community how much the coin daemon RPC has improved over the years and what can be done with it.
• Of course on the business side, it is great PR. It can help draw attention to CCE and the upcoming 4.0 update.

In my view, the open source version is not particularly special in terms of coding and I think any moderately skilled programmer could have wrote it.
The open source version is very basic and truly meant as a foundation to build on. The full closed source version is more robust and much more feature rich.
CCE 3.99 already has many more features and a full API vs the open source version, including some of the commands needed for "lite" clients, the API will be further expanded in full version 4.0.

The open source version fits into CCE as I am planning on offering 4 tiers of explorers on CCE once the full version 4.0 is finished.
The free tier will basically be the open source version minus the two API commands and supported by ad revenue.
The removal of the API is mostly due to the fact the API calls do not generate ad revenue.
I may start setting up explorers on the free tier before the full version 4.0 is released on CCE.

More information on the full version 4.0 and cryptocoinexplorer.com will be in the official thread at:  https://bitcointalk.org/index.php?topic=922515.0.

[dreamwatcher]

There are couple of fixes in the documentation that will show up in the next commit.

The coin daemon needs to run with the daemon option. Either add the option "daemon=1" in the daemon configuration file or run the daemon with the --daemon switch.
It has become habit for me to always run the coin daemons with the switch which is why I overlooked it in the documentation. 

The apache2 http proxy module is actually called proxy_http. The command to activate it should read: a2enmod proxy_http. In the instructions I refer to it incorrectly as http_proxy.

[mullick]

There are couple of fixes in the documentation that will show up in the next commit.

The coin daemon needs to run with the daemon option. Either add the option "daemon=1" in the daemon configuration file or run the daemon with the --daemon switch.
It has become habit for me to always run the coin daemons with the switch which is why I overlooked it in the documentation. 

The apache2 http proxy module is actually called proxy_http. The command to activate it should read: a2enmod proxy_http. In the instructions I refer to it incorrectly as http_proxy.

Great work

I was able to get the db loader loading a chain last night. Will test out the webserver tonight and let you know how everything goes.

Documentation was great. Almost no trouble getting it configured and loaded into mysql

[dreamwatcher]

New commit pushed:

• Fixed timeout error bug (issue #1)
• Documentation corrections

[cinnamon_carter]

taking a tumble at it tonight , looks tight , still don't have it woking yet though  !! lol

[dreamwatcher]

taking a tumble at it tonight , looks tight , still don't have it woking yet though  !! lol

How is it going along?

Any questions?

[Hoshimaru]

Hi

Thanks again for the PM. I finally managed to get some time to give OS-CCE a try.
Configuring your explorer is really a piece of cake. Great!

I took a me a minute to realize I had to manually create the tables using mysql cce -u <user> -p < newdb.sql
Contrary to Bitcoin-abe, it seems that loading the DB with ./dbload.py -n -v is much slower?
It takes like 10 minutes to load 1000 blocks on a virtual machine with Ubuntu Server 14.04 with 2GB RAM. Does it heavily rely on RAM? I might have to increase it?

Another reason I can think of is the checking back 250 blocks to detect orphaned blocks? This might be too far in the past for my coin? My blockchain has over 1.7 million blocks (just over 1 year old)

Or is this due to me having reniced the wallet daemon to "19"? It's a habit I took while working with VPS servers as these companies often don't like high loads and will charge you for it or terminate your contract.

Anyway, I'm looking forward to the day it's synched to see what it's going to be like \(^_^)/

Hmmz... I had to restart dbload ..

Code:

akira@positron:~/CCE$ ./dbload.py -n -v
Processing Block:  6368  of  1637812 
Main Loop cannot concatenate 'str' and 'int' objects
akira@positron:~/CCE$ ./dbload.py -v
Recheck Called
Processing Block:  7188  of  1637812 

A little later: ~1000 blocks

Code:

Processing Block:  10192  of  1637812 
Main Loop (1062, u"Duplicate entry '-1' for key 'PRIMARY'")

Or

Code:

Processing Block:  13430  of  1637812 
Main Loop sequence index must be integer, not 'str'

I'm confused now

Edit: Looks like it's solved by adding the -l argument.

[dreamwatcher]

Code:

akira@positron:~/CCE$ ./dbload.py -n -v
Processing Block:  6368  of  1637812 
Main Loop cannot concatenate 'str' and 'int' objects
akira@positron:~/CCE$ ./dbload.py -v
Recheck Called
Processing Block:  7188  of  1637812 

This is due to the time-out  in normal mode , you later discovered to correct with the -l option to extend the time-out to 24 hours.
"Main Loop cannot concatenate 'str' and 'int' objects" is a bug corrected in the latest commit. It was purely a cosmetic bug with sending the error string to stderr, no function involved.

Edit - After second look, the error occurred during a proper build (-n). My guess is your coin daemon is running slow or is sluggish to respond. The RPC interface in comm.py will timeout the connection
after 10 seconds. If the daemon is responding slow, it would also account for the slow build time.

Code:

Processing Block:  10192  of  1637812 
Main Loop (1062, u"Duplicate entry '-1' for key 'PRIMARY'")

Odd error that pops up once in a great while when restarting during the initial load. Just remove the row in the block table with the height "-1" and continue.




Or

Code:

Processing Block:  13430  of  1637812 
Main Loop sequence index must be integer, not 'str'

Related to the first to question. Make sure you are using the latest version.
 
I'm confused now

Edit: Looks like it's solved by adding the -l argument. - This is correct



The 250 block check takes only a second or two. It is asking for the block hash from the coin daemon and checking it against the database. If you feel that is too long it can be adjusted in the configuration file.
There is also a 15 second delay when starting the loader without the new database option. This is to allow the coin daemon time to catch orphans it discovers shortly after calling the loader through block notify.
Both of these functions run once when the loader is started, when you see the count-up they are no longer being called.

As to questions about speed of the initial load, it is hard to answer without actually running it on whatever system you are using. The CCE servers have about a half dozen explorers each running and generally have a CPU usage around 0.8 - 1.5 on a six CPU system. This is running the coin daemons, loaders and web servers.
I recently built a database on a 600K + block chain and the initial load took about 10 hours. The initial load can take some time, but after the initial load is finished it runs smooth and quick.

[dreamwatcher]

I will address some of these annoyances mentioned in the above posts on the next commit 

[Hoshimaru]

Thank you for replying dreamwatcher

I've doubled the amount of RAM for the VM and it did accelerate the process. It loaded over 200000 blocks the past half hour.
I must agree with you that the daemon and qt wallet is very slow. Changing the rpc timeout to 60 seconds in comm.py and the deamon's config also seems to have a positive effect.

We discovered that our coin was forked from Hackcoin and have managed to more or less clean the code and take control, but that never solved the performance issues. In fact, it has been like this since the beginning and staking puts a considerable amount of stress on the system, which shouldn't happen. I'm working with a coin that has been abandoned by the original developer and the two of us trying to keep it afloat, are no developers.

The VM is basically running on top of a computer that I used for mining altcoins last year. I extended the memory to 16GB, but the CPU is still the old one, an Intel Celeron G1610, which probably acts as a bottleneck for quite demanding processes. I intend on replacing it soon.

The VPS that runs bitcoin-abe atm is a 4 vCPU VM with 6GB RAM and 30GB SSD storage and has an offloaded MySQL DB running on Dual E5 2620 CPUs, 384GB RAM and 8 x 480GB SSD on RAID10. I don't worry about its performance in running CCE.

It's just that before deploying something, I tend to test it first and see what I can expect

The things like the rich list and largest tx... is that processed after all the blocks are loaded or when invoking stats.py? I imagine it will take some time to process it on 1.7M blocks.

[oniromancia]

Thank you for replying dreamwatcher

I've doubled the amount of RAM for the VM and it did accelerate the process. It loaded over 200000 blocks the past half hour.
I must agree with you that the daemon and qt wallet is very slow. Changing the rpc timeout to 60 seconds in comm.py and the deamon's config also seems to have a positive effect.

We discovered that our coin was forked from Hackcoin and have managed to more or less clean the code and take control, but that never solved the performance issues. In fact, it has been like this since the beginning and staking puts a considerable amount of stress on the system, which shouldn't happen. I'm working with a coin that has been abandoned by the original developer and the two of us trying to keep it afloat, are no developers.

The VM is basically running on top of a computer that I used for mining altcoins last year. I extended the memory to 16GB, but the CPU is still the old one, an Intel Celeron G1610, which probably acts as a bottleneck for quite demanding processes. I intend on replacing it soon.

The VPS that runs bitcoin-abe atm is a 4 vCPU VM with 6GB RAM and 30GB SSD storage and has an offloaded MySQL DB running on Dual E5 2620 CPUs, 384GB RAM and 8 x 480GB SSD on RAID10. I don't worry about its performance in running CCE.

It's just that before deploying something, I tend to test it first and see what I can expect

The things like the rich list and largest tx... is that processed after all the blocks are loaded or when invoking stats.py? I imagine it will take some time to process it on 1.7M blocks.

This is happening on my server after every few weeks . We have a getblock call to altcoin daemon for each blocks. After few block numbers bitcoind RPC hangs (I have tried waiting for 30 min, but doesn't help, have to kill daemon and restart). I'm also using a node to make rpc calls.

[cinnamon_carter]

i spent a few hours the other night looking it over in detail setting up the environment, may have time this weekend to fire it up