Bitcoin Forum

Open Source CCE 4.0: https://github.com/cryptocoinexplorer/Open-Source-CCE-4.0 (https://github.com/cryptocoinexplorer/Open-Source-CCE-4.0)


CryptoCoin explorer was started in late 2012 as a PPC explorer shortly after the PPC block chain went live.
The PPC explorer was based on ABE with some modifications to allow ABE to work with the then new POS block chain.
Eventually more explorers were added and the site became www.cryptocoinexplorer.com
 (http://www.cryptocoinexplorer.com)
After the first year a few things became apparent.

ABE (Created by John Tobey - https://github.com/bitcoin-abe/bitcoin-abe (https://github.com/bitcoin-abe/bitcoin-abe)):

Before I go further, I want to give a big credit to John Tobey. At the time ABE was written, coin daemon RPC commands were not as advanced as they are today.
ABE was a brilliant piece of software and is still largely used as a base for explorers to this day.
In no way do I want to imply otherwise. John Tobey and his team have contributed greatly to the Cryptocoin community.

ABE quite often would need to have its code modified to work with new emerging block chains. At the time, ABE would get its information by directly reading the blk database files.
As new block chain technologies came out and fields were added or modified, ABE would be unable correctly read the files as it expected fields to be in certain byte positions from certain markers.
If the database structure was not Bitcoin or Litecoin, it would have a difficult time parsing the block chain. ABE also needed to be able to natively hash. At the time it could hash SHA-256 and Scrypt.

CCE (CryptoCoin Explorer):
With CCE expanding, it was becoming increasing difficult to keep altering the code to work with more and more block chains.
RPC commands were becoming more advanced and reached a point where all the information needed for an explorer could be retrieved from the coin daemon RPC.
I decided to write a new explorer from the ground up that would get all the information from the daemon RPC thereby negating the need to have the explorer
understand the block structure or hash algorithm. Using just configuration options, CCE can be made to work with almost all coins without the need to patch code.

CCE was always a full block chain explorer, storing data in a mysql database as new blocks were accepted into the block chain. The database loader is triggered by the
block notify daemon function and populates the database. The web server works with the database exclusively and does not communicate with the coin daemon.

NOTE: Later versions added a couple of functions where the web server does communicate with the daemon, but those functions are not in the open source version.

The first release of the new CCE was named CCE 3.0 and released on CCE in 2013. Later CCE 3.5 and eventually CCE 3.99 (A final patch version adding some of the core CCE 4.0 improvements) were released on CCE.
These were all closed source.

At the beginning of this year, I started working on CCE 4. This was to be a basic rewrite of CCE 3.5 with many features added and core code improvements.
I also decided to release an open source version of CCE 4. This is a VERY BASIC version of CCE 4 and meant to act as a foundation for building an explorer, though it can be used as is.
The closed source version being developed for CCE is based on the open source version, but is more robust and feature rich.

Open Source CCE 4.0: https://github.com/cryptocoinexplorer/Open-Source-CCE-4.0 (https://github.com/cryptocoinexplorer/Open-Source-CCE-4.0)

That's a nice piece of work.

I like the top 100 rich list and the list of peers. I have read countless requests for nodes in the altcoin threads here.

All the main features I would want from a block explorer are there, like searches for balances in addresses, and searches for transactions. If I ever set one up I will use your latest block explorer.

Fantastic work!
I have been using the sponsored explorer for over a year and it works great. As mentioned, the open sourced explorer has lots of good features so definitely going to try it for some coins.

FYI:

The official BCT thread for the Cryptocoin Explorer website:

https://bitcointalk.org/index.php?topic=922515.0 (https://bitcointalk.org/index.php?topic=922515.0)

If you are looking for a professionally hosted and maintained block explorer, please see the sponsorship page.

http://www.cryptocoinexplorer.com/sponsor (http://www.cryptocoinexplorer.com/sponsor)

These explorers are CCE version 3.99 as the full version of CCE 4.0 is in development.

New commit pushed to the repository.

• Bug Fix: Replaced a line of code that was unintentionally removed. This line checked the stats Boolean in the configuration file.
• Documentation: Various corrections. Moved the database loader and web server documentation to separate folder "docs".

looks great / love your work

Great work, thanks!

This is great news. I have followed your work on them for some time. Im a bit surprised it went open source but am happy you did.

Well I guess my weekend is booked now lol

There are a few reasons I released an open source version of CCE.

• To contribute to the open source community. My knowledge of development has been expanded by the open source community. I felt it was appropriate to give something back.
• I wanted to show the cryptocoin community how much the coin daemon RPC has improved over the years and what can be done with it.
• Of course on the business side, it is great PR. It can help draw attention to CCE and the upcoming 4.0 update.

In my view, the open source version is not particularly special in terms of coding and I think any moderately skilled programmer could have wrote it.
The open source version is very basic and truly meant as a foundation to build on. The full closed source version is more robust and much more feature rich.
CCE 3.99 already has many more features and a full API vs the open source version, including some of the commands needed for "lite" clients, the API will be further expanded in full version 4.0.

The open source version fits into CCE as I am planning on offering 4 tiers of explorers on CCE once the full version 4.0 is finished.
The free tier will basically be the open source version minus the two API commands and supported by ad revenue.
The removal of the API is mostly due to the fact the API calls do not generate ad revenue.
I may start setting up explorers on the free tier before the full version 4.0 is released on CCE.

More information on the full version 4.0 and cryptocoinexplorer.com will be in the official thread at:  https://bitcointalk.org/index.php?topic=922515.0. (https://bitcointalk.org/index.php?topic=922515.0.)

There are couple of fixes in the documentation that will show up in the next commit.

The coin daemon needs to run with the daemon option. Either add the option "daemon=1" in the daemon configuration file or run the daemon with the --daemon switch.
It has become habit for me to always run the coin daemons with the switch which is why I overlooked it in the documentation.  :)

The apache2 http proxy module is actually called proxy_http. The command to activate it should read: a2enmod proxy_http. In the instructions I refer to it incorrectly as http_proxy.

Great work :)

I was able to get the db loader loading a chain last night. Will test out the webserver tonight and let you know how everything goes.

Documentation was great. Almost no trouble getting it configured and loaded into mysql

New commit pushed:

• Fixed timeout error bug (issue #1)
• Documentation corrections

taking a tumble at it tonight , looks tight , still don't have it woking yet though  !! lol

How is it going along?

Any questions?

Hi :)

Thanks again for the PM. I finally managed to get some time to give OS-CCE a try.
Configuring your explorer is really a piece of cake. Great!

I took a me a minute to realize I had to manually create the tables using mysql cce -u <user> -p < newdb.sql ;)
Contrary to Bitcoin-abe, it seems that loading the DB with ./dbload.py -n -v is much slower?
It takes like 10 minutes to load 1000 blocks on a virtual machine with Ubuntu Server 14.04 with 2GB RAM. Does it heavily rely on RAM? I might have to increase it?

Another reason I can think of is the checking back 250 blocks to detect orphaned blocks? This might be too far in the past for my coin? My blockchain has over 1.7 million blocks (just over 1 year old) :-X

Or is this due to me having reniced the wallet daemon to "19"? It's a habit I took while working with VPS servers as these companies often don't like high loads and will charge you for it or terminate your contract.

Anyway, I'm looking forward to the day it's synched to see what it's going to be like \(^_^)/

Hmmz... I had to restart dbload ..


A little later: ~1000 blocks

Or

I'm confused now ???

Edit: Looks like it's solved by adding the -l argument.

This is due to the time-out  in normal mode , you later discovered to correct with the -l option to extend the time-out to 24 hours.
"Main Loop cannot concatenate 'str' and 'int' objects" is a bug corrected in the latest commit. It was purely a cosmetic bug with sending the error string to stderr, no function involved.

Edit - After second look, the error occurred during a proper build (-n). My guess is your coin daemon is running slow or is sluggish to respond. The RPC interface in comm.py will timeout the connection
after 10 seconds. If the daemon is responding slow, it would also account for the slow build time.



Odd error that pops up once in a great while when restarting during the initial load. Just remove the row in the block table with the height "-1" and continue.




Or

Related to the first to question. Make sure you are using the latest version.
 
I'm confused now ???

Edit: Looks like it's solved by adding the -l argument. - This is correct



The 250 block check takes only a second or two. It is asking for the block hash from the coin daemon and checking it against the database. If you feel that is too long it can be adjusted in the configuration file.
There is also a 15 second delay when starting the loader without the new database option. This is to allow the coin daemon time to catch orphans it discovers shortly after calling the loader through block notify.
Both of these functions run once when the loader is started, when you see the count-up they are no longer being called.

As to questions about speed of the initial load, it is hard to answer without actually running it on whatever system you are using. The CCE servers have about a half dozen explorers each running and generally have a CPU usage around 0.8 - 1.5 on a six CPU system. This is running the coin daemons, loaders and web servers.
I recently built a database on a 600K + block chain and the initial load took about 10 hours. The initial load can take some time, but after the initial load is finished it runs smooth and quick.

I will address some of these annoyances mentioned in the above posts on the next commit  :)

Thank you for replying dreamwatcher

I've doubled the amount of RAM for the VM and it did accelerate the process. It loaded over 200000 blocks the past half hour.
I must agree with you that the daemon and qt wallet is very slow. Changing the rpc timeout to 60 seconds in comm.py and the deamon's config also seems to have a positive effect.

We discovered that our coin was forked from Hackcoin and have managed to more or less clean the code and take control, but that never solved the performance issues. In fact, it has been like this since the beginning and staking puts a considerable amount of stress on the system, which shouldn't happen. I'm working with a coin that has been abandoned by the original developer and the two of us trying to keep it afloat, are no developers.

The VM is basically running on top of a computer that I used for mining altcoins last year. I extended the memory to 16GB, but the CPU is still the old one, an Intel Celeron G1610, which probably acts as a bottleneck for quite demanding processes. I intend on replacing it soon.

The VPS that runs bitcoin-abe atm is a 4 vCPU VM with 6GB RAM and 30GB SSD storage and has an offloaded MySQL DB running on Dual E5 2620 CPUs, 384GB RAM and 8 x 480GB SSD on RAID10. I don't worry about its performance in running CCE.

It's just that before deploying something, I tend to test it first and see what I can expect :)

The things like the rich list and largest tx... is that processed after all the blocks are loaded or when invoking stats.py? I imagine it will take some time to process it on 1.7M blocks.

This is happening on my server after every few weeks . We have a getblock call to altcoin daemon for each blocks. After few block numbers bitcoind RPC hangs (I have tried waiting for 30 min, but doesn't help, have to kill daemon and restart). I'm also using a node to make rpc calls.

i spent a few hours the other night looking it over in detail setting up the environment, may have time this weekend to fire it up

How much space does the DB require?
Guess I'll just have to test in on the VPS this weekend. My test VM with 16GB storage ran out of diskspace after ~800k blocks.

Also, I can't find a way to specity the mysql host in the configuration file. Does it absolutely need a local mysql server to function? I'd prefer using a remote mysqldb that could handle all the queries more easily. Can I change the host address in comm.py without breaking it?


Changed the host and it works with a remote server :)
Lol... with abe it was a little harder to get this working...

Lookin' forward to a completely loaded DB \(^_ ^)/

The size of the database depends greatly on the number of transactions in the block chain.
Using HoboNickels as an example:
~2.3 Million blocks - ~6.5 million TX_IN and ~6.5 million TX_OUT
Size of the active database: 6.7 GB

I will add an option to set the host address for the database in the next commit. Personally, I have never had need for it so I never thought to put it in.






This is why there are two timers in the database loader. The main loop timer is set at 5 minutes while individual RPC calls are set at 10 seconds. Since adding the timers I have not had to reset the loader or the daemons. If the loader gets interpreted/timeout, the next time it is called it re-parses the last 5 blocks including transactions, this helps guarantee information integrity if the loader is interrupted in the middle of parsing blocks/transactions.  I have not noticed the coin daemons RPC completely locking up where it will not answer subsequent calls, they appear only to ignore/hang on one call.





The stats module is called when the database loader has completed its cycle. However, the stats module can be run independently, but the information will only be accurate to the point the database is loaded. Largest TX is populated on the fly as the loader is parsing transactions, again only will be acurate to the point where the database is loaded.

Added - Running the stats.py module while the loader is running might interrupt the loader due to the way Python handles module imports. If one really wants to run the stats module while the loader is running I would suggest making a copy of comm.py named something like comm2.py and change the import in stats.py to import the copied file instead of comm.py.

New commit pushed.  :)

Added:

• Configuration options to set MySQL server IP address and port
• Configuration option to set RPC timeout
• Instructions on how to set up an empty database from the command line

Fixed:

• Type errors when logging or sending messages to stderr. Exception descriptors are now always cast to string
• The database loader will now throw an exception when a block  height of "-1" is sent to the block parser instead of attempting to process it.

thanks , will have another go at it this week i think

Hello \(^_^)/
The whole blockchain got loaded :)

I've got some questions concerning how to continue now:
After it finished, it calls the statistics module:

This statistics steps in incredibly fast, so I was kind of suspicious that it did it's job.
Unlike the other modules, it didn't create a log file and I only have these:

• comm.log
• loader.log
• server.log

When I launch python websrv.py, I can only connect to it using lynx localhost:8222 (I did not setup Apache yet, as it's currently serving Abe)
Shouldn't I be able to connect from a remote client as well even without Apache involved? Opening this port 8222 in iptables doesn't change its behaviour.

server.log contains:

Using this text based browser, I noticed that the peers and richlist pages are not populated.
https://i.imgur.com/DrYr5tS.png (http://imgur.com/DrYr5tS)

I believe the last lines in comm.log are from this statistics module:

I can query the table address and it looks completely filled as one would expect.

The stats module runs quickly as it is using MySql itself to sort most of the results. Hash rate, difficulty and peer info are directly queried from the coin daemon.

Except for the sever log, it is normal for there not to be logs as they are only written to when warnings/errors occur.

The SQL error, I have not seen. However it appears you are using MariaDB, CCE and the SQL file I have included are written with InnoDB.

Hash rate can only be obtained if the coin daemon provides it. As outlined in the instructions:



The CCE web server uses Apache for http proxy and to serve static files.

It also requires bootstrap 3 be served statically or hosted (In which case change the header in base.html)


As outlined in the instructions, you can change the port the CherryPy server runs on in the server.conf file


If  one wishes to use another web server then Apache, one will need to configure according to the server. It needs to proxy/reverse proxy and serve static files.( Bootstrap 3 and any images)


I believe the issues with the text based browser may be related to not having or using bootstrap 3.

I also wanted to add that CherryPy is configured to be run through a proxy. If you want to directly access it , most likely you will need to change the server.conf file.


tools.encode.on: True
tools.gzip.on: True
tools.gzip.mime_types: ['text/html', 'text/plain', 'application/json', 'text/javascript', 'application/javascript']
tools.proxy.on: True

Thank for your reply. I got over exited today because of passing a screening for a job I'd really like to do, hence the stupid questions.
The client I use doesn't have a hashrate field to be returned and that's indeed what went wrong. Peers and richlist are now populated as well.

I'll do the webserver part sometime tomorrow or the following days. So far I love OS CCE v4.0. You've created a fantastic piece of software m(._.)m

Concerning InnoDB vs MariaDB, there is not much I can do about it, except install a private instance of mysql on the VPS. The offloaded DB is running as:

mysql  Ver 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64) using readline 6.3

Connection id:          25126661
Current database:
Current user:           mine1_cce@23.92.65.26
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server version:         5.5.5-10.0.14-MariaDB-1~precise-log mariadb.org binary distribution
Protocol version:       10
Connection:             daldb.vpsdime.com via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:               3306
Uptime:                 5 days 8 hours 42 min 20 sec


I thought it was InnoDB due to the engine being set to default:

+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+                                                                                                                              
| Engine             | Support | Comment                                                                    | Transactions | XA   | Savepoints |                                                                                                                              
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+                                                                                                                              
| CSV                | YES     | CSV storage engine                                                         | NO           | NO   | NO         |                                                                                                                              
| MRG_MyISAM         | YES     | Collection of identical MyISAM tables                                      | NO           | NO   | NO         |                                                                                                                              
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables                  | NO           | NO   | NO         |                                                                                                                              
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears)             | NO           | NO   | NO         |                                                                                                                              
| MyISAM             | YES     | MyISAM storage engine                                                      | NO           | NO   | NO         |                                                                                                                              
| InnoDB             | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES          | YES  | YES        |                                                                                                                              
| ARCHIVE            | YES     | Archive storage engine                                                     | NO           | NO   | NO         |                                                                                                                              
| FEDERATED          | YES     | FederatedX pluggable storage engine                                        | YES          | NO   | YES        |                                                                                                                              
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                                         | NO           | NO   | NO         |                                                                                                                              
| Aria               | YES     | Crash-safe tables with MyISAM heritage                                     | NO           | NO   | NO         |                                                                                                                              
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+

The reported SQL error ... will it break something in the fuctionality or stability while running your block explorer?

It's up & running dreamwatcher \(^_^)/ ^{(* (http://cce.morningminerals.com/))}

At first I had some trouble with it because I'm already running a vhost on that server for bitcoin-abe.
The configuration of this virtual host is slightly different to what's written in the docs on github:

<VirtualHost <something>.morningminerals.com:80>
    ServerName <something>.morningminerals.com
    ServerAlias <something>.morningminerals.com
    ProxyPreserveHost On
    ProxyErrorOverride On
    DocumentRoot /home/akira/www/cce
    <Directory /home/akira/www/cce>
        Order allow,deny
        Allow from all
    </Directory>
    ProxyPass /images !
    ProxyPass /robots.txt !
    ProxyPass /css !
    ProxyPass /js !
    <Proxy <something>.morningminerals.com>
        Order allow,deny
        Allow from all
    </Proxy>
    ProxyPass / http://localhost:8222/
    ProxyPassReverse / http://localhost:8222/
</VirtualHost>

Without the added green directive it won't load the bootstrap files located in documentroot.
Also, the domain/host must specified in each individual vhost's conf or only one of them works ;-)

Hi,

try your script but currently getting  "Main Loop label empty or too long" after running ./dbload.py -n -v

Any ideas?

Hi, dreamwatcher. You not answer any email or PM.
What status of scifi exchange and KED, GPL, UFC?
If you abandon it, why not return our coins from exchange?

dreamwatcher has been around a long while,  i am not sure what you are asking of him but I trust he will return

Hi,
Tell me how to connect to the coin Open Source CCE 4.0 Block Explorer ?
There are detailed instructions?

can you responsible adults pls take over the git on OSC?   (open source coin)

folder Open-Source-CCE-4.0-master\docs there are 2 files dbload.rst and web_server.rst
I edit them under their own coin ?
On the server you want to install Apache and to put in the directory WWW all files from html ?
And before that, you need to start a coin purse ?

Only after that I do not understand what to do ?

Your request has zero /nothing to do with this topic.

This thread is for Dreamwatchers open source CCE "State of the Art" Block Crawler

not for the coin you mention, 


btw I did not report the post to a mod but it really does not belong here,

get in touch with r3wt , it was his coin , before he turned it over to a developer who cannot even compile hello world............... I doubt anyone here wants anything to do with that project.




Please don't spam this thread.

thank

Nice work!  ;)

Using it now: http://45.32.67.42 (http://45.32.67.42)

https://bitcointalk.org/index.php?topic=743731.msg13046478#msg13046478 (https://bitcointalk.org/index.php?topic=743731.msg13046478#msg13046478)