Open Source CCE 4.0 Block Explorer

[Hoshimaru]

How much space does the DB require?
Guess I'll just have to test in on the VPS this weekend. My test VM with 16GB storage ran out of diskspace after ~800k blocks.

Also, I can't find a way to specity the mysql host in the configuration file. Does it absolutely need a local mysql server to function? I'd prefer using a remote mysqldb that could handle all the queries more easily. Can I change the host address in comm.py without breaking it?

Code:

conn = pymysql.connect(db=CONFIG["database"]["dbname"], host='127.0.0.1', port=3306, user=CONFIG["database"]["dbuser"],passwd=CONFIG["database"]["dbpassword"])

Changed the host and it works with a remote server
Lol... with abe it was a little harder to get this working...

Lookin' forward to a completely loaded DB \(^_ ^)/

[dreamwatcher]

How much space does the DB require?
Guess I'll just have to test in on the VPS this weekend. My test VM with 16GB storage ran out of diskspace after ~800k blocks.

Also, I can't find a way to specity the mysql host in the configuration file. Does it absolutely need a local mysql server to function? I'd prefer using a remote mysqldb that could handle all the queries more easily. Can I change the host address in comm.py without breaking it?

Code:

conn = pymysql.connect(db=CONFIG["database"]["dbname"], host='127.0.0.1', port=3306, user=CONFIG["database"]["dbuser"],passwd=CONFIG["database"]["dbpassword"])

Changed the host and it works with a remote server
Lol... with abe it was a little harder to get this working...

Lookin' forward to a completely loaded DB \(^_ ^)/

The size of the database depends greatly on the number of transactions in the block chain.
Using HoboNickels as an example:
~2.3 Million blocks - ~6.5 million TX_IN and ~6.5 million TX_OUT
Size of the active database: 6.7 GB

I will add an option to set the host address for the database in the next commit. Personally, I have never had need for it so I never thought to put it in.

This is happening on my server after every few weeks . We have a getblock call to altcoin daemon for each blocks. After few block numbers bitcoind RPC hangs (I have tried waiting for 30 min, but doesn't help, have to kill daemon and restart). I'm also using a node to make rpc calls.

This is why there are two timers in the database loader. The main loop timer is set at 5 minutes while individual RPC calls are set at 10 seconds. Since adding the timers I have not had to reset the loader or the daemons. If the loader gets interpreted/timeout, the next time it is called it re-parses the last 5 blocks including transactions, this helps guarantee information integrity if the loader is interrupted in the middle of parsing blocks/transactions.  I have not noticed the coin daemons RPC completely locking up where it will not answer subsequent calls, they appear only to ignore/hang on one call.

The things like the rich list and largest tx... is that processed after all the blocks are loaded or when invoking stats.py? I imagine it will take some time to process it on 1.7M blocks.

The stats module is called when the database loader has completed its cycle. However, the stats module can be run independently, but the information will only be accurate to the point the database is loaded. Largest TX is populated on the fly as the loader is parsing transactions, again only will be acurate to the point where the database is loaded.

Added - Running the stats.py module while the loader is running might interrupt the loader due to the way Python handles module imports. If one really wants to run the stats module while the loader is running I would suggest making a copy of comm.py named something like comm2.py and change the import in stats.py to import the copied file instead of comm.py.

[dreamwatcher]

New commit pushed.  

Added:

• Configuration options to set MySQL server IP address and port
• Configuration option to set RPC timeout
• Instructions on how to set up an empty database from the command line

Fixed:

• Type errors when logging or sending messages to stderr. Exception descriptors are now always cast to string
• The database loader will now throw an exception when a block  height of "-1" is sent to the block parser instead of attempting to process it.

[cinnamon_carter]

thanks , will have another go at it this week i think

[Hoshimaru]

Hello \(^_^)/
The whole blockchain got loaded

I've got some questions concerning how to continue now:
After it finished, it calls the statistics module:

Code:

akira@Mineralscoin:~/CCE$ python dbload.py -v -l
Processing Block:  1749237  of  1749236 
Calling Statistics Module
Database load complete

This statistics steps in incredibly fast, so I was kind of suspicious that it did it's job.
Unlike the other modules, it didn't create a log file and I only have these:

• comm.log
• loader.log
• server.log

When I launch python websrv.py, I can only connect to it using lynx localhost:8222 (I did not setup Apache yet, as it's currently serving Abe)
Shouldn't I be able to connect from a remote client as well even without Apache involved? Opening this port 8222 in iptables doesn't change its behaviour.

server.log contains:

Code:

[29/Jun/2015:11:18:35] ENGINE Daemonized to PID: 11612
[29/Jun/2015:11:18:35] ENGINE Started monitor thread '_TimeoutMonitor'.
[29/Jun/2015:11:18:35] ENGINE PID 11612 written to '/home/akira/CCE/cherrypy.pid'.
[29/Jun/2015:11:18:35] ENGINE Serving on 127.0.0.1:8222
[29/Jun/2015:11:18:35] ENGINE Bus STARTED

Using this text based browser, I noticed that the peers and richlist pages are not populated.


I believe the last lines in comm.log are from this statistics module:

Code:

ERROR:root:06-29 16:25:05 (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ': '74.15099844', u'proof-of-stake': '0.00024414'}' at line 1") : query no return
ERROR:root:06-29 16:25:05 UPDATE stats SET curr_diff = %s : query no return
ERROR:root:06-29 16:25:05 'networkhashps' : General stat module error

I can query the table address and it looks completely filled as one would expect.

[dreamwatcher]

The stats module runs quickly as it is using MySql itself to sort most of the results. Hash rate, difficulty and peer info are directly queried from the coin daemon.

Except for the sever log, it is normal for there not to be logs as they are only written to when warnings/errors occur.

The SQL error, I have not seen. However it appears you are using MariaDB, CCE and the SQL file I have included are written with InnoDB.

ERROR:root:06-29 16:25:05 'networkhashps' : General stat module error

Hash rate can only be obtained if the coin daemon provides it. As outlined in the instructions:

Code:

hashrate:

        Boolean indicating if the network hash is available by the daemon.
        Network hash rate can only be obtained from the coin daemon.

hashfield: (Suggested: networkhashps)

        Label of the field network hash rate is returned by the coin daemon.

ratelabel: (Suggested: MH or GH)

        Label to use for the hash rate on the index web page.

hashmult:

        Hash rate multiplier to use for storing hash rate at desired level.
        Example: Daemon output is 5678 H/s. Use the multipler 0.001 to store as 5.678.

The CCE web server uses Apache for http proxy and to serve static files.

It also requires bootstrap 3 be served statically or hosted (In which case change the header in base.html)

Code:

Install Bootstrap3 css and js files to their appropriate css and js directory in document root.

        Alternately, find a host for bootstrap3 and change the base.html header to reflect.

As outlined in the instructions, you can change the port the CherryPy server runs on in the server.conf file

Code:

[global]
server.socket_port:8222

If  one wishes to use another web server then Apache, one will need to configure according to the server. It needs to proxy/reverse proxy and serve static files.( Bootstrap 3 and any images)


I believe the issues with the text based browser may be related to not having or using bootstrap 3.

[dreamwatcher]

I also wanted to add that CherryPy is configured to be run through a proxy. If you want to directly access it , most likely you will need to change the server.conf file.


tools.encode.on: True
tools.gzip.on: True
tools.gzip.mime_types: ['text/html', 'text/plain', 'application/json', 'text/javascript', 'application/javascript']
tools.proxy.on: True

[Hoshimaru]

Thank for your reply. I got over exited today because of passing a screening for a job I'd really like to do, hence the stupid questions.
The client I use doesn't have a hashrate field to be returned and that's indeed what went wrong. Peers and richlist are now populated as well.

I'll do the webserver part sometime tomorrow or the following days. So far I love OS CCE v4.0. You've created a fantastic piece of software m(._.)m

Concerning InnoDB vs MariaDB, there is not much I can do about it, except install a private instance of mysql on the VPS. The offloaded DB is running as:

mysql  Ver 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64) using readline 6.3

Connection id:          25126661
Current database:
Current user:           mine1_cce@23.92.65.26
SSL:                    Not in use
Current pager:          stdout
Using outfile:          ''
Using delimiter:        ;
Server version:         5.5.5-10.0.14-MariaDB-1~precise-log mariadb.org binary distribution
Protocol version:       10
Connection:             daldb.vpsdime.com via TCP/IP
Server characterset:    latin1
Db     characterset:    latin1
Client characterset:    utf8
Conn.  characterset:    utf8
TCP port:               3306
Uptime:                 5 days 8 hours 42 min 20 sec


I thought it was InnoDB due to the engine being set to default:

+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+                                                                                                                              
| Engine             | Support | Comment                                                                    | Transactions | XA   | Savepoints |                                                                                                                              
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+                                                                                                                              
| CSV                | YES     | CSV storage engine                                                         | NO           | NO   | NO         |                                                                                                                              
| MRG_MyISAM         | YES     | Collection of identical MyISAM tables                                      | NO           | NO   | NO         |                                                                                                                              
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables                  | NO           | NO   | NO         |                                                                                                                              
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears)             | NO           | NO   | NO         |                                                                                                                              
| MyISAM             | YES     | MyISAM storage engine                                                      | NO           | NO   | NO         |                                                                                                                              
| InnoDB             | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES          | YES  | YES        |                                                                                                                              
| ARCHIVE            | YES     | Archive storage engine                                                     | NO           | NO   | NO         |                                                                                                                              
| FEDERATED          | YES     | FederatedX pluggable storage engine                                        | YES          | NO   | YES        |                                                                                                                              
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                                         | NO           | NO   | NO         |                                                                                                                              
| Aria               | YES     | Crash-safe tables with MyISAM heritage                                     | NO           | NO   | NO         |                                                                                                                              
+--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+

The reported SQL error ... will it break something in the fuctionality or stability while running your block explorer?

[Hoshimaru]

It's up & running dreamwatcher \(^_^)/ ^(*)

At first I had some trouble with it because I'm already running a vhost on that server for bitcoin-abe.
The configuration of this virtual host is slightly different to what's written in the docs on github:

<VirtualHost <something>.morningminerals.com:80>
    ServerName <something>.morningminerals.com
    ServerAlias <something>.morningminerals.com
    ProxyPreserveHost On
    ProxyErrorOverride On
    DocumentRoot /home/akira/www/cce
    <Directory /home/akira/www/cce>
        Order allow,deny
        Allow from all
    </Directory>
    ProxyPass /images !
    ProxyPass /robots.txt !
    ProxyPass /css !
    ProxyPass /js !
    <Proxy <something>.morningminerals.com>
        Order allow,deny
        Allow from all
    </Proxy>
    ProxyPass / http://localhost:8222/
    ProxyPassReverse / http://localhost:8222/
</VirtualHost>

Without the added green directive it won't load the bootstrap files located in documentroot.
Also, the domain/host must specified in each individual vhost's conf or only one of them works ;-)

[lahlor]

Hi,

try your script but currently getting  "Main Loop label empty or too long" after running ./dbload.py -n -v

Any ideas?

[vampirus]

Hi, dreamwatcher. You not answer any email or PM.
What status of scifi exchange and KED, GPL, UFC?
If you abandon it, why not return our coins from exchange?

[cinnamon_carter]

dreamwatcher has been around a long while,  i am not sure what you are asking of him but I trust he will return

[MorGan-id]

Hi,
Tell me how to connect to the coin Open Source CCE 4.0 Block Explorer ?
There are detailed instructions?

[BitcoinNational]

can you responsible adults pls take over the git on OSC?   (open source coin)

[MorGan-id]

can you responsible adults pls take over the git on OSC?   (open source coin)

folder Open-Source-CCE-4.0-master\docs there are 2 files dbload.rst and web_server.rst
I edit them under their own coin ?
On the server you want to install Apache and to put in the directory WWW all files from html ?
And before that, you need to start a coin purse ?

Only after that I do not understand what to do ?

[cinnamon_carter]

Your request has zero /nothing to do with this topic.

This thread is for Dreamwatchers open source CCE "State of the Art" Block Crawler

not for the coin you mention, 


btw I did not report the post to a mod but it really does not belong here,

get in touch with r3wt , it was his coin , before he turned it over to a developer who cannot even compile hello world............... I doubt anyone here wants anything to do with that project.




Please don't spam this thread.

can you responsible adults pls take over the git on OSC?   (open source coin)

[MorGan-id]

Your request has zero /nothing to do with this topic.

This thread is for Dreamwatchers open source CCE "State of the Art" Block Crawler

not for the coin you mention, 


btw I did not report the post to a mod but it really does not belong here,

get in touch with r3wt , it was his coin , before he turned it over to a developer who cannot even compile hello world............... I doubt anyone here wants anything to do with that project.




Please don't spam this thread.

can you responsible adults pls take over the git on OSC?   (open source coin)

thank

[oldcoinguru]

Nice work!  

Using it now: http://45.32.67.42

https://bitcointalk.org/index.php?topic=743731.msg13046478#msg13046478

Block explorer

http://45.32.67.42

Nice work mate block explorer up and running 

It's not my work.  I used some open source software provided by dreamwatcher for the block explorer  https://bitcointalk.org/index.php?topic=1088291.0