Hoshimaru
|
|
June 26, 2015, 06:14:17 AM Last edit: June 26, 2015, 07:06:52 AM by Hoshimaru |
|
How much space does the DB require? Guess I'll just have to test in on the VPS this weekend. My test VM with 16GB storage ran out of diskspace after ~800k blocks. Also, I can't find a way to specity the mysql host in the configuration file. Does it absolutely need a local mysql server to function? I'd prefer using a remote mysqldb that could handle all the queries more easily. Can I change the host address in comm.py without breaking it?conn = pymysql.connect(db=CONFIG["database"]["dbname"], host='127.0.0.1', port=3306, user=CONFIG["database"]["dbuser"],passwd=CONFIG["database"]["dbpassword"]) Changed the host and it works with a remote server Lol... with abe it was a little harder to get this working... Lookin' forward to a completely loaded DB \(^_ ^)/
|
|
|
|
dreamwatcher (OP)
Legendary
Offline
Activity: 1064
Merit: 1000
|
|
June 26, 2015, 06:03:47 PM Last edit: June 26, 2015, 06:40:16 PM by dreamwatcher |
|
How much space does the DB require? Guess I'll just have to test in on the VPS this weekend. My test VM with 16GB storage ran out of diskspace after ~800k blocks. Also, I can't find a way to specity the mysql host in the configuration file. Does it absolutely need a local mysql server to function? I'd prefer using a remote mysqldb that could handle all the queries more easily. Can I change the host address in comm.py without breaking it?conn = pymysql.connect(db=CONFIG["database"]["dbname"], host='127.0.0.1', port=3306, user=CONFIG["database"]["dbuser"],passwd=CONFIG["database"]["dbpassword"]) Changed the host and it works with a remote server Lol... with abe it was a little harder to get this working... Lookin' forward to a completely loaded DB \(^_ ^)/ The size of the database depends greatly on the number of transactions in the block chain. Using HoboNickels as an example: ~2.3 Million blocks - ~6.5 million TX_IN and ~6.5 million TX_OUT Size of the active database: 6.7 GB I will add an option to set the host address for the database in the next commit. Personally, I have never had need for it so I never thought to put it in. This is happening on my server after every few weeks . We have a getblock call to altcoin daemon for each blocks. After few block numbers bitcoind RPC hangs (I have tried waiting for 30 min, but doesn't help, have to kill daemon and restart). I'm also using a node to make rpc calls.
This is why there are two timers in the database loader. The main loop timer is set at 5 minutes while individual RPC calls are set at 10 seconds. Since adding the timers I have not had to reset the loader or the daemons. If the loader gets interpreted/timeout, the next time it is called it re-parses the last 5 blocks including transactions, this helps guarantee information integrity if the loader is interrupted in the middle of parsing blocks/transactions. I have not noticed the coin daemons RPC completely locking up where it will not answer subsequent calls, they appear only to ignore/hang on one call. The things like the rich list and largest tx... is that processed after all the blocks are loaded or when invoking stats.py? I imagine it will take some time to process it on 1.7M blocks.
The stats module is called when the database loader has completed its cycle. However, the stats module can be run independently, but the information will only be accurate to the point the database is loaded. Largest TX is populated on the fly as the loader is parsing transactions, again only will be acurate to the point where the database is loaded. Added - Running the stats.py module while the loader is running might interrupt the loader due to the way Python handles module imports. If one really wants to run the stats module while the loader is running I would suggest making a copy of comm.py named something like comm2.py and change the import in stats.py to import the copied file instead of comm.py.
|
|
|
|
dreamwatcher (OP)
Legendary
Offline
Activity: 1064
Merit: 1000
|
|
June 28, 2015, 02:51:00 PM |
|
New commit pushed. Added: - Configuration options to set MySQL server IP address and port
- Configuration option to set RPC timeout
- Instructions on how to set up an empty database from the command line
Fixed: - Type errors when logging or sending messages to stderr. Exception descriptors are now always cast to string
- The database loader will now throw an exception when a block height of "-1" is sent to the block parser instead of attempting to process it.
|
|
|
|
cinnamon_carter
Legendary
Offline
Activity: 1148
Merit: 1018
It's about time -- All merrit accepted !!!
|
|
June 29, 2015, 04:41:30 AM |
|
thanks , will have another go at it this week i think
|
Check out my coin Photon Merge Mine 5 other Blake 256 coins - 6x your hash power https://www.blakecoin.org/The obvious choice is not always the best choice. LOOK DEEPER - Look into the Blake 256 Family -- CC
|
|
|
Hoshimaru
|
|
June 29, 2015, 04:39:34 PM |
|
Hello \(^_^)/ The whole blockchain got loaded I've got some questions concerning how to continue now: After it finished, it calls the statistics module: akira@Mineralscoin:~/CCE$ python dbload.py -v -l Processing Block: 1749237 of 1749236 Calling Statistics Module Database load complete This statistics steps in incredibly fast, so I was kind of suspicious that it did it's job. Unlike the other modules, it didn't create a log file and I only have these: - comm.log
- loader.log
- server.log
When I launch python websrv.py, I can only connect to it using lynx localhost:8222 (I did not setup Apache yet, as it's currently serving Abe) Shouldn't I be able to connect from a remote client as well even without Apache involved? Opening this port 8222 in iptables doesn't change its behaviour. server.log contains: [29/Jun/2015:11:18:35] ENGINE Daemonized to PID: 11612 [29/Jun/2015:11:18:35] ENGINE Started monitor thread '_TimeoutMonitor'. [29/Jun/2015:11:18:35] ENGINE PID 11612 written to '/home/akira/CCE/cherrypy.pid'. [29/Jun/2015:11:18:35] ENGINE Serving on 127.0.0.1:8222 [29/Jun/2015:11:18:35] ENGINE Bus STARTED Using this text based browser, I noticed that the peers and richlist pages are not populated. I believe the last lines in comm.log are from this statistics module: ERROR:root:06-29 16:25:05 (1064, u"You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ': '74.15099844', u'proof-of-stake': '0.00024414'}' at line 1") : query no return ERROR:root:06-29 16:25:05 UPDATE stats SET curr_diff = %s : query no return ERROR:root:06-29 16:25:05 'networkhashps' : General stat module error
I can query the table address and it looks completely filled as one would expect.
|
|
|
|
dreamwatcher (OP)
Legendary
Offline
Activity: 1064
Merit: 1000
|
|
June 29, 2015, 05:41:35 PM |
|
The stats module runs quickly as it is using MySql itself to sort most of the results. Hash rate, difficulty and peer info are directly queried from the coin daemon. Except for the sever log, it is normal for there not to be logs as they are only written to when warnings/errors occur. The SQL error, I have not seen. However it appears you are using MariaDB, CCE and the SQL file I have included are written with InnoDB. ERROR:root:06-29 16:25:05 'networkhashps' : General stat module error Hash rate can only be obtained if the coin daemon provides it. As outlined in the instructions: hashrate:
Boolean indicating if the network hash is available by the daemon. Network hash rate can only be obtained from the coin daemon.
hashfield: (Suggested: networkhashps)
Label of the field network hash rate is returned by the coin daemon.
ratelabel: (Suggested: MH or GH)
Label to use for the hash rate on the index web page.
hashmult:
Hash rate multiplier to use for storing hash rate at desired level. Example: Daemon output is 5678 H/s. Use the multipler 0.001 to store as 5.678.
The CCE web server uses Apache for http proxy and to serve static files. It also requires bootstrap 3 be served statically or hosted (In which case change the header in base.html) Install Bootstrap3 css and js files to their appropriate css and js directory in document root.
Alternately, find a host for bootstrap3 and change the base.html header to reflect.
As outlined in the instructions, you can change the port the CherryPy server runs on in the server.conf file [global] server.socket_port:8222 If one wishes to use another web server then Apache, one will need to configure according to the server. It needs to proxy/reverse proxy and serve static files.( Bootstrap 3 and any images) I believe the issues with the text based browser may be related to not having or using bootstrap 3.
|
|
|
|
dreamwatcher (OP)
Legendary
Offline
Activity: 1064
Merit: 1000
|
|
June 29, 2015, 05:55:49 PM |
|
I also wanted to add that CherryPy is configured to be run through a proxy. If you want to directly access it , most likely you will need to change the server.conf file.
tools.encode.on: True tools.gzip.on: True tools.gzip.mime_types: ['text/html', 'text/plain', 'application/json', 'text/javascript', 'application/javascript'] tools.proxy.on: True
|
|
|
|
Hoshimaru
|
|
June 29, 2015, 08:57:25 PM |
|
Thank for your reply. I got over exited today because of passing a screening for a job I'd really like to do, hence the stupid questions. The client I use doesn't have a hashrate field to be returned and that's indeed what went wrong. Peers and richlist are now populated as well. I'll do the webserver part sometime tomorrow or the following days. So far I love OS CCE v4.0. You've created a fantastic piece of software m(._.)m Concerning InnoDB vs MariaDB, there is not much I can do about it, except install a private instance of mysql on the VPS. The offloaded DB is running as: mysql Ver 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64) using readline 6.3
Connection id: 25126661 Current database: Current user: mine1_cce@23.92.65.26 SSL: Not in use Current pager: stdout Using outfile: '' Using delimiter: ; Server version: 5.5.5-10.0.14-MariaDB-1~precise-log mariadb.org binary distribution Protocol version: 10 Connection: daldb.vpsdime.com via TCP/IP Server characterset: latin1 Db characterset: latin1 Client characterset: utf8 Conn. characterset: utf8 TCP port: 3306 Uptime: 5 days 8 hours 42 min 20 sec
I thought it was InnoDB due to the engine being set to default: +--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+ | Engine | Support | Comment | Transactions | XA | Savepoints | +--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+ | CSV | YES | CSV storage engine | NO | NO | NO | | MRG_MyISAM | YES | Collection of identical MyISAM tables | NO | NO | NO | | MEMORY | YES | Hash based, stored in memory, useful for temporary tables | NO | NO | NO | | BLACKHOLE | YES | /dev/null storage engine (anything you write to it disappears) | NO | NO | NO | | MyISAM | YES | MyISAM storage engine | NO | NO | NO | | InnoDB | DEFAULT | Percona-XtraDB, Supports transactions, row-level locking, and foreign keys | YES | YES | YES | | ARCHIVE | YES | Archive storage engine | NO | NO | NO | | FEDERATED | YES | FederatedX pluggable storage engine | YES | NO | YES | | PERFORMANCE_SCHEMA | YES | Performance Schema | NO | NO | NO | | Aria | YES | Crash-safe tables with MyISAM heritage | NO | NO | NO | +--------------------+---------+----------------------------------------------------------------------------+--------------+------+------------+The reported SQL error ... will it break something in the fuctionality or stability while running your block explorer?
|
|
|
|
Hoshimaru
|
|
July 01, 2015, 10:39:55 AM |
|
It's up & running dreamwatcher \(^_^)/ (*)At first I had some trouble with it because I'm already running a vhost on that server for bitcoin-abe. The configuration of this virtual host is slightly different to what's written in the docs on github: <VirtualHost <something>.morningminerals.com:80> ServerName <something>.morningminerals.com ServerAlias <something>.morningminerals.com ProxyPreserveHost On ProxyErrorOverride On DocumentRoot /home/akira/www/cce <Directory /home/akira/www/cce> Order allow,deny Allow from all </Directory> ProxyPass /images ! ProxyPass /robots.txt ! ProxyPass /css ! ProxyPass /js ! <Proxy <something>.morningminerals.com> Order allow,deny Allow from all </Proxy> ProxyPass / http://localhost:8222/ ProxyPassReverse / http://localhost:8222/ </VirtualHost>
Without the added green directive it won't load the bootstrap files located in documentroot. Also, the domain/host must specified in each individual vhost's conf or only one of them works ;-)
|
|
|
|
lahlor
|
|
July 07, 2015, 08:06:18 AM |
|
Hi,
try your script but currently getting "Main Loop label empty or too long" after running ./dbload.py -n -v
Any ideas?
|
|
|
|
vampirus
|
|
August 03, 2015, 05:27:29 PM |
|
Hi, dreamwatcher. You not answer any email or PM. What status of scifi exchange and KED, GPL, UFC? If you abandon it, why not return our coins from exchange?
|
|
|
|
cinnamon_carter
Legendary
Offline
Activity: 1148
Merit: 1018
It's about time -- All merrit accepted !!!
|
|
August 04, 2015, 09:06:24 AM |
|
dreamwatcher has been around a long while, i am not sure what you are asking of him but I trust he will return
|
Check out my coin Photon Merge Mine 5 other Blake 256 coins - 6x your hash power https://www.blakecoin.org/The obvious choice is not always the best choice. LOOK DEEPER - Look into the Blake 256 Family -- CC
|
|
|
MorGan-id
|
|
October 30, 2015, 03:28:11 PM |
|
Hi, Tell me how to connect to the coin Open Source CCE 4.0 Block Explorer ? There are detailed instructions?
|
|
|
|
BitcoinNational
Legendary
Offline
Activity: 1470
Merit: 1010
Join The Blockchain Revolution In Logistics
|
|
October 30, 2015, 03:42:21 PM |
|
can you responsible adults pls take over the git on OSC? (open source coin)
|
|
|
|
MorGan-id
|
|
October 30, 2015, 06:44:51 PM |
|
can you responsible adults pls take over the git on OSC? (open source coin)
folder Open-Source-CCE-4.0-master\docs there are 2 files dbload.rst and web_server.rst I edit them under their own coin ? On the server you want to install Apache and to put in the directory WWW all files from html ? And before that, you need to start a coin purse ? Only after that I do not understand what to do ?
|
|
|
|
cinnamon_carter
Legendary
Offline
Activity: 1148
Merit: 1018
It's about time -- All merrit accepted !!!
|
|
October 31, 2015, 07:08:53 AM |
|
Your request has zero /nothing to do with this topic. This thread is for Dreamwatchers open source CCE "State of the Art" Block Crawler not for the coin you mention, btw I did not report the post to a mod but it really does not belong here, get in touch with r3wt , it was his coin , before he turned it over to a developer who cannot even compile hello world............... I doubt anyone here wants anything to do with that project. Please don't spam this thread. can you responsible adults pls take over the git on OSC? (open source coin)
|
Check out my coin Photon Merge Mine 5 other Blake 256 coins - 6x your hash power https://www.blakecoin.org/The obvious choice is not always the best choice. LOOK DEEPER - Look into the Blake 256 Family -- CC
|
|
|
MorGan-id
|
|
October 31, 2015, 10:47:39 AM |
|
Your request has zero /nothing to do with this topic. This thread is for Dreamwatchers open source CCE "State of the Art" Block Crawler not for the coin you mention, btw I did not report the post to a mod but it really does not belong here, get in touch with r3wt , it was his coin , before he turned it over to a developer who cannot even compile hello world............... I doubt anyone here wants anything to do with that project. Please don't spam this thread. can you responsible adults pls take over the git on OSC? (open source coin)
thank
|
|
|
|
|
|