toxic0n
Member
 Offline
Activity: 413
Merit: 10
|
 |
April 23, 2014, 05:41:56 PM |
|
so I may have missed this in the many pages here, but have we ever gotten an API?
The whole cloudflare/ddos protection stuff is nice to keep the pool up, but incredibly annoying for both checking hte pages and trying to write data scrapers to track my performance.
If there is no API (still) any one have any suggestions on how to scrape data from their miner pages? I cant seem to get around the cloudflare junk
I am having the same problem in trying to "data scrape" as you call it. I read on the CloudFlare site that their protection does not prevent web crawlers from scanning the site (so I am confident we can find a way to do it), and I'm thinking what I will try next is to write code to accept their cookie(s). Of course, my guess is worthless until I have it running. What I unsuccessfully tried in php was to file_get_contents, then do a javascript setTimeout of 10 seconds, after which I do a window.location.reload. hmmm, well i guess if they say web crawlers can scan it, there has to be a way. what's the link where you read this? I went the python route a few weeks back, but got no where. After reading your post, i fired wireshark back up and it looks like there's a hidden form that needs some special key to return in order to let us through ... so I also think we'll need some code to accept their cookie/whatever and respond back similar to how a browser would. still kind of pain to grab some data  update: ok so i did some digging. got this from the cloudflare site: Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed. The page which your visitors see when in IUAM can be fully customized to reflect your branding. I'm Under Attack mode does not block search engine crawlers or your existing CloudFlare whitelist. My guess is they whitelisted known and allowed search engine crawlers which is why those bots work where as ours dont. Add to it that it looks like Terk opted to put the domain in a permanent I'm Under Attack Mode and we are where we are. So pretty much we need to mimic the expected behavior if we want anything to work ... lovely I neednt link you now, your text is exactly what I read. I did not reach the conclusion that you did about the crawlers being whitelisted, thanks for helping me think!  As to "kind of a pain", well, it would be worth it so I could get some sleep, I'm plain stupid about having to check every hour and enter data into a spreadsheet. Auto-reading from CM is the only missing link. I'm databasing the exchange rates of 5 different coins once every five minutes, making graphs, all automatically. See https://github.com/jordoh/miner-monitor/blob/master/pools/clever_mining.rb#L84-L102 for ruby code. The basic process is: 1. Get the 503 page from CloudFlare - it contains a verification code (in the HTML) and simple equation to solve (in the JS). 2. Make a get request for http://www.clevermining.com/cdn-cgi/l/chk_jschl?jschl_vc=<verification code from step 1>&jschl_answer=<answer to equation from step 1> with the referer header set to the page you are trying to access. If you got the answer right, you'll a 302 (redirect) response with a cf_clearance cookie. 3. Follow the redirect (i.e. request the original page you were trying to access), including the cf_clearance cookie. Thanks for this, I was looking into making an Android widget for CM and looks like I will have a way to do so. |
|
|
|
|
|
|
|
|
|
|
|
"The nature of Bitcoin is such that once version 0.1 was released, the
core design was set in stone for the rest of its lifetime." -- Satoshi
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
GolfCabalist
Newbie
Offline
Activity: 36
Merit: 0
|
|
April 23, 2014, 09:40:13 PM |
|
so I may have missed this in the many pages here, but have we ever gotten an API?
The whole cloudflare/ddos protection stuff is nice to keep the pool up, but incredibly annoying for both checking hte pages and trying to write data scrapers to track my performance.
If there is no API (still) any one have any suggestions on how to scrape data from their miner pages? I cant seem to get around the cloudflare junk
I am having the same problem in trying to "data scrape" as you call it. I read on the CloudFlare site that their protection does not prevent web crawlers from scanning the site (so I am confident we can find a way to do it), and I'm thinking what I will try next is to write code to accept their cookie(s). Of course, my guess is worthless until I have it running. What I unsuccessfully tried in php was to file_get_contents, then do a javascript setTimeout of 10 seconds, after which I do a window.location.reload. hmmm, well i guess if they say web crawlers can scan it, there has to be a way. what's the link where you read this? I went the python route a few weeks back, but got no where. After reading your post, i fired wireshark back up and it looks like there's a hidden form that needs some special key to return in order to let us through ... so I also think we'll need some code to accept their cookie/whatever and respond back similar to how a browser would. still kind of pain to grab some data update: ok so i did some digging. got this from the cloudflare site: Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed. The page which your visitors see when in IUAM can be fully customized to reflect your branding. I'm Under Attack mode does not block search engine crawlers or your existing CloudFlare whitelist. My guess is they whitelisted known and allowed search engine crawlers which is why those bots work where as ours dont. Add to it that it looks like Terk opted to put the domain in a permanent I'm Under Attack Mode and we are where we are. So pretty much we need to mimic the expected behavior if we want anything to work ... lovely I neednt link you now, your text is exactly what I read. I did not reach the conclusion that you did about the crawlers being whitelisted, thanks for helping me think! As to "kind of a pain", well, it would be worth it so I could get some sleep, I'm plain stupid about having to check every hour and enter data into a spreadsheet. Auto-reading from CM is the only missing link. I'm databasing the exchange rates of 5 different coins once every five minutes, making graphs, all automatically. See https://github.com/jordoh/miner-monitor/blob/master/pools/clever_mining.rb#L84-L102 for ruby code. The basic process is: 1. Get the 503 page from CloudFlare - it contains a verification code (in the HTML) and simple equation to solve (in the JS). 2. Make a get request for http://www.clevermining.com/cdn-cgi/l/chk_jschl?jschl_vc=<verification code from step 1>&jschl_answer=<answer to equation from step 1> with the referer header set to the page you are trying to access. If you got the answer right, you'll a 302 (redirect) response with a cf_clearance cookie. 3. Follow the redirect (i.e. request the original page you were trying to access), including the cf_clearance cookie. Jordo! Thanks 10^6. I was getting the answer, but knew nothing about the cf_clearance cookie. You take a huge risk putting yourself in a support position |
|
|
|
|
Monayman
Newbie
Offline
Activity: 33
Merit: 0
|
|
April 23, 2014, 11:21:38 PM |
|
I noticed something amusing today. Can you spot a difference between these two fragments of cgminer logs: [2014-04-23 04:08:23] Network diff set to 766
[2014-04-23 04:08:23] Stratum from pool 0 detected new block
[2014-04-23 04:09:00] Network diff set to 901
[2014-04-23 04:09:00] Stratum from pool 0 detected new block
[2014-04-23 04:09:14] Network diff set to 1.01K
[2014-04-23 04:09:14] Stratum from pool 0 detected new block
[2014-04-23 04:09:15] Network diff set to 314
[2014-04-23 04:09:15] Stratum from pool 0 detected new block
[2014-04-23 04:09:40] Network diff set to 288
[2014-04-23 04:09:40] Stratum from pool 0 detected new block
[2014-04-23 04:09:49] Network diff set to 299
[2014-04-23 04:09:49] Stratum from pool 0 detected new block
[2014-04-23 04:10:40] Network diff set to 317
[2014-04-23 04:10:40] Stratum from pool 0 detected new block
[2014-04-23 04:11:26] Network diff set to 319
[2014-04-23 04:11:26] Stratum from pool 0 detected new block
[2014-04-23 04:11:29] Network diff set to 325
[2014-04-23 04:11:29] Stratum from pool 0 detected new block
[2014-04-23 04:08:23] Network diff set to 766
[2014-04-23 04:08:23] Stratum from pool 0 detected new block
[2014-04-23 04:09:00] Network diff set to 901
[2014-04-23 04:09:00] Stratum from pool 0 detected new block
[2014-04-23 04:09:14] Network diff set to 1.01K
[2014-04-23 04:09:14] Stratum from pool 0 detected new block
[2014-04-23 04:09:18] Network diff set to 314
[2014-04-23 04:09:18] Stratum from pool 0 detected new block
[2014-04-23 04:09:39] Network diff set to 288
[2014-04-23 04:09:39] Stratum from pool 0 detected new block
[2014-04-23 04:09:48] Network diff set to 299
[2014-04-23 04:09:48] Stratum from pool 0 detected new block
[2014-04-23 04:10:40] Network diff set to 317
[2014-04-23 04:10:40] Stratum from pool 0 detected new block
[2014-04-23 04:11:25] Network diff set to 319
[2014-04-23 04:11:25] Stratum from pool 0 detected new block
[2014-04-23 04:11:29] Network diff set to 325
[2014-04-23 04:11:29] Stratum from pool 0 detected new block
The first one is from CleverMining, the second one from WafflePool (or maybe it's the other way around). Great minds think alike? You had me spend a couple of minutes looking for the smallest discrepancy between both ... and then i gave up and read your final comment -_- |
|
|
|
|
|
Ibistru
|
|
April 24, 2014, 11:22:26 AM |
|
I have noticed that on this pool there is a long delay, several hours usually, before mined coins are exchanged for bitcoins.
My question is: doesn't this undermine the very concept of multi-coin mining, i.e. mining the coins that are most profitable in that moment? If you wait several hours before trading at the exchange don't you risk to lose profitability (or to take a loss)?
The pool marks coins as exchanged after they are withdrawn from exchanges plus three confirmations. Actual exchanging into BTC is done much earlier and is ongoing. Withdrawing is done few times per day and at this moment balances are marked as exchanged. Thanks, this sounds reasonable and answers my question. EDIT: you may wish to write this on the website, other people may wonder. Great job on your pool! |
|
|
|
|
|
Kapz786
|
|
April 24, 2014, 11:25:19 AM |
|
Would really like a Hong Kong or Australian based server - get a lot of rejects at the moment  |
|
|
|
|
|
Xenocyde
|
|
April 24, 2014, 12:32:04 PM |
|
Would really like a Hong Kong or Australian based server - get a lot of rejects at the moment You are getting fake rejects most likely. If you see "Untracked share" warning in your miner, then that's a fake reject. Real rejects rate is 1-2%. |
|
|
|
muchdogecpumine
Newbie
Offline
Activity: 36
Merit: 0
|
|
April 24, 2014, 03:04:43 PM |
|
Is anybody else having trouble accessing their user page? I can connect to the other clevermining pages, but when I try to load my user page I get a 504 error.
|
|
|
|
|
|
dragonmike
|
|
April 24, 2014, 03:24:59 PM |
|
Is anybody else having trouble accessing their user page? I can connect to the other clevermining pages, but when I try to load my user page I get a 504 error.
same. |
|
|
|
|
|
PCMiner
|
|
April 24, 2014, 03:29:03 PM |
|
I noticed something amusing today. Can you spot a difference between these two fragments of cgminer logs: [2014-04-23 04:08:23] Network diff set to 766
[2014-04-23 04:08:23] Stratum from pool 0 detected new block
[2014-04-23 04:09:00] Network diff set to 901
[2014-04-23 04:09:00] Stratum from pool 0 detected new block
[2014-04-23 04:09:14] Network diff set to 1.01K
[2014-04-23 04:09:14] Stratum from pool 0 detected new block
[2014-04-23 04:09:15] Network diff set to 314
[2014-04-23 04:09:15] Stratum from pool 0 detected new block
[2014-04-23 04:09:40] Network diff set to 288
[2014-04-23 04:09:40] Stratum from pool 0 detected new block
[2014-04-23 04:09:49] Network diff set to 299
[2014-04-23 04:09:49] Stratum from pool 0 detected new block
[2014-04-23 04:10:40] Network diff set to 317
[2014-04-23 04:10:40] Stratum from pool 0 detected new block
[2014-04-23 04:11:26] Network diff set to 319
[2014-04-23 04:11:26] Stratum from pool 0 detected new block
[2014-04-23 04:11:29] Network diff set to 325
[2014-04-23 04:11:29] Stratum from pool 0 detected new block
[2014-04-23 04:08:23] Network diff set to 766
[2014-04-23 04:08:23] Stratum from pool 0 detected new block
[2014-04-23 04:09:00] Network diff set to 901
[2014-04-23 04:09:00] Stratum from pool 0 detected new block
[2014-04-23 04:09:14] Network diff set to 1.01K
[2014-04-23 04:09:14] Stratum from pool 0 detected new block
[2014-04-23 04:09:18] Network diff set to 314
[2014-04-23 04:09:18] Stratum from pool 0 detected new block
[2014-04-23 04:09:39] Network diff set to 288
[2014-04-23 04:09:39] Stratum from pool 0 detected new block
[2014-04-23 04:09:48] Network diff set to 299
[2014-04-23 04:09:48] Stratum from pool 0 detected new block
[2014-04-23 04:10:40] Network diff set to 317
[2014-04-23 04:10:40] Stratum from pool 0 detected new block
[2014-04-23 04:11:25] Network diff set to 319
[2014-04-23 04:11:25] Stratum from pool 0 detected new block
[2014-04-23 04:11:29] Network diff set to 325
[2014-04-23 04:11:29] Stratum from pool 0 detected new block
The first one is from CleverMining, the second one from WafflePool (or maybe it's the other way around). Great minds think alike? Actually what you are reading is from one pool, not from both. whichever one "pool 0" is in your settings is just reporting a change even if it's not the pool actively being mined on. I only figured this out (by accident) after having two machines side by side, one on a stable LTC pool with Clevermining as a fall over and the other on Clevermining with the LTC pool as it's backup. The network dif changes (in my case, from clevermining) were reporting instantly on both machines at the same time even though they were technically mining on different pools. |
|
|
|
|
|
byt411
|
|
April 24, 2014, 03:29:51 PM |
|
Is anybody else having trouble accessing their user page? I can connect to the other clevermining pages, but when I try to load my user page I get a 504 error.
same. I can't even access the homepage. CloudFlare is working fine, so CleverMining might have gotten DDOSed? I thought the frontend couldn't be DDOSed, since CloudFlare is supposed to protect, but I guess there's still the possibility. |
|
|
|
|
muchdogecpumine
Newbie
Offline
Activity: 36
Merit: 0
|
|
April 24, 2014, 03:43:41 PM |
|
Is anybody else having trouble accessing their user page? I can connect to the other clevermining pages, but when I try to load my user page I get a 504 error.
same. I can't even access the homepage. CloudFlare is working fine, so CleverMining might have gotten DDOSed? I thought the frontend couldn't be DDOSed, since CloudFlare is supposed to protect, but I guess there's still the possibility. They really should post updates to their twitter or here. If the web page is down but stratum is up, we get a bit jumpy. |
|
|
|
|
|
Xenocyde
|
|
April 24, 2014, 04:21:03 PM |
|
Cloudflare is doing some maintenance.
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3654
Merit: 8922
https://bpip.org
|
|
April 24, 2014, 04:48:01 PM |
|
Actually what you are reading is from one pool, not from both. whichever one "pool 0" is in your settings is just reporting a change even if it's not the pool actively being mined on. I only figured this out (by accident) after having two machines side by side, one on a stable LTC pool with Clevermining as a fall over and the other on Clevermining with the LTC pool as it's backup. The network dif changes (in my case, from clevermining) were reporting instantly on both machines at the same time even though they were technically mining on different pools.
That's just not true. Pool 0 is not the same in my two configs and there is no other pool reporting anything. This from ~20 minutes ago:
0: Enabled Alive Quota 1 Prio 0: stratum+tcp://us.clevermining.com:3333 User:abcd
[2014-04-24 18:19:40] Network diff set to 588
[2014-04-24 18:19:40] Stratum from pool 0 detected new block
[2014-04-24 18:19:46] Network diff set to 693
[2014-04-24 18:19:46] Stratum from pool 0 detected new block
[2014-04-24 18:19:49] Network diff set to 897
[2014-04-24 18:19:49] Stratum from pool 0 detected new block
[2014-04-24 18:20:18] Network diff set to 912
[2014-04-24 18:20:18] Stratum from pool 0 detected new block
[2014-04-24 18:20:23] Network diff set to 1.01K
[2014-04-24 18:20:23] Stratum from pool 0 detected new block
[2014-04-24 18:20:28] Stratum from pool 0 requested work restart
[2014-04-24 18:21:06] Network diff set to 721
[2014-04-24 18:21:06] Stratum from pool 0 detected new block
[2014-04-24 18:21:14] Network diff set to 850
[2014-04-24 18:21:14] Stratum from pool 0 detected new block
[2014-04-24 18:21:16] Stratum from pool 0 requested work restart
[2014-04-24 18:21:53] Network diff set to 1.01K
[2014-04-24 18:21:53] Stratum from pool 0 detected new block
[2014-04-24 18:22:05] Network diff set to 1.05K
[2014-04-24 18:22:05] Stratum from pool 0 detected new block
[2014-04-24 18:23:31] Network diff set to 1.03K
0: Enabled Alive Quota 1 Prio 0: stratum+tcp://useast.wafflepool.com:3333 User:abcd
[2014-04-24 18:19:39] Network diff set to 588
[2014-04-24 18:19:39] Stratum from pool 0 detected new block
[2014-04-24 18:19:46] Network diff set to 693
[2014-04-24 18:19:46] Stratum from pool 0 detected new block
[2014-04-24 18:19:47] Network diff set to 897
[2014-04-24 18:19:47] Stratum from pool 0 detected new block
[2014-04-24 18:20:17] Network diff set to 912
[2014-04-24 18:20:17] Stratum from pool 0 detected new block
[2014-04-24 18:20:28] Network diff set to 1.01K
[2014-04-24 18:20:28] Stratum from pool 0 detected new block
[2014-04-24 18:20:28] Stratum from pool 0 requested work restart
[2014-04-24 18:21:05] Network diff set to 721
[2014-04-24 18:21:05] Stratum from pool 0 detected new block
[2014-04-24 18:21:14] Network diff set to 850
[2014-04-24 18:21:14] Stratum from pool 0 detected new block
[2014-04-24 18:21:16] Stratum from pool 0 requested work restart
[2014-04-24 18:21:53] Network diff set to 1.01K
[2014-04-24 18:21:53] Stratum from pool 0 detected new block
[2014-04-24 18:22:05] Network diff set to 1.05K
[2014-04-24 18:22:05] Stratum from pool 0 detected new block
[2014-04-24 18:23:28] Network diff set to 1.03K
|
|
|
|
|
bigradeon
Member
Offline
Activity: 98
Merit: 10
|
|
April 24, 2014, 04:49:20 PM |
|
hello all
along with 503 status i got an message from cg: " rejected untracked stratum from clevemining" , so what does it mean?
|
|
|
|
|
|
byt411
|
|
April 24, 2014, 04:53:12 PM |
|
hello all
along with 503 status i got an message from cg: " rejected untracked stratum from clevemining" , so what does it mean?
It means nothing, ignore it. |
|
|
|
|
bigradeon
Member
Offline
Activity: 98
Merit: 10
|
|
April 24, 2014, 05:11:06 PM |
|
@byt41 Tk! just had a panic  |
|
|
|
|
uberua
Member
Offline
Activity: 179
Merit: 10
|
|
April 24, 2014, 05:36:34 PM |
|
Cloudflare is doing some maintenance.
Amsterdam CloudFlare Working www.clevermining.comHost Error What's wrong with web page, any comments? |
|
|
|
|
GolfCabalist
Newbie
Offline
Activity: 36
Merit: 0
|
|
April 24, 2014, 06:00:05 PM |
|
Cloudflare is doing some maintenance.
Amsterdam CloudFlare Working www.clevermining.comHost Error What's wrong with web page, any comments? Back up at 13:59 EDT. |
|
|
|
|
|
Terk (OP)
|
|
April 24, 2014, 06:01:18 PM |
|
Website has been down for two hours but mining was unaffected. Web is back online. Sorry for inconvenience.
|
|
|
|
|
|
|
