[POOL][Scrypt][Scrypt-N][X11] Profit switching pool - wafflepool.com

[Starscream]

Hashrate : 0...

Problem with the stats page I think as I am still submitting shares, no real problem.

By the way, thank you waffle for the pool !

Getting the same.

[The Fat Miner]

Hashrate : 0...

Problem with the stats page I think as I am still submitting shares, no real problem.

By the way, thank you waffle for the pool !

Getting the same.

Glad to see I'm not the only one same issue

[northwind6]

Hashrate : 0...

Problem with the stats page I think as I am still submitting shares, no real problem.

By the way, thank you waffle for the pool !

Getting the same.

Glad to see I'm not the only one same issue

Same here... hashrate 0   

[gaalx]

miners work, but statistics 0. Need to wake wafflepool)))

[poolwaffle]

I'm the one who dropped it to 0 temporarily.

I was concerned with the profitability as well (its stupid low), and started digging around.  Looks like our main database started acting funny at some point this morning, causing some chain reactions down the line.  Once I noticed (30min ago), I did an emergency restart of everything (db, endpoints, etc), which involved a few minutes of downtime (0 hashrate).

Everything should be up and running now, and I'm looking to see how much of the damage that was done is recoverable (just not processing blocks), how much of it is just stat reporting (not being able to pull stats from the db and thus not reflecting correctly), and how much is essentially actually lost (profitability stats couldn't be updated, thus switcher has bad data to use, thus picking incorrect coins, and just mining for shit for a while).

Essentially, I'm looking into it as we speak, will hopefully know more in the coming few hours.  Sorry ahead of time.

[ugrash]

I'm the one who dropped it to 0 temporarily.

I was concerned with the profitability as well (its stupid low), and started digging around.  Looks like our main database started acting funny at some point this morning, causing some chain reactions down the line.  Once I noticed (30min ago), I did an emergency restart of everything (db, endpoints, etc), which involved a few minutes of downtime (0 hashrate).

Everything should be up and running now, and I'm looking to see how much of the damage that was done is recoverable (just not processing blocks), how much of it is just stat reporting (not being able to pull stats from the db and thus not reflecting correctly), and how much is essentially actually lost (profitability stats couldn't be updated, thus switcher has bad data to use, thus picking incorrect coins, and just mining for shit for a while).

Essentially, I'm looking into it as we speak, will hopefully know more in the coming few hours.  Sorry ahead of time.

Ok, thanks for the update

[The Fat Miner]

Ohh dear thanks for the update.

[poolwaffle]

OK.  So there are unfortunately a whole bunch of things going into the profitability for today, some of which are unrecoverable, some are essentially display issues (recoverable), some are outside of our control (some recoverable, some unrecoverable).

First, the unrecoverable:
So from what I can tell, we had about 3-4 hours early this morning of less than optimal mining (the definition of less than optimal I'm not sure on, around -25% seems likely).  This was due to two factors.  First, the database issues caused our switcher to make less than optimal choices.  Second, the database issues caused a problem in our block propagation strategy.  We typically relay blocks to remote nodes to better help with propagation time (fewer orphans).  Due to the database issues, this failed as well, so we ended up having more orphaned blocks during that time (20% instead of 10%), these happened more on lower difficulty coins than high ones (as expected), so its not a hard 10%, but its definitely a factor.  This is obviously mining time spent, that cannot be recovered, and we're expected to have _actual_ lower earnings today than normal because of it (guessing 10% or so).

Second, the display issues:
We track balances every few minutes.  Its what gives us the running tallies of coins, earnings for the day, and overall profitability numbers.  The way in which we track them is essentially snapshots of what we _think_ we have in earnings.  This is coins that we've already exchanged (known), coins that we think we can exchange (confirmed blocks, waiting for deposit to the exchange to convert), and coins that we hope we can exchange (blocks that aren't confirmed yet, could still easily be orphaned).  When we solve a block, we calculate what we think it is worth (coins x price), and add to our balance.  If in the future the block gets orphaned, the balance in a future snapshot will be less than the one previous.  What this means, is that if a block is earned in a previous day, it will be added to the earnings for that day, but if it is orphaned today, would be removed from earnings for today (yesterday is higher than it should have been, albeit slightly, today is lower than it should have been).

This compounds with the third point, issues outside of our control:
A few days ago (3-4) we earned a bunch of coins (Reddcoin, Karmacoin, Mintcoin, Smartcoin), these were all on a new exchange (coinmarket.io).  We ran them in testing and everything was fine.  We started mining them in rotation, everything was fine.  We started mining them fully, everything was fine for a day.  Then coinmarket went offline, and has been claiming they'll be coming back "today" for about 3 days straight.  In that time, all of the coins that we earned while mining, have been stuck there, unable to be exchanged.  These coins were essentially "earned" 3-4 days ago (higher earnings on those days), and have been dropping in expected price (other exchanges prices included) over those 3-4 days (mostly today/yesterday), leading to today's balance being lower than it should be.

The last piece is something that I expect is fully recoverable (more of a display issue):
The way we calculate price is still a relic of the way it came to be.  We originally (day 1) started tracking prices as a method of determining the value of coins (imagine that!).  This includes depth analysis.  Essentially depth analysis can be explained with a simple example: There are 2 bids for a coin, one bid for 10 of that coin for 1.0btc each, one bid for 10 of that coin for 0.1btc each.  Assume each block is worth 10 coins.  The first block you mine is worth a good bit (10x1 = 10btc!), however, the 2nd block you mine is worth _significantly_ less (10x0.1 = 1btc).  This is extremely important to be able to tell how much of the coins you mine you'll actually be able to sell (and their expected prices).  We've always tracked prices as a "what is the next block worth", as that was always the most important piece from a switcher point of view.  This is the price we use across the board for things like "expected btc value".

Normally, using this price isn't an issue, its a conservative estimate of actual prices.  In the previous example, if we had mined 2 blocks (20 coins), we would show the expected value as 20x0.1 = 2btc (note the low price), when in actuality, it would be 10x1 + 10x0.1 = 11btc (actual price).  This normally isn't a problem because we normally don't have that many outstanding coins, and as they convert to btc, we get the actual price they were converted at, and our stats stay very close to actual.  The time that this fails, is times like today.  Cryptsy's API is down (not sure why), so we haven't been able to auto-trade coins overnight.  This means coins have been piling up (they've actually been traded now, but our scripts can't see that until the API is back up), and because they've been piling up, and earnings are using the conservative estimate, earnings show as lower than in actuality.


All of these things combined are essentially factors in the current earnings looking rediculously low for the day.  And unfortunately, telling how much is each of them is a really challenging thing.  I'm still digging through it all, just thought I'd give you guys an update to what I've found so far.

[vladman]

Thanks for the update Waffle, as ever. And thanks for working tirelessly on improving this already awesome pool of yours.

[haploid23]

Most of what you said went over my head, but thanks for the update. If I'm understanding the update correctly, a lot of it was because of the database issue? What exactly happened with the database, and when/how will it be fixed? A 25% dip in profit is a good chunk that we're all losing.

[poolwaffle]

Most of what you said went over my head, but thanks for the update. If I'm understanding the update correctly, a lot of it was because of the database issue? What exactly happened with the database, and when/how will it be fixed? A 25% dip in profit is a good chunk that we're all losing.

It already has been fixed.  It happened at roughly 6am, was fixed at roughly 9:30am.  The cause was (in simple terms) it got close to running out of disk space, essentially from me enabling logging to track down some slow-downs, and the logs not being rotated properly (about 30gb of logging overnight).  My alerts went off as a warning, but I didn't hear them apparently (asleep).

[oktay50000]

why btc per mh dropped badly to 0.00780040


*edit : lets wait and see fixes will work or not

[sve]

<Big status update>

poolwaffle, thanks again for your great updates on what's going on. This gives me great confidence in the pool and in you as pooloperator.
Keep up the good work!!!

[gtraah]

<Big status update>

poolwaffle, thanks again for your great updates on what's going on. This gives me great confidence in the pool and in you as pooloperator.
Keep up the good work!!!

Just curious if the Pay time is 7am EST as the 1 page says , Is this still the case?? As last couple days not the case, If the pay time or schedule or pay frequency has changed, can some one let me know of please update the 1st page ?

PS: Thank you Poolwaffle for giving us updates and not leaving us in the dark, Very good work!! You must be very proud of yourself for what are doing and accomplishing  AWESOME is not even the word

Thanks
A H

[igoork]

Too many unconverted, what is going on?

From 1am to 10am last confirmed, from 10am to 2pm not a single confirmed?

http://www.dodaj.rs/f/1V/rQ/2OBLMHUj/waffle.jpg

[gtraah]

Too many unconfirmed, what is going on?

Since 28th Feb In AEST [my time zone]... THe graphs/stats have been looking bad.. And I think its due to the explanation that was just given..

It has just clicked over to the 2/3/2014 12:28am And I am still waiting for things to start looking good , Not so good so far

[diwskwmx]

I read it as the numbers will continue to look bad until the cryptsy API comes back up.  That way the scripts can get caught up.

[sve]

<Big status update>

poolwaffle, thanks again for your great updates on what's going on. This gives me great confidence in the pool and in you as pooloperator.
Keep up the good work!!!

Just curious if the Pay time is 7am EST as the 1 page says , Is this still the case?? As last couple days not the case, If the pay time or schedule or pay frequency has changed, can some one let me know of please update the 1st page ?

PS: Thank you Poolwaffle for giving us updates and not leaving us in the dark, Very good work!! You must be very proud of yourself for what are doing and accomplishing  AWESOME is not even the word

Thanks
A H

http://wafflepool.com/news

first post

[gtraah]

<Big status update>

poolwaffle, thanks again for your great updates on what's going on. This gives me great confidence in the pool and in you as pooloperator.
Keep up the good work!!!

Just curious if the Pay time is 7am EST as the 1 page says , Is this still the case?? As last couple days not the case, If the pay time or schedule or pay frequency has changed, can some one let me know of please update the 1st page ?

PS: Thank you Poolwaffle for giving us updates and not leaving us in the dark, Very good work!! You must be very proud of yourself for what are doing and accomplishing  AWESOME is not even the word

Thanks
A H

http://wafflepool.com/news

first post

Ahhh >> multiple times per day at no specific times?

Well first post does not state this and because I usually only look at the  " better stats page" I did not see the news Anyways all I can see here is this -

Note: Payouts are being changed slighty (multiple times per day), please see here for more information.

Those of you still mining at our default domain (wafflepool.com instead of one of our geo-located hosts), please update your configuration as we will be disabling mining on the base domain in the next couple days.
Payouts (Feb 26th)
Over the past few days we have been having delays on payouts from one of our major exchanges that we trade coins on (Cryptsy). Essentially we request bitcoin withdrawals for all of the amount we have held with them (enough to cover all of our payouts), however, by the time the withdrawal clears (6-12 hour delay recently), we have accumulated enough new earnings, that we no longer have enough funds from that withdrawal to complete payouts, and have to start another withdrawal (perpetually).

Starting today (until this is fixed, or we have a better solution), we'll do rolling payouts each time one of the withdrawals clears, and will continue to pay users with the same threshold (0.01btc). We will pay groups of users at a time, and will select users on the basis who hasn't been paid in the longest time. Everyone will still be paid their full earnings, there are no coins missing, etc. We just need to start paying in smaller batches, a few times per day (instead of one large batch).

I'll try my best to get this resolved as quickly as possible so we can get to a state of consistently timed payouts, but right now, the majority of this is out of my hands, and I'm waiting on responses from Cryptsy.


Have I missed something This says the 26th Feb Which I thought was still normal pay time Hmm maybe my timezone is different... Anyway It shows 2 payouts on the 27th March and the 1st march 2nd March just started 1 hour ago so I think I should be expecting 2 payouts today again

[mr_fakename]

why btc per mh dropped badly to 0.00780040


*edit : lets wait and see fixes will work or not

That's only a partial day it says on the table, that figure probably wont be accurate representation of the final number for that day as it could have been a bad morning but the afternoon is better. Although Poolwaffle mentioned some server errors earlier that would also explain a bad number/day.

Poolwaffle how accurate are the partial day numbers compared to the final number? if they can be a bit off might be worth not showing it until a day is finalised.