[4+ EH] Slush Pool (slushpool.com); Overt AsicBoost; World First Mining Pool

[allinvain]

Hmm, this is weird..I have "0.63640126 BTC" showing as unconfirmed and it has been that way for many hours

I need to check three blocks manually and mark them in database as valid or invalid, which I'll do today. Until then, reward from those three blocks is pending as 'unconfirmed'...

OK slush, thanks for clearing that up. I am in no rush so do that at your earliest convenience.

[LobsterMan]

Eagerly awaiting full status report 

[cablepair]

ive been mining at 375 m/hashs since slush came back and got it back up and running...

it seems like unconfirmed keeps going up, but confirmed is going no where

I definitely should of made something in the last 15 or so hours

whats up with the confirmations ?

[xenon481]

ive been mining at 375 m/hashs since slush came back and got it back up and running...

it seems like unconfirmed keeps going up, but confirmed is going no where

I definitely should of made something in the last 15 or so hours

whats up with the confirmations ?

Block Generation requires 120 confirmations (subsequent blocks) before the 50BTC is spendable.

It takes ~20 hours for 120 confirmations.

[cablepair]

thanks for the info, it just seems so much slower than it was before

I'll keep mining away though...

on another note, im looking to buy a new righ with two cross fired 5770's what kind of speeds you think I could get from that?

[dafishman]

thanks for the info, it just seems so much slower than it was before

I'll keep mining away though...

on another note, im looking to buy a new righ with two cross fired 5770's what kind of speeds you think I could get from that?

Everything seems to be "business as usual" at this point.

Two 5770's should net you ~340-370.

Might want to check the price difference between 5770's and 5850's. Although the 5850's each require 2x pcie power.

I have 2 5850's in one rig and am getting about 580 with them. I also have a few 5770's running and they net about 160-180 each.

Just pay attention to cooling when using two cards in one machine.

[slush]

Eagerly awaiting full status report  

What happen?
On Saturday's evening (UTC), bitcoin daemons on two independent pool servers ate all available memory (RAM & swap) and fall to massive I/O operations, which make servers extremely slow. Central server switched traffic to the last machine, but this machine was unable to handle all traffic and some RPC calls started to timeout. Then those miners getting RPC timeouts probably started to ask server even more frequently (miners don't implement any "polite waiting" in the code and send new RPC call immediately after error/timeout). Those thousands of requests per second were the last piece to shut down last working machine .

Why so many invalid blocks?
Then central server tried to balance traffic between all three overloaded servers, which lead to partially working, but invalid blocks generating pool. This was caused by huge I/O latencies on first two crashed machines: when bitcoind is waiting to I/O operations (usually processing new bitcoin block), it is still accepting getwork submits, but submitted work is compared to non-actual blockchain.

Why bitcoind initially crashed?
I never see this before, which is also the reason why there wasn't any watchdog or another mechanism recovering pool from this type of issue. Actually restarting of bitcoind when they started to eat too much memory *should* be enough for the future, but threshold for doing that automatically is of course hard to test.

Was it an attack?
I have no evidence for following claims, but makes sense for me: I heard some rumors on the forum/IRC that is possible to overload bitcoind by sending malicious messages to its P2P port. I can imagine attacker waiting till I'll be offline and then started with such attack. Fix was pretty easy: bitcoind restart helped, so attack while I'm online could be recovered almost instantly. I also know that attacker who did previous DoS is on #bitcoin-dev channel, so detecting that I'm offline is pretty easy (and I also randomly mentioned that I'm going to few day holiday).  I'm usually online all the time, this was second weekend since December 2010 (pool start) in the row when I was offline without any chance to reach Internet connection.

Lesson learn
a) Don't take holiday
b) If I'll take holiday, don't tell it anybody .
c) Try to setup watchdog restarting bitcoind when it started to use too much memory.

Finally I'm very sorry for such troubles, but everything is resolved now, pool hashrate is again slowly climbing up and I'll try to setup such watchdog before I'll go offline next time.

[allinvain]

Also maybe add more RAM to your servers? With the fees that you're earning I'm sure you can afford to upgrade the server infrastructure But I think the main problem is the buggy bitcoind you use.

[slush]

Also maybe add more RAM to your servers? With the fees that you're earning I'm sure you can afford to upgrade the server infrastructure

I'm sure that it will eat everything what I'll 'offer'. It filled few GB of swap space before it finally crash...

But I think the main problem is the buggy bitcoind you use.

*If* there is such bug, it is *very* likely included also in upstream. Don't forget that bitcoind on pools is used under extreme load, so there might be hidden troubles which don't appear in your personal wallet... There were already similar issues: deadlocks in 'send'/'sendmany' RPC, huge performance issues in getwork & spam transactions in the network etc...

[allinvain]

Also maybe add more RAM to your servers? With the fees that you're earning I'm sure you can afford to upgrade the server infrastructure

I'm sure that it will eat everything what I'll 'offer'. It filled few GB of swap space before it finally crash...

But I think the main problem is the buggy bitcoind you use.

*If* there is such bug, it is *very* likely included also in upstream. Don't forget that bitcoind on pools is used under extreme load, so there might be hidden troubles which don't appear in your personal wallet... There were already similar issues: deadlocks in 'send'/'sendmany' RPC, huge performance issues in getwork & spam transactions in the network etc...

Hmm I see. So what's needed is an "industrial strength" bitcoin daemon - a lot of work and bug testing involved there. Would it be a good idea to setup the system in such a way that when the servers get overloaded to a certain point no new workers will be accepted to join the pool? Also is it possible to rewrite the bitcoind so it operates in a cluster configuration? This way you can have one bitcoind loaded to say only maximum 50% capacity.

[slush]

So what's needed is an "industrial strength" bitcoin daemon

Exactly. I know some people are working on highly scalable bitcoind replacement, but nothing for production use yet. It's sad that all pool failures for whole service history were related to bitcoind; I cannot remember single critical error in my code .

[marcus_of_augustus]

Maybe a small business partner or other worker to watch over pool when you are off-line? Meatspace watchdog.

bitcoind is open source so bug fixes are welcomed i'm sure.

[warweed]

see this ...



this just sucks

[dishwara]

Eagerly awaiting full status report 

Lesson learn
a) Don't take holiday
b) If I'll take holiday, don't tell it anybody .
c) Try to setup watchdog restarting bitcoind when it started to use too much memory.

Welcome back, rise again.
I differ slightly. Take holiday(It's must, coz humans born to live, not to work) after setting up watchdog & don't share that you in holiday.

[LobsterMan]

I was using deepbit's pool for a little while while yours was down slush, and I was wondering if you are planning to implement a pay per share model?

To me it almost seems like the people who are doing pay per share are getting slightly ripped off, and I much prefer the proportional method, but I am wondering anyway 

[slush]

Welcome back, rise again.
I differ slightly. Take holiday(It's must, coz humans born to live, not to work) after setting up watchdog & don't share that you in holiday.

My previous point about not taking holiday was of course joke. I'm currently working on monitoring measures - what is yet OK and what is beginning of the failure. Active monitoring will be working soon. And yes, I won't say anybody that I'm going off next time...

[Nythain]

I was using deepbit's pool for a little while while yours was down slush, and I was wondering if you are planning to implement a pay per share model?

To me it almost seems like the people who are doing pay per share are getting slightly ripped off, and I much prefer the proportional method, but I am wondering anyway 

Pay per share has its ups and downs, just not as drastic as score and proportional. I made more in 12 hours on deepbit doing pps with my 68Mh/s a few days ago than i have in 2x as much time on either slush or eligius.

[slush]

I was using deepbit's pool for a little while while yours was down slush, and I was wondering if you are planning to implement a pay per share model?

PPS is vulnerable to some kind of attacks (similar to classic sabotage), it was already discussed here. I know people like PPS (mostly beginners, before they realize that everything *really* converge to expected values and they are ripped of by 7%), but I don't want to open yet another doors to hurt the pool. I believe that proportional & score system is best option for serious miners, which is what I offer.

[slush]

Pay per share has its ups and downs, just not as drastic as score and proportional. I made more in 12 hours on deepbit doing pps with my 68Mh/s a few days ago than i have in 2x as much time on either slush or eligius.

Well, don't take it personally, but my previous post was exactly about people like you .

[Nythain]

Pay per share has its ups and downs, just not as drastic as score and proportional. I made more in 12 hours on deepbit doing pps with my 68Mh/s a few days ago than i have in 2x as much time on either slush or eligius.

Well, don't take it personally, but my previous post was exactly about people like you .

Heh, I'm not taking anything personally. I don't want to see pps on your pool, and you're pool is my pool of preference. I'm also a complete and total newb to the scene, but I'm picking it up fairly decently.

Side note, good work and thanks for getting things back up and running and really smooth so quickly since getting back from holiday. As soon as I actually turn over enough for a payout there's definitely a donation coming your way from me.