Good to know that somebody is using sierrachart . I should release new version fixing some issues soon...

Pool is going fine, today was just bad luck, nothing more. warweed, you don't need to apologize, it's your rig.

Initial problem was that bitcoind ate few GB of memory. It was first time I've seen that and cannot tell if it is bug or some kind of P2P attack as mentioned in my previous post.

Yes, it would be very nice from miner developers...

In fact, I started with repair immediately when I came home; I still had jacket and shoes when pool started to work again

You should open separate thread in marketplace or technical discussion - 15 BTC is going to be serious wealth and I'm sure there are some skilled programmers interested in getting their first bitcoins for some fun with Twisted...

Well, don't take it personally, but my previous post was exactly about people like you .

PPS is vulnerable to some kind of attacks (similar to classic sabotage), it was already discussed here. I know people like PPS (mostly beginners, before they realize that everything *really* converge to expected values and they are ripped of by 7%), but I don't want to open yet another doors to hurt the pool. I believe that proportional & score system is best option for serious miners, which is what I offer.

My previous point about not taking holiday was of course joke. I'm currently working on monitoring measures - what is yet OK and what is beginning of the failure. Active monitoring will be working soon. And yes, I won't say anybody that I'm going off next time...

I meet one man which speculated thousands of dollars on bitcoin/mtgox. I was surprised that he doesn't know too much about how this exchange works technically. For example that he can place order to some price level and wait until it is filled by others and NOT move the market by his order...

I can confirm, it start to idle after some time (but previously reported connection troubles are solved with this release).

Exactly. I know some people are working on highly scalable bitcoind replacement, but nothing for production use yet. It's sad that all pool failures for whole service history were related to bitcoind; I cannot remember single critical error in my code .

I'm sure that it will eat everything what I'll 'offer'. It filled few GB of swap space before it finally crash...


*If* there is such bug, it is *very* likely included also in upstream. Don't forget that bitcoind on pools is used under extreme load, so there might be hidden troubles which don't appear in your personal wallet... There were already similar issues: deadlocks in 'send'/'sendmany' RPC, huge performance issues in getwork & spam transactions in the network etc...

What happen?
On Saturday's evening (UTC), bitcoin daemons on two independent pool servers ate all available memory (RAM & swap) and fall to massive I/O operations, which make servers extremely slow. Central server switched traffic to the last machine, but this machine was unable to handle all traffic and some RPC calls started to timeout. Then those miners getting RPC timeouts probably started to ask server even more frequently (miners don't implement any "polite waiting" in the code and send new RPC call immediately after error/timeout). Those thousands of requests per second were the last piece to shut down last working machine .

Why so many invalid blocks?
Then central server tried to balance traffic between all three overloaded servers, which lead to partially working, but invalid blocks generating pool. This was caused by huge I/O latencies on first two crashed machines: when bitcoind is waiting to I/O operations (usually processing new bitcoin block), it is still accepting getwork submits, but submitted work is compared to non-actual blockchain.

Why bitcoind initially crashed?
I never see this before, which is also the reason why there wasn't any watchdog or another mechanism recovering pool from this type of issue. Actually restarting of bitcoind when they started to eat too much memory *should* be enough for the future, but threshold for doing that automatically is of course hard to test.

Was it an attack?
I have no evidence for following claims, but makes sense for me: I heard some rumors on the forum/IRC that is possible to overload bitcoind by sending malicious messages to its P2P port. I can imagine attacker waiting till I'll be offline and then started with such attack. Fix was pretty easy: bitcoind restart helped, so attack while I'm online could be recovered almost instantly. I also know that attacker who did previous DoS is on #bitcoin-dev channel, so detecting that I'm offline is pretty easy (and I also randomly mentioned that I'm going to few day holiday).  I'm usually online all the time, this was second weekend since December 2010 (pool start) in the row when I was offline without any chance to reach Internet connection.

Lesson learn
a) Don't take holiday
b) If I'll take holiday, don't tell it anybody .
c) Try to setup watchdog restarting bitcoind when it started to use too much memory.

Finally I'm very sorry for such troubles, but everything is resolved now, pool hashrate is again slowly climbing up and I'll try to setup such watchdog before I'll go offline next time.

jedi95, I tested new version and works perfecly on my pool. Thanks a lot, 15 BTC on the way!

I need to check three blocks manually and mark them in database as valid or invalid, which I'll do today. Until then, reward from those three blocks is pending as 'unconfirmed'...

Warweed, I absolutely understand your feelings, but please wait to full report of outage. I'm doing my best, but I cannot assume all kinds of failures. This was serious one and I solved it ASAP. I'm not sitting 24/7 on the line waiting on troubles, I need holiday time to time.  Trust me that you're not alone who is upset with weekend troubles... Btw I'm not taking fees from failed blocks, of course.

This happen when fails some database writes in time of finding block. I have to fix those things manually, which I didn't do yet. Mining works, but I need to fix this stuff today.

I'm too much busy to care about your pool, my friend. I never tried anything with your pool. No hoping, no DDoS, no breaking passwords. I'm really interested in your evidence, because you're talking bullshits. I totally don't understand why you hate me so much.



Oh my god, why? I don't care about you at all.


My god, how is this related to DDoS, password breaking and all other shits? I wrote you that *if* I'll try pool hoping, I'll do it in open way, as real proof, not because I want to earn bitcoins.

Please, ignore me, right? And please stop saying that I'm an attacker without any evidence. Btw I was (probably) attacked during this weekend, too. There is somebody trying to shut down some important bitcoin services (like mtgox and, ehm, me). Why are you so sure that *I* am the attacker?

Pool was in unstable condition for more than 2/3 time of the round #4288, hard to take those numbers seriously. I just wake up to check pool status and everything operates normally, new round started few minutes ago.