Eligius: 0% Fee BTC, 105% PPS NMC, No registration, CPPSRB

[wizkid057]

Greetings,

So, an update on last night's DDoS attack, in timeline form.

• at 04:01 UTC An email was received threatening to DoS eligius with 5 BTC ransom.  The full message was "you pay 5 bitcoin or x.x.x.x x.x.x.x x.x.x.x go down you have 1 hour" where x.x.x.x is replaced with some mostly non-public Eligius server IPs, specifically two core bitcoind nodes and the webserver's actual IP.  They didn't even provide a contact method or way to pay the ransom (not that I would have done so anyway).  Email was filtered as spam and I only found it after the fact also (sent from some bogus email address with a name of "Tina Turner")
• at 04:48 UTC There was a physical fiber connectivity issue at Eligius's primary data center.  This resulted in nearly 70% of the data center's local bandwidth capacity being lost.  Secondary connections were immediately put to use and were covering the mostly off-peak load and most services, including Eligius, were not impacted.  (Spoiler: this turned out to be unrelated to the attackers)
• at 05:16 UTC a DDoS began on Eligius.  External traffic saturated the secondary fiber connections from outside.  To make matters worse, some servers in the same facility as Eligius (unrelated to us) appear to have been compromised and used to DoS Eligius from inside the data center as well, causing latency, but not triggering my normal DDoS alerts since the external DDoS was mostly UDP amplification and filtered upstream.
• at 05:36 UTC Eligius miners, mostly unaffected by the DDoS's at this point aside from some work update latency, found a block, #387439 with hash 000000000000000008e051b41e7ada11e9931153a0bb02960ebb4a9e0374e404.  Eligius nodes attempted to submit this block to our primary nodes, secondary external nodes, and BlueMatt's relay network.  The first of these to be able to accept this block happened 46 seconds later due to extreme network latency within the data center.  By then the bitcoin P2P network had already seen a competing block, 000000000000000003e98e022c09c263e6b28f79cbc973a094444f649bbe4bcf by AntPool.  Eligius miners continued to mine on top of *e404, but about 10 minutes later BitFury found a block that built on top of AntPool's block and Eligius's block was officially stale.
• at 05:57 UTC The NOC staff, dealing mostly with the fiber loss issue, isolated the compromised machines inside the facility (no Eligius machines) that were contributing to the DDoS attack on Eligius.
• at 06:02 UTC The external DDoS against Eligius expanding to include core data center hardware and further saturated the secondary fiber connections beyond usability.
• at 06:03 UTC Eligius's mining servers automatically switched to a tertiary link with extremely limited bandwidth when connectivity was lost.  Coinbaser was disabled and block size limited while on this connection to save bandwidth, again all automatically.
• at 06:40 UTC Data center staff had mitigated the DDoS to the point where things were mostly stable on the secondary fiber link, with patches of extreme latency as the pipe was periodically saturated by either the DDoS or normal traffic.
• at 08:03 UTC The majority of the primary fiber connection was restored via a temporary link bypassing the physically damaged fiber link.  Latency with Eligius was back to normal at this point.
• at 11:02 UTC Primary fiber link was fully restored via a temporary link with full capacity.
• at 16:30 UTC Data center NOC staff's failure analysis concluded that faulty wiring in an underground conduit near to this section of fiber had caused excess heat which damaged the primary fiber link.

Initially I had thought it possible that the people threatening the attack may had been responsible for the fiber cut on the data center's primary connections.  After sharing the timing of the incident and the ransom note with the NOC, they were also skeptical of the coincidental timing.  Turns out the two were completely unrelated and were just independent issues.  However, if just one or the other had happened there would have been little to no impact to services, Eligius's or other's at the facility.  But since the attackers were able to leverage a partly crippled data center when they attacked, their attack was at least partly successful.  Fortunately non-Eligius servers at the data center were only mildly affected by this whole situation or I might be accelerating my migration to the new data center beyond my planned time table.

Total actual down time for Eligius's mining servers was under 45 seconds, measured from a remote monitoring server, around 06:02 UTC when the changeover to the tertiary link happened.  Unfortunately latency was very bad in bursts between 05:16 and 08:03 UTC, as high as 1 minute latency.  For the most part, miner connections were not affected aside from delayed work changes and delayed share submission responses.

The above is all for the mining servers.  The web server was unavailable for several minutes at a time while the lower priority DDoS mitigation for that setup took effect.

None of the "wake wizkid057 up" alerts I had setup triggered, since in the eyes of all of my monitoring the mining servers were all online, available, working on up to date work, etc, with DDoS mitigation doing its job correctly.  The switch to the tertiary link caused small connectivity loss, but not enough to drop all miner connections, and wasn't quite long enough to trigger an alert either.  Overall, the system performed well under the circumstances, IMO, and I'd be surprised if many other hosted services could survive such an incident.  Had I been awake and available from the beginning, I don't think there would have been much else I would have been able to do anyway.

But, in any case, I'm working on hardening the back end servers against a similar incident and adding even more redundancy to the block submission path out of Eligius's network.  Once the migration to our new data center is complete I'll even have a 4G LTE backup link available for this purpose.

TLDR:  There was a DDoS along with some data center network issues last night. All is well now.  No Eligius servers were compromised.

[1713266649]

1713266649

[1713266649]

1713266649

[btcmner]

last night's DDoS attack

By any chance, could there be some relation with the test servers you had recently set up ? Could their IPs have been nearby the IPs of the main servers, therefore giving some direction for location to the vandals ?

[wizkid057]

last night's DDoS attack

By any chance, could there be some relation with the test servers you had recently set up ? Could their IPs have been nearby the IPs of the main servers, therefore giving some direction for location to the vandals ?

The test servers would still be behind Eligius's normal DDoS protection, just on alternate ports.  I also hadn't provided info on them to anyone yet.

[btcmner]

last night's DDoS attack

By any chance, could there be some relation with the test servers you had recently set up ? Could their IPs have been nearby the IPs of the main servers, therefore giving some direction for location to the vandals ?

The test servers would still be behind Eligius's normal DDoS protection, just on alternate ports.  I also hadn't provided info on them to anyone yet.

Is there a recently joined worker or set of workers which stopped mining just at about the time the saturation attempt began ?

[paputa]

can i earn with nicehash and eligius?

[mopped]

Are there eu based stratum servers for Eligius? Did some digging and saw there use to be a totally separate eu pool... which I believe is no longer. I'm guessing there isn't since I'm not finding anything but thought I'd ask anyway. Thanks!

[wizkid057]

Are there eu based stratum servers for Eligius? Did some digging and saw there use to be a totally separate eu pool... which I believe is no longer. I'm guessing there isn't since I'm not finding anything but thought I'd ask anyway. Thanks!

There is no dedicated EU server, but I use geo-dns to try and place people on the closest relay possible, and I do have a relay in Europe.

can i earn with nicehash and eligius?

You can mine on Eligius from nicehash if you like... I personally don't see the point, since at best I think you'll break even or come out ahead by a few tenths of a percent on a good day.  But I won't argue with it, whatever works.

[Biodom]

Greetings,
TLDR:  There was a DDoS along with some data center network issues last night. All is well now.  No Eligius servers were compromised.

well, that's not all, right?
we lost a block due to...things, which may or may not be directly related to ddos.
did not exactly get the point why the block was not submitted on a timely basis ("because of extreme latency at the datacenter").
i don't think that the datacenter was ddosed, right? maybe you need to be at a better datacenter?
My personal loss ~$25
oh, well

[wizkid057]

Greetings,
TLDR:  There was a DDoS along with some data center network issues last night. All is well now.  No Eligius servers were compromised.

well, that's not all, right?
we lost a block due to...things, which may or may not be directly related to ddos.
did not exactly get the point why the block was not submitted on a timely basis ("because of extreme latency at the datacenter").
i don't think that the datacenter was ddosed, right? maybe you need to be at a better datacenter?
My personal loss ~$25
oh, well

So, I publicly post a detailed explanation of exactly what transpired that day (over two weeks ago now) with to-the-minute timestamps of everything that happened... and that's not good enough for you?

Exactly what I posted is exactly what happened.  The loss of primary connectivity at the datacenter caused what would have normally been an easily filtered DDoS attack to severely saturate secondary connectivity for the entire datacenter with hundreds thousands of machines competing for tiny slices of remaining bandwidth with latency as high as two minutes for a short period.  Nothing else that could have been done to prevent this.   Aside from putting in my own 4G LTE backup connection or having some alternate connectivity for the block submission paths in the future there would be nothing to be done to fix against this.

The arrangement I have with the two main datacenters that Eligius uses keep Eligius's cost well below what they should be for very stable connectivity and hosting.  I'm talking something like 10% of retail here, and 100% of donations go to covering hosting.  Sometime's it's not enough, and they let it slide because they know what is being run. They essentially subsidize our hosting a bit asking for nothing much in return mostly due to my prior unrelated relationships with them.  Get a better datacenter?  Good luck finding better ones than these at the prices we pay for hosting all of the equipment needed for Eligius along with the massive amounts of bandwidth needed for normal operation AND mitigating DoS attacks.

Put it this way.  In the past two years Eligius's downtime due to datacenter issues: About 45 seconds, 100% of which was from the incident two weeks ago that was well beyond their control.  That's 99.99993% connectivity uptime.  Electric power has also never been down for a single second in the nearly full decade that I've had a relationship with them.  I have a personal server there that's highly outdated with an uptime of something like 1500 days.

On top of that, I've gone out of pocket on 4G LTE backup connectivity for Eligius's rack (plus some unrelated servers) to ensure that this is extremely unlikely to happen again by always submitting blocks via the LTE connection in addition to the normal channels.

If you're asking for more than that from a 0% fee pool, please just mine elsewhere.

[Biodom]

Greetings,
TLDR:  There was a DDoS along with some data center network issues last night. All is well now.  No Eligius servers were compromised.

well, that's not all, right?
we lost a block due to...things, which may or may not be directly related to ddos.
did not exactly get the point why the block was not submitted on a timely basis ("because of extreme latency at the datacenter").
i don't think that the datacenter was ddosed, right? maybe you need to be at a better datacenter?
My personal loss ~$25
oh, well

So, I publicly post a detailed explanation of exactly what transpired that day (over two weeks ago now) with to-the-minute timestamps of everything that happened... and that's not good enough for you?

Exactly what I posted is exactly what happened.  The loss of primary connectivity at the datacenter caused what would have normally been an easily filtered DDoS attack to severely saturate secondary connectivity for the entire datacenter with hundreds thousands of machines competing for tiny slices of remaining bandwidth with latency as high as two minutes for a short period.  Nothing else that could have been done to prevent this.   Aside from putting in my own 4G LTE backup connection or having some alternate connectivity for the block submission paths in the future there would be nothing to be done to fix against this.

I was not trying to scald you, just pointing out that TLDR portion was incomplete without stating a fact that one block was lost.
Yes, it was stated in the long version as it being "stale". I did not see a statement like this: TLDR: was ddosed, lost one block, instead it was: pool/datacenter was ddosed, all is good now, nothing was compromised...hence my original post.

Essentially, apart from commenting I have no other issues. I don't come to this thread often, so again, apologies for bringing something of two weeks ago-this shows simply when i last looked at the thread than anything else. I am sorry that this caused some distress, which was NOT intended.

[wizkid057]

Greetings,
TLDR:  There was a DDoS along with some data center network issues last night. All is well now.  No Eligius servers were compromised.

well, that's not all, right?
we lost a block due to...things, which may or may not be directly related to ddos.
did not exactly get the point why the block was not submitted on a timely basis ("because of extreme latency at the datacenter").
i don't think that the datacenter was ddosed, right? maybe you need to be at a better datacenter?
My personal loss ~$25
oh, well

So, I publicly post a detailed explanation of exactly what transpired that day (over two weeks ago now) with to-the-minute timestamps of everything that happened... and that's not good enough for you?

Exactly what I posted is exactly what happened.  The loss of primary connectivity at the datacenter caused what would have normally been an easily filtered DDoS attack to severely saturate secondary connectivity for the entire datacenter with hundreds thousands of machines competing for tiny slices of remaining bandwidth with latency as high as two minutes for a short period.  Nothing else that could have been done to prevent this.   Aside from putting in my own 4G LTE backup connection or having some alternate connectivity for the block submission paths in the future there would be nothing to be done to fix against this.

I was not trying to scald you, just pointing out that TLDR portion was incomplete without stating a fact that one block was lost.
Yes, it was stated in the long version as it being "stale". I did not see a statement like this: TLDR: was ddosed, lost one block, instead it was: pool/datacenter was ddosed, all is good now, nothing was compromised...hence my original post.

Essentially, apart from commenting I have no other issues. I don't come to this thread often, so again, apologies for bringing something of two weeks ago-this shows simply when i last looked at the thread than anything else. I am sorry that this caused some distress, which was NOT intended.

It was a TLDR version for a reason.  I just won't post a summary anymore and there won't be any problems.

I recall the question posed as to why I was moving things to a new data center, but can't seem to find it to quote.  The new data center is run by the same company as the old one, basically... but is physically closer to my location so it's easier for me to work on things when I need physical access to the hardware.  Plus the new data center has much more capacity than the current one, which has pretty much reached the limits of physical space for hardware.  That said, physical proximity came in handy last night.

As for news, at about 3:45AM my time this morning I was alerted to a connectivity loss of one of the new Eligius servers that was handling some of the pool's stratum traffic.  (There are multiple, and the others were picking up the slack for the most part).  It wouldn't respond at all via the network or KVM, and remote power cycling wasn't bringing it back online.  I headed over to the data center and got there at about 4:15 AM (it's only about 15-20 minutes from me... closer to 15 at 4AM on a Sunday morning) and investigated.  Power supply in the machine wasn't working, so swapped it out with a spare from the data center's stock, and all was well.  At about 5:30AM I was back in bed.

Miners were mostly unaffected aside from a connection drop and reconnect around the time the one server went offline.

I much prefer my own hands to some NOC tech's grubby remote hands, not that they've ever been a problem. The staff at the new data center doesn't even have a key to the rack that has Eligius's servers while I'm in town (which is most of the time).

Anyway, all servers are online and functioning normally and I'm slowly continuing the longest migration in history to the new data center.  I'm going to keep a couple of servers online at the original data center for backup purposes for as long as they allow me to do so.  Mainly I've been working to make sure mining is mostly unaffected by the whole migration process.

Happy Holidays!

[Questor]

Anyway, all servers are online and functioning normally and I'm slowly continuing the longest migration in history to the new data center.  I'm going to keep a couple of servers online at the original data center for backup purposes for as long as they allow me to do so.  Mainly I've been working to make sure mining is mostly unaffected by the whole migration process.

Happy Holidays!

Thanks wizkid057.  Due in part to the skyrocketing difficulty making solo mining less viable, share issues, and the desire to keep the little guy alive, we have been throwing a bunch of hash back this way, and with appropriate donations that are still less than other for profit pools with (somewhat unrealistic) hope for the future!

Cheers all!  It's been a bumpy ride, but we're still here

[papadrams]

Hi guys,
I bought my first miner 2 days ago and (after a lot of reading) point it to eligius.. I have one question. I can see in my profile that i am going to enter payout queue when i reach 0.04BTC. I suppose i cant hold my BTC there after that point.. Furthermore, i see in payout queue someone who hasn;t been paid for over 11 months now..
Can anyone explain to me how the payment system works in eligius?

[wizkid057]

Hi guys,
I bought my first miner 2 days ago and (after a lot of reading) point it to eligius.. I have one question. I can see in my profile that i am going to enter payout queue when i reach 0.04BTC. I suppose i cant hold my BTC there after that point.. Furthermore, i see in payout queue someone who hasn;t been paid for over 11 months now..
Can anyone explain to me how the payment system works in eligius?

Payout queue is prioritized based on payout age.  So, the people at the top of the list (with multi-month balance ages) likely mined very very slowly to get to their minimum payout, or they stopped mining for a while with a tiny balance and then started back up a long time later, and thus when they finally did they entered the queue at a pretty high priority since their balance age (basically last payout date) was a long time ago.

[M8BWNNRFMNdak68c]

can you please explain if this payout is correct, or if there are 0.4 BTC lost because of a very long round?

you can see the growing estimate on 03. January
then on 4. January it fades away just like on a normal PPLNS pool
on 5. January there was again mining - and luckily a block was found, so that the mining was directly payed out. ( 0.25999983 BTC )
some hours ago there was also mining, and a block found. here the Recent Backpay kicks in..
now there are 0.8 BTC ( and growing ) in the http://eligius.st/~wizkid057/newstats/payoutqueue.php

so let's sum up:
5. January: correct
4. January: correct
3. January: 0.4 BTC still missing

so i wonder if they are lost or will show up in a confusing manner? we are already on the third block after this event..
they are neither in the payout table, nor in the balance graph, nor in the payout queue..
it looks like the "shelved shares" got deleted without payment? ( http://eligius.st/~gateway/faq-page#t14n35 )
my last hope is the green "maximum reward" graph.. what does it indicate? when could i get this?

[wizkid057]

can you please explain if this payout is correct, or if there are 0.4 BTC lost because of a very long round?

http://eligius.st/~wizkid057/newstats/userstats.php/12QVpo48NA5LGgcz28qt1dLX7bpa9fqLJr

Looks like you started mining on the 3rd when the pool was about ~6 hours into the long round for block 391,783.  You mined for a few hours and built up some shares in the top of the share log and stopped mining at about ~18:15.  All of those shares remained in the top 25 BTC of the share log until about ~23:00.  Then active miner shares started burying your shares in the share log, resulting in the gradual estimate drop.  At about ~02:40 you no longer had any shares in the top 25 BTC of the share log since you had stopped mining over 8 hours prior.  It was another 21 hours after this that block 391,783 was finally found, with your shares from the previous day well buried in the share long under active miners.

Then you mined a little on the 5th.  100% of those shares were paid by block 391,811.  You started mining again later on, and soon after the pool found block 391,880 and paid for the shares you submitted since starting again.  And the remaining of your graph includes estimates for the mining you've done since that block was found (estimates only, not rewarded until a block is found).

You have some amount of shares from 2015-01-03 buried in the share log still.  Perhaps we'll get some lucky blocks and un-bury them, perhaps not.  You're not a full time Eligius miner, so, your percentage of shares rewarded will vary greatly since your shares are in pockets of the share log and not evenly distributed like a full time miner's would be.

In any case, looks completely normal to me. 

Edit/update:  Note that no shares at Eligius are ever "deleted" or otherwise forgotten, unlike PPLNS type pools.  The maximum reward graph is the maximum you could possibly be paid by Eligius for the work you've submitted if 100% of your shares in the share log are rewarded.  The share log is a LIFO buffer, so the latest shares submitted by miners are paid first, which is why active miners always have shares to be paid and miners who go inactive may or may not have shares in the top of the share log when a block is found.

[M8BWNNRFMNdak68c]

ah okay.. thank you for your fast and detailed reply..

Last In - First Out.. i think i got it.. when we are very lucky, we can pay older and even older shares ( the newest unpaid ones have the highest priority ). so as we are working with statistics: in an ideal world the average luck for each round should be 100% - so if we are having "bad luck" now, it might possible that there is much luck in the future.. so my shares could get paid in 1 day, 1 month or possibly even in one year.. everything is possible..

but what is the difference between the "share log" and the "payout queue"? obviously not every share from the log will get in the queue, but why?
my shares from today all get in the queue.. but why can't we see the shares from the 3rd? as the queue is 150 btc long, they should be in a quite high position? ( like "age: 2 days" )
is it possible for one address to be on 2 positions in the payout queue?

[wizkid057]

ah okay.. thank you for your fast and detailed reply..

Last In - First Out.. i think i got it.. when we are very lucky, we can pay older and even older shares ( the newest unpaid ones have the highest priority ). so as we are working with statistics: in an ideal world the average luck for each round should be 100% - so if we are having "bad luck" now, it might possible that there is much luck in the future.. so my shares could get paid in 1 day, 1 month or possibly even in one year.. everything is possible..

but what is the difference between the "share log" and the "payout queue"? obviously not every share from the log will get in the queue, but why?
my shares from today all get in the queue.. but why can't we see the shares from the 3rd? as the queue is 150 btc long, they should be in a quite high position? ( like "age: 2 days" )
is it possible for one address to be on 2 positions in the payout queue?

Well, I'm going to clean up the queue shortly, so, that should help.

The payout queue is for actual on-the-blockchain payments to miners.  The share log is a record of every share ever submitted to miners, which could be rewarded (earned and ready to be paid) or unrewarded (not actually earned yet).

The payout queue basically looks at the share log and sums up everyone's rewarded but not yet paid out shares, checks to see if they have reached their minimum payout amount, and then prioritizes them based on the time of their last actual payment from the pool.  So, if the last time you were paid was months ago due to slow mining or whatever reason, then when you reach the payout queue that is the time used for prioritizing and you enter the queue at that point (probably pretty close to the top of the list).  If you get paid regularly or even in each block, your payout age is relatively fresh.

A simple somewhat incomplete summary is that the share log is internal accounting, and the payout queue is for actual payouts based on that internal accounting.

Hope this makes sense.

[M8BWNNRFMNdak68c]

hmm okay ..  but now we have two contradicting statements:
1. The Shares are paid LIFO
2. Shares from a user without payment gain priority over shares from a user with continous payments.

it's hard to combine that in my mind.. wouldn't i get an advantage if i change my address contiously..
so my shares will always be older than the shares of the honest miners?

and still not clear why my 0.4 BTC are not listed in the payout queue?

but nevertheless, i think we can trust you and believe that the algorithm produces fair payments..
maybe there should just be some more documentation.. the FAQ does't speak about the priority gain though aging..

[wizkid057]

hmm okay ..  but now we have two contradicting statements:
1. The Shares are paid LIFO
2. Shares from a user without payment gain priority over shares from a user with continous payments.

it's hard to combine that in my mind.. wouldn't i get an advantage if i change my address contiously..
so my shares will always be older than the shares of the honest miners?

and still not clear why my 0.4 BTC are not listed in the payout queue?

but nevertheless, i think we can trust you and believe that the algorithm produces fair payments..
maybe there should just be some more documentation.. the FAQ does't speak about the priority gain though aging..

Oh, no.  Let me clarify.

Your #1 point is correct.  #2 is a little off.

The payout queue only cares about shares already rewarded (or shares that would be rewarded if the pool found a block at that moment, but let's keep it simple).  Shares are rewarded regardless of balance/payout age to whatever address they go with from the share log LIFO.  The payout queue simply prioritizes full payouts of already rewarded shares based on balance age.  Generally the payout queue is kept to 1 block worth of payouts anyway, so this usually doesn't matter.

So, the shares that are in the share log but not yet rewarded (like the shares you submitted on the 3rd during that awful round we had) are not yet in the payout queue since a block has not been found that has rewarded them yet.

Unrewarded: In share log, but not yet earned.  Shelved shares waiting to be rewarded by a block.  No block has been found to reward these shares.  These are not owed payment by the pool.
Rewarded: A block has been found where the share was in the top 25 BTC of the share log and these shares are earned and are added to the user's balance due, and thus the balance that is used by the payout queue.  The pool owes the miner payment for these shares (assuming the block rewarding them remains valid, not orphaned/stale) per payout rules (minimum payout threshold, etc).
Paid: Paid to the miner on the blockchain.  Pool has no further obligation to these shares.

The shares you submitted on the 3rd are still unrewarded (shelved shares).

Hopefully that will clear things up.