Eligius: 0% Fee BTC, 105% PPS NMC, No registration, CPPSRB

[AbiTxGroup]

Something strange is happening again with the payout queue.

I entered the payout queue in the late teens of blocks ahead and then it went all the way to 26 blocks ahead.  I have since dropped down to 22 blocks ahead and now I am starting to climb back up again, meaning more blocks ahead of me.

Why does the blocks before me in the payput queue go both up and down.  I thought that once you enter the queue, any new blocks solved would just move you closer to the payout, not farther back in the queue.

If this has been answered before, then assisting with a link would be helpful, no need to repost the info.

Thanks.

[nottm28]

Mitigating DDoS attacks is pretty basic, depending on whether they're trying to attack the TCP stack (like SYN floods) or the protocol (like that NTP reflection attack that was going on a couple of months ago).  More servers won't help unless you have a genuine scaling problem.

Care to elaborate on the pretty basic way a DDoS can be mitigated? I'm sure plenty of people will be interested...

[wizkid057]

I'm going to repost/quote baddw's explanation here, since it is a decent run down of the process.  Could use a little editing/fixes, but I'll leave it alone for now since it gets the idea across.

I am going to attempt here to write a full description of how Eligius pays pool contributors, including a description of both the CPPSRB system and the payment queue.  My goal is something that can be linked or copied and pasted whenever somebody comes with a question like this.  I want to cover a lot of bases and give some solid examples.  Everybody, please let me know if there are any corrections needed.

CPPSRB SYSTEM:
First, some terms.  Given the Eligius hashrate, and the Bitcoin difficulty, there is an expected number of shares before Eligius finds a block.  Let's call this number N.  A pure PPS (Proportional Payment) would immediately credit each and every share with exactly 25/N BTC.  This is called the PPS value.

Every share submitted goes into a stack.  The Eligius system refers to it as a "shelf" system, but in basic computer science terms, it is a stack.  It is a LIFO (Last-in First-Out) system, like a stack of plates in a cafeteria.  Each share is "worth" its PPS value, but it might or might not ever be credited to your balance.

Each time a share is found, the system immediately awards the top N shares in the stack, and each of them is credited with its PPS value.  In this way, it is somewhat like a PPLNS (Payment Per Last N Shares) system.  However, it differs in that once a share is awarded, it is removed from the stack permanently.  A PPLNS system will double-pay some shares some of the time (when the round is shorter than N shares, it will double-pay the more recent shares from the last round), and discard shares some of the time (when the round is longer than N shares, any shares older than N are discarded and never paid).

So let's say that N is 100 shares, i.e. we expect to find a block every 100 shares.  (I'll keep the values low here just to keep things simple.  Multiply by some billions to get the actual numbers.)  Shares are added to the top of the stack as soon as they come in.  First round, the pool finds a block at 120 shares, longer than expected.    So the most recent 100 shares are credited with their PPS value, and the first 20 shares are kept on the stack.  The next round, the shares are added to the stack as they come in.  This round, the pool finds a block at 60 shares, shorter than expected.  Since those most recent 60 shares are on top of the stack, they are credited first.  Since 60 is less than N, the next 40 shares in the stack are paid out.  This includes the 20 shares that were put there in the last block.  The next 20 shares in the stack will then be credited, from whenever they were accumulated.

So, you assume credits in your account in this way.  This is the first stage.  The next stage is the payout queue. 

PAYOUT QUEUE:
You can enter the payout queue once your account balance reaches its threshold, which can be set manually.  Once you hit your threshold, you are entered into the payout queue.  The queue is just that, a queue, not a stack.  It is generally not FIFO or LIFO like the CPPSRB system.  Instead, your position in the queue is determined solely by when you were last paid out.  If you were last paid out 36 hours ago, you will be ahead of anybody who was last paid out 35:59:59 or less recently.  So your position in the queue will change all the time, as the payout queue is reduced (when blocks are found and people are paid out) or when the queue ahead of you gets bigger (because people who have not been paid as recently, have hit their threshold and entered the queue ahead of you).

In setting your payout threshold, you have to weigh the benefits of making it long or short, relative to your hashrate.  If you set it to be paid more frequently, your payouts will be very inconsistent as there will constantly be people jumping the queue ahead of you as they hit their thresholds.  It can be quite discouraging to be close to the front of the payout queue, and then for some whale to hit their threshold and their payments will be 150BTC or something, consuming the next 6 blocks and putting you 6 blocks further back in the queue.  On the other hand, if your threshold is set high, so that you are paid out (say) once a week, you will generally be at or near the very front of the queue every time you hit your threshold.  It seems that the typical choice is to set it to pay out approximately once every day, or 24 hours. 

It is important to note that as your unpaid balance grows, the amount that you will receive in your next payout grows as well.  Just because you hit your threshold doesn't mean the system stops crediting your shares; and the system will pay out the full owed amount (as of the previous block) whenever you come up to the front of the queue.  So say have your threshold set to pay you out roughly every 24 hours, which is equal to ~0.1 BTC given your hashrate.  Somehow there is a backlog in the queue, and you don't actually get paid for 48 hours.  When that payout occurs, you will be paid out ~0.2 BTC.  The next time, it takes 36 hours for you to make your way to the front of the queue, so you are paid out ~.15 BTC.  So your payment per mining hour will be consistent, but the payments might be inconsistently spaced.

Payouts are typically handled in the coinbase transaction when the block is mined.  This means that the coins are mined directly to your mining address, and the pool never holds the coins in its wallet addresses.  However, sometimes something gets screwed up, and the CPPSRB system goes into failsafe mode, and all mined coins are sent into Eligius's cold wallet until they are paid out in a manual payout by wizkid057.

[wizkid057]

Mitigating DDoS attacks is pretty basic, depending on whether they're trying to attack the TCP stack (like SYN floods) or the protocol (like that NTP reflection attack that was going on a couple of months ago).  More servers won't help unless you have a genuine scaling problem.

Care to elaborate on the pretty basic way a DDoS can be mitigated? I'm sure plenty of people will be interested...

Mitigating DDoS attacks is never "basic".  Mitigating the DDoS attacks that have been ongoing for quite some time against Eligius is time consuming, requires cooperation of many different entities, and is a genuine pain in the ass.  But, I'd much rather mitigate than give the attackers any satisfaction.

[AbiTxGroup]

Thanks for reposting the payout queue info wizkid057, that was the info I was questioning. 

I have my payout set to low as I have added a lot of miners without readjusting the threshold. I will adjust it now. Currently I vary from receiving a payout every 2-to-3 days but sitting in the queue for most of that time bouncing around.

[bolverk]

Mitigating DDoS attacks is pretty basic, depending on whether they're trying to attack the TCP stack (like SYN floods) or the protocol (like that NTP reflection attack that was going on a couple of months ago).  More servers won't help unless you have a genuine scaling problem.

Care to elaborate on the pretty basic way a DDoS can be mitigated? I'm sure plenty of people will be interested...

Mitigating DDoS attacks is never "basic".  Mitigating the DDoS attacks that have been ongoing for quite some time against Eligius is time consuming, requires cooperation of many different entities, and is a genuine pain in the ass.  But, I'd much rather mitigate than give the attackers any satisfaction.

It depends on the attack, but most of the garden variety DDoS is basic, but you seem adverse to accepting anyone's help or expertise, Wiz.

SYN floods are very basic unless the packet rate actually exceeds your available bandwidth.  There's not much you can do without the help of your backbone provider if they can actually fill your pipe.  But if it's just a socket threshold limit on the server itself there's a couple of ways to protect yourself, even without a decent accelerator or IDP.  Basic steps like enabling SYN cookies, increasing your syn backlog queue, lowering SYN and SYN/ACK retries, etc.

Beyond that you could enable packet marking so you can log stale SYNs and feed them to a program (like autofwd) in order to semi-permanently firewall hosts generating those packets.  If those packets are forged, and not generated directly by bots, there may be some collateral damage.

All of this is even easier if you're comfortable with OpenBSD.

Application protocol level DDoS, depending on the protocol, can be a bit hairier.

[Luke-Jr]

When it comes to Bitcoin, DDoS means 30+ Gbit/sec :p

[bolverk]

When it comes to Bitcoin, DDoS means 30+ Gbit/sec :p

Are you saying you have a cluster of servers sitting on multiple OC-255s?  Like I said, if they can actually fill the pipe with SYNs you're pretty much screwed, you need help from your backbone provider.

That said, I haven't seen any significant detail divulged on the attack.  No one's said what they're attacking, the IP stack, the application, what.  Are they overrunning the pipe or server resources?  If the latter there's more than likely a few things you can do to scale just to make it harder for them to effectively DDoS you.

[wizkid057]

A semi-well known fact about me:  my real life job includes DDoS mitigation at the ISP level for multiple ISPs.

This particular attack against Eligius has taken almost every form possible... UDP reflection attacks (DNS, NTP, SNMP, etc... 30+ gigabit at times), TCP SYN attacks (over 20 gigabit peaks), botnets directly flooding pool ports (multiple gigabit), botnets attempting application layer (stratum and HTTP) attacks (varies up to several gigabit and > 100k connections), HTTP request floods from botnets and other amplification (wordpress being one), hanging TCP connection attacks, various attack attempts against public facing bitcoinds, flood attacks against upstream routers, social engineering attempts (someone has contacted the abuse@ addresses for some nodes claiming Eligius is DoS attacking them, lol, presumably in an attempt to stir trouble with our hosts), and probably a ton of other things that are just automatically filtered/ignored.

[bolverk]

A semi-well known fact about me:  my real life job includes DDoS mitigation at the ISP level for multiple ISPs.

This particular attack against Eligius has taken almost every form possible... UDP reflection attacks (DNS, NTP, SNMP, etc... 30+ gigabit at times), TCP SYN attacks (over 20 gigabit peaks), botnets directly flooding pool ports (multiple gigabit), botnets attempting application layer (stratum and HTTP) attacks (varies up to several gigabit and > 100k connections), HTTP request floods from botnets and other amplification (wordpress being one), hanging TCP connection attacks, various attack attempts against public facing bitcoinds, flood attacks against upstream routers, social engineering attempts (someone has contacted the abuse@ addresses for some nodes claiming Eligius is DoS attacking them, lol, presumably in an attempt to stir trouble with our hosts), and probably a ton of other things that are just automatically filtered/ignored.

It's nice to get some hard info on what's going on.  I, BTW, work for a backbone provider, specifically for managed IP services.

I am surprised that Eligius is operating with that level of network connectivity, as a mostly unfunded operation I has thought your resources were a lot more meager.  That said, at the point your pipe is getting filled your provider should be willing to either provide some sort of CoS shaping and/or basic filtering for you, to keep pipe congestion down.  For what gets past that I would assume you've got some basic stateful firewalling in place, at least.

Some of your other attacks, like the HTTP requests, while intended as a DDoS, are great at pointing out scalability problems, for which there's probably some low hanging fruit.  Some squid boxes, ICAP, memcached, etc.  I haven't looked at the stratum protocol so I couldn't guess what your problems would be there.

Sounds like you guys are definitely getting a trial by fire.

[DPoS]

How well does CloudFlare deal with many of the attacks listed?

could some things be gained from using them or building something similar and more specific for mining pools?

[wizkid057]

Cloudflare only helps with http/https last I checked.  And I would rather not centralize yet another bitcoin related service behind Cloudflare.

As for the effectiveness of the attacks, the only noticeable effects lately have been slightly delayed response times with the web server.  The pool servers have been generally unaffected.

The web server setup definitely is not as scalable as I would like, but that falls back to the mostly unfunded part of the problem.

[wizkid057]

Awesome work wizkid. Just a small thanks for all of the hard work.

https://blockchain.info/tx/abb641b1c23447050c7582bddeb92e3e6882f5365ec69656ac67d2d4a066b571

This almost slipped through the bitcointroll.org cracks...

Thank you very much, gigavps.  Much appreciated.  Definitely will put towards upcoming improvements.

[jamesg]

Awesome work wizkid. Just a small thanks for all of the hard work.

https://blockchain.info/tx/abb641b1c23447050c7582bddeb92e3e6882f5365ec69656ac67d2d4a066b571

This almost slipped through the bitcointroll.org cracks...

Thank you very much, gigavps.  Much appreciated.  Definitely will put towards upcoming improvements.

Cold storage catch up would be really nice. 

[wizkid057]

Awesome work wizkid. Just a small thanks for all of the hard work.

https://blockchain.info/tx/abb641b1c23447050c7582bddeb92e3e6882f5365ec69656ac67d2d4a066b571

This almost slipped through the bitcointroll.org cracks...

Thank you very much, gigavps.  Much appreciated.  Definitely will put towards upcoming improvements.

Cold storage catch up would be really nice. 

I'm actually finalizing some code that will make cold storage catch ups much less frequent and I was planning on a catch as soon as I finish phase 1 of that code

[gorgatron]

Okay, I see some wonkiness took place last night for a few others, but not many.

My stats were just a bit off (not an issue, could be latency), but there was a stat/issue that was troubling. My KnC (moved to the stadard port after FW update by WK's writ!) just dropped out of the pool. No failover. Hashrate dropped from 143 GH/s to ~800 MH/s. Tried to get it going, and no dice.

My BFL units still showed they were hashing, but the stats for that ~175 GH/s dropped out, too. All that was showing was ~70 GH/s from my TechnoBit miners.

This was at around 2 am CST, I want to say, but can't be exact, unfortunately. BFGminer showed that the BFL units were hashing again at Eligius, but stats still didn't reflect this. This was an hour ago when I tried again... or was able to try again. I'm sure it was hshing, as I saw a bit of buildup in shares, so it seems the hashrate stat was mainly affected.

I've mainly just stuck to the pool despite stats, but when a miner drops the pool and won't reconnect, I decided to switch. I've taken my small op to another pool for the next hours until I hear more feedback from the pool. If all is kosher again, I'll try to move the KnC. If it will connect to the pool, I'll be good. As mentioned, I don't worry about the stats as long as I can see shares being processed. That stopped for both my BFL units (using bfgminer) and the KnC (also bfgminer). maybe just a fluke on my end with the BFL units. I just had to rebuild my system, and it's buggy. Doesn't explain the KnC instance, nor why the TechnoBit units still showed in my stats. Just odd. I've read the rest of the thread from the days I was out of commission, and noticed all the posts regarding DDoS, but didn't think it would affect the pool, nor do I still think it necessarily would. I might attribute the oddness I saw in the stats page to that, but the rest is odd. I've witnessed the network drop before, where I couldn't access Eligius nor two failovers - switched to PPC for a few hours - but this was isolated to Eligius, as far as I could tell. I had forgotten my BTCguild account and made a new one last night and connected right away. I'm there for now (until I can build up a small payout, and then try Eligius again).

Anyone else notice anything remotely similar? Just trying to figure out what might be on my end, what's up with the KnC, and why the TechnoBits never fell. lol - No panicking, just curious if anyone noticed the same issues, and if they somehow resolved them. Hope to switch back later. Just upped my donation, but guess I wasn't quick enough. lol jk  Again, some of this may be on my end. I've had a lot of issues getting the BFL units working properly after rebuilding y system, and my KnC has been wonky off-and-on for a few weeks. You know, the usual.

I seen same thing but on TechnoBits units. It went from 1TH to almost zero. but my total heshrate was constant. It looks like shares were not credited to worker but to my general address only. No idea why this happen. Switching off and then back on the pool solved that.

yeah, tried that. no dice. Oddly enough, two of my TechnoBit units burned up (almost literally) about 48 hrs later. I believe they were the source of another issue plaguing my system. I'm down 50 GH/s now, but at least my gear is stable.

I think I was in the middle of a post when they finally fried and tripped the PSU, causing the PC to power down. No more gear seems to be affected, so I'm cool with that.

Still, despite WK's mentioning that any loss of connectivity with the pool had to be on my side, I find it odd that the KnC just dropped out completely (also didn't switch to failover, which is odd and annoying, but likely because I was supposedly still hashing @ ~800 MH/s like an over-powered BFL FPGA lol!!!), and that then my BFL gear did the same. As I've mentioned in other posts long ago, I've had it happen that I cannot connect to any pool [that I trust], and since I use no other pools on the same port Eligius uses, or at least not BTC pools on that port and that's all I tried, it was strange that only Eligius was affected on my end, except the now deceased TechnoBit units. I've switched back to Eligius after taking care of the aforementioned HW issues, and assuming since WK says the problem was on my end, I'd likely hook up and be hashing away once more... And so it is.

I just don't like that this was the only pool affected on my end out of 5 I will dare connect to. I just went to BTC Guild for a few days to keep mining and now Eligius is working. Does anyone have ideas as to why Eligius may have been blocked by my hardware (KnC/BFL, both running bfgminer)? In case you've read one of my long-ass posts, I don't talk about/bitch about tech issues at the pool. I'm in no way qualified to do so. I can make things work, and can solve local problems, for the most part, but as to why hardware my bock access, I have no understanding. I appreciate WK making the short comment that the problem isn't on the pool's end, as that's a start, but it doesn't tell me anything, as I lack the background knowledge to understand why it would come from my end and not the pool. If it had been the pool, that would have been simple to explain away, and I wouldn't really worry, since someone else with experience is running the show. I want to know how to avoid the issue on MY end, or if it's possible.

I know I haven't given much technical info to pinpoint a cause or narrow group of causes, but I've made it clear that I don't know where to start. TO ANYONE: If you have a spare moment to put some thought into this, let me know what kind if info you'd need from me, and if I can obtain it, I will. I've essentially rebuilt my PC over the past week, and then had to install a new hard drive a day after I thought i was finished (that's when the TechnoBit units fried, and then my hard drive decided it didn't like to boot... just thankful I was able to connect it later to extract some data to save some time getting the replacement drive prepared). Sorry, I go off on tangents.

Others said the experienced similar issues, not just my fellow TechnoBit user, so if any of you have a handle on why this happens, I'm all ears.

I thought perhaps it had something to do with the attacks, but read in the meantime that the two things are completely unrelated (wasn't fully competent on the nature of DDoS attacks and the extent of damage they can do - glad they don't fuck up the pool itself, but sorry to read WK is spending his free time dealing with that nonsense, getting ransom requests, and having a shitty time from the sound of things... Hope your girlfriend hasn't been too upset about the DDoS attackers taking all your free time!!! ;P).

Glad to be connected again, and thanks to anyone with any thoughts on how I might avoid the above-described problem in the future.

[gorgatron]

so ... the device running cgminer worked as expected but ................. the ones that lost you lotsa hash rate were running what?

nothing was using cgminer. I used Hex Miner for the TechnoBit devices. Those kept going. I use bfgminer for everything else. It's embedded in the mess of text I wrote, but that part is in there.

The ones that lost the hash rate stopped hashing at Eligius. The BFL units on one instance of bfgminer went into failover, and the KnC on a separate instance dropped to less than 800 MH/s

Oddly enough, I have a total of 330 GH/s after the two TechnoBit units died the other day. I'm still at the computer after writing the postyou've responded to. My hashrate at Eligius has dropped again from ~330 GH/s to ~265 GH/s. bfgminer shows the KnC and BFL units to be hashing fine. Curious if this is a stats issue, si I've switched everything to another pool again for 24 hrs to see what kind of hashrate stat i get. Perhaps i have more HW issues than I thought, though bfgminer is reflecting this in no way whatsoever.

I've been messing with this crap for hours every evening for over a week. I'm burnt out for now. I'll check in tomorrow. Appreciate the help.

[bolverk]

Cloudflare only helps with http/https last I checked.  And I would rather not centralize yet another bitcoin related service behind Cloudflare.

As for the effectiveness of the attacks, the only noticeable effects lately have been slightly delayed response times with the web server.  The pool servers have been generally unaffected.

The web server setup definitely is not as scalable as I would like, but that falls back to the mostly unfunded part of the problem.

I'll throw this out there one more time:  I know my 1% isn't much given my tiny rig, but I'd be happy to donate some time to help out the cause.  Scalability is a big part of my job.

[Multipulty]

pool is work?

[davebodger]

pool is work?

Yes, pool has been working for me all morning.