[ANN][LTC][Pool][PPLNS][STRATUM] - ltc.kattare.com - burnside's Mining Pool

[tacotime]

I'm sorry but I had to move to litecoinpool.org until you can figure out what is wrong. I don't like switching teams in the middle of the game but the 40% average stale rate is killing me. I get <1% there.

Come to ltcmine.ru and get 0.27% stales, that's what I'm getting right now.  

[1714113418]

1714113418

[1714113418]

1714113418

[1714113418]

1714113418

[1714113418]

1714113418

[1714113418]

1714113418

[1714113418]

1714113418

[Bogart]

I'm sorry but I had to move to litecoinpool.org until you can figure out what is wrong. I don't like switching teams in the middle of the game but the 40% average stale rate is killing me. I get <1% there.

Just did the same.  So far I'm getting maybe 3-4% stales on my well-connected workers, and 6-10% on my bad-internet ones.

Sure beats 40%.

I'd come back if the situation changes.

[tacotime]

Yeah, I switched about 3 months ago.  I know burnside works really hard to keep the place running, but he needs to implement stratum and vardiff at this point because he can't compete with the larger pools.   It will be more beneficial to him in the end to use stratum anyhow, more miners, more blocks, more fees for him (if he had vardiff and stratum at 1.5% fees PPLNS I'd be back in an instant).

[burnside]

Yeah, I switched about 3 months ago.  I know burnside works really hard to keep the place running, but he needs to implement stratum and vardiff at this point because he can't compete with the larger pools.   It will be more beneficial to him in the end to use stratum anyhow, more miners, more blocks, more fees for him (if he had vardiff and stratum at 1.5% fees PPLNS I'd be back in an instant).

I'd love to get stratum going.  I was playing with the stratum-mining-proxy, got it all installed and running, did the ./mining-proxy.py --help and realized that it's for connecting getwork (old skool) miners to a stratum pool.  Cripes.  I need an scrypt version of stratum-mining. 

Anyone aware of any LTC mining pool software that can do stratum?

[baggyp]

https://github.com/CryptoManiac/stratum-mining-proxy  ??

[burnside]

https://github.com/CryptoManiac/stratum-mining-proxy  ??

Is not what you think it is.

It's for legacy mining software to be able to connect to a stratum pool.   

[butjust41day]

Hello Burnside

Hopefully this information is helpful.

Something I've noticed is that most of the rejected shares I receive on your pool are not stales as everyone seems to be claiming. They may be listed as round stales under My Stats; but they are not. I have 4 rigs and I run both cgminer and bfgminer. When running on your pool all four rigs are seeing an increased unknown work rejection. I have tried running theses rigs on several different pools and never see this error. On your pool my unknown work rejects have hit as high as 25 percent in a round.

I don't claim to be knowlegable of any of these systems and software; but, I am a capable troubleshooter in my day job. As a trouble shooter I try to break problems down into sections to solve them. The fact that my rigs work else where eliminates one of the sections and puts the problem at your end.

Dividing your end into two sections to start:

1- Your server is sending my rigs work that is not legitamate work. My miner solves and sends the work back and it is rejected as unknown work as it should be. It shows up as a stale under My Stats and is rejected.

or

2- The work your server sends is legitimate work. My end solves correctly and sends in the work. Your server rejects this legitimate work. Maybe its the DDos filters. Maybe some other error. It shows up as a stale under My Stats and is rejected.

I hope the above information somehow points you in the right direction. If it was legitimate stales it wouldn't be much of an issue. I'm running on a  Stratum pool right now and get some stales there too. Spending time working on Stratum when the problem may lay elsewhere will gain very little for the time being. I'm looking forward to this problem being solved. I like the excellent work you've put into your pool.

[Zedster]

Yeah, I switched about 3 months ago.  I know burnside works really hard to keep the place running, but he needs to implement stratum and vardiff at this point because he can't compete with the larger pools.   It will be more beneficial to him in the end to use stratum anyhow, more miners, more blocks, more fees for him (if he had vardiff and stratum at 1.5% fees PPLNS I'd be back in an instant).

I'd love to get stratum going.  I was playing with the stratum-mining-proxy, got it all installed and running, did the ./mining-proxy.py --help and realized that it's for connecting getwork (old skool) miners to a stratum pool.  Cripes.  I need an scrypt version of stratum-mining. 

Anyone aware of any LTC mining pool software that can do stratum?

I don't know if this helps but Coinotron uses stratum on his LTC pool.  Ask them maybe?

[burnside]

Hello Burnside

Hopefully this information is helpful.

Something I've noticed is that most of the rejected shares I receive on your pool are not stales as everyone seems to be claiming. They may be listed as round stales under My Stats; but they are not. I have 4 rigs and I run both cgminer and bfgminer. When running on your pool all four rigs are seeing an increased unknown work rejection. I have tried running theses rigs on several different pools and never see this error. On your pool my unknown work rejects have hit as high as 25 percent in a round.

I don't claim to be knowlegable of any of these systems and software; but, I am a capable troubleshooter in my day job. As a trouble shooter I try to break problems down into sections to solve them. The fact that my rigs work else where eliminates one of the sections and puts the problem at your end.

Dividing your end into two sections to start:

1- Your server is sending my rigs work that is not legitamate work. My miner solves and sends the work back and it is rejected as unknown work as it should be. It shows up as a stale under My Stats and is rejected.

or

2- The work your server sends is legitimate work. My end solves correctly and sends in the work. Your server rejects this legitimate work. Maybe its the DDos filters. Maybe some other error. It shows up as a stale under My Stats and is rejected.

I hope the above information somehow points you in the right direction. If it was legitimate stales it wouldn't be much of an issue. I'm running on a  Stratum pool right now and get some stales there too. Spending time working on Stratum when the problem may lay elsewhere will gain very little for the time being. I'm looking forward to this problem being solved. I like the excellent work you've put into your pool.

Thank you for that, it got me thinking about the block boundary issues we've seen in the past and I realized that the litecoind process that is supposed to send a HUP to pushpool might only be reading the PID file at startup.

I restarted litecoind, which caused it to re-read the PID file, and the stales have dropped significantly now.  I thought it would just pick up changes to the PID file as they occur.  So now I know that anytime pushpoold is restarted... I have to also restart litecoind.  At least until I can alter the custom patch to re-read it on each block boundary.

We still have other issues.  Now that I've found this one I need to set the difficulty back where it was.  I'd changed the difficulty thinking it'd help matters, but I think in reality it made them a percent or two worse than they were before.

The other thing I can do is I can change the values around for how long work is considered current.  Right now you only get two minutes from the time you get the work to when you submit it.  If I make that 3-5 minutes then some of the cpu miners will probably see lower stales.

Cheers.

[baggyp]

great work guys. glad to see some progress! I loved mining at your pool and I look forward to returning!

That said. 83 difficulty in 3333min? WHAT IN THE SEVEN HELLZ!

We need FPGA. Power is quickly drawing closer to profit :\

[butjust41day]

Glad to be able to help.

I'm working right now but will check things out at my end this evening. I'll watch for a while and see if the problem is still there. One more thing that I didn't mention was that the error seems almost random. It's not always at the end of a round. Sometimes I'll get 10 to 40 rejects in a row all saying unknown work. Then right back to accepted shares.

I wish I was more knowledgeable about this so I could be more help on my end. I think bgminer might come with a logging option. If it does I'll try and record something for you.

[burnside]

Glad to be able to help.

I'm working right now but will check things out at my end this evening. I'll watch for a while and see if the problem is still there. One more thing that I didn't mention was that the error seems almost random. It's not always at the end of a round. Sometimes I'll get it 10 to 40 rejects in a row all saying unknown work. Then right back to accepted shares.

I wish I was more knowledgeable about this so I could be more help on my end. I think bgminer might come with a logging option. If it does I'll try and record something for you.

You're definitely right, there is another ongoing issue.  That is that pushpool gets overloaded periodically.  (a few seconds every few minutes.)  I've expanded file descriptors and every single restriction I can think of to work around this, but have been unsuccessful.

So, unable to expand pushpool I ended up spinning up two pushpools and load balance them using nginx.  I made that change ~6 months ago.  Now we're back to one pushpool or the other getting overloaded periodically, and with the load balancing what happens is the overloaded pushpool gets removed from the balancing and you get sent to the pushpool that is still answering.  If you're making a request to submit work, and that work came from the other pushpool, then the current pushpool doesn't recognize it and it gets flagged as invalid.

Oddly enough, pushpool has memcache functionality and both pushpools are pointed at the same memcache.  I thought initially this was so that you could run a bunch of them and have them share the work between them, but clearly that is not the case.  I'm not really sure what pushpool is using memcache for.

As you digest all this you're probably wondering why then can't I just add a third pushpool to the balancing.  The problem is that in order to make sure that you always get sent to the same backend pushpool (because of the issue where your work is invalid if you don't) I had to configure the balancing to be by IP address.  And, naturally, since we're behind a DDoS service, 80% of our traffic comes from... the same IP address.  Ugh.  So even with the two pushpools, one takes like 80% of the traffic and I have no way to split it out beyond that.  I need like 3 DDoS services with each one running a pushpool behind 'em. 

[Tranz]

Network Hash Rate   -2,147.484 GH/s

An Int over run obv. But this is just reporting right?

[baggyp]

Glad to be able to help.

I'm working right now but will check things out at my end this evening. I'll watch for a while and see if the problem is still there. One more thing that I didn't mention was that the error seems almost random. It's not always at the end of a round. Sometimes I'll get it 10 to 40 rejects in a row all saying unknown work. Then right back to accepted shares.

I wish I was more knowledgeable about this so I could be more help on my end. I think bgminer might come with a logging option. If it does I'll try and record something for you.

You're definitely right, there is another ongoing issue.  That is that pushpool gets overloaded periodically.  (a few seconds every few minutes.)  I've expanded file descriptors and every single restriction I can think of to work around this, but have been unsuccessful.

So, unable to expand pushpool I ended up spinning up two pushpools and load balance them using nginx.  I made that change ~6 months ago.  Now we're back to one pushpool or the other getting overloaded periodically, and with the load balancing what happens is the overloaded pushpool gets removed from the balancing and you get sent to the pushpool that is still answering.  If you're making a request to submit work, and that work came from the other pushpool, then the current pushpool doesn't recognize it and it gets flagged as invalid.

Oddly enough, pushpool has memcache functionality and both pushpools are pointed at the same memcache.  I thought initially this was so that you could run a bunch of them and have them share the work between them, but clearly that is not the case.  I'm not really sure what pushpool is using memcache for.

As you digest all this you're probably wondering why then can't I just add a third pushpool to the balancing.  The problem is that in order to make sure that you always get sent to the same backend pushpool (because of the issue where your work is invalid if you don't) I had to configure the balancing to be by IP address.  And, naturally, since we're behind a DDoS service, 80% of our traffic comes from... the same IP address.  Ugh.  So even with the two pushpools, one takes like 80% of the traffic and I have no way to split it out beyond that.  I need like 3 DDoS services with each one running a pushpool behind 'em. 

fairly new to litecoin, been a bitcoin fanboi since I heard about it right after the crash in '11. I'm no developer so I often just talk out of my ass, but it is in an attempt to lend my problem solving skills to the effort... I just don't always succeed!

I know CGMiner has a failswitch configuration, you can change it to round robin, etc. Maybe if you had several pools set up behind the ddos and just had people config to round robin between them?

[baggyp]

Also, this is for bitcoin, so I'm not sure how much work it would take to implement it into scrypt mining, but here:

http://mining.bitcoin.cz/stratum-mining
https://github.com/slush0/stratum-mining
https://docs.google.com/document/d/17zHy1SUlhgtCMbypO8cHgpWH73V5iUQKk_0rWvMqSNs/edit?hl=en_US&pli=1
https://github.com/slush0/stratum
http://webchat.freenode.net/?randomnick=0&channels=#stratum&prompt=1
https://bitcointalk.org/index.php?topic=55842

[burnside]

Network Hash Rate   -2,147.484 GH/s

An Int over run obv. But this is just reporting right?

Hah, that's awesome.  Whatever it is, that's how it's coming out of litecoind:

# litecoind getmininginfo
{
    "blocks" : 312928,
    "currentblocksize" : 1225,
    "currentblocktx" : 1,
    "difficulty" : 54.19159781,
    "errors" : "",
    "generate" : false,
    "genproclimit" : -1,
    "hashespersec" : 0,
    "networkhashps" : -2147483648,
    "pooledtx" : 3,
    "testnet" : false
}

[butjust41day]

You're definitely right, there is another ongoing issue.  That is that pushpool gets overloaded periodically.  (a few seconds every few minutes.)  I've expanded file descriptors and every single restriction I can think of to work around this, but have been unsuccessful.

So, unable to expand pushpool I ended up spinning up two pushpools and load balance them using nginx.  I made that change ~6 months ago.  Now we're back to one pushpool or the other getting overloaded periodically, and with the load balancing what happens is the overloaded pushpool gets removed from the balancing and you get sent to the pushpool that is still answering.  If you're making a request to submit work, and that work came from the other pushpool, then the current pushpool doesn't recognize it and it gets flagged as invalid.

Oddly enough, pushpool has memcache functionality and both pushpools are pointed at the same memcache.  I thought initially this was so that you could run a bunch of them and have them share the work between them, but clearly that is not the case.  I'm not really sure what pushpool is using memcache for.

As you digest all this you're probably wondering why then can't I just add a third pushpool to the balancing.  The problem is that in order to make sure that you always get sent to the same backend pushpool (because of the issue where your work is invalid if you don't) I had to configure the balancing to be by IP address.  And, naturally, since we're behind a DDoS service, 80% of our traffic comes from... the same IP address.  Ugh.  So even with the two pushpools, one takes like 80% of the traffic and I have no way to split it out beyond that.  I need like 3 DDoS services with each one running a pushpool behind 'em.  

Firstly things do seem to be running much smoother this morning.

Thank you for your explanation above. I have to admit I had to read it twice; but, your explanation is very clear. If pushpool keeps a running log of what work it has issued and that log is not being shared in the memcache where is it? Do both running pushpools share a common database where information about each workers contributing shares ect. are maintained? And is the log of work issued also apart of that database? If it was might this be a database issue?

One other thing if the memcache is not for sharing of information of what work has been issued should both pushpools be pointing at the same cache or should they have seperate memory blocks?

Forgive my ignorance here because I really don't know. I've just found in my line of work it's good to bounce ideas off each other and sometimes even a wrong idea triggers a right solution.


Edit: I don't know if this really means much; but, I notice something today when I was monitoring the situation. When I went to the My Stats tab it seemed to hang up. While it was hanging I checked one of my rigs and the unknown work rejects we're happening. I had been running at below 0.5 percent stales up to that point (suddenly I was at 20 percent; although my miner calls them unknown work). The My Stats tab has to access the database to display. Interesting coincidence?

[baggyp]

Edit: I don't know if this really means much; but, I notice something today when I was monitoring the situation. When I went to the My Stats tab it seemed to hang up. While it was hanging I checked one of my rigs and the unknown work rejects we're happening. I had been running at below 0.5 percent stales up to that point (suddenly I was at 20 percent; although my miner calls them unknown work). The My Stats tab has to access the database to display. Interesting coincidence?

this seems right to me. perhaps the stats are autoupdating on the page and its causing conflict?

[burnside]

You're definitely right, there is another ongoing issue.  That is that pushpool gets overloaded periodically.  (a few seconds every few minutes.)  I've expanded file descriptors and every single restriction I can think of to work around this, but have been unsuccessful.

So, unable to expand pushpool I ended up spinning up two pushpools and load balance them using nginx.  I made that change ~6 months ago.  Now we're back to one pushpool or the other getting overloaded periodically, and with the load balancing what happens is the overloaded pushpool gets removed from the balancing and you get sent to the pushpool that is still answering.  If you're making a request to submit work, and that work came from the other pushpool, then the current pushpool doesn't recognize it and it gets flagged as invalid.

Oddly enough, pushpool has memcache functionality and both pushpools are pointed at the same memcache.  I thought initially this was so that you could run a bunch of them and have them share the work between them, but clearly that is not the case.  I'm not really sure what pushpool is using memcache for.

As you digest all this you're probably wondering why then can't I just add a third pushpool to the balancing.  The problem is that in order to make sure that you always get sent to the same backend pushpool (because of the issue where your work is invalid if you don't) I had to configure the balancing to be by IP address.  And, naturally, since we're behind a DDoS service, 80% of our traffic comes from... the same IP address.  Ugh.  So even with the two pushpools, one takes like 80% of the traffic and I have no way to split it out beyond that.  I need like 3 DDoS services with each one running a pushpool behind 'em.  

Firstly things do seem to be running much smoother this morning.

Thank you for your explanation above. I have to admit I had to read it twice; but, your explanation is very clear. If pushpool keeps a running log of what work it has issued and that log is not being shared in the memcache where is it? Do both running pushpools share a common database where information about each workers contributing shares ect. are maintained? And is the log of work issued also apart of that database? If it was might this be a database issue?

One other thing if the memcache is not for sharing of information of what work has been issued should both pushpools be pointing at the same cache or should they have seperate memory blocks?

Forgive my ignorance here because I really don't know. I've just found in my line of work it's good to bounce ideas off each other and sometimes even a wrong idea triggers a right solution.


Edit: I don't know if this really means much; but, I notice something today when I was monitoring the situation. When I went to the My Stats tab it seemed to hang up. While it was hanging I checked one of my rigs and the unknown work rejects we're happening. I had been running at below 0.5 percent stales up to that point (suddenly I was at 20 percent; although my miner calls them unknown work). The My Stats tab has to access the database to display. Interesting coincidence?

The work is stored in pushpool's memory I'm pretty sure.  It's all internal until a share is submitted, then the result of that share (stale or not) plus the username submitting the share is submitted to the db as an insert.  I use insert delayed even, so the db would have to be completely down for it to impact work submissions.  Edit... probably worth noting here too that the db server is a different box.  We have three servers right now, the webserver, db server, and pushpool/litecoind server.

The other possibility though is that there are network issues or rate limiting going on at the DDoS provider level.  Eg, if you can't load the site and the shares delay on submission, then your requests might not even be getting to us.

I'm tempted to just turn off the DDoS protection, but I know as soon as I do we're gonna get caught with our pants down again.     (for those of you new to LTC, we have been one of the few pools that have been up through most of the DDoS attacks, though it does impact us too.)

[Internet151]

Right now I have automatic payouts set up on my account. Is there any way to disable the email notifications that come with these payouts?

Also, is there an IRC channel for this pool?