Hey guys, just a quick update (sorry for the lack of them over the past day or so).
We've unfortunately had a terrible amount of server issues (a few of which were mentioned here) over the past few days, and have been in 100% firefighting mode. A coin update caused a bunch of chain reactions, eating up too much memory, causing our work processors to crash, causing the queue of shares to shoot up, causing too much memory usage (again), causing our db to become starved, causing bitcoind to crash, causing our profitability calculator to reference everything compared to a 0 difficulty bitcoin, causing profitabilities to all be 0, causing it to stick on luckycoin for about 2 hours while I fixed all the issues.
No shares should have been lost, they just got piled up for a while.
Everything is back up and working, and better than ever. We've added precautions to avoid this happening again ideally, but these things never happen in the same way twice.
During all this time, since the endpoints had to be restarted thirty million times, I threw in some upgrades in the process.
1) Stratum endpoints should be much faster, and previously could lock up for a second or so when submitting a found block, which on low difficulty coins could lower profitability pretty substantially (for that few seconds).
2) Our switcher to prioritize certain things a bit heavier than we were before.
- Price-stable coins (doge, etc) are given a small boost in relative weight.
- Coin price (depth-based) is weighted by network hash power. For example, if on some coin, we look to be about 25% of the network hash power, we require a depth of 4x standard block reward, plus our outstanding balance, in calculating depth required to sell. Assuming that the other 75% of network hash power will eat through depth as well. This causes prices to be a bit more accurate (and with low price coins, this can be a +-20% swing in price).
- We prioritize staying on the same coin a bit more heavily. If the switch to the new coin isn't worth 5% more, we'll now stay on the same coin (fewer stale shares during the switch, etc). 5% was picked semi-randomly, so this will continue to be tuned based on what we see over the next few days.
3) Better wafflepool connectivity. We've bumped the number of connections per coin daemon to give us a bit better network propagation (without exhausting bandwidth), as well as making sure our endpoints (and main server) are connected to each other, making sure our servers see a block solved by us first (without going through the whole network first).
4) New coins, we've got a few new coins syncing as it is, hopefully will be added tomorrow
Long story short, sorry about the downtimes, but we're back and better than ever. Hopefully profitability starts reflecting it soon
