People who want big shares should set eu2 as primary, people who want little shares should set eu as primary, both setting the other as backup.  The rationale being that if eu gets ddos'd, you'll fail over to eu2 and work on big shares for a little while.

No eu2 is the final address.  eu.multipool.us will remain up as the non-protected EU pool.  I suggest anyone mining on eu set eu2 as backup.

In case anyone didn't notice the news update, the DDOS-protected EU high hashrate server is now available at eu2.multipool.us.

You need to use east or west not [east|west], that's just shorthand.

Yes, what was wrong was that doge payments could not be calculated because the query was timing out due to the 60 million rows in the shares table.  Once those shares were reimported and summarizeed, payments went out as normal.

There was a period of about 3.5 hours between blocks 71226 and 71424 where we didn't find any blocks.  This was right after the DB came back up.  The db queries block everything else, so blocks found during that time were probably not submitted fast enough and wound up being stale.

Note that this all happened long after the DB had been down for 2+ hours and I had advised people to switch off the pool so I don't know that I should be held 100% accountable for that.  Obviously there were some mistakes made, but to suggest malice is irresponsible.   I had to choose between killing those stratum instances and losing all the shares they were holding in memory, or letting them run at a lower efficiency.  I chose the former; after learning what I have learned from this incident I would most likely choose the other option if this were to happen again.

So just to clear up some misconceptions:

This all happened Friday night.  It's Sunday morning.  It's been a day and a half.

The DB didn't go down until approx. 5pm Central time Friday Night.  Once it became clear that things weren't going to recover in a reasonable time period, I shut down the pools, this was at around 8 or 9pm.  So there couldn't have been more than a few hours worth of shares lost (some stratum instances recovered and dumped their shares into the DB after the DB came back up)

All of the blocks that were found paid out without any manual intervention, once the shares were reimported.  This means that there were shares in the DB for all of those blocks.

It's possible that there were some anomalies, but for the most part payouts should be correct.  It's also possible that many people switched off after I changed the website notification at ~7pm recommending that people switch to another pool and there was not enough hashrate left to find many blocks.

I am behind approximately 500 emails, send me a new email with your username and I will take a look at your account.

BTW, having extremely public downtime and not paying people for shares submitted would be an extremely inefficient way to steal money.

Doge blocks scoring now..  Not sure what happened there, a lock got set in the db somehow.

It can take up to an hour for the difficulty to refresh from the DB.  You can try creating a new worker.

There are now about 15 million shares remaining to be imported and 27 blocks left to be scored from before the outage.   I expect this will complete within 2-3 hours.

Nope, just an extortion attempt.  I should have some DDOS protected IPs up by tonight or tomorrow.

The attacks are coming from gptdown.  I just received an email from someone claiming to be the admin there that stated they have stopped the attacks.

Dunno, I just asked DigitalOcean the same question, if they won't restore service I'll need to spin up proxy VMs on another service.  hopefully less than an hour.

Hey guys, I didn't forget about you guys.  I was actually in the IRC channel last night and asked people to pass along the message but apparently it didn't get out very well.

The problem was basically that I was without good internet access again for about 7-8 hours.  I was in a car with people from my "real" job for about 4 hours and then I had to go out and socialize with work people.  I was working feverishly to fix the site via my smartphone the entire time, but had to limit my time spent on the phone.

I'll be returning home later today and then it's back to "normal" support until the next time I need to travel.

What's ironic is that the root cause of all this was me trying to bring down the stales on DOGE by twiddling with the DB thresholds on the DOGE pools.  Rather than committing shares every 30 seconds, I brought that down to 10 seconds.  The old DB just could not handle 3x the number of connections and started getting behind on shares.  I didn't notice this until about 2pm the next day and that's when I finally decided to bring down the website to try to allow the DB to catch up.  But when I saw that there were over 60 million shares in the DOGE shares table, I knew that it was going to take hours and hours and I thought I could speed things up by bringing down the old DB and migrating it to a larger system.  At the time DOGE blocks could not even be scored because the queries were taking so long that they were timing out.

The provider took over 2 hours to make a snapshot of a 160GB VPS and then another 45 minutes to bring up the new db.  During the snapshot was when I posted the update that was visible on the site.   When the new DB server finally came up I was still out with my colleagues and had to wait until I got back to my hotel room to complete bringing things back up.

Right now, all 3 pools are under DDoS attack, I suppose by either people who are pissed, or competitors that want to maximize the impact of this downtime to try to get people to switch pools.

DDoS protected US-West and EU pools are coming early next week, those will be the high difficulty pools I spoke about last week.  I already have the EU server set up, just waiting till I get home to configure things.

As far as support expectations, it's all best effort.  One thing I know I did wrong was that I should have brought down the pools before I started the DB rebuild.  But I had no idea it was going to take almost 3 hours to redeploy that VPS.  If I knew that, I definitely would have brought down the pools first.  But in the grand scheme of things people should not be depending on just one pool (even if it's the awesome Multipool).

I haven't read most of the comments because they'll probably just bum me out, and get me distracted from finishing the work that's needed, which is to get all of the DOGE shares into the DB from yesterday and get all the blocks scored and paid out.  But rest assured that all the shares that were submitted will be accounted for, even shares that were submitted during the downtime.  If any shares are missing I may need to use an average over multiple blocks, but the blocks should be paid fairly in any case.

Again I apologize for this downtime, please help me make Multipool even more successful so that I can leave this "day job" and hire some more support people

I can't really speak to that.  At my own meager 600 kH/sec, I made 2,965 DOGE today mining on the multiport.  The multiport was on DOGE 63% of the time (per multipool stats page), which means if I mined DOGE for 24 hours I should have made ~4706 DOGE which is almost exactly (actually a little more than) the estimate that coinwarz gives me for 600kH (4646).

Our fee is 1.5% as stated in OP.

Please keep your pool/service spam in your own threads.  This thread is for multipool support.

MOON withdrawals were delayed today because there was unprecedented withdrawal demand and I was without Internet access for several hours (traveling).

Unfortunately I cannot predict when withdrawal demand will peak (If I could, I'd be a trader, rather than a pool op).  I'm also not willing to leave more coins in the hot wallets where they could be subject to theft.  MOON payments are going out now.

If you want to trade coins and ensure quick access, it's best to set an auto pay to your own wallet.  This gives the pool the benefit of predictable payouts and also gives you more control over your coins.

It's coming.

If your username on the pool is the same as on here, you've only ever earned ~29 DOGE on Multipool.  Not 100k or even thousands.

If you have multiple accounts, let me know which account is missing the DOGE and I can look into it.

It's not necessarily an issue with Multipool.  It may be an issue of there just being too much hashrate on the network and/or poor connectivity.  I can't find anything in particular wrong on the pool and/or coin daemon side, unless there's a stratum exploit that isn't known yet.

I'm going to be addnoding Middlecoin as soon as h2o adds our wallet IPs into his firewall.

There have been a lot of stale blocks over the past couple days.  I may need to tweak Doge downward to make up for this if it continues.  The reset of the pools earlier when I  changed the payout address seems to have helped a bit but I'm keeping an eye on it.

We've also got an average block reward (all time) of about 472K.  This is substantially lower than it should be and the only thing I can think of that would explain it is if "someone" with a very high hashrate is only mining DOGE when the block reward is over 500k.