Bitcoin Forum

Posted on our blog

http://blog.bitpay.com/


Our site was back online for about 12 hours today. It worked great during the London Conference, then the attacker found a way to defeat our DDOS protection.  

I am still in Europe and Steve is also away.  We have taken the site offline until we can return home and properly deploy new servers.  Or until the attacker gives up and moves on.

i would think a payment processing business with a thousand merchants relying on them would do better than allow the situation where not even one principal is available to keep the machines running.

How long do you think this will take? Are we talking tomorrow or days? As we count on your servers to process payments we are kind of dead in the water with bitcoins till then unless we pull out manual payments.

That is the beauty of bitcoin.  Anyone who wants any of my product can message me here or through my store.  I can take your bitcoin myself and verify the payment. 

It is less convenient and I hope the attackers are know they are damaging bitcoin by making it appear less reliable.   

If Visa/MC is down, I can not take the payment myself without them.   

This really depends on the attacker at this point…we'll keep trying to bring the site back online and improve our anti-DDOS capabilities as best we can…I expect he'll continue to attack.  There's really nothing else we can do.

We have deployed some additional DDOS protection so https://bitpay.com is back online.  Not all merchants are able to get through the firewall just yet, but we are working with each one.

Email stephen@bitpay.com with any issues.  Steve cannot access PM on this forum so you will not get through unless you email him directly.

If the attacker steps  the attack the site may go down again without warning.

We will be deploying a much more robust system in a few weeks.  But this protection may hold us over until that can come online.

Didn't the attacker state his motives? Is it for money, or what?

We cannot elaborate on any specific information like this.

Here are a couple additional things to note:
- bit-pay.com no longer bounces to bitpay.com (we'll re-enable it when I have more time)
- http://bitpay.com no longer redirects to https://bitpay.com (again, I'll re-enable when I have more time) …you have to go directly to https://bitpay.com

Also, one of the issues that this has created for us is that the old SSL client certificate based method of API authentication no longer works (due to the anti-DDOS proxying of connections).  We were planning to keep that method active until all merchants had a chance to migrate.  Unfortunately we've not yet had a chance to upgrade all shopping cart plugins to use the new API key based method of authentication.  If you're using one of our shopping carts and are unable to process a checkout, send me a private message.

Im getting an invalid invoice error with my WHMCS api?

Regards,
Staff @ QuantumKiwi

I'm getting the same thing, have been all day. I think it's with the API, I emailed steve and still waiting for a response back.

I too am getting an error when the customer is trying to checkout.  I am guessing the API has to be updated.  Steve could you please post/let us know when and how we should update our api.  I might have a really old one as it is directing me to bit-pay.com/checkout
Thanks

The redirect you can actually fix yourself in the php library file, under the curl class. I did this awhile ago as the updated WHMCS module wasn't on the site. But I have had this fix for awhile now and it is still giving me the invoice error. So it probably is the API as the original module is kind of old.

I have a question, is Bitpay still doing ACH transfers out on a daily basis? Or are these also suspended during the attack? As I have a lot of cash in there right now that need's to be transferred out at the end of the day :S

He asked for 1k BTC.

ACH transfers are not interrupted.  All funds are fully accounted for and banks are just fine.

Any source?

Got this but saw it elsewhere too where I can't remember now
http://bitcoinmagazine.net/walletbit-under-ddos-1000btc-demanded/

It's about WalletBit (which is already back online), not about BitPay, mate.

Sorry! I think I should go to sleep now  :-[

But tbh I wouldn't be surprised if this was not the same person / group of people.

Especially given recent DDoS ''offers'' in the marketplace.

I just was able to get on the site and download the most recent whmcs module but no go, its still the old version. I emailed steve a while ago but no response yet, i know he's busy but if the merchants could at least use their sites. I sent in my servers ips under the hope that they would be allowed access maybe. A response or status report would be nice. I mean you have 1000 merchants just to be offline like this completely.

Well now my password is not working and the password reset page is not working. Anybody got any updates on this?

So back to the point, how do we get our API's working properly again???

I wouldn't mind accepting peoples payments for services and get some bills paid :)

I wouldn't mind it either, my previous idea had reference, but nobody wants to respond from the company not even by email. Give them the server IPs and let them ok those IPs.

We have resorted to making our own automated BTC payment system, we're almost done with it for WHMCS. Though we offer many other form's of payments most people don't like paying with CC from the BTC community. So instead of loosing clientele we have resorted to making our own, if it work's look's like bitpay has lost their first customer over this. Sorry guys but no communication is not good news.

The DDOS protection we setup is breaking the client SSL certificate used in our API and all of the shopping cart plugins.  However, it is blocking the attack and our site is reachable now.

We have recently added API keys as an easier authentication method for the API.  The API keys are passing through the firewalls just fine.

if you are using our API, then please update your authentication method to replace the client SSL with the API key.  instructions can be found here:

https://bitpay.com/downloads/bitpayApi.pdf (https://bitpay.com/downloads/bitpayApi.pdf)

if you are using one of our cart plugins like Magento, WHMCS, opencart, or zen cart, we need to update all of these plugins to use the new auth method.  If you are a programmer, you can update the plugin yourself using the instructions in the above link.

We are still under attack (for the 4th day in a row) so things are extremely busy right now.  We will update the plugins as soon as we get time.

Bitpay is down right now. You should use global distributed servers to prevent DDoS.

we will get there.  one problem at a time.

the CDN from cloudflare looks like a nice service to do just that.

Really, im not impressed bitpay.


You have lost another customer.

No support? No replys? No direct support to merchants who GAIN you the profits, you take a percentage of our cut and we are getting little to no support.

Your telling us merchants with API's to modify the API ourselves...... Get real. Im getting a coder to code my own module and take my own payments.

Companies who do not respect clients and look after them in hard times - LOOSE CLIENTS.

Bye BitPay! Hello Quantum Kiwi BTC Pay!

Keep up the good work guys, this is really starting too look more like an attack on bitcoin, 4 days already  :o




@QuantumKiwi go troll somewhere else, welcome to my "ignore"

Bitpay will emerge stronger from this one stronger and harder to attack.

Im not trolling, im a very pissed off business owner that cannot take payments from clients.

Clients are looking at us like fools.

I think im entitled to be fairly annoyed - No support, being told to recode the API myself??

And Bitpay takes a percentage from our profits...... Im not willing to give the company that benefit after this.

Sorry BitPay, but while shit hits the fan you have to keep your clients INFORMED. Leaving merchants with API's just hanging there is not acceptable.

our site is under a massive DDOS attack, has been for 4 days now, and our DDOS protection service says it is massive, in the top 5% of attacks they see.  Someone is very, very determined to knock out bitcoin and bitpay right now.

Our site will be up when the attacker stops, or when we keep adding enough firewalls to block all of the botnets.

Once we add the firewalls up, the API and all of the plugins will need to be reconfigured.  We can only do one thing at a time.  First thing is to stop or block the attack.  We can't get to steps 3 and 4 until that is done, otherwise its a waste of time.

There are other bitcoin payment solutions out there, or you can build your own.

Is there any indication where this is coming from or why they are doing it?
Are they asking for a ransom?

Top 5%? Are you guys the ones that are being hit with the 65Gbps?
http://blog.cloudflare.com/65gbps-ddos-no-problem

That would be us, yes.  Steve is in touch with them on a regular basis.  This botnet attacker is just relentless.

How does this mesh with the end of the article?

Why dont you set up a host of private IPs for your customers to use to process payments? I might be missing something, but let the criminal DDoS your public site all he wants, if he doesnt know the IPs your customers use, then the impact should be minimal, no?

Perhaps a biased article?

That which does not kill you, makes you stronger.

If this doesn't apply to any activity on the Internet, then it can't apply to anything.

Though to be honest I am very unhappy with not being able to use my payment system and no communication from the company I do understand priority, especially with a 65Gbps attack, MY GOD!!! I am still an unhappy customer, but priority comes first some times. With us when a server goes offline the customer just want's their site back up, while we're still trying to figure out why it went offline and save data. So I do understand but am still a bit of an unhappy customer.  Maybe instill a company policy of when there are problem's like this, send out a mass customer email, that's what we do. All our customer's got emails when we couldn't do bitcoin payment's anymore using a mass mail program.  Just a thought.

I cannot do the API configuration (Not I do not agree with us having to do it but again priority) because bitpay.com is still offline and I cannot get in to get my Key :) so we cannot reconfigure the script yet for this reason. But from what I can tell it's just a single edit, change from SSL to HTTP Auth with is a header field, real quick and simple if we could get in  :'(

In our case we do offer CC payment's, so if a customer needed to pay before we got our own system up 3 hours ago they could have real simple using CC.

My theory is it's not a regular attacker, who has that money to attack for 72 hours at 65Gbps? We're talking something around $20 a min. here in cost or more! My personal opinion is it's Paypal or Visa going after you guys, but that's just my theory. They are the only ones looking to gain from this. Any other attacker no matter what the ransom it wouldn't pay their cost of running that big of an attack for 4 days. Plus if you don't pay and they are busted they just lost their money for the attack ha ha ha

So it's gotta be a big corp doing it that stands to gain from you going down and Bitcoin's being hurt.

I agree with the private IP routing, that would actually work.

We will get it fixed.  The right solution is a network of private servers with proxies and DDOS protection for everything.  This will take time to deploy properly.

For large merchants, we would consider installing their own instance of bitpay for their payment processing.  This is something we could offer as a premium service.

This has made us think of ways to make our service more decentralized and more robust.  I agree that someone is trying to attack bitcoin and slow down its rapid adoption.

If we can solve this attack issue, I think everyone will be glad it is now when we have 1,000 merchants and not a year from now when we have 10,000 or more and we're doing $1M/day in processing.

This is REALLY crazy!!  :o

Just heard from a source that a little monkey in paypal threw his toys out of his cott when bitpay hit 1,000 merchants.

Maybe you have a big pissed off billionaire paying some russian to ddos?

After seeing the 65gbps.. Far out, thats one impressive attack... And its not a normal attack...?

A normal attack is about 10 ;) Maybe 20 if they hate you with a passion only an ex-lover could muster up ;)

Is $20/min accurate for what it would cost to buy a DDoS of this scale?

No more, if I remember right the article said something around $30 for 30 seconds. So $60 a min. would be more like it if that is correct. But if your buying 4 days worth they probably gave them a discount ;) ha ha ha

It depends.  Reading the cloudflare article, it appears as though there are ways to DDOS that are [virtually] free, one of which is DNS reflection.  If DNS reflection is used heavily in this case, it might be relatively cheap to keep the DDOS going.

My personal suspicion is that it is a ransom (despite bitpay's statement to the contrary), just because the timing is too coincidental with the walletbit DDOS.  The DDOS's on both started at the same time.

One user in the comments who claims familiarity with botnets says a typical rental rate for one of this size might be $1/second. (he specifically mentions $30/30 seconds).

That's what it was! Sorry, I knew I read it somewhere, just for looking:
http://www.zdnet.com/blog/security/study-finds-the-average-price-for-renting-a-botnet/6528

Just as a said note, I bet if the botter is not a large corp, I bet their paying in bitcoin's for the service ha ha ha ha ha

So I know you said that the HTTP Auth was the only thing that needed to be changed, but it's not. We changed that and no matter what we change we always get echoed back out the CloudFlare checking your browser page. So it won't let us past cloudflare to get to the API.

https://i.imgur.com/Wav1r.png

yeah that's the challenge screen.

email support@bitpay.com so my partner and CTO Steve can look into it.  There may be other changes needed, I am not 100% sure. 

I think curl needs the redirect option but it wasn't working for us either :S I'll email then now. Thanks.

well bitpay is online again :)

CloudFlare says the website is offline again.  I cannot view my invoice :(

Working for the moment. I was able to pay for a Jalapeno  :)
Is the attack over or I was just being lucky?

Yeah it seems to be working now. We ended up not only having to add the http Auth part but remove any reference to port 8443 from our module. Problem being is as of right now Steve has to allow any server IP access for it to work so it seem's. But it is working just great right now.

I updated the OpenCart module code to restore functionality, so those with opencarts should check out this thread: https://bitcointalk.org/index.php?topic=110690.0

this did NOT work for me

I have since  had to take tens of thousands of dollars in transactions manually

and have offered a 50BTC  bounty for someone to develop an opencart plugin that would interact directly with Bitcoind

I am sure something like this could be very simple to do, but it is beyond  my expertise.

This kinda makes it sound like MY extension made you do those transactions manually and not BitPay's service outage.    :-\

I've been working on a bitcoind OpenCart extension. The trick is the server-side bitcoind implementation. Even if someone makes an extension, there will be some server-specific bitcoind setup required, so it won't be a simple FTP and done install. Now that I know there's a bounty on it, I might speed up my development efforts.

I think I got your patched open cart extension "working"

I did a test transaction, and send the coins - although I have not seen the coins show up in my bit-pay account yet. hopefully I am not out these coins.

Transaction ID: f5cf1ad733e935232d920e4033ee447ffb74e37a36750e06bff631e90988f1be




As far as an opencart / bitcoind extension this is something that is really needed my friend.

and who needs a simply ftp and done install?

I have setup, modified, and transacted thousands of bitcoins in transactions with that very primitive zencart module that has been around forever

git://github.com/jalder/Zen-Cart-Bitcoin-Payment-Module.git



Yes we all love Bitpay (too much apparently)

but we really really really need a way to be able to conduct retail transactions with Bitcoin without being dependent on a third party
and I am SURE this is the way Satoshi wanted it. To be able to conduct these transactions on our own without being dependent on anyone else...

think about it bro.

Still down?

Seriously, so many clients asking why they get " cannot connect to host "

Very frustrating.

Contact steve or support, the IPN isn't working and posting back. So the transactions will go through and get to bitpay, but bitpay isn't posting back to the sites to let your system know that there has been a payment. Something to do with the cloudflare proxy. So steve is having to manually send payment notifications via email to each client till it's fixed.

On the manual system, send me a PM with requirements. When the kitty litter hit the fan we made our own Bitcoin payment module for WHMCS that interacts with bitcoind without a third party. It's quite simple really just need a DB to keep track of invoices and transaction's. So it's already made by us to work with bitcoind, we would just need to change it to opencart.

Plus if you don't already have a bitcoind instance running we have bitcoind cloud hosting to offer. So PM me, it's already out there and made and working :)

Hi! I'm trying to integrate the whmcs module, but it is not working. I don't know if it is because the ddos problem, changes to the available download of the module or because I don't know what is the "Secret Key To Verify Comm."

Can you tell what is "Secret Key To Verify Comm."? And if the available download for whmcs module is updated to the new changes?

Thanks

Secret key is what ever you make it :) "asdfasdfaolj23lkj12lk3jlkajsad" or "asldkjfl2k3kaalkdj2l3klasd" really.

Download the WHMCS module and open bp_lib.php, if it has the curl_opt for password there and the two SSL gone or commented your good to go.  Steve asked me to send over the recent updated version after we tested it for a few days. Those few are done so here in a bit I will be sending them the working version we have. So shortly it should be up.

sounds good guys, making some progress.

very interested in the independent bitcoin whmcs module though.

well... I will be waiting for bitpay module. Also I'm building my own module for my site bitcoinmkt.com.

last update on your blog was the 16th, its now the 22 here in NZ.

When is the WHMCS module fix going to be released?

Does anyone else have a problem with credit card? Mine just stopped working. Not giving any error. Pin looks to be accepted, just cash is not getting out of ATM.

I haven't had any problems.  Are you still having issues?