Bitinstant- Processing Instant Transfers 24/7. LR|Gox|TH|Dwolla|LR|PP|PX| More..

[Yankee (BitInstant)]

Good Afternoon!

Over the past four days Bitinstant has been offline for 6-10 hours at a time.

Right now, we're working around the clock to get all of these issues resolved. During our scheduled downtime we had found a few security holes, coupled with 2 hacking attempts at our database and OMS software. We've also been subject to daily DoS attacks against our servers.

All attempts have been unsuccessful. Our databases and software have not been compromised nor do we hold any funds on our servers.

All of our systems are behind firewalls, load balancers, and numerous fail-safes to prevent anything from happening.
Bitinstant and this community share the same beliefs of transparency so sometime between Sunday night and Monday morning EST we will be posting a fully detailed report of what went wrong, and how we fixed them.

We've also been working on a nice amount of new features such as adding many more exchanges, payment processors and an updated user-friendly web interface.

While our Web UI and API are down, Manual Transfers are available 24/7. To initiate a transfer, email support@bitinstant.com or use the info below to contact us and we will get you started. We will continue to process transfers during our downtime. In addition, if you believe you have funds tied up with us or need support, please do not hesitate to call, email, or get in touch on IRC at any time.  

At this point, our ETA to be live is sometime over Sunday night, so we can be rolling by the start of the business day, Monday.

Thank you for all your support and patience.

Charlie Shrem & Gareth Nelson, founders.

Telephone: 1.716.712.4846
Fax:          1.347.223.5251
Support@BitInstant.com
/server irc.bitinstant.com
/chanell #Bitinstant

[1714039289]

1714039289

[1714039289]

1714039289

[nmat]

Damn... a 5% arbitrage opportunity and you're down! Any idea when the site will be up again?

[Yankee (BitInstant)]

Damn... a 5% arbitrage opportunity and you're down! Any idea when the site will be up again?

Our Web UI and API are down but we are available 24/7 to process manual transfers. Send an email with your request to me, charlie@bitinstant.com or Query me on IRC above and we will get that straightaway!

-Charlie

[pent]

https://www.bitinstant.com/


502 Bad Gateway

[Yankee (BitInstant)]

https://www.bitinstant.com/


502 Bad Gateway

Thanks! We were down earlier for about 1 hour. All good now

[pent]

Consider using monit daemon to keep all services up automatically

[Yankee (BitInstant)]

Consider using monit daemon to keep all services up automatically

Cool! Thanks for the tip 

[Gareth Nelson]

Consider using monit daemon to keep all services up automatically

That wouldn't be suitable for what caused the earlier downtime (FastCGI daemon was overloaded - we had about 400 requests a minute coming in and it simply died).
What I will be implementing is a custom watchdog that simply hits our website and if there's anything other than an HTTP 200, fires up more servers.

[pent]

That wouldn't be suitable for what caused the earlier downtime (FastCGI daemon was overloaded - we had about 400 requests a minute coming in and it simply died).
What I will be implementing is a custom watchdog that simply hits our website and if there's anything other than an HTTP 200, fires up more servers.

Monit can do this

Code:

check process apache with pidfile /var/run/httpd.pid
       start "/etc/init.d/httpd start"
       stop  "/etc/init.d/httpd stop"
       if failed host btcinstant.com port 80
          protocol http request "/testfile.php"
       then restart
if 5 restarts within 5 cycles then timeout

replace apache to fpm backend. it also could monitor process consumed resources and restart it if above limit.
or even restart can be replaced with custom script which increases workers count.

[Gareth Nelson]

That wouldn't be suitable for what caused the earlier downtime (FastCGI daemon was overloaded - we had about 400 requests a minute coming in and it simply died).
What I will be implementing is a custom watchdog that simply hits our website and if there's anything other than an HTTP 200, fires up more servers.

Monit can do this

Code:

check process apache with pidfile /var/run/httpd.pid
       start "/etc/init.d/httpd start"
       stop  "/etc/init.d/httpd stop"
       if failed host btcinstant.com port 80
          protocol http request "/testfile.php"
       then restart
if 5 restarts within 5 cycles then timeout

replace apache to fpm backend. it also could monitor process consumed resources and restart it if above limit.
or even restart can be replaced with custom script which increases workers count.

We don't use apache or PHP and restarting the server would not help in most cases if we're being hammered by requests - in this scenario we need to boot more servers (not worker threads - servers).

The basic setup would look like this in pseudocode:
while(1) {
    get https://www.bitinstant.com
    if (http_result_code != 200) {
      boot another server
      add to load balancer
      wait for server setup
      check for HTTP 200
      if (http_result_code != 200) {
        restart webserver
        wait
        check
        if (http_result_code != 200) {
           remotely signal a robot to poke me in the leg while saying "bitinstant is down" using a text to speech engine
        }
      }
    }
}

[pent]

anyway its your job, cheers

[Gareth Nelson]

anyway its your job, cheers

Thanks for the tip anyway

[RaggedMonk]

http://www.youtube.com/watch?v=PxMazut6-0U&list=UUOv9LmEtLR-HzDcupGT6UIw

I see BitInstant is promoting itself on the webshow hosted by Bruce Wagner, who was convicted of mortgage fraud and fined a quarter million dollars.

BitInstant does not support or condone fraud, does it?
Have you or any other member of BitInstant ever been convicted of fraud?

Is your indifference towards Bruce's past indiscretions, viewing going on his show a "means to an ends," indicative of your business values as a whole?
Do you believe self-promotion, and the resulting profits, is worth associating your business with someone convicted of fraud?

Thank you,
-Ragged

[Yankee (BitInstant)]

http://www.youtube.com/watch?v=PxMazut6-0U&list=UUOv9LmEtLR-HzDcupGT6UIw

I see BitInstant is promoting itself on the webshow hosted by Bruce Wagner, who was convicted of mortgage fraud and fined a quarter million dollars.

BitInstant does not support or condone fraud, does it?
Have you or any other member of BitInstant ever been convicted of fraud?

Is your indifference towards Bruce's past indiscretions, viewing going on his show a "means to an ends," indicative of your business values as a whole?
Do you believe self-promotion, and the resulting profits, is worth associating your business with someone convicted of fraud?

Thank you,
-Ragged

Yes...Yes...and double YES baby!!!

fyi- by hotlinking Bruce's youtube video you just made him some more money! God I love capitalism 

Cheers mate,

Charlie

[RaggedMonk]

http://www.youtube.com/watch?v=PxMazut6-0U&list=UUOv9LmEtLR-HzDcupGT6UIw

I see BitInstant is promoting itself on the webshow hosted by Bruce Wagner, who was convicted of mortgage fraud and fined a quarter million dollars.

BitInstant does not support or condone fraud, does it?
Have you or any other member of BitInstant ever been convicted of fraud?

Is your indifference towards Bruce's past indiscretions, viewing going on his show a "means to an ends," indicative of your business values as a whole?
Do you believe self-promotion, and the resulting profits, is worth associating your business with someone convicted of fraud?

Thank you,
-Ragged

Yes...Yes...and double YES baby!!!

fyi- by hotlinking Bruce's youtube video you just made him some more money! God I love capitalism 

Cheers mate,

Charlie

Quoting to save your response.

[Yankee (BitInstant)]

http://www.youtube.com/watch?v=PxMazut6-0U&list=UUOv9LmEtLR-HzDcupGT6UIw

I see BitInstant is promoting itself on the webshow hosted by Bruce Wagner, who was convicted of mortgage fraud and fined a quarter million dollars.

BitInstant does not support or condone fraud, does it?
Have you or any other member of BitInstant ever been convicted of fraud?

Is your indifference towards Bruce's past indiscretions, viewing going on his show a "means to an ends," indicative of your business values as a whole?
Do you believe self-promotion, and the resulting profits, is worth associating your business with someone convicted of fraud?

Thank you,
-Ragged

Yes...Yes...and double YES baby!!!

fyi- by hotlinking Bruce's youtube video you just made him some more money! God I love capitalism  

Cheers mate,

Charlie

Quoting to save your response.

Yes, we know we went on the Bruce Wagner show and yes we know of the controversy that swirls around Mr. Wagner, unfortunately there are not that many other good sources of free traffic, so as a business that wants to succeed, we were forced to make a decision. However, we fully stand behind this decision.

We apologize if this made you feel uncomfortable or think that BitInstant's services were in any way fraudulent. I'd like to stress that BitInstant is not associated with Bruce Wagner or the Bitcoin Show other than appearing on his show (mainly for the free traffic).

I don't think I need to prove Bitinstants legitimacy here as we've transferred over half a million dollars so far and have had nothing but 100% positive feedback. You can read some reviews here: resellerratings.com/store/bitinstant

If you'd like to argue or discuss the merits of our decision this, feel free to reach me on skype, irc, phone or any other means. My information is freely available on Bitinstant.com
In addition, your more then welcome to contact my Legal team at Aini & Lazar PLLC in New York City.

-Charlie

[Anonymous]

http://www.youtube.com/watch?v=PxMazut6-0U&list=UUOv9LmEtLR-HzDcupGT6UIw

I see BitInstant is promoting itself on the webshow hosted by Bruce Wagner, who was convicted of mortgage fraud and fined a quarter million dollars.

BitInstant does not support or condone fraud, does it?
Have you or any other member of BitInstant ever been convicted of fraud?

Is your indifference towards Bruce's past indiscretions, viewing going on his show a "means to an ends," indicative of your business values as a whole?
Do you believe self-promotion, and the resulting profits, is worth associating your business with someone convicted of fraud?

Thank you,
-Ragged

Yes...Yes...and double YES baby!!!

fyi- by hotlinking Bruce's youtube video you just made him some more money! God I love capitalism  

Cheers mate,

Charlie

Quoting to save your response.

Yes, we know we went on the Bruce Wagner show and yes we know of the controversy that swirls around Mr. Wagner, unfortunately there are not that many other good sources of free traffic, so as a business that wants to succeed, we were forced to make a decision. However, we fully stand behind this decision.

We apologize if this made you feel uncomfortable or think that BitInstant's services were in any way fraudulent. I'd like to stress that BitInstant is not associated with Bruce Wagner or the Bitcoin Show other than appearing on his show (mainly for the free traffic).

I don't think I need to prove Bitinstants legitimacy here as we've transferred over half a million dollars so far and have had nothing but 100% positive feedback. You can read some reviews here: resellerratings.com/store/bitinstant

If you'd like to argue or discuss the merits of our decision this, feel free to reach me on skype, irc, phone or any other means. My information is freely available on Bitinstant.com
In addition, your more then welcome to contact my Legal team at Aini & Lazar PLLC in New York City.

-Charlie

Bruce is one shady guy, but he is right, free traffic is free traffic, and I'd take that any day.

[Gareth Nelson]

I hope Yankee does not mind me mentioning this in public but I did ask him if it's wise to go on Bruce's show - he convinced me it was, and seeing the result i'm glad that he did.
Bruce may have a dodgy history, but nothing in that show should be taken as representing BitInstant as being associated with Bruce's past.

[RaggedMonk]

I hope Yankee does not mind me mentioning this in public but I did ask him if it's wise to go on Bruce's show - he convinced me it was, and seeing the result i'm glad that he did.
Bruce may have a dodgy history, but nothing in that show should be taken as representing BitInstant as being associated with Bruce's past.

Thanks for the insight. Glad to hear this.