burnside
Legendary
 Offline
Activity: 1106
Merit: 1004
Lead Blockchain Developer
|
 |
May 22, 2013, 01:56:00 AM |
|
The bottleneck is bitcoind. I spent a bunch of time last night working on shifting requests away from bitcoind, that code is live now on litecoinglobal.com. If it holds up, I'll roll it out to btct.co soon. Eventually we'll shift everything but deposits/withdrawals out of bitcoind, but it will be a very slow process unfortunately.
Why don't you run bitcoind on a seperated server? It doesn't fix the problem but at least bitcoind won't slow down normal trading. Personally I don't care if I have to wait a few minutes for my withdrawal or deposit to show up but I hate it when I can't execute trades. The site is currently made up of 4 servers. Not going to go into detail regarding the layout, but bitcoind as of last week (when I posted about working on a server improvement) has been running on a dedicated SSD-backed server. It has a 140 MB wallet and still takes ages to process getBalance requests. My guess is that it's iterating through all of the thousands of transactions in the BerkeleyDB on every request or something because it performs very poorly. Cheers. |
|
|
|
|
|
|
|
|
|
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
|
|
|
|
|
stslimited
|
|
May 22, 2013, 11:53:12 AM |
|
errrrr not sure if I should transfer more to my btct.co wallet with site being wonky
|
|
|
|
|
sharky101
Member
Offline
Activity: 108
Merit: 10
|
|
May 22, 2013, 12:22:11 PM |
|
Is the site down for maintenance? Or under DDOS? Hate to see Cloudflare.
|
|
|
|
|
bizz
|
|
May 22, 2013, 01:35:06 PM |
|
Looks to me like growing pains. Pretty serious volume last few days.
|
|
|
|
|
usagi
VIP
Hero Member
Offline
Activity: 812
Merit: 1000
13
|
|
May 22, 2013, 01:40:23 PM |
|
Error:
backend server failure at 145 What's this? |
|
|
|
|
kokojie
Legendary
Offline
Activity: 1792
Merit: 1003
|
|
May 22, 2013, 01:48:07 PM |
|
The bottleneck is bitcoind. I spent a bunch of time last night working on shifting requests away from bitcoind, that code is live now on litecoinglobal.com. If it holds up, I'll roll it out to btct.co soon. Eventually we'll shift everything but deposits/withdrawals out of bitcoind, but it will be a very slow process unfortunately.
Why don't you run bitcoind on a seperated server? It doesn't fix the problem but at least bitcoind won't slow down normal trading. Personally I don't care if I have to wait a few minutes for my withdrawal or deposit to show up but I hate it when I can't execute trades. The site is currently made up of 4 servers. Not going to go into detail regarding the layout, but bitcoind as of last week (when I posted about working on a server improvement) has been running on a dedicated SSD-backed server. It has a 140 MB wallet and still takes ages to process getBalance requests. My guess is that it's iterating through all of the thousands of transactions in the BerkeleyDB on every request or something because it performs very poorly. Cheers. Why not regularly shrink the wallet size? 140MB wallet is pretty outrageous. I don't think anyone else is running bitcoind with a wallet of that size. |
btc: 15sFnThw58hiGHYXyUAasgfauifTEB1ZF6
|
|
|
|
parseval
|
|
May 22, 2013, 01:49:45 PM |
|
Looks to me like growing pains. Pretty serious volume last few days.
All of this growth is great for BTC-TC. By my count, it looks like BTC-TC has traded more than 20k BTC worth of securities so far for May. I hope the DDoS issue resolves itself. I understand that DDoS is a particularly difficult/tedious problem to resolve (it's akin to playing whack-a-mole), but I have faith in burnside's ability. Just be patient, and don't lose confidence. If this is another DDoSer, that's exactly what they want. |
|
|
|
|
gramma
|
|
May 22, 2013, 01:51:09 PM |
|
My guess is that in addition to any ddos that may be happening, people are trying to get in a last-minute purchase before ASICminer dividends hit. Or hammering the server trying to see how rich they are this morning.
|
BTC: 1MrNRPo7p8DEyxn87c9BCGwrbatBQeCHc1
|
|
|
TaxReturn
Member
Offline
Activity: 67
Merit: 10
|
|
May 22, 2013, 01:56:43 PM |
|
Given the track record of former exchanges even the slightest bump makes me a little uncomfortable. I really hope this one isn't going the way of the dodo too...
|
|
|
|
|
burnside
Legendary
Offline
Activity: 1106
Merit: 1004
Lead Blockchain Developer
|
|
May 22, 2013, 02:53:48 PM |
|
Hi all. More DDoS this morning. I'm pretty sure they got around the DDoS protection by extracting the server's IP from the emails the site sends. They were attacking the server's base IP and email is the only real give-away I can think of. (open to suggestions from you other site operators out there.) We've had to move the server and all outbound emails have been suspended for the time being. Given the track record of former exchanges even the slightest bump makes me a little uncomfortable. I really hope this one isn't going the way of the dodo too...
Not if I can help it. Looks to me like growing pains. Pretty serious volume last few days.
All of this growth is great for BTC-TC. By my count, it looks like BTC-TC has traded more than 20k BTC worth of securities so far for May. I hope the DDoS issue resolves itself. I understand that DDoS is a particularly difficult/tedious problem to resolve (it's akin to playing whack-a-mole), but I have faith in burnside's ability. Just be patient, and don't lose confidence. If this is another DDoSer, that's exactly what they want. Thank you, I definitely appreciate that. I'll work today on figuring out how to restore the email system. Cheers. |
|
|
|
|
|
ThickAsThieves
|
|
May 22, 2013, 02:58:43 PM |
|
Hi all.
More DDoS this morning. I'm pretty sure they got around the DDoS protection by extracting the server's IP from the emails the site sends. They were attacking the server's base IP and email is the only real give-away I can think of. (open to suggestions from you other site operators out there.)
We've had to move the server and all outbound emails have been suspended for the time being.
Talk to topace/lightbox, he figured this out I think. |
|
|
|
|
|
donut
|
|
May 22, 2013, 02:59:59 PM |
|
The site is currently made up of 4 servers. Not going to go into detail regarding the layout, but bitcoind as of last week (when I posted about working on a server improvement) has been running on a dedicated SSD-backed server. It has a 140 MB wallet and still takes ages to process getBalance requests. My guess is that it's iterating through all of the thousands of transactions in the BerkeleyDB on every request or something because it performs very poorly.
Cheers.
Have you thought about using a ramdrive to host the wallet? Should improve things quite significantly speed-wise. Obviously it should have sane backup procedures in case of power loss. |
|
|
|
|
|
CanadianGuy
|
|
May 22, 2013, 03:10:42 PM |
|
If this attacker thinks he can lower the share price, he's mistaken. I think people are more likely to hold onto a good dividend paying stock through something like this. If the share price goes down, it becomes even dumber to sell because the dividend % will be higher. If we only depended on share price, then maybe this would have more of an affect. If manipulating price is his motivation, he's a fool  |
|
|
|
|
|
Carnth
|
|
May 22, 2013, 03:11:14 PM |
|
Given the track record of former exchanges even the slightest bump makes me a little uncomfortable. I really hope this one isn't going the way of the dodo too...
I have confidence in Burnside. Hosting servers is his business (seriously, look at his sig). Besides, even if there is a catastrophic event, all the asset issuers have API access to all the shareholder data (and emailed twice a day just to be sure). It would be very easy to move on--this is one of the best features of btct.co/litecoinglobal.com. Have you thought about using a ramdrive to host the wallet? Should improve things quite significantly speed-wise. Obviously it should have sane backup procedures in case of power loss.
I think the only way to really resolve the bitcoind wallet problem is to move the wallet from a Berkeley DB file into a dedicated database like MySQL or MariaDB. I have no idea how other sites do this, but it would take care of the problem. |
|
|
|
|
burnside
Legendary
Offline
Activity: 1106
Merit: 1004
Lead Blockchain Developer
|
|
May 22, 2013, 03:31:48 PM |
|
The site is currently made up of 4 servers. Not going to go into detail regarding the layout, but bitcoind as of last week (when I posted about working on a server improvement) has been running on a dedicated SSD-backed server. It has a 140 MB wallet and still takes ages to process getBalance requests. My guess is that it's iterating through all of the thousands of transactions in the BerkeleyDB on every request or something because it performs very poorly.
Cheers.
Have you thought about using a ramdrive to host the wallet? Should improve things quite significantly speed-wise. Obviously it should have sane backup procedures in case of power loss. I'm actually leaning toward the bottleneck being something CPU related. When the site slows down, the cpu on bitcoind pegs at 100% and disk IO does not seem to spike a whole lot. (I've been graphing the system via collectd.) Since bitcoind is not multithreaded, I think I'm pretty much at it's limit. |
|
|
|
|
Garr255
Legendary
Offline
Activity: 938
Merit: 1000
What's a GPU?
|
|
May 22, 2013, 04:01:27 PM |
|
I'm actually leaning toward the bottleneck being something CPU related. When the site slows down, the cpu on bitcoind pegs at 100% and disk IO does not seem to spike a whole lot. (I've been graphing the system via collectd.)
Since bitcoind is not multithreaded, I think I'm pretty much at it's limit.
I've never seen it done before, but would running two bitcoind servers (and subsequently moving half of the accounts to the new one) mitigate the problem? |
“First they ignore you, then they laugh at you, then they fight you, then you win.” -- Mahatma Gandhi
Average time between signing on to bitcointalk: Two weeks. Please don't expect responses any faster than that!
|
|
|
|
jjdub7
|
|
May 22, 2013, 06:18:58 PM |
|
Hi all.
More DDoS this morning. I'm pretty sure they got around the DDoS protection by extracting the server's IP from the emails the site sends. They were attacking the server's base IP and email is the only real give-away I can think of. (open to suggestions from you other site operators out there.)
We've had to move the server and all outbound emails have been suspended for the time being.
Thanks for the prompt update burnside |
|
|
|
|
|
InstantBTC
|
|
May 22, 2013, 06:47:35 PM |
|
I'm unable to get the /transfers to load at all. /security/*, /account, and /portfolio load fine now, most of the time, but /transfers won't load at all.
|
|
|
|
|
gimme_bottles
|
|
May 22, 2013, 06:59:57 PM |
|
anyone else not able to login using 2FA? leaving the 2FA field blank gives me a missing 2FA error (obviously), using 2FA takes forever to load and finally redirects me to a cloudflare site telling me the requested site is offline.
|
|
|
|
|
|
carnitastaco
|
|
May 22, 2013, 07:22:10 PM |
|
anyone else not able to login using 2FA? leaving the 2FA field blank gives me a missing 2FA error (obviously), using 2FA takes forever to load and finally redirects me to a cloudflare site telling me the requested site is offline.
Even if I can log in successfully, once I get to trying to make a trade it hangs and then eventually comes back with an invalid gauth message. pretty frustrating, been trying to swap some of my 1/100 for 1/1 all morning |
|
|
|
|
|
