For those who weren't on #bitcoin-otc last night during the MtGox downtime, MagicalTux discussed the issue and after bringing the exchange back online, stayed on for an impromptu Q&A. Since I don't know if #bitcoin-otc is archive anywhere, I figured I would trim the discussion and post it here for posterity. What follows is the most relevant of what I was able to capture.
We can agree that variety of active exchanges (decentralized exchange) is in the spirit of bitcoin. But I would like to thank MagicalTux and MtGox for bringing his site back online quickly and transparently. I'd also like to congratulate him on the success of MtGox. The mainstream coverage of bitcoin over the past month contributed to the surge in price, and is further evidence that bitcoin is being adopted as a new standard, and is here to stay. MtGox has played a crucial role in its success, and deserves a special thanks. Also, a congratulations to MagicalTux on his apparent marriage and new family.
Oh, and if anyone wondered what the name "MtGox" means, it is embedded below
[21:08] <+MagicalTux> in 0.05 seconds, I'm getting 313 connection attempts
[21:09] <+MagicalTux> that's ~6200 connections/second
[21:09] <Glyph_Melancholi> sorry, 312 of those are probably me hitting the reload button\
[21:09] <+MagicalTux> stamit: from various sources
[21:09] <inductor> MagicalTux, have you identified how many computers are targeting you?
[21:09] <+MagicalTux> stamit: probably going to block some legitimate ips too
[21:10] <+MagicalTux> I'll be storing bits of flood from tcpdump, and blocking those
[21:10] <+MagicalTux> please stop hammering mtgox during this time
[21:10] <+MagicalTux> or you'll be blocked too
[21:11] <+MagicalTux> stamit: I get ~6000 syn/second
[21:11] <+MagicalTux> from tons of ips
[21:11] <+MagicalTux> need to do some analysis
[21:37] <ultramancool> MagicalTux: yeah, any luck with the DDoS issue?
[21:42] <+MagicalTux> meh
[21:42] <+MagicalTux> I created an iptable with ~4000 rules, uses too much cpu
[21:49] <+MagicalTux> in fact when I tried to limit the ddos, the amount of traffic increased
[21:49] <+MagicalTux> I think there are people out there who are having fun
[21:59] <Dark_Apostrophe> Right, MagicalTux - you need a DDoS-proof cloud storage provider
[22:01] <+MagicalTux> Dark_Apostrophe: this is being done
[22:02] <+MagicalTux> btw traffic continues to increase~
[22:03] <shooter_mcgavin> MagicalTux : is there anything the community can do to assist you?
[22:03] <+MagicalTux> shooter_mcgavin: for now I guess we need patience
[22:04] <shooter_mcgavin> Are all of the connections coming from a country or IP range?
[22:04] <+MagicalTux> shooter_mcgavin: multiple
[22:21] <+MagicalTux> mtgox should be back "soonish"
[22:28] <+MagicalTux> mh
[22:29] <+MagicalTux> the attack seems to be slowing down a bit
[22:30] <+MagicalTux> djbaniel: I'm evaluating options
[22:30] <kornholi> MagicalTux: are they just synflooding
[22:31] <+MagicalTux> kornholi: no, it's real tcp connection attemps, if I unblock the ips they hit some urls on mtgox
[22:31] <+MagicalTux> kornholi: right now I'm dropping all incoming traffic, so I only get ~20Mbps worth of connection attempts
[22:32] <kornholi> MagicalTux: all of them are port 80?
[22:32] <+MagicalTux> kornholi: port 443 only
[22:32] <datathe1st> MagicalTux you run MtGox?
[22:32] <+MagicalTux> datathe1st: I try to
[22:34] <datathe1st> MagicalTux: 1 month ago I learned about MtGox and though wow. Now I'm chatting with the person who created it. Really cool.
[22:34] <+MagicalTux> datathe1st: I didn't created it
[22:34] <+MagicalTux> I handle it right now however
[22:35] <Dark_Apostrophe> What does the name Mt. Gox signify? Is it an actual mountain?
[22:45] <+MagicalTux> [11:44:53] <shooter_mcgavin> MagicalTux: I recommend blocking at the country IP range level to help get things under control <- I guess I'll start with russia
[22:46] <Dark_Apostrophe> MagicalTux: Not Vietnam?
[22:46] <+MagicalTux> Dark_Apostrophe: vietnam is #2
[22:46] <BaltarNZ> I've dealt with a few DDOS attacks over the years, generally I block everything, then open up slowly to certain geos
[22:46] <+MagicalTux> looks like those guys (btcex/etc) are really behind it
[22:46] <+MagicalTux> BaltarNZ: could be an idea too
[23:04] <+MagicalTux>
http://bitcointalk.org/index.php?topic=6931.0 <-
[23:05] <shooter_mcgavin> MagicalTux: did you add the IP range blocks? There should be entire blocks you can add
[23:06] <+MagicalTux> shooter_mcgavin: I'll add extra ddos protections first, if I just block they'll try different kinds of ddos and I'll have to do lots of stuff again
[23:10] <aaa3> man wtf no ddoss protection? what kind of business is magictux running here\
[23:10] <+MagicalTux> aaa3: we got ddos protectiosn up to some point
[23:49] <gdoteof_home_> any word from MagicalTux ?
[23:50] <+MagicalTux> gdoteof_home: check the forum
[23:51] <+MagicalTux> anyway that was a stupid move, if the goal was really to get valuation down
[23:52] <Omnifarious> MagicalTux: Is your upstream working to help you with the DDoS?
[23:55] <+MagicalTux> Omnifarious: upstream here is not ready to stop so small DDoS, need to use different upstream (move planned)
[00:15] <+MagicalTux> in the train now
[00:20] <+MagicalTux> mh
[00:20] <+MagicalTux> in fact I may be able to mitigate this easier than thought
[00:21] <+MagicalTux>
http://bitcointalk.org/index.php?topic=6827.msg101542#msg101542 <- I think I like this guy
[00:21] <Cusipzzz> lol...any ETA, MT ?
[00:22] <+MagicalTux> Cusipzzz, less than 24 hours, and less than 2 hours if I'm successful blocking all except some
[00:32] <+MagicalTux> who manages gribble ?
[00:32] <+sgornick> MagicalTux: nanotube
[00:33] * MagicalTux pokes nanotube
[00:33] <+nanotube> hehe sup guys
[00:33] <+MagicalTux> nanotube, can you tell me from which ip gribble connects to mtgox?
[00:33] <+nanotube> MagicalTux: stand by for pm
[00:35] <+MagicalTux> ;;bc.mtgox
[00:35] <@gribble> Error: "bc.mtgox" is not a valid command.
[00:35] <+MagicalTux> ;;bc,mtgox
[00:35] <@gribble> {"ticker":{"high":4.15,"low":2.75,"vol":65304,"buy":3.421,"sell":3.57,"last":3.5}}
[00:35] <Rothgar> wow it works!
[00:35] <+MagicalTux> that's gribble only
[00:36] <+MagicalTux> the ddos is currently at 1/6th of what it was a bit earlier
[00:38] <Androgynous> DDoSers still raping it?
[00:38] <Androgynous> D=
[00:38] <+MagicalTux> Androgynous, ddos has calmed down a bit
[00:38] <Diablo-D3> MagicalTux: seriously, move to rapidxen
[00:39] <Diablo-D3> MagicalTux: what vps provider are you on now?
[00:39] <+MagicalTux> Diablo-D3, running on 4 dedicated servers
[00:39] <Diablo-D3> MagicalTux: are you fucking kidding me?
[00:39] <Diablo-D3> MagicalTux: you're doing it seriously wrong
[00:40] <Diablo-D3> MagicalTux: mtgox must have the shittiest code base ever
[00:40] <+MagicalTux> Diablo-D3, it is
[00:40] <+MagicalTux> that's why I'm rewriting it
[00:40] <Diablo-D3> I mean, fuck, my pool software should take like
[00:40] <Diablo-D3> half of a server
[00:40] <+MagicalTux> initially when Jed wrote it, it wasn't planned it'd get that many accesses
[00:40] <Diablo-D3> and service 10 times more people
[00:40] <Diablo-D3> MagicalTux: dude
[00:40] <Diablo-D3> hes doing it wrong
[00:40] <Diablo-D3> you ALWAYS assume infinite users
[00:40] <+MagicalTux> Diablo-D3, that's one of the reasons why I'm here now
[00:40] <Diablo-D3> because thats how ddoses fuck you
[00:41] <ghshephard> Diablo-D3 - I think MagicalTux agreed with you.
[00:41] <ghshephard> :-)
[00:41] <Diablo-D3> ghshephard: Im just bitching
[00:43] <+MagicalTux> Diablo-D3, I see at least 4 different patterns in the attacks I'm getting here
[00:45] <+MagicalTux> mh
[00:46] <+MagicalTux> Diablo-D3, the botnet in front of me is now presenting over 65000 different ips with traffic coming from all upstream providers at the same time
[00:46] <+MagicalTux> ip spoof + syn attack + regular attack
[00:46] <Diablo-D3> MagicalTux: yuck
[00:46] <mrb_> MagicalTux: how much traffic do you see?
[00:46] <+MagicalTux> mrb_, not that much, it's just traffic specially crafted to use the most ram possible
[00:47] <ntosme2> MagicalTux: well I can load the page now so you're doing something right
[00:47] <+MagicalTux> TCP: Possible SYN flooding on port 443. Sending cookies.
[00:47] <+MagicalTux> net_ratelimit: 7032 callbacks suppressed
[00:47] <+MagicalTux> TCP: Possible SYN flooding on port 443. Sending cookies.
[00:47] <+MagicalTux> net_ratelimit: 16646 callbacks suppressed
[00:47] <Glyph_Melancholi> Houston, We have liquidity!
[00:47] <Glyph_Melancholi>
[00:47] <Diablo-D3> MagicalTux: syncookies will help with syn attacks
[00:48] <Cusipzzz> MagicalTux: pretty fast now for me
[00:48] <+MagicalTux> Diablo-D3, it does
[00:48] <Glyph_Melancholi> all hail magicaltux!
[00:48] <kgo> da2ce7: If you believe the forums, it's because mtgox cost a competing exchange 160 bucks.
[00:48] <+MagicalTux> kgo, let me give the guy his 160 or 320 bucks, and let's end it
[00:49] <Rothgar> ;;bc,mtgox
[00:49] <@gribble> {"ticker":{"high":4.15,"low":2.75,"vol":62528,"buy":3.421,"sell":3.5699,"last":3.55}}
[00:49] <BitcoinExchange> yay!
[00:49] * Glyph_Melancholi starts dumping his 0.23 bitcoins
[00:49] <+MagicalTux> it seems to hold in the meantime, and I'll move mtgox to a special ddos-resisting system
[00:50] <mewantsbitcoins> MagicalTux don't give money away. Instead sort it out. It won't be long before others come along to ddos you
[00:50] <amiller> MagicalTux, good show
[00:50] <sneak> MagicalTux: what is the etymology of the name Mt Gox?
[00:50] <+MagicalTux> mewantsbitcoins, I was jocking
[00:50] <sneak> i've had several people ask me
[00:51] <+MagicalTux> sneak, initially it was "Magic The Gathering Online eXchange", it was adapted to bitcoin a while ago, and now changing the name would make the exchange lose part of its charm
[00:51] <cyphase> *comes down from heart attack*
[00:51] <+MagicalTux> mewantsbitcoins, just joking about how 160 bucks can cost a few hours of uptime
[00:51] <ghshephard> MagicalTux - Congratulations. Good working bringing it back online.
[00:51] <cyphase> i fell asleep with my mtgox money in bitcoin. luckily it didnt plummet
[00:52] <Cusipzzz> i like to think of it as Mt. Gox, the Mountain of Gox where moses brought down the tablets that defined what a currency should be
[00:52] <DavidSJ> I always thought the "mt" in "mtGox" stood for MagicalTux, not "Magic The".
[00:52] <+MagicalTux> :p
[00:52] <+MagicalTux> DavidSJ, that works too
[00:53] <+MagicalTux> someone sent me a photoshop of "Mount Gox"
[00:53] <+MagicalTux> I have that somewhere in the archives
[00:53] <Glyph_Melancholi> I always thought it was "Magical Tux's Gathering for Online Exchange"
[00:53] <Rothgar> ;;bc,mtgox
[00:53] <@gribble> {"ticker":{"high":4.15,"low":2.75,"vol":62573,"buy":3.422,"sell":3.5698,"last":3.55}}
[00:54] <+nanotube> yay it is up. MagicalTux ++
[00:54] <aninoni> hey, I just got home and I'm trying to follow what's up with mtgox and btcex. What's the latest word on the street?
[00:55] <sneak> MagicalTux: thanks for the info
[00:55] <noagendamarket> he should rename it to btsex as now he will be fucked
[00:55] <+MagicalTux> aninoni, for now it seems stable, trades at 3.55 on mtgox
[00:56] <+MagicalTux> ;;bc,mtgox
[00:56] <@gribble> {"ticker":{"high":4.15,"low":2.75,"vol":62576,"buy":3.5,"sell":3.564,"last":3.5}}
[00:56] <aninoni> oh nice...I didn't know it was back up. I thought mtgox decided to pull their site for 24h
[00:56] <noagendamarket> idnar
[00:56] <+MagicalTux> aninoni, I'm trying to have it up
[00:58] <+MagicalTux> I have to thank those russian for ruining my post-wedding reunion with my new family
[00:58] <Diablo-D3> MagicalTux: seriously, move to rapidxen
[00:59] <+MagicalTux> Diablo-D3, their live chat didn't answer when I asked about how they'd handle this kind of ddos
[00:59] <Diablo-D3> MagicalTux: just go on #systeminplace on irc.staticbox.net
[01:00] <Diablo-D3> wait
[01:00] <Diablo-D3> MagicalTux: what livechat?
[01:00] <Rothgar> ;;bc,mtgox
[01:00] <@gribble> {"ticker":{"high":4.15,"low":2.75,"vol":62651,"buy":3.54,"sell":3.57,"last":3.54}}
[01:00] <+MagicalTux> Diablo-D3, on their site
[01:01] <Diablo-D3> hahah
[01:01] <Diablo-D3> wow
[01:01] <Diablo-D3> it has one now
[01:01] <Diablo-D3> LOL
[01:01] <+MagicalTux> I was "connected to an operator" but it wasn't really live
[01:01] <Diablo-D3> MagicalTux: just go on the irc channel
[01:01] <warpi> "Loadavg too high" @ mtgox
[01:02] <+MagicalTux> warpi, I added a protection based on loadavg
[01:02] <warpi> MagicalTux, oki
[01:02] <+MagicalTux> warpi, it has the intended effect
[01:02] <warpi> MagicalTux, i was thinking, is it a lot of traffic for
http://mtgox.com/code/data/getDepth.php ?
[01:03] <+MagicalTux> warpi, normally high
[01:03] <datathe1st> is there a live ticker for mtgox?
[01:03] <Glyph_Melancholi> me too
[01:03] <Glyph_Melancholi> loadavg too high[01:04] <byronbb> #bitcoin-market streams mtgox
[01:04] <+MagicalTux> Glyph_Melancholi, do not refresh the "loadavg too high" message too much, or you'll be firewalled out
[01:04] <Glyph_Melancholi> .... too late
[01:04] <datathe1st> 3.54
[01:04] <+MagicalTux> heh
[01:05] <+MagicalTux> loadavg reducing
[01:06] <warpi> MagicalTux, isnt it possible to se which ips is bad and which one is good, and give "loadavg" to the bad ips?
[01:06] <joecool> MagicalTux: I was hearing you take dwolla now somehow?
[01:06] <+MagicalTux> joecool, yep, manually for now, interested?
[01:06] <VADiUM> warpi, probably not because its SYN flood with fake ip
[01:07] <+MagicalTux> warpi, loadavg is back to normal
[01:07] <warpi> ok
[01:07] <+MagicalTux> the loadavg message is mainly made to appear when loadavg is abnormal, so the blocking scripts are still working
[01:07] <toffoo> does "Loadavg too high due to ddos, please retry in ~5 min" mean i've been firewalled out?
[01:08] <+MagicalTux> ddos is still subsiding at same rate of -6Mbps/hour
[01:08] <+MagicalTux> toffoo, no
[01:08] <+MagicalTux> toffoo, firewalled out = timeout
[01:09] <joecool> MagicalTux: should I contact you again when I plan to do it?
[01:10] <+MagicalTux> joecool, yep
[01:10] <+MagicalTux> joecool, just mail
info@mtgox.com[01:11] <da2ce7> just read:
https://www.bitcoin.org/smf/index.php?topic=6864.0[01:11] <da2ce7> crazy stuff
[01:12] <da2ce7> I don't think that anyone dissagrees that we need a more decentralized extange system... and people are busy building those systems RIGHT NOW... but they are NOT READY YET!
[01:15] <+MagicalTux> da2ce7, I agree
[01:37] <zbanks> Debtors always get pissed when deflation happens. bitcoinex is acting pretty rationally egotistically ddos'ing, even though its a complete asshole move
[01:38] <+MagicalTux> new decrease speed for ddos: -1Mbps/hour
[01:38] <+MagicalTux> zbanks, he just risks causing the reverse result
[01:38] <+MagicalTux> ie. people who own bitcoins stopping sale, and value increasing even more
[01:38] <zbanks> Yeah, I'm curious to see what happens
[01:39] <+MagicalTux> for now I could get things back up before anything major happens
[01:39] <+MagicalTux> I hope
[01:40] <zbanks> Good luck!
[01:44] <llama> true that bitcoinex is DDOSing mtgox?
[01:44] <+MagicalTux> llama, sometimes he says so, sometimes he says it's not him
[01:45] <+MagicalTux> I got an email from him yesterday, about one hour before the first ddos
[01:45] <Lachesis> MagicalTux, what location are the requests coming from?
[01:45] <+MagicalTux> Lachesis, worldwide
[01:45] <Lachesis> country, i mean
[01:45] <Lachesis> hmm
[01:45] <+MagicalTux> also a good deal of spoofed ips
[01:46] <cyphase> ah, so they're just flooding the connection, not necessarily blocking the server(s)
[01:46] <shooter_mcgavin> mtgox.com is back up! !
[01:46] <shooter_mcgavin> lol jk.
[01:46] <shooter_mcgavin> >_> /cry
[01:46] <+MagicalTux> nah, there was also direct attacks on the application
[01:46] <Lachesis> cyphase, that's how ddoses usually work
[01:46] <+MagicalTux> shooter_mcgavin, it's been up for a while
[01:46] <shooter_mcgavin> =O
[01:46] <shooter_mcgavin> WHAAAAT
[01:46] == aninoni [~Eric@69.76.219.125] has quit [Ping timeout: 260 seconds]
[01:46] <+MagicalTux> ;;bc,mtgox
[01:46] <@gribble> {"ticker":{"high":4.15,"low":2.75,"vol":62950,"buy":3.56,"sell":3.7146,"last":3.54}}
[01:47] <Beremat> ;;gpg register Beremat 6D6796B4AB8D00DE
[01:47] <donchate> ;;gpg verify
http://pastebin.com/raw.php?i=yLPnXU0f[01:47] <@gribble> Request successful for user Beremat. Your challenge string is: b2d88c9af8849a142a22d262a070ec05d2e4bcb1567733675b5e39884a853f1f
[01:47] <@gribble> Error: Signature verification failed.
[01:47] <shooter_mcgavin> what happened, MagicalTux
[01:47] <+MagicalTux> shooter_mcgavin, ddos strnenght has dropped enough so I could the the service back up
[01:47] <shooter_mcgavin> for now at least?
[01:48] <+MagicalTux> yep
[01:48] <spz5> need ddos proof hosting, or xfer to amazon's elastic cloud during an attack
[01:48] <+MagicalTux> will move to ddos-proofed servers soon
[01:48] <Beremat> ..okay, I've never used gpg before. is that error from gribble up there a bad thing?
[01:48] <Beremat> or is it normal
[01:48] <+MagicalTux> Beremat, normal
[01:48] <Beremat> okay, thanks
[01:48] <+MagicalTux> Beremat, read the help on how to identify with gpg
[01:49] <+MagicalTux> the error was for donchate
[01:49] <Beremat> ah I see
[01:49] <Beremat> thanks
[01:50] <casascius> I have a couple of questions about MtGox... I would be interested to know how much BTC is kept "online" (versus on removable media), percent to total. OK to ask?
[01:51] <+MagicalTux> casascius, the amount kept online depends on the observed rate of deposit vs withdraw, plus some security, so I cannot disclose it
[01:52] <plato_> ha this WaPo mention was funny
[01:53] <plato_> "these payment systems would be only as reliable as the individual companies’ software and servers...nothing would prevent the operators of the platform from issuing more...That’s why we have the Fed."
http://www.washingtonpost.com/opinions/imagining-a-world-without-the-dollar/2011/04/26/AFjawKEF_story_1.html[01:54] <+MagicalTux> plato_, nice humoristic article
[01:54] <casascius> magicaltux, sure, thanks. Second lingering question is, can you give any opinion as to the security of mtgoxUSD? I mean, what if Fed decides MtGox is "aiding money laundering" or some crock, and seizes USD.
[01:54] <donchate> ;;gpg verify
http://pastebin.com/raw.php?i=gj2SQ4yY[01:54] <@gribble> Registration successful. You are now authenticated for user 'donchate' with key 4CB28DB851A84E1B
[01:54] <casascius> magicaltux, I realize MtGox is not in USA, but I imagine it has accounts in USA
[01:55] <+nanotube> casascius: nobody is secure against the fed. even multibillion dollar corps like pokerstars and fulltiltpoker
[01:55] <+MagicalTux> casascius, legally they cannot preven you from withdrawing your funds back, unless they get specific rights (ie. terrorist activity, etc)
[01:55] <+MagicalTux> we do legal bookkeeping and keep everything tidy so we should be able to make them happy enough to avoid being in trouble
[01:56] <Quetzalcoatl_> I thought Mt. Gox was up.
[01:56] <+Keefe> Unirgy: not here
[01:56] <+MagicalTux> Quetzalcoatl_, it is somewhat
[01:56] <Unirgy> it's so damn hard to buy bitcoin
[01:56] <noitev> ill sell
[01:56] <+MagicalTux> Unirgy, you try to buy with what payment method?
[01:57] <casascius> magicaltux, that makes sense. Third and final lingering question was, do you have any willingness or means to undergo a sort of self-audit similar to what financial online service providers in the US go through, currently known as SAS 70...(it's a lot less intrusive than it sounds)
[01:57] <+MagicalTux> casascius, we have plans to implement proper auditing, once we finish dealing with the new software/db
[01:57] <Unirgy> MagicalTux: until now i used only standard methods of payment, like paypal
[01:58] <+MagicalTux> Unirgy, try dwolla
[01:58] <Unirgy> since i was learning about bitcoin, i've registered on numerous exchanges and got LR account
[01:58] <Unirgy> MagicalTux: i do have dwolla
[01:58] <casascius> magicaltux, Example, I run a business that provides payroll-related services. One thing our customers demand from us is a SAS 70 document, which is nothing more than a 30-50 page document from a CPA who visited us and basically went through a checklist, mainly that covered operational controls.
[01:58] <Unirgy> MagicalTux: how can i use it to get bitcoins?
[01:59] <+MagicalTux> Unirgy, mtgox.com, then mail
info@mtgox.com to add funds via dwolla
[01:59] <casascius> magicaltux, They focus little on the software, and focus more on getting our ANSWERS to a list of pre-defined questions someone has already thought would be worthy for online service providers, and then renders an opinion as to how accurate our answers are.
[01:59] <Quetzalcoatl_> MagicalTux: Will the new software allow us to deposit into Mt. Gox with Dwolla?
[01:59] <+MagicalTux> Quetzalcoatl_, soon
[01:59] <+MagicalTux> casascius, could you mail me all that?
[01:59] <mibitcoin> MagicalTux: Are there any plans to implement KYC (Know-Your-Customer) requirements for MtGox users?
[02:00] <+MagicalTux> mibitcoin, optionnally, for big traders
