Bitcoin Forum

..

There are some very clever hackers hiding on this forum and in bitcoin-world in general. You need to stay on your toes at all times.

Yeah, especially around members that haven't posted in 3 years and then post 20 posts in 1 day, like you.

You can always leave and also the OP can always leave http://techforum.it/styles/default/xenforo/smilies/asd.gif. However, yes this time due a social engineering attack... really impressive how it was easy.

Actually Theymos already said it wasn't a social engineering attack, they just havent said what it was yet.

Ignore this if you want to read something interesting. I only want to say thank you to owners of BT. There were some downs but you did repair it, and if someone is angry of you, why ? You can't protect forum of 100% and we should know that you are doing it well as you can.

Well, it's impossible to be completely secure from an attack... they admins ARE human.  My concern is how easily the hackers were able to get in.

How do you know it was easy? How do you know that this was the first attempt?

I would hope that previous social engineering attacks would have been reported both internally (within the company) and to theymos. This would have allowed both to take additional precautions to prevent this kind of attack.

It is however possible that getting KVM access was attempted multiple times.

You have to realize that hackers hate BCT. They've been hacked and had long downtimes quite a lot in the history.

Dude, they're making money hand over nutsack. You don't have to thank them for anything. They should be thanking us.

Yeah, Bitcointalk is one of the larger forums in the world, I don't know the actual statistic, but I'm sure it gets a lot more attempted attacks than is publicly known. I dont think the forum's track record is all that bad though, two or three hacks come to mind in 5 years. Some DDOS too, but you can't really prevent that.


Heh, what do you think that money that is made hand over nutsack is used for? Creating new forum software, security bounties, etc.

And your salaries, is it not?

The Staff and Admins split somewhere around 10-15% of monthly advertising revenues. Donators funds have never been touched. Mod payments aren't considered salaries, they are considered tips as they aren't guaranteed, and for the time spent moderating, staff members are far better off getting a minimum wage job at a McDonalds. It is however a nice gesture, and a result of the forums not really needing any more money (the same reason donations are no longer solicited).

im betting theymos's password is "theymos is AWESOME" but seriously the amount this forum has earned for him you think hed keep on his toes about stuff. ddos attacks and hacking.

what would you suggest for preventing both? I'm sure theymos would be very grateful if you could lend your ideas. he might have money to invest but it doesn't grant him unlimited knowledge. its nt as simple as you make it sound. many multi million websites have been exploited and ddos

not going to say much but donations and stuff are there to support and fix issues in time of need. i dont know maybe everyones passwords getting stolen is a time of need. dont know why theymos sits on a large stack of BTCBTCBTC maybe he wants his own island or something. as for mod payments i dont want to upset you but recently its not the hardest job   :o but you still do a good job :) please dont shout qt me

How do the donators feel about that? I guess its better knowing they're still somewhere than not knowing what was done with them.

iv actually been through ddoss suggestions on here in the past. dont know if they didnt like my idea or was just ignored. id set up a few cheap vps load balancers. set software can only respond to certain requests so it filters out damaging traffic to the main website. i know ddos attacks are getting bigger and more complex but so are defenses. and in fairness this isnt a massive site so dosnt attract the worst ddos or hackers mostly because its a forum and little info/money to be gained from it. i do however respect the fact passwords where atleast encrypted... see  alot bigger sites fall at that point

id be pissed :) if donations are just sitting there then whats the point in donating? even worse when they could be being used to improve/tighten security.

Yep, donations and such are to fix issues in time of need. Some issues though aren't fixed by throwing all of your money at them. For example, this recent hack, there is no reasonable solution that Theymos could have done to prevent this. If you know the solution, let us know  ;)

I didn't say that moderating was hard, it just takes time. My point was that moderators are still technically volunteers, the staff (Theymos included) aren't bleeding the forums dry of funds. Donations have never been used to pay any Moderators/Admins, and funds or lack of funds isn't the issue. People hack websites that are targets, Bitcointalk is a target. The fact that the hackers didn't find a vulnerability in the site, and instead targeted the host says a lot. Facebook was hacked with a 1.05 Billion dollar private datacenter. We could spend every last penny to set up a tiny data center on an isolated island and hire one armed guard to prevent this same issue from happening again, but we can't really say that is reasonable solution.

*edit*


Donators were told that Donations would be used for new forum software, which is in production. I meant that donator's funds have never been touched by Staff/Admins.

Again, how would you increase security? There isn't a hole in which to throw money that gives you what you want. New more secure forum software is in production, but it wouldn't have helped in this case.



The forums does have multiple hosts to help mitigate DDOS attacks. One of those hosts is what allowed the latest hack to happen.

I posted the simple solutions elsewhere, I'll repost it here:

https://bitcointalk.org/index.php?topic=1069837.msg11453289#msg11453289

Easily preventable on two levels:

1) collocate your own equipment in a remote data center. The customer service staff will simply have no access to it besides being able to press buttons on the box.

2) use non-commodity hardware like Oracle SPARC or IBM POWER or HP Integrity/Itanium.  Then even if they manage to steal it they most likely will not be able to get the data off of it without specialized assistance.

Also, don't run Linux on those machines, but their native OS: Solaris, AIX, HP/UX respectively.

just had a look at the article and all i can say is what the hell. either root password was piss poor or he got lucky. and ontop of that no white list for ip login? thats asking to get ass raped. all ROOT logins should be white listed its a basic security feature or even 2 levels of security... like 2 passwords... be interesting to know if it was being brute forced to? and if it was how that many attempts went unnoticed maybe a failed login attempt warning would be a good idea

theres loads of defenses out there and now iv read the article im pretty shocked. i was expecting a datacenter backdoor being used not walking in the front door

https://bitcointalk.org/index.php?topic=1067985

a few suggesting it was an inside job by someone xd think its unlikely unless they where blackmailed into it. but still get a damn white list in place and do it now :P maybe i should come up with ideas for security its fun :D

maybe have a table in a secret location all mods get a key and only 2 mods turning the keys at the same time can change any forum code or access the servers... ooo with hand scanners and retinal scanners to confirm the changes... just a thought

I suppose its not entirely out of the question, but collocating our own equipment probably isn't the best idea either. It would be less than cost effective, and forum up time and reliability would be far less than it is with professional datacenters. What country to place the datacenter would be another issue, and hiring employees to manage it doesn't sound too appealing. It sounds like a complete mess, and something unnecessary for a forum. This is a website, a large one at that, but the Bitcoin forum isn't Google.

You are just bullshitting. I've been doing exactly that professionally (mostly as a consultant) for many years. Yeah, it is somewhat more expensive, especially in the upfront capital cost, but the operating expenses are frequently actually lower. It is a perfect solution for "a website" even with much less traffic than this one.

In particular the reliability is better if the owner of the equipment is conscientious and willing to learn, because there is no blaming "somebody else". Also, the customer service staff for the physical collocation customers is typically way more responsible and conscientious.

The "professional datacenters" that have equipment leasing included in their rental fees are the dreck of the datacenter business, because they by necessity serve mostly fly-by-nights. It seem to me like you've never owned the equipment in any datacenter so you don't really have a base to make a real judgment.

Tell me then, how much would it cost to set up a datacenter in a couple of countries, buy the equipment, and hire employees? I think you are vastly overestimating how much the advertising revenue brought in could support.

I'm not going to give you a quote just to prove myself, I charge for such services and I'm positive that you are neither serious nor authorized to purchase anything.

But one thing is worth mentioning: "hire employees". For a physical collocation "remote hands" services are usually available in increments of 15 minutes. What I'm positive is that after buying and paying for "remote hands" a couple of times, which normally involves a telephone/facetime/skype conversation with the remote contractor, the possibility "social engineering" essentially disappears.

The worst "hack"s that did happen on my watch was nothing more than an equipment destruction or theft (for wipe & resale).

Heh, no it would not had the forums not had years of reserves. What it brings in for advertising revenue is public, just check the closed auction threads to see what it could support.


I wasn't asking for something that you spent 20 minutes or more figuring out, I meant a rough figure, because for someone who knows what they are doing, you are either grossly overestimating the forum's budget or underestimating the cost of setting up multiple datacenters in various countries and the unpleasantness that would come with that. The forum could support a single full time employee perhaps, not multiple + building expenses + interesting tax implications for owning physical property, and more tax issues for having physical property in multiple countries. I'm no expert on the matter, but even with absurdly and unreasonably low cost assumptions, we are still vast sums of money apart. I could set up a datacenter in a shed in my backyard for $5k. If Theymos wants to take me up on that offer, I'd be happy to oblige.

Hey Mods, do you realize some people work in IT, some at small businesses who are renting servers too, knowing that it's not that big of a deal these days?

It's a pretty good record, that said the hackers could have made some coin from the bounties but I guess they thought hacking a userbase was a better ROI in the long run either way it is a lot of work
It does make me wonder if this is the last hack per se of the old forum, the new forum software's launch is getting closer as will a ton of holes in all likelihood.

is there something missing ? since bitcointalk forum hacked?
what is the major effect on this attack?

you should change your password.

other effects may be old accounts coming back to life...

***wonders if iv yet again been ignored on suggestions for forum security*** white list ips that can access the server or very least who can use root.

again very simple solution to a not so complex hack and add email alerts to every single attempted login and successful login

That really won't help much against something like this. The hackers would just ask the hosting company to change the whitelisted IP's too, they already reset the root password for the hackers, I don't see why they wouldn't change the whitelisted IP's too.

Giving you a quote is dependent upon many variables but as a rule of thumb Colocation is more secure and less expensive in the long term in almost all cases. Leasing a dedicated server is less expensive initially because you don't need to purchase the hardware, install it and have some backup parts on hand/or have funds set aside for a smart hand. Colocation has higher upfront costs and complexities but most small businesses can pay 50-150USD a month on colocation fees.

Example-
Leasing a dedicated server may cost ~150 USD a month for the first 6 month promotion and than ~250 USD a month thereafter. You could purchase a refurbished 2u server for a couple grand and spend 80 dollars a month in colocation fees.

A higher traffic site like Bitcointalk would need more expensive servers purchased and higher colocation fees  but would have much higher levels of savings as the dedicated server lease fees and bandwidth costs would be much more as well.

Colocation would likely offer better security and large savings in the longterm.

Dude, what can I say? You are not only a compulsive bullshit artist, but you've also mostly lost touch with reality. What buildings? What full time employees? What tax implications? One is true: .

I am an expert, but I'm not really interested in learning the finances of this forum. It is up to theymos to scan his tax returns for bitcointalk and call Dell Small Business (or any other large reseller of electronics) financial department and ask them how much credit he's going to get for his non-profit organization. Literally millions of small business' owners done that before him.

I could then discuss various technical details and options, but I'm too ethical to even joke about $5k hosting in a shed.

All I have to say is my school had lots of wisdom retaining and maintaining the old mainframe. It allowed us to learn not only the technical details of virtualization (it was called VM/370 then, not KVM or whatever) but also experience first hand the bullshit from the time-share salesmen. The "cloud" terminology was not invented then, everyone used "time-share". Nowadays the "time-share" is a dirty word related to the vacation package sales. But the infectious anti-technical sales bullshit permeating the business is the same as it was through the 1960-1980 when it was popular.

But before he's going to even scan the tax returns for the forum he'll need to ask himself a question "Do I give a flying fuck on a rolling donut about the information security of the members of this forum?" Maybe the true answer really is "I like to have a convenient 'scapegoat in the cloud'. I can always point to the sky and say 'It was their fault, not mine!'".

But in this case it was 'used' also a soc. eng. practice... and you can build the security that you want but if an employee will reset the root password it will be really a problem  :-\.

However I think (all) we are waiting more info. from theymos about this situation.

We are discussing the feasibility of creating our own hosting location, having our own facilities, not giving another random 3rd party access to the server. I thought you were aware that the hosting company staff were the weak link in this hack. The way to get around that, is to change to a facility operated by an employee of the Bitcointalk. That would involve building our own infrastructure, hiring staff to monitor its physical location, etc. That would also involve owning property to build on.

If we just rent server space from an already established company, we face the same issues. Not having complete trust of the people who have access to the server. So if we are talking about just changing hosts to something that isn't a large operation in a giant datacenter to a shared location with a couple of other people, we still have to worry about the human factor.

Ecatly, I have proposed in the past days the creation of an "home made" server to hosting the forum but I do not know if it is a really *possibilty or not.


*With all the money spent in the creation of the epochtalk forum software.

This is all quite clear to me.  ???
It's expensive to set up. It's not like you DL a wordpress style and host it on GoDaddy. Thanks for explaining.

This is an unusual statement to make as I don't think anyone is suggesting bitcointalk get into the expensive datacenter business or 2112 suggestions are 100% bulletproof. Every option has various tradeoffs and inherent costs.

Some other options besides fully managed/un-managed leased dedicated servers -
1) managed colocated servers
2) unmanaged colocated servers
3) leasing a cabinet with your own servers
4) leasing your own cage

I personally think it is a bit unusual that Theymos is paying for multiple managed or un-managed dedicated server leases, especially based upon the inherent security considerations of this forum and costs.  

There are ways to protect colocated servers as well from tampering.
https://www.racksolutions.com/secure-server-unit.html
is one example amongst many. Additionally, every datacenter I have worked in had many security cameras, armed guards, tracking keypasses, ect...

Oh, I know what you mean now. Heh, too much prior discussion of private built hosting centers on islands had me thinking in extremes.

My bad.

Its fine, I have often made the mistake of overly complicating solutions unnecessarily before realizing that simplicity is often superior which seems to be what is happening with epochtalk , but that is a whole other conversation.

Man, you really have comprehension problems.

I'll repeat: colocation a privately owned server solves nearly all the avenues of "social engineering" attack. By "privately owned" I mean server not leased from the hosting company but a server owned outright or leased independently from the server manufacturer or distributor. The colocation staff will then only have as much access as you decide to provide them, typically limited to pushing buttons, inserting media into tray and connecting cables. The hacking risk is limited to intentional damage or physical theft.

And please quit your "large operation" "giant datacenter" bullshit. Colocation space can be bough in 1U units (1.75 inch height, 19" wide, varying depth around 1meter). Employee time can be bought in quarter-hour increments. The "human factors" are limited to physical theft that is extremely rare and easier caught and prosecuted.

I can tell you have worked as a network administrator before, and agree with your comments. Additionally, collocation for a high traffic site like bitcointalk could save thousands of dollars in lease fees a month.

There are some downsides of course. More work setting it up, initial costs of servers, the need to occasionally upgrade the hardware and sell off old servers, ect... all these things can intimidate someone without experience but really isn't difficult and well worth it for high traffic sites that need better security. I am sure there must be one member of staff or moderator with a bit of data center experience?

^^ THIS + 100. as enterprise admin i can sign it. obviously you know what you are talking about.) regarding damage and theft, also these factors can be highly minimized, I visited couple of T4 high sec. data-centers around the world and except some military invasion or massive terrorist attack, I really can't even imagine simple physical "theft" from some rack. there were so many checks, scanners, cameras everywhere, guards everywhere, nobody alone in room rules..etc..

Regarding HW damage, this is solved by clustering in different buildings or even in different towns/countries..running services on fully redundant HW is not problem at all..

Yes, we have probably visited the same datacenters and the security is insane and way more involved than the average person expects. Multiple armed guards on each floor, cameras everywhere, lengthy background checks for network administrators , keypasses tracking me and only allowing me access to certain parts of the building, extensive backup power, EMP protection, backup cooling , ect--- layers and layers of security.

 RAID is no replacement for offsite backups or redundancy plans. Additionally, one should plan to have a backup server/s located in a country that doesn't have a history of being a lapdog to the US.