Bitcoin Forum

The Bitcoin.org site and associated forum were down for about 9 hours today, I think.

Some of us, using the IRC Chatroom, have expressed concern for this single point of failure.

There has been discussion about somehow making a distributed Forum... or an auto-syncing forum...

But, in the meantime, we have reacted swiftly.

On this page  http://www.bitcoinme.com/discussion-forum.html  I have listed no less that FIVE Bitcoin Forums we now have access to.   

It would be great to cross-link as much as possible, so that in the event of a major outage, people would instantly know where ELSE they could go for information.

What do you think?

I agree, currently in case of down of bitcoin.org the only remaining official information source is IRC, I think alternative urls should be either known, or included (for example on an help page in the bitcoin client) :)

Even if they were just listed, in as many places as possible...  Users would likely "bookmark" them in their browsers...  But it'd also be nice if they had access to a copy of the list-of-forums on several major Bitcoin sites...  (similar to how there are lists-of-exchange-sites all over the place).


That sounds like a great idea too.   The more well-written resources we have, the less chance we'll ever get so cut off again... as we were today.

Didn't it feel odd....!?   I think a LOT of people were feeling very cut off from their community....   It can be a little like crack....  :)

Might be a good idea to mirror the existing forum somewhere else as a backup.

Why was the site down?

It would be great if somebody make a mirror as a tor hidden service and in I2P network. Doing so we could survive gov/hackers attacks. I was very nervous and can't sleep due to this recent bitcoin.org shutdown.

This page seems down too :)

The new .p2p distributed dns system they are creating might hold the answer. The government will be unable to shut it down even if they seize the domain name.  ;D

A distributed, auto syncing forum?

What about Usenet?

Feel free to set that up.  :)

Usenet newsgroups can be accessed via the web using Google Groups, but to set up a new Usenet newsgroup (not just a "Google group") I believe you need a Usenet subscription.

Is there anyone here who has one?

So, are people actually serious about this idea?

This outage was a bit strange as it seemed to impact a couple of different Bitcoins related websites.  Perhaps that was coincidence as well.  The coincidence with getting a Reddit link seems to be more than mere coincidence too.  That is why I wasn't sure if this was a denial-of-service attack or simply a bad server that may have given up due to a crush of incoming volume.


Setting up a usenet newsgroup isn't too hard, particularly if you run a usenet server.  The problem is mainly one of if anybody wants to bother as Usenet has been so full of spam that it generally isn't worth reading anymore.  The nice thing about Usenet is that it is decentralized... not quite peer to peer but pretty close and allows "live" synchronization between the various archives so no single server being down will necessarily keep participants from continuing conversations on the various threads or discussion groups.  I've run some quality usenet server software that takes all of about 5-10 minutes of configuration, and if all you are doing is a "private" or "custom" newsgroups with peers that you trust or know about, you can ignore the other newsfeeder services altogether.  I would suggest avoiding Google unless you are more into the Google Groups themselves, not because of usenet services.

Most quality e-mail reader software like Thunderbird has built-in usenet readers as well.  It just takes getting used to using the forums in that manner.

Despite the downtime, we achieved 315 posts that last day of November. :D

I'd prefer the mirror approach too, rather than starting from scratch with something like usenet.  Just so that discussions can be picked up where they were left off.

Even a once daily backup wouldn't be too onerous, would it?

A failover would be nice. So we can continue talking while the original server is getting its shit together.

Just do backups.

Have nothing else, if something happens to the information will not be affected and community survive.

But a backup is just that, a backup.

There's no communication on the forum happening until SMF is running.

So, have we reach a decision making point on backup?

I think the consensus is:

1. Offsite backup, encrypted. Keys held by elder members.


Nice to have.

1. Failover solution. Some way to get the forum back up and running.

Freenet.

Thats not a backup, thats a separate forum all together, its an agorist marketplace

How long does it take to restore a backup on another SMF install?  Can't be that big a job, and Bruce Wagner has setup an SMF board over at his site already.

Tell us about Freenet. 

How often is this synced?

I run the earthsociety.org portal and the digital coin forum (small beans compared to bitcoin forum!). Would be willing to give a subdomain with ftp access to a maintainer to manage a mirror of bitcoin forum.

Painfully difficult to set up properly on your computer, and painfully slow once it is working.  It also makes a poor data haven (it's primary purpose) because the network has the habit of "forgeting" pieces of the file that you desire over time.  This happens because nodes have limited storage space, and favor popular content; and because there is no way to compensate anyone for their contributions, so nodes drop off without warning.  It's really mostly used as a huge encrypted BBS, distributing messages and content, much of it because it is objectionable to someone. 

FreeNet is a good idea.
Tor is a good idea.
Distriubted forum is a good idea (I have never heard of such a thing before :) )
IRC is a good idea.
Providing a list of multiple forums/points of bitcoin communicating inside each bitcoin distribution is a good idea.

Also maybe having a daily-updated tar.gz of the entire history of the forum would be good...and then everyone can download it as a torrent.

While it is true that the FreeNet network will forget stuff put up there once and never viewed, however, there are enough people using bitcoin forum and site that there will be plenty of people who can reseed it on FreeNet if it ever gets forgotten (which is unlikely, since hopefully there will be plenty of people accessing it regularly).

So how hard it is to get the forum data mirrored, anyone?

Yes, we should do something better than a HTML scrape.

This is the second time that the forum goes down. We should definitely do something about distributing and mirroring it.

Has anyone yet figured out why it is going down?

 No one seems to even know who runs this site!?   ???

Starting today, I will be double-posting every comment I post, on both this forum and the bitcoinme.com forum.   If I have time, I'll post it on the other 2 forums as well.... bitcoin.in and the Google Groups Bitcoin group.  

 We need our forums of communication to be decentralized as well.
 

It's Sirius and he speculate that there were too much memory usage. In other words, the bitcoin community is growing.

For the last recent downtime, Sirius did an upgrade on the server. No worry.

That's good to hear.  :)

 It would be nice if he could read a wrecked the page to a down for maintenance page.

( I love android voice to text. That mistake was so cute I thought I'd leave it in. :)  )

I recommend adding this to your forum signature....


    ( alternate bitcoin forums: http://bitcoinme.com ● http://bitcoin.in ● http://goo.gl/bzze1 )


...so everyone knows that there is more than ONE bitcoin forum on the internet.

Or we can Sirius here and discuss about how we're going to mirror the bitcoin forum database...

Resurrecting this topic; we can't get complacent about this subject.


So, the simplest solution to this problem is to simply back up the database to somewhere safe, no?

Can we get Sirius onboard with this idea?

The main site could be mirrored, but keeping the forum database in sync on multiple servers sounds difficult. Automated backups of the database to a remote location would be useful if something unexpected happens with the hosting. Any ideas how to store the backups? Maybe encrypt them and distribute to multiple servers around the world. Distributing them through a single gateway (Dropbox, Tahoe-LAFS?) would save bandwidth.

That would be a good start. Tahoe-LAFS would be nice to use also. (There's also possible synergy between Tahoe and Bitcoin)

I think we can handle trusting the seniors not to abuse the database data.

Any possible timetable?

Anybody here ever used Collanos, it might hold the answer.

I havent used it for 2-3years, but it is P2p, distributed etc. (Very similar to Microsoft Groove, now know as Sharepoint Workspace.)

Cheers

EDIT: Had a download and play, and it is probably not appropriate for what we want to do.

Why so difficult ? Why make it complicated ?

Make it simple. RSYNC, a simple-yet-advanced Linux program can distribute almost anything (even whole Linux partition) over SSH from server to server using public keys (passwords are not needed).
An automated script ( about 10-30 lines of code) can automatically ping main server, restore last backup and start backup forum once the main forum fails.

RSYNC is blazingly fast. It can distribute whole directory structure over SSH, being waaaaaaaaaaaaaaay much faster than any FTP.

Quite easy to do actually.

Since I know something about Linux administration (and i have done something similiar already), i think can help here.

OK, here is what needs to be done:

1. Create a dedicated read-only MySQL or PostgreSQL (whichever is being used) database slave on the main bitcoin server.
This slave will be used for doing quick full backups every hour or half an hour. That backups will be automatically packaged and put in a certain folder on the server.

2. Create a bash or php script (about 10 lines) that will
a) Backup the database and pack it to .tar.gz
b) Backup directory structure of the webserver and pack it to tar.gz
Every X minutes.

3. Create a low-privileged user on the main Bitcoin server, using which RSYNC/SSH will connect to the server and download backups.

4. Setup another backup webserver - with apache/nginx/mysql/postrgre/whatever.

5. Create a script that will regularly download current backups from the main server using RSYNC over SSH.

6. Create a script that will ping/wget the main server to check if it works. If not, then the script will unpack last posessed backup, and then mstart apache/nginx/mysql

7. Setup SSH public key authentication so that RSYNC can connect.

8. ?? ??

9. PROFIT !

I think we could implement Shadow's idea.

Sirius?

Will be sufficient to WAL-log of database and send log files to gmail account at fixed intervals

It is a reliable way, even in the case of hacking the server, this backup can not be deleted.

1. Gmailing backups is too complicated and slow.

2. RSYNCed Backups on 10 other servers also cannot be deleted. RSYNC is an extremely powerful, stable, fast & totally command-line-scriptable app. You can use it to create a system of webserver backups.
In fact it is widely used for Linux mirrors (Gentoo Linux uses this in its package management system).

I can help writing some rsync scripts as i am experienced in it, if you're interested.

5 Mb attached WAL-log file per day can be sended very fast


why?

I'll look more into this next weekend. Rsync sounds good.

1. Simple. Because the main server will not have access to the mirror servers. Only the mirror servers need access to root server, not the other way around.
Neither the main server, or possible hackers don't need to know anything about what mirror servers are there, or how many of them is out there.

2. Once you do one backup, you can create a chain of backup servers, where server A backups ROOT server, server B backups A server, and server C backups B server, making it nearly impossible to hack or even trace where all the backups are. All of this can be done with few simple RSYNC scripts.

If you are thinking about government trying to hack or interfere with main site & mirror servers to bring them down, then I'm telling you: not a chance. Just look at the wikileaks case. There are thousands of wikileaks mirrors now. And it is quite easy to create new ones. Very powerful people had interest in killing wikileaks, yet they failed.
Internet is already far more powerful & resilient than everybody expected.

Even Russia government with all their dirty methods would not be able to make all wikileaks mirrors go offline if they wanted to. The only way to do that is shut down the internet completely.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

WikiLeaks not only involved in the dissemination of information but also promote themselves for profit. Otherwise they could just put all files in eMule and gradually publish passwords to them without any kind of 'mirrors'.

It is also very strange to hear about what methods of intelligence agencies of our countries are very different. :)

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

Sorry to interrupt the discussion, only to let you guys know that I have just created another forum:

http://bitcoin.ie/forum/ (http://bitcoin.ie/forum/)

It is going to be mostly focused on the Irish Bitcoin community, however it's all in English so always another place to talk about Bitcoins in case the main forum fails again. I am still in the process of setting it up, so there are no rules posted yet etc, but it should be more or less fully functional now.

Thanks.

So ?

Hence, the original server pushes backups to other servers. And backups will be erased when the original server has been hacked.

That is not a problem when using RSYNC.
By default, RSYNC does not delete files that are deleted from remote server.

This implies that the attacker have permissions to delete. Hence it is - a threat.

The only thing that implies is that you do not understand how rsync works.
RSYNC uses privileges of given SSH user when accessing remote server. So if you create an unprivileged user, only give SSH public keys only to trusted people, and make all backups read - only, really nothing can happen.

Also, you can setup a separate server with SSH daemon, which will be used for storing backups... So potential attacker will never have access to the main server.

If you are really really really paranoid, You can even set up a chain of servers ROOT -> A -> B -> C, where ROOT SSH-connects to A and stores backups there, then B connects to A to download the backups, and C connects to B to download backup copies. And you only give access to C to the outside world.
But i seriously doubt such a security measure will be ever needed.

nightmare

An attacker could mess up the original file that is pulled by the backup servers, but I'm not too worried about that happening. Old backups can be stored for at least a week or two.

The server now makes daily backups automatically, but doesn't rsync them yet. Bzipped size about 50 MB.

How long will this process take?

Don't worry, not long :)

Yes, but it is also easy to do automatic MD5/SHA1/SHA256 signing of every file. So this is a non-issue really.
Just add 5 more lines to backup script.


Awesome. I can create a backup site myself easily.

--------
Also, here is the examplary RSYNC command to backup a folder of one server to another:

If you want to check what will the command do, without actually executing it, add "-n" parameter, which will do a so-called DRY RUN:

If the remote sever is on a different port than standard 22, then use this:

This is also a dry run, no files will be transferred. 40022 is naturally the port number and blowfish is type of encryption used for connection (it's the fastest one for transferring files).

The "-rltgoDvzx" parameters will make rsync transfer entire directory recursively with all possible subdirectories, without files/dirs from other mounted volumes in the path.
Try # man rsync to see what all the options mean.

It's now making daily backups to remote servers. Thanks for help ShadowOfHarbringer.

Thanks for resolving this issue!

That mean that the senior develooper can redeploy to a different server with the backups, if neccesary. Right?

How many servers are serving as backup/mirror?

No problem.
Are you planning to make some public site from which other people (such as myself) could download backups ?

I can setup a backup server too (and act as a mirror backup), as i rent a few servers in datacenters here and there.

You could probably setup a dedicated RSYNC server with bandwidth per user limit. There is even no need to give people SSH access actually.

The database backups contain private stuff like password hashes (=passwords), PM's and email addresses, so we can't give them out. Only bitcoin.org admins (Satoshi, Gavin, me) have the access.


Yes. We'll make sure there are always backups available.

Ah sure, forgot about that.
Well, there is always a possibility of cutting out all sensitive data from the database to make a public read-only forum backup (nobody will be able to login to it since it would contain no hashes, so just read-only).

But i guess that would be probably too much work.

Sirius, would you care to create a no-sensitive-data version of backups so everyone can backup it like wikileaks ?

So, how mirrors are backing up the forum db?

Is there a way to do a db dump of non-sensitives information?

There is a way, the question is if sirius has time or will to do it.

Er, how many mirrors are backing up the database?

The forum database is now being automatically compress, encrypted, and backed up to another of sirius' servers and to Amazon S3 every day (one per day, I don't delete the old backups, yet).

Perhaps we should build a Forum-Backup capability into the standard bitcoin block chain..... so that a current backup of the entire forum database is embedded into the block chain.   ;)

And have my password hash included? No way!
EDIT: You can use google cache if it ever goes down btw. :)

Could be done, but not very high priority. We don't need Wikileaks-grade security measures yet.

Hey, nobody is in a hurry here.
I was just asking if You can do it at all.