Yes, the Bitcoin.org Site and Forum were Down

[srb123]

Anybody here ever used Collanos, it might hold the answer.

I havent used it for 2-3years, but it is P2p, distributed etc. (Very similar to Microsoft Groove, now know as Sharepoint Workspace.)

Cheers

EDIT: Had a download and play, and it is probably not appropriate for what we want to do.

[ShadowOfHarbringer]

Automated backups of the database to a remote location would be useful if something unexpected happens with the hosting. Any ideas how to store the backups? Maybe encrypt them and distribute to multiple servers around the world. Distributing them through a single gateway (Dropbox, Tahoe-LAFS?) would save bandwidth.

That would be a good start. Tahoe-LAFS would be nice to use also. (There's also possible synergy between Tahoe and Bitcoin)

I think we can handle trusting the seniors not to abuse the database data.

Any possible timetable?

Why so difficult ? Why make it complicated ?

Make it simple. RSYNC, a simple-yet-advanced Linux program can distribute almost anything (even whole Linux partition) over SSH from server to server using public keys (passwords are not needed).
An automated script ( about 10-30 lines of code) can automatically ping main server, restore last backup and start backup forum once the main forum fails.

RSYNC is blazingly fast. It can distribute whole directory structure over SSH, being waaaaaaaaaaaaaaay much faster than any FTP.

Quite easy to do actually.

[ShadowOfHarbringer]

Since I know something about Linux administration (and i have done something similiar already), i think can help here.

OK, here is what needs to be done:

1. Create a dedicated read-only MySQL or PostgreSQL (whichever is being used) database slave on the main bitcoin server.
This slave will be used for doing quick full backups every hour or half an hour. That backups will be automatically packaged and put in a certain folder on the server.

2. Create a bash or php script (about 10 lines) that will
a) Backup the database and pack it to .tar.gz
b) Backup directory structure of the webserver and pack it to tar.gz
Every X minutes.

3. Create a low-privileged user on the main Bitcoin server, using which RSYNC/SSH will connect to the server and download backups.

4. Setup another backup webserver - with apache/nginx/mysql/postrgre/whatever.

5. Create a script that will regularly download current backups from the main server using RSYNC over SSH.

6. Create a script that will ping/wget the main server to check if it works. If not, then the script will unpack last posessed backup, and then mstart apache/nginx/mysql

7. Setup SSH public key authentication so that RSYNC can connect.

8. ?? ??

9. PROFIT !

[kiba]

I think we could implement Shadow's idea.

Sirius?

[bitcoinex]

The main site could be mirrored, but keeping the forum database in sync on multiple servers sounds difficult. Automated backups of the database to a remote location would be useful if something unexpected happens with the hosting. Any ideas how to store the backups? Maybe encrypt them and distribute to multiple servers around the world. Distributing them through a single gateway (Dropbox, Tahoe-LAFS?) would save bandwidth.

Will be sufficient to WAL-log of database and send log files to gmail account at fixed intervals

It is a reliable way, even in the case of hacking the server, this backup can not be deleted.

[ShadowOfHarbringer]

The main site could be mirrored, but keeping the forum database in sync on multiple servers sounds difficult. Automated backups of the database to a remote location would be useful if something unexpected happens with the hosting. Any ideas how to store the backups? Maybe encrypt them and distribute to multiple servers around the world. Distributing them through a single gateway (Dropbox, Tahoe-LAFS?) would save bandwidth.

Will be sufficient to WAL-log of database and send log files to gmail account at fixed intervals

It is a reliable way, even in the case of hacking the server, this backup can not be deleted.

1. Gmailing backups is too complicated and slow.

2. RSYNCed Backups on 10 other servers also cannot be deleted. RSYNC is an extremely powerful, stable, fast & totally command-line-scriptable app. You can use it to create a system of webserver backups.
In fact it is widely used for Linux mirrors (Gentoo Linux uses this in its package management system).

[ShadowOfHarbringer]

I think we could implement Shadow's idea.

Sirius?

I can help writing some rsync scripts as i am experienced in it, if you're interested.

[bitcoinex]

The main site could be mirrored, but keeping the forum database in sync on multiple servers sounds difficult. Automated backups of the database to a remote location would be useful if something unexpected happens with the hosting. Any ideas how to store the backups? Maybe encrypt them and distribute to multiple servers around the world. Distributing them through a single gateway (Dropbox, Tahoe-LAFS?) would save bandwidth.

Will be sufficient to WAL-log of database and send log files to gmail account at fixed intervals

It is a reliable way, even in the case of hacking the server, this backup can not be deleted.

1. Gmailing backups is too complicated and slow.

5 Mb attached WAL-log file per day can be sended very fast

2. RSYNCed Backups on 10 other servers also cannot be deleted.

why?

[sirius]

I'll look more into this next weekend. Rsync sounds good.

[ShadowOfHarbringer]

2. RSYNCed Backups on 10 other servers also cannot be deleted.

why?

1. Simple. Because the main server will not have access to the mirror servers. Only the mirror servers need access to root server, not the other way around.
Neither the main server, or possible hackers don't need to know anything about what mirror servers are there, or how many of them is out there.

2. Once you do one backup, you can create a chain of backup servers, where server A backups ROOT server, server B backups A server, and server C backups B server, making it nearly impossible to hack or even trace where all the backups are. All of this can be done with few simple RSYNC scripts.

If you are thinking about government trying to hack or interfere with main site & mirror servers to bring them down, then I'm telling you: not a chance. Just look at the wikileaks case. There are thousands of wikileaks mirrors now. And it is quite easy to create new ones. Very powerful people had interest in killing wikileaks, yet they failed.
Internet is already far more powerful & resilient than everybody expected.

Even Russia government with all their dirty methods would not be able to make all wikileaks mirrors go offline if they wanted to. The only way to do that is shut down the internet completely.

[bitcoinex]

2. RSYNCed Backups on 10 other servers also cannot be deleted.

why?

1. Simple. Because the main server will not have access to the mirror servers. Only the mirror servers need access to root server, not the other way around.
Neither the main server, or possible hackers don't need to know anything about what mirror servers are there, or how many of them is out there.

2. Once you do one backup, you can create a chain of backup servers, where server A backups ROOT server, server B backups A server, and server C backups B server, making it nearly impossible to hack or even trace where all the backups are. All of this can be done with few simple RSYNC scripts.

If you are thinking about government trying to hack or interfere with main site & mirror servers to bring them down, then I'm telling you: not a chance. Just look at the wikileaks case. There are thousands of wikileaks mirrors now. And it is quite easy to create new ones. Very powerful people had interest in killing wikileaks, yet they failed.
Internet is already far more powerful & resilient than everybody expected.

Even Russia government with all their dirty methods would not be able to make all wikileaks mirrors go offline if they wanted to. The only way to do that is shut down the internet completely.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

WikiLeaks not only involved in the dissemination of information but also promote themselves for profit. Otherwise they could just put all files in eMule and gradually publish passwords to them without any kind of 'mirrors'.

It is also very strange to hear about what methods of intelligence agencies of our countries are very different.

[ShadowOfHarbringer]

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

[Mahkul]

Sorry to interrupt the discussion, only to let you guys know that I have just created another forum:

http://bitcoin.ie/forum/

It is going to be mostly focused on the Irish Bitcoin community, however it's all in English so always another place to talk about Bitcoins in case the main forum fails again. I am still in the process of setting it up, so there are no rules posted yet etc, but it should be more or less fully functional now.

Thanks.

[bitcoinex]

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

[ShadowOfHarbringer]

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

So ?

[bitcoinex]

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

So ?

Hence, the original server pushes backups to other servers. And backups will be erased when the original server has been hacked.

[ShadowOfHarbringer]

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

So ?

Hence, the original server pushes backups to other servers. And backups will be erased when the original server has been hacked.

That is not a problem when using RSYNC.
By default, RSYNC does not delete files that are deleted from remote server.

[bitcoinex]

And to secure WAL-log (if its required, which is doubtful) it is better to use a pgp encryption

You can use simple automated script to compress the files, encrypt them with PGP or Truecrypt, and then RSYNC them.
Simple enough, no need to involve any emails in this.

The problem is that source of "creation_a_new_WAL_log_file" event is DB daemon, not a remote backup server.

So ?

Hence, the original server pushes backups to other servers. And backups will be erased when the original server has been hacked.

That is not a problem when using RSYNC.
By default, RSYNC does not delete files that are deleted from remote server.

This implies that the attacker have permissions to delete. Hence it is - a threat.

[ShadowOfHarbringer]

This implies that the attacker have permissions to delete. Hence it is - a threat.

The only thing that implies is that you do not understand how rsync works.
RSYNC uses privileges of given SSH user when accessing remote server. So if you create an unprivileged user, only give SSH public keys only to trusted people, and make all backups read - only, really nothing can happen.

Also, you can setup a separate server with SSH daemon, which will be used for storing backups... So potential attacker will never have access to the main server.

If you are really really really paranoid, You can even set up a chain of servers ROOT -> A -> B -> C, where ROOT SSH-connects to A and stores backups there, then B connects to A to download the backups, and C connects to B to download backup copies. And you only give access to C to the outside world.
But i seriously doubt such a security measure will be ever needed.

[bitcoinex]

nightmare