Free not-quite-a-CDN for non-profit projects

[joepie91]

I decided not to drop this in Project Development as it's not just specific to Bitcoin.

I've been working on a 'development collective' for a while now (http://www.cryto.net/) and by now there is a fairly solid not-quite-a-CDN running. Right now it's essentially a redundant Tahoe-LAFS storage grid with 5 geographically distributed "gateways" - it's essentially a bit like a CDN without the geotargeting. GeoDNS *is* planned for the future, but right now we don't run our own DNS servers yet.

What it does: it delivers files. Any and every kind of files. You can essentially upload a file to the storage grid, and get back a 'read URI' that can be translated to a 'gateway URL' - which is essentially just a URL that anyone can access to download your file. It provides provider-independent security (meaning that you don't have to trust a third party to use it) as the only way to read a file, is using the read URI - which is also a decryption key. While using the standard gateways you have to trust the gateway provider (which right now is pretty much me) to not intercept your data, you can easily set up your own gateway.

There's about 500GB of free space across the grid right now, which should translate into about 200GB of usable space with default settings. Usually any 4 servers can disappear from the grid (which right now consists of 10 servers) and your files should still be retrievable - if you change your settings you can even make the reliability higher.

Now what am I offering? Basically free (unlimited, within reason) usage of that not-quite-a-CDN for non-profit projects. Tahoe-LAFS relies on Python and Twisted, and provides a web API, so it should be easy to implement it into your application. An I2P tunnel for the main gateway is available on http://cryto-gateway.i2p/, while the clearnet gateway runs at http://tahoe-gateway.cryto.net:3719/ (links are interchangeable across gateways, so you can just change the gateway address to make gateway links work on I2P).

An example snippet of PHP code that I am using in the new version of AnonNews to store uploaded images on that grid:

Code: (PHP)

$tahoe_server = "http://localhost:3456";
$tahoe_gateway = "http://tahoe-gateway.cryto.net:3719";
$upload_result = curl_put("{$tahoe_server}/uri", $_FILES['file']['tmp_name']);
if($upload_result !== false)
{
	$upload_b64 = urlsafe_b64encode($upload_result);
	$upload_url = "{$tahoe_gateway}/download/{$upload_b64}/{$_FILES['file']['name']}";
}

... where $upload_url is the public HTTP gateway location of the file - anyone can go to that URL (or for example set it as img src) and see the uploaded file.

If you're working on a non-profit project that could use redundant file storage, hop on to the Cryto IRC and let me know

Of course donating server space or even running a gateway (small 1-file Python script) is also welcomed very much.

EDIT: A quick diagram I made of the current grid setup (minus the 'tahoe-lafs clients' that actually upload data, this is just the delivery part): http://tahoe-gateway.cryto.net:3719/download/VVJJOkNISzp1eHJyc3huemo3ZzYzdmR0N3E3cGY2bzd1aTp2ZG5oa2d3eTV3cmxjcHlvb3B3ZmFzb3duNjRoMmFkNm4zZ3dnZHppenV3ZXJpdnc3bDNhOjM6NjoyODk0NjA=/cryto-tahoe.png

[Vladimir]

.

[joepie91]

Check out zerigo.com using their service you can get geotargeting going in a few hours. Can stop using (and paying) them once you build own DNS with geotargeting. Probably using something like PowerDNS, geotargeted DNS can be bullt in a few days tops. Just a thought ...


.

I'm very low on funds... and the main idea is to do everything using existing/donated resources rather than paying for an additional hosted service. The main challenge right now is to set up a redundant and reliable set of DNS servers (geographically distributed of course) and have all of them serve the exact same DNS records - and base their load-balancing on both geographical location and server load of individual servers. Another important thing is that caching would have to be implemented into the gateway, whereas it now just retrieves files from the grid for every request (with a cache expiry time of 1 year, considering the files are immutable) I would like to see it cache the most used (small) files locally to cut down on the request latency, which is around 200-500ms per request now (due to polling all storage servers for shares). For now HE DNS (which I'm currently using) will do well enough for everything to at least be reachable and reasonably fast and responsive