Poll on solving the imgur issue

[PowerGlove]

I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link), and then modify the image proxy to make use of that file (maybe by importing it into a SQLite database, or loading it into a hash table at startup, or something).

That way, everything (mostly; banning future Imgur links was a good move) stays the same (image proxy still shielding IP addresses, images stay Tor-friendly, etc.) and the problem gets fixed transparently (and Imgur can then set itself on fire, for all we care). Also lays the technical foundation for the next time something like this happens.

Just my 2 sats.

[NotATether]

I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link), and then modify the image proxy to make use of that file (maybe by importing it into a SQLite database, or loading it into a hash table at startup, or something).

That way, everything (mostly; banning future Imgur links was a good move) stays the same (image proxy still shielding IP addresses, images stay Tor-friendly, etc.) and the problem gets fixed transparently (and Imgur can then set itself on fire, for all we care). Also lays the technical foundation for the next time something like this happens.

Just my 2 sats.

A CSV file or SQLite database won't do, it's gotta be a dedicated database server which can scale if Bitcointalk and consequentially the image proxy server is hammered with incoming requests.

Also if we are going to rely on TalkImg as a backbone, I still believe that the hosting costs should be subsidized by the forum treasury if required for network capacity to be met.

[PowerGlove]

A CSV file or SQLite database won't do, it's gotta be a dedicated database server which can scale if Bitcointalk and consequentially the image proxy server is hammered with incoming requests.

This is not my first rodeo NAT, I've written a decent amount of performance-critical software in my time. I wouldn't have suggested something that I didn't have first-hand experience with, so trust me when I say that slurping up a text file into a hash table at startup or relying on SQLite (either file-backed or :memory: style) will scale just fine for this particular problem.

[joker_josue]

I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link), and then modify the image proxy to make use of that file (maybe by importing it into a SQLite database, or loading it into a hash table at startup, or something).

A CSV file or SQLite database won't do, it's gotta be a dedicated database server which can scale if Bitcointalk and consequentially the image proxy server is hammered with incoming requests.

His suggestion, from what I understand is the proxy to redirect a link to another link.
And in that case, the proxy just has to know that the link X happens to point to Y. This could be a solution that would avoid messing with the DB of forum posts.


@LoyceV
I know you made a backup of all the images linked on the site, which made a total of 800k.

But, can you tell me how many images are from imgur?

[LoyceV]

I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link)

If it were up to me, that would be as easy as replacing "m.imgur.com" by "m.talkimg.com". My filesystem seems to perform just fine with 800k images in one directory.

@LoyceV
I know you made a backup of all the images linked on the site, which made a total of 800k.

But, can you tell me how many images are from imgur?

All of them are from imgur I'm also trying to download all other images, but it gets stuck on (failing) timeouts for many different websites.

[joker_josue]

If it were up to me, that would be as easy as replacing "m.imgur.com" by "m.talkimg.com". My filesystem seems to perform just fine with 800k images in one directory.

OK. I thought it was all the pictures on the forum.
And in terms of bandwidth, how do you handle that?

[LoyceV]

And in terms of bandwidth, how do you handle that?

I don't: I don't want to have to deal with possible copyright issues
Maybe theymos can share how much bandwidth the image proxy consumes in total.

[joker_josue]

And in terms of bandwidth, how do you handle that?

I don't: I don't want to have to deal with possible copyright issues
Maybe theymos can share how much bandwidth the image proxy consumes in total.

You can look at my data to get an idea. 

I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective. 

[LoyceV]

You can look at my data to get an idea.  

That's only recent images, I'm curious how often old images are downloaded too.

I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective.  

Unfortunately, being selective doesn't work to replace all links.

[joker_josue]

I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective.  

Unfortunately, being selective doesn't work to replace all links.

I know! Nor is it because of space or bandwidth. It's because of the number of files to be loaded.

I've been thinking about how I can help with that.

[PrimeNumber7]

It looks like imgur has an api, but it doesn't look like the API can retrieve images currently. Perhaps they may be willing to add such functionality.

If they add it as a paid feature, I don't think the forum should go down that road. It would set a precendent for other image hosts to do the same.

I disagree. Imgur is providing a service (hosting images), and deserves to earn a fair profit. It would cost money for the forum to host images on its own infrastructure, and as long as the cost to use imgur (and all other image hosting sites) is less than the cost of self-hosting, it will make sense to pay.

I presume that imgur usually makes its money from ads when people access images on its imgur site (not the raw image that is accessed via the forum image proxy). I presume that the image proxy complicates various metrics that imgur uses.

[LoyceMobile]

I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective.  

Unfortunately, being selective doesn't work to replace all links.

I know! Nor is it because of space or bandwidth. It's because of the number of files to be loaded.

I've been thinking about how I can help with that.

If you have the space, why would the number of pictures be a problem? Just disable directory view in your webserver and it should just be able to handle it.

[joker_josue]

If you have the space, why would the number of pictures be a problem? Just disable directory view in your webserver and it should just be able to handle it.

But I have no problems with space or bandwidth. The limit is on the number of files I can have on the server. Right now I can have around 600k files, a very comfortable number for the next months/years. You know that hosting services talk about unlimited, but there is always some kind of limit. I preferred to have unlimited disk space and bandwidth.

So entering 800k files now would require a new server upgrade. And I didn't want to do that now. But, I'm checking several options that can make everything viable, without having to do a new upgrade. Therefore, what I ask now is patience to be able to support this type of action. I am studying what I can do to accommodate these images.

[LoyceV]

Right now I can have around 600k files, a very comfortable number for the next months/years.

I see. That's tiny
For comparison: a "normal" filesystem on a 1 TB disk has 58 million inodes (which means it could handle that many small files).

You know that hosting services talk about unlimited, but there is always some kind of limit. I preferred to have unlimited disk space and bandwidth.

I learned a long time ago to stay away from "unlimited". I prefer to know what I get.

So entering 800k files now would require a new server upgrade. And I didn't want to do that now. But, I'm checking several options that can make everything viable, without having to do a new upgrade. Therefore, what I ask now is patience to be able to support this type of action. I am studying what I can do to accommodate these images.

I used to use shared hosting, until my last one disappeared. Since then, I've become much more comfortable with VPS servers, which in general give better specs for money, but don't come with a pre-installed selection of scripts.

[ibminer]

It sounds like we're piping a bunch of traffic through 1 IP, which might create problems from any site or image host out there, but maybe depending on the level of traffic we get up to on the forum?

Without exceptions, I'd imagine firewalls or other security devices out there potentially interpreting this activity as a weak DoS attempt from 1 IP, or just odd/malicious traffic of some sort which is taking enough resources to get blocked.

Maybe even just a curious IT guy browsing logs and seeing a shit load of unusual traffic from 1 IP and blocking it?   I've done this plenty in my world, but I obviously don't run any image hosts, so I'm not sure how they (or their providers) would interpret or handle this any differently.

[joker_josue]

I see. That's tiny

It's not that tiny, for the service in question. Especially at this early stage.
You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images.
This is a large number, but it was gathered over 13 years, which averages out to 150k images per year. Let's round it up to 200k images per year.
That means I can maintain these conditions for the next 2-3 years. It will be time when I'm planning to do a new upgrade.

But, as I said, I'm evaluating several possibilities, we'll see what I get. 

I used to use shared hosting, until my last one disappeared. Since then, I've become much more comfortable with VPS servers, which in general give better specs for money, but don't come with a pre-installed selection of scripts.

I'm using one of the biggest hosting services on the market. Without any intermediary company, that is I can choose: shared server; vps; or dedicated server.
Therefore, this possibility is more remote.

But even a VPS has file limits. The service you use goes up to 58,000,000!? 

[dkbit98]

It's not that tiny, for the service in question. Especially at this early stage.
You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images.

It's totally unrealistic to think TalkImg can all of the sudden replace all Imgur images after just few weeks of existence  
In theory everything sounds great, just replace the links and everything will work perfectly, but nobody is financially helping this project and thinking about server costs.
Maybe it's better to replace all old Imgur images with Imgbb or Postimages for now, and later we an think about alternative slow moving to TalkImg.

[Synchronice]

You know that hosting services talk about unlimited, but there is always some kind of limit. I preferred to have unlimited disk space and bandwidth.

I remember the time when there was a boom of unlimited hosting ads from companies like Hostgator, iPage, Bluehost and I wanted to make a file sharing website, similar to wetransfer and I was always receiving warning from these hosting companies that I shouldn't upload or store high size (500MB and higher) files on my website and when I was asking them then why it's called unlimited, they were telling me that it was marked as unlimited because most businesses upload some kilobytes and they don't limit them, etc. They were just saying it's unlimited while it was actually almost as limited as free web hosting. That's why I always prefer dedicated server but in your case, AWS S3 is really the most cost-effective option.
Btw as far as I know, digitalocean doesn't have inodes limitation.

I see. That's tiny

It's not that tiny, for the service in question. Especially at this early stage.
You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images.
This is a large number, but it was gathered over 13 years, which averages out to 150k images per year. Let's round it up to 200k images per year.
That means I can maintain these conditions for the next 2-3 years. It will be time when I'm planning to do a new upgrade.

But, as I said, I'm evaluating several possibilities, we'll see what I get. 

How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?

[LoyceV]

You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images.

I count about 2.5 million image links in my data collection. Some of those are dead already, so your estimate is close enough.

But even a VPS has file limits. The service you use goes up to 58,000,000!? 

The "58 million" is the number of inodes on a 1 TB disk. Most files aren't that small, so the number of images you can fit in 1 TB will be significantly lower. So in most cases, the number of files shouldn't be a problem before you run out of diskspace.

AWS S3 is really the most cost-effective option.

I did a quick price check:

First 50 TB / Month: $0.023 per GB

AWS is very reliable, but quite expensive on bandwidth. 10 TB doesn't sound unrealistic, and would cost $230 per month. Contabo (I think TryNinja still uses this) gets you 32 TB bandwidth for €17.49 per month. And unlike AWS, that includes the rest of the server too. But it probably won't reach the same very high uptime AWS has.

[joker_josue]

hat's why I always prefer dedicated server but in your case, AWS S3 is really the most cost-effective option.
~~
How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?

No, it is not very economical, because bandwidth is very expensive.
In this type of service, the focus is not so much on disk space, but on bandwidth.

The "58 million" is the number of inodes on a 1 TB disk. Most files aren't that small, so the number of images you can fit in 1 TB will be significantly lower. So in most cases, the number of files shouldn't be a problem before you run out of diskspace.

I think I didn't explain it well. I can have 500 files with 1TB on the server, no problem. As I said, I have neither space nor bandwidth concerns at this time.

Either way, thanks for your suggestion, which I'll explore.