Poll on solving the imgur issue

[Synchronice]

AWS S3 is really the most cost-effective option.

I did a quick price check:

First 50 TB / Month: $0.023 per GB

AWS is very reliable, but quite expensive on bandwidth. 10 TB doesn't sound unrealistic, and would cost $230 per month. Contabo (I think TryNinja still uses this) gets you 32 TB bandwidth for €17.49 per month. And unlike AWS, that includes the rest of the server too. But it probably won't reach the same very high uptime AWS has.

You are right, I completely ignored that they make money from bandwidth. Btw, how big can be the demand on bandwidth in our case? I mean, this image hosting is dedicated for this forum, one will rarely use it outside of bitalk, so, not every uploaded image will waste bandwidth.
Btw what about Hetzner? It's in Europe (Germany or Finland) and is pretty cheap. I can't swear but I'm sure it's one of the cheapest option out there, it's not as cheap as Contabo but performs way better. Probably what makes AWS more attractive for joker_josue is probably the pay-as-you approach payment model.

No, it is not very economical, because bandwidth is very expensive.
In this type of service, the focus is not so much on disk space, but on bandwidth.

If it's not a secret, I'll ask this question again: How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?

[1713988869]

1713988869

[1713988869]

1713988869

[PrimeNumber7]

AWS S3 is really the most cost-effective option.

I did a quick price check:

First 50 TB / Month: $0.023 per GB

AWS is very reliable, but quite expensive on bandwidth. 10 TB doesn't sound unrealistic, and would cost $230 per month. Contabo (I think TryNinja still uses this) gets you 32 TB bandwidth for €17.49 per month. And unlike AWS, that includes the rest of the server too. But it probably won't reach the same very high uptime AWS has.

You are right, I completely ignored that they make money from bandwidth. Btw, how big can be the demand on bandwidth in our case? I mean, this image hosting is dedicated for this forum, one will rarely use it outside of bitalk, so, not every uploaded image will waste bandwidth.
Btw what about Hetzner? It's in Europe (Germany or Finland) and is pretty cheap. I can't swear but I'm sure it's one of the cheapest option out there, it's not as cheap as Contabo but performs way better. Probably what makes AWS more attractive for joker_josue is probably the pay-as-you approach payment model.

No, it is not very economical, because bandwidth is very expensive.
In this type of service, the focus is not so much on disk space, but on bandwidth.

If it's not a secret, I'll ask this question again: How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?

If the image proxy keeps served images cached for a period of time, bandwidth probably won’t be an issue. But if the image proxy queries the storage bucket every time someone accesses the image, egress charges will quickly add up. For example, an attacker/troll could potentially add the same image to a single post 1000 times, and subsequently access the post 1000 times — this would result in the same image being accessed a million times.

[joker_josue]

If it's not a secret, I'll ask this question again: How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?

It has such a constant freshness, about 300 images a day.
And when someone uses the script to migrate the images, the growth increases that day.
As I mentioned, if the current pace is maintained, it will be very pleasant.

[LoyceV]

If the image proxy keeps served images cached for a period of time, bandwidth probably won’t be an issue. But if the image proxy queries the storage bucket every time someone accesses the image, egress charges will quickly add up.

Caching is up to the browser, so it's a per-user thing:

The image proxy isn't a caching proxy. Proxied images are never saved to disk, and only small chunks of images are stored in memory at any one time. The proxy does work with client-side caching (it passes on appropriate cache-related headers, etc.), so you may cache images.

[TryNinja]

Contabo (I think TryNinja still uses this) gets you 32 TB bandwidth for €17.49 per month. And unlike AWS, that includes the rest of the server too. But it probably won't reach the same very high uptime AWS has.

I do, and they stopped scheduling random maintance to my server (always accompanied with some downtime) so I have no complains.

[ibminer]

Caching is up to the browser, so it's a per-user thing:

The image proxy isn't a caching proxy. Proxied images are never saved to disk, and only small chunks of images are stored in memory at any one time. The proxy does work with client-side caching (it passes on appropriate cache-related headers, etc.), so you may cache images.

Also in that post:

Increasing fault-tolerance is a long-term goal, but (re)creating a truly decentralized and uncensorable forum is outside of bitcointalk.org's scope.

Now I don't really believe theymos was specifically talking about the image proxy or images here, but..
Should the preservation of posted images be part of the fault-tolerance goals of the forum?  Does the removal of an image present a fault in the context of a thread/post?

As an example, let's look at the reference thread of a flag theymos created.

Without the images, as it stands now, does the flag still hold water?? or.. at least as much as it does with images?  (I'm sure there are better examples but I'm just being dramatic )


 

[Mate2237]

The number two is the best option for me because linking the embedded image with link is the best option for me. And another thing again, till now my images uploaded before the imgur drama is still invalid. Because for me to go back and upload those pictures again is not easy and another thing is that I have lost all those images. So there is no way again for me to redo all over. I tried the TryNinja image recovery method but it could not work for me. So I just gave it for some times.

[PrimeNumber7]

If the image proxy keeps served images cached for a period of time, bandwidth probably won’t be an issue. But if the image proxy queries the storage bucket every time someone accesses the image, egress charges will quickly add up.

Caching is up to the browser, so it's a per-user thing:

The image proxy isn't a caching proxy. Proxied images are never saved to disk, and only small chunks of images are stored in memory at any one time. The proxy does work with client-side caching (it passes on appropriate cache-related headers, etc.), so you may cache images.

I suspect that is the underlying root cause of the imgur problems. If there is an image in a particular thread, the image proxy server may be asking for imgur to serve the same image multiple times over a short time if there is a conversation among several people. Replying to a post with an image may result in the same person causing the image proxy server to call the image three times, and this would not count the times the person reads a thread, but does not reply.

Imgur (in the past) uses AWS, so they are paying the quoted prices for serving images to the forum image proxy. Serving only the image means imgur doesn't get any ad revenue from the forum.

Someone is going to have to pay to get images from wherever they are stored to forum users. It doesn't look like imgur is wanting to pay for this anymore.

[PowerGlove]

I know! Nor is it because of space or bandwidth. It's because of the number of files to be loaded.

But I have no problems with space or bandwidth. The limit is on the number of files I can have on the server.

As I said, I have neither space nor bandwidth concerns at this time.

If bandwidth and space are not a problem for you, and you're only trying to work around an artificial limit on the number of files that your hosting allows, then there are some ways around that...

You could store the images (just the Imgur ones) as BLOBs in a database table, or pack them into a tar-like file format. Both of those approaches will require serving those images from behind a script (either to do a database lookup, or to seek to a specific offset and then read from the amalgamated file), but having them on their own kind of URL (distinct from other TalkImg images) actually makes a lot of sense. That way, my earlier idea of modifying the image proxy to use a "substitutions table" simplifies to just mechanically translating the links on the fly (e.g. https://i.imgur.com/7wqVXzD.png -> https://talkimg.com/imgur.php?id=7wqVXzD, or something like that).

Actually, thinking about this more carefully, why even do this through TalkImg at all? Maybe it makes more sense to set up some dedicated hosting to serve those images without entangling them with TalkImg? I'm not sure who wants that challenge, but working on the problem in isolation will probably lower (and mostly fix) the costs, which might make it easier to find financial support.

[LoyceV]

Actually, thinking about this more carefully, why even do this through TalkImg at all? Maybe it makes more sense to set up some dedicated hosting to serve those images without entangling them with TalkImg? I'm not sure who wants that challenge, but working on the problem in isolation will probably lower (and mostly fix) the costs, which might make it easier to find financial support.

A subdomain of talkimg could be on another webhost (that keeps the DNS responsibility in one place), say the ".m" that imgur uses most of the time. A €5.99/month or double this $58.88/year budget webhost will (probably) be enough, and since the amount of storage if fixed I expect hosting costs to go further down in the future.
The only reason I'm not doing this myself is because I don't want to deal with copyright takedown requests.

[joker_josue]

A subdomain of talkimg could be on another webhost (that keeps the DNS responsibility in one place), say the ".m" that imgur uses most of the time. A €5.99/month or double this $58.88/year budget webhost will (probably) be enough, and since the amount of storage if fixed I expect hosting costs to go further down in the future.
The only reason I'm not doing this myself is because I don't want to deal with copyright takedown requests.

And that's exactly what I'm thinking of doing. I just need to reorganize and check which is the best option for everything to work correctly. 

Tell me something (you thought you already said it, but remember), how much does that 800k occupy?

[LoyceV]

Tell me something (you thought you already said it, but remember), how much does that 800k occupy?

It's 143 GB. It could have some cleaning, like the 30k images that only say "removed", but I hardlinked all duplicates to the same file already, so it doesn't take much diskspace.

[Timelord2067]

More than a week has gone past without a single vote or comment having been made, so it's time to call it for option two

2: Turn imgur images into links    - 51 (54.8%)

Time to lock the thread as there are other threads which canvas the topics being discussed here.

[OgNasty]

More than a week has gone past without a single vote or comment having been made, so it's time to call it for option two

2: Turn imgur images into links    - 51 (54.8%)

Time to lock the thread as there are other threads which canvas the topics being discussed here.

I think that is probably the best solution even if it isn't the one I voted for.  Mostly, I don't want to download any software and I don't think it makes sense for end users to do anything. The amount of man hours that would be wasted by so many people each having to individually solve this issue would be a shocking waste of time.  Sure, having the images just transferred to talkimg would be the best solution for end users, but it would require trusting another service not to do the same thing as imgur.  Replacing with links seems to be the least time consuming solution that would also not add additional points of failure.  I'll be glad when it's done.

[Timelord2067]

The benefit I see with talkimg.com is that it was created by an active user of the Forum ( joker_josue ) specifically for the Forum to use and is relying on what I would call meager donations to maintain the website, but is not asking for any other payment in kind for their services to actually run the site.  Already there is an "unofficial" support person ( TryNinja ) who created a script to assist port images URL's in posts.  While that might not be everyone's cup of tea trusting scripts it's help which again has been done without being asked, or paid for.

Kudos to both I say.

[JeromeTash]

I have a question regarding adding Imgur links in new posts. I believe this was disabled, as one will get an error stating that the imgur links are no longer supported in new posts when they try to create a post containing the links.

So how come some members are still able to add imgur links in their posts, like the OP in this post?

[ibminer]

So how come some members are still able to add imgur links in their posts, like the OP in this post?

They are not able to embed an imgur image in a post, embedded images use the image proxy. But you can still post the link for the image itself. Clicking the link itself does not use the image proxy and goes directly to imgur.

[NotATether]

More than a week has gone past without a single vote or comment having been made, so it's time to call it for option two

2: Turn imgur images into links    - 51 (54.8%)

Time to lock the thread as there are other threads which canvas the topics being discussed here.

I think that is probably the best solution even if it isn't the one I voted for.  Mostly, I don't want to download any software and I don't think it makes sense for end users to do anything. The amount of man hours that would be wasted by so many people each having to individually solve this issue would be a shocking waste of time.  Sure, having the images just transferred to talkimg would be the best solution for end users, but it would require trusting another service not to do the same thing as imgur.  Replacing with links seems to be the least time consuming solution that would also not add additional points of failure.  I'll be glad when it's done.

I think it's the best solution. The only problem I can think of is that Imgur hijacks the direct links on mobile browsers to take you to their homepage (on iOS and Android I believe, though at least in Android you have the possibility of using a browser that bypasses the Imgur pages UI).

[aoluain]

More than a week has gone past without a single vote or comment having been made, so it's time to call it for option two

2: Turn imgur images into links    - 51 (54.8%)

Time to lock the thread as there are other threads which canvas the topics being discussed here.

I think that is probably the best solution even if it isn't the one I voted for.  Mostly, I don't want to download any software and I don't think it makes sense for end users to do anything. The amount of man hours that would be wasted by so many people each having to individually solve this issue would be a shocking waste of time.  Sure, having the images just transferred to talkimg would be the best solution for end users, but it would require trusting another service not to do the same thing as imgur.  Replacing with links seems to be the least time consuming solution that would also not add additional points of failure.  I'll be glad when it's done.

I voted for this option purely because it seems like the best solution and it means there
will be no need anyone to back track through all their links to fix the errors which would
indeed be such a waste of time. Personally if I had used IMGUR in the past I wouldnt even
bother going back to fix the errors and I'm sure there are others who would feel te same.

The benefit I see with talkimg.com is that it was created by an active user of the Forum ( joker_josue ) specifically for the Forum to use and is relying on what I would call meager donations to maintain the website, but is not asking for any other payment in kind for their services to actually run the site.  Already there is an "unofficial" support person ( TryNinja ) who created a script to assist port images URL's in posts.  While that might not be everyone's cup of tea trusting scripts it's help which again has been done without being asked, or paid for.

Kudos to both I say.

Absolutely, this is really the best solution and option for hosting images going forward.
The fact that it is run by active members ² of the forum is a massive benefit.

[IIrik11]

can images hosted at google-drive, one-drive, etc can be embedded into the posts?

if yes, then in my view that would be the best way of going forward, no third party websites required.

who ever wants to post image into their topic can upload them to their cloud-drive and post it in?

 

btw, u made me watch 'my cousin vinny' again, such a good movie..