PowerGlove
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 12:03:17 PM |
|
I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link), and then modify the image proxy to make use of that file (maybe by importing it into a SQLite database, or loading it into a hash table at startup, or something).
That way, everything (mostly; banning future Imgur links was a good move) stays the same (image proxy still shielding IP addresses, images stay Tor-friendly, etc.) and the problem gets fixed transparently (and Imgur can then set itself on fire, for all we care). Also lays the technical foundation for the next time something like this happens.
Just my 2 sats.
|
|
|
|
NotATether
Legendary
Offline
Activity: 1722
Merit: 7244
In memory of o_e_l_e_o
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 12:17:43 PM |
|
I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link), and then modify the image proxy to make use of that file (maybe by importing it into a SQLite database, or loading it into a hash table at startup, or something).
That way, everything (mostly; banning future Imgur links was a good move) stays the same (image proxy still shielding IP addresses, images stay Tor-friendly, etc.) and the problem gets fixed transparently (and Imgur can then set itself on fire, for all we care). Also lays the technical foundation for the next time something like this happens.
Just my 2 sats.
A CSV file or SQLite database won't do, it's gotta be a dedicated database server which can scale if Bitcointalk and consequentially the image proxy server is hammered with incoming requests. Also if we are going to rely on TalkImg as a backbone, I still believe that the hosting costs should be subsidized by the forum treasury if required for network capacity to be met.
|
|
|
|
PowerGlove
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 12:40:15 PM |
|
A CSV file or SQLite database won't do, it's gotta be a dedicated database server which can scale if Bitcointalk and consequentially the image proxy server is hammered with incoming requests.
This is not my first rodeo NAT, I've written a decent amount of performance-critical software in my time. I wouldn't have suggested something that I didn't have first-hand experience with, so trust me when I say that slurping up a text file into a hash table at startup or relying on SQLite (either file-backed or :memory: style) will scale just fine for this particular problem.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 12:41:48 PM Merited by PowerGlove (1) |
|
I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link), and then modify the image proxy to make use of that file (maybe by importing it into a SQLite database, or loading it into a hash table at startup, or something).
A CSV file or SQLite database won't do, it's gotta be a dedicated database server which can scale if Bitcointalk and consequentially the image proxy server is hammered with incoming requests. His suggestion, from what I understand is the proxy to redirect a link to another link. And in that case, the proxy just has to know that the link X happens to point to Y. This could be a solution that would avoid messing with the DB of forum posts.
@LoyceV I know you made a backup of all the images linked on the site, which made a total of 800k. But, can you tell me how many images are from imgur?
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17356
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 01:55:07 PM |
|
I think a sensible way to respond to the Imgur issue would be for theymos to ask LoyceV and joker_josue to come up with a CSV file of TalkImg "fallback" links (i.e. old Imgur link -> new TalkImg link) If it were up to me, that would be as easy as replacing "m.imgur.com" by "m.talkimg.com". My filesystem seems to perform just fine with 800k images in one directory. @LoyceV I know you made a backup of all the images linked on the site, which made a total of 800k.
But, can you tell me how many images are from imgur? All of them are from imgur ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) I'm also trying to download all other images, but it gets stuck on (failing) timeouts for many different websites.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 04:02:09 PM |
|
If it were up to me, that would be as easy as replacing "m.imgur.com" by "m.talkimg.com". My filesystem seems to perform just fine with 800k images in one directory.
OK. I thought it was all the pictures on the forum. And in terms of bandwidth, how do you handle that?
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17356
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 04:13:30 PM |
|
And in terms of bandwidth, how do you handle that? I don't: I don't want to have to deal with possible copyright issues ![Sad](https://bitcointalk.org/Smileys/default/sad.gif) Maybe theymos can share how much bandwidth the image proxy consumes in total.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 04:19:22 PM |
|
And in terms of bandwidth, how do you handle that? I don't: I don't want to have to deal with possible copyright issues ![Sad](https://bitcointalk.org/Smileys/default/sad.gif) Maybe theymos can share how much bandwidth the image proxy consumes in total. You can look at my data to get an idea. ![Roll Eyes](https://bitcointalk.org/Smileys/default/rolleyes.gif) I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective. ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif)
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17356
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 04:22:18 PM |
|
You can look at my data to get an idea. ![Roll Eyes](https://bitcointalk.org/Smileys/default/rolleyes.gif) That's only recent images, I'm curious how often old images are downloaded too. I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective. ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif) Unfortunately, being selective doesn't work to replace all links.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 04:30:22 PM |
|
I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective. ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif) Unfortunately, being selective doesn't work to replace all links. I know! Nor is it because of space or bandwidth. It's because of the number of files to be loaded. I've been thinking about how I can help with that. ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif)
|
|
|
|
PrimeNumber7
Copper Member
Legendary
Offline
Activity: 1652
Merit: 1900
Amazon Prime Member #7
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 04:59:54 PM |
|
It looks like imgur has an api, but it doesn't look like the API can retrieve images currently. Perhaps they may be willing to add such functionality. If they add it as a paid feature, I don't think the forum should go down that road. It would set a precendent for other image hosts to do the same. I disagree. Imgur is providing a service (hosting images), and deserves to earn a fair profit. It would cost money for the forum to host images on its own infrastructure, and as long as the cost to use imgur (and all other image hosting sites) is less than the cost of self-hosting, it will make sense to pay. I presume that imgur usually makes its money from ads when people access images on its imgur site (not the raw image that is accessed via the forum image proxy). I presume that the image proxy complicates various metrics that imgur uses.
|
|
|
|
LoyceMobile
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 05:18:21 PM |
|
I don't mind receiving the images, but at the moment I can't receive 800k all at once. We had to be selective. ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif) Unfortunately, being selective doesn't work to replace all links. I know! Nor is it because of space or bandwidth. It's because of the number of files to be loaded. I've been thinking about how I can help with that. ![Undecided](https://bitcointalk.org/Smileys/default/undecided.gif) If you have the space, why would the number of pictures be a problem? Just disable directory view in your webserver and it should just be able to handle it.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 05:36:25 PM |
|
If you have the space, why would the number of pictures be a problem? Just disable directory view in your webserver and it should just be able to handle it.
But I have no problems with space or bandwidth. The limit is on the number of files I can have on the server. Right now I can have around 600k files, a very comfortable number for the next months/years. You know that hosting services talk about unlimited, but there is always some kind of limit. I preferred to have unlimited disk space and bandwidth. So entering 800k files now would require a new server upgrade. And I didn't want to do that now. But, I'm checking several options that can make everything viable, without having to do a new upgrade. Therefore, what I ask now is patience to be able to support this type of action. I am studying what I can do to accommodate these images.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17356
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 05:54:10 PM |
|
Right now I can have around 600k files, a very comfortable number for the next months/years. I see. That's tiny ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) For comparison: a "normal" filesystem on a 1 TB disk has 58 million inodes (which means it could handle that many small files). You know that hosting services talk about unlimited, but there is always some kind of limit. I preferred to have unlimited disk space and bandwidth. I learned a long time ago to stay away from "unlimited". I prefer to know what I get. So entering 800k files now would require a new server upgrade. And I didn't want to do that now. But, I'm checking several options that can make everything viable, without having to do a new upgrade. Therefore, what I ask now is patience to be able to support this type of action. I am studying what I can do to accommodate these images. I used to use shared hosting, until my last one disappeared. Since then, I've become much more comfortable with VPS servers, which in general give better specs for money, but don't come with a pre-installed selection of scripts.
|
|
|
|
ibminer
Legendary
Offline
Activity: 1856
Merit: 2896
Goonies never say die.
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 06:14:38 PM |
|
It sounds like we're piping a bunch of traffic through 1 IP, which might create problems from any site or image host out there, but maybe depending on the level of traffic we get up to on the forum? Without exceptions, I'd imagine firewalls or other security devices out there potentially interpreting this activity as a weak DoS attempt from 1 IP, or just odd/malicious traffic of some sort which is taking enough resources to get blocked. Maybe even just a curious IT guy browsing logs and seeing a shit load of unusual traffic from 1 IP and blocking it? ![Tongue](https://bitcointalk.org/Smileys/default/tongue.gif) I've done this plenty in my world, but I obviously don't run any image hosts, so I'm not sure how they (or their providers) would interpret or handle this any differently.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 08, 2023, 06:29:28 PM |
|
I see. That's tiny ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) It's not that tiny, for the service in question. Especially at this early stage. You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images. This is a large number, but it was gathered over 13 years, which averages out to 150k images per year. Let's round it up to 200k images per year. That means I can maintain these conditions for the next 2-3 years. It will be time when I'm planning to do a new upgrade. But, as I said, I'm evaluating several possibilities, we'll see what I get. ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) I used to use shared hosting, until my last one disappeared. Since then, I've become much more comfortable with VPS servers, which in general give better specs for money, but don't come with a pre-installed selection of scripts. I'm using one of the biggest hosting services on the market. Without any intermediary company, that is I can choose: shared server; vps; or dedicated server. Therefore, this possibility is more remote. But even a VPS has file limits. The service you use goes up to 58,000,000!? ![Huh](https://bitcointalk.org/Smileys/default/huh.gif)
|
|
|
|
dkbit98
Legendary
Offline
Activity: 2352
Merit: 7430
|
It's not that tiny, for the service in question. Especially at this early stage. You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images.
It's totally unrealistic to think TalkImg can all of the sudden replace all Imgur images after just few weeks of existence ![Cheesy](https://bitcointalk.org/Smileys/default/cheesy.gif) In theory everything sounds great, just replace the links and everything will work perfectly, but nobody is financially helping this project and thinking about server costs. Maybe it's better to replace all old Imgur images with Imgbb or Postimages for now, and later we an think about alternative slow moving to TalkImg.
|
|
|
|
Synchronice
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 09, 2023, 07:13:11 AM |
|
You know that hosting services talk about unlimited, but there is always some kind of limit. I preferred to have unlimited disk space and bandwidth.
I remember the time when there was a boom of unlimited hosting ads from companies like Hostgator, iPage, Bluehost and I wanted to make a file sharing website, similar to wetransfer and I was always receiving warning from these hosting companies that I shouldn't upload or store high size (500MB and higher) files on my website and when I was asking them then why it's called unlimited, they were telling me that it was marked as unlimited because most businesses upload some kilobytes and they don't limit them, etc. They were just saying it's unlimited while it was actually almost as limited as free web hosting. That's why I always prefer dedicated server but in your case, AWS S3 is really the most cost-effective option.
Btw as far as I know, digitalocean doesn't have inodes limitation. I see. That's tiny ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) It's not that tiny, for the service in question. Especially at this early stage. You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images. This is a large number, but it was gathered over 13 years, which averages out to 150k images per year. Let's round it up to 200k images per year. That means I can maintain these conditions for the next 2-3 years. It will be time when I'm planning to do a new upgrade. But, as I said, I'm evaluating several possibilities, we'll see what I get. ![Wink](https://bitcointalk.org/Smileys/default/wink.gif) How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?
|
| CHIPS.GG | | | ▄▄███████▄▄ ▄████▀▀▀▀▀▀▀████▄ ▄███▀░▄░▀▀▀▀▀░▄░▀███▄ ▄███░▄▀░░░░░░░░░▀▄░███▄ ▄███░▄░░░▄█████▄░░░▄░███▄ ███░▄▀░░░███████░░░▀▄░███ ███░█░░░▀▀▀▀▀░░░▀░░░█░███ ███░▀▄░▄▀░▄██▄▄░▀▄░▄▀░███ ▀███░▀░▀▄██▀░▀██▄▀░▀░███▀ ▀███░▀▄░░░░░░░░░▄▀░███▀ ▀███▄░▀░▄▄▄▄▄░▀░▄███▀ ▀████▄▄▄▄▄▄▄████▀ █████████████████████████ | | ▄▄███████▄▄ ▄███████████████▄ ▄█▀▀▀▄█████████▄▀▀▀█▄ ▄██████▀▄█▄▄▄█▄▀██████▄ ▄████████▄█████▄████████▄ ████████▄███████▄████████ ███████▄█████████▄███████ ███▄▄▀▀█▀▀█████▀▀█▀▀▄▄███ ▀█████████▀▀██▀█████████▀ ▀█████████████████████▀ ▀███████████████████▀ ▀████▄▄███▄▄████▀ ████████████████████████ | | 3000+ UNIQUE GAMES | | | 12+ CURRENCIES ACCEPTED | | | VIP REWARD PROGRAM | | ◥ | Play Now |
|
|
|
LoyceV
Legendary
Offline
Activity: 3430
Merit: 17356
Thick-Skinned Gang Leader and Golden Feather 2021
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 09, 2023, 09:00:25 AM |
|
You have gathered 800k from a hosting, let's imagine that there are more than 800k together with the other services. Which indicates 1600k images listed from the forum. Let's round it up to 2000k images. I count about 2.5 million image links in my data collection. Some of those are dead already, so your estimate is close enough. But even a VPS has file limits. The service you use goes up to 58,000,000!? ![Huh](https://bitcointalk.org/Smileys/default/huh.gif) The "58 million" is the number of inodes on a 1 TB disk. Most files aren't that small, so the number of images you can fit in 1 TB will be significantly lower. So in most cases, the number of files shouldn't be a problem before you run out of diskspace. AWS S3 is really the most cost-effective option. I did a quick price check: First 50 TB / Month: $0.023 per GB AWS is very reliable, but quite expensive on bandwidth. 10 TB doesn't sound unrealistic, and would cost $230 per month. Contabo (I think TryNinja still uses this) gets you 32 TB bandwidth for €17.49 per month. And unlike AWS, that includes the rest of the server too. But it probably won't reach the same very high uptime AWS has.
|
|
|
|
joker_josue
Legendary
Offline
Activity: 1778
Merit: 4838
**In BTC since 2013**
|
![](https://bitcointalk.org/Themes/custom1/images/post/xx.gif) |
June 09, 2023, 09:35:18 AM |
|
hat's why I always prefer dedicated server but in your case, AWS S3 is really the most cost-effective option. ~~ How is the demand on your service? Is it increasing? Or stabilizing? Or decreasing since imgur links aren't broken anymore?
No, it is not very economical, because bandwidth is very expensive. In this type of service, the focus is not so much on disk space, but on bandwidth. The "58 million" is the number of inodes on a 1 TB disk. Most files aren't that small, so the number of images you can fit in 1 TB will be significantly lower. So in most cases, the number of files shouldn't be a problem before you run out of diskspace.
I think I didn't explain it well. I can have 500 files with 1TB on the server, no problem. As I said, I have neither space nor bandwidth concerns at this time. Either way, thanks for your suggestion, which I'll explore.
|
|
|
|
|