TalkImg.com - Image hosting for BitcoinTalk

[Smartvirus]

Nope, I was just trying to upload an image from the regular version but couldn't. The size was just too big, exceeded the limit, so I was given the message you know what and a redirect link to the zip tool. Clicked that link, and I was at the error page!

I assume, the redirect link to the zip tool had the "www" part included! Yes, by the looks of it you didn't fix that!

Thanks for the heads up.
That one escaped me. It's already resolved. I believe that everywhere. 

Somehow I’ve always had to ignore this and source for other means to ensure the image is acceptable. Scale it to acceptable range and be back with it, that’s for GIF files I mean. It would be great to have the ZIP code working perfectly so, we don’t have to go through the huddles of rescaling and uploading over and over.
I guess it’s all worked out now and we would be having these problems no more.

[joker_josue]

This weekend, a new backup file was released. Did you happen to manage to make a copy or haven't I tried yet?

I'm waiting for a full backup first

But do you want a full backup from day one, or from the last backup day you have?
Would you rather wait until I have the new backup type plan, is that it?

Somehow I’ve always had to ignore this and source for other means to ensure the image is acceptable. Scale it to acceptable range and be back with it, that’s for GIF files I mean. It would be great to have the ZIP code working perfectly so, we don’t have to go through the huddles of rescaling and uploading over and over.
I guess it’s all worked out now and we would be having these problems no more.

Converting GIFs is not possible with this tool yet. Converting GIFs is more complicated and can greatly impact the quality. So I got it by not setting up GIF conversion yet.

[LoyceV]

But do you want a full backup from day one, or from the last backup day you have?

I have nothing left:

I don't think I made any local backups, so my backups disappeared with my server.

Would you rather wait until I have the new backup type plan, is that it?

That's probably easier than changing it again later

[joker_josue]

Would you rather wait until I have the new backup type plan, is that it?

That's probably easier than changing it again later

I wanted to create a more forum-oriented solution. Because, no matter how many instructions and controls you make, it will always be difficult to guarantee that all images from current backups are being used on the forum. I would like to create a model that guarantees good management of resources and is viable for everyone.

I've been thinking of two ways we can do this:
a) Do a weekly scraping of all images used on the forum. Then make a backup of all these images.
b) Use the monthly logs to check for new images used on the forum and create a backup.

In the case of option A, which seems to me to be the most viable, but if a post is edited with a new image, this information may fail.
The same can happen in option B, because the record may not be well done. Also, since I don't keep large logs, the information is temporary and limited in information.

Perhaps the solution could be a combination of the two.

For the forum scraping experts (LoyceV and TryNinja), was it very difficult for you to provide a weekly list of all the new images used on the forum?

[LoyceV]

In the case of option A, which seems to me to be the most viable, but if a post is edited with a new image, this information may fail.
The same can happen in option B, because the record may not be well done.

None of those solutions guarantee catching 100% of all images.

For the forum scraping experts (LoyceV and TryNinja), was it very difficult for you to provide a weekly list of all the new images used on the forum?

I can get you such a list, but the will look like this:

Code:

https://ip.bitcointalk.org/?u=https%3A%2F%2Fwww.talkimg.com%2Fimages%2F2023%2F04%2F30%2Flogo_v3_big15552fcf6c36af30.png&t=673&c=8BvZ3oqOv4uklg

The list will be incomplete because it's based on unedited posts.

[NotATether]

The easiest way to make a backup of only Bitcointalk images is to scrape your web server's log files for image URLs. Then you can filter for bitcointalk.org URLs that have a status code of 200. Make a cron job that runs every few minutes that checks for new log entries, and save the image URLs into a file, one per line. From there, I think you know how to back up the images. Most web servers keep archived, zipped versions of their old log files too, unless you explicitly deleted them.

[joker_josue]

In the case of option A, which seems to me to be the most viable, but if a post is edited with a new image, this information may fail.
The same can happen in option B, because the record may not be well done.

None of those solutions guarantee catching 100% of all images.

For the forum scraping experts (LoyceV and TryNinja), was it very difficult for you to provide a weekly list of all the new images used on the forum?

I can get you such a list, but the will look like this:

Code:

https://ip.bitcointalk.org/?u=https%3A%2F%2Fwww.talkimg.com%2Fimages%2F2023%2F04%2F30%2Flogo_v3_big15552fcf6c36af30.png&t=673&c=8BvZ3oqOv4uklg

The list will be incomplete because it's based on unedited posts.

That's a little bit what I want to test in the coming weeks.

This information for me would already be a good start.

How long does it take to do a scraping?
Imagine that today we were doing a scrape of the entire forum to collect this information, how long does it take?

From the analysis I've been doing, the log I'm managing to record, indicates all the images that at some point were seen via the forum proxy.
Now we will be adding to that all the images that are on the forum to date.

~

Yes, the problem is not in knowing the images seen by the forum proxy, but that information has to be compiled in order to back it up the way I want it

AnIn the case of logs, no, my server doesn't save it. I made a point of not keeping it.
I believe in the possibility that the host company will keep some backup in case there is a problem with the server on their part. But I no longer have access to this and I don't even want to ask for it.

[TryNinja]

How long does it take to do a scraping?
Imagine that today we were doing a scrape of the entire forum to collect this information, how long does it take?

From the analysis I've been doing, the log I'm managing to record, indicates all the images that at some point were seen via the forum proxy.
Now we will be adding to that all the images that are on the forum to date.

There are 65284650 posts. If you scrape one at a time, every second, that's 755 days. But with every page access you can have up to 20 posts (because each page can hold up to 20 posts), so that's at least 38 days (most likely more because lots of posts are on topics with < 20 posts on the page).

Could tecnically do more than one request per second, but theymos has explicitly said bots are only allowed with this limit of 1 req/sec.

Those IPs are not blocked currently. But your other abusive IPs were blocked. Just your quotefast requests (which are only part of what your crawler does) were occurring at an average frequency of 7.6 requests per second in the most recent access logging period. Your requests constituted 3.4% of all forum requests in this period. This is entirely unacceptable and of course resulted in those IPs being banned.

The maximum allowed bot request frequency is 1 request per second. Those IPs are now accessing pages at an average of 2.5 requests per second combined. If you continue exceeding the allowed request limit, we will continue banning your IPs.

[Upgrade00]

There are 65284650 posts. If you scrape one at a time, every second, that's 755 days. But with every page access you can have up to 20 posts (because each page can hold up to 20 posts), so that's at least 38 days (most likely more because lots of posts are on topics with < 20 posts on the page).

Considering that there is the option to view pages less than 25 all at the same time, is it possible for a bot to scrape posts that way to significantly reduce the time it will take?

And if that's not possible what's the limitation?

[joker_josue]

There are 65284650 posts. If you scrape one at a time, every second, that's 755 days. But with every page access you can have up to 20 posts (because each page can hold up to 20 posts), so that's at least 38 days (most likely more because lots of posts are on topics with < 20 posts on the page).

Could tecnically do more than one request per second, but theymos has explicitly said bots are only allowed with this limit of 1 req/sec.

Maybe just starting from a certain date and moving forward is better.


TryNinja / LoyceV
Can you give me a list of all the images you have in your comics from the images hosted on TalkImg? I wanted it by March 31st.
It could be in the format that LoyceV presented. Just a txt file with this list.

[LoyceV]

A fresh scrape from the forum, 1 page per second, will probably take about 5 months. I'd say that's the wrong approach for this purpose, it'll be endless scraping and still incomplete.

I could give you this (incomplete) list from my archives, but considering it's in about 10 million files, reading all data takes a while. TryNinja can probably pull this from his database a lot quicker.

March 31 was a few weeks ago

[joker_josue]

A fresh scrape from the forum, 1 page per second, will probably take about 5 months. I'd say that's the wrong approach for this purpose, it'll be endless scraping and still incomplete.

I don't intend to do this constantly! It would just be a start, for what I want, based on the information already collected. It is not for doing new scraping.

If the tests I'm doing go well, I won't need external information to know which images were accessed by the forum's image proxy, at least once. The plan, if it works, is that whenever an image is accessed via the forum proxy, it will be registered in a DB, which will then be used to make backups.

Additionally, current backups will not change. I will continue to make full backups available weekly.

[TryNinja]

Can you give me a list of all the images you have in your comics from the images hosted on TalkImg? I wanted it by March 31st.
It could be in the format that LoyceV presented. Just a txt file with this list.

Here is for starters. Only original scraped posts: https://files.ninjastic.space/talkimg.txt

Not 100% sure if everything is there, but I think so. No duplicates, one per line. Total 19099.

edit: That's for all posts, do you only want since march 31?

[joker_josue]

Can you give me a list of all the images you have in your comics from the images hosted on TalkImg? I wanted it by March 31st.
It could be in the format that LoyceV presented. Just a txt file with this list.

Here is for starters. Only original scraped posts: https://files.ninjastic.space/talkimg.txt

Not 100% sure if everything is there, but I think so. No duplicates, one per line. Total 19099.

edit: That's for all posts, do you only want since march 31?

Thanks TryNinja, this is exactly what I'll need. Thanks for your help.

Yes, I just need until the end of March, from that month on I will start capturing all the images that were only accessed at least once via the forum proxy.

Don't you have any of the topics that were edited when TalkImg was launched?

[joker_josue]

A little update on the topic. Between the TryNinja archive and the data I was able to gather directly from my server, he was able to gather a set of over 99k links. Thanks for your help. 

For now, it will not be necessary to gather more indirect data. Now let the system work normally.

Now we will do some tests and soon launch a new backup system.

[joker_josue]

I ask the moderators, not to mesquelate/merge posts that I make in a row in this thread.
Especially post that are about service updates.

If you have questions about this situation, talk to me. Thank you for your cooperation.

[LoyceV]

I ask the moderators, not to mesquelate/merge posts that I make in a row in this thread.

I think that only works for posts at least 24 hours apart, or very large posts.

[DYING_S0UL]

Dear joker_josue

I am not able to upload images! The situation is same as I had previously reported several times!

Error report
rec.png - Request denied (401)
Forum time: April 25, 2025, 08:29:52 AM

Using windows, no VPN connected, no firewall or anything of sorts, just the default settings.
Also tried to upload using VPN, but this time nothing happens no error report, it just stuck at 0% upload.
Note that I have a stable internet connection. I am surfing the internet, using the forum just fine.

[joker_josue]

Dear joker_josue

I am not able to upload images! The situation is same as I had previously reported several times!

Error report
rec.png - Request denied (401)
Forum time: April 25, 2025, 08:29:52 AM

Using windows, no VPN connected, no firewall or anything of sorts, just the default settings.
Also tried to upload using VPN, but this time nothing happens no error report, it just stuck at 0% upload.
Note that I have a stable internet connection. I am surfing the internet, using the forum just fine.

Does it only happen with this image?

Did you have the site open in a tab for a long time?
For security reasons, sometimes, when the page is open for a long time without activity, the server turns off the credentials connection (let's call it that). Even when browsing the site, the upload system may not work correctly.
If this is the case, try closing the tab and reopening it or force refreshing the page so that it loads new credentials.

[DYING_S0UL]

Did you have the site open in a tab for a long time?

For security reasons, sometimes, when the page is open for a long time without activity, the server turns off the credentials connection (let's call it that). Even when browsing the site, the upload system may not work correctly.

If this is the case, try closing the tab and reopening it or force refreshing the page so that it loads new credentials.

No.

It looks like you were right!
To confirm your theory I forced closed the browser from task manager, then restarted the laptop, opened a fresh tab and tried uploading different images!
And apparently it worked.

Anyway thanks, I'll keep your suggestion in mind, in case the next time I face something similar!