Title: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on June 20, 2020, 07:42:25 AM Archiving a thread is now as easy as posting the right link anywhere on the forum (and waiting a bit)!
Short version: Get a topicID you want to see, for instance 5145594 (https://bitcointalk.org/index.php?topic=5145594.0). Insert the topicID into the following link and post it on any public board on Bitcointalk: https://loyce.club/archive/details/topic_5145594.html Wait a bit, then click the link! Full version Almost a year ago, I opened 35M posts! View unedited/deleted posts (search per post, per user or per topic) (https://bitcointalk.org/index.php?topic=5167469.0). This has a lot of data (currently around 60 GB), but it's painstaking to manually find exactly which posts in a topic are edited or deleted. I started archiving posts in July 2019. Especially at the beginning I missed some posts due to down time, and even now I occasionally miss some posts due to connection problems. Since February 2020 (https://bitcointalk.org/index.php?topic=5167469.msg53759604#msg53759604), I'm scraping and archiving all older posts too What it does I've created an on-demand service to get details from any topic for:
The Topic Details If a new update for the same topic is requested, I'll include a list of all previous Topic Details. You'll have to make a new post to be detected by my scraper. Editing an existing post won't get detected. Please don't quote the archive link, it'll trigger another update. Sample output explained https://loyce.club/other/details.gif Post 50988796 (https://bitcointalk.org/index.php?topic=1813624.msg50988796#msg50988796) links to the post on Bitcointalk (even if the post itself has been deleted). Post 235 (https://loyce.club/archive/details/topic_1813624_2020-06-20_Sat_08.52h.html#msg51134694) is older than my archive (https://bitcointalk.org/index.php?topic=5167469.0). I don't have an unedited backup, so I created a new backup. Post 236 is Deleted! I have an unedited backup, no need for a new backup. Post 242 was Edited! I have an unedited backup, and created a new backup of the current post. Post 246 is Unedited! No need for a new backup. Post 251 doesn't have an unedited backup (which means my scraper was offline at the moment), so I created a new backup. The (link (https://loyce.club/archive/details/topic_1813624_2020-06-20_Sat_08.52h.html#msg52297446)) at the end of each line points at that specific row in my list. https://loyce.club/other/details0.gif If I have no If a post was removed before I tried to archive it, I (obviously) can't list it. Limitations
Test it! Please try it, and let me know if it works as expected. Bugs Please post! This is far from finished. Intended use I'm hoping this can be useful to expose certain scammers. Please don't turn this into a(nother) witch hunt. Be nice Don't try to abuse this. I don't want to make a blacklist, but I will if I have to. Todo
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Rikafip on June 21, 2020, 08:15:08 AM Bump: I need testers :) Hm, I don't know what I am doing wrong, but this thing doesn't work for me, all I get is 404 error message.55 second early bump, that's how much I need testers I tried inserting topicID 5256136 ([ANN] DSF - The SoFi Blockchain - Redefine Social Network with Blockchain (https://bitcointalk.org/index.php?topic=5256136.0), but nothing. Tried waiting for a few minutes, as you said to wait a bit, but that didn't help either. This is the link http://loyce.club/archive/details/topic_5256136.html edit: it works now :) So I guess I had to wait for it to get processed/updated, as you said. I like this new feature a lot, it makes it so much easier to find deleted/edited posts. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: acroman08 on June 21, 2020, 08:38:31 AM I'm not sure if I am doing this correctly. will delete the post later if I made a mistake but if not, I'll update if it's working.
testing: http://loyce.club/archive/details/topic_5257024.html the link works! thanks! great job as always! edit: it works now :) trueSo I guess I had to wait for it to get processed/updated, as you said. I like this new feature a lot, it makes it so much easier to find deleted/edited posts. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: SFR10 on June 21, 2020, 08:39:42 AM Any way to get a "permanent archive" link prior to its current date or up to a specific date only?
Edited! Can your scraper detect edits within the 10-minute mark?Bump: I need testers :) Another test for "this topic (https://bitcointalk.org/index.php?topic=5243791.0)" :)- Link: http://loyce.club/archive/details/topic_5243791.html Update: It's working. Wish I had some sMerits for you. Update 2: ~Snipped~ Thank you for these :)Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on June 21, 2020, 08:44:39 AM Hm, I don't know what I am doing wrong, but this thing doesn't work for me, all I get is 404 error message. Did you post the link in a new post?In your post history (http://loyce.club/archive/members/265/2658890.html), I can't find the link anywhere else than in your current post (http://loyce.club/archive/posts/5465/54657119.html). My scraper didn't miss any recent posts (http://loyce.club/archive/posts/5465/), so I can only conclude you didn't post it. If you don't post it, my scraper won't create it. I've underlined the "post it" in my OP. Quote Tried waiting for a few minutes If all goes well, after a few seconds there should be a page telling you it's now scraping your topic. Depending on the number of pages that can take a while, but it auto-refreshes.Quote So I guess I had to wait for it to get processed/updated I think all you had to do was posting the link. When you posted about it, it worked :PTitle: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Rikafip on June 21, 2020, 08:51:19 AM Did you post the link in a new post? No I didn't, I was just trying it without actually posting, somehow I completely missed that underlined "post it" part, and I posted it when I thought it's not working. And then it worked :pIf you don't post it, my scraper won't create it. I've underlined the "post it" in my OP. I think all you had to do was posting the link. When you posted about it, it worked :P Yep, that was the issue. My bad, all good now! Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on June 21, 2020, 08:52:42 AM Any way to get a "permanent archive" link prior to its current date or up to a specific date only? I'm not sure what you're looking for. If I have an older version available, the link shows up.If there is no older version, I'll mention that from now on. Example: http://loyce.club/archive/details/topic_5256854.html Quote Can your scraper detect edits within the 10-minute mark? Yes. Your post (http://loyce.club/archive/posts/5465/54657224.html) for instance was scraped after 4 seconds.No I didn't, I was just trying it without actually posting, somehow I completely missed that underlined "post it" part, and I posted it when I thought it's not working. I'm not so advanced that I can do cool server-side stuff. I need a post to trigger a request.The "no unedited backup (http://loyce.club/archive/details/topic_5256854_2020-06-21_Sun_10.52h.html#msg54657268)" shows up for posts that are less than a few minutes old. Even though I archived the post (http://loyce.club/archive/posts/5465/54657268.html) already, my topic list (http://loyce.club/archive/topics/525/5256854.html) gets updated only once every 5 minutes. This is a shortcoming I can live with :) Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: alani123 on July 13, 2020, 09:59:53 AM What better place to contribute to the testing other than this thread?
I'll try one of my own posts (https://bitcointalk.org/index.php?topic=797299.0): https://loyce.club/archive/details/topic_797299.html Let's see. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 13, 2020, 10:06:43 AM I'll try one of my own posts (https://bitcointalk.org/index.php?topic=797299.0): Unfortunately, since I scraped this thread only this year, I can't highlight older deleted posts.Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Timelord2067 on July 13, 2020, 12:48:38 PM Only one of three requests seems to have worked: https://bitcointalk.org/index.php?topic=2544574.msg54786188#msg54786188
Mk III has 117 pages (and counting) - far more than the Mk II & Mk I combined. (and all three threads have multiple posts that get reviewed and edited at later dates) Do more pages get scraped at a later date if a new request is made? Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: TheBeardedBaby on July 13, 2020, 12:53:46 PM Let's try it :)
https://loyce.club/archive/details/topic_996318.html Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Timelord2067 on July 13, 2020, 12:55:00 PM Let's try it :) https://loyce.club/archive/details/topic_996318.html Perhaps yours is in a queue behind my three (two of which might be in a holding pattern at the moment?) Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: TheBeardedBaby on July 13, 2020, 12:58:04 PM Mine is quite a huge thread -> Stake your Bitcoin address here.
I hope I haven't killed the server :D (if so, send me the bill in PM...) It will take quite some time to analyze it myself but it could be worth it. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: alani123 on July 13, 2020, 03:18:01 PM Doesn't quoting the link re-trigger the scrapper? You might have killed it twice or thrice for good measure. :D
Just kidding. LoyceV probably had considered the possibility. ;) Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 13, 2020, 03:24:34 PM Only one of three requests seems to have worked: Update: I now allow up to 5 requests (from one user or from different users) at once. So what you and TheBeardedBaby did earlier shouldn't be a problem from now on. I also noticed I forgot to activate the limitation on number of pages. I'll increase it to 250 pages, just know that it will take a very long time if a topic is that long. Quote Do more pages get scraped at a later date if a new request is made? No, not at the moment. However, I've quoted your posts to trigger your previously ignored requests.Perhaps yours is in a queue behind my three (two of which might be in a holding pattern at the moment?) Correct. TheBeardedBaby posted this while you occupied my scraper.IF it works, you should see something on the URL you posted within a few seconds. That placeholder stays until it's done. I restarted TheBeardedBaby's request. Doesn't quoting the link re-trigger the scrapper? Correct.I'll make some changes: when I'm done, I'll allow more than one request at once. But each additional request will slow down the other requests, I don't want to cause too much load on the forum. Updates done :) Feel free to test it. See status.txt (https://loyce.club/archive/details/status.txt) for currently running tasks. Mine is quite a huge thread -> Stake your Bitcoin address here. It becomes interesting from the moment I started archiving all posts (https://loyce.club/archive/details/topic_996318_2020-07-13_Mon_16.50h.html#msg51929114). It also shows the startup problems I've had with failed archives due to down time (https://loyce.club/archive/details/topic_996318_2020-07-13_Mon_16.50h.html#msg52135405).I currently can't catch older deleted posts (that may have been deleted after I scraped them). I'll add this to my Todo. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: TheBeardedBaby on July 14, 2020, 12:23:56 PM I can say this is an amazing tool. I finally got the time to go trough the long threads and yes, it's working perfectly well.
My bookmark list of LoyceVs threads is growing, soon start to be difficult to get oriented in it, I have to start dividing them in sections based on categories. This is only a good news. Perfect tool to spot a bumping services, I'll give it more tests after we are back from vacation, traveling this weekend :) Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 17, 2020, 06:23:46 PM TheBeardedBaby (https://bitcointalk.org/index.php?action=profile;u=1291828) made this suggestion:
Can you add a link to the deleted posts from each user in the list. I'm talking about this > I've added it, click "more (http://)" to see all unedited/deleted (https://bitcointalk.org/index.php?topic=5167469.0) posts:Code: https://loyce.club/archive/members/267/2670747.html https://loyce.club/other/details2.gif Disclaimer: I don't check if I actually have archived older posts for that user. Users who haven't posted in the last year will lead to a 404 for now. When I'm done scraping older posts (https://bitcointalk.org/index.php?topic=5167469.msg53759604#msg53759604), I may add all of them. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: kire - cryptzino on July 21, 2020, 05:45:12 PM Great tool, and I'm trying to utilize it.
It seems my last topic is deleted. But as I don't have PM or other messages, I don't know the topic ID. Is there a way to check it? Thank you Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 21, 2020, 06:15:35 PM I don't know the topic ID. You can find your old posts using your userID here (https://bitcointalk.org/index.php?topic=5167469.0):Is there a way to check it? Viewing unedited/deleted posts
How to use it
So in your case, that brings you to unedited (or deleted) posts made by kire - cryptzino (https://loyce.club/archive/members/282/2822262.html). This (https://loyce.club/archive/posts/5482/54829792.html) post was deleted, and I guess this is what you're looking for: https://loyce.club/archive/details/topic_5263237.html Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: kire - cryptzino on July 21, 2020, 08:41:54 PM ... Thank you! Precisely what I was looking for. Now I will go after the mystery of why the topic was deleted. I just hope this is not a standard practice around. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Rizzrack on July 29, 2020, 11:36:01 PM https://loyce.club/archive/details/topic_2647654.html and bump :P
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: alani123 on July 30, 2020, 02:29:53 AM https://loyce.club/archive/details/topic_5264620.html
Can this be used to see if a thread was moved to a different board? Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 30, 2020, 06:56:08 AM ~snip link to prevent it from archiving again~ My "later backups" ended here (https://loyce.club/archive/details/topic_2647654_2020-07-30_Thu_01.36h.html#msg37976149). That's because I haven't processed (https://loyce.club/archive/oldposts/) all data yet, but I should have it archived already. If you're looking for an older post that was deleted recently, you can try again later. I need some time to finalize this data.Can this be used to see if a thread was moved to a different board? No, I don't keep track of boards. The thread you requested was on the Reputation board when I first saw it.Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Jet Cash on July 30, 2020, 07:43:56 AM There are two types of edit. One is a change to the text because the poster made a typo, or posted the wrong info. The other edit is an addition to a post, and this is something I do. Your post may be the last in a thread, and you may want to add additional information without creating a series of posts by a single member. It would be interesting to be able to find these posts. Maybe we could tag them to make them easier to list. Tags such as [UPDATE] or [ADDITION] could be used.
Another advantage of this is that such threads would not be bumped, but an alert would allow interested readers to find them. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Rizzrack on July 30, 2020, 07:54:29 AM My "later backups" ended here (https://loyce.club/archive/details/topic_2647654_2020-07-30_Thu_01.36h.html#msg37976149). That's because I haven't processed (https://loyce.club/archive/oldposts/) all data yet, but I should have it archived already. If you're looking for an older post that was deleted recently, you can try again later. I need some time to finalize this data. Was actually looking for a recent post that was recently deleted (few days old) Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 30, 2020, 08:10:09 AM There are two types of edit. One is a change to the text because the poster made a typo, or posted the wrong info. The other edit is an addition to a post, and this is something I do. Your post may be the last in a thread, and you may want to add additional information without creating a series of posts by a single member. It would be interesting to be able to find these posts. Maybe we could tag them to make them easier to list. Tags such as [UPDATE] or [ADDITION] could be used. I won't even try to automate detecting the difference, but it happens a lot indeed. People adding a comma or fixing a typo. Those can easily be spotted by comparing the old and new backup, and those cases are not the reason I created this topic.Quote Another advantage of this is that such threads would not be bumped, but an alert would allow interested readers to find them. If the last post in a topic gets edited after I've read it, it shows as https://bitcointalk.org/Themes/custom1/images/english/new.gif on the board overview (https://bitcointalk.org/index.php?board=24.0). But it doesn't show as https://bitcointalk.org/Themes/custom1/images/english/new.gif on the Watchlist (https://bitcointalk.org/index.php?action=watchlist). That could be a good feature to add to the forum.Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: alani123 on July 30, 2020, 10:31:52 PM Can this be used to see if a thread was moved to a different board? No, I don't keep track of boards. The thread you requested was on the Reputation board when I first saw it.Now I wonder about how the archive tool behaves on moved (and trash-canned) threads, redirection messages and the likes. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on July 31, 2020, 06:34:59 AM Hm, it might be useful to note the board. It's too late for that, unless I re-scrape all topics everything, and I don't really want to do that.Quote Now I wonder about how the archive tool behaves on moved (and trash-canned) threads, redirection messages and the likes. Moving to a different board doesn't change the topicID, and I search everything based on the topicID. If the topic gets deleted, my details still work on the original topicID.I'm not sure if Mods can merge or split topics, but that would mess it up. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on August 12, 2020, 10:32:53 AM Update:
Since February 2020 (https://bitcointalk.org/index.php?topic=5167469.msg53759604#msg53759604), I'm scraping and archiving all older posts too I now have (almost) all posts archived. However, I still don't highlight deleted posts from my "older" archive. I'll add this after listing all posts per topic (and per member).Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: namarang on August 17, 2020, 04:30:18 PM I think the update should include the content and generally everything included in the post.
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: alani123 on August 18, 2020, 11:28:31 PM I think it would be super useful if we could see which posts/replies of a user profile have been deleted by having it highlighted on the user page also...
It can be done for a post, so maybe in a similar fashion also for a profile. Perhaps on a per-request basis to reduce load by extra scraping/maching. Thanks. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on August 19, 2020, 08:50:58 AM I think it would be super useful if we could see which posts/replies of a user profile have been deleted by having it highlighted on the user page also... I agree, but:My initial plan was to make this for a user's post history too, but it was much more work than anticipated, so I skipped that. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: Rikafip on October 26, 2020, 09:57:19 AM Bump for this excellent tool!
https://loyce.club/archive/details/topic_5265765.html Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on October 31, 2020, 09:05:54 AM https://loyce.club/archive/details/topic_5282911.html
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: DarkStar_ on November 29, 2020, 05:42:51 PM https://loyce.club/archive/details/topic_3243635.html
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: DarkStar_ on January 24, 2021, 07:40:56 PM https://loyce.club/archive/details/topic_3243635.html
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: DarkStar_ on May 13, 2021, 02:03:37 AM https://loyce.club/archive/details/topic_3243635.html
Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: suchmoon on May 13, 2021, 02:38:06 AM https://loyce.club/archive/details/topic_3243635.html Something seems to be broken... the thread is 59 pages but the scraper is at 220+ already. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: DarkStar_ on May 13, 2021, 02:45:14 AM https://loyce.club/archive/details/topic_3243635.html Something seems to be broken... the thread is 59 pages but the scraper is at 220+ already. It seems to have finally stopped, with just the minor side effect of posts on the last thread showing 192 times (https://loyce.club/archive/details/2021-05-13_Thu_04.03h/56928249.html). I wonder if some sort of rate limit was hit to stop it; I was kinda expecting it to run infinitely until Loyce stops it :P Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: suchmoon on May 13, 2021, 02:58:34 AM It seems to have finally stopped, with just the minor side effect of posts on the last thread showing 192 times (https://loyce.club/archive/details/2021-05-13_Thu_04.03h/56928249.html). I wonder if some sort of rate limit was hit to stop it; I was kinda expecting it to run infinitely until Loyce stops it :P I think it hit the 250-page hard limit: This service is currently limited to scraping the first 250 pages of a topic. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on May 13, 2021, 09:31:06 AM Something seems to be broken... the thread is 59 pages but the scraper is at 220+ already. It seems to have finally stoppedI think it hit the 250-page hard limit: Correct.It should be fixed now, let's test: https://loyce.club/archive/details/topic_3243635.html Update: all fine :) Please post if something else fails. Title: Re: LoyceV's Topic Details: highlight deleted and edited posts (forum wide) Post by: LoyceV on June 05, 2021, 08:31:51 AM This service is temporarily out of order. I'm fixing shortcomings in some of the data (https://bitcointalk.org/index.php?topic=5167469.msg57133811#msg57133811) it uses, and this will take a while (at least a week, possibly longer).
When done, I'll add this functionality: list deleted "older" posts. I only found out now (May 29, 2021) that this doesn't work. |