Ninjastic.space - BitcoinTalk Post/Address archive + API

[BitcoinGirl.Club]

Hey @TryNinja, just wanted to give you a heads up that your bot seems to have taken a break from scraping pages since yesterday. The last post it scraped was #62522464. Not sure if this has anything to do with the ongoing lag issue on bitcointalk, but thought you should know.

Not just Ninjastic Space but also Loyce Club too. Where are you Bot?

Interesting, Loyce club is not archiving: http://loyce.club/archive/topics/545/5459232.html
Ninjastic Space is not archiving too: https://ninjastic.space/topic/5459232

You’re right, I have no idea how I (or anybody else) never noticed that.

Acknowledging it but can’t do anything to fix it now for reasons already disclosed on the other thread (rip my PC and now I’m also back traveling). Seems like most people just type it case-perfectly.

I have been following it since long time. I think long ago I noticed it and I thought it was designed such way so I never thought to inform you or make a post about it LOL

[1714602449]

1714602449

[Csmiami]

----

---

Already replied on the other thread

Cloudflare on the forum is blocking scrapers, mine can't do anything either
Once in a while Cloudflare goes into impossible-mode. To test: visit the forum in a private browser window, if you see Cloudflare, scrapers won't work.

[Stalker22]

Already replied on the other thread
~

Thanks! I was not aware that the issue had already been reported in other threads. I really hope that the CloudFlare problem gets resolved quickly because I would really hate if ninjastic.space go offline. Maybe theymos could consider making an exception for scrapers so they can access the site regardless of CloudFlare?

[nutildah]

I just wanted to say this is a fantastic utility and I use it probably more than any other Bitcointalk utility out there (although BPIP and loyce.club are also both extremely useful). It makes it super easy to research stuff for my own projects as well as stuff I do on the forum and I appreciate the fact that @TryNinja has kept it going all this time. Kudos to you sir for a job well done.

[Bitcoin_Arena]

Something I noticed

Let's say I wanted to search about an announcement of MegaEther.co but then I don't quite remember the extension, so I type only <MegaEther> in the post title search space and hit the search button. I will receive no results at all.
It's not until I type the whole domain as part of the post title <MegaEther.co>  that I will see the results.

I think the domain name even without the extension should be able to pull up results. Those who want an exact match can use the quotes to wrap the full domain name

[light_warrior]

Something I noticed

Let's say I wanted to search about an announcement of MegaEther.co but then I don't quite remember the extension, so I type only <MegaEther> in the post title search space and hit the search button. I will receive no results at all.
It's not until I type the whole domain as part of the post title <MegaEther.co>  that I will see the results.

I think the domain name even without the extension should be able to pull up results. Those who want an exact match can use the quotes to wrap the full domain name

You can search for the word you are looking for in the "Post Contetnt" field. Then you will get what you are looking for. Including topics that contain this word in their title. I just checked by typing in just MegaEther without the domain and got a list of 15 posts that mention the word. And your post is at the top of the list. The first forum mention of this MegaEther in this thread.

I purposely said post title search space and not post content search space.

Check again.

Yes, it doesn't work in the Post Title field. But why not use the search the way I used?

[Bitcoin_Arena]

You can search for the word you are looking for in the "Post Contetnt" field. Then you will get what you are looking for. Including topics that contain this word in their title. I just checked by typing in just MegaEther without the domain and got a list of 15 posts that mention the word. And your post is at the top of the list. The first forum mention of this MegaEther in this thread.

I purposely said post title search space and not post content search space.

Check again.

[hosseinimr93]

I just checked by typing in just MegaEther without the domain and got a list of 15 posts that mention the word.

As mentioned above by Bitcoin_Arena, he was talking about post title, not post content.

And note that even if we search for post content, that's still the same.
All the posts displayed in your search result contain the word MegaEther. If you search for MegaEther, it wouldn't give you the posts that contain MegaEther.co and don't contain "MegaEther".

[Bitcoin_Arena]

Yes, it doesn't work in the Post Title field. But why not use the search the way I used?

Because not every reply or post content might contain the keyword I am looking for (in this case it's a domain)
Just see for yourself. The search results count when you use MegaEther.co as an example are completely different on the Post content field vs Title content field

[light_warrior]

Yes, it doesn't work in the Post Title field. But why not use the search the way I used?

Because not every reply or post content might contain the keyword I am looking for (in this case it's a domain)
Just see for yourself. The search results count when you use MegaEther.co as an example are completely different on the Post content field vs Title content field

You can use the "*" symbol (like this MegaEther*) in the "Post Title" field if you don't remember the domain name. You will get the same result if you enter the full name of the site with the domain MegaEther.co. in the "Post Title" field. The output will give you the same 35 mentions of the site with the domain.

[LoyceV]

I purposely said post title search space and not post content search space.

I can get you a 113 MB text file with all known titles, it will tell you this:

Code:

5177960:🚀 ⭐ MegaEther.co ⭐ Lottery Game Based On The Ethereum Blockchain Platform 🚀
5177981:[BOUNTY]🚀 ⭐ MegaEther.co ⭐ Lottery Game Based On The ETH Blockchain Platform 
5177993:⭐ MegaEther.co ⭐ Lottery Swap Token Swap Event
5178036:MegaEther.co - fake team and plagiarized whitepaper

[TryNinja]

New Ninjastic.space: which features would you like to see?

[light_warrior]

TryNinja, I have a question about the post edit display feature in ninjastic. Using ninjastic I have encountered the following phenomenon. Some posts that have been edited, ninjastic shows the old content and the new content of the post, and some do not. It says "No edit history has been found for this post." even though I know for a fact that the post was edited. I can tell by the underline line. What does this depend on? I know that the forum gives some time to edit a post (how much time, by the way?) without showing the underscore line. After that time passes, that underscore line appears. But ninjastic doesn't register it for some reason.

I guess that a certain amount of time has to pass between writing a post and editing it. In that case, what should be the time interval for ninjastic to commit the post edit. Or am I wrong? Then what is the reason?

[TryNinja]

Some posts that have been edited, ninjastic shows the old content and the new content of the post, and some do not. It says "No edit history has been found for this post." even though I know for a fact that the post was edited. I can tell by the underline line. What does this depend on?

My bot scrapes every post two times: once when it is posted and another time after 5 minutes. If the post was edited at that point, after the first 5 minutes, the changes will show up on the edit history. That’s it.

If you edit a post 10 times, or edit a post after 1 day, 7 days, … there is no way for me (and my bot) to know since keeping track of every change is impossible.

[JayJuanGee]

Some posts that have been edited, ninjastic shows the old content and the new content of the post, and some do not. It says "No edit history has been found for this post." even though I know for a fact that the post was edited. I can tell by the underline line. What does this depend on?

My bot scrapes every post two times: once when it is posted and another time after 5 minutes. If the post was edited at that point, after the first 5 minutes, the changes will show up on the edit history. That’s it.

If you edit a post 10 times, or edit a post after 1 day, 7 days, … there is no way for me (and my bot) to know since keeping track of every change is impossible.

Why 5 minutes?  Why not 2 hours or 6 hours or 12 hours or even 24 hours to really be inclusive of most of the times that edits might happen.

Most of the time, if I am going to edit, I will edit many of my posts within 5 to 20 minutes, but there are some very substantive edits that I might make several hours later, sometimes even 8-16 hours later. Probably less than 1% of any of my edits would end up happening beyond 16 hours after the initial post... and maybe only right around 50% would actually happen within 5 minutes of the initial post.

Of course, I am kind of guessing about my own behavior, but it seems that a second scraping of a few hours would end up being more inclusive to capture a lot more of the edited posts rather than doing it within 5 minutes. 

Don't get me wrong.. I can see that maybe you are correct if you have an assumption that most of the edits happen within 5 minutes of the original post, and the more I sit here and think about it, there are many times that I do edit right away.. but since my posts tend to be long, sometimes it will take longer than 5 minutes just to perform whatever edits that I have.. but sure some of them take less than a minute.. so maybe you have more data or reasons for why you choose 5 minutes for your second scrape.  and also choose not to do a third scrape.

[TryNinja]

Why 5 minutes?  Why not 2 hours or 6 hours or 12 hours or even 24 hours to really be inclusive of most of the times that edits might happen.

One of the reasons I added a post recheck was to catch mentions that only happened in edits. Sometimes you misstype someones name, or you forget to put it on the post, etc… I don’t want people getting a notification of a mention after 2h, 24h, … so 5 min (or even 10) seemed like the sweet spot.

I do plan an overhaul of this system with extra rechecks after 24h and/or 1 week, and/or letting others keep check of a post for changes.

[Lafu]

Hey @TryNinja , would it be possible to make another search field in the Post options on your Webpage?
I was thinking maybe to search after registered Date , that would be for sure helpfull to search the lately Accounts that posting malware.
I dont know if that is possible and just wanted to let you know.
The Webpage is awesome

[LoyceV]

that would be for sure helpfull to search the lately Accounts that posting malware.

May I remind you Finding spam and scams by keyword exists?

[Lafu]

May I remind you Finding spam and scams by keyword exists?

I know LoyceV and i havnt forgot that service from you dont worry.
But in this case i just looking specially for the registered Date from this Accounts.
Because all the Accounts the last 4 weeks was Registered since September 30, 2021 and they still have a lot of them looks like.
It was just an idea that TryNinja maybe can add it if possible.

[libert19]

Hi @TryNinja, https://ninjastic.space/user/username?merits >> latest merit transactions >> images in posts appear as invalid image on ninjastic [1], while on bitcointalk they appear just fine.

For example, this post of yours: Re: TalkImg.com - Image hosting for BitcoinTalk, image appears invalid on Ninjastic [1].


[1] https://www.talkimg.com/image/6fUYI