Ninjastic.space - BitcoinTalk Post/Address archive + API

[skarais]

10.000.000, maybe a bit too much...?

I tried running it for the entire archive (no filters) and I gave up before the query could finish.

Hahaha, by the way, thanks for the latest upgrade for your awesome tool. I almost asked you for this improvement when updating the board's local activity stats last month, but I forgot about it so quickly. But @Rikafip do it here, everything was fine. Thanks, mate. 

[Rikafip]

Like @indah rezqi said above, it's an arbitrary number to keep everything running fine. But since there hasn't been any abuse, I'm uncapping it (at least for now). Let me know if it works.

Thanks for uncapping it!

I did some testing on Bitcoin Discussion board (date range set for the whole 2018) and  it works as long as I set the cap at 32,000 users. Everything above that (or not setting it at all) gives me an error after some time of processing it but don't know if cause of error is on my side.

I started testing it for the first time, now it's over 10K. From these results, there are 10997 users who actively post on bitcoin discussion boards including child boards during 2020. I think the tool is better now, is there still a limit to the number?

I just did a little bit of comparison regarding active users in the first 7 months (up until July 17th) of 2018 and in the same period during this year and difference is huge: 29815 users active in Bitcoin Discussion board (including child boards) during that period compared to only 3204 in 2022. I did similar comparison for my local board some time ago and results were pretty much similar, decline was around 90% as well.

[NeuroticFish]

I have some sort of bug report, I think. It may be nothing, or just a glitch in the matrix, I don't know.

It has 2 parts:
1. If I search for: author GazetaBitcoin, post title paraipan, it gives no returns, hence missing https://bitcointalk.org/index.php?topic=5266812
Also if I search for: author GazetaBitcoin, post content paraipan, it doesn't return that topic either, and also for example it's missing https://bitcointalk.org/index.php?topic=5135232.msg60637543#msg60637543

2. For the last 2-3 days not all the mention notifications seem to come and some 6 merits are missed too (meaning 2 or 3 meriting events)


Both are no biggie, you've done great job on both tools, still, I thought that you may want to know about this.

[JeromeTash]

I have some sort of bug report, I think. It may be nothing, or just a glitch in the matrix, I don't know.

It has 2 parts:
1. If I search for: author GazetaBitcoin, post title paraipan, it gives no returns, hence missing https://bitcointalk.org/index.php?topic=5266812
Also if I search for: author GazetaBitcoin, post content paraipan, it doesn't return that topic either, and also for example it's missing https://bitcointalk.org/index.php?topic=5135232.msg60637543#msg60637543

It's probably because the post was made in Investigations board. If you remember, Investigations board is off limits to those who have not logged in and newbie ranked users. I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.

[LoyceV]

I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.

It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.

[TryNinja]

I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.

It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.

Yep.

AFAIK, they won't show up in the "recent posts" page? If that's the case, it explains why it's missing.

[LoyceV]

AFAIK, they won't show up in the "recent posts" page? If that's the case, it explains why it's missing.

I just checked for this post, and I found it back in Recent Posts. But only if you log in, which I assume your scraper doesn't do for this.

[NeuroticFish]

It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.

I've missed this and it makes perfect sense. Thanks for the explanation.

[decodx]

@TryNinja, I think the Ninjastic.space scraper bot may be down.

Last post scrapped: 60738192 posted on 2022-08-12 10:56:09 UTC
The forum is currently at: 60739433, which means that the last 1200 or so posts have not been scraped.

[LoyceV]

the last 1200 or so posts have not been scraped.

Don't worry, I have a backup
Seriously though: @TryNinja: let me know which posts you missed, and I'll get you a compressed file once your scraper is back. Let's face it: your search system is much better

[hosseinimr93]

@TryNinja, I think the Ninjastic.space scraper bot may be down.

Seems that there's a problem with the search engine too.
I get an error saying "Something went wrong..." when I try to search for posts or addresses.

[TryNinja]

Last post scrapped: 60738192 posted on 2022-08-12 10:56:09 UTC
The forum is currently at: 60739433, which means that the last 1200 or so posts have not been scraped.

Thanks.

My VPS space was full.

I got some extra time by deleting my logs.

let me know which posts you missed, and I'll get you a compressed file once your scraper is back.

The scraper wasn't actually down, so [AFAIK] nothing is missing. I even got notified for the posts above.

My elasticsearch instance (the one I use for the indexing/searching) wasn't being updated because of the lack of space.

edit: ok I lied, seems like the scraper missed around 570 posts. Please give me 60738193-60738767.

[LoyceV]

Please give me 60738193-60738767.

Click
It's scheduled to be deleted in 7 days.

[Xal0lex]

TryNinja, is it possible to make a link to your service, with which could search for text without going to your site and without inserting the text in the box? That is, select the text you want, right-click and choose to search your site. Like Google, for example:

Code:

https://google.com/search?q=%s

[TryNinja]

is it possible to make a link to your service, with which could search for text without going to your site and without inserting the text in the box? That is, select the text you want, right-click and choose to search your site.

You mean like a browser extension that adds some kind of “search on ninjastic.space” button to the context menu?

You can already use a direct link, btw: https://ninjastic.space/search?content=without%20inserting%20the%20text%20in%20the%20box

[Stalker22]

Xal0lex, I installed a custom context search extension in my browser and this works:

Code:

https://ninjastic.space/search?content=%s

You can do the same for BTC/ETH addresses:

Code:

https://ninjastic.space/addresses?address=%s

I do not want to recommend any specific browser extensions, as you will most likely find them on your own.

[Xal0lex]

is it possible to make a link to your service, with which could search for text without going to your site and without inserting the text in the box? That is, select the text you want, right-click and choose to search your site.

You mean like a browser extension that adds some kind of “search on ninjastic.space” button to the context menu?

You can already use a direct link, btw: https://ninjastic.space/search?content=without%20inserting%20the%20text%20in%20the%20box

No no, I'm not asking you to make a browser extension. That would be too much In my browser, in the search engine settings, I have the ability to add multiple search engines and use them as needed. If I highlight text and right-click, I am given the option to search the highlighted text using one of the search engines listed. This is the list I want to add your service to.



In your link after the "=" sign there must be a variable of some kind, like Google's "%s".

[Stalker22]

In your link after the "=" sign there must be a variable of some kind, like Google's "%s".

Xal0lex did you see what I suggested in the previous post?

That variable "%s" is not related to Google search or any other service. It is used within the extension so that the text you select is transferred to the url, which is then picked up by the search engine (in this case google). The same applies to ninjastic.space.

[Xal0lex]

In your link after the "=" sign there must be a variable of some kind, like Google's "%s".

Xal0lex did you see what I suggested in the previous post?

That variable "%s" is not related to Google search or any other service. It is used within the extension so that the text you select is transferred to the url, which is then picked up by the search engine (in this case google). The same applies to ninjastic.space.

Sorry, I didn't see your post as the thread went to a new page. Unfortunately, the "%s" variable doesn't work for me. When i use such a variable, it generates the following address (example): https://ninjaastic.space/search?content=I+installed+a+custom+context+search+extension+in%E2%80%A6 The browser "thinks" for a few seconds and finally generates an error.

[Stalker22]

In your link after the "=" sign there must be a variable of some kind, like Google's "%s".

Xal0lex did you see what I suggested in the previous post?

That variable "%s" is not related to Google search or any other service. It is used within the extension so that the text you select is transferred to the url, which is then picked up by the search engine (in this case google). The same applies to ninjastic.space.

Sorry, I didn't see your post as the thread went to a new page. Unfortunately, the "%s" variable doesn't work for me. When i use such a variable, it generates the following address (example): https://ninjaastic.space/search?content=I+installed+a+custom+context+search+extension+in%E2%80%A6 The browser "thinks" for a few seconds and finally generates an error.

You have an error in your url. Double "a" in ninjastic.space.

It should be:

Code:

https://ninjastic.space/search?content=%s

or like this for addresses:

Code:

https://ninjastic.space/addresses?address=%s