skarais
Legendary
Offline
Activity: 2590
Merit: 2155
|
|
July 17, 2022, 07:50:21 PM |
|
10.000.000, maybe a bit too much...? I tried running it for the entire archive (no filters) and I gave up before the query could finish. Hahaha, by the way, thanks for the latest upgrade for your awesome tool. I almost asked you for this improvement when updating the board's local activity stats last month, but I forgot about it so quickly. But @Rikafip do it here, everything was fine. Thanks, mate.
|
|
|
|
Rikafip
Legendary
Offline
Activity: 1862
Merit: 6279
|
|
July 17, 2022, 08:48:54 PM |
|
Like @indah rezqi said above, it's an arbitrary number to keep everything running fine. But since there hasn't been any abuse, I'm uncapping it (at least for now). Let me know if it works. Thanks for uncapping it! I did some testing on Bitcoin Discussion board (date range set for the whole 2018) and it works as long as I set the cap at 32,000 users. Everything above that (or not setting it at all) gives me an error after some time of processing it but don't know if cause of error is on my side. I started testing it for the first time, now it's over 10K. From these results, there are 10997 users who actively post on bitcoin discussion boards including child boards during 2020. I think the tool is better now, is there still a limit to the number?
I just did a little bit of comparison regarding active users in the first 7 months (up until July 17th) of 2018 and in the same period during this year and difference is huge: 29815 users active in Bitcoin Discussion board (including child boards) during that period compared to only 3204 in 2022. I did similar comparison for my local board some time ago and results were pretty much similar, decline was around 90% as well.
|
|
|
|
NeuroticFish
Legendary
Offline
Activity: 3780
Merit: 6480
Looking for campaign manager? Contact icopress!
|
|
July 27, 2022, 02:13:21 PM |
|
I have some sort of bug report, I think. It may be nothing, or just a glitch in the matrix, I don't know. It has 2 parts: 1. If I search for: author GazetaBitcoin, post title paraipan, it gives no returns, hence missing https://bitcointalk.org/index.php?topic=5266812Also if I search for: author GazetaBitcoin, post content paraipan, it doesn't return that topic either, and also for example it's missing https://bitcointalk.org/index.php?topic=5135232.msg60637543#msg606375432. For the last 2-3 days not all the mention notifications seem to come and some 6 merits are missed too (meaning 2 or 3 meriting events) Both are no biggie, you've done great job on both tools, still, I thought that you may want to know about this.
|
|
|
|
JeromeTash
Legendary
Offline
Activity: 2254
Merit: 1229
Heisenberg
|
|
July 27, 2022, 02:40:00 PM |
|
It's probably because the post was made in Investigations board. If you remember, Investigations board is off limits to those who have not logged in and newbie ranked users. I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17268
Thick-Skinned Gang Leader and Golden Feather 2021
|
I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule. It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2940
Merit: 7368
|
|
July 27, 2022, 03:05:39 PM |
|
I believe even tools like Ninjastic space are not able to pick any information from that board due to that same rule. It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped. Yep. AFAIK, they won't show up in the "recent posts" page? If that's the case, it explains why it's missing.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17268
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
July 27, 2022, 04:35:11 PM |
|
AFAIK, they won't show up in the "recent posts" page? If that's the case, it explains why it's missing. I just checked for this post, and I found it back in Recent Posts. But only if you log in, which I assume your scraper doesn't do for this.
|
|
|
|
NeuroticFish
Legendary
Offline
Activity: 3780
Merit: 6480
Looking for campaign manager? Contact icopress!
|
|
July 27, 2022, 06:46:28 PM |
|
It's possible to scrape hidden boards too, but those posts aren't supposed to be accessible by anyone (including search engines). That's why they're not scraped.
I've missed this and it makes perfect sense. Thanks for the explanation.
|
|
|
|
decodx
|
@TryNinja, I think the Ninjastic.space scraper bot may be down. Last post scrapped: 60738192 posted on 2022-08-12 10:56:09 UTC The forum is currently at: 60739433, which means that the last 1200 or so posts have not been scraped.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17268
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
August 12, 2022, 02:48:26 PM |
|
the last 1200 or so posts have not been scraped. Don't worry, I have a backup Seriously though: @TryNinja: let me know which posts you missed, and I'll get you a compressed file once your scraper is back. Let's face it: your search system is much better
|
|
|
|
hosseinimr93
Legendary
Offline
Activity: 2506
Merit: 5563
|
|
August 12, 2022, 06:47:37 PM |
|
@TryNinja, I think the Ninjastic.space scraper bot may be down.
Seems that there's a problem with the search engine too. I get an error saying "Something went wrong..." when I try to search for posts or addresses.
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2940
Merit: 7368
|
|
August 12, 2022, 09:49:26 PM |
|
Last post scrapped: 60738192 posted on 2022-08-12 10:56:09 UTC The forum is currently at: 60739433, which means that the last 1200 or so posts have not been scraped. Thanks. My VPS space was full. I got some extra time by deleting my logs. let me know which posts you missed, and I'll get you a compressed file once your scraper is back.
The scraper wasn't actually down, so [AFAIK] nothing is missing. I even got notified for the posts above. My elasticsearch instance (the one I use for the indexing/searching) wasn't being updated because of the lack of space. edit: ok I lied, seems like the scraper missed around 570 posts. Please give me 60738193-60738767.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3416
Merit: 17268
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
August 13, 2022, 05:15:29 AM |
|
Please give me 60738193-60738767. Click It's scheduled to be deleted in 7 days.
|
|
|
|
Xal0lex
Staff
Legendary
Offline
Activity: 2562
Merit: 2527
|
|
August 20, 2022, 07:16:51 PM |
|
TryNinja, is it possible to make a link to your service, with which could search for text without going to your site and without inserting the text in the box? That is, select the text you want, right-click and choose to search your site. Like Google, for example: https://google.com/search?q=%s
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2940
Merit: 7368
|
|
August 20, 2022, 07:25:05 PM |
|
is it possible to make a link to your service, with which could search for text without going to your site and without inserting the text in the box? That is, select the text you want, right-click and choose to search your site.
You mean like a browser extension that adds some kind of “search on ninjastic.space” button to the context menu? You can already use a direct link, btw: https://ninjastic.space/search?content=without%20inserting%20the%20text%20in%20the%20box
|
|
|
|
Stalker22
Legendary
Offline
Activity: 1610
Merit: 1381
|
|
August 20, 2022, 08:18:36 PM |
|
Xal0lex, I installed a custom context search extension in my browser and this works: https://ninjastic.space/search?content=%s You can do the same for BTC/ETH addresses: https://ninjastic.space/addresses?address=%s I do not want to recommend any specific browser extensions, as you will most likely find them on your own.
|
|
|
|
Xal0lex
Staff
Legendary
Offline
Activity: 2562
Merit: 2527
|
is it possible to make a link to your service, with which could search for text without going to your site and without inserting the text in the box? That is, select the text you want, right-click and choose to search your site.
You mean like a browser extension that adds some kind of “search on ninjastic.space” button to the context menu? You can already use a direct link, btw: https://ninjastic.space/search?content=without%20inserting%20the%20text%20in%20the%20boxNo no, I'm not asking you to make a browser extension. That would be too much In my browser, in the search engine settings, I have the ability to add multiple search engines and use them as needed. If I highlight text and right-click, I am given the option to search the highlighted text using one of the search engines listed. This is the list I want to add your service to. In your link after the "=" sign there must be a variable of some kind, like Google's "%s".
|
|
|
|
Stalker22
Legendary
Offline
Activity: 1610
Merit: 1381
|
|
August 20, 2022, 09:22:08 PM |
|
In your link after the "=" sign there must be a variable of some kind, like Google's "%s".
Xal0lex did you see what I suggested in the previous post? That variable "%s" is not related to Google search or any other service. It is used within the extension so that the text you select is transferred to the url, which is then picked up by the search engine (in this case google). The same applies to ninjastic.space.
|
|
|
|
Xal0lex
Staff
Legendary
Offline
Activity: 2562
Merit: 2527
|
|
August 20, 2022, 09:30:03 PM |
|
In your link after the "=" sign there must be a variable of some kind, like Google's "%s".
Xal0lex did you see what I suggested in the previous post? That variable "%s" is not related to Google search or any other service. It is used within the extension so that the text you select is transferred to the url, which is then picked up by the search engine (in this case google). The same applies to ninjastic.space. Sorry, I didn't see your post as the thread went to a new page. Unfortunately, the "%s" variable doesn't work for me. When i use such a variable, it generates the following address (example): https://ninjaastic.space/search?content=I+installed+a+custom+context+search+extension+in%E2%80%A6 The browser "thinks" for a few seconds and finally generates an error.
|
|
|
|
Stalker22
Legendary
Offline
Activity: 1610
Merit: 1381
|
|
August 20, 2022, 09:32:38 PM |
|
In your link after the "=" sign there must be a variable of some kind, like Google's "%s".
Xal0lex did you see what I suggested in the previous post? That variable "%s" is not related to Google search or any other service. It is used within the extension so that the text you select is transferred to the url, which is then picked up by the search engine (in this case google). The same applies to ninjastic.space. Sorry, I didn't see your post as the thread went to a new page. Unfortunately, the "%s" variable doesn't work for me. When i use such a variable, it generates the following address (example): https://ninjaastic.space/search?content=I+installed+a+custom+context+search+extension+in%E2%80%A6 The browser "thinks" for a few seconds and finally generates an error. You have an error in your url. Double "a" in ninjastic.space. It should be: https://ninjastic.space/search?content=%s or like this for addresses: https://ninjastic.space/addresses?address=%s
|
|
|
|
|