Ninjastic.space - BitcoinTalk Post/Address archive + API

[TryNinja]

The new ninjastic.space website is out! Remade from scratch.

Features:

- New darkmode design.
- Search posts by author, topic id, content, date range and/or board.
- Search addresses for posts and connections between users (Bitcoin and Ethereum).
- How many posts/merits were made/sent every day.
- RTL support for RTL-languages (e.g Arabic).
- Check the unedited/archived posts along with their edited version made up to 5 minutes later. ^NEW
- Check user stats (e.g posts per board, posts per day), mentioned addresses, etc.... ^NEW

Pages:

- Unedited/archived post: https://ninjastic.space/post/55139442
- Unedited/archived posts on specific topic: https://ninjastic.space/topic/5273824
- Unedited/archived posts by address: https://ninjastic.space/address/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L
- User stats, favorite topics, deleted/edited posts and addresses: https://ninjastic.space/user/TryNinja ^NEW

API:

If you have a cool project or project idea that requires any posts/addresses data, I can help you with my REST API.

Here is the documentation: https://docs.ninjastic.space

Source Code:

- https://bitcointalk.org/index.php?topic=5273824.msg57711630#msg57711630

Donations

100% of the donations will be used to pay the bot and website expenses (~$18/month):

BTC Bech32: bc1qlfzjqgleh3pg7l63p9fc596uqv30hqr9dpg59q
BTC Legacy: 1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L

New ideas are highly appreciated!

[1696149260]

1696149260

[1696149260]

1696149260

[1696149260]

1696149260

[Tytanowy Janusz]

Great tool. I needed something like that.

Maybe I'm doing something wrong but it looks to me that search function needs a little bug-fix. When i put Topic ID and content bot finds me all post with this content without filtering to given topic.

[TryNinja]

Great tool. I needed something like that.

Maybe I'm doing something wrong but it looks to me that search function needs a little bug-fix. When i put Topic ID and content bot finds me all post with this content without filtering to given topic.

Oopss.. forgot an "&" in the query string. Please, try again.

[Tytanowy Janusz]

Oopss.. forgot an "&" in the query string. Please, try again.

Still the same. Put some more "&"



edit (answering post below to avoid spam):

@TryNinja You were right. Cashed version was the case. It works great now.

[TryNinja]

Still the same. Put some more "&"

I'm 100% sure you are looking at a cached version of the page.

Try Ctrl + F5 to clear the cache and try once again.

[LoyceV]

- New darkmode design.

For archiving purposes, I don't like editing the font colors away from the original.

- Unedited/archived post: https://ninjastic.space/post/55139442

That's easier than https://loyce.club/archive/posts/5513/55139442.html Databases FTW

(thanks to @LoyceV for his oldposts archive).

You're welcome Under "archived", it's safe to change "This may or may not represent the time and date the post was made" to something like "This was long after the post was made".
Also: the word is scraped, not scrapped, right?

- Unedited/archived posts by address

I'll see what I can pull out of your data dump when I have the time

[TryNinja]

For archiving purposes, I don't like editing the font colors away from the original.

You have a point. I will change that.

You're welcome Under "archived", it's safe to change "This may or may not represent the time and date the post was made" to something like "This was long after the post was made".
Also: the word is scraped, not scrapped, right?

Thanks. I still make this mistake more often than I should.

- Unedited/archived posts by address

I'll see what I can pull out of your data dump when I have the time

I plan to always make a new dump every few days so anyone can download it when they want/need it. Make sure to take a look at http://public.ninjastic.space to get the latest one when you decide to use it.

[SFR10]

And you can always suggest/ask for a different kind of data. Just post here/PM me.

Is it possible to get the "general statistics for each user ^{[e.g. like this]}"?
^{- I'm after the last two stats (Most Popular Boards By Posts and Activity).}
I noticed it shows "(Unknown Title)" for threads written in languages other than English ^{[not a big deal though]}.
^{- Apart from that, it looks great. Bookmarked.}

[protrader786]

<..>

Hey SRF, can I know how you were able to manage to view user stats please? Or is it just an old archive ? When I click on my profile or others profiles to see the stats, it tells me that user stats are disabled. Please let me know.

[Stalker22]

This is a great tool and I'm really glad I came across this thread. Well done, TryNinja!
This can really make it easier to search forum posts, especially when discovering alt accounts of ban evaders or merit abusers.

Hey SRF, can I know how you were able to manage to view user stats please? Or is it just an old archive ? When I click on my profile or others profiles to see the stats, it tells me that user stats are disabled. Please let me know.

I believe it is an old archive. That's why SFR10 asked if TryNinja could do something similar with his data.

[TryNinja]

I really like the feature search post, it's more efficient that using google search and have some unique search option.

Thanks! It was hard to make it usable with 47m+ posts, so I'm glad it's working. I had to learn a lot about how databases work and indexes. It's not perfect, though.

Is it possible to get the "general statistics for each user ^{[e.g. like this]}"?
^{- I'm after the last two stats (Most Popular Boards By Posts and Activity).}

Yes! Or kinda of. I'm already working on something like this.

Some stats won't be available, since I can't get them (votes casted, polls created, time spent online, topics created, most popular boards per activity, etc...). But you will be able to see stuff like in which boards you posted the most (and when you did it).

I noticed it shows "(Unknown Title)" for threads written in languages other than English ^{[not a big deal though]}.

The "Unknown Title" shows up for the posts Loyce provided to me. His data didn't include any title, so I would need to scrape everything from zero to get them. New posts and posts which got notified to a bot user have their title.

[LoyceV]

Some stats won't be available, since I can't get them (~topics created

Just a thought: you can probably get this if you check which user made the first post in each topic.

The "Unknown Title" shows up for the posts Loyce provided to me. His data didn't include any title, so I would need to scrape everything from zero to get them. New posts and posts which got notified to a bot user have their title.

Most topics have been deleted, so if you scrape those titles, you can skip a few million topics to speed things up.

[Stalker22]

I can't even image how difficult it is.

It's just matter of convenience, but it is possible to bold text on the posts that matches the content/keyword? For example, if i use content/keyword "blockchain", all texts containing "blockchain" would be bolded automatically.

You mean, something like, how bitcointalk search function does? Yeah, that would be nice. I'm sure TryNinja can make the search term highlighted in the results.

[Aveatrex]

Amazing! I see that you are making use of Reactjs, way to go .  I'm interested in your RESTful API, can you elaborate on how it works?

[masulum]

Everything is good, the search post feature is very helpful, to make this feature are different with show the latest post from Bitcointalk, maybe you can add filter posts there. Not an important thing, just additional features maybe will help in the future.

Also, Link color need to change from blue with more soft color. I try to use this style:

Code:

a {
    color: darkorange;
    text-decoration: none;
    background-color: transparent;
    outline: none;
    cursor: pointer;
    -webkit-transition: color .3s;
    transition: color .3s;
    -webkit-text-decoration-skip: objects;
}

For me (maybe different experience with other users), background and the text color its doesn't match. if you are happy with that, its okay nothing to change.

example result:
       Test -       Test         

[suchmoon]

Nitpick: for the front-page charts perhaps it makes sense to exclude today otherwise the charts have a bit of a misleading dip at the end there.

Looks great. Very fast. Not sure how accurate it is though. I can't believe I said "cunt" only 7 times. Does it search only outside of quotes? Only full words?

[TryNinja]

I can't even image how difficult it is.

It's just matter of convenience, but it is possible to bold text on the posts that matches the content/keyword? For example, if i use content/keyword "blockchain", all texts containing "blockchain" would be bolded automatically.

I think it is. I'll look at it later today.

Amazing! I see that you are making use of Reactjs, way to go .  I'm interested in your RESTful API, can you elaborate on how it works?

I have a few endpoints which will return the data you need. For example:

GET: /posts/55141939

Code:

{
  "id": "1c25054c-b1b8-41eb-8c66-697c8b697179",
  "post_id": 55142446,
  "topic_id": 5273824,
  "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API",
  "author": "Aveatrex",
  "author_uid": 950474,
  "content": "Amazing! I see that you are making use of Reactjs, way to go <img src=\"https://bitcointalk.org/Smileys/default/smiley.gif\" alt=\"Smiley\" border=\"0\"> .  I'm interested in your RESTful API, can you elaborate on how it works? ",
  "date": "2020-09-06T12:45:48.000Z",
  "boards": [
    "Other",
    "Meta"
  ],
  "archive": false
}

For me (maybe different experience with other users), background and the text color its doesn't match. if you are happy with that, its okay nothing to change.

It will be fixed. Thanks!

Nitpick: for the front-page charts perhaps it makes sense to exclude today otherwise the charts have a bit of a misleading dip at the end there.

Makes sense. I'll remove today's data.

Looks great. Very fast. Not sure how accurate it is though. I can't believe I said "cunt" only 7 times. Does it search only outside of quotes? Only full words?

Goddammit, you are right. I messed up big with my algo. I hate regex.

I will fix it and the results should increase considerably for every other search. It does look inside quotes, but the regex is eating a few parts of the post it shouldn't when there are multiple quotes.

[Ratimov]

The most luxurious solution, especially the search by address function, where a list of all users who published these addresses is immediately displayed. Many thanks to the author for this site.



Of course, it would be great if such a function worked not only on wallet addresses, but also on social media addresses (twitter, facebook, telegram, etc.)

[suchmoon]

Goddammit, you are right. I messed up big with my algo. I hate regex.

I will fix it and the results should increase considerably for every other search. It does look inside quotes, but the regex is eating a few parts of the post it shouldn't when there are multiple quotes.

Maybe don't rush with that. Ignoring quotes could be a feature, at least an optional one. Most of the time I would probably want someone the user posted themselves, not when they quoted something.

However searching for partial words would be great. E.g. "bitcoin" should find "bitcoins". Or perhaps it should be an option too, for those cases where you don't want "ninja" to find "tryninja".


Let me know if you'd like some help with the unknown titles. I can give you a dump of post IDs and titles that could significantly reduce the number of posts you'd need to re-scrape.

[alani123]

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.