Ninjastic.space - BitcoinTalk Post/Address archive + API

[TryNinja]

I thought you had the more recent posts already

Nope. I only have your 'old posts' archive and the very most recent ones since the bot v2 was released around August 24th. I think I need the posts from around March 22th until that date.

[1714776687]

1714776687

[1714776687]

1714776687

[1714776687]

1714776687

[1714776687]

1714776687

[1714776687]

1714776687

[LoyceV]

I think I need the posts from around March 22th until that date.

I got you posts 54060000-55069999 (this file will self-destruct in 7 days). Does this work?

[TryNinja]

I got you posts 54060000-55049998 (this file will self-destruct in 7 days). Does this work?

Could you get me a few more (+1917 to be exact)?

54060000-55051915 (You can also round it up to whatever makes it easier for you)

It looks like they are already in this file. Thanks.

[LoyceV]

It looks like they are already in this file. Thanks.

You're right, I fixed the total count

[TheBeardedBaby]

Just a note,
The arrangement of the results of the address search in the mobile version of the website is OK expect for the users. See for yourself and search the address from my profile page.

[TryNinja]

You're right, I fixed the total count

Thank you! I just added most of the missing posts from this year. Now I have more archived posts then my current post count. (8,362 vs 8,314)

To compensate the good news, I learned a few new things about how databases work, but not before missing around ~400 posts from the last hour because of 'table locks'. This means that the bot may have missed a few notifications. I'll try to somehow scrape them later for the archive.

Just a note,
The arrangement of the results of the address search in the mobile version of the website is OK expect for the users.

Thanks. I think I fixed it.

[LoyceMobile]

I learned a few new things about how databases work, but not before missing around ~400 posts from the last hour because of 'table locks'. This means that the bot may have missed a few notifications. I'll try to somehow scrape them later for the archive.

If it's just 400, you can easily get them from my online post archive.

[TheBeardedBaby]

Thanks. I think I fixed it.

Confirm, it's fixed now, and it looks nice. Thank you

[TryNinja]

If it's just 400, you can easily get them from my online post archive.

All good now. It had missed 394 posts to be exact.

Would you consider also scraping the post's title and last board id for the next posts? That's the only thing that is always missing for my archive.

[LoyceV]

Would you consider also scraping the post's title and last board id for the next posts? That's the only thing that is always missing for my archive.

I don't want to change my storage format, it might break searches that cover old and new posts. I should have done this from the start though.

Bumping my own post with a quote from another topic:

https://bitcointalk.org/index.php?topic=5190513 returns 1.
https://bitcointalk.org/index.php?topic=5190513.0 returns 306.

Request: can you add the option to omit the post from the results? I'd like to see a list of headers only so I can quickly see who posted all this. It looks like a few users posted the same thing a couple hundred times, but it takes a lot of scrolling to check all posts.

I'm impressed by the performance! I know nothing about databases, but searching this much data in a few seconds is impressive.

[TryNinja]

Request: can you add the option to omit the post from the results? I'd like to see a list of headers only so I can quickly see who posted all this. It looks like a few users posted the same thing a couple hundred times, but it takes a lot of scrolling to check all posts.

I have added 2 new designs. There is now: normal, header only and compact.

Header only:


Compact:

I'm impressed by the performance! I know nothing about databases, but searching this much data in a few seconds is impressive.

You are not the only one.

[LoyceV]

I have added 2 new designs. There is now: normal, header only and compact.

Thanks! One more request: can you remove the empty line between lines in Compact mode?

[TryNinja]

can you remove the empty line between lines in Compact mode?

Does it look better now?

[LoyceV]

can you remove the empty line between lines in Compact mode?

Does it look better now?

Yes, it's good now

[TryNinja]

A few new changes since the last update:

-> Added to the topics page:

- List of users participating on a topic: who posted on a topic and how many times they did it.
- Normal, header-only and compact viewing mode for the list of posts.

-> Added to the search page:

- Normal, header-only and compact viewing mode for the list of posts (announced above).
- Search page URL params. E.g: http://ninjastic.space/search?author=TryNinja&board=24

-> Added to the user profiles:

- Top 5 favorite topics.
- Deleted posts and edited posts (within 5 mins of created).
- All time, 30 days and 7 days periods for the overview of posts per board.
- Graph of posts per week (last year).

If anyone has other ideas, please let me know.

Let's test something

Nice try, but there are many security measures already in place to prevent XSS attacks.

[hatshepsut93]

Let's test something

Nice try, but there are many security measures already in place to prevent XSS attacks.

I want to build smth with your API, do you escape HTML in the posts provided by it? Though it shouldn't matter that much, cause I should be escaping it anyway. Plus the forum escapes it too, so it should be pretty safe to use the provided HTML to display posts. Still I would like to test it more, but I'm afraid to trigger some sort of forum's automatic response if I'll just paste a ton of XSS payloads here.

[PrimeNumber7]

You will likely be prevented from posting anything malicious on the forum that might cause a scrapper to be caused to run arbitrary code. I have been prevented from posting non-malicous code of scripts I have written in the past, and I believe from SQL statements are also blocked.

If you are looking for vulnerabilities (and TryNinja has given you permission to do so), I believe a SQL injection attack has a higher probability of finding one.

[LoyceMobile]

I'm afraid to trigger some sort of forum's automatic response if I'll just paste a ton of XSS payloads here.

Chances are Cloudflare blocks certain posts, I haven't seen the forum itself restrict posts.

[TryNinja]

I want to build smth with your API,

Feel free to PM me if you need help with that. I already changed a few things on some of my API endpoints and will change a few others soon. I'll also create a page to show and explain all available endpoints.

do you escape HTML in the posts provided by it?

Yes. Here is how your post was saved: https://api.ninjastic.space/posts/55349523

The forum also blocks most of them through Cloudflare.

I'm afraid to trigger some sort of forum's automatic response if I'll just paste a ton of XSS payloads here.

I don't think you would get banned because of that, so no worries. You just get a Cloudflare error page and they block you from making the post.

Try writing this and clicking "Preview":

Code:

j4vascript:alert(1)

(replacing the 4 with an 'a')

Result:

Sorry, you have been blocked
You are unable to access bitcointalk.org

Scary message, but you can just go back and keep using the forum normally.

[PrimeNumber7]

I'm afraid to trigger some sort of forum's automatic response if I'll just paste a ton of XSS payloads here.

Chances are Cloudflare blocks certain posts, I haven't seen the forum itself restrict posts.

CF blocking certain posts is one and the same of the forum blocking posts. The forum is using CF as it's agent to block DDoS and other attacks. (the posts being blocked would fall under what could possibly be "other" attacks).