Ninjastic.space - BitcoinTalk Post/Address archive + API

[suchmoon]

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.

[1715509924]

1715509924

[1715509924]

1715509924

[1715509924]

1715509924

[1715509924]

1715509924

[1715509924]

1715509924

[1715509924]

1715509924

[TheBeardedBaby]

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

[TryNinja]

Hello again.
I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions.
Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram.
In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.

Thank you!

That's a good idea. I'm indeed thinking about making a user's "profile page" where you will be able to see some stats about anyone's. Possibly stuff like where he posts and when he posts, his addresses, social links, possible connections, etc... all of that will be easier with the changes I'm making. Feel free to give more suggestions!


The number of posts without a title - shown as "(Unknown Title)" - has dropped from 42 millions to just 8k! Chances are that every post you check now has its title.

[Aveatrex]

I have a few endpoints which will return the data you need. For example:

GET: /posts/55141939

Code:

{
  "id": "1c25054c-b1b8-41eb-8c66-697c8b697179",
  "post_id": 55142446,
  "topic_id": 5273824,
  "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API",
  "author": "Aveatrex",
  "author_uid": 950474,
  "content": "Amazing! I see that you are making use of Reactjs, way to go <img src=\"https://bitcointalk.org/Smileys/default/smiley.gif\" alt=\"Smiley\" border=\"0\"> .  I'm interested in your RESTful API, can you elaborate on how it works? ",
  "date": "2020-09-06T12:45:48.000Z",
  "boards": [
    "Other",
    "Meta"
  ],
  "archive": false
}

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

[suchmoon]

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

It's a simple HTTP request:

https://api.ninjastic.space/posts/55141939

Status code 200 means the post exists and you can parse JSON from the response. 404 means not found, etc.

[TryNinja]

This ^

You can use them as you wish for now. But I would appreciate if you consulted me before doing many requests or implementing it in any kind of project. This way we can optimize things to keep the server working without too much workload. I could also maybe implement some kind of authentication in the future.

Some endpoints:

Post: https://api.ninjastic.space/posts/55141939
Posts: https://api.ninjastic.space/posts/46250414,50646799,55163653,50966531
Posts on topic: https://api.ninjastic.space/posts/topic/5273824

Address: https://api.ninjastic.space/addresses/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L
Address authors: https://api.ninjastic.space/addresses/1NinjabXd5znM5zgTcmxDVzH4w3nbaY16L/authors
Addresses on post: https://api.ninjastic.space/addresses/post/50966531

I will probably create a page to document them better.

[FatFork]

How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it

You can find a useful online API testing tool here: https://reqbin.com/
Just send a request to one of the ninjastic.space endpoints (like these examples TryNinja gave) and watch the responses. You can also find code samples for popular programming languages. Try it. RESTful APIs are quite simple and easy to implement.

[Vod]

The new ninjastic.space website is out! Remade from scratch.

Nice job.  I have some ideas that could help in scam busting.   I left you merit, but even better than that, I've left you my trust.

- The post archive is still incomplete as many posts from this year are missing. It has, however, a lot more posts than its previous version: 42,785,512 posts! Mostly from the previous years. (thanks to @LoyceV for his oldposts archive).

Any chance you can recover other posts from the Internet Archive or one of the bitcointalk clone sites?     Some scammers have deleted hundreds of posts of illegal activities.

[TryNinja]

Nice job.  I have some ideas that could help in scam busting.   I left you merit, but even better than that, I've left you my trust.

Thanks, Vod.

Let me know and I'll implement them if possible.

Any chance you can recover other posts from the Internet Archive or one of the bitcointalk clone sites?     Some scammers have deleted hundreds of posts of illegal activities.

It's technically possible, but I'm not sure how hard that would be. I'm priorizing scraping all the live posts that are missing from the database. When everything is working and most features are done, I may think about doing that.

[Aveatrex]

It's a simple HTTP request:

https://api.ninjastic.space/posts/55141939

Status code 200 means the post exists and you can parse JSON from the response. 404 means not found, etc.

Oh if it's just a simple HTTP request then I am familiar with that   I thought it was some kind of special interface with mandatory authorization via an api key....

You can use them as you wish for now. But I would appreciate if you consulted me before doing many requests or implementing it in any kind of project. This way we can optimize things to keep the server working without too much workload.

I'm still working on the bot right now, if I deem it useful enough to release it publicly one day, I will definitely let you know before so we can optimize it.

I could also maybe implement some kind of authentication in the future.

I think you should. Otherwise you are just asking for a DoS attack

[Vod]

Let me know and I'll implement them if possible.

How about a blockchain search?

4 values - min,max number of bitcoins transferred.   Start,end date to search.
For example - I want to search for any transfers between 450-500 bitcoin between Sep 1 and Oct 31 2015.

If you integrate the crypto price in the search, I could also search for transfers of $40-$50 for example.

[FatFork]

I could also maybe implement some kind of authentication in the future.

I think you should. Otherwise you are just asking for a DoS attack

Lack of authentication doesn't mean there are no other DDoS mitigation measures implemented. Just saying...

Btw, you messed the quotes up, that was TryNinja's quote.

[LoyceV]

How about a blockchain search?

4 values - min,max number of bitcoins transferred.   Start,end date to search.

Blockchair has this data. It might take more disk space than all Bitcointalk posts.

I have several topics based on it already (but I don't do databases):

Related topics
Bitcoin block data available in CSV format
List of all Bitcoin addresses with a balance
List of all Bitcoin addresses ever used

[suchmoon]

Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.

Quick update on this: the timezone mess is messier than I thought so it will take some time to sort through it. Basically some posts got scraped with +0200 instead of UTC but I don't know which ones, so I'll probably need to scrape some "checkpoints" and find posts between them that are "time traveling" (e.g. created later than the next post).

[TryNinja]

Quick update on this: the timezone mess is messier than I thought so it will take some time to sort through it. Basically some posts got scraped with +0200 instead of UTC but I don't know which ones, so I'll probably need to scrape some "checkpoints" and find posts between them that are "time traveling" (e.g. created later than the next post).

Would you also have the boards of the posts? The old archive also doesn't contain them.

I finished setting up the new database and already have some cool new features to announce. But I'm waiting for the timestamps and potentially the boards so I can index the data.

Here is a sneak peek of one of them (WIP): https://talkimg.com/images/2023/05/14/blobf707e32c89df6b5f.png

[Vod]

One of the things I had wanted to with BPIP was breakdown posts per hour per section of the forum.  People could see when the best time to post would be. Obviously this can be done easily using the infrastructure you have set up. 

[suchmoon]

Would you also have the boards of the posts? The old archive also doesn't contain them.

I do.

[TryNinja]

I do.

Would you be able to send them to me in this format (along with the post date)?

Code:

postid, date, boards
55179038, "2020-04-13 12:03:00", "{Other, Meta}"

One of the things I had wanted to with BPIP was breakdown posts per hour per section of the forum.  People could see when the best time to post would be. Obviously this can be done easily using the infrastructure you have set up.  

That sounds like an easy one. I will add it!

[suchmoon]

I do.

Would you be able to send them to me in this format (along with the post date)?

Code:

postid, date, boards
55179038, "2020-04-13 12:03:00", "{Other, Meta}"

I have only the ID of the direct "parent" board (24 in your example). You would need to scrape the board hierarchy if you need the full path and the names of the boards.

Also that timestamp stuff will take me a day or two so if you want boards sooner - let me know, I can send it separately.

[TryNinja]

I have only the ID of the direct "parent" board (24 in your example). You would need to scrape the board hierarchy if you need the full path and the names of the boards.

Also that timestamp stuff will take me a day or two so if you want boards sooner - let me know, I can send it separately.

In this case, you can send me the boards right away so I can figure out how to do that.

Thanks!