Ninjastic.space - BitcoinTalk Post/Address archive + API

[Stalker22]

Of course, it would be great if such a function worked not only on wallet addresses, but also on social media addresses (twitter, facebook, telegram, etc.)

I don't think it's that simple. There is no uniform format for social media addresses. For example, someone can post only their profile name as opposed to the full url to the profile page. Searching for such data is no different than searching for any other words.

[TryNinja]

New Update!

- Post edit history.

If you go to a post's page, you can see the original unedited post and its edited version (edited up to 5 minutes later). Also, you will be notified by the bot if someone edits the post and mentions you/quotes you later (again, up to 5 min).

Example: https://ninjastic.space/post/55146574


It is possible to get the bounty profiles (at least some of them). I also save the post's board, so I can filter them to the "Bounties" board. I will see if I can take a look at this when I have more time.

[Stalker22]

So your statement that searching for addresses is no different from searching for ordinary words is incorrect.

Not really. As TryNinja replied, it is probably possible to get some bounty profiles but the results will not be complete or completely accurate.
Anyone who has ever searched for alt accounts of ban evaders and bounty cheaters knows that it requires a bit of creativity as well. 

[LoyceV]

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

Not exactly. Depends on the database schema. I'm already working on this, but I still didn't find a good solution that is fast and effective.

Maybe you can include userIDs in the data dump? Or make a separate data dump for "userID,addresses".

When I search for a simple text ("test"), I quickly get 200 results. However, when I search for a rare text (such as "1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD") without Author or Topic ID, the Search button keeps spinning for what seems like indefinitely. I assume it's still searching, but this might cause a problem with server load at some point.

- Post edit history.

If you go to a post's page, you can see the original unedited post and its edited version (edited up to 5 minutes later).

This is cool Maybe make it 10 minutes though, that's the time limit after which the forum will show a post as edited.

[TheBeardedBaby]

This is amazing tool, I love it.
Sorry, No more merit to give... and theymos is silent on my Merit Source Application

I'll add as small suggestion.

I don't have enough free time, and sometimes I do research/post from the phone, while in bed, so having a bbcode format of the results from the search to copy from will be a very handy tool at least for me.

[mole0815]

I had already mentioned your old website in the German section. If it is desired I can translate the big new update and advertise it in our section.
Your effort and invested time should help as many users as possible

[suchmoon]

Let me know if you'd like some help with the unknown titles. I can give you a dump of post IDs and titles that could significantly reduce the number of posts you'd need to re-scrape.

That would be great.

https://bpip.org/titles_20200907.zip

Format is CSV because it's relatively compact for this volume of data. The zip file is ~800MB, uncompressed size ~3GB. It'd be possible to make it smaller by grouping the titles together ("Title",<list of post IDs with that title>) since most replies typically have the same title but it would probably be a pain to import. Let me know if you prefer a different format.

Sample (note the double quote escapes):

Code:

"PostId","Title"
28,Welcome to the new Bitcoin forum!
29,Repost: Bitcoin Maturation
30,Repost: Request: Make this anonymous?
31,Re: Repost: Bitcoin Maturation
32,Re: Repost: Request: Make this anonymous?
33,Repost: How anonymous are bitcoins?
34,Re: Repost: How anonymous are bitcoins?
36,Repost: Linux/UNIX compile
37,Re: Repost: Linux/UNIX compile
38,[OLD THREAD] Bitcoin version 0.2 development status
40,A few suggestions
41,Re: A few suggestions
42,Re: A few suggestions
43,Questions about Bitcoin
44,Re: A few suggestions
45,Re: A few suggestions
46,Re: Questions about Bitcoin
47,Re: A few suggestions
48,Re: Questions about Bitcoin
49,Re: Questions about Bitcoin
50,Re: A few suggestions
51,Re: A few suggestions
52,Re: A few suggestions
53,Break on the supply's increase
54,Re: A few suggestions
55,Re: A few suggestions
56,Re: Break on the supply's increase
57,Re: A few suggestions
58,Re: A few suggestions
59,Re: A few suggestions
60,Re: A few suggestions
61,Re: A few suggestions
62,Re: A few suggestions
63,Re: A few suggestions
64,Re: A few suggestions
65,"New Exchange Service: ""BTC 2 PSC"""
66,Re: A few suggestions
67,Re: A few suggestions
68,Re: A few suggestions
69,Re: A few suggestions
70,Re: A few suggestions
71,Re: A few suggestions
72,Re: A few suggestions
73,Bitcoin 0.2 released!
74,Re: Bitcoin 0.2 released!
75,FreeBSD build patch
76,Re: A few suggestions
77,Re: A few suggestions
78,Re: A few suggestions
79,Re: A few suggestions
81,Re: A few suggestions
82,"Re: New Exchange Service: ""BTC 2 PSC"""
83,Is my second Transaction working correctly? +Transfer Question
84,Re: Is my second Transaction working correctly? +Transfer Question
85,Re: Is my second Transaction working correctly? +Transfer Question
86,Re: Is my second Transaction working correctly? +Transfer Question
87,"64bit support"
88,Re: Bitcoin 0.2 released!
90,Web UI ideas
91,Re: Web UI ideas
92,Re: Web UI ideas
93,Re: Web UI ideas
94,Re: Web UI ideas
95,"Re: New Exchange Service: ""BTC 2 PSC"""
96,"Re: New Exchange Service: ""BTC 2 PSC"""
97,Re: 64bit support
100,New exchange (Bitcoin Market)

[PrimeNumber7]

Since it's all in a DB, it would be possible to associate a user with all of the addresses they've posted, no? I see the opposite being available and I can't help but think of also searching by user.

Not exactly. Depends on the database schema. I'm already working on this, but I still didn't find a good solution that is fast and effective.

Your 'posts' table should be as follows:
postID (P) - int
time
UID - int
posttext - str
addressposted - bool (this is optional, but may make searching for addresses easier)
(any other information that does not appear in your DB elsewhere)

You will have an 'addresses' table as follows:
addressID (P) - int (this is an arbitrary number)
address - str
(any other information that does not already appear elsewhere in your DB that you want to keep track of, such as address type, or coin)

postedaddresses table:
ID (P) - int (arbitrary number)
postID - int
addressID - int

[TryNinja]

Maybe you can include userIDs in the data dump? Or make a separate data dump for "userID,addresses".

I'll make some changes in the way I store the addresses which will probably allow me to get this data.

When I search for a simple text ("test"), I quickly get 200 results. However, when I search for a rare text (such as "1NXYoJ5xU91Jp83XfVMHwwTUyZFK64BoAD") without Author or Topic ID, the Search button keeps spinning for what seems like indefinitely. I assume it's still searching, but this might cause a problem with server load at some point.

You're right. The database should have a timeout for long queries but it wasn't working. It is now.

You also made me realize that the query was taking too long in some cases when there were little results. For example, pretty much any address search would take forever. Now you can (mostly) search them and it is way faster.

This is cool Maybe make it 10 minutes though, that's the time limit after which the forum will show a post as edited.

I'll think about it. I'm also thinking about making more checks, like another one after 1 or 2 hours.

I don't have enough free time, and sometimes I do research/post from the phone, while in bed, so having a bbcode format of the results from the search to copy from will be a very handy tool at least for me.

Will do! Thanks!

I had already mentioned your old website in the German section. If it is desired I can translate the big new update and advertise it in our section.
Your effort and invested time should help as many users as possible

Thank you! I really appreciate that.

~

You gave me some ideas. I won't do it exactly like this since I would need to make some big chances, but I may have found a possible solution.


Edit: Just saw the bot/website received a big donation. Thanks however did it! That pretty much pays the bot for a whole year.

[Lafu]

Awesome nice work and useful tool buddy 

Maybe you can add something in the search post section .
Would be helpful if there would be a Ignore field for Usernames so if you search for some content that the posts from this User dont show up in the results.

Dont know if that is possible and easy to code into the webpage and bot.

 

[bitmover]

hello TryNinja

I tried to view your webpage but I couldn't pass the loading screen using Edge (a chromium browser) and firefox. I am using adblock and privacy badger.




Edit:
I clicked Search Post and Addresses and it worked. Just the first screen that is spinnig forever.

[TryNinja]

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

[TheBeardedBaby]

The server crashed earlier today for ~1 hour while processing a big amount of data. It's working again.

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

Thanks, i hate breaking things up.

[TryNinja]

Sorry if i broke it, i thought to use it for the experiment i made earlier today, the first two address searches were ok but on the third one stuck...

It's ok, you didn't break anything (I think).

I just messed up when importing suchmoon's file with the post titles. But it's now in progress. You just made your searches at the wrong time.

[Pmalek]

This is getting bookmarked and definitely translated into my local forum for everyone to see when I get around to it.

I found the Addresses feature very interesting so I decided to give it a try. 
I did a search on "site:bitcointalk.org avoiding ban" and found this thread by YOSHIE.
I copied the ETH address and ran a search. The results were instant! Very surprised by the speed. It showed all 5 users who posted that address and links to their posts.



I have 2 questions though.

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?
2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

[TryNinja]

1. Why is Irfan156's name in yellow and the rest in green? What do the colors mean?

Each user gets a random color so you can scroll up through the list and differentiate them easier. It doesn't mean anything.

2. Why does it say unknown title above the posts? Did a search on myself and found that posts from July 2020 and older show unknown title.

Because I don't know their titles.

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

[Pmalek]

Snip

Cool, thanks for the clarification.
I just translated your topic and posted it in our local Croatian forum. We have a few scam busters and private investigators who could find it useful. Although the majority speaks English, it's still a plus to have a reference in your own language.   

[suchmoon]

I'm currently running a script to get most of them. For example, satoshi's first post already has his title showing up (it didn't before): https://ninjastic.space/post/28

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.


I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

[LoyceV]

I'm also thinking about making more checks, like another one after 1 or 2 hours.

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

[TryNinja]

I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.

I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...).

I tried searching for invalid (non-numeric) topic ID and it seems to spin forever. If combined with user it seems to return all posts for that user.

I'll fix that when I finish changing everything to search the new database layer (more below).

I assume you're scraping the page with the post to check for updates, right? That means you're downloading up to 19 other posts at the same time. If you download them anyway, can you add those posts to your "check for editing" feature?

Yes! You are absolutely right. I'll do that.


I found out that updating a lot of data in a database full of indices is a pain in the ass. The database needs to re-create every index, which takes  a lot of time (mainly due to the reverse index that allows the "content" search).

I'm currently setting up a second database (using Elasticsearch) which will make searching and data analysing a shit-ton easier and faster. Because of that, the "search post" page will be unavailable for some time in a few hours.