suchmoon
Legendary
Offline
Activity: 3808
Merit: 9028
https://bpip.org
|
|
September 09, 2020, 10:48:31 PM |
|
I just realized you're not showing create/edit timestamps for posts. Do you have them? I can give you that too.
I don't. That would also be great. Can you send me only the date they were posted? (no edit timestamps, etc...). Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.
|
|
|
|
TheBeardedBaby
Legendary
Offline
Activity: 2240
Merit: 3150
₿uy / $ell ..oeleo ;(
|
|
September 10, 2020, 07:32:50 AM |
|
Hello again. I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions. Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram. In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2968
Merit: 7397
|
|
September 10, 2020, 07:24:39 PM |
|
Hello again. I may repeat myself but this is one of my favorite projects/tools ever. I have a few more suggestions. Since this tool will be mainly used by people like me, hunting for spammers, alt accounts, etc. It would be very convenient to have a search of social media links cross posted by different users, like it is now the search for the addresses. You only need Twitter, Facebook, and Telegram. In addition, maybe make like a profile page for each user with just links to all addresses he posted and all the social media links.
Thank you! That's a good idea. I'm indeed thinking about making a user's "profile page" where you will be able to see some stats about anyone's. Possibly stuff like where he posts and when he posts, his addresses, social links, possible connections, etc... all of that will be easier with the changes I'm making. Feel free to give more suggestions!
The number of posts without a title - shown as "(Unknown Title)" - has dropped from 42 millions to just 8k! Chances are that every post you check now has its title.
|
|
|
|
Aveatrex
|
|
September 10, 2020, 08:57:08 PM |
|
I have a few endpoints which will return the data you need. For example: GET: /posts/55141939 { "id": "1c25054c-b1b8-41eb-8c66-697c8b697179", "post_id": 55142446, "topic_id": 5273824, "title": "Re: Ninjastic.space - BitcoinTalk Post/Address archive + API", "author": "Aveatrex", "author_uid": 950474, "content": "Amazing! I see that you are making use of Reactjs, way to go <img src=\"https://bitcointalk.org/Smileys/default/smiley.gif\" alt=\"Smiley\" border=\"0\"> . I'm interested in your RESTful API, can you elaborate on how it works? ", "date": "2020-09-06T12:45:48.000Z", "boards": [ "Other", "Meta" ], "archive": false } How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3808
Merit: 9028
https://bpip.org
|
|
September 10, 2020, 09:18:39 PM |
|
How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it It's a simple HTTP request: https://api.ninjastic.space/posts/55141939Status code 200 means the post exists and you can parse JSON from the response. 404 means not found, etc.
|
|
|
|
|
FatFork
Legendary
Offline
Activity: 1736
Merit: 2644
Top Crypto Casino
|
|
September 10, 2020, 09:40:09 PM |
|
How to get access to these end points?/ Can you give me access to one of them? I will be messing around with data a bit and try to integrate it into my bot. I'm not too familiar with RESTful APIs so please include as much details as possible about it You can find a useful online API testing tool here: https://reqbin.com/Just send a request to one of the ninjastic.space endpoints (like these examples TryNinja gave) and watch the responses. You can also find code samples for popular programming languages. Try it. RESTful APIs are quite simple and easy to implement.
|
|
|
|
Vod
Legendary
Offline
Activity: 3836
Merit: 3130
Licking my boob since 1970
|
|
September 10, 2020, 10:15:27 PM |
|
Nice job. I have some ideas that could help in scam busting. I left you merit, but even better than that, I've left you my trust. - The post archive is still incomplete as many posts from this year are missing. It has, however, a lot more posts than its previous version: 42,785,512 posts! Mostly from the previous years. (thanks to @LoyceV for his oldposts archive).
Any chance you can recover other posts from the Internet Archive or one of the bitcointalk clone sites? Some scammers have deleted hundreds of posts of illegal activities.
|
I post for interest - not signature spam. https://vod.fan - fast/free image sharing - coming Oct! Will Theymos finish his $100,000,000 forum before this one shuts down?
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2968
Merit: 7397
|
|
September 10, 2020, 11:22:19 PM |
|
Nice job. I have some ideas that could help in scam busting. I left you merit, but even better than that, I've left you my trust.
Thanks, Vod. Let me know and I'll implement them if possible. Any chance you can recover other posts from the Internet Archive or one of the bitcointalk clone sites? Some scammers have deleted hundreds of posts of illegal activities.
It's technically possible, but I'm not sure how hard that would be. I'm priorizing scraping all the live posts that are missing from the database. When everything is working and most features are done, I may think about doing that.
|
|
|
|
Aveatrex
|
|
September 11, 2020, 01:11:33 PM Last edit: September 11, 2020, 03:44:37 PM by Aveatrex |
|
Oh if it's just a simple HTTP request then I am familiar with that I thought it was some kind of special interface with mandatory authorization via an api key.... You can use them as you wish for now. But I would appreciate if you consulted me before doing many requests or implementing it in any kind of project. This way we can optimize things to keep the server working without too much workload.
I'm still working on the bot right now, if I deem it useful enough to release it publicly one day, I will definitely let you know before so we can optimize it. I could also maybe implement some kind of authentication in the future.
I think you should. Otherwise you are just asking for a DoS attack
|
|
|
|
Vod
Legendary
Offline
Activity: 3836
Merit: 3130
Licking my boob since 1970
|
|
September 11, 2020, 02:30:34 PM |
|
Let me know and I'll implement them if possible.
How about a blockchain search? 4 values - min,max number of bitcoins transferred. Start,end date to search. For example - I want to search for any transfers between 450-500 bitcoin between Sep 1 and Oct 31 2015. If you integrate the crypto price in the search, I could also search for transfers of $40-$50 for example.
|
I post for interest - not signature spam. https://vod.fan - fast/free image sharing - coming Oct! Will Theymos finish his $100,000,000 forum before this one shuts down?
|
|
|
FatFork
Legendary
Offline
Activity: 1736
Merit: 2644
Top Crypto Casino
|
|
September 11, 2020, 03:26:24 PM |
|
I could also maybe implement some kind of authentication in the future.
I think you should. Otherwise you are just asking for a DoS attack Lack of authentication doesn't mean there are no other DDoS mitigation measures implemented. Just saying... Btw, you messed the quotes up, that was TryNinja's quote.
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3444
Merit: 17471
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
September 11, 2020, 04:07:13 PM |
|
How about a blockchain search?
4 values - min,max number of bitcoins transferred. Start,end date to search. Blockchair has this data. It might take more disk space than all Bitcointalk posts. I have several topics based on it already (but I don't do databases):
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3808
Merit: 9028
https://bpip.org
|
|
September 11, 2020, 06:09:05 PM |
|
Sure. I need to double-check a few things first. At one point I had some issues with timezones so I'll verify if I need to make any adjustments. It will all be in UTC once it's ready.
Quick update on this: the timezone mess is messier than I thought so it will take some time to sort through it. Basically some posts got scraped with +0200 instead of UTC but I don't know which ones, so I'll probably need to scrape some "checkpoints" and find posts between them that are "time traveling" (e.g. created later than the next post).
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2968
Merit: 7397
|
|
September 11, 2020, 09:29:39 PM Last edit: May 14, 2023, 02:46:28 AM by TryNinja |
|
Quick update on this: the timezone mess is messier than I thought so it will take some time to sort through it. Basically some posts got scraped with +0200 instead of UTC but I don't know which ones, so I'll probably need to scrape some "checkpoints" and find posts between them that are "time traveling" (e.g. created later than the next post).
Would you also have the boards of the posts? The old archive also doesn't contain them. I finished setting up the new database and already have some cool new features to announce. But I'm waiting for the timestamps and potentially the boards so I can index the data. Here is a sneak peek of one of them (WIP): https://talkimg.com/images/2023/05/14/blobf707e32c89df6b5f.png
|
|
|
|
Vod
Legendary
Offline
Activity: 3836
Merit: 3130
Licking my boob since 1970
|
|
September 11, 2020, 11:34:05 PM |
|
One of the things I had wanted to with BPIP was breakdown posts per hour per section of the forum. People could see when the best time to post would be. Obviously this can be done easily using the infrastructure you have set up.
|
I post for interest - not signature spam. https://vod.fan - fast/free image sharing - coming Oct! Will Theymos finish his $100,000,000 forum before this one shuts down?
|
|
|
suchmoon
Legendary
Offline
Activity: 3808
Merit: 9028
https://bpip.org
|
|
September 11, 2020, 11:50:25 PM |
|
Would you also have the boards of the posts? The old archive also doesn't contain them. I do.
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2968
Merit: 7397
|
|
September 11, 2020, 11:57:04 PM |
|
I do.
Would you be able to send them to me in this format (along with the post date)? postid, date, boards 55179038, "2020-04-13 12:03:00", "{Other, Meta}" One of the things I had wanted to with BPIP was breakdown posts per hour per section of the forum. People could see when the best time to post would be. Obviously this can be done easily using the infrastructure you have set up.
That sounds like an easy one. I will add it!
|
|
|
|
suchmoon
Legendary
Offline
Activity: 3808
Merit: 9028
https://bpip.org
|
|
September 12, 2020, 12:11:00 AM |
|
I do.
Would you be able to send them to me in this format (along with the post date)? postid, date, boards 55179038, "2020-04-13 12:03:00", "{Other, Meta}" I have only the ID of the direct "parent" board (24 in your example). You would need to scrape the board hierarchy if you need the full path and the names of the boards. Also that timestamp stuff will take me a day or two so if you want boards sooner - let me know, I can send it separately.
|
|
|
|
TryNinja (OP)
Legendary
Offline
Activity: 2968
Merit: 7397
|
|
September 12, 2020, 12:14:27 AM |
|
I have only the ID of the direct "parent" board (24 in your example). You would need to scrape the board hierarchy if you need the full path and the names of the boards.
Also that timestamp stuff will take me a day or two so if you want boards sooner - let me know, I can send it separately.
In this case, you can send me the boards right away so I can figure out how to do that. Thanks!
|
|
|
|
|