Bitcoin Forum
January 28, 2020, 09:40:33 AM *
News: Latest Bitcoin Core release: 0.19.0.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 [2]  All
  Print  
Author Topic: Additional data dumps?  (Read 833 times)
iasenko
Hero Member
*****
Offline Offline

Activity: 812
Merit: 1572


Zombie mode:ON! o_0


View Profile WWW
February 08, 2019, 08:37:25 AM
 #21

It's not a necrobump.
Can we have the modlog and seclog dumps instead everyone to scrape the data from the server?

Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1580204433
Hero Member
*
Offline Offline

Posts: 1580204433

View Profile Personal Message (Offline)

Ignore
1580204433
Reply with quote  #2

1580204433
Report to moderator
100bitcoin
Sr. Member
****
Offline Offline

Activity: 690
Merit: 343


View Profile
February 08, 2019, 12:15:11 PM
 #22

It's not a necrobump.
Can we have the modlog and seclog dumps instead everyone to scrape the data from the server?


Can one still able to scrape data from the server? I thought theymos prohibited it since bitcointalk.to started to scrape the whole forum.

iasenko
Hero Member
*****
Offline Offline

Activity: 812
Merit: 1572


Zombie mode:ON! o_0


View Profile WWW
February 08, 2019, 12:23:34 PM
 #23

Can one still able to scrape data from the server? I thought theymos prohibited it since bitcointalk.to started to scrape the whole forum.

Both LoyceV and Vod are doing it, also i've seen other users too, so I think there is no any prohibition, yet.
I think if those dumps are available for download directly from the forum, more people can benefit out of it and there will be less traffic to the server.

LoyceV
Legendary
*
Online Online

Activity: 1736
Merit: 5638


Most of loyce.club works again


View Profile WWW
February 08, 2019, 12:27:18 PM
 #24

Scraping works, but the current modlog covers only a limited time. It's not possible to get a complete overview of all banned users. From various sources, I have a list of 170k banned users now, but it's far from complete.

iasenko
Hero Member
*****
Offline Offline

Activity: 812
Merit: 1572


Zombie mode:ON! o_0


View Profile WWW
February 08, 2019, 12:44:52 PM
Last edit: February 08, 2019, 02:03:45 PM by iasenko
 #25

Scraping works, but the current modlog covers only a limited time. It's not possible to get a complete overview of all banned users. From various sources, I have a list of 170k banned users now, but it's far from complete.

Is there any place to have the modlog in raw format available? Even for the limited time. I want to check some things Smiley

LoyceV
Legendary
*
Online Online

Activity: 1736
Merit: 5638


Most of loyce.club works again


View Profile WWW
February 08, 2019, 05:02:00 PM
 #26

Is there any place to have the modlog in raw format available? Even for the limited time. I want to check some things Smiley
Do you mean older versions? I used archive.li and archive.org.

tranthidung
Hero Member
*****
Offline Offline

Activity: 700
Merit: 1213


Manager? See my profile page, please


View Profile WWW
January 09, 2020, 03:48:01 AM
Last edit: January 09, 2020, 05:54:43 AM by tranthidung
 #27

UID -> name, merit, potential activity, posts
 post ID -> topic ID, time, UID
 topic ID -> board ID, first post ID
 board ID -> board name
The new year, so I bump it to ask for additional data dump granted by theymos.

Besides these formats above, I ask for this one (for merit data):
Code:
time amount msg user_from user_to boardid
1516831941  1 2818066.msg28853325 35 877396 24
I already collected the boardid, so if the merit data has only one additional variable for board's ID (boardid), it will eliminate the need to scrap data (from LoyceV's help) each 6 months. Although I don't know the others need such variable in data dumps or not.

For some sorts of analyses like these:




▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄                  ▄▄▄   ▄▄▄▄▄        ▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄
 ▀████████████████▄  ████                 █████   ▀████▄    ▄████▀  ▄██████████████   ████████████▀  ▄█████████████▀  ▄█████████████▄
              ▀████  ████               ▄███▀███▄   ▀████▄▄████▀               ████   ████                ████                   ▀████
   ▄▄▄▄▄▄▄▄▄▄▄█████  ████              ████   ████    ▀██████▀      ██████████████▄   ████████████▀       ████       ▄▄▄▄▄▄▄▄▄▄▄▄████▀
   ██████████████▀   ████            ▄███▀     ▀███▄    ████        ████        ████  ████                ████       ██████████████▀
   ████              ████████████▀  ████   ██████████   ████        ████████████████  █████████████▀      ████       ████      ▀████▄
   ▀▀▀▀              ▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀  ▀▀▀▀        ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀        ▀▀▀▀       ▀▀▀▀        ▀▀▀▀▀

#1 CRYPTO CASINO & SPORTSBOOK
  WELCOME
BONUS
.INSTANT & FAST.
.TRANSACTION.....
.PROVABLY FAIR.
......& SECURE......
.24/7 CUSTOMER.
............SUPPORT.
BTC      |      ETH      |      LTC      |      XRP      |      XMR      |      BNB      |     more
tranthidung
Hero Member
*****
Offline Offline

Activity: 700
Merit: 1213


Manager? See my profile page, please


View Profile WWW
January 11, 2020, 03:36:40 AM
Merited by LoyceV (1), PrimeNumber7 (1)
 #28

From the reply of admin yesterday, I think now it is a very good time to think of a consistent format for the forum's data dumps. Each dataset has different variables inside, but I think all of them should be connected with only common variable (at least one variable) - userid.

Username, no matter it is username or display name or both will result in differences when connecting different datasets dumped by the forum.

For additional data dumps, it is not the priority and I am not in a position to ask for it too much, but for current data formats, a small adjustment: from username to userid will be good.

LoyceV asked for this change too: https://bitcointalk.org/index.php?topic=5104467.msg53551686#msg53551686




▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄                  ▄▄▄   ▄▄▄▄▄        ▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄
 ▀████████████████▄  ████                 █████   ▀████▄    ▄████▀  ▄██████████████   ████████████▀  ▄█████████████▀  ▄█████████████▄
              ▀████  ████               ▄███▀███▄   ▀████▄▄████▀               ████   ████                ████                   ▀████
   ▄▄▄▄▄▄▄▄▄▄▄█████  ████              ████   ████    ▀██████▀      ██████████████▄   ████████████▀       ████       ▄▄▄▄▄▄▄▄▄▄▄▄████▀
   ██████████████▀   ████            ▄███▀     ▀███▄    ████        ████        ████  ████                ████       ██████████████▀
   ████              ████████████▀  ████   ██████████   ████        ████████████████  █████████████▀      ████       ████      ▀████▄
   ▀▀▀▀              ▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀  ▀▀▀▀        ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀        ▀▀▀▀       ▀▀▀▀        ▀▀▀▀▀

#1 CRYPTO CASINO & SPORTSBOOK
  WELCOME
BONUS
.INSTANT & FAST.
.TRANSACTION.....
.PROVABLY FAIR.
......& SECURE......
.24/7 CUSTOMER.
............SUPPORT.
BTC      |      ETH      |      LTC      |      XRP      |      XMR      |      BNB      |     more
Pages: « 1 [2]  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!