LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16584
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
May 19, 2020, 12:02:53 PM Last edit: September 08, 2023, 08:40:29 AM by LoyceV Merited by Welsh (4), actmyname (3), malevolent (2), DdmrDdmr (2), marlboroza (2), babo (1), Daniel91 (1), SFR10 (1), Koal-84 (1), hosseinimr93 (1), tranthidung (1), Bthd (1), TheBeardedBaby (1), morvillz7z (1), DireWolfM14 (1), Bitcoin_Arena (1), joker_josue (1), Nellayar (1) |
|
Bitcointalk's Security log only shows data from the last 30 days. At actmyname's request, I've just started keeping longer records. I expected to be done quickly, but sorting out archived data and converting it back to the same format turned out to be a lot of work and took several days.LoyceV's seclog logSee loyce.club/seclog/! (Warning: current size 168 MB, Firefox works better than Chrome) Or loyce.club/seclog/index.html.gz (Recommended, currently 18 MB) Machine readable: seclog.txt (32 MB) or seclog.txt.gz (7.3 MB) This contains all data I could find from seclog, archived posts and data provided by a user. I don't think this has much discussion value, I just leave it here for whoever can use it. IncompleteI show what I have, but this data is not complete because I started scraping 10 years too late. From now on it should include all new seclog-entries. UpdatesOnce a day. This isn't meant to be real-time. For more recent data, see seclog. BugsPlease post if there is any problem with the data. No spamPlease
See [overview] LoyceV's useful data on Bitcointalk for more of my forum-related topics
|
|
|
|
|
|
|
|
|
Bitcoin addresses contain a checksum, so it is very unlikely that mistyping an address will cause you to lose money.
|
|
|
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction.
|
SFR10
Legendary
Offline
Activity: 2996
Merit: 3416
Crypto Swap Exchange
|
|
May 19, 2020, 02:17:46 PM |
|
LoyceV's seclog logSee loyce.club/seclog/! (Warning: current size 115 MB) ~Snipped~ BugsPlease post if there is any problem with the data. Not sure if it's my internet speed or it really loads slowly [it took me a very long time just to be able to load the data for the previous day and scroll down without having lags]... - Highlighting a certain part seems to contribute to its lag.
- After a certain time, it stops loading the data (I waited for 5 minutes but it doesn't load any data past may 16th, on my side).
- I was able to replicate this issue twice (tested in Chrome).
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16584
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
May 19, 2020, 02:26:14 PM |
|
(tested in Chrome) From my Trust list viewer, I know Firefox can handles large HTML pages better than Chrome. The page has 638000+ entries, browsers don't really like it, but if I'd make it a text file the links won't work. I'll add a compressed file in a bit, it might work better from local storage.
|
|
|
|
Welsh
Staff
Legendary
Offline
Activity: 3262
Merit: 4110
|
|
May 19, 2020, 02:28:09 PM |
|
Not sure if it's my internet speed or it really loads slowly [it took me a very long time just to be able to load the data for the previous day and scroll down without having lags]... - Highlighting a certain part seems to contribute to its lag.
- After a certain time, it stops loading the data (I waited for 5 minutes but it doesn't load any data past may 16th, on my side).
- I was able to replicate this issue twice (tested in Chrome).
Its slow because as far as I can tell it continuously loads the data on the page. However, I made it to November 2019 fairly quickly, and the slow loading is partly because of the way its designed to load data, your client, and connection speed. As LoyceV stated, Firefox handles it slightly better, and that's what I tested it on. I'll add a compressed file in a bit, it might work better from local storage.
Is it worth considering implementing pagination with the option of displaying all data on one singular page if desired?
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16584
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
May 19, 2020, 03:01:33 PM |
|
Is it worth considering implementing pagination with the option of displaying all data on one singular page if desired? I kinda expect the people who use this, to want to search the whole thing at once: wait for it to load, then CTRL-F and open links in a New Tab. I've added a compressed file. If it's slow after downloading, you have your own computer to blame
|
|
|
|
coinlocket$
Legendary
Offline
Activity: 2366
Merit: 1512
#1 VIP Crypto Casino
|
|
May 19, 2020, 03:12:18 PM |
|
The file is way too big, I've tried to use CTRL F or scrolling up and down but got a white page for loading. I have a 200MB connection that should be faster.
What about splitting the page in alphabetic order?
|
|
|
|
PrimeNumber7
Copper Member
Legendary
Offline
Activity: 1624
Merit: 1899
Amazon Prime Member #7
|
|
May 19, 2020, 03:26:51 PM |
|
This is probably too large for most web browsers. I found this stack overflow question where someone claims that browsers tend to die after loading a 60 MB text file. I think a better solution might be to use a search function that would allow for someone to search if/when a username had a security event. Sending 100+ MB of data to the internet is also expensive. Sending a 100 MB file to the internet will cost about 0.9 cents, or US$0.009 and this can add up if you send this file many times.
|
|
|
|
SFR10
Legendary
Offline
Activity: 2996
Merit: 3416
Crypto Swap Exchange
|
|
May 19, 2020, 04:43:26 PM |
|
I know Firefox can handles large HTML pages better than Chrome.
As LoyceV stated, Firefox handles it slightly better, and that's what I tested it on.
Just tried testing it on Firefox and the difference was night and day [quite responsive]. I've added a compressed file. If it's slow after downloading, you have your own computer to blame I have a beefy computer but the improvement wasn't that significant in Chrome - Perhaps as you said earlier, it has to do with browser limitations.
I disabled "hardware acceleration" in Chrome and noticed some improvements...
|
|
|
|
hosseinimr93
Legendary
Online
Activity: 2380
Merit: 5235
|
|
May 19, 2020, 07:43:29 PM Last edit: May 19, 2020, 07:56:44 PM by hosseinimr93 |
|
Why don't you separate it into different months? Then there will be no loading problem. Perhaps, you are trying to make it easy to find a certain user by using CTRL+F? But we have BPIP for that. Also, for that purpose, you may consider creating a separate seclog page for every user. (I don't know how time consuming creating that is.)
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
ChuckBuck
|
|
May 19, 2020, 08:53:00 PM |
|
Also, for that purpose, you may consider creating a separate seclog page for every user. (I don't know how time consuming creating that is.)
LoyceV saved simply the entire Bitcointalk seclog for a longer period of time. Meanwhile, BPIP only stores the time when users change passwords, other data seems to be ignored. The most specific example, the moment the user wakes up, BPIP does not save it. This data is really important and necessary in investigations. Why create a separate seclog page for each user? It is not really necessary What about splitting the page in alphabetic order?
No, it seems to be a bad idea. Sorting in real time would be better. You won't want to find a guy at the bottom of the alphabet who changed his password yesterday But add some sort options looks good to search if you want. Anyway, Ctrl + F is the best
|
|
|
|
hosseinimr93
Legendary
Online
Activity: 2380
Merit: 5235
|
|
May 19, 2020, 09:24:17 PM Last edit: May 19, 2020, 09:34:26 PM by hosseinimr93 |
|
Meanwhile, BPIP only stores the time when users change passwords, other data seems to be ignored.
BPIP saves all data reported in seclog. Woke up:
Source: BPIPManual recovery:
Source: BPIPWhy create a separate seclog page for each user? It is not really necessary You are right. That's not necessary. I suggested having separate monthly seclog to avoid loading problem. I guess LoyceV didn't implement that so all data for a certain user can be found with a simple CTRL+F. So, I suggested separate seclog for users (only if the purpose of LoyceV was to make it easy to find all data for a specific user) instead of having all data in a single page.
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
Chlotide
|
|
May 20, 2020, 12:16:57 AM |
|
It's a good initiative but this page will get very big very fast!
This is why all the implementations on the forum have limitations of last 30 days, last 120 days, 200 posts etc The way I see it being able to scale is with search and filters. By default last 30 days and then filter by username, action, date, time frame etc with a limit of 200-300 rows (maybe add fewer rows with pagination).
IMHO it's great but not scalable unfortunately. It will probably bsod my pc in a few months of growing
|
|
|
|
tranthidung
Legendary
Offline
Activity: 2268
Merit: 4005
Farewell o_e_l_e_o
|
|
May 20, 2020, 09:21:18 AM Last edit: May 20, 2020, 09:45:59 AM by tranthidung |
|
The solution is import data directly from the loyce club to your computer, then save it in the format you plan to use. According to seclog page, there are less than 600 users per day over the last year so it is less than median of merit transactions. If you guys can play with merit data, you can play with this one too.
Thanks for another data but honestly once again you 'ruin' my life with data, LoyceV. Usecase: find strange thing if we see spike. Maybe a new spam campaign or bounty is launched and force spammers to wake up their accounts. Last year, we had some spike relates to Yobit. However, statistics should be connected to user ranks to have better data interpretation. - Woke-up users are high ranked: likely a new spam campaign/ bounty.
- Woke-up users are mostly newbies: a new bump service (that is killed by the bump score) or someone try to harm the forum with spam/attack.
You guys on the forum will fastly report such phenomenon but it is better if we can judge something from data-based results. And on this perspective, the data type is helpful.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16584
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
May 20, 2020, 04:55:01 PM |
|
What about splitting the page in alphabetic order? I spent too much time on this already, and I really don't think many people will use this. I think a better solution might be to use a search function that would allow for someone to search if/when a username had a security event. Agreed. But out of my skills-league. If anyone wants to make it, feel free to take my data and get daily updates from the forum Why don't you separate it into different months? Then there will be no loading problem. Perhaps, you are trying to make it easy to find a certain user by using CTRL+F? But we have BPIP for that. BPIP didn't scrape seclog as early as the archived data I've included. Let's see if BPIP can add this data I suggested having separate monthly seclog to avoid loading problem. You can find all monthly data on archive.is. It's a lot of work to find a certain username with 90 different pages. It's a good initiative but this page will get very big very fast! Not really, it's big already, but it will take several years to double in size. It will probably bsod my pc in a few months of growing Don't use Windows
|
|
|
|
ChuckBuck
|
|
May 20, 2020, 06:11:00 PM |
|
Meanwhile, BPIP only stores the time when users change passwords, other data seems to be ignored.
Do you think it only saved that information in the most recent time? I changed my email and woke up after a long period of inactivity (end of 2019), and watching me on BPIP, it didn't display that information.
|
|
|
|
tranthidung
Legendary
Offline
Activity: 2268
Merit: 4005
Farewell o_e_l_e_o
|
|
May 21, 2020, 07:13:09 AM |
|
Loyce, could you save the seclog data in .txt format as same as what you usually do with merit and btc blockchain data?
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16584
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
May 21, 2020, 03:37:39 PM Last edit: May 22, 2020, 12:04:49 PM by LoyceV Merited by tranthidung (1) |
|
Loyce, could you save the seclog data in .txt format as same as what you usually do with merit and btc blockchain data? Sure, all it took was one long line of code: See seclog.txt (or seclog.txt.gz): UnixTime UserID String 1590075009 2490711 woke up 1590074446 2491035 woke up 1590074179 1706830 woke up 1590073677 83586 woke up 1590073568 1115137 woke up 1590073086 2612874 password reset via email 1590072435 1817851 woke up 1590072232 1715288 woke up 1590071571 459378 woke up 1590071559 1304409 woke up 1590071550 459378 password reset via email 1590070728 998158 woke up 1590070330 366549 woke up 1590070316 949002 woke up 1590070280 1161921 woke up 1590069550 2612498 woke up 1590069117 2433138 woke up 1590068811 1064700 woke up 1590068499 1073616 woke up 1590068456 2603243 password reset via email 1590068448 1073616 password reset via email 1590067842 1421454 woke up 1590067829 1421454 password reset via email 1590067306 1748392 woke up 1590066644 1229436 woke up 1590066629 1229436 password reset via email 1590066209 842426 password changed 1590066182 842426 woke up 1590066059 1049269 password changed 1590066029 1748388 woke up 1590066025 1049269 woke up 1590065265 2272659 woke up 1590065031 331517 woke up 1590065013 290762 woke up 1590064921 322576 woke up 1590064854 2579450 woke up 1590064771 862915 woke up 1590064752 1380772 woke up 1590064532 2715207 woke up 1590064129 2809340 password reset via email 1590063915 2187418 password reset via email 1590063455 1111537 woke up 1590063406 2651918 woke up 1590063359 1194657 woke up 1590063197 2457299 woke up 1590062684 49320 woke up 1590062406 1797438 woke up 1590062156 1420329 woke up 1590062080 1099430 woke up 1590061969 1866874 password reset via email 1590061568 2710692 woke up 1590061371 2685759 woke up 1590061145 2579651 woke up 1590060861 1740285 woke up 1590060757 2653941 woke up 1590059965 2703192 woke up 1590059598 20145 woke up 1590059482 1831063 woke up 1590059466 1733619 woke up 1590058854 2573487 woke up 1590058343 2111196 woke up 1590058308 979231 woke up 1590058300 2324268 woke up 1590058262 1232629 password reset via email 1590058223 104892 woke up 1590057824 2517791 woke up 1590057718 23696 manual recovery, ownership change queued 1590057578 2567195 woke up 1590057238 77542 manual recovery, ownership change queued 1590056801 1720837 woke up 1590056358 1158590 password changed 1590056182 336754 woke up 1590056178 950934 woke up It gets daily updates.
|
|
|
|
babo
Legendary
Offline
Activity: 3598
Merit: 4125
|
|
May 22, 2020, 11:04:43 AM |
|
Bitcointalk's Security log only shows data from the last 30 days. At actmyname's request, I've just started keeping longer records. I expected to be done quickly, but sorting out archived data and converting it back to the same format turned out to be a lot of work and took several days.~~~~ as always you have done a great job compliments i can't imagine this forum without you -- we could also create a log for uidso as to see if the user resets the password several times or remains dormant for a long time and several timesis only an example of application
|
. .BLACKJACK ♠ FUN. | | | ███▄██████ ██████████████▀ ████████████ █████████████████ ████████████████▄▄ ░█████████████▀░▀▀ ██████████████████ ░██████████████ █████████████████▄ ░██████████████▀ ████████████ ███████████████░██ ██████████ | | CRYPTO CASINO & SPORTS BETTING | | │ | | │ | ▄▄███████▄▄ ▄███████████████▄ ███████████████████ █████████████████████ ███████████████████████ █████████████████████████ █████████████████████████ █████████████████████████ ███████████████████████ █████████████████████ ███████████████████ ▀███████████████▀ ███████████████████ | | .
|
|
|
|
Shimmiry
Full Member
Offline
Activity: 840
Merit: 105
★Bitvest.io★ Play Plinko or Invest!
|
|
May 22, 2020, 12:05:30 PM |
|
Can you teach me or somehow publish a thread on how to do scrapping? I've been amazed on how you manage to sort data depending on your likes. I would also wanted to make my own thread/scraps on newbies that aren't been active for couple of weeks (as I found it a good idea to somehow help the Community that is bombarded with fake newbie accounts just to make common topics and threads then being abandoned right after), and suggests an action corresponding with it. Also, I would like to apply my Front-end development skills in making those logs. I do programming as well hence I can understand it well.
Also, I do suggest that logs would be depend on month so that all users can view it with no lags.
|
|
|
|
LoyceV (OP)
Legendary
Offline
Activity: 3304
Merit: 16584
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
May 22, 2020, 02:31:06 PM |
|
Can you teach me or somehow publish a thread on how to do scrapping? Scraping is nothing more than just automating downloading data. You can use countless different methods, and my method won't likely fit many other people. So just find what works for you, and use that.
|
|
|
|
|