LoyceV's seclog log

[LoyceV]

Bitcointalk's Security log only shows data from the last 30 days. At actmyname's request, I've just started keeping longer records. I expected to be done quickly, but sorting out archived data and converting it back to the same format turned out to be a lot of work and took several days.

LoyceV's seclog log
See loyce.club/seclog/! (Warning: current size 168 MB, Firefox works better than Chrome)
Or loyce.club/seclog/index.html.gz (Recommended, currently 18 MB)
Machine readable: seclog.txt (32 MB) or seclog.txt.gz (7.3 MB)

This contains all data I could find from seclog, archived posts and data provided by a user.

I don't think this has much discussion value, I just leave it here for whoever can use it.

Incomplete
I show what I have, but this data is not complete because I started scraping 10 years too late. From now on it should include all new seclog-entries.

Updates
Once a day. This isn't meant to be real-time.
For more recent data, see seclog.

Bugs
Please post if there is any problem with the data.

No spam
Please

See [overview] LoyceV's useful data on Bitcointalk for more of my forum-related topics

[1713569806]

1713569806

[SFR10]

LoyceV's seclog log
See loyce.club/seclog/! (Warning: current size 115 MB)

~Snipped~

Bugs
Please post if there is any problem with the data.

Not sure if it's my internet speed or it really loads slowly ^{[it took me a very long time just to be able to load the data for the previous day and scroll down without having lags]}...

• Highlighting a certain part seems to contribute to its lag.
• After a certain time, it stops loading the data (I waited for 5 minutes but it doesn't load any data past may 16^th, on my side).
  ^{- I was able to replicate this issue twice (tested in Chrome).}

[LoyceV]

(tested in Chrome)

From my Trust list viewer, I know Firefox can handles large HTML pages better than Chrome. The page has 638000+ entries, browsers don't really like it, but if I'd make it a text file the links won't work.

I'll add a compressed file in a bit, it might work better from local storage.

[Welsh]

Not sure if it's my internet speed or it really loads slowly ^{[it took me a very long time just to be able to load the data for the previous day and scroll down without having lags]}...

• Highlighting a certain part seems to contribute to its lag.
• After a certain time, it stops loading the data (I waited for 5 minutes but it doesn't load any data past may 16^th, on my side).
  ^{- I was able to replicate this issue twice (tested in Chrome).}

Its slow because as far as I can tell it continuously loads the data on the page. However, I made it to November 2019 fairly quickly, and the slow loading is partly because of the way its designed to load data, your client, and connection speed. As LoyceV stated, Firefox handles it slightly better, and that's what I tested it on.

I'll add a compressed file in a bit, it might work better from local storage.

Is it worth considering implementing pagination with the option of displaying all data on one singular page if desired?

[LoyceV]

Is it worth considering implementing pagination with the option of displaying all data on one singular page if desired?

I kinda expect the people who use this, to want to search the whole thing at once: wait for it to load, then CTRL-F and open links in a New Tab. I've added a compressed file. If it's slow after downloading, you have your own computer to blame

[coinlocket$]

The file is way too big, I've tried to use CTRL F or scrolling up and down but got a white page for loading.
I have a 200MB connection that should be faster.

What about splitting the page in alphabetic order?

[PrimeNumber7]

LoyceV's seclog log
See loyce.club/seclog/! (Warning: current size 115 MB)
Or loyce.club/seclog/index.html.gz (Recommended, currently 12 MB)

This is probably too large for most web browsers. I found this stack overflow question where someone claims that browsers tend to die after loading a 60 MB text file.

I think a better solution might be to use a search function that would allow for someone to search if/when a username had a security event.

Sending 100+ MB of data to the internet is also expensive. Sending a 100 MB file to the internet will cost about 0.9 cents, or US$0.009 and this can add up if you send this file many times.

[SFR10]

I know Firefox can handles large HTML pages better than Chrome.

As LoyceV stated, Firefox handles it slightly better, and that's what I tested it on.

Just tried testing it on Firefox and the difference was night and day ^{[quite responsive]}.

I've added a compressed file. If it's slow after downloading, you have your own computer to blame

I have a beefy computer but the improvement wasn't that significant in Chrome
^{- Perhaps as you said earlier, it has to do with browser limitations.}
I disabled "hardware acceleration" in Chrome and noticed some improvements...

[hosseinimr93]

Why don't you separate it into different months? Then there will be no loading problem.
Perhaps, you are trying to make it easy to find a certain user by using CTRL+F? But we have BPIP for that. Also, for that purpose, you may consider creating a separate seclog page for every user. (I don't know how time consuming creating that is.)

[ChuckBuck]

Also, for that purpose, you may consider creating a separate seclog page for every user. (I don't know how time consuming creating that is.)

LoyceV saved simply the entire Bitcointalk seclog for a longer period of time. Meanwhile, BPIP only stores the time when users change passwords, other data seems to be ignored. The most specific example, the moment the user wakes up, BPIP does not save it. This data is really important and necessary in investigations.
Why create a separate seclog page for each user? It is not really necessary 

What about splitting the page in alphabetic order?

No, it seems to be a bad idea. Sorting in real time would be better. You won't want to find a guy at the bottom of the alphabet who changed his password yesterday  But add some sort options looks good to search if you want. Anyway, Ctrl + F is the best 

[hosseinimr93]

Meanwhile, BPIP only stores the time when users change passwords, other data seems to be ignored.

BPIP saves all data reported in seclog.

Woke up:


Source: BPIP


Manual recovery:


Source: BPIP

Why create a separate seclog page for each user? It is not really necessary  

You are right. That's not necessary. I suggested having separate monthly seclog to avoid loading problem. I guess LoyceV didn't implement that so all data for a certain user can be found with a simple CTRL+F. So, I suggested separate seclog for users (only if the purpose of LoyceV was to make it easy to find all data for a specific user) instead of having all data in a single page.

[Chlotide]

It's a good initiative but this page will get very big very fast!

This is why all the implementations on the forum have limitations of last 30 days, last 120 days, 200 posts etc
The way I see it being able to scale is with search and filters. By default last 30 days and then filter by username, action, date, time frame etc with a limit of 200-300 rows (maybe add fewer rows with pagination).

IMHO it's great but not scalable unfortunately. It will probably bsod my pc in a few months of growing

[tranthidung]

The solution is import data directly from the loyce club to your computer, then save it in the format you plan to use.

According to seclog page, there are less than 600 users per day over the last year so it is less than median of merit transactions. If you guys can play with merit data, you can play with this one too.
Thanks for another data but honestly once again you 'ruin' my life with data, LoyceV.

Usecase: find strange thing if we see spike. Maybe a new spam campaign or bounty is launched and force spammers to wake up their accounts. Last year, we had some spike relates to Yobit.

However, statistics should be connected to user ranks to have better data interpretation.

• Woke-up users are high ranked: likely a new spam campaign/ bounty.
• Woke-up users are mostly newbies: a new bump service (that is killed by the bump score) or someone try to harm the forum with spam/attack.

You guys on the forum will fastly report such phenomenon but it is better if we can judge something from data-based results. And on this perspective, the data type is helpful.

[LoyceV]

What about splitting the page in alphabetic order?

I spent too much time on this already, and I really don't think many people will use this.

I think a better solution might be to use a search function that would allow for someone to search if/when a username had a security event.

Agreed. But out of my skills-league. If anyone wants to make it, feel free to take my data and get daily updates from the forum

Why don't you separate it into different months? Then there will be no loading problem.
Perhaps, you are trying to make it easy to find a certain user by using CTRL+F? But we have BPIP for that.

BPIP didn't scrape seclog as early as the archived data I've included. Let's see if BPIP can add this data

I suggested having separate monthly seclog to avoid loading problem.

You can find all monthly data on archive.is. It's a lot of work to find a certain username with 90 different pages.

It's a good initiative but this page will get very big very fast!

Not really, it's big already, but it will take several years to double in size.

It will probably bsod my pc in a few months of growing

Don't use Windows

[ChuckBuck]

Meanwhile, BPIP only stores the time when users change passwords, other data seems to be ignored.

Do you think it only saved that information in the most recent time? I changed my email and woke up after a long period of inactivity (end of 2019), and watching me on BPIP, it didn't display that information.

[tranthidung]

Loyce, could you save the seclog data in .txt  format as same as what you usually do with merit and btc blockchain data?

[LoyceV]

Loyce, could you save the seclog data in .txt  format as same as what you usually do with merit and btc blockchain data?

Sure, all it took was one long line of code:
See seclog.txt (or seclog.txt.gz):

Code:

UnixTime UserID String
1590075009 2490711 woke up
1590074446 2491035 woke up
1590074179 1706830 woke up
1590073677 83586 woke up
1590073568 1115137 woke up
1590073086 2612874 password reset via email
1590072435 1817851 woke up
1590072232 1715288 woke up
1590071571 459378 woke up
1590071559 1304409 woke up
1590071550 459378 password reset via email
1590070728 998158 woke up
1590070330 366549 woke up
1590070316 949002 woke up
1590070280 1161921 woke up
1590069550 2612498 woke up
1590069117 2433138 woke up
1590068811 1064700 woke up
1590068499 1073616 woke up
1590068456 2603243 password reset via email
1590068448 1073616 password reset via email
1590067842 1421454 woke up
1590067829 1421454 password reset via email
1590067306 1748392 woke up
1590066644 1229436 woke up
1590066629 1229436 password reset via email
1590066209 842426 password changed
1590066182 842426 woke up
1590066059 1049269 password changed
1590066029 1748388 woke up
1590066025 1049269 woke up
1590065265 2272659 woke up
1590065031 331517 woke up
1590065013 290762 woke up
1590064921 322576 woke up
1590064854 2579450 woke up
1590064771 862915 woke up
1590064752 1380772 woke up
1590064532 2715207 woke up
1590064129 2809340 password reset via email
1590063915 2187418 password reset via email
1590063455 1111537 woke up
1590063406 2651918 woke up
1590063359 1194657 woke up
1590063197 2457299 woke up
1590062684 49320 woke up
1590062406 1797438 woke up
1590062156 1420329 woke up
1590062080 1099430 woke up
1590061969 1866874 password reset via email
1590061568 2710692 woke up
1590061371 2685759 woke up
1590061145 2579651 woke up
1590060861 1740285 woke up
1590060757 2653941 woke up
1590059965 2703192 woke up
1590059598 20145 woke up
1590059482 1831063 woke up
1590059466 1733619 woke up
1590058854 2573487 woke up
1590058343 2111196 woke up
1590058308 979231 woke up
1590058300 2324268 woke up
1590058262 1232629 password reset via email
1590058223 104892 woke up
1590057824 2517791 woke up
1590057718 23696 manual recovery, ownership change queued
1590057578 2567195 woke up
1590057238 77542 manual recovery, ownership change queued
1590056801 1720837 woke up
1590056358 1158590 password changed
1590056182 336754 woke up
1590056178 950934 woke up

It gets daily updates.

[babo]

Bitcointalk's Security log only shows data from the last 30 days. At actmyname's request, I've just started keeping longer records. I expected to be done quickly, but sorting out archived data and converting it back to the same format turned out to be a lot of work and took several days.
~~~~

as always you have done a great job
compliments

i can't imagine this forum without you
--
we could also create a log for uid
so as to see if the user resets the password several times or remains dormant for a long time and several times

is only an example of application

[Shimmiry]

Can you teach me or somehow publish a thread on how to do scrapping? I've been amazed on how you manage to sort data depending on your likes. I would also wanted to make my own thread/scraps on newbies that aren't been active for couple of weeks (as I found it a good idea to somehow help the Community that is bombarded with fake newbie accounts just to make common topics and threads then being abandoned right after), and suggests an action corresponding with it. Also, I would like to apply my Front-end development skills in making those logs. I do programming as well hence I can understand it well.

Also, I do suggest that logs would be depend on month so that all users can view it with no lags.

[LoyceV]

Can you teach me or somehow publish a thread on how to do scrapping?

Scraping is nothing more than just automating downloading data. You can use countless different methods, and my method won't likely fit many other people. So just find what works for you, and use that.