[UPDATED] bitcoin/bitcointalk.org Stats Animation - all in one thread

[DdmrDdmr]

Just in case you prefer using a more recent dataset, I’ve uploaded the file with all TXs as of last Friday. I’ve not overwritten my prior file just in case you prefer to use that dataset.

The newly uploaded files are located here:

CSV version (77Mb):
https://drive.google.com/file/d/1kOuW7gO-Om2X8zvwJhSrEet5eNjlvSRu/view?usp=sharing

Rar version (9Mb):
https://drive.google.com/file/d/1UY7YON7y-DzUGYIWh1V3jOt-P7l7kmS8/view?usp=sharing

[1714799948]

1714799948

[1714799948]

1714799948

[1714799948]

1714799948

[1714799948]

1714799948

[bitmover]

Made some updates:

- Added DdmrDdmr data. Now i can make animations and graphics per board. I am open to requests
- Made Wall Observer board animation
- Added Avatars to all animations.

How can I do this for the local Turkish section?

Hello Tom Bombadil. Yes, I can do that animation now. I will do as soon as I can.
TUrkish and any other request board (I can do any local)

[bitmover]

Just added Super Merit table and Turkish merit race.

To do:

-More local boards animations per request. Portuguese is the obvious next one
-More tables and different graphic visualizations with the new DdmrDdmr data. As now I can get data per board/topic I have wider possibilities

[bitmover]

Just in case you prefer using a more recent dataset, I’ve uploaded the file with all TXs as of last Friday. I’ve not overwritten my prior file just in case you prefer to use that dataset.

The newly uploaded files are located here:

CSV version (77Mb):
https://drive.google.com/file/d/1kOuW7gO-Om2X8zvwJhSrEet5eNjlvSRu/view?usp=sharing

Rar version (9Mb):
https://drive.google.com/file/d/1UY7YON7y-DzUGYIWh1V3jOt-P7l7kmS8/view?usp=sharing

Hello DdmrDdmr.

Can you share with me your last updated data from merit dashboard again?
I am using it in my merit table (https://www.bitcoindata.science/supertable.html) and in some other projects as well.

[DdmrDdmr]

<...>

I've uploaded all the TXs to the following file, including the TXs from today’s merit.txt file:
https://drive.google.com/file/d/1MnRa6e2O0RH7K2gw7AUoT7FwK8rHbX1-/view?usp=sharing

Observations:
-   I’ve only uploaded the .rar file, which is 16 Mb. If you’d rather have I upload the csv file without compression, tell me (264 Mb).
-   The files is csv, with quote qualifiers.
-   Code page is 1252 (ANSI – Latin 1) and Unicode.
-   The file contains the additional fields described here: https://bitcointalk.org/index.php?topic=5265298.msg54895308#msg54895308

Let's see if you can load this file properly. I has to make it Unicode because of the non Latin characters - i.e. russian charactrs on path_l4, which is the first childboard level. RowId number 20 is the first case with such characters.

[bitmover]

Let's see if you can load this file properly. I has to make it Unicode because of the non Latin characters - i.e. russian charactrs on path_l4, which is the first childboard level. RowId number 20 is the first case with such characters.

Thanks for that.

This file was not easily parsed... I had to change a lot of configurations in python notebook, even data rate limit... I learned a lot parsing your data.

I also had to open it in Notepad and I changed it encoding manually to UTF-8, breaking all those special characters (as I won't use them anyway)

My final csv was like this:

Code:

"8","2018-01-24 22:29:38","1516832978","1344962","dspencer","Member","1101839","Scout Georgenson","Jr. Member","1","https://bitcointalk.org/index.php?topic=2384335.msg28854772#msg28854772","Alternate cryptocurrencies","Announcements (Altcoins)","Tokens (Altcoins) (Moderators: mprep, Welsh)","","","Bitcoin Forum > Alternate cryptocurrencies > Announcements (Altcoins) > Tokens (Altcoins) (Moderators: mprep, Welsh) > [ANN] [ICO] Xwin – CryptoBet betting platform based on blockchain"

then I was finally able to open your dataset

Thank you again. I will work in that data to update my Super Merit table soon.

[DdmrDdmr]

<...>

<…>

Yep, I sometimes encounter issues with encodings I receive on files. Takes some time fighting to overcome it. If you’ve now got a method to load the file, that is not too much of a hassle, we can leave the file as is (or go over it if you want me to change the format).

The "path" field (last field) does indeed look weird (*) now on your quoted example, but the rest seem ok.

Check if row number 20 shows as this:

"20","2018-01-24 22:33:27","1516833207","1189487","Prometheu$","Full Member","1028592","nikola22","Full Member","1","https://bitcointalk.org/index.php?topic=2772292.msg28837085#msg28837085","Local","Russian","Aльтepнaтивныe кpиптoвaлюты ","","","Bitcoin Forum > Local > Pyccкий (Russian) > Aльтepнaтивныe кpиптoвaлюты (Moderators: xandry, Xal0lex) > [ANN][ICO] 🌀TOKENGO🌀 BUSINESS TOKENIZATION PLATFORM - Bonus up to 50%"

If the fields are displayed as above (and not as a bunch of question marks), you’re good to go.

(*) The weird looking character seems to be a result of this:

The problem is being caused by UTF-8 bytes being interpreted as Windows-1252 (or ISO 8859-1) bytes.

https://www.i18nqa.com/debug/utf8-debug.html

[bitmover]

The "path" field (last field) does indeed look weird (*) now on your quoted example, but the rest seem ok.

Check if row number 20 shows as this:

"20","2018-01-24 22:33:27","1516833207","1189487","Prometheu$","Full Member","1028592","nikola22","Full Member","1","https://bitcointalk.org/index.php?topic=2772292.msg28837085#msg28837085","Local","Russian","Aльтepнaтивныe кpиптoвaлюты ","","","Bitcoin Forum > Local > Pyccкий (Russian) > Aльтepнaтивныe кpиптoвaлюты (Moderators: xandry, Xal0lex) > [ANN][ICO] TOKENGO BUSINESS TOKENIZATION PLATFORM - Bonus up to 50%"

If the fields are displayed as above (and not as a bunch of question marks), you’re good to go.

This is my line 20

Code:

"20","2018-01-24 22:33:27","1516833207","1189487","Prometheu$","Full Member","1028592","nikola22","Full Member","1","https://bitcointalk.org/index.php?topic=2772292.msg28837085#msg28837085","Local","Russian","Альтернативные криптовалюты ","","","Bitcoin Forum > Local > Русский (Russian) > Альтернативные криптовалюты (Moderators: xandry, Xal0lex) > [ANN][ICO] í ¼í¼€TOKENGOí ¼í¼€ BUSINESS TOKENIZATION PLATFORM - Bonus up to 50%"

I can clean this with regex. I will just lose the pathl4 column in some rows (specially locals), which is the one with this crazy "    ÐÐ»ÑŒÑ‚ернативные криптоваÐ...    "...
But, overall I will can use pathl4 column, unless it has some crazy encoding.

How did you manage to work with this kind of characters?     льтepнaтивныe кpип and     . Are you using UTF-8?

[DdmrDdmr]

<...>

Actually, the data I scrape and store goes to a table where the fields are defined as nVarchar (as opposed to varchar). nVarchar supports Unicode characters, which is what I need to store the scraped information whilst preserving the content’s format.

Due to the above, the export I created requires me to mark Unicode in the export parametrization (forget the UTF8  -> Unicode overrides it even if I select it).

The resulting file opens with Excel seamlessly, as does notepad (notepad indicates that the format is actually UTF 16). You probable need to see how to treat Unicode files in your environment without all those prior conversions. Ideally, without needing to edit the file in between, since the editor may change the format and be part of the cause. This is normally a small painful process of getting the environment to interpret this kind of file properly.

[bitmover]

Ideally, without needing to edit the file in between, since the editor may change the format and be part of the cause. This is normally a small painful process of getting the environment to interpret this kind of file properly.

The process was so painful that I intentionally edited the file in notepad to change it, intentionally breaking unwanted characters.

I was able to update my table using your data, you can check it on https://www.bitcoindata.science/supertable.html

I will try to work in the new categories (path l4 first) on Monday.



Edit:

The resulting file opens with Excel seamlessly, as does notepad (notepad indicates that the format is actually UTF 16).

I just changed encoding to UTF-16 and it worked without too much a hashle (just did a few adjustments). My line 20 is intact now and I can see Aльтepнaтивныe кpиптoвaлюты correctly

[DdmrDdmr]

<…> I was able to update my table using your data, you can check it on https://www.bitcoindata.science/supertable.html <...>

Looks good. We differ a bit on the last decimal of the percentages, but minimal, likely due to using different rounding precision.

<…> I just changed encoding to UTF-16 and it worked without too much a hashle (just did a few adjustments). My line 20 is intact now and I can see Aльтepнaтивныe кpиптoвaлюты correctly <…>

Great! slayed the Unicode dragon pretty swiftly there.

[examplens]

The process was so painful that I intentionally edited the file in notepad to change it, intentionally breaking unwanted characters.

I was able to update my table using your data, you can check it on https://www.bitcoindata.science/supertable.html

I will try to work in the new categories (path l4 first) on Monday.

Wow, well done, glad to see this

I have a suggestion for you. Maybe you can disable report from zero value column. it only causes additional scrolling. Also when we talking about the scroll, if it's possible to change horizontal to vertical scroll, it is easier to use on desktop pc. That will be fine.
Tried on a mobile device, and there is okay. Table fields are placed one below the other.

[bitmover]

Wow, well done, glad to see this

I have a suggestion for you. Maybe you can disable report from zero value column. it only causes additional scrolling.

Thanks examplens

I liked your suggestion, however I am not able to do it for now.
As I explained here, I tried to generate the table myself using bootstrap, which would allow a lot of customization options. But the loading time was terrible. I think I need to learn more about SQL or AJAX or soemthing like that to make a server side processing.

I am not able to do it.

I tried to use bootstrap + datatable.net to create the table, so I would have more customization options. However, I do not know how to efficiently parse the data, and my HTML file had 30mb and consumed a lot of processing power.

Flourish is very efficient in parsing the data, so I will keep the table in this format and move on to other projects. At least for now. Maybe in the future as I learn more about JS and SQL...

Also when we talking about the scroll, if it's possible to change horizontal to vertical scroll, it is easier to use on desktop pc. That will be fine.
Tried on a mobile device, and there is okay. Table fields are placed one below the other.

This I can do. This is a dataframe transpose. And it is really a nice suggestion. I will make some tests

[bitmover]

<...>

I've uploaded all the TXs to the following file, including the TXs from today’s merit.txt file:
https://drive.google.com/file/d/1MnRa6e2O0RH7K2gw7AUoT7FwK8rHbX1-/view?usp=sharing

Hello DdmrDdmr.
Do you have an updated file for merit transactions? So i can update the merit table.
It doesn't need to be so big like the old one. I decided not to include those child boards, and focus more in other projects. Please share the link with me when you can.

Thank you

[DdmrDdmr]

<…>

Since tomorrow we receive the new Merit.txt file, it’s probably better to wait for that update, in order to have the most recent version. It should therefore be ready tomorrow (max. Saturday). I’ll let you know when it’s done, and I’ll trim the information on child boards from it (leaving the granularity at subsection level again).

Edit: File-> https://drive.google.com/file/d/1meWPgG_bIyChkUIcxHhTQnB4WPwp1oAV/view?usp=sharing

[bitmover]

<…>

Since tomorrow we receive the new Merit.txt file, it’s probably better to wait for that update, in order to have the most recent version. It should therefore be ready tomorrow (max. Saturday). I’ll let you know when it’s done, and I’ll trim the information on child boards from it (leaving the granularity at subsection level again).

Edit: File-> https://drive.google.com/file/d/1meWPgG_bIyChkUIcxHhTQnB4WPwp1oAV/view?usp=sharing

Thank you DdmrDdmr.

The page was just updated with the latest data
 link

[bitmover]

You can use the following file: https://drive.google.com/file/d/1rOvFPbd9T6q5qrtLtAD1gy4fBMcUrg3Z/view?usp=sharing
It contains all the historical TXs, including the board description (section/subsection) It occupies some 75 MBs. You’ll have to load them to a DB or whatever repository you are going to use, and filter by subsection = Turkish (last column).

The data is as of last Friday 19/06/2020. How you create a the race itself is up to you (python, Tableau, etc.).

Hello @DdmrDdmr,

Can you update this file again, please?
I have some ideas about new charts, a network chart for some local boards.

Thank you.

[DdmrDdmr]

<...>

I’ve uploaded a file with the same structure as last time to the following link:
https://drive.google.com/file/d/1dANGltunbyBo1JwNaVLUhdGfQVnTdR4O/view?usp=sharing

The file is a compressed file, that contains the flat file "TXs_Bitmover_export_20210228.txt", with data as of last Friday.

Bear in mind that I haven’t had my morning coffee yet, so please check that the content is as expected…

[fillippone]

Hey,
I know this thread hasn’t been updated for a long time.
Is there the possibility to have a merit race for the Italian board?
Or any new idea you came up with in those few months?
Thanks!

[bitmover]

Hey,
I know this thread hasn’t been updated for a long time.
Is there the possibility to have a merit race for the Italian board?
Or any new idea you came up with in those few months?
Thanks!

Of course!
For local boards I need some extra data from @DdmrDdmr. Can you please send me an updated file of including Section/subsection column? no need to have too many details, I will consider any local child board as local only.
Thank you DdmrDdmr