My journey from user, to cyborg, to maybe coding a bot.

[DirtyKeyboard]

Ok.  So.  Here's where we're at.  I have moved to posting ChartBuddy's daily recap at UTC midnight.  Either with my phone, from the road, using Chrome Remote to a home laptop, or I get Task Scheduler to work more consistently tomorrow.

Storylog:
1.Download I have changed, with help, the regular expression to download the previous day's images only.  Whether that is zero or 24.  Next would be what to do with missing frames, and what the minimum frame count would be to even do a recap that day.  This also allows a variable, total_downloads to be set for later commands.  I need to somehow pass the information to the GIMP plugin also.

2.Import  I'm going to try to save the total_downloads variable to a file, so that the gimp plug in can load in the correct number of frames for the gif.  Or just run the same code inside the GIMP plugin to count the number of files/frames/images to import.

3.Export Secret API keys working consistently.  I think I need to refresh one of them soon somehow.

4.Posting Currently testing having Windows Task Scheduler run the script, through a batch file.
5.Archive Starting tomorrow all frames will hopefully be saved as, dates  yyyy-mm-dd.  As suggested above, and yes, should make sorting so much easier going forward.  I'll post the code after doing a renaming, and clean up effort tomorrow.

For the Top20 Days of Bitcoin thread, we've had some back and forth.  I get things working, and then can't leave well enough alone.  It's up to 9 currencies, 1 BTC  $ volume column, and 3 moving averages columns.  It's created by 3 table maker scripts, which call 12 seperate column making scripts.  4 of the 12 are somewhat custom but the other 8 are virtually identical in function, that they should be inputs to a function.  Soon.
1.Download  Now instead of downloading the whole 5 year data set each day, the script has added functionality of now only downloaded the latest data and appending the main list.

2.Import Figured out a new way to keep track of the newest date, when sorting by VWAP, and the list continues to grow in length each day. I put a counter in while processing all the files to look for the oldest date in the top 100.  

3.Export  Learned how to have one script call another script.

4.Posting  It's one click action, but I'm currently testing having Windows Task Scheduler run the script, through a batch file.

5.Archive  I'm trying not to hoard things this project, but the ever expanding price and volume history for BTC in each currency should be backed up regularly  

Here is the lead table maker script which calls the other 2 table making scripts, and handles almost all of the formatting, headers, and closers.

Code:

import pyautogui, pyperclip, shutil, time, webbrowser

# sets a sleep timer allowing for easier setting of faster runs during testing, and the data has been downloaded already
sleep_time = 15

# calls the scripts which create each column to be zipped together below
import Top20USD
time.sleep(sleep_time)
import Top20GBP
time.sleep(sleep_time)
import Top20EUR
time.sleep(sleep_time)
import Top20CAD
time.sleep(sleep_time)
import Top20JPY
time.sleep(sleep_time)

# location of the columns created above
usd = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_USD/top100usd.txt'
gbp = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_GBP/top100gbp.txt'
eur = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_EUR/top100eur.txt'
cad = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_CAD/top100cad.txt'
jpy = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_JPY/top100jpy.txt'

file_paths = [usd, gbp, eur, cad, jpy]
print("importingfiles")
with open('C:/PyProjects/Top20/Top20_Total_03_24/top20table.txt', 'w', encoding='utf-8') as output_file:

    # Read lines from all files simultaneously
    with open(file_paths[0], 'r', encoding='utf-8') as file1, \
            open(file_paths[1], 'r', encoding='utf-8') as file2, \
            open(file_paths[2], 'r', encoding='utf-8') as file3, \
            open(file_paths[3], 'r', encoding='utf-8') as file4, \
            open(file_paths[4], 'r', encoding='utf-8') as file5:
        
        for line1, line2, line3, line4, line5 in zip(file1, file2, file3, file4, file5):

            # Write combined lines to the output file
            output_line = f"| {line1.strip()}| {line2.strip()}| {line3.strip()}| {line4.strip()}| {line5.strip()}|\n"
            output_file.write(output_line)
# progress check
print("clipboarding")

# putting the trad5 part together, setting links as variables to limit code line length
with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20table.txt', 'r') as top100:
    ul = 'https://bitcoincharts.com/charts/bitstampUSD'
    gl = 'https://data.bitcoinity.org/markets/volume/5y/GBP/kraken?r=day&t=b'
    el = 'https://data.bitcoinity.org/markets/volume/5y/EUR/kraken?r=day&t=b'
    cl = 'https://data.bitcoinity.org/markets/volume/5y/CAD/kraken?r=day&t=b'
    jl = 'https://data.bitcoinity.org/markets/volume/5y/JPY/kraken?r=day&t=b'
    list = top100.read()
    pre = f"[pre]   US Dollar              British Pound          EU Euro                Canadian Dollar        Japanese Yen"
    prelude = f"[size=9pt][b]| [url={ul}]Rank BitStamp   USD/BTC[/url]| [url={gl}]Rank Kraken     GBP/BTC[/url]| [url={el}]Rank Kraken     EUR/BTC[/url]| [url={cl}]Rank Kraken     CAD/BTC[/url]| [url={jl}]Rank Kraken        JPY/BTC[/url]|[/b]"
    explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                                                    *  Chart  Legends  *                                                       |[/url]"
    JimboToronto = "|                                                          GoBTCGo™                                                             |[/pre]"
    full_post1 = f"{pre}\n{prelude}\n{list}{explanation}\n{JimboToronto}"

# on to the next set of tablesk, this script calls 4 seperate currency scripts
import tm_T2_03_21

# making the second table 
with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20tableT2.txt', 'r') as top100T2:
   
    il = 'https://data.bitcoinity.org/markets/volume/5y/IDR/bitcoin.co.id?r=day&t=b'
    kl = 'https://data.bitcoinity.org/markets/volume/5y/KRW/korbit?r=day&t=b'
    al = 'https://data.bitcoinity.org/markets/volume/5y/AUD/btcmarkets?r=day&t=b'
    ual = 'https://data.bitcoinity.org/markets/volume/5y/UAH/exmo?r=day&t=b'
    bv = 'https://data.bitcoinity.org/markets/volume/5y/USD/bitcoin.co.id?r=day&t=b' # this the highest BTC*USD price*volume
    listT2 = top100T2.read()
    pre = f"[pre]  Korean Wan               Australian Dollar    Ukrainian Hryvnia      Indonesian Rupiah          Bitstamp Exchange"
    prelude = f"[size=9pt][b]| [url=https://data.bitcoinity.org/markets/volume/5y/KRW/korbit?r=day&t=b]#  Korbit         KRW/BTC[/url]| [url=https://data.bitcoinity.org/markets/volume/5y/AUD/btcmarkets?r=day&t=b]#  btcmarkets AUD/BTC[/url]| [url=https://data.bitcoinity.org/markets/volume/5y/UAH/exmo?r=day&t=b]#  exmo         UAH/BTC[/url]| [url=https://data.bitcoinity.org/markets/volume/5y/IDR/bitcoin.co.id?r=day&t=b]#  bitcoin.co.id    IDR/BTC[/url]| [url=https://bitcoincharts.com/charts/bitstampUSD]#  BTC vol  in MUSD[/url]|[/b]"
    explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                                                    *  Chart  Legends  *                                                    |[/url]"
    JimboToronto = "|                                                          GoBTCGo™                                                          |[/pre]"
    full_post2 = f"{pre}\n{prelude}\n{listT2}{explanation}\n{JimboToronto}"

# the final set of newest moving day average columns
import tm_T3_03_30

with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20tableT3.txt', 'r') as top100T3:
    ul = 'https://bitcoincharts.com/charts/bitstampUSD'
    listT3 = top100T3.read()
    pre = f"[pre]Moving Average VWAP, the date listed is the final day of the given time span\n|  7 Day MA             30 Day MA           200 Day MA"
    prelude = f"[size=9pt][b]|  [url={ul}]#     7 DMA  USD/BTC[/url]|  [url={ul}]#     30 DMA USD/BTC[/url]| [url={ul}]#     200 DMA USD/BTC[/url]|[/b]"
    explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                         *  Chart  Legends  *                       |[/url]"
    JimboToronto = "|                               GoBTCGo™                       |[/pre]"
    full_post3 = f"{pre}\n{prelude}\n{listT3}{explanation}\n{JimboToronto}"
    full_post = f"Daily [BTC] Volume Weighted Average Prices\n{full_post1}\n{full_post2}\n{full_post3}"
    
    pyperclip.copy(full_post)

    # can use this link for the reply page to top20 thread
    url = 'https://bitcointalk.org/index.php?action=post;topic=138109.0'
    webbrowser.open(url)
    time.sleep(5)
    pyautogui.hotkey('f11')
    time.sleep(5)
    pyautogui.hotkey('tab')
    time.sleep(2)
    pyautogui.hotkey('tab')
    time.sleep(2)
    pyautogui.hotkey('ctrl', 'v')
    time.sleep(2)
    pyautogui.hotkey('tab')
    time.sleep(2)
    # we're doing it live if the next command is #ed out
    pyautogui.hotkey('tab')
    time.sleep(5)
    pyautogui.hotkey('enter')
    time.sleep(5)
    pyautogui.hotkey('f11')
    
# if i don't delete the cache each time, the minor changes I make don't show up the next run, i think
shutil.rmtree('C:/PyProjects/Top20/Top20_Total_03_24/__pycache__')

Here is a sample of the code that should be a function.  This one gathers the newest data to append for Great Britain's Pound, sets the newest red item, the oldest green item, bolds items from the last 30 days, and makes sure VWAPs with different numbers of digits still line up.  Check out the outrageous use of commenting to choose your own adventure.  

Code:

import csv, os, requests, time
from datetime import datetime, timedelta
import pandas as pd

# vwap_digits varialbe is used later to have, say, 100_000 and 99_999  be lined up in a column.
currency = "GBP"
exchange = "kraken"
next_vwap_digits = "6"  # not used yet, set manually for each currency.

# current_unix_time = int(time.time())
unix_time_month = 60 * 60 * 24 * 31
unix_time_day = 60 * 60 * 24
today = datetime.today()

###########################################################
# RUN this if the previous days are lost, otherwise just download the last 2 days, jump ahead 
###########################################################

# url = f"https://data.bitcoinity.org/export_data.csv?currency={currency}&data_type=volume&exchange={exchange}&r=day&t=b&timespan=5y&vu=curr"
# response = requests.get(url)
# print(response)
# # Check if the request was successful (status code 200)
# if response.status_code == 200:
#     # Assuming the response contains CSV data, you can save it to a file
#     with open(f"C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/{currency}_volume.csv", "w", newline="") as csvfile:
#         csvfile.write(response.text)
#     print("CSV file downloaded successfully!")
# else:
#     print(f"Error: {response.status_code} - Unable to download CSV data.")
# time.sleep(15)

# url2 = f"https://data.bitcoinity.org/export_data.csv?currency={currency}&data_type=volume&exchange={exchange}&r=day&t=b&timespan=5y"
# response = requests.get(url2)
# print(response)
# # Check if the request was successful (status code 200)
# if response.status_code == 200:
#     # Assuming the response contains CSV data, you can save it to a file
#     with open(f"C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/BTC_volume.csv", "w", newline="") as csvfile:
#         csvfile.write(response.text)
#     print(f"CSV file downloaded successfully!")
# else:
#     print(f"Error: {response.status_code} - Unable to download CSV data.")


# ###########################################################
# # run this to only append the latest data              
# ###########################################################


filename = f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/{currency}_volume.csv'   
url = f"https://data.bitcoinity.org/export_data.csv?currency={currency}&data_type=volume&exchange={exchange}&r=day&t=b&timespan=3d&vu=curr"
response = requests.get(url)
print(response)
# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Split the response text into lines
    lines = response.text.split('\n')
    print(lines)

    # Get the last line (excluding the potential trailing empty line)
    last_line = lines[-2] if lines[-1] == '' else lines[-1]
    print(last_line)              
    # Assuming the response contains CSV data, you can save it to a file
    with open(filename, "a", newline="") as csvfile:
        csvfile.write(last_line + '\n')
    print("CSV file downloaded successfully!")
else:
    print(f"Error: {response.status_code} - Unable to download CSV data.")
time.sleep(15)


filename = f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/BTC_volume.csv'
url2 = f"https://data.bitcoinity.org/export_data.csv?currency={currency}&data_type=volume&exchange={exchange}&r=day&t=b&timespan=3d"
response = requests.get(url2)
print(response)
# Check if the request was successful (status code 200)
if response.status_code == 200:
    # Split the response text into lines
    lines = response.text.split('\n')

    # Get the last line (excluding the potential trailing empty line)
    last_line = lines[-2] if lines[-1] == '' else lines[-1]
    print(last_line)  
    # Assuming the response contains CSV data, you can save it to a file
    with open(filename, "a", newline="") as csvfile:
        csvfile.write(last_line + '\n')
    print(f"CSV file downloaded successfully!")
else:
    print(f"Error: {response.status_code} - Unable to download CSV data.")


###########################################################
# end of commenting choice                                 
###########################################################


# Read the CSV files
datestamp = pd.read_csv(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/{currency}_volume.csv', usecols=[0])  # Assuming timestamp is in the first column
curr_df = pd.read_csv(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/{currency}_volume.csv', usecols=[1])
btc_df = pd.read_csv(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/BTC_volume.csv', usecols=[1])

# Perform division
result_df = curr_df / btc_df

# Save the  combined DataFrame to a new CSV file
result_df.to_csv(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/result_with_timestamp.csv', index=False)

# Extract the date part from the timestamp
result_df['Date'] = pd.to_datetime(datestamp.iloc[:, 0]).dt.date

# Save the combined DataFrame to a new CSV file
result_df.to_csv(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/result_with_timestamp.csv', index=False)

print("Results with timestamp written to 'result_with_timestamp.csv'.")

filename = f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/result_with_timestamp.csv'
top = 100
red_rank = 0
# Read data from the CSV file for green finding later
rows = []
with open(filename, 'r') as file:
    reader = csv.reader(file)
    next(reader)  # Skip the header row

    # Fill empty cells in column [0] before sorting, luckily this only happened when prices were low.  Here is where the red_rank variable is counted.
    # the last item in the list will be the red item if it makes the top 100 daily vwap.
    previous_value = 10000
    for row in reader:
        vwapcurr = row[0] or previous_value  # Fill empty cell with previous value
        time = row[1]
        rows.append((time, vwapcurr))
        previous_value = vwapcurr  # Update previous value for the next iteration
        red_rank += 1

    # Sort the data by VWAPcurr
    tuples = [(timestamp, vwap, i + 1) for i, (timestamp, vwap) in enumerate(rows)]
    sorted_tuples = sorted(tuples, key=lambda x: float(x[1]), reverse=True)
    rank = 1
    red_rank -= 1
    print(f"red rank is {red_rank}")
    with open(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/forgreen{currency}.txt', 'w') as top100:
        
        for time, vwapcurr, rows in sorted_tuples[:top]:
            formatted_date = datetime.strptime(time, '%Y-%m-%d').strftime('%Y-%m-%d')
            print(time)
            date_difference = today - datetime.strptime(formatted_date, '%Y-%m-%d')
            vwapcurr_float = float(vwapcurr)
            formatted_output = f"{rank:2d}, {time}, {vwapcurr_float:.0f}"
            top100.write(formatted_output + '\n')
            rank += 1

    os.replace (f"C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/forgreen{currency}.txt", f"C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/forgreen{currency}.csv")  
    filename = f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/forgreen{currency}.csv'

    # Read data from the to find green rank
    rows = []
    with open(filename, 'r') as file:
        reader = csv.reader(file)
        for row in reader:
            rows.append(row)

    # Sort the data by date, top entry is oldest rank in top 100, which is the green row
    tuples = [(rank, timestamp, vwap) for rank, timestamp, vwap in rows]  
    sorted_tuples = sorted(tuples, key=lambda x: x[1]) 
    green_rank, _, _ = sorted_tuples[0]
    print(green_rank)


    # sort for rank, bolding, coloring
    filename = f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/result_with_timestamp.csv'
    top = 100

    # Read data from the CSV file
    rows = []
    with open(filename, 'r') as file:
        reader = csv.reader(file)
        next(reader)  # Skip the header row

        # Fill empty cells in column [0] before sorting
        previous_value = 1000
        for row in reader:
            vwapcurr = row[0] or previous_value  # Fill empty cell with previous value
            time = row[1]
            rows.append((time, vwapcurr))
            previous_value = vwapcurr  # Update previous value for the next iteration

    # Sort the data by VWAPcurr
    tuples = [(timestamp, vwap, i + 1) for i, (timestamp, vwap) in enumerate(rows)]
    sorted_tuples = sorted(tuples, key=lambda x: float(x[1]), reverse=True)
    rank = 1

    with open(f'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_{currency}/top100{currency}.txt', 'w') as top100:
        
        for time, vwapcurr, rows in sorted_tuples[:top]:
            formatted_date = datetime.strptime(time, '%Y-%m-%d').strftime('%Y-%m-%d')
            date_difference = today - datetime.strptime(formatted_date, '%Y-%m-%d')
            vwapcurr_float = float(vwapcurr)
            if rank <= 99:
                spacing = " "
            if rank == 100:
                spacing = "" 
            if date_difference <= timedelta(days = 30):
                bolding = "[b]"
                unbolding = "[/b]"
            if date_difference >= timedelta(days = 31):
                bolding = ""
                unbolding = ""
            if rows == red_rank:
                redcoloring = "[color=red][u]"
                reduncoloring = "[/u][/color]"
            if rows != red_rank:
                redcoloring = ""
                reduncoloring = ""     
            if  rank < int(green_rank):
                greencoloring = ""
                greenuncoloring = ""
            if  rank > int(green_rank):
                greencoloring = ""
                greenuncoloring = ""
            if vwapcurr_float <= 99_999:
                    endspace = " "
            if vwapcurr_float >= 100_000:
                    endspace = ""
            if rank - int(green_rank) == 0:
                greencoloring = "[color=#0BC505][u]"
                greenuncoloring = "[/u][/color]"
            formatted_output = f"{redcoloring}{greencoloring}{bolding}{rank:2d}{spacing} {formatted_date}  {endspace}{vwapcurr_float:,.0f}{unbolding}{reduncoloring}{greenuncoloring}"
            top100.write(formatted_output + '\n')
            rank += 1

EDIT: Phrasing

[1715103155]

1715103155

[LoyceV]

what to do with missing frames

Maybe make the previous frame gray? That way, it's still clear the hour passed, but the data is missing. Or a graph without data.

[DirtyKeyboard]

what to do with missing frames

Maybe make the previous frame gray? That way, it's still clear the hour passed, but the data is missing. Or a graph without data.

The graph with no data sounds cool.  I think I've previously seen a blank ChartBuddy graph posted like you describe.  Maybe grey with a moonlay prototype, mentioned above, frame inserted.  Exciting times for sure.  Next rank I think I can ensmallen my signature characters.  We'll see.  FlyMangoFly

[DirtyKeyboard]

It's funny how things work sometimes, or don't work rather.  All week long I was ready to go full automated posting.  I've set up a laptop as a 'home posting machine', for if I'm out on the road at UTC midnight, when it all goes down.    Top20 list at 00:20 UTC, Buddy recap at 00:30 UTC  I can remote into the laptop using my phone, but it's tough to do much.  So Monday, was just lucky somehow, maybe because I rebooted Sunday night, but the laptop had a banner day.  See, this is what happens to the big developers.  They code the game on a supercomputer, and then expect it to run on general hardware.  The ChartBuddy recap scrip which has to load 24 images into GIMP was written on a computer with an SSD, while the laptop is on a much slower spinning disk drive.  So the overabundance of caution giving that step 20 seconds, to happen with the SSD, proved enough time, just once for the laptop.  We're kinda jumping around though, becuase I just figured out that was the problem today.  Right here is press to start loading images, making gifs, and then exporting.  It's worse than I thought.

Code:

pyautogui.hotkey('enter')
time.sleep(20)
# sequence to quit, and agree not to save
pyautogui.hotkey('ctrl', 'q')

I have since increased the time to 120 seconds, not 20.  
So Monday: shortly before the end of the hour, I checked on how it went with my phone.  I was very happy to see at least the CB Recap went off, I squinted a little closer at my phone and saw the Top20 list hadn't crashed, it was waiting for an input.  I remembered that I put in there to pause the code during error checking the previous night.  I was able to, click into the terminal and send an 'enter' over the wires, to have the script proceed and just made it under the wire.  What a world.  The end of the hour deadline is self imposed, but it would make things messy to run the ChartBuddy script an hour late.

Tuesday:  I had fixed the Top20 input problem, and that went off perfectly.  But then, you guessed it, the CB script threw a wierd error about how the help file wasn't installed and I need to choose the online version in the preferences or download a local copy.  It was like an ominous threat.  But thankfully the images had successfully downloaded and the gifs made.  I just reproduced the error and what happens is the files start loading, but hitting quit opens another menu.  The gifs get made, but GIMP never closes and all the pyautogui I use to post the post fall apart, because GIMP is opened fullscreen.  Luckily I've been in the crashing business for some time, and I knew that if I could somehow only run the second half of the script it should work fine.  Luckily the phone touch screen is okay at 'clicking and dragging' so, knowing I had backups, I deleted the already done steps, hit File, Save, and reran it successfully.  

Wednesday, Thursday, Friday:  Same story.  Autoposting rate 5/10 successful auto launches to completion, but I would say a solid 7.5 if giving partial credit.  10/10 deadlines met.

Saturday:  Do you see it coming?  I do some practice runs with the ChartBuddy script and notice how I was putting the bigger time.sleep in the wrong place.  I had it after pressing the keys to load the plug in, not the key to run the plug in.  Then I got to work changing the script that had worked all week.  In fairness, I was waiting til the weekend to change some tables, but 3 minutes before go time is not the time to be doing the final save before run.  See what had happened was, to test the Top20 BTC weeks Monday to Monday, months and years, it would just be easiest to move the incomplete data from this week, month, and year's partial data, run the script and then put the data back.  I then set those new scripts to only run if it is the first day of a new time period.  Except you don't put the data back, and you get bad tables.  Luckily my experience with crashing allowed me to grab the latest backup of all the files, fix a few things and rerun it from home base, 7 minutes late.  50 % Saturday grade.  I need this upcoming 100 % tomorrow for confidence going into next week.  Difficulty:  I want another new row of tables tomorrow.

I've got a new project as a tribute to my late friend Mango.  It's bringing me joy, and I'm looking forward to my next rank increase, when I think i can make the 'pixels' in my signature smaller than they are now.  But I've got my signature auto updating every hour.  Mango gets to fly across the screen everyday, and some fun stuff happens along the way occasionally.  As the sky grows behind him, it shrinks ahead. At the end of the signature  I set up a 3 by 7 'display' that is fairly easy to update.  The following is a blank display.  By changing sky_color to "green", that pixel will be green.

Code:

if utch == 18 or 19: # sky_color is a call to a dictionary of changing colors depending on hour of day
    a0 = sky_color; a1 = sky_color; a2 = sky_color; a3 = sky_color; a4 = sky_color; a5 = sky_color; a6 = sky_color
    b0 = sky_color; b1 = sky_color; b2 = sky_color; b3 = sky_color; b4 = sky_color; b5 = sky_color; b6 = sky_color
    c0 = sky_color; c1 = sky_color; c2 = sky_color; c3 = sky_color; c4 = sky_color; c5 = sky_color; c6 = sky_color

top_line = f"[b][color={sky_color}]{'█' * 2 * utch}███[/color]Mango      ####below
mid_line = f"[b][color={sky_color}]{'█' * 2 * utch}[/color]      Ascii
low_line = f"[b][color={sky_color}]{'█' * 2 * utch}███[/color]Art

[color={sky_color}]{'█' * (50 - (2 * utch))}[/color][/b]  #### then the display this is all on the same line actually

[color={a0}]█[/color][color={a1}]█[/color][color={a2}]█[/color][color={a3}]█[/color][color={a4}]█[/color][color={a5}]█[/color][color={a6}]█[/color][color={sky_color}]██[/color]"

I promised the latest ChartBuddy code last post I think, so here it is.  I think it's good for awhile.  I would be a lot less nervous if I could get the plug in to run with out the GIMP GUI involved, but all efforts have been in vain, although now...maybe...the headless batch command is not working because I need a longer pause in the plug in itself?
Secret Keys Removed

Code:

import os, pyautogui, pyperclip, re, requests, time, urllib.request, webbrowser
from datetime import datetime, timezone, timedelta
from os import rename

startTime = time.perf_counter()

# set UTC dates for links and new folder, so 'today' is around 30 minutes old when this is run
today = datetime.now(timezone.utc)
yesterday = today - timedelta(1)
print(today)

# get the final 20 gif layers in reverse order, starting with 24, if there is no cloudfare
number = 24
url4 = 'https://bitcointalk.org/index.php?action=profile;u=110685;sa=showPosts;start=0'
time.sleep(20)
response = requests.get(url4)

# turn page source of ChartBuddy's latest posts page into textfile.
source_code = response.text
textfile = open('C:/PyProjects/CB/Temp/CBSource.txt', 'w+')
textfile.write(source_code)
textfile.seek(0)
filetext = textfile.read()
textfile.close()

# find matches using regex for 1 image posted today at around 00:02 UTC...
pattern1 = r'https:\/\/www\.talkimg\.com\/images\/\w{{4}}\/\w{{2}}\/{0}\/\w{{5}}\.png'.format(today.strftime('%d'))
print(pattern1)

# and from yesterday, set a counter for total downloads, plenty of copilot help with my regex
pattern2 = r'https:\/\/www\.talkimg\.com\/images\/\w{{4}}\/\w{{2}}\/{}\/\w{{5}}\.png'.format(yesterday.strftime('%d'))
print(pattern2)
matches = re.findall(pattern1, filetext) + re.findall(pattern2, filetext)

# python is so cool you can just cycle though stuff, without setting counters...
total_dl =  0
for link in matches:
    dl_number = f"{number:02d}"
    total_dl += 1

    # status update, and the GIMP plug in wants these names, will rename below
    print(number, link)
    urllib.request.urlretrieve(link, 'C:/PyProjects/CB/images/download ({}).png'.format(dl_number))
    number = number - 1

    # wait between downloads to not stress talkimg.com.  I did work for a donation to them :)
    time.sleep(5)

# something I'm saving, why not?  might be useful one day...
os.remove('C:/PyProjects/CB/Temp/CBSource.txt')
print("going on")

# get the first 4 images on the next page, which are the last frames of the gif
url5 = 'https://bitcointalk.org/index.php?action=profile;u=110685;sa=showPosts;start=20'
time.sleep(20)
response5 = requests.get(url5)
source_code = response5.text
textfile5 = open('C:/PyProjects/CB/Temp/CBSource2.txt', 'a+')
textfile5.write(source_code)
textfile5.seek(0)
filetext2 = textfile5.read()
textfile5.close()

# these could only be from yesterday, since 'today' is only 30 minutes old
matches = re.findall(pattern2, filetext2) 
for link in matches:
    if number >= 1:
        dl_number = f"{number:02d}"
        total_dl += 1
        print(number, link)
        urllib.request.urlretrieve(link, 'C:/PyProjects/CB/images/download ({}).png'.format(dl_number))
        number = number - 1
        time.sleep(2)
    if number <= 0:
        print("skipping link")
os.remove('C:/PyProjects/CB/Temp/CBSource2.txt')
time.sleep(5)

# hot keys for gif making, main monitor focusing click
pyautogui.click(1, 1)
time.sleep(5)

# open GIMP, be paitient
pyautogui.hotkey('ctrl', 'alt', 'g')
time.sleep(60)
pyautogui.hotkey('f11')
time.sleep(20)

# click on middle of GIMP window to be sure
pyautogui.click(820, 446)
time.sleep(20)

# hotkeys to plugin that loads layers, exports, resizes
pyautogui.hotkey('ctrl', 'alt', 'l')
time.sleep(20)

# need to learn GUIless GIMP, keys to finalize plug
pyautogui.hotkey('tab')
time.sleep(5)
pyautogui.hotkey('tab')
time.sleep(5)
pyautogui.hotkey('tab')
time.sleep(5)
pyautogui.hotkey('enter')
time.sleep(120)

# sequence to quit, and agree not to save
pyautogui.hotkey('ctrl', 'q')
time.sleep(10)
pyautogui.hotkey('shift', 'tab')
time.sleep(5)
pyautogui.hotkey('enter')
time.sleep(60)

# uploading big gif and getting link to use later, 
url = "https://api.imgur.com/3/image"
payload = {'name': f'{yesterday:%m}-{yesterday:%d}-{yesterday.year}'}
files=[('image',('gif.gif',open('C:/PyProjects/CB/Temp/gif.gif','rb'),'image/gif'))]
headers = {'Authorization': 'Bearer xxxxxxxxxxxxxxxxxxxx'}
response = requests.post(url, headers=headers, data=payload, files=files)
data = response.json()
imgur_big_gif = data.get("data", {}).get("link")

# uploading talkimg gif and getting link to use later
url = "https://talkimg.com/api/1/upload"
headers = {"X-API-Key": "xxxxxxxxxxxxxxxxxxxxxxxxxxx"}
files = {"source": open("C:/PyProjects/CB/Temp/gif2.gif", "rb")}
payload = {"title": f'{yesterday:%m}-{yesterday:%d}-{yesterday.year}', "album_id": "UFbj"}
response = requests.post(url, headers=headers, data=payload, files=files)
data = response.json()
talkimg_gif = data["image"]["url"]

# add post to clipboard for btctalk
pyperclip.copy(f"ChartBuddy's Daily Wall Observation recap\n[url={imgur_big_gif + "v"}].[img]{talkimg_gif}[/img].[/url]\nAll Credit to [url=https://bitcointalk.org/index.php?topic=178336.msg10084622#msg10084622]ChartBuddy[/url]")

# can use this link for the reply button, keys to get to post entry box
url7 = 'https://bitcointalk.org/index.php?action=post;topic=178336.0'
webbrowser.open(url7)
time.sleep(60)
pyautogui.hotkey('f11') 
time.sleep(20)
pyautogui.hotkey('tab')
time.sleep(5)
pyautogui.hotkey('tab')
time.sleep(5)

# paste the post and head to post or preview button
pyautogui.hotkey('ctrl', 'v')
time.sleep(5)
pyautogui.hotkey('tab')
time.sleep(5)

# we're doing it live if the next command is #ed out
pyautogui.hotkey('tab')
time.sleep(5)
pyautogui.hotkey('enter')
time.sleep(5)
pyautogui.hotkey('f11') 
time.sleep(20)

# name new image folder with date, if not already created
directory = f"{yesterday.year}/{yesterday.year}-{yesterday:%m}/"
parent_dir = "C:/PyProjects/CB/CBuddyDaily/"
newf = os.path.join(parent_dir, directory)
present = os.path.exists(newf)
if not newf:
    os.mkdir(newf)
src = "C:/PyProjects/CB/images/"
files = os.listdir(src)
os.chdir(src)
image = 1
for file in files:

    # this names the images yyyy-mm-dd (01) through (24), making the most recent image hour 24 because its the last hour, not the first hour
    name = os.path.join(src, file)
    rename (name, f"C:/PyProjects/CB/CBuddyDaily/{yesterday.year}/{yesterday.year}-{yesterday:%m}/{yesterday.year}-{yesterday:%m}-{yesterday:%d}({image:02d}).png")
    image += 1

giff = f"C:/PyProjects/CB/gifs/{yesterday.year}/{yesterday.month}-{yesterday.year}/"
present = os.path.exists(giff)
if not giff:
    os.mkdir(giff)
rename ("C:/PyProjects/CB/Temp/gif.gif", f"C:/PyProjects/CB/gifs/{yesterday.year}/{yesterday.month}-{yesterday.year}/b{yesterday:%y}-{yesterday:%m}-{yesterday:%d}.gif")
rename (f"C:/PyProjects/CB/Temp/gif2.gif", f"C:/PyProjects/CB/gifs/{yesterday.year}/{yesterday.month}-{yesterday.year}/{yesterday:%y}-{yesterday:%m}-{yesterday:%d}.gif")




###########                           ☰☰☰             ▟█
# The End                              ☰☰☰☰☰☰  ⚞▇▇▇██▛(°)>
###########                           ☰☰☰             ▜█

[DirtyKeyboard]

All righty then.  I've got ChartBuddy with extended pauses ready to go. I've got the 4th table in the Top20 Days of Bitcoin thread ready to go, and have put the newest data back, unlike yesterday.  I don't have the Mango signature totally figured out, and it's breaking my brain.  I can not figure out why some hours engage the 'display' and some just fill it with sky_color.  Here is the rising moon in the 'display' portion of the sig.

███████████████████████████████████████████            ▟██           ███████████████████
████████████████████████████████████████        ⚞▇▇▋███▎▎(⚬)>       █████████████████
███████████████████████████████████████████            ▜██           ███████████████████ 
If you scroll to the end you can see the two white 'pixels'.  This is a working example, but most of the rest just take the sky_color variable, and ignore the color command if present.

Code:

if utcu == 20:
    a0 = sky_color; a1 = sky_color; a2 = sky_color; a3 = sky_color; a4 = sky_color; a5 = sky_color; a6 = sky_color
    b0 = sky_color; b1 = sky_color; b2 = sky_color; b3 = sky_color; b4 = sky_color; b5 = sky_color; b6 = sky_color
    c0 = sky_color; c1 = sky_color; c2 = sky_color; c3 = "white"; c4 = "white"; c5 = sky_color; c6 = sky_color

Code:

[b][color=#114054]███████████████████████████████████████████[/color]            [color=green]▟██[/color]           [color=#114054]██████████[/color][/b][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]██[/color]
[b][color=#114054]████████████████████████████████████████[/color]  [color=green]      ⚞[/color][color=#0197d2]▇▇[/color][color=green]▋███[/color][color=#e7cc3d]▎▎[/color][color=#dd7e3e]([/color]⚬[color=orange])[/color]>       [color=#114054]████████[/color][/b][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]██[/color]
[b][color=#114054]███████████████████████████████████████████[/color]            [color=green]▜██[/color]           [color=#114054]██████████[/color][/b][color=#114054]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=white]█[/color][color=white]█[/color][color=#114054]█[/color][color=#114054]█[/color][color=#114054]██[/color] 

This and the next few hours work, but I've got more code checking to do.

So I've got that to chew on, and apparently I'm going to try get a script to scrape the 100 Push-Ups A Day Until Bitcoin Is $100K Challenge thread, to keep track of everyones' pushups.  It sounds like how I search for ChartBuddy's posts in the Page Source, I should be able to key in on this data as well. I've asked them to post like
"format:
100kchallenge,UserName,daysreporting,pushupscompleted

example:
100kchallenge,JayJuanGee,63,10550"
I'm sure that is going to end well.     And now JayJuanGee wants to include the date, that sure simplifies things.    Come join us everyone!

[DirtyKeyboard]

Missed it by that much.  {editor: that was supposed to be 6 pt font, but we'll let it run because reasons}  
I changed the Task Schedule times, and the scripts overlapped.  So the ChartBuddy script ran great starting at 00:15 UTC, and then the Top20 script started at 00:20 UTC, which is not enough of a delay.  What was I thinking?  

I checked the status from my phone at  00:30 UTC and was happy to see the recap posted, but the script that had been flawless the week prior didn't work.  I could see through my phone the terminal had passed the message "No Table 44U" which happens when it is not a new year, month, or week, so don't post those stats.  Wow this got complicated.

So when I figured out how to 'Ctrl'-'v' over the phone, the ChartBuddy post showed up.  Well, short story long, I got home, and simply had to rerun the parts of the script that aren't downloaded.  Part of the script downloads the data and formats the columns, the other part of the script takes the columns and makes the tables.  Updated main table maker with date choice posted at end.

For the newest pushup counter script I went head long in to copy paste mode with Copilot and came up with this last night, plus one change by me.    The tabulate python script wasn't accepting the decimal place inputs i believe I was inputting.  The average pushups column kept coming up with too many decimal places.  Well I'm chuffed i was able to make this change to the script to have it look how I wanted.

Code:

average_pushup = pushups_completed / days_reporting
        average_pushups = f"{average_pushup:.2f}"

The copilot suggested code was just passing along the average_pushups variable originally, which I renamed average_pushup, so I could modify the variable average_pushups and not have to change anything else.
Pushup script

Code:

import csv, json, os, pyautogui, pyperclip, re, requests, time, urllib.request, webbrowser
from datetime import datetime, timezone, timedelta
from os import rename
from tabulate import tabulate

# set UTC dates for links and new folder, so 'today' is around 30 minutes old when this is run
today = datetime.now(timezone.utc)
yesterday = today - timedelta(1)
print(today)

json_filename = "C:/PyProjects/PU/PU/latest_data.json"
with open(json_filename, "r") as jsonfile:
    latest_data = json.load(jsonfile)

url = 'https://bitcointalk.org/index.php?topic=5484350.740'
time.sleep(5)
response = requests.get(url)

# turn page source of Pushups page into textfile.
source_code = response.text
textfile = open('C:/PyProjects/PU/PUSource.txt', 'w+', encoding="UTF-8")
textfile.write(source_code)
textfile.seek(0)
filetext = textfile.read()
textfile.close()

# Define the regex pattern for 100kchallenge
pattern = r"100kchallenge,(\w+),(\d+),(\d+),(\d{4}-\d{2}-\d{2})"
matches = re.findall(pattern, filetext)
print(matches)
# input()
csv_data = []

# Load the latest_data dictionary from the JSON file (if it exists)
json_filename = "C:/PyProjects/PU/PU/latest_data.json"
try:
    with open(json_filename, "r") as jsonfile:
        latest_data = json.load(jsonfile)
except FileNotFoundError:
    # If the file doesn't exist, start with an empty dictionary
    latest_data = {}
    

# for username, data in latest_data.items():
#     days_reporting, pushups_completed, _, = data[1:]  # Ignore the date
#     average_pushups = int(pushups_completed) / int(days_reporting)
#     latest_data[username] = (*data, average_pushups)

# Process each match
for match in matches:
    username, days_reporting, pushups_completed, date = match
    if username not in latest_data or date > latest_data[username][3]:
        latest_data[username] = (username, days_reporting, pushups_completed, date)

# Write the data to a CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
    writer = csv.writer(csvfile)
    for entry in latest_data.values():
        writer.writerow(entry)

print(f"Data written to {csv_filename}")
# Read data from the CSV file

json_filename = "C:/PyProjects/PU/PU/latest_data.json"
with open(json_filename, "w") as jsonfile:
    json.dump(latest_data, jsonfile)

print(f"Latest data saved to {json_filename}")

# read data from csv, and calculate average pushups per day
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        # calculate average pushups per day place it in the row
        days_reporting, pushups_completed = int(row[1]), int(row[2])
        average_pushup = pushups_completed / days_reporting
        average_pushups = f"{average_pushup:.2f}"
        row.append(average_pushups)
        rows.append(row)

# Sort the rows by pushups completed (descending order)
sorted_rows = sorted(rows[0:], key=lambda x: int(x[2]), reverse=True)

for i, row in enumerate(sorted_rows):
    row.insert(0, i + 1)

new_csv_filename = "C:/PyProjects/PU/100kpushups_rank.csv"
with open(new_csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    for entry in sorted_rows:
        writer.writerow(entry)


rows = []
with open(new_csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Print the data as a table
headers = ["Order", "Username", "Days In", "Pushups", "Last Date", "Daily Average"]
print(tabulate(rows[0:], headers=headers, tablefmt="rounded_grid"))
post = (tabulate(rows[0:], headers=headers, tablefmt="rounded_grid"))
pyperclip.copy(f"[pre]{post}[/pre]")













###########                            ☰☰☰        ▟█
# The End #                       ☰☰☰☰☰☰  ⚞▇▇▇██▛(°)>
###########                            ☰☰☰        ▜█

Current Top20 Main Tablemaker code with optional 4th table highlighting the all time Top20 Years, Months, and Weeks of BTC

Code:

if todayutc.isoweekday() == 1 or todayutc.day == 1 or todayutc.month == 1 and todayutc.day == 1:
        import t4_table

Code:

import datetime, pyautogui, pyperclip, shutil, time, webbrowser
from datetime import datetime, timezone

todayutc = datetime.now(timezone.utc)

# sets a sleep timer for faster runs during testing
sleep_time = 15

# calls the scripts which creat each column to be zipped together below
import Top20USD
time.sleep(sleep_time)
import Top20GBP
time.sleep(sleep_time)
import Top20EUR
time.sleep(sleep_time)
import Top20CAD
time.sleep(sleep_time)
import Top20JPY
time.sleep(sleep_time)

# location of the columns created above
usd = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_USD/top100usd.txt'
gbp = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_GBP/top100gbp.txt'
eur = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_EUR/top100eur.txt'
cad = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_CAD/top100cad.txt'
jpy = 'C:/PyProjects/Top20/Top20_Total_03_24/VWAP_JPY/top100jpy.txt'

file_paths = [usd, gbp, eur, cad, jpy]
print("importingfiles")
with open('C:/PyProjects/Top20/Top20_Total_03_24/top20table.txt', 'w', encoding='utf-8') as output_file:

    # Read lines from all files simultaneously
    with open(file_paths[0], 'r', encoding='utf-8') as file1, \
            open(file_paths[1], 'r', encoding='utf-8') as file2, \
            open(file_paths[2], 'r', encoding='utf-8') as file3, \
            open(file_paths[3], 'r', encoding='utf-8') as file4, \
            open(file_paths[4], 'r', encoding='utf-8') as file5:
        
        for line1, line2, line3, line4, line5 in zip(file1, file2, file3, file4, file5):

            # Write combined lines to the output file
            output_line = f"| {line1.strip()}| {line2.strip()}| {line3.strip()}| {line4.strip()}| {line5.strip()}|\n"
            output_file.write(output_line)
# progress check
print("clipboarding")

# putting the trad5 part together, setting links as variables to limit code line length
with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20table.txt', 'r') as top100:
    ul = 'https://bitcoincharts.com/charts/bitstampUSD'
    gl = 'https://data.bitcoinity.org/markets/volume/5y/GBP/kraken?r=day&t=b'
    el = 'https://data.bitcoinity.org/markets/volume/5y/EUR/kraken?r=day&t=b'
    cl = 'https://data.bitcoinity.org/markets/volume/5y/CAD/kraken?r=day&t=b'
    jl = 'https://data.bitcoinity.org/markets/volume/5y/JPY/kraken?r=day&t=b'
    list = top100.read()
    pre = f"[pre]   US Dollar              British Pound          EU Euro                Canadian Dollar        Japanese Yen"
    prelude = f"[size=9pt][b]| [url={ul}]Rank BitStamp   USD/BTC[/url]| [url={gl}]Rank Kraken     GBP/BTC[/url]| [url={el}]Rank Kraken     EUR/BTC[/url]| [url={cl}]Rank Kraken     CAD/BTC[/url]| [url={jl}]Rank Kraken        JPY/BTC[/url]|[/b]"
    explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                                                    *  Chart  Legends  *                                                       |[/url]"
    JimboToronto = "|                                                          GoBTCGo™                                                             |[/pre]"
    full_post1 = f"{pre}\n{prelude}\n{list}{explanation}\n{JimboToronto}"

# on to the next set of tablesk, this script calls 4 seperate currency scripts
import tm_T2_03_21

# making the second table 
with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20tableT2.txt', 'r') as top100T2:
   
    il = 'https://data.bitcoinity.org/markets/volume/5y/IDR/bitcoin.co.id?r=day&t=b'
    kl = 'https://data.bitcoinity.org/markets/volume/5y/KRW/korbit?r=day&t=b'
    al = 'https://data.bitcoinity.org/markets/volume/5y/AUD/btcmarkets?r=day&t=b'
    ual = 'https://data.bitcoinity.org/markets/volume/5y/UAH/exmo?r=day&t=b'
    bv = 'https://data.bitcoinity.org/markets/volume/5y/IDR/bitcoin.co.id?r=day&t=b'
    listT2 = top100T2.read()
    pre = f"[pre]  Korean Wan               Australian Dollar    Ukrainian Hryvnia      Indonesian Rupiah          Bitstamp Exchange"
    prelude = f"[size=9pt][b]| [url=https://data.bitcoinity.org/markets/volume/5y/KRW/korbit?r=day&t=b]#  Korbit         KRW/BTC[/url]| [url=https://data.bitcoinity.org/markets/volume/5y/AUD/btcmarkets?r=day&t=b]#  btcmarkets AUD/BTC[/url]| [url=https://data.bitcoinity.org/markets/volume/5y/UAH/exmo?r=day&t=b]#  exmo         UAH/BTC[/url]| [url=https://data.bitcoinity.org/markets/volume/5y/IDR/bitcoin.co.id?r=day&t=b]#  bitcoin.co.id    IDR/BTC[/url]| [url=https://bitcoincharts.com/charts/bitstampUSD]#  BTC vol  in MUSD[/url]|[/b]"
    explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                                                    *  Chart  Legends  *                                                    |[/url]"
    JimboToronto = "|                                                          GoBTCGo™                                                          |[/pre]"
    full_post2 = f"{pre}\n{prelude}\n{listT2}{explanation}\n{JimboToronto}"

# the set of newest moving day average columns
import tm_T3_03_30

with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20tableT3.txt', 'r') as top100T3:
    ul = 'https://bitcoincharts.com/charts/bitstampUSD'
    listT3 = top100T3.read()
    pre = f"[pre]Moving Average VWAP, date listed is final day in span\n|  7 Day MA              30 Day MA            200 Day MA            100 Week MA            200 Week MA         |"
    prelude = f"[size=9pt][b]|  [url={ul}]#      7 DMA  USD/BTC[/url]|  [url={ul}]#      30 DMA USD/BTC[/url]| [url={ul}]#      200 DMA USD/BTC[/url]| [url={ul}]#      100 WMA USD/BTC[/url]|  [url={ul}]#     200 WMA USD/BTC[/url]|[/b]"
    explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                                                  *  Chart  Legends  *                                                 |[/url]"
    JimboToronto = "|                                                        GoBTCGo™                                                       |[/pre]"
    full_post3 = f"{pre}\n{prelude}\n{listT3}{explanation}\n{JimboToronto}"

    if todayutc.isoweekday() == 1 or todayutc.day == 1 or todayutc.month == 1 and todayutc.day == 1:
        import t4_table

        with open('C:/PyProjects/Top20/Top20_Total_03_24/Top20tableT4.txt', 'r') as top100T4:
            ul = 'https://bitcoincharts.com/charts/bitstampUSD'
            listT4 = top100T4.read()
            pre = f"[pre]It must be a new Year, Month, or Week!\nVWAP for Calendar ranges = sum of vwap for days in range / # of days in range.\n  Top Year        Top Month          Top Week ending on"
            prelude = f"[size=9pt][b]| [url={ul}]#        USD/BTC[/url]| [url={ul}]#           USD/BTC[/url]| [url={ul}]#              USD/BTC[/url]|[/b]"
            explanation = "[url=https://bitcointalk.org/index.php?topic=138109.msg54917391#msg54917391]|                     *  Chart  Legends  *                     |[/url]"
            JimboToronto = "|                           GoBTCGo™                           |[/pre]"
            full_post4 = f"{pre}\n{prelude}\n{listT4}{explanation}\n{JimboToronto}"
            full_post = f"Daily [BTC] Volume Weighted Average Prices\n{full_post1}\n{full_post2}\n{full_post3}\n{full_post4}"
    else:
        print("No table 44U!")
        full_post = f"Daily [BTC] Volume Weighted Average Prices\n{full_post1}\n{full_post2}\n{full_post3}"
    
    pyperclip.copy(full_post)

    # can use this link for the reply page to top20 thread
    url = 'https://bitcointalk.org/index.php?action=post;topic=138109.0'
    webbrowser.open(url)
    time.sleep(60)
    pyautogui.hotkey('f11')
    time.sleep(20)
    pyautogui.hotkey('tab')
    time.sleep(2)
    pyautogui.hotkey('tab')
    time.sleep(2)
    pyautogui.hotkey('ctrl', 'v')
    time.sleep(2)
    pyautogui.hotkey('tab')
    time.sleep(2)
    # we're doing it live if the next command is #ed out
    # pyautogui.hotkey('tab')
    time.sleep(5)
    pyautogui.hotkey('enter')
    time.sleep(60)
    pyautogui.hotkey('f11')
    
# if i don't delete the cache each time, the minor changes I make don't show up the next run, i think
shutil.rmtree('C:/PyProjects/Top20/Top20_Total_03_24/__pycache__')


















# ###########                            ☰☰☰         ▟██
# # The End #                       ☰☰☰☰☰☰  ⚞▇▇▇▋███▎▎(⚬)>
# ###########                            ☰☰☰         ▜██

[DirtyKeyboard]

Greetings and salutations, what a beautiful day for a code run, let's run two.    I love it when a plan comes together-Hannibal, A-Team.
 I'd say, I'm not touching a thing, but...the Top20 script shows the top year, month, and week of BTC vwap, only on the first day after the respective periods.  But with the 4th table the post hits the character limit.  Might have to lose the niceties and sidebars, do spaces count?

I mentioned heavily leaning on Co-Pilot for the push up code and figured someone might want to read the conversation.

Code:

## You
We've picked up another project.  My friend convinced people to exercise and report their pushups to a public forum, 
then I will use Python to gather and put the data into a table.  I was looking for help to develop the regex expression search, 
starting with the key 100kpushups, from the souce page of the forum.  I figured we could turn that into a csv pretty easily.   
Here is the template following by the current code, template = "format:
100kchallenge,UserName,daysreporting,pushupscompleted,Date-Reporting-In(yyyy-mm-dd)
example:
100kchallenge,DirtyKeyboard,57,5081,2024-04-06"   starting code = '''import os, pyautogui, pyperclip, re, requests, time, urllib.request, webbrowser
from datetime import datetime, timezone, timedelta
url4 = 'https://bitcointalk.org/index.php?topic=5484350.720'
time.sleep(20)
response = requests.get(url4)
turn page source of forum into pagesourcesource_code = response.text
textfile = open('C:/PyProjects/PU/Temp/PUSource.txt', 'w')
textfile.write(source_code)
textfile.seek(0)
filetext = textfile.read()
textfile.close()
find matches using regex for 1 image posted today at around 00:02 UTC...pattern1 = r"100kpushups, \w{{.}}  
#then only numbers,  only numbers, and date only in yyyy-mm-dd format
print(pattern1)'''
## Copilot
Certainly! Let's create a regular expression (regex) pattern to extract the relevant information from the forum source page related to the 100kpushups challenge. Based on your template, we'll look for lines that match the following format:

100kchallenge, UserName, daysreporting, pushupscompleted, Date-Reporting-In(yyyy-mm-dd)

Here's the regex pattern you can use:

import re

# Define the regex pattern for 100kpushups
pattern = r"100kpushups, (\w+), (\d+), (\d+), (\d{4}-\d{2}-\d{2})"

# Example source text (you can replace this with the actual source code from the forum)
source_text = """
100kpushups, DirtyKeyboard, 57, 5081, 2024-04-06
100kpushups, FitGuru, 42, 3750, 2024-04-06
"""

# Find all matches using the regex pattern
matches = re.findall(pattern, source_text)

# Print the matches
for match in matches:
    print(f"Username: {match[0]}, Days Reporting: {match[1]}, Pushups Completed: {match[2]}, Date: {match[3]}")

When you run this code, it will extract the relevant data from the source_text and print it out. You can replace the source_text with the actual content from the forum page. Adjust the file paths and other details as needed for your specific use case.
Feel free to modify the pattern or adapt it further based on any variations you encounter in the forum data. Let me know if you need any additional assistance! 
## You
That's great, thank you!  Instead of printing to terminal could we start a dictionary entry for each unique UserName?  Or another way might be to write the matches to a 4 column csv first?
## Copilot
Certainly! Let's organize the data either into a dictionary or write it to a CSV file.
- Dictionary Approach:
We can create a dictionary where each unique UserName is a key, and the corresponding value is a list of tuples containing (days_reporting, pushups_completed, date) for that user. Here's how you can modify the code to achieve this:

import re
import csv

# Define the regex pattern for 100kpushups
pattern = r"100kpushups, (\w+), (\d+), (\d+), (\d{4}-\d{2}-\d{2})"

# Example source text (you can replace this with the actual source code from the forum)
source_text = """
100kpushups, DirtyKeyboard, 57, 5081, 2024-04-06
100kpushups, FitGuru, 42, 3750, 2024-04-06
"""

# Initialize a dictionary to store data for each unique UserName
user_data = {}

# Find all matches using the regex pattern
matches = re.findall(pattern, source_text)

# Process each match
for match in matches:
    username, days_reporting, pushups_completed, date = match
    if username not in user_data:
        user_data[username] = []
    user_data[username].append((days_reporting, pushups_completed, date))

# Print the dictionary entries
for username, data in user_data.items():
    print(f"Username: {username}")
    for entry in data:
        print(f"  Days Reporting: {entry[0]}, Pushups Completed: {entry[1]}, Date: {entry[2]}")

# Write the data to a CSV file
csv_filename = "100kpushups_data.csv"
with open(csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(["Username", "Days Reporting", "Pushups Completed", "Date"])
    for username, data in user_data.items():
        for entry in data:
            writer.writerow([username, entry[0], entry[1], entry[2]])

print(f"Data written to {csv_filename}")

- CSV Approach:
If you prefer a CSV file directly, we can write the matches to a 4-column CSV file. The code above already includes writing the data to a CSV file named 100kpushups_data.csv.
Feel free to choose the approach that best suits your needs! 
## You
That works great.  So the dictionary is not permanent in anyway?, but I have the csv file to be appended or updated. I see a problem though, someone quoted my format in the thread and so can we deal with identical entries not being written to the csv?
## Copilot
Certainly! Let's address both concerns:
- Dictionary Approach (Non-Permanent):
The dictionary approach I provided is not permanent. It exists only during the execution of your Python script. If you close the script or restart your program, the dictionary will be lost. However, you can always recreate it by processing the data again from the source text.
- Avoiding Identical Entries in CSV:
To prevent identical entries from being written to the CSV file, we can keep track of the unique usernames encountered. If a username has already been added to the CSV, we'll skip adding it again. Here's the modified code:

import re
import csv

# Define the regex pattern for 100kpushups
pattern = r"100kpushups, (\w+), (\d+), (\d+), (\d{4}-\d{2}-\d{2})"

# Example source text (you can replace this with the actual source code from the forum)
source_text = """
100kpushups, DirtyKeyboard, 57, 5081, 2024-04-06
100kpushups, FitGuru, 42, 3750, 2024-04-06
100kpushups, DirtyKeyboard, 58, 5120, 2024-04-07
"""

# Initialize a set to store unique usernames
unique_usernames = set()

# Initialize a list to store data for writing to CSV
csv_data = []

# Find all matches using the regex pattern
matches = re.findall(pattern, source_text)

# Process each match
for match in matches:
    username, days_reporting, pushups_completed, date = match
    if username not in unique_usernames:
        unique_usernames.add(username)
        csv_data.append([username, days_reporting, pushups_completed, date])

# Write the data to a CSV file
csv_filename = "100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
    writer = csv.writer(csvfile)
    for entry in csv_data:
        writer.writerow(entry)

print(f"Data written to {csv_filename}")

In this modified code:
    - We use a set called unique_usernames to keep track of usernames encountered.
    - If a username is already in the set, we skip adding it to the CSV.
    - Otherwise, we add the username to the set and append the data to the csv_data list.
Now, the CSV file will only contain unique entries based on the usernames. Feel free to adjust the code further to fit your specific requirements! 
## You
Okay, so far so good.  Now I'm trying to repurpose some for our code to rank the users by greatest pushups done.  Is this close?  '''print(f"Data written to {csv_filename}")
Read data from the CSV file for ranking pushups donerows = []
with open(csv_filename, 'r') as file:
reader = csv.reader(file)
for row in reader:
tuples = [(username, days_reporting, pushups_completed, date, i + 1) for i, (username, days_reporting, pushups_completed, date) in enumerate(rows)]
sorted_tuples = sorted(tuples, key=lambda x: float(x[1]), reverse=True)'''
## Copilot
Your approach is on the right track! However, there are a couple of adjustments needed to achieve the desired ranking of users by the greatest number of pushups completed. Let's make the necessary modifications:
- Reading Data from CSV and Ranking Users:
We'll read the data from the CSV file and rank users based on the number of pushups completed. Here's the updated code snippet:

import csv

# Read data from the CSV file
csv_filename = "100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Sort the rows by pushups completed (descending order)
sorted_rows = sorted(rows[1:], key=lambda x: int(x[2]), reverse=True)

# Print the sorted rows
print("\nRanking by Pushups Completed:")
for i, row in enumerate(sorted_rows):
    print(f"{i+1}. {row[0]}: {row[2]} pushups")

In this code:
    - We read the data from the CSV file into the rows list.
    - We sort the rows based on the number of pushups completed (column index 2) in descending order.
    - Finally, we print the sorted rows along with the user rankings.
- Handling Duplicate Usernames:
If there are duplicate usernames in the CSV (as you mentioned someone quoted your format), the code above will handle them correctly. It will rank users based on the total pushups completed, even if there are multiple entries for the same username.
Feel free to incorporate this code snippet into your existing script, and it should give you the desired ranking of users by pushups completed! 
## You
Look what i added :)  '''for i, row in enumerate(sorted_rows):
print(f"{i+1}. {row[0]}: {row[2]} pushups done in {row[1]} day(s).")'''  And don't think I didn't notice that FitGuru has a slightly higher pushups per day average than I do.  Haha. :D  Was that on purpose?
## Copilot
Certainly! I see you've added a nice touch to display the rankings with the number of pushups done and the corresponding days reported. 
As for FitGuru's slightly higher pushups per day average, well, it seems even in the world of pushups, there's friendly competition! FitGuru might have been secretly doing extra reps when no one was looking.  Keep up the great work, and remember, consistency is key! ️‍♂️
If you need any more assistance or have further questions, feel free to ask! 
## You
Oh wow, can we use python tabulate with the pushups csv file or do we need a panda.dataframe to auto make a table?
## Copilot
Certainly! You have a couple of options for creating a table from your CSV data:
- Using tabulate:
The tabulate library is a great choice for creating formatted tables from tabular data. It works well with lists of lists (which you can extract from your CSV data). Here's how you can use it:

from tabulate import tabulate
import csv

# Read data from the CSV file
csv_filename = "100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Print the data as a table
headers = ["Username", "Days Reporting", "Pushups Completed", "Date"]
print(tabulate(rows[1:], headers=headers, tablefmt="grid"))

Make sure you have the tabulate library installed (pip install tabulate) before running this code.
- Using pandas DataFrame:
If you prefer working with DataFrames, you can use the pandas library. It provides more advanced data manipulation capabilities. Here's an example using pandas:

import pandas as pd

# Read data from the CSV file into a DataFrame
csv_filename = "100kpushups_data.csv"
df = pd.read_csv(csv_filename)

# Print the DataFrame (optional)
print(df)

# If you want to display it as a table:
print(df.to_string(index=False))

Make sure you have the pandas library installed (pip install pandas) before running this code.
Choose the approach that best fits your needs. Both methods allow you to create nicely formatted tables from your CSV data! 
## You
Amazing again.  Thank you!  But it only seems to work on the second run, which I think means during the sort it is skipping the header row, but the csv file has no headers.  Do you think we could add matching headers during this step?  '''# Process each match
for match in matches:
username, days_reporting, pushups_completed, date = match
if username not in unique_usernames:
unique_usernames.add(username)
csv_data.append([username, days_reporting, pushups_completed, date])
Write the data to a CSV filecsv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
writer = csv.writer(csvfile)
for entry in csv_data:
writer.writerow(entry)'''
## Copilot
Certainly! To address the issue of skipping the header row during sorting, we can add the headers to the CSV file before writing the data. Here's the modified code snippet:

import csv

# Define the headers for the CSV file
csv_headers = ["Username", "Days Reporting", "Pushups Completed", "Date"]

# Read data from the CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Sort the rows by pushups completed (descending order)
sorted_rows = sorted(rows[1:], key=lambda x: int(x[2]), reverse=True)

# Write the data to a new CSV file with headers
new_csv_filename = "C:/PyProjects/PU/100kpushups_data_with_headers.csv"
with open(new_csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(csv_headers)  # Write the headers
    for entry in sorted_rows:
        writer.writerow(entry)

print(f"Data written to {new_csv_filename}")

In this modified code:
- We define the csv_headers list containing the column names.
- Before writing the data to the new CSV file, we write the headers using writer.writerow(csv_headers).
- The rest of the code remains the same.
Now the new CSV file will have headers, and the sorting should work as expected. Feel free to give it a try, and let me know if you need any further assistance! 
## You
Super, but this is confusing me.  Does the Terminal in Visual Studio have some persistent memory?  I ran your code and it turned out great.  But then I decide to shorten the headers to just Pushups, deleted the csv file.  Reran the script and this came out on the terminal "| Username      |   Days Reporting |   Pushups Completed | Date       |
+===============+==================+=====================+============+
| DirtyKeyboard |               57 |                5081 | 2024-04-06 |
+---------------+------------------+---------------------+------------+"  which is cool, but the headers didn't change.  But in the new csv file, the headers are changed.  What should I be doing between script runs to clear all memory before runs?
## Copilot
The behavior you're experiencing is likely due to the way Python handles file I/O and memory. Let me explain what's happening:
- File I/O and CSV Headers:
    - When you read data from a CSV file using csv.reader, it reads the entire file (including the header row) into memory as a list of lists (each inner list representing a row).
    - When you sort the rows, you're sorting the entire list (including the header row) based on the specified column.
    - When you write the sorted data back to a new CSV file, it includes the header row from the original data.
- Memory Persistence:
    - Python scripts run in a single session, and variables and data persist in memory until the script terminates.
    - When you re-run the script, any data loaded into memory during the previous run (including the header row) remains accessible.
    - Deleting the CSV file doesn't clear the data from memory; it only removes the file from disk.
- Solution:
    - To ensure that the header row is updated when you modify the headers, you can explicitly write the headers to the new CSV file before writing the data.
    - Additionally, consider using a separate variable for the data (excluding the header row) to avoid mixing it with the header during sorting.
Here's an updated version of the code that addresses these points:

import csv

# Define the headers for the CSV file
csv_headers = ["Username", "Days Reporting", "Pushups Completed", "Date"]

# Read data from the CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Sort the rows by pushups completed (excluding the header row)
sorted_rows = sorted(rows[1:], key=lambda x: int(x[2]), reverse=True)

# Write the data to a new CSV file with headers
new_csv_filename = "C:/PyProjects/PU/100kpushups_data_with_headers.csv"
with open(new_csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(csv_headers)  # Write the headers
    for entry in sorted_rows:
        writer.writerow(entry)

print(f"Data written to {new_csv_filename}")

This updated code ensures that the header row is correctly written to the new CSV file, even if you modify the headers. Feel free to give it a try, and let me know if you have any further questions! 
## You
I'm so silly sometimes.  I thought tabulate pulls the table headings from the csv, but there is a seperate input line for the table headings.  I understand now.  Thanks again!
## Copilot
No need to feel silly! Learning and understanding new concepts is all part of the process. I'm glad we got it sorted out! If you have any more questions or need further assistance, feel free to ask. 
## You
I see now we lost our ranking.  :)  Can we add the rank to the csv file as we sort that data here? '''# Sort the rows by pushups completed (descending order) adding rank
sorted_rows = sorted(rows[0:], key=lambda x: int(x[2]), reverse=True)
I'm saving this for later to congratulate new results from peopleprint("\nRanking by Pushups Completed:")
for i, row in enumerate(sorted_rows):
print(f"{i+1}. {row[0]}: {row[2]} pushups done in {row[1]} day(s).")
Read data from the CSV file'''## Copilot
Certainly! To add the rank to the CSV file while sorting the data, we can modify the code as follows:

import csv

# Define the headers for the CSV file
csv_headers = ["Username", "Days Reporting", "Pushups Completed", "Date", "Rank"]

# Read data from the CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Sort the rows by pushups completed (excluding the header row)
sorted_rows = sorted(rows[1:], key=lambda x: int(x[2]), reverse=True)

# Add rank to each row
for i, row in enumerate(sorted_rows):
    row.append(i + 1)  # Rank is 1-based

# Write the data to a new CSV file with headers
new_csv_filename = "C:/PyProjects/PU/100kpushups_data_with_rank.csv"
with open(new_csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(csv_headers)  # Write the headers
    for entry in sorted_rows:
        writer.writerow(entry)

print(f"Data with rank written to {new_csv_filename}")

In this modified code:
- We add a new column called "Rank" to the csv_headers.
- During sorting, we append the rank (1-based) to each row.
- The data is then written to a new CSV file with the updated headers.
Now the new CSV file will include the rank along with the other data. Feel free to give it a try, and let me know if you have any further questions! 
## You
Is there a way to change append here with lead?  '''for i, row in enumerate(sorted_rows):
row.append(i + 1)'''    Or otherwise we need to move the csv file rank column from [4] to column [0] for the table to come out correct.  Or it would be easiest if we can tell tabulate to put the columns in this order [4] [0] [1] [2] [3] from the new_csv_filename = "C:/PyProjects/PU/100kpushups_data_with_rank.csv".  Either way would work.
## Copilot
Certainly! To add the rank as the first column (leading) in your CSV file, you can use the insert method to place it at the beginning of each row. Here's how you can modify the code:

import csv

# Define the headers for the CSV file
csv_headers = ["Rank", "Username", "Days Reporting", "Pushups Completed", "Date"]

# Read data from the CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        rows.append(row)

# Sort the rows by pushups completed (excluding the header row)
sorted_rows = sorted(rows[1:], key=lambda x: int(x[2]), reverse=True)

# Add rank to each row (insert at the beginning)
for i, row in enumerate(sorted_rows):
    row.insert(0, i + 1)  # Insert rank at the beginning

# Write the data to a new CSV file with headers
new_csv_filename = "C:/PyProjects/PU/100kpushups_data_with_rank.csv"
with open(new_csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(csv_headers)  # Write the headers
    for entry in sorted_rows:
        writer.writerow(entry)

print(f"Data with rank written to {new_csv_filename}")

Now the new CSV file will have the rank as the first column. Feel free to give it a try, and let me know if you have any further questions! 
## You
The ranking is working well, but there seems to be a problem.  With the matches writing to the csv.  I just posted my daily progress, 
and the regex found it, but didn't put the most recent result in the brand new csvfile.  I put a print(matches) command in the middle 
and this came back "[('DirtyKeyboard', '57', '5081', '2024-04-06'), ('DirtyKeyboard', '57', '5081', '2024-04-06'), ('DirtyKeyboard', '58', '5106', '2024-04-07')]"  But in the csv files there is only the first result.  Now that we have a better idea,
maybe is there a different way to do these steps or is something else wrong? 
'''# Define the regex pattern for 100kchallenge
pattern = r"100kchallenge,(\w+),(\d+),(\d+),(\d{4}-\d{2}-\d{2})"
matches = re.findall(pattern, filetext)
print(matches)
input()
Initialize a set to store unique usernamesunique_usernames = set()
Initialize a list to store data for writing to CSVcsv_data = []
Process each matchfor match in matches:
username, days_reporting, pushups_completed, date = match
if username not in unique_usernames:
unique_usernames.add(username)
csv_data.append([username, days_reporting, pushups_completed, date])
Write the data to a CSV filecsv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
writer = csv.writer(csvfile)
for entry in csv_data:
writer.writerow(entry)'''
## Copilot
I apologize for the inconvenience. It seems that the duplicate entries are not being filtered out correctly. Let's modify the code to ensure that only the most recent entry for each username is included in the CSV file. We'll use a dictionary to keep track of the latest data for each user. Here's the updated code:

import csv

# Define the regex pattern for 100kchallenge
pattern = r"100kchallenge,(\w+),(\d+),(\d+),(\d{4}-\d{2}-\d{2})"
matches = re.findall(pattern, filetext)

# Initialize a dictionary to store the latest data for each username
latest_data = {}

# Process each match
for match in matches:
    username, days_reporting, pushups_completed, date = match
    if username not in latest_data or date > latest_data[username][3]:
        latest_data[username] = (username, days_reporting, pushups_completed, date)

# Write the data to a CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
    writer = csv.writer(csvfile)
    for entry in latest_data.values():
        writer.writerow(entry)

print(f"Data with latest entries written to {csv_filename}")

In this updated code:
- We use a dictionary called latest_data to store the most recent entry for each username.
- If a username is encountered again, we compare the date to determine whether the new entry is more recent.
- Only the latest entry for each username is included in the CSV file.
Please give this modified code a try, and let me know if it resolves the issue! 
## You
That worked great, but I think because I had two matches in one forum page, and I'm the first to respond.  Do we need to save the dictionary to a file, and then reload it at the begining of the script for tomorrow's run?
## Copilot
Certainly! Saving the dictionary to a file and reloading it at the beginning of each script run is a good approach to maintain continuity across multiple days. Here's how you can modify your script to achieve this:
- Save Dictionary to File (End of Today's Run):
    - After processing the matches and creating the latest_data dictionary, save it to a file (e.g., a JSON file) using the json module.
    - This file will store the most recent data for each username.
- Reload Dictionary from File (Start of Tomorrow's Run):
    - At the beginning of tomorrow's script run, load the saved dictionary from the file.
    - Use this loaded dictionary as the starting point for processing new data.
Here's an example of how you can implement this:

import csv
import json

# ... (previous code remains unchanged)

# Save the latest_data dictionary to a JSON file
json_filename = "latest_data.json"
with open(json_filename, "w") as jsonfile:
    json.dump(latest_data, jsonfile)

print(f"Latest data saved to {json_filename}")

At the start of tomorrow's script:

import csv
import json

# Load the latest_data dictionary from the JSON file
json_filename = "latest_data.json"
with open(json_filename, "r") as jsonfile:
    latest_data = json.load(jsonfile)

# ... (rest of your script)

Remember to adjust the file paths and filenames as needed for your specific setup. This way, you'll have a persistent record of the latest data even across different script runs. 
## You
Great.  I think this is gonig to be close to finished if we can figure this out.  If you can figure out how to code my comments in the following snippet would be incredible '''# include the latest page_number in the json somehow       json_filename = "latest_data.json"
with open(json_filename, "r") as jsonfile:
latest_data = json.load(jsonfile)
using page_number = latest_ page_number + 20url = f'https://bitcointalk.org/index.php?topic=5484350.{page_number}'
time.sleep(5)
response = requests.get(url)
if page_number += 20 returns a valid web page response append it to the listturn multiple page sources  into the textfile.source_code = response.text
textfile = open('C:/PyProjects/PU/Temp/PUSource.txt', 'a')
textfile.write(source_code)
when done with the loop textfile.seek(0) and move onfiletext = textfile.read()
textfile.close()
Define the regex pattern for 100kchallengepattern = r"100kchallenge,(\w+),(\d+),(\d+),(\d{4}-\d{2}-\d{2})"
matches = re.findall(pattern, filetext)'''    So we want to cycle through the web addresses, finding matches and, then adding 20 and seeing if that webpage exists, else: move on   # add latest_page_number = page_number to json file at the end, or a different file if easier.
## Copilot
Certainly! Let's break down the steps and implement them in your script:
- Include the Latest Page Number in the JSON:
    - After processing the matches and creating the latest_data dictionary, add the latest page number to it.
    - Save the updated latest_data dictionary to the JSON file.
- Cycle Through Web Addresses and Append Valid Pages:
    - Start with the latest page number plus 20 (as you mentioned).
    - Check if the URL with the new page number returns a valid web page response (status code 200).
    - If it does, append the page source to the list of all page sources.
    - Repeat this process until an invalid page response is encountered.
- Combine All Page Sources into a Single Text File:
    - After the loop, combine all page sources into a single text file (similar to what you're already doing).
Here's how you can modify your code to achieve this:

import csv
import json
import requests
import time

# Load the latest_data dictionary from the JSON file
json_filename = "latest_data.json"
with open(json_filename, "r") as jsonfile:
    latest_data = json.load(jsonfile)

# Calculate the next page number (assuming it's 20 more than the latest page number)
latest_page_number = latest_data.get("latest_page_number", 700)  # Example initial value
page_number = latest_page_number + 20

# Initialize an empty list to store page sources
all_page_sources = []

# Loop through pages
while True:
    url = f"https://bitcointalk.org/index.php?topic=5484350.{page_number}"
    time.sleep(5)
    response = requests.get(url)
    if response.status_code == 200:
        # Valid page response, append it to the list
        all_page_sources.append(response.text)
        page_number += 20
    else:
        # Invalid page response, break the loop
        break

# Combine all page sources into a single text file
with open("C:/PyProjects/PU/Temp/PUSource.txt", "a") as textfile:
    for source_code in all_page_sources:
        textfile.write(source_code)

# Read the combined source text
with open("C:/PyProjects/PU/Temp/PUSource.txt", "r") as textfile:
    filetext = textfile.read()

# Define the regex pattern for 100kchallenge
pattern = r"100kchallenge,(\w+),(\d+),(\d+),(\d{4}-\d{2}-\d{2})"
matches = re.findall(pattern, filetext)

# Initialize a dictionary to store the latest data for each username
latest_data = {}

# Process each match
for match in matches:
    username, days_reporting, pushups_completed, date = match
    if username not in latest_data or date > latest_data[username][3]:
        latest_data[username] = (username, days_reporting, pushups_completed, date)

# Save the latest_data dictionary to a JSON file
with open(json_filename, "w") as jsonfile:
    json.dump(latest_data, jsonfile)

print(f"Latest data saved to {json_filename}")

This updated code should handle cycling through web addresses, appending valid pages, and maintaining the latest page number in the JSON file. Feel free to give it a try, and let me know if you have any further questions! 
## You
Hmm.  It doens't return page_number=740 as invalid, it just serves up page_number=720, again.   Oh. well.  But the json data saved great, but I think we need to load the json data here, instead of starting empty.  '''# Initialize a dictionary to store the latest data for each username
latest_data = {}
Process each matchfor match in matches:
username, days_reporting, pushups_completed, date = match
if username not in latest_data or date > latest_data[username][3]:
latest_data[username] = (username, days_reporting, pushups_completed, date)'''
## Copilot
I apologize for the oversight. You're absolutely right; we should load the existing JSON data at the beginning of the script to maintain continuity. Let's modify the code accordingly:

import csv
import json
import requests
import time

# Load the latest_data dictionary from the JSON file (if it exists)
json_filename = "latest_data.json"
try:
    with open(json_filename, "r") as jsonfile:
        latest_data = json.load(jsonfile)
except FileNotFoundError:
    # If the file doesn't exist, start with an empty dictionary
    latest_data = {}

# Calculate the next page number (assuming it's 20 more than the latest page number)
latest_page_number = latest_data.get("latest_page_number", 700)  # Example initial value
page_number = latest_page_number + 20

# Initialize an empty list to store page sources
all_page_sources = []

# Loop through pages
while True:
    url = f"https://bitcointalk.org/index.php?topic=5484350.{page_number}"
    time.sleep(5)
    response = requests.get(url)
    if response.status_code == 200:
        # Valid page response, append it to the list
        all_page_sources.append(response.text)
        page_number += 20
    else:
        # Invalid page response, break the loop
        break

# Combine all page sources into a single text file
with open("C:/PyProjects/PU/Temp/PUSource.txt", "a") as textfile:
    for source_code in all_page_sources:
        textfile.write(source_code)

# Read the combined source text
with open("C:/PyProjects/PU/Temp/PUSource.txt", "r") as textfile:
    filetext = textfile.read()

# Define the regex pattern for 100kchallenge
pattern = r"100kchallenge,(\w+),(\d+),(\d+),(\d{4}-\d{2}-\d{2})"
matches = re.findall(pattern, filetext)

# Process each match
for match in matches:
    username, days_reporting, pushups_completed, date = match
    if username not in latest_data or date > latest_data[username][3]:
        latest_data[username] = (username, days_reporting, pushups_completed, date)

# Save the latest_data dictionary to a JSON file
with open(json_filename, "w") as jsonfile:
    json.dump(latest_data, jsonfile)

print(f"Latest data saved to {json_filename}")

This updated code will load the existing JSON data (if available) and continue from there. Thank you for catching that, and I appreciate your attention to detail! 
## You
Everything is going well so far, by my friends want to know their dailly average pushups.  I was hopeful this would work, any help?  '''# Write the data to a CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
writer = csv.writer(csvfile)
for entry in latest_data.values():
average = latest_data.values[2]/latest_data.values[1]
writer.writerow(entry,average)
## Copilot
I apologize for the oversight in the code snippet. To calculate the daily average pushups for each user and write it to the CSV file, you can modify the code as follows:

import csv
import json

# Load the latest_data dictionary from the JSON file (if it exists)
json_filename = "latest_data.json"
try:
    with open(json_filename, "r") as jsonfile:
        latest_data = json.load(jsonfile)
except FileNotFoundError:
    # If the file doesn't exist, start with an empty dictionary
    latest_data = {}

# Calculate the average pushups per day for each user
for username, data in latest_data.items():
    days_reporting, pushups_completed, _ = data[1:]  # Ignore the date
    average_pushups = int(pushups_completed) / int(days_reporting)
    latest_data[username] = (*data, average_pushups)

# Write the updated latest_data dictionary to a JSON file
with open(json_filename, "w") as jsonfile:
    json.dump(latest_data, jsonfile)

print(f"Latest data with average pushups saved to {json_filename}")

In this updated code:
- We calculate the average pushups per day for each user by dividing the total pushups completed by the number of days reporting.
- The average pushups value is added to the existing data for each user.
- The updated latest_data dictionary is saved back to the JSON file.
Feel free to give this modified code a try, and let me know if you have any further questions! 
## You
So then this snippet with average_pushups included would go next?  '''# Process each match
for match in matches:
username, days_reporting, pushups_completed, date = match
if username not in latest_data or date > latest_data[username][3]:
latest_data[username] = (username, days_reporting, pushups_completed, date, average_pushups)   # Write the data to a CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
writer = csv.writer(csvfile)
for entry in latest_data.values():
writer.writerow(entry)
print(f"Data written to {csv_filename}")'''
## Copilot
Certainly! Let's include the average_pushups value in the latest_data dictionary and write it to the CSV file. Here's the modified snippet:

# Calculate the average pushups per day for each user
for username, data in latest_data.items():
    days_reporting, pushups_completed, _ = data[1:]  # Ignore the date
    average_pushups = int(pushups_completed) / int(days_reporting)
    latest_data[username] = (*data, average_pushups)

# Write the data to a CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
with open(csv_filename, "a", newline="") as csvfile:
    writer = csv.writer(csvfile)
    for entry in latest_data.values():
        writer.writerow(entry)

print(f"Data with average pushups written to {csv_filename}")

In this updated code:
- We calculate the average_pushups value for each user.
- The latest_data dictionary is updated to include the average pushups value.
- The data (including the average) is written to the CSV file.
Feel free to give this modified code a try, and let me know if you have any further questions! 
## You
I think we should leave the average pushups out of the json file.  Can we follow my comments here, and calculate 'average = entry[2]/entry[1]' and put it in the row?  '''# read data from csv, and calculate average pushups per day
rows = []
with open(csv_filename, 'r') as file:
reader = csv.reader(file)
for row in reader:
# calculate average pushups per day place it in the row
# average = entry[2]/entry[1]
rows.append(row)
Sort the rows by pushups completed (descending order)sorted_rows = sorted(rows[0:], key=lambda x: int(x[2]), reverse=True)"
## Copilot
Certainly! Let's calculate the average pushups per day for each user and include it in the row. Here's the modified snippet:

import csv

# Read data from the CSV file
csv_filename = "C:/PyProjects/PU/100kpushups_data.csv"
rows = []
with open(csv_filename, 'r') as file:
    reader = csv.reader(file)
    for row in reader:
        # Calculate average pushups per day
        days_reporting, pushups_completed = int(row[1]), int(row[2])
        average_pushups = pushups_completed / days_reporting
        # Append the average to the row
        row.append(average_pushups)
        rows.append(row)

# Sort the rows by pushups completed (descending order)
sorted_rows = sorted(rows[1:], key=lambda x: int(x[2]), reverse=True)

# Write the data to a new CSV file with headers
new_csv_filename = "C:/PyProjects/PU/100kpushups_data_with_average.csv"
with open(new_csv_filename, "w", newline="") as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(["Username", "Days Reporting", "Pushups Completed", "Date", "Average Pushups"])
    for entry in sorted_rows:
        writer.writerow(entry)

print(f"Data with average pushups written to {new_csv_filename}")

In this updated code:
- We calculate the average_pushups value for each user based on the total pushups completed and the number of days reporting.
- The average pushups value is appended to the existing data for each user.
- The data (including the average) is written to the new CSV file.
Feel free to give this modified code a try, and let me know if you have any further questions! 

Haha, I messed up my own format, with first saying 100kpushups, then correctly saying 100kchallenge.  I have now changed teh regex to accept only 100k or 100k plus any 9 word characters to handle mispellings.

[DirtyKeyboard]

Look Ahead:  Functions!

Updates:  Chart Buddy is no longer posting.  Hopefully just for now.

The Top 20 scripts have been running great.  Just 1 crash in the last 2 weeks.  I believe because I ran the script too close to UTC midnight and the Canadian $ data came back blank.  So I moved back the run time until 12:45 am instead of 12:20 am UTC.  And so far so good.

The 100 Push-Ups A Day Until Bitcoin Is $100K Challenge in Speculation has been taking up most of my free time.

The regex format is working great, when people type it correctly.   Even I mess it up time to time.  There needs to be some sanity checks, like is the date from the future, or is this less pushups than the last entry.  But we've added a second table and little graph tracking everyone's pushup progress, partly inspired by LoyceV's sick BBC code merit graphs I saw the other week, the progress bar I figured out, and the animated Mango sig sky.

But now I wanted to do more with the data.  Graphs for each user.  One Total Pushups vs Time and another, Pushups per Given Report.  What to do with them though?

Main goal:  To provide individual statistics to people taking part in the 100 Push-Ups A Day Until Bitcoin Is $100K Challenge

Delivery Options:  I noticed the Tabulate Python module I'm using has an output format option of html, so in theory I could make websites for everyone.  I tested one with free hosting, but I was going to have to get everyone free SSL certificates, so I went with option B.  Our own website of websites, er basically.  I've set up an SMF forum.  I know, I know, put that in the things that could only go wrong category.  Haha!  But when all you have is a hammer...  We'll see how long the open parts of the forum last, but there is one board that only I can post the graphs in, and guests and members can view.  Then there is the On topic and Off topic boards that members can post in.  I'm ready to delete those boards if need be.

Method:  I had some co-pilot help, and I was able to modify old code for new purposes, but I've got my first 2 functions on the books.  grapher, which plots the total pushups, and graphper (looks better than grapherper ) which plots the number of pushups per each report.  I don't think I quite have the idea though.  My function is still basically a one purpose only function, or rather a function that does many things to acheive one specific task. 
A] There is a dictionary of usernames, and their forum thread numbers at the top. 
B] The function, which takes each user's CSV file that I've been appending the last 2 weeks, and calculates and graphs the total pushups, and pushups per report using matplotlib scatter and stem graphs respectively.
C] and then an for loop that runs the function for every user in the dictionary.

Actuality:  Now there is only 1 function.  I strung them back to back so only one forum post needed to be made.  I'm trying not to attact bots so I'm not going to link the forum, but I will spell it out in the Speculation thread. I removed it, and my super secret API keys from the code below. 
both_graphs code:

Code:

import csv, json, pyautogui, pyperclip, requests, time, webbrowser
import matplotlib.pyplot as plt
import numpy as np
import matplotlib as mpl
import pandas as pd
from datetime import timedelta

SMALL_SIZE = 8
MEDIUM_SIZE = 12
BIGGER_SIZE = 14

plt.rc('font', size=MEDIUM_SIZE)          # controls default text sizes
plt.rc('axes', titlesize=MEDIUM_SIZE)     # fontsize of the axes title
plt.rc('axes', labelsize=MEDIUM_SIZE)    # fontsize of the x and y labels
plt.rc('xtick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('ytick', labelsize=SMALL_SIZE)    # fontsize of the tick labels
plt.rc('legend', fontsize=MEDIUM_SIZE)    # legend fontsize
plt.rc('figure', titlesize=BIGGER_SIZE)  # title fontsize

pusher_dictionary = {"Team": 2, "OgNasty": 3, "JayJuanGee": 4, "DirtyKeyboard": 5, "I_Anime": 6, "Tmoonz": 7, "Troytech": 8, "7juju": 9, "Ambatman":  10,  "promise444c5": 11, "Gallar": 12,  "Mayor of Ogba": 13, "Zackz5000": 14, "Ricardo11": 15, "Tungbulu": 16, "Bravut": 17,  "Kwarkam": 18, "Judith87403": 19, "adultcrypto": 20, "teamsherry": 21, "Notalony": 22, "Dailyscript": 23,  "Obim34": 24, "proty": 25, "Antonil": 27, "Bd officer": 28, "SickDayIn": 29}

plt.style.use('ggplot')
def both_graphs(username):
    json_filename = "C:/PyProjects/PU/PU/latest_data.json"
    with open(json_filename, "r") as jsonfile:
        latest_data = json.load(jsonfile)
        for pusher, data in latest_data.items():
            if pusher == username:
                username, days_reporting, pushups_completed, date = data
                list_of_pushups = []
                x = []  # list to hold dates
                y = []  # list to hold pushups_completed
                filename = f"C:/PyProjects/PU/PU/{pusher}.csv"
                df = pd.read_csv(filename)
                entries = len(df)
                if entries >= 1:
                    with open(filename, "r") as personal_csv:
                        reader = csv.reader(personal_csv)
                        
                        prev_pushups = 0  # initialize previous pushups count
                        for row in reader:
                            current_pushups = int(row[2])  # current total pushups
                            if prev_pushups != 0:  # if not the first row
                                x.append(row[3])  # append date
                                daily_pushups = current_pushups - prev_pushups  # calculate daily pushups
                                y.append(daily_pushups)  # append daily pushups
                                # list_of_pushups.append(daily_pushups[-1])
                            prev_pushups = current_pushups  # update previous pushups count
                        print(daily_pushups)   
                        x = [pd.to_datetime(d) for d in x]  # convert dates after x has been populated
                        
                        #set plot options
                        yesterday = min(x) - timedelta(1)
                        tomorrow = max(x) + timedelta(1)
                        mpl.pyplot.close()
                        fig, ax = plt.subplots(figsize=(8,8),  layout="constrained")
                        markerline, stemline, baseline, = ax.stem(x,y,linefmt='k-',markerfmt='ko',basefmt='k.')
                        plt.setp(stemline, linewidth = 2.5)
                        plt.setp(markerline, markersize = 8)
                        ax.set(xlim=(yesterday, tomorrow), ylim=(0, max(y)), yticks=np.arange(0, max(y)*1.4, 50))
                        ax.set_xlabel('Report Date')
                        ax.set_ylabel('Pushups in Report')
                        plt.savefig(f'C:/PyProjects/PU/{pusher}.png')
                        list_of_pushups = y

                        #upload and get link
                        url = "https://api.imgur.com/3/image"
                        payload = {'name': f'{pusher}'}
                        files=[('image',(f'{pusher}.png',open(f'C:/PyProjects/PU/{pusher}.png','rb'),'image/png'))]
                        headers = {'Authorization': 'Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}
                        response = requests.post(url, headers=headers, data=payload, files=files)
                        data = response.json()
                        pusher_graph = data.get("data", {}).get("link")
                        print(pusher_graph)
                else:
                     pusher_graph = "2 data points needed"

    with open(json_filename, "r") as jsonfile:
       latest_data = json.load(jsonfile)
       for pusher, data in latest_data.items():
              if pusher == username:
                username, days_reporting, pushups_completed, date = data
                x = []  
                y = [] 
                with open(f"C:/PyProjects/PU/PU/{pusher}.csv", "r") as personal_csv:
                    reader = csv.reader(personal_csv)
                    for row in reader:
                        x.append(row[3])  
                        y.append(int(row[2]))  
                    x = [pd.to_datetime(d) for d in x]
                    yesterday = min(x) - timedelta(1)
                    tomorrow = max(x) + timedelta(1)
                    mpl.pyplot.close()
                    fig, ax = plt.subplots(figsize=(8,8),  layout="constrained")
                    ax.scatter(x, y, s=100)

                    #this part deals with how the axes should be scaled relative to how many pushups have been completed
                    if max(y) >= 2000:
                            y_space = 1000
                    else:
                            y_space = 100
                    ax.set(xlim=(yesterday, tomorrow), ylim=(0, max(y)*1.4), yticks=np.arange(0, max(y)*1.4, y_space))
                    ax.set_title(f"{pusher}'s Total Pushups Reported")
                    ax.set_xlabel('Report Date')
                    ax.set_ylabel('Total Pushups')
                    plt.savefig(f'C:/PyProjects/PU/{pusher}2.png')

                    # grabs the forum thread number from the dictionary
                    page = pusher_dictionary[pusher]
                    print(page)

                    # upload graph to imgur
                    url = "https://api.imgur.com/3/image"
                    payload = {'name': f'{pusher}'}
                    files=[('image',(f'{pusher}.png',open(f'C:/PyProjects/PU/{pusher}2.png','rb'),'image/png'))]
                    headers = {'Authorization': 'Bearer xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'}
                    response = requests.post(url, headers=headers, data=payload, files=files)
                    data = response.json()
                    pusher_graph2 = data.get("data", {}).get("link")
                    print(pusher_graph2)

                    # add post to clipboard
                    pyperclip.copy(f"[img]{pusher_graph}[/img]\n[img]{pusher_graph2}[/img]")

                    # can use this link for the reply button
                    url7 = f'https://pushups/index.php?action=post;topic={page}.0'
                    webbrowser.open(url7)
                    time.sleep(20)
                    pyautogui.hotkey('f11') 
                    time.sleep(10)
                    pyautogui.hotkey('tab')
                    time.sleep(5)
                    pyautogui.hotkey('tab')
                    time.sleep(5)
                    pyautogui.hotkey('tab')
                    time.sleep(5)
                    pyautogui.hotkey('ctrl', 'v')
                    time.sleep(5)
                    pyautogui.hotkey('tab')
                    time.sleep(5)
                    
                    # we're doing it live if the next command is #ed out
                    # pyautogui.hotkey('tab')
                    time.sleep(5)
                    pyautogui.hotkey('enter')
                    time.sleep(5)
                    pyautogui.hotkey('f11') 

for username, page in reversed(pusher_dictionary.items()):
    print(username)
    both_graphs(f"{username}")