Major change: I have added finance_history

* pulled old values of data via restic backups
    * inserted them into a new finance_history table
    * added a finplan user, use of sudo & better ENV (container/production) & wrapper.sh into Dockerfile (like I do for other projects)
    * recalculate the bills / Living Expenses now we have real bills for a year
    * remove tax back for now (need to handle quit date vs. end of financial year better)
    * hard-coded / hacked in 2026 for pay cycle dates / should be dynamic, but not sure I'll work in 2027+
    * refactor front-end to handle 2026 as the current year / its more or less dynamic (via hard-coded FIRST_YEAR) variable - could not use datetime, as it was in late Dec. when I noticed the issue ('next pay' was in 2026, current year was 2025)
    * also added history to graphs, changed formatting to make the history /
    * savings projections to be orannge with circles, but historical is a solid line, future is a dash, also made all lines have lineWidth: 1 for aesthetics

Dockerfile

@@ -3,13 +3,15 @@ FROM python:latest
 # this forces /code to be owned by the user specified in the docker-compose file, we could ignore issues without this (but I am writing log files to the current dir with gunicorn as a non-priv'd user)
 ARG USERID
 ARG GROUPID
-RUN apt-get update && apt-get install sqlite3 && apt-get dist-upgrade -y
+RUN apt-get update && apt-get -y install cron sqlite3 sudo && apt-get dist-upgrade -y
 WORKDIR /code
 COPY requirements.txt .
-RUN pip3 install -r requirements.txt
+COPY crontab /etc/crontab
+RUN pip3 install --upgrade pip && pip3 install -r requirements.txt
+RUN groupadd -g ${GROUPID} finplan && useradd -m -u ${USERID} -g ${GROUPID} finplan
 COPY . .
-RUN chown -R ${USERID}:${GROUPID} /code
+RUN chown -R finplan:finplan /code
-EXPOSE 8080
+EXPOSE 80
-CMD gunicorn --bind=0.0.0.0:8080 --workers=2 --threads=2 main:app --error-logfile /code/gunicorn.error.log --access-logfile /code/gunicorn.log --capture-output
+# NOTE, wrapper.sh will use sudo to work in PROD and DEV AS the correct
-# comment out the gunicorn line & uncomment the sleep so you can run gunicorn by hand in case it does not start on init (via docker exec...)
+# BOOK_UID/BOOK_GID as pybook user and group
-#CMD sleep 99999
+CMD ["./wrapper.sh"]

TODO

@@ -1,24 +1,19 @@
-bills html, and growth types are very lame... could I do something more like:
+bills:
+    bills html, and growth types are poor code repitition / lame... could I do something more like:
         {% for gt in growth %}
             {% if gt.name == 'Min' %}
                 <option value='min'
                     {% if bt.which_growth == gt.name %}selected{% endif %}
                 >{{'%.2f'|format(bt.ann_growth_min)}}% {{gt.name}}</option>
 
-
 CALC:
     * if I quit at different times of the financial year, technically the amount I earn will be taxed
     differently, but hard to calc - slides around with tax brackets in future
 
-UI:
+    * still get double health insurance bills sometimes (just viewing a new date might trigger this??? or at least when I changed years)
-    * to allow <yr> total on bill summary, need to re-work annual growth to a drop-down
-        - [DONE] mock-up
-        - need to change 'which growth' to accommodate new growth models (cpi, override*)
-            - [DONE] do this in DB with new table - then don't do crazy pattern matching/making up overrides in the UI code
-            - get UI to use bill_growth_types table 
-            - get UseGrowth() to use bill_growth_types table
 
-For bills:
+UI:
-    * might need to be able to mark a specific bill as an outlier:
+    * add FIXED Annotations:
-        - so we ignore the data somehow (think Gas is messing with my bills)
+        * historical annotations (at least M_quit_date, Trip_date, school_fees in 25)
-        - and even electricity, water, etc. for when we were away in Europe but mostly gas/elec
+    * should try AI with how to distribute annotations better
+    * make bills tabs a vertical navbar instead of horizontal

calc.py

@@ -6,7 +6,7 @@ from defines import END_YEAR
 LEASE = 0
 
 # Dates that don't change
-first_pay_date = datetime(2025,1,8)
+first_pay_date = datetime(2026,1,8)
 school_fees_date = datetime(2025, 12, 5)
 car_balloon_date = datetime(2026, 11, 15)
 mich_present_date = datetime(2026,10,15)
@@ -82,7 +82,7 @@ def calculate_savings_depletion(finance, bill_data, bill_type):
     payout = 83115.84
     print( f"leave payout gross={payout}" )
 
-    # However, if I quit in the next fin year - tax for 2025 will be: $4,288 plus 30c for each $1 over $45,000
+    # However, if I quit in the next fin year - tax will be: $4,288 plus 30c for each $1 over $45,000
     #   (assuming the 7830.42 * ~90/bus_days_in_fortnight = ~ $64k - > 45k and < $135k bracket is 30%)
     #   Given, I probably can't stop Deakin doing PAYG deductions, I won't get
     #   the tax back until the end of the financial year, so work out the
@@ -95,6 +95,9 @@ def calculate_savings_depletion(finance, bill_data, bill_type):
 
     tax_diff_D_leave = payout - D_leave_after_tax
 
+    ### FIXME: for now, assume no tax back after leave - think this may be needed if I quit anytime nowish until end of Jun
+    tax_diff_D_leave = 0
+
     print( f"tax_diff_D_leave: {tax_diff_D_leave}") 
 
     ### leave / tax items finished ###
@@ -116,6 +119,7 @@ def calculate_savings_depletion(finance, bill_data, bill_type):
     yr=str(current_date.year)
     for b in bill_data:
         if yr in b['bill_date']:
+            print( f"Seems {yr} is in {b['bill_date']} -- add {b['amount']}" )
             total += b['amount']
     
     print( f"this yr={current_date.year} - total={total}" )
@@ -138,7 +142,7 @@ def calculate_savings_depletion(finance, bill_data, bill_type):
     car_buyout_date = datetime.strptime( finance['Car_buyout_date'], "%Y-%m-%d") 
 
     # to force deakin pay cycles to match reality, we work from the 8th of Jan as our "day-zero" so we are paid on the 8/1/25, 22/1/25, etc.
-    days_count = ( current_date - datetime(2025,1,1) ).days
+    days_count = ( current_date - datetime(2026,1,1) ).days
 
     # Track the fortnight, and monthly interest
     fortnight_income = 0
@@ -219,7 +223,7 @@ def calculate_savings_depletion(finance, bill_data, bill_type):
             # monthly increase living expenses by a monthly inflation multiplier
             Living_Expenses += (Inflation/100.0)/12 * Living_Expenses
             daily_living_expenses = Living_Expenses / 365
-#            print(f"{current_date}: Living Exp inceased - ${Living_Expenses}")
+            print(f"{current_date}: Living Exp inceased - ${Living_Expenses}")
 
         if current_date.date() == school_fees_date.date():
             current_savings -= School_Fees

crontab

@@ -0,0 +1,2 @@
+# run once every 5 days or so
+0 23 2-27/5 * * finplan /code/snapshot.sh

db.py

@@ -177,10 +177,20 @@ def get_finance_data():
     conn.close()
     return dict(finance)
 
+def get_historical_data():
+    conn = connect_db(False)
+    # this treats the returns as dicts
+    conn.row_factory = sqlite3.Row
+    cur = conn.cursor()
+    cur.execute('SELECT * FROM finance_history order by snapshot_date')
+    hist = [dict(row) for row in cur.fetchall()]
+    conn.close()
+    return hist
+
 def get_budget_data(finance_data):
     # annual bills - health ins (5k), rates (2.6), electricity (1.2), gas (2.1) - but 1.4 in 2025 due to EU trip, internet (1.6), car insurance (.7), rego (.8), house insurance (2.4), GFC (2.6), water (1.1), eweka (.1), phones (.5), melb. pollen (.03), nabu casa (.1)   --- noting phone is elevated presuming I also go onto Aldi plan, but that there is no family discount, and health will be extra after stop working
     # fudging below - its more like 15.2 + health, and really gas will be more than 2.1 than 1.4, so about 16+5
-    bills = 21000
+    bills = 25357.07
     BUDGET=[]
     BUDGET.append( ('Bills', f"${bills:,.2f}") )
     BUDGET.append( ('Buffer', f"${finance_data['CBA']*finance_data['CBA_price']+finance_data['TLS']*finance_data['TLS_price']:,.2f}") )

main.py

@@ -1,7 +1,7 @@
 # main.py
 from flask import Flask, render_template, request, redirect, url_for, Response, jsonify
 from calc import calculate_savings_depletion, calc_key_dates
-from db import init_db, get_finance_data, update_finance, get_budget_data
+from db import init_db, get_historical_data, get_finance_data, update_finance, get_budget_data
 from db import insert_cset, get_comp_set_data, get_comp_set_options, delete_cset
 from db import get_bill_freqs, get_bill_growth_types
 from db import get_bill_data, new_bill, update_bill_data, delete_bill, delete_estimated_bills, delete_estimated_bills_for
@@ -26,6 +26,7 @@ init_db()
 
 @app.route('/')
 def index():
+    hist = get_historical_data()
     finance_data = get_finance_data()
     get_comp_set_options(finance_data)
     bill_data = get_bill_data("order_by_date_only")
@@ -106,7 +107,7 @@ def index():
     # now work out how much padding we need in the first year to align the last dates for all years
     padding=second_count - first_count
     key_dates = calc_key_dates( finance_data )
-    return render_template('index.html', now=now, first_yr=first_yr, padding=padding, finance=finance_data, depletion_date=depletion_date, savings=savings_per_fortnight, BUDGET=BUDGET, COMP=COMP, DISP=DISP, key_dates=key_dates)
+    return render_template('index.html', now=now, first_yr=first_yr, padding=padding, finance=finance_data, depletion_date=depletion_date, savings=savings_per_fortnight, BUDGET=BUDGET, COMP=COMP, DISP=DISP, key_dates=key_dates, hist=hist)
 
 @app.route('/save', methods=['POST'])
 def save():
@@ -185,7 +186,10 @@ def DisplayBillData():
     total=calc_future_totals(bill_info, bill_types)
     # update/re-get bill_data now that new estimated bills have been added
     bill_data = get_bill_data("order_by_bill_type_then_date")
-    return render_template('bills.html', bill_data=bill_data, bill_types=bill_types, bill_freqs=bill_freqs, bill_ui=bill_ui, this_year=datetime.today().year, END_YEAR=END_YEAR, total=total, key_dates=key_dates, growth=bill_growth_types, cpi=finance_data['Inflation'] )
+    finance_data = get_finance_data()
+    # FIXME: really need to do this better, but for now, hard code til I quit seems okay to me
+    FIRST_YEAR=2026
+    return render_template('bills.html', bill_data=bill_data, bill_types=bill_types, bill_freqs=bill_freqs, bill_ui=bill_ui, this_year=datetime.today().year, FIRST_YEAR=FIRST_YEAR, END_YEAR=END_YEAR, total=total, key_dates=key_dates, growth=bill_growth_types, cpi=finance_data['Inflation'] )
 
 @app.route('/newbilltype', methods=['POST'])
 def InsertBillType():

snapshot.sh

@@ -0,0 +1,41 @@
+#!/bin/bash
+
+DB_FILE="finance.db"
+HISTORY_TABLE="finance_history"
+# Current date in the format you've been using
+DATE_STR=$(date +%Y-%m-%d)
+
+# 1. Identify which table exists (finance or finance_data)
+SRC_TABLE=finance
+
+if [ -z "$SRC_TABLE" ]; then
+    echo "Error: Source table not found."
+    exit 1
+fi
+
+# 2. Get columns dynamically (excluding 'id')
+COLS=$(sqlite3 "$DB_FILE" "PRAGMA table_info($SRC_TABLE);" | cut -d'|' -f2 | grep -v '^id$' | xargs | tr ' ' ',')
+
+# 3. Change Detection Logic
+# We grab the latest row from history and compare it to the current source data
+CURRENT_VALS=$(sqlite3 "$DB_FILE" "SELECT $COLS FROM $SRC_TABLE ORDER BY id DESC LIMIT 1;")
+LAST_VALS=$(sqlite3 "$DB_FILE" "SELECT $COLS FROM $HISTORY_TABLE ORDER BY snapshot_date DESC LIMIT 1;")
+
+if [ "$CURRENT_VALS" == "$LAST_VALS" ]; then
+    echo "Data identical to last snapshot ($DATE_STR). Skipping."
+    exit 0
+fi
+
+# 4. Auto-Migration
+# Ensure any new columns in 'finance' also exist in 'finance_history'
+for col in ${COLS//,/ }; do
+    sqlite3 "$DB_FILE" "ALTER TABLE $HISTORY_TABLE ADD COLUMN $col NUMERIC;" 2>/dev/null
+done
+
+# 5. The Snapshot Insert
+# INSERT OR IGNORE ensures that if Docker restarts today, we don't duplicate the row.
+echo "Changes detected. Recording snapshot for $DATE_STR..."
+sqlite3 "$DB_FILE" <<EOF
+INSERT OR IGNORE INTO $HISTORY_TABLE (snapshot_date, $COLS)
+SELECT '$DATE_STR', $COLS FROM $SRC_TABLE ORDER BY id DESC LIMIT 1;
+EOF

templates/index.html

@@ -16,7 +16,7 @@
     <script src="https://code.highcharts.com/themes/adaptive.js"></script>
     <style>
         .col-form-label { width:140px; }
-        html { font-size: 80%; }
+        html { font-size: 75% !important; }
     </style>
 </head>
 <body>
@@ -222,6 +222,8 @@
         // make these global so we can also use them in the /save route (via modal)
         const savingsData = JSON.parse('{{ savings | tojson }}');
         const vars = JSON.parse('{{ finance | tojson }}');
+        const rawHistData = JSON.parse('{{ hist | tojson }}');
+        const histData = rawHistData.map(entry => [ new Date(entry.snapshot_date).getTime(), parseFloat(entry.Savings || 0) ]);
 
         $(function() { $('[data-bs-toggle="popover"]').popover(); });
         window.onload = function() { 
@@ -259,6 +261,7 @@
             $('#tab-but-findata').click()
             // Parse the savings_data from Flask
             const chartData = savingsData.map(entry => [new Date(entry[0]).getTime(), parseFloat(entry[1])]);
+            const histChartData = histData.map(entry => [new Date(entry[0]).getTime(), parseFloat(entry[1])]);
             {% if COMP %}
                 const compSavingsData = JSON.parse('{{ COMP['savings_data'] | tojson }}');
                 const compChartData = compSavingsData.map(entry => [new Date(entry[0]).getTime(), parseFloat(entry[1])]);
@@ -348,7 +351,10 @@
                     }, shared:true 
                 },
                 annotations: annotations, // Add annotations
-                series: [ { name: "Savings", data: chartData, marker: { radius: 2 } } ]
+                series: [ 
+                    { name: "Savings", data: chartData, marker: { enabled: true, symbol: 'circle', radius: 2}, lineWidth:1, dashStyle: 'ShortDash', color: 'orange' },
+                    { name: "Historical", data: histChartData, marker: { enabled: true, symbol: 'circle', radius: 2 }, lineWidth:1 }
+                ]
             });
 
             {% if COMP %}
@@ -374,8 +380,9 @@
                     }, shared:true 
                 },
                 series: [
-                    { name: "Savings", data: chartData, marker: { radius: 2 } }
+                    { name: "Savings", data: chartData, marker: { enabled: true, symbol: 'circle', radius: 2}, lineWidth:1, dashStyle: 'ShortDash', color: 'orange' }
-                    ,{ name: "{{COMP['vars']['name']}}", data: compChartData, marker: { radius: 2 } }
+                    ,{ name: "{{COMP['vars']['name']}}", data: compChartData, marker: { enabled: true, symbol: 'diamond', radius: 2 }, lineWidth:1, color: 'cyan' },
+                    { name: "Historical", data: histChartData, marker: { enabled: true, symbol: 'circle', radius: 2 }, lineWidth:1 }
                 ]
             });
             {% endif %}

wrapper.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+
+if [ "$ENV" == "production" ]; then
+    export WORKERS=2
+    export THREADS=2
+    export RELOAD=""
+else
+    cd /pybook_mapped_volume
+    export WORKERS=1
+    export THREADS=1
+    export RELOAD="--reload"
+fi
+
+# start cron as root to do historical snapshots
+/usr/sbin/cron -f &
+
+# start finplan with right amount of threads/workers, etc.
+sudo -u finplan gunicorn --bind=0.0.0.0:80 --timeout 300 --workers=$WORKERS --threads=$THREADS main:app --env ENV="$ENV" --error-logfile gunicorn.error.log --access-logfile gunicorn.log --capture-output $RELOAD --enable-stdio-inheritance
+
+# just in case it fails this keeps the container up so you can check gunicorn logs
+sleep 99999