Commit Graph

201 Commits

Author SHA1 Message Date
Davide Scaini b23b3de1bb feat: include gear in activity index summaries; generate OG images in serve rebuild 2026-05-24 12:51:57 +02:00
Davide Scaini 5bf426df29 fix: use Strava gear ID prefix (b/g) to determine gear type, not missing primary_type field 2026-05-24 12:44:25 +02:00
Davide Scaini 40ccec0e2d fix: generate OG images in serve rebuild worker, not on every deploy 2026-05-24 12:39:38 +02:00
Davide Scaini e553e08663 feat: gear registry — manage bikes/shoes per athlete, set per activity
- New /api/gear CRUD endpoints (gear.json per user)
- Gear tab in AthleteView (owner-only): add, edit, retire items
- EditDrawer gear field becomes a dropdown when registry has items
- Strava API sync now resolves gear_id → name, adds to registry automatically
- Strava ZIP import reads Gear column from activities.csv
- POST /api/strava/import-gear for one-time backfill from stored originals
2026-05-24 12:33:41 +02:00
Davide Scaini 693f720cbd feat: OG link previews — track image + meta tags for Telegram/WhatsApp
- bincio/render/ogimage.py: generate 400x400 elevation-coloured PNG with Pillow
- bincio/serve/routers/ogimage.py: /activity/{id}/ OG HTML stub for bot UAs;
  /og-image/{user}/{id}.png serves pre-generated images with on-demand fallback
- scripts/generate_og_images.py: batch pre-generation, incremental (mtime skip)
- scripts/strava_elevation_audit.py: add source/threshold/MA columns and pct stats
- pyproject.toml: add Pillow>=10 to serve extras
2026-05-23 21:44:19 +02:00
Davide Scaini 56932f7f25 perf: add patch_index flag to recalculate_elevation_hysteresis
Allows bulk callers to skip per-activity index.json rewrites and
batch the update themselves, reducing O(n²) index churn to O(n).
2026-05-23 21:05:00 +02:00
Davide Scaini 02edb0b0f9 fix: per-source elevation params — strava_export vs barometric vs raw GPS
Previous thresholds (10 m GPS, 5 m barometric, 30 s MA) were calibrated for
raw noisy GPS. Strava-exported FIT files carry elevation already pre-processed
by Strava (smooth 1 m quantisation, no steps > 5 m), so the aggressive
filtering suppressed real climbing — avg −17 % error across 37 reference
activities.

New strategy, keyed on source + altitude_source:
  strava_export           → MA 5 s, threshold 1.0 m
  fit_file / barometric   → no MA, threshold 1.5 m
  fit_file / gps          → MA 5 s, threshold 2.0 m
  unknown non-strava      → MA 5 s, threshold 1.5 m

Result on 37 cross-referenced activities: avg −2.8 %, std 4.6 %,
37/37 within ±15 % (was 0/37).

Both paths — initial import (metrics._elevation) and bulk recalculate
(dem.recalculate_elevation_hysteresis) — now use the same elevation_params()
function from metrics.py.
2026-05-23 20:12:11 +02:00
Davide Scaini d4e5b11f71 admin: add Total imported and Last sync columns to Garmin sync table
Matches the Strava sync table layout. Accumulates total_imported in
garmin_sync.json state on each sync run; admin API exposes last_sync_at
and total_imported from that file.
2026-05-21 20:34:25 +02:00
Davide Scaini 84eff1f3b0 perf: spatial 10 m downsampling for timeseries
Extract _haversine_m from the inline block in _gps_speed_kmh, add
_spatial_downsample (keep one sample per 10 m traveled, GPS haversine
primary / speed×Δt fallback, indoor activities unchanged), and wire it
into build_timeseries() after the 1 s dedup loop.

Add --downsample-timeseries migration flag to bincio render that applies
the same downsampling to existing stored timeseries files without
re-extracting from original FIT/GPX files.
2026-05-19 20:11:00 +02:00
Davide Scaini 835968e8fe perf: unblock event loop for segment_efforts scan
Extract the synchronous segment-file scan into a plain function and
dispatch it via asyncio.to_thread so it runs in a thread pool instead
of blocking the event loop during concurrent fetches.
2026-05-19 19:53:26 +02:00
Davide Scaini adaa075e6e Add usage stats script and /api/admin/stats endpoint
scripts/usage_stats.py: standalone script (PEP 723, runs via uv run)
that parses all nginx access.log files, filters bots, maps Referer
headers to feature labels, and produces a 3-panel matplotlib figure:
daily logins + 7-day rolling mean, hour×weekday API heatmap, and
weekly feature usage stacked area. Output saved to
/var/bincio/stats/latest.png. Intended for a weekly cron job.

bincio/serve/routers/admin.py: GET /api/admin/stats serves the PNG
via the existing _require_admin() check — no new auth logic or nginx
changes needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-05-18 20:54:17 +02:00
Davide Scaini c0f6c4da6d render: add --recompute-vam to backfill climbing_time_s into existing activities
Reads each activity's timeseries, re-runs the VAM algorithm (which now returns
both climbing_vam_mh and climbing_time_s), and patches activities/*.json and
index.json in-place. Run once after upgrading to the new schema so NerdCorner
can filter and opacity-encode existing data.
2026-05-17 10:15:29 +02:00
Davide Scaini 766da0320b NerdCorner VAM: filter short climbs, opacity-encode confidence, add climbing time to tooltip
- Exclude per-activity VAM contributions where climbing_time_s < 10 min; short
  punchy efforts don't represent aerobic fitness and were skewing monthly averages
- Store climbing_time_s alongside climbing_vam_mh in metrics, detail JSON, and
  summary JSON so the frontend has the data to reason about confidence
- Accumulate total climbing time per period; opacity scales from 0.25 (10 min,
  minimum threshold) to 1.0 (≥ 1 h) so thin-evidence months read as faint dots
- Render VAM as dots only (no lines) since each period is an independent average,
  not a cumulative — lines implied continuity that isn't there
- Tooltip now shows "1060 m/h · 38 min climbing"
2026-05-17 10:13:39 +02:00
Davide Scaini 14a4a0b994 Activity detail: layout refactor + GPS-derived speed for map coloring
Layout: map + charts stacked left, stats panel (2-col) on the right.
Cadence moved to last stat. Charts sit directly below the map.

Speed coloring: most FIT files don't record per-second speed, leaving
timeseries speed_kmh all-null and the hover link dead. Fix: derive speed
from consecutive GPS coordinates (haversine + 5-pt moving average) when
the device didn't record it. Add --backfill-speed render flag to retrofit
existing timeseries files.
2026-05-16 23:24:29 +02:00
Davide Scaini 003b540481 VAM: drop duration curve, show avg climbing VAM in Nerd Corner
Remove the per-duration VAM curve everywhere (metrics, summaries, detail
JSON, athlete.json, VamChart.svelte, AthleteView VAM tab). Keep only
climbing_vam_mh per activity. Add it to activity summaries so NerdCorner
can plot average climbing VAM per week/month year-over-year alongside
distance/elevation/time. Add --backfill-vam-summary flag to copy the
field from existing detail JSONs into index.json without re-extracting.
2026-05-16 22:03:40 +02:00
Davide Scaini 7cd8a6b030 VAM: add --recompute-vam flag and compute_vam_from_timeseries helper
Refactors core VAM logic into _vam_from_ele_1hz() and _build_ele_1hz()
so both the DataPoint-based extract path and the timeseries-based backfill
path share the same implementation.

render --recompute-vam reads stored *.timeseries.json files and updates
climbing_vam_mh + vam_curve in activities/*.json and index.json in-place,
without re-parsing the original FIT/GPX files.
2026-05-16 21:37:51 +02:00
Davide Scaini baf20b51ba Add VAM (climbing velocity) metric and per-duration curve
Extract pipeline now computes two VAM metrics per activity (cycling,
running, hiking, walking):
- climbing_vam_mh: VAM on ascending segments only, using 30 s forward
  lookahead to classify climbing vs. flat/descent (stored in detail JSON)
- vam_curve: [[duration_s, vam_mh], ...] best VAM per standard duration
  (60 s – 1 h), sliding window on 30 s smoothed elevation, only windows
  with ≥ 10 m net gain count (stored in summary + detail)

Athlete JSON aggregates vam_curve across all activities (all_time,
last_365d, last_90d), same structure as power_curve.

Frontend:
- ActivityDetail shows "Climbing VAM" stat (grouped with elevation)
- AthleteView adds a "VAM Curve" tab that appears only when the athlete
  has climbing data; renders VamChart (new component, mirrors MmpChart)

vam_curve stripped from combined global feed; kept in user year shards
for season-based on-the-fly aggregation in VamChart.

Requires bincio reextract to backfill existing activities.
2026-05-16 21:34:06 +02:00
Davide Scaini de602ff5d9 Settings: per-user default for download_disabled
New pref download_disabled_default (stored in user_prefs + mirrored to
_user_settings.json for the render pipeline). When true, apply_sidecar
marks all activities as download_disabled unless the sidecar explicitly
sets download_disabled: false (per-activity opt-in from the edit drawer).

Settings page gets an "Activity defaults" card with the toggle.
2026-05-16 20:51:23 +02:00
Davide Scaini 2d9620c6d1 Admin: add Garmin sync status panel
New /api/admin/garmin-sync (GET) and /api/admin/garmin-sync/run (POST)
endpoints mirror the Strava equivalents, reading _garmin_sync_status.json
per user and exposing a run-now button. Admin page shows the Garmin table
below the Strava one, with auth_error/api_error/ok badges and live polling
while a sync is running.
2026-05-16 20:31:02 +02:00
Davide Scaini 2c69e75842 Show orange upload button when Strava/Garmin auth fails
GET /api/me/sync-status reads _strava_sync_status.json and
_garmin_sync_status.json for the logged-in user. On page load the nav
script checks this endpoint and, if either service has status=auth_error,
turns the upload arrow orange with a tooltip naming the disconnected
service(s).
2026-05-16 20:27:43 +02:00
Davide Scaini 0eb25620ef Add headless Garmin background sync (systemd timer)
New `bincio sync-garmin` command mirrors sync-strava: discovers all users
with garmin_creds.json, refreshes cached garth OAuth2 session, imports new
activities, and optionally POSTs to the rebuild endpoint.

systemd timer fires every 3h offset by 1h30m from Strava to avoid
simultaneous rebuilds. Status written to _garmin_sync_status.json per user.
2026-05-16 20:13:12 +02:00
Davide Scaini 307f1fbbc1 download bas: embed timeseries into the JSON so the file is self-contained 2026-05-15 18:38:53 +02:00
Davide Scaini c465e518e5 Add activity file downloads with per-activity download_disabled flag
New endpoint: GET /api/activity/{id}/download/{bas|original|gpx}
- bas: streams the BAS detail JSON as an attachment
- original: streams the original FIT or GPX file from originals/
- gpx: generates a GPX from the timeseries (always available when GPS exists)

download_disabled flag stored in sidecar (edits/{id}.md), propagated to
the merged BAS detail JSON. When set, only the owner can download.

Backend: ops.py writes flag to sidecar; merge.py propagates it to detail
JSON; download.py implements the endpoint; server.py registers the router.
Frontend: EditDrawer gets a "No download" toggle button; ActivityDetail
shows a Download section (hidden when disabled and viewer is not the owner).
2026-05-15 18:35:40 +02:00
Davide Scaini fe437626e6 Global feed: switch from sequential pages to month-based BAS shards
feed.json is now a BAS shard index pointing to feed-YYYY-MM.json files
(~150 activities / ~25 KB gzip each) instead of 400+ sequential feed-N.json
pages. The frontend can now jump directly to a specific month when filtering
by year or date range, without loading all newer data first.

- merge.py: write_combined_feed groups by YYYY-MM and emits a shard index
- dataloader.ts: isYearShardUrl matches feed-YYYY-MM.json; loadCombinedFeed
  returns pendingShards; FeedPage interface and loadCombinedFeedPage removed
- ActivityFeed.svelte: _yearFromShard handles both index-YYYY and feed-YYYY-MM;
  feedNextPage/feedTotalPages/loadingAllFeedPages removed; infinite-loop bug
  fixed (toLoad.length guard before setting loadingAllShards); onMount uses
  pendingShards from loadCombinedFeed
2026-05-15 10:25:01 +02:00
Davide Scaini d2151a4acf Ideas: add reopen button when awaiting; add /reopen endpoint 2026-05-15 09:07:49 +02:00
Davide Scaini 15e9969ca2 Ideas: add 'won't implement' status with decline/reopen button 2026-05-15 08:36:31 +02:00
Davide Scaini ed6a7ed39c Ideas: add 'awaiting feedback' status with amber section + admin comment
Status cycles open → awaiting → done → reopen.
Awaiting ideas float to the top in a 'Waiting for your feedback' section
with an amber border (#f59e0b).

Admin can attach an implementation note to any awaiting idea via
POST /api/ideas/{id}/comment. The note appears inside the same card
in a distinct sub-box with a subtle amber tint border, editable inline.
The sub-box is visible to all users once a note exists.
2026-05-15 08:18:44 +02:00
Davide Scaini 3b675a68b0 Elevation: skip near-zero dropout values mid-recording
Devices (Apple Watch, some GPS units) record 0.0 when they lose barometric/GPS
lock mid-activity. The old accumulation committed these as real sea-level points,
inflating both gain and loss by the current elevation (e.g. 792m dropout on the
Cosmo Walk added ~1584m of phantom gain+loss).

Fix: skip any elevation value < 1.0m when the current committed elevation is
significantly above zero (> threshold). Gradual legitimate descents to sea level
are unaffected because intermediate values are committed along the way.

Add --recompute-elevation flag to bincio render to backfill existing activities.
2026-05-15 01:21:34 +02:00
Davide Scaini 4ea2292e2b Indoor detection: title-based inference in merge layer + fix _merge_all_locked
- Add _INDOOR_TITLE_RE / _infer_indoor_title() to writer.py (matches zwift,
  ftp-builder, turbo-trainer, rodillo); replaces the narrower zwift-only regex
  that was local to write_athlete_json
- _is_outdoor now delegates to _infer_indoor_title so all four keywords are
  excluded from records and MMP aggregation
- apply_sidecar and _apply_sidecar_summary both set sub_sport=indoor when the
  title matches and no explicit sub_sport is already present
- _merge_one_locked: detect title-inferred activities as needs_merge and call
  apply_sidecar({},{}) so the _merged copy gets sub_sport=indoor written
- _merge_all_locked: read index upfront to populate to_merge with title-inferred
  IDs; call apply_sidecar({},{}) for activities in to_merge without sidecars;
  apply _apply_sidecar_summary to ALL summary entries (not only sidecar ones)
2026-05-15 01:03:17 +02:00
Davide Scaini a863cdd663 Records: exclude Zwift activities by title; show Saved confirmation before closing drawer
- _is_outdoor now also excludes activities whose title matches /\bzwift\b/i,
  covering the ~50 Strava-imported Zwift rides that lack sub_sport metadata.

- EditDrawer waits 900ms after a successful save before dispatching 'saved'
  (which closes the drawer), so the green "Saved" confirmation is visible.
2026-05-15 00:49:46 +02:00
Davide Scaini 9f1e9e4d3b Records: apply sidecars before computing; fix best_climb_m for long mountain climbs
- _rebuild_athlete_json now applies sidecar edits (sub_sport, sport, etc.)
  in-memory before passing summaries to write_athlete_json, so activities
  marked indoor via sidecar are correctly excluded from records.

- _best_climb now runs Kadane's over cumulative distance (not 1Hz dense
  time) so recording pauses don't create None gaps that falsely reset the
  climbing window. Grappa: 811m→1603m; Nivolet: 311m→2009m.

- Add bincio render --recompute-climbs to backfill existing activities
  from their stored timeseries.
2026-05-15 00:30:58 +02:00
Davide Scaini de07d8d4cf activities: trigger rebuild after edit so records update immediately 2026-05-15 00:09:51 +02:00
Davide Scaini 1ce94b8536 records: exclude indoor/treadmill/virtual sub_sport; rebuild athlete.json on bake
- fit.py: map FIT sub_sport 'treadmill' and 'virtual' to 'indoor'
- writer.py: broaden _is_outdoor to catch all indoor sub_sport variants
- render/cli.py: rebuild athlete.json from index.json on every bake so
  records never go stale when the exclusion logic changes
2026-05-14 23:37:54 +02:00
Davide Scaini 5167f2a988 explore: shard tracks into per-year files for progressive loading
bake_tracks now writes tracks_YYYY.json shards + tracks_index.json manifest
instead of a single monolithic tracks.json. API /api/me/tracks returns the
manifest; /api/me/tracks/{year} serves individual shards. Explore.svelte
fetches the two most recent years eagerly then streams the rest in the
background so the map renders immediately with recent data.
2026-05-14 18:34:53 +02:00
Davide Scaini c36b95e041 segments detect: add --fresh flag to clear efforts before re-detecting 2026-05-14 17:11:11 +02:00
Davide Scaini 862226305a Fix segment avg_speed: derive from distance/time; tighten speed bounds to reject false matches 2026-05-14 17:09:41 +02:00
Davide Scaini 8ff781661e Fix feedback JSON encoding: use ensure_ascii=False to preserve accented characters 2026-05-14 17:04:44 +02:00
Davide Scaini 5593764fdb explore: skip legacy bare-timestamp geojsons; type pill colors visible when inactive 2026-05-14 15:55:10 +02:00
Davide Scaini 537d1bb712 explore: exclude indoor/virtual activities from tracks.json 2026-05-14 14:34:44 +02:00
Davide Scaini 5307ae287c Explore: personal GPS heatmap tab under Athlete page
- bincio/explore.py: bake_tracks() simplifies GPS coords (RDP ε=0.0001),
  strips to [lng,lat], groups by sport type, writes per-handle tracks.json
- bake-tracks CLI command; render CLI calls _bake_tracks() after each build;
  strava_zip runs it once at end of batch
- /api/me/tracks endpoint serves the baked file; wipe_user cleans it up
- Explore.svelte: MapLibre full-screen map with sidebar — type pills,
  year/month date filter, Lines / Heatmap (global or by-type) view modes
- AthleteView: Explore tab visible only to profile owner (checks __bincioMe)
- Base.astro: fullscreen prop + Planner nav link
2026-05-14 14:31:21 +02:00
Davide Scaini 1a7d1dc8c3 serve: complete CurrentUserResponse model (add wiki_access, activity_access, dem_configured) 2026-05-14 11:06:35 +02:00
Davide Scaini a10164b932 metrics: fall back to GPS-derived speed in compute_best_efforts when device speed absent
FIT files from older devices (and GPX/TCX files) often omit the speed field.
The sliding-window best-effort algorithm was treating all such points as speed=0,
so no records were ever produced for these activities.

Fix: when p.speed_kmh is None but consecutive lat/lon are available, compute
haversine segment speed and spread it evenly across the 1Hz interval slots.
This mirrors what _gps_stats already does for avg/max speed computation.
2026-05-14 09:16:01 +02:00
Davide Scaini c85d2edf39 dev: fix _start_serve to set deps.data_dir/site_dir (not srv.*) after Step 3 split 2026-05-14 00:09:41 +02:00
Davide Scaini 27f6d141f7 Refactor step 4: narrow broad except Exception catches
Replaced 28 bare `except Exception` catches across 8 files with specific
exception types reflecting the actual failure modes:

- JSON file reads → (OSError, json.JSONDecodeError)
- datetime parsing → ValueError
- base64 decoding → ValueError
- YAML parsing → (OSError, yaml.YAMLError); import moved above try
- GeoJSON coord extraction → (TypeError, IndexError, AttributeError)
- Startup temp-file cleanup → OSError
- Single JSON line parsing (SSE batch) → json.JSONDecodeError

Kept broad catches only where intentional:
- Background thread top-level guards (tasks.py, admin.py) with log.exception
- SSE stream generator tops (strava.py, garmin.py, uploads.py)
- Per-item batch loops that must not abort the whole operation
- Explicitly non-fatal post-upload merge steps with log.warning
2026-05-13 23:58:14 +02:00
Davide Scaini 8380b1d2cc Refactor: split serve/server.py (3220 lines) into focused modules
serve/server.py is now 69 lines — app factory, middleware, and router
registration only.

New modules:
  deps.py    (168 lines) — module-level globals + auth dependency functions
  models.py   (85 lines) — all Pydantic request/response models
  tasks.py   (136 lines) — background workers and job tracker
  routers/               — one file per domain (10 routers, ~2750 lines total)
    auth.py, me.py, admin.py, activities.py, uploads.py,
    segments.py, strava.py, garmin.py, ideas.py, feed.py

cli.py updated to set globals on deps instead of server.

88 new regression tests in tests/serve/ cover auth guards and key
behaviours for every router; 294 total passing after the split.
2026-05-13 23:47:19 +02:00
Davide Scaini 2ec4d9157c Refactor: extract edit UI HTML into bincio/edit/templates/edit.html
The 285-line _HTML string literal in edit/server.py is replaced by a
template file loaded at request time. The route handler is unchanged in
behaviour — it still substitutes __SITE_URL__, __SPORT_OPTIONS__, and
__STAT_CHECKBOXES__ before returning the response.

Five new tests cover: 200 response, form presence, site_url injection,
no unresolved placeholders, and template file existence on disk.
2026-05-13 23:19:19 +02:00
Davide Scaini 9dd533825f Fix pre-existing test failures in test_writer and test_metrics
test_writer: _dummy_metrics() and test_build_summary_required_fields were
missing np_power_w=None after the field was added to ComputedMetrics.

test_metrics: the leading-zero elevation heuristic fired on a single 0.0
start value, incorrectly skipping the first legitimate elevation step.
Guard now requires at least 2 consecutive near-zero leading values before
activating the Apple Watch lock-acquisition workaround.
2026-05-13 23:15:26 +02:00
Davide Scaini e61d05fc41 Refactor: extract shared image upload utilities into bincio/shared/images.py
ALLOWED_IMAGE_TYPES, MAX_IMAGE_BYTES, and unique_image_name() were
duplicated identically in both the edit and serve servers. Centralising
them means a single change point for any future extension (e.g. adding
image/avif support).

Tests added in tests/test_shared_images.py cover no-collision, single
and chained collisions, no-suffix filenames, and constant values.
2026-05-13 23:13:08 +02:00
Davide Scaini cd97e4cc87 CORS: allow all *.bincio.org origins (for planner.bincio.org) 2026-05-13 22:51:00 +02:00
Davide Scaini 58a5d5b450 Strava sync: skip import when a FIT-file upload already covers the same start time
Before importing each Strava activity, build a set of existing timestamp
prefixes (YYYY-MM-DDTHHMMSSZ) from the activities directory. If the incoming
Strava activity matches an existing prefix, record its Strava ID as done and
skip — preventing duplicate entries when a FIT file and a Strava sync both
cover the same ride.

Also reports skipped-existing count in the summary line.
2026-05-13 21:52:07 +02:00