astro build resolves the public/data symlink and copies all activity JSON
into dist/; rsync then copied that to the webroot — but nginx already serves
/data/ directly from /var/bincio/data/ via alias, so both copies were dead
weight. Freed 36 GB → 14 GB on the live server.
- post-receive hook: prune dist/data/ before rsync, add --exclude=data/
- docs: update manual rebuild command and nginx comment to match
- serve/server.py: _mb() now uses lstat() to count symlinks at face value
rather than following them to targets, so admin storage panel no longer
double-counts _merged/ (which is mostly symlinks into activities/)
- serve/server.py GET adds private:bool to the response (true when
privacy is "unlisted" or legacy "private") so EditDrawer can read it
- edit/server.py GET: same fix for the single-user edit server
- EditDrawer: fall back to d.privacy if d.private is absent; rename
"Private" toggle label to "Unlisted"
rewriteActivityUrls now skips URLs that are already absolute (start with
/ or http). Before this fix, the new user→year two-level nesting caused
year-shard URLs (/data/brut/_merged/activities/X.json) to be prepended
again at the user-shard level, producing broken doubled paths and making
every activity show "Activity not found".
delete_activity now updates data_dir/index.json so merge_all no longer
re-adds the summary for a deleted activity, preventing the broken
"Activity not found" state after deletion.
ActivityDetailLoader switches from loadIndex (all year shards) to
loadIndexPaged (first year shard only) + direct file fallback, so
opening an activity detail page no longer downloads the entire history.
merge_all/_merged/index.json is now a shard manifest; activities are
split into index-{year}.json files. The feed loads only the most-recent
year on first paint (~200 activities instead of all of them). Older
years are fetched lazily when the user clicks "Load older activities".
Also strips best_efforts / best_climb_m / source from shard files —
these fields are aggregation inputs only, never read by the feed UI.
Add application/gzip, application/x-gzip, and .gz to the accept
attribute so browsers show .gpx.gz / .fit.gz / .tcx.gz in the picker.
Browsers often ignore double-extension filters (.gpx.gz) without the
matching MIME type.
- CHANGELOG.md: add [Unreleased] 2026-04-16 section covering settings
page, admin tools, password reset, re-extract, community page, SSE
upload progress, and all bug fixes since 2026-04-10
- Remove docs-proposal.md (internal planning doc, not user-facing)
- Remove publish/ directory (leftover artefacts from publish.sh, not
meant to be tracked)
- scripts/pull_feedback.sh: replace hardcoded default VPS IP with a
required positional argument to avoid leaking server address
- docs/squash-for-github.md: document the squash-for-github commit
strategy for future reference
- EditDrawer: stop auto-inserting  into description on
upload — images are tracked via custom.images; the refs only cluttered
the textarea. Strip any pre-existing refs on load so old sidecars are
also cleaned up when the drawer is opened.
- ActivityDetail: imageBase now treats detail_url that starts with '/'
as already-absolute (same fix pattern as track_url / detail_url);
was prepending ${base}data/ on top of /data/... → double path.
Move "Delete original files" out of the Storage card and into the
Danger zone as a first, less-destructive step (simple confirm, no
password needed). "Delete all activity data" and "Delete account"
follow below it, both still password-gated. Descriptions clarify
exactly what each action does and does not remove.
- user_prefs table in db.py with get/set helpers
- GET/PUT /api/me/prefs endpoints for bulk pref management
- GET/PUT/DELETE /api/me/strava-credentials; PUT preserves existing
secret when client_secret field is left blank
- _strava_creds() helper resolves per-user → instance fallback across
all five Strava endpoints
- Settings page: Navigation card (hide Feed/Community/About toggles)
and Strava credentials card
- Base.astro: ids on feed/community/about nav links; applies
nav_hide_* prefs after login
API endpoints (all auth-gated to the logged-in user):
- GET /api/me/storage — per-category disk breakdown
- DELETE /api/me/originals — free originals/ dir (post-extraction cleanup)
- DELETE /api/me/activities — wipe all activity data (password confirm)
- DELETE /api/me — delete account + all data (password confirm)
- PUT /api/me/display-name — update display name
- PUT /api/me/password — change password (requires current password)
Page at /settings/:
- Storage card: activities / originals / Strava originals / photos / total
with one-click 'Delete original files' when originals exist
- Profile card: display name field with inline save
- Password card: change password form
- Danger zone: delete all activities or delete account (both require
password confirmation in a modal before proceeding)
Nav: 'Settings' link appears in the top bar after login (same as Admin).
When 'Overwrite existing activities' is checked, duplicate activities are
re-extracted and replaced instead of silently skipped:
- Deletes {id}.json, .geojson, .timeseries.json from activities/ and _merged/
- Removes the stale index summary and dedup cache entry
- Ingests the new file fresh via ingest_parsed
- Reports 'overwritten' (↺) status in the SSE stream vs 'imported' (↓)
- done event includes 'overwritten' count; UI shows it alongside 'added'
- /api/admin/disk now includes in_db flag per user (true if account exists in DB)
- Ghost users (directory exists, no DB account) show amber 'ghost' badge and only
Diag + Delete dir buttons (no Re-extract, Rebuild, Reset pwd, Reset data)
- DELETE /api/admin/users/{handle}/directory wipes the entire directory and updates
the root manifest; refuses if the account still exists in the DB
- Wires up rmdir-btn with a window.confirm before calling the new endpoint
resolveShards also rewrites track_url to absolute paths (/data/…).
The trackUrl reactive statement only handled http:// prefixes,
producing double /data//data/… for the GeoJSON fetch → map had no track.
resolveShards rewrites detail_url to absolute paths (e.g. /data/brut/_merged/activities/{id}.json)
when fetching from a user shard. loadActivity and loadTimeseries only checked for http:// prefixes
and treated /data/... paths as relative, producing double /data//data/... in the fetch URL → 404.
Fix: treat URLs starting with / as already absolute, same as http:// URLs.
Strava originals with absolute Unix timestamps stored as elapsed-second
offsets produce a t_max of ~1.6 billion. compute_mmp and compute_best_efforts
both create dense 1Hz arrays via range(t_min, t_max+1), which for a 1.6B
span allocates 44+ GB and OOM-kills the process. Add a >1-week sanity
check and return None early for corrupt streams.
Root cause: old Strava activities (seen from 1970-epoch start_date)
where the time stream contains absolute Unix timestamps instead of
elapsed seconds.
One Python process for 2015 activities exhausts all RAM + swap on a
cheap VPS. Split into sequential batches of 100: each subprocess handles
100 activities and exits, returning all memory to the OS before the
next batch starts. The server chains batches in the SSE event_stream
and triggers a single rebuild when all batches complete.
CPython's allocator holds freed memory in arenas and doesn't return it to
the OS, causing RSS to grow throughout the 2015-activity loop even when
each iteration's objects are freed. Call gc.collect() + malloc_trim(0)
every 50 activities to return freed pages to the kernel and keep RSS flat.
uv is unreliable in systemd environments where PATH omits ~/.local/bin.
Use sys.executable's parent directory to find the venv's bincio script
directly — this always works since the server itself runs from the venv.
The in-process approach loaded all 2015 Strava originals into the server
process memory, causing OOM kills. Now spawns `bincio reextract-originals`
as a child process; heavy work runs in an isolated Python interpreter that
exits when done, freeing all memory.
Also adds `bincio reextract-originals` as a standalone CLI command that
prints JSON-lines progress to stdout — useful for running directly on the
VPS via SSH for large backlogs.
The per-call run_in_executor pattern caused network errors.
New approach: one thread runs the entire extraction loop and puts
SSE strings into an asyncio.Queue via call_soon_threadsafe; the
async generator drains the queue. This is the correct pattern for
background-thread + SSE streaming in FastAPI.
The sync generator was failing with a network error because Starlette's
iterate_in_threadpool doesn't properly propagate exceptions from sync
generators — the connection resets with no body.
Fix: convert event_stream to an async generator (Starlette handles these
natively without thread wrapping), move imports to the endpoint function
scope so failures raise HTTPException before the stream starts, and run
CPU-intensive work (parse + write) via loop.run_in_executor so the
async generator can actually yield between activities.
- Generator now yields a 'status' event immediately so the client can
distinguish 'working' from 'failed silently before first event'
- Batch mode: call write_activity per file but write index.json and
athlete.json only once at the end (was O(n²) — 2015 rewrites)
- JS: check r.ok before reading the body stream; show HTTP error detail
instead of staying stuck at 'Starting…'
- Handle 'status' event type in the progress log
- POST /api/admin/users/{handle}/reextract-originals: reads stored
originals/strava/*.json and re-runs strava_to_parsed + ingest_parsed
without hitting the Strava API; streams SSE progress; calls merge_all
and rebuild on completion
- GET /api/admin/users/{handle}/diag: now shows _merged/activities/
file counts, a sample of filenames in activities/ (with symlink flag),
and lists pending_files by name
- Admin page: Re-extract button per user with live SSE progress modal
- bincio.serve logger wired into uvicorn output: rebuild steps, upload
errors, strava-zip progress all now appear in the server log
- _trigger_rebuild: capture stdout/stderr, log errors instead of silently
discarding; exceptions logged with traceback instead of swallowed
- upload handler: log per-file errors with traceback; include error detail
in the SSE event sent back to the browser
- strava-zip handler: log imported/error counts on completion
- GET /api/admin/users/{handle}/diag: snapshot of a user's data dir
(file counts, sizes, index activity counts, pending uploads)
- POST /api/admin/users/{handle}/rebuild-sync: blocking rebuild that
returns full stdout/stderr — for debugging without SSH log access
- Admin page: Diag button per user opens a modal showing the diag JSON
vis.js requires a pixel-sized container — flex:1 is ignored.
Use position:fixed toolbar + JS-measured height for the graph div,
stored as window._network for resize handling.
getStaticPaths now returns [] — all /activity/{id}/ URLs are served by
the activity/index.html shell via nginx try_files and hydrated by
ActivityDetailLoader. Pre-rendering thousands of pages was exhausting
server RAM and killing the build. The dynamic loader already handles
public, unlisted, and local activities identically.
db.py: reset_codes table (code, handle, created_by, created_at,
expires_at, used_at); create_reset_code() invalidates any prior unused
code for the same handle; use_reset_code() validates handle match,
expiry (24 h), and single-use; change_password() updates the hash.
server.py: POST /api/admin/users/{handle}/reset-password-code (admin)
returns a code; POST /api/auth/reset-password (public) validates the
code + handle and sets the new password.
Admin page: "Reset pwd" button per user — shows the code inline on
click (monospace, click-to-copy).
/reset-password/ page: handle + code + new password form.
Login page: "Forgot password?" link.
The old DELETE /api/admin/users/{handle}/activities only removed *.json
files and _merged/, leaving originals/ (Strava FIT files) and edits/
untouched — causing the 968 MB disk usage after a delete.
_wipe_user_activities() now removes activities/, edits/, originals/,
_merged/, index.json, athlete.json, and .bincio_cache.json. Admin page
button renamed to "Reset data" with updated confirmation text.
Server endpoint removes the activity JSON, GeoJSON, timeseries, sidecar
edit, and images directory. Also purges the dedup cache entry so the
file can be re-uploaded if needed. Runs merge_all + rebuild afterwards.
EditDrawer: two-click delete button (click once → "Confirm delete?",
click again → deletes). On success, dispatches 'deleted' event.
ActivityDetail navigates back to the feed on delete.
FIT parser: try enhanced_altitude before altitude. Barometric altimeters
on modern Garmins (Edge 540, 840, etc.) write enhanced_altitude in
record messages and total_ascent in lap messages. The old code read only
altitude, producing null elevation_m per point → null elevation_gain_m
at the activity root while laps had correct values from total_ascent.
ActivityMap: use preview_coords (passed from ActivitySummary) to
initialise the map at the activity's location on mount, eliminating the
flash of world-view before the async detail JSON / bbox arrives.
Single-activity writes now trigger a fast merge_one instead of a full
user rebuild. post_activity was fixed earlier; this completes the fix
for upload_image and delete_image endpoints.
- "unlisted" = not shown in the public feed, but GPS track, timeseries
and detail JSON are all accessible by direct URL (security by obscurity)
- "private" accepted as legacy alias everywhere (backward compat with
existing data on disk)
- New writes from Strava sync / ZIP upload / sidecar use "unlisted"
- Only "no_gps" now suppresses the GPS track
- isUnlisted() helper in format.ts used by all Svelte/Astro components
- SCHEMA.md and CLAUDE.md document the privacy model and the distinction
between "unlisted" and "no_gps"
If the index-based lookup fails (shard fetch silently failed, stale
index state, etc.), try fetching the activity detail file directly from
each user shard's _merged/activities/ directory. This makes private
activities and newly-synced activities accessible even when the index
resolution fails.
Also add console.error logging when shards fail in resolveShards to
help diagnose root causes.
added 3256 more.
- danilo: _merged/ is 8 KB — basically empty. merge_all likely ran concurrently (multiple file uploads trigger multiple rebuilds without a lock in --no-build mode),
causing a race where shutil.rmtree(merged_acts) from one run wiped what another run was writing.
Two fixes: serialize --no-build rebuilds with the same lock, and add a "Rebuild" button to the admin page.
Root causes fixed:
1. merge_all race condition — --no-build rebuilds now hold _rebuild_lock, same as full builds
2. The SSE rebuild-trigger bug (already fixed earlier) was brut's original cause
next server restart.
Admin page now shows:
- Overall disk bar (used/free/%)
- Per-user table: total, activities (with file count), originals (with Strava breakdown), merged, images
- A mini bar per user showing relative size
- Red ⚠ warning if orphaned temp ZIPs are still present for a user
- Delete activities button (reloads sizes after)
Davide Scaini