Ideas and votes are stored as flat JSON files in /var/bincio/_ideas/,
following the same filesystem-first philosophy as segments and efforts.
Vote toggling uses fcntl exclusive locking to prevent concurrent writes.
Both trigger_detect and me_segment_rescan were appending-only, so false
efforts recorded before the geometric speed check fix remained after
rescan. Now each rescan path clears the effort file first, making the
result authoritative.
- detect.py: truncate started_at to seconds so dedup key survives JSON round-trip
- store.py: dedup by (activity_id, iso-started_at) string key, not object equality
- server.py: extract _scan_segment_for_user helper; trigger background scan
for the creating user's activities when a new segment is saved
- best_activity_id now included in segment_summary API response
- Best time is a direct link to the activity that produced it
- Clicking a row expands an inline effort list (lazy-loaded from
/api/segments/{id}/efforts): date linked to activity, time, Δ vs PR
- Clicking again collapses; ▲/▼ chevron shows state
New API endpoints:
- GET /api/segments/{id} — single segment metadata
- GET /api/activities/{id}/segment_efforts — efforts for an activity (auth)
- GET /api/users/{handle}/segment_summary — public best time + count per segment
New components:
- SegmentDetail.svelte — map + metadata + effort table (with PR/Δ) + rescan button
- SegmentsPage.svelte — URL router: shows detail when /segments/{id}/, list otherwise
Updated:
- segments/index.astro — now uses SegmentsPage router
- nginx-activity.conf — add /segments/ try_files rule for client-side routing
- ActivityDetail.svelte — segment efforts block below laps
- AthleteView.svelte — Segments tab with best time + effort count per segment
- format.ts — add formatElapsed() for compact m:ss display
When a user saves new Strava credentials with a different client_id,
auto-delete the existing token (it belongs to a different OAuth app
and will always fail on refresh). Add POST /api/strava/disconnect
endpoint and a "Disconnect from Strava" button in settings, visible
only when connected.
Immediate: deleted diego_p's stale token so he can reconnect.
Apple Watch and similar devices record exactly 0.0 for elevation while
waiting for barometric/GPS lock, then jump to the real altitude. The
hysteresis accumulator was seeding from 0.0, counting the full jump as
ascent. Fix: detect a leading near-zero run followed by a large jump
and seed the accumulator from the first real value instead.
Applied in both _elevation() (fresh extractions) and
recalculate_elevation_hysteresis() (recompute path). Added a bulk
admin endpoint POST /api/admin/users/{handle}/recompute-elevation and
corresponding button to fix existing stored activities.
Each sync run now writes _strava_sync_status.json per user (status,
imported count, error message). New admin endpoints expose this data
and allow triggering an on-demand sync. The admin page gains a Strava
Sync section showing per-user token/credentials state, total imported,
last sync time, and last-run status with inline error messages.
- Add bincio sync-strava command: headless multi-user Strava sync
designed for systemd timer. Discovers users via strava_token.json,
skips users without their own strava_credentials.json, respects
Strava visibility (only_me → unlisted). Treats 404 stream errors as
no-GPS activities rather than retrying every run.
- Add deploy/systemd/bincio-sync.{service,timer}: runs every 3 hours,
Persistent=true to catch up after downtime.
- Add POST /api/internal/rebuild: webhook for sync timer to trigger
site rebuild, authenticated via X-Sync-Secret header.
- Add suspended column to users table with auto-migration on open_db.
Suspended users are blocked at login and session lookup (covers both
activity site and wiki, which share instance.db).
- Add POST /api/admin/users/{handle}/suspend|unsuspend and
DELETE /api/admin/users/{handle}/account endpoints.
- Admin panel: Suspend/Unsuspend toggle, Del account button, suspended
badge on user row.
Replace per-upload Astro build threads with a single background worker
(_site_rebuild_worker) that waits on an event, sleeps 60 s to let upload
bursts settle, then runs one full build + rsync. 271 concurrent uploads now
produce one build instead of 271 serialised builds, eliminating the OOM kill.
--webroot is re-enabled; merge-only path still runs immediately per upload.
Also: date filter row added to ActivityFeed.svelte (sport + date presets
with dynamic year pills); deploy/vps gitignored for VPS config backups.
Adds edits_json column (migration v3) to store user overrides separately
from detail_json so Option A server re-extraction never clobbers them.
- Tap the title in the detail screen to edit (local activities only, shown
with a ✎ hint). Saves on keyboard dismiss via onEndEditing.
- Cards and search display user_title ?? title.
- Raw upload: user_title sent to server -> sidecar written so web UI shows
the correct title (server re-extracts from FIT, which has Karoo's title).
- BAS upload: detail.title overridden before sending, no sidecar needed.
Four related issues made uploading 271+ activities unreliable:
1. merge_all/write_combined_feed were inside the extraction try/except —
any merge race returned 422 even though the file was on disk, causing
the mobile app to permanently mark the upload as failed. Fixed by
moving them to a separate best-effort try/except after the extraction
block. Switch to merge_one (single-activity symlink) instead of
merge_all (full rebuild) so each upload is O(1) FS ops, not O(N).
2. The dev watcher fired merge_all for every activity .json write AND the
upload endpoint also ran merge_all — O(N²) symlink operations during
bulk uploads. Watcher now skips activities/*.json changes (upload
endpoint handles those directly).
3. Vite/Chokidar followed the public/data symlink and opened a handle per
activity file; constant merge rebuilds exhausted file descriptors and
crashed the Astro dev server. Fixed with watch.ignored on public/data.
4. _write_year_shards and write_combined_feed used f.unlink() without
missing_ok=True — concurrent callers racing the same file threw
FileNotFoundError which propagated as a false extraction failure.
Three bugs fixed:
- /api/upload/bas and /api/upload/raw never updated user_dir/index.json, so
merge_all couldn't include uploaded activities in year shards — they existed
on disk but were invisible to the browser feed. Fixed by _upsert_index_summary()
called before merge_all().
- Silent catch {} in uploadLocalActivities swallowed all per-activity errors;
replaced with console.warn so failures are visible in Expo logs.
- After a server wipe, synced_at flags on the device caused "Nothing to upload"
forever. uploadFeed() now reconciles against GET /api/feed at the start of each
upload: local activities not found on the server get synced_at cleared.
Also: live upload progress ("Uploading N / M…"), failed count in result message,
onProgress callback on uploadFeed(), countPendingUploads() helper.
merge_all(user_dir) updates the per-user _merged/ shard but the home page
loads feed.json first via loadCombinedFeed. write_combined_feed was only
called by the CLI render command, not by the API upload endpoints or the
dev watcher, leaving feed.json permanently stale after any runtime upload.
Add write_combined_feed(_get_data_dir()) after every merge_all call in
/api/upload/bas, /api/upload/raw, the dev.py file watcher, and dev startup.
- Skip MapLibre on Android <29 (Karoo): SELinux denies kgsl-3d0 access
from untrusted_app context, crashing the GPU driver on any OpenGL
surface. Replace with SvgRouteView — equirectangular SVG route trace
using react-native-svg, no native GL surface needed.
- Add +/- zoom buttons to full-screen MapLibre map on modern devices
via Camera ref and onRegionDidChange.
- Skip PyodideWebView on Android <29: same GPU driver conflict; set
_engineUnavailable at module init via API level gate (< 29).
- Add engine_unavailable fast path in PyodideWebView: post message
immediately if WebAssembly.Global is absent (Chrome <69) instead of
attempting 30 MB Pyodide download.
- Add server-side extraction fallback (extractServer.ts): when engine
unavailable, POST raw file as base64 to /api/upload/raw; server runs
full Python pipeline and returns extracted data.
- Add /api/upload/raw endpoint in server.py.
- Add pre-flight auth check (checkServerAuth) before batch import so
an expired token errors immediately rather than after N files.
- Fix uploadLocalActivities in sync.ts: was reading original_path as
JSON (binary FIT file, always threw), silently skipping every upload.
Now reads detail_json from DB directly.
- Redesign Feed header: replace single Sync button with Upload /
Download / Refresh. Pull-to-refresh and Refresh button are local-only.
Auto-refresh on tab focus via useFocusEffect.
- Replace ActivityIndicator with plain Text everywhere (native animation
also crashes Karoo GPU driver).
- Raise macOS open-file limit in dev_test.py to prevent EMFILE errors
from Astro file watcher.
- Document all Karoo hardware constraints in docs/mobile-app.md.
- Server: POST /api/upload/bas accepts pre-extracted BAS JSON (activity + optional timeseries/geojson), writes files and triggers merge_all
- sync.ts: uploadLocalActivities reads unsynced local activities by original_path, POSTs to /api/upload/bas, marks synced_at on success
- Settings: Upload toggle (Off / Upload local activities) in Sync section with subLabel dividers for Download / Upload groups
- Feed: sync message includes uploaded count when activities are pushed
- Add /api/activity/{id}/geojson and /api/activity/{id}/timeseries endpoints
(bearer-token-gated, falls back from _merged to raw activities dir)
- Rewrite activity detail screen with MapLibreGL v11 API (Map, Camera,
GeoJSONSource, Layer) and react-native-svg area chart with gradient fill
- On-demand fetch for remote activities that have no local geojson/timeseries
- Add react-native-svg dependency; requires dev build (npx expo run:android)
_merged/index.json is a shard manifest with activities:[] when the user
has >FEED_PAGE_SIZE activities. The endpoint now collects from all
index-{year}.json shard files before returning.
SyncResult gains a `total` field (activities received from server) so the
feed screen can distinguish "server returned nothing" from "all already
stored locally". Messages: "No activities on instance" / "Up to date (N)"
/ "X of N activities synced".
Server (bincio/serve/server.py):
- Add _require_auth: accepts session cookie OR Authorization: Bearer token
- POST /api/auth/token: same as /api/auth/login but returns token in body
(password used once, not stored; mobile stores only the session token)
- GET /api/feed: auth-gated; reads _merged/index.json for the user and
returns the activities array as JSON
Mobile:
- db/sync.ts: syncFeed(db) fetches /api/feed, upserts each summary into
local SQLite as origin='remote'; skips locally-imported activities
- db/queries.ts: add upsertRemoteActivity (INSERT ... ON CONFLICT DO UPDATE
WHERE origin='remote' — never overwrites local imports); fix feed sort
order to started_at DESC instead of insertion order
- settings.tsx: Connect section — password field (not persisted) + Connect
button calls POST /api/auth/token and stores token; Disconnect clears it
- index.tsx: ↓ Sync button + pull-to-refresh both trigger syncFeed; cloud
badge on remote activities; empty state updated
- Add GET /api/wheel/download to serve/server.py and edit/server.py:
serves dist/bincio-*.whl via FileResponse; in production nginx takes
the request before FastAPI, so this is a no-op there but works locally
- wheel_version response now includes api_url: "/api/wheel/download"
alongside the nginx-served url field
- Bundle mobile/assets/bincio.whl (built from dist/) as an offline
fallback for Pyodide testing before the first instance sync
- docs/mobile-app.md: document dev setup — bundled asset, local server
endpoint, and how to refresh the bundle with uv build + cp
Same fix as cea1dbc (post-receive hook) but missed in server.py: Astro resolves
the public/data symlink and copies all activity JSON into dist/; without pruning,
every Strava sync / upload / edit that triggers a full build + rsync duplicates
GBs of data into the nginx webroot.
Both rsync callsites now rm -rf dist/data + pass --exclude=data/ to rsync.
- dem.py: apply 45s median filter before hysteresis to suppress SRTM
tile-boundary steps that were accumulating through the 5m threshold;
raise DEM hysteresis threshold from 5m to 10m
- dem.py: back up elevation_m as elevation_m_original in timeseries
before the first DEM overwrite, so original sensor data is preserved
- dem.py: add recalculate_elevation_hysteresis() — recomputes gain/loss
from original recorded elevation (reads elevation_m_original if a DEM
run already replaced elevation_m) using source-aware thresholds
(5m barometric, 10m GPS/unknown); does not touch the elevation array
- edit/server.py, serve/server.py: split /recalculate-elevation into
two endpoints: /recalculate-elevation/dem and
/recalculate-elevation/hysteresis
- EditDrawer.svelte: replace single DEM button with two side-by-side
buttons — "Recalculate (hysteresis)" (fast, offline) and
"Recalculate (DEM)" (SRTM lookup)
Adds a "Recalculate from terrain map (DEM)" button to the activity edit
drawer. On click it queries an Open-Elevation-compatible API to replace
GPS altitude with SRTM terrain data, applies 5m hysteresis, and updates
the activity's elevation stats and timeseries chart in place.
- bincio/extract/dem.py: lookup_elevations() (batched HTTP POST) +
recalculate_elevation() (subsample → DEM → interpolate → hysteresis →
patch activity JSON, timeseries JSON, index.json)
- POST /api/activity/{id}/recalculate-elevation on both serve and edit
servers; serve endpoint is auth-gated and triggers merge + rebuild
- --dem-url flag (also DEM_URL env var) on bincio serve and bincio edit;
logged at startup; missing URL returns a clear 503 with setup instructions
- /api/me response gains dem_configured bool
- EditDrawer: button with loading state, shows new ↑/↓ values on success
astro build resolves the public/data symlink and copies all activity JSON
into dist/; rsync then copied that to the webroot — but nginx already serves
/data/ directly from /var/bincio/data/ via alias, so both copies were dead
weight. Freed 36 GB → 14 GB on the live server.
- post-receive hook: prune dist/data/ before rsync, add --exclude=data/
- docs: update manual rebuild command and nginx comment to match
- serve/server.py: _mb() now uses lstat() to count symlinks at face value
rather than following them to targets, so admin storage panel no longer
double-counts _merged/ (which is mostly symlinks into activities/)
- serve/server.py GET adds private:bool to the response (true when
privacy is "unlisted" or legacy "private") so EditDrawer can read it
- edit/server.py GET: same fix for the single-user edit server
- EditDrawer: fall back to d.privacy if d.private is absent; rename
"Private" toggle label to "Unlisted"
delete_activity now updates data_dir/index.json so merge_all no longer
re-adds the summary for a deleted activity, preventing the broken
"Activity not found" state after deletion.
ActivityDetailLoader switches from loadIndex (all year shards) to
loadIndexPaged (first year shard only) + direct file fallback, so
opening an activity detail page no longer downloads the entire history.
- user_prefs table in db.py with get/set helpers
- GET/PUT /api/me/prefs endpoints for bulk pref management
- GET/PUT/DELETE /api/me/strava-credentials; PUT preserves existing
secret when client_secret field is left blank
- _strava_creds() helper resolves per-user → instance fallback across
all five Strava endpoints
- Settings page: Navigation card (hide Feed/Community/About toggles)
and Strava credentials card
- Base.astro: ids on feed/community/about nav links; applies
nav_hide_* prefs after login
API endpoints (all auth-gated to the logged-in user):
- GET /api/me/storage — per-category disk breakdown
- DELETE /api/me/originals — free originals/ dir (post-extraction cleanup)
- DELETE /api/me/activities — wipe all activity data (password confirm)
- DELETE /api/me — delete account + all data (password confirm)
- PUT /api/me/display-name — update display name
- PUT /api/me/password — change password (requires current password)
Page at /settings/:
- Storage card: activities / originals / Strava originals / photos / total
with one-click 'Delete original files' when originals exist
- Profile card: display name field with inline save
- Password card: change password form
- Danger zone: delete all activities or delete account (both require
password confirmation in a modal before proceeding)
Nav: 'Settings' link appears in the top bar after login (same as Admin).
When 'Overwrite existing activities' is checked, duplicate activities are
re-extracted and replaced instead of silently skipped:
- Deletes {id}.json, .geojson, .timeseries.json from activities/ and _merged/
- Removes the stale index summary and dedup cache entry
- Ingests the new file fresh via ingest_parsed
- Reports 'overwritten' (↺) status in the SSE stream vs 'imported' (↓)
- done event includes 'overwritten' count; UI shows it alongside 'added'
- /api/admin/disk now includes in_db flag per user (true if account exists in DB)
- Ghost users (directory exists, no DB account) show amber 'ghost' badge and only
Diag + Delete dir buttons (no Re-extract, Rebuild, Reset pwd, Reset data)
- DELETE /api/admin/users/{handle}/directory wipes the entire directory and updates
the root manifest; refuses if the account still exists in the DB
- Wires up rmdir-btn with a window.confirm before calling the new endpoint
One Python process for 2015 activities exhausts all RAM + swap on a
cheap VPS. Split into sequential batches of 100: each subprocess handles
100 activities and exits, returning all memory to the OS before the
next batch starts. The server chains batches in the SSE event_stream
and triggers a single rebuild when all batches complete.
uv is unreliable in systemd environments where PATH omits ~/.local/bin.
Use sys.executable's parent directory to find the venv's bincio script
directly — this always works since the server itself runs from the venv.
The in-process approach loaded all 2015 Strava originals into the server
process memory, causing OOM kills. Now spawns `bincio reextract-originals`
as a child process; heavy work runs in an isolated Python interpreter that
exits when done, freeing all memory.
Also adds `bincio reextract-originals` as a standalone CLI command that
prints JSON-lines progress to stdout — useful for running directly on the
VPS via SSH for large backlogs.
The per-call run_in_executor pattern caused network errors.
New approach: one thread runs the entire extraction loop and puts
SSE strings into an asyncio.Queue via call_soon_threadsafe; the
async generator drains the queue. This is the correct pattern for
background-thread + SSE streaming in FastAPI.
The sync generator was failing with a network error because Starlette's
iterate_in_threadpool doesn't properly propagate exceptions from sync
generators — the connection resets with no body.
Fix: convert event_stream to an async generator (Starlette handles these
natively without thread wrapping), move imports to the endpoint function
scope so failures raise HTTPException before the stream starts, and run
CPU-intensive work (parse + write) via loop.run_in_executor so the
async generator can actually yield between activities.
- Generator now yields a 'status' event immediately so the client can
distinguish 'working' from 'failed silently before first event'
- Batch mode: call write_activity per file but write index.json and
athlete.json only once at the end (was O(n²) — 2015 rewrites)
- JS: check r.ok before reading the body stream; show HTTP error detail
instead of staying stuck at 'Starting…'
- Handle 'status' event type in the progress log
- POST /api/admin/users/{handle}/reextract-originals: reads stored
originals/strava/*.json and re-runs strava_to_parsed + ingest_parsed
without hitting the Strava API; streams SSE progress; calls merge_all
and rebuild on completion
- GET /api/admin/users/{handle}/diag: now shows _merged/activities/
file counts, a sample of filenames in activities/ (with symlink flag),
and lists pending_files by name
- Admin page: Re-extract button per user with live SSE progress modal
- bincio.serve logger wired into uvicorn output: rebuild steps, upload
errors, strava-zip progress all now appear in the server log
- _trigger_rebuild: capture stdout/stderr, log errors instead of silently
discarding; exceptions logged with traceback instead of swallowed
- upload handler: log per-file errors with traceback; include error detail
in the SSE event sent back to the browser
- strava-zip handler: log imported/error counts on completion
- GET /api/admin/users/{handle}/diag: snapshot of a user's data dir
(file counts, sizes, index activity counts, pending uploads)
- POST /api/admin/users/{handle}/rebuild-sync: blocking rebuild that
returns full stdout/stderr — for debugging without SSH log access
- Admin page: Diag button per user opens a modal showing the diag JSON
db.py: reset_codes table (code, handle, created_by, created_at,
expires_at, used_at); create_reset_code() invalidates any prior unused
code for the same handle; use_reset_code() validates handle match,
expiry (24 h), and single-use; change_password() updates the hash.
server.py: POST /api/admin/users/{handle}/reset-password-code (admin)
returns a code; POST /api/auth/reset-password (public) validates the
code + handle and sets the new password.
Admin page: "Reset pwd" button per user — shows the code inline on
click (monospace, click-to-copy).
/reset-password/ page: handle + code + new password form.
Login page: "Forgot password?" link.
The old DELETE /api/admin/users/{handle}/activities only removed *.json
files and _merged/, leaving originals/ (Strava FIT files) and edits/
untouched — causing the 968 MB disk usage after a delete.
_wipe_user_activities() now removes activities/, edits/, originals/,
_merged/, index.json, athlete.json, and .bincio_cache.json. Admin page
button renamed to "Reset data" with updated confirmation text.
Davide Scaini