Four related issues made uploading 271+ activities unreliable:
1. merge_all/write_combined_feed were inside the extraction try/except —
any merge race returned 422 even though the file was on disk, causing
the mobile app to permanently mark the upload as failed. Fixed by
moving them to a separate best-effort try/except after the extraction
block. Switch to merge_one (single-activity symlink) instead of
merge_all (full rebuild) so each upload is O(1) FS ops, not O(N).
2. The dev watcher fired merge_all for every activity .json write AND the
upload endpoint also ran merge_all — O(N²) symlink operations during
bulk uploads. Watcher now skips activities/*.json changes (upload
endpoint handles those directly).
3. Vite/Chokidar followed the public/data symlink and opened a handle per
activity file; constant merge rebuilds exhausted file descriptors and
crashed the Astro dev server. Fixed with watch.ignored on public/data.
4. _write_year_shards and write_combined_feed used f.unlink() without
missing_ok=True — concurrent callers racing the same file threw
FileNotFoundError which propagated as a false extraction failure.
When the dev file-watcher and an upload endpoint both trigger merge_all on the
same user directory at the same time, the shutil.rmtree/_merged/activities/
sequence collides: one thread deletes the directory while the other is mid-walk,
causing ENOTEMPTY or ENOENT crashes.
Added a per-user-dir threading.Lock (_merge_lock) at the top of merge.py so
all callers (upload endpoints, watcher, admin rebuild) are serialized without
changing any call site. Also made the rmtree ignore_errors=True and the
symlink_to conditional to survive any stale leftover from a prior crash.
Instead of the browser resolving 20+ user shards recursively (~27 MB),
generate a pre-sorted feed.json at merge time with 50 activities per
page. The global feed loads one ~30 KB file on first paint; "Load more"
fetches subsequent pages (feed-2.json, feed-3.json, etc.).
Per-user profile pages still use year-sharded loadIndexPaged as before.
merge_all/_merged/index.json is now a shard manifest; activities are
split into index-{year}.json files. The feed loads only the most-recent
year on first paint (~200 activities instead of all of them). Older
years are fetched lazily when the user clicks "Load older activities".
Also strips best_efforts / best_climb_m / source from shard files —
these fields are aggregation inputs only, never read by the feed UI.
- "unlisted" = not shown in the public feed, but GPS track, timeseries
and detail JSON are all accessible by direct URL (security by obscurity)
- "private" accepted as legacy alias everywhere (backward compat with
existing data on disk)
- New writes from Strava sync / ZIP upload / sidecar use "unlisted"
- Only "no_gps" now suppresses the GPS track
- isUnlisted() helper in format.ts used by all Svelte/Astro components
- SCHEMA.md and CLAUDE.md document the privacy model and the distinction
between "unlisted" and "no_gps"
stripping them; privacy filtering is now done client-side
- ActivityFeed: detect logged-in user via bincio:me event; show private
activities only when viewing your own profile; private cards get a lock
badge
- writer.py: timeseries is now written to {id}.timeseries.json as a separate file. The detail JSON gets a timeseries_url field instead. finalize_pending and cleanup_pending handle the extra file.
- merge.py (merge_one): symlinks the .timeseries.json file alongside the detail JSON. merge_all already handles it transparently (the .timeseries.json stem doesn't match any activity
ID in to_merge, so it falls through to the symlink branch).
- types.ts: timeseries is now timeseries?: Timeseries | null, and timeseries_url?: string | null added.
- dataloader.ts: new loadTimeseries(url, detailUrl, base) function that resolves paths correctly in both single- and multi-user modes (uses the fetched detail URL's directory as the base).
- ActivityDetail.svelte: loads timeseries separately after detail loads; uses detail.timeseries for IDB activities (embedded) or fetches via detail.timeseries_url for server activities. Charts show a pulse placeholder while loading.
2 — GZip
- GZipMiddleware (min 1 KB) added to both bincio/serve/server.py and bincio/edit/server.py — all API JSON responses are now gzip-compressed.
- For static files (the big timeseries JSONs), nginx should be configured with gzip on; gzip_types application/json application/geo+json; — no code change needed on the server side.
Net effect: opening an activity page now fetches ~1.4 KB (detail) instead of ~586 KB. The timeseries fetches ~60–150 KB gzip-compressed shortly after (it loads concurrently with the map rendering).
- Add bincio/extract/ingest.py as a facade over the extract internals (ingest_parsed, strava_sync), reducing coupling from 6+ imports to one
- Add merge_one() to merge.py — fast single-activity path for interactive edits (rewrites one file + index, skips full directory rebuild)
- Rewrite edit/ops.py to delegate to the new facade; fix broken run_strava_sync return (was referencing undefined locals)
- Remove duplicated SPORTS, STAT_PANELS, VALID_ACTIVITY_ID from edit/server.py — now imported from ops.py
- bincio/render/merge.py: parse sidecar .md files (YAML frontmatter +
markdown body), produce data/_merged/ with symlinks for unmodified
activities and real merged files for overridden ones; filters private
activities from index.json; sorts highlighted activities first.
Keeps extracted data pristine — re-running extract never clobbers edits.
- bincio/edit/: FastAPI edit server (port 4041) with embedded HTML/JS
edit UI; GET/POST /api/activity/{id} reads/writes sidecars; multipart
image upload to edits/images/{id}/; DELETE for image cleanup.
- bincio render now calls merge_all() before build/serve and symlinks
public/data → data/_merged/ instead of data/ directly.
- ActivityDetail.svelte: edit button (links to edit server) when
PUBLIC_EDIT_URL env var is set; respects custom.hide_stats to suppress
stat panels; description supports whitespace-preserving rendering.
- 15 unit tests covering parse_sidecar, apply_sidecar, and merge_all.
Davide Scaini