diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ff36fca..8bb4095 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,7 +14,7 @@ jobs: steps: - uses: actions/checkout@v6 - uses: astral-sh/setup-uv@v8.0.0 - - run: uv sync + - run: uv sync --extra serve --extra edit --extra strava - run: uv run pytest frontend: diff --git a/.gitignore b/.gitignore index d98680e..f157861 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,7 @@ build/ htmlcov/ .coverage .idea* +feedback* # uv uv.lock @@ -20,6 +21,9 @@ site/node_modules/ site/dist/ site/.astro/ +# MkDocs +mkdocs-site/ + # BAS data stores (user data, not committed to the tool repo) bincio_data/ *.bincio_cache.json @@ -28,6 +32,19 @@ bincio_data/ .env extract_config.yaml +# Local working / scratch files +advice.md +issues.md +todo.md +site/public/data +site/public/*.whl +.claude/settings.local.json +dns.md +ngix_bincio.md +publish.sh +docs/squash-for-github.md +CLAUDE.md + # Capacitor native projects # Commit these if you want to track native code changes; # omit these lines if you regenerate them from `npx cap add` diff --git a/CHANGELOG.md b/CHANGELOG.md index 2b6ffc0..6e2b646 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,342 @@ # Changelog -## [Unreleased] — 2026-04-06 +## [0.1.0] — 2026-04-22 + +### Improvement — DEM & hysteresis algorithm refinements + +**Hysteresis-only recalculation** (`recalculate_elevation_hysteresis`) reworked: + +- Pre-smooths the elevation series with a **30 s centred moving average** (O(n) + cumsum implementation) before accumulation. Pre-smoothing suppresses barometric + quantization steps and GPS jitter without discarding real terrain. +- Hysteresis thresholds reduced to **1 m (barometric)** / **3 m (GPS/unknown)** + — safe after pre-smoothing, and accurate enough to capture genuine small climbs + that the previous 5 m / 10 m thresholds were swallowing. +- Response key renamed `source` → `altitude_source` for consistency with the + detail JSON field. + +**DEM recalculation** median-filter window widened from 45 s → **60 s** to more +reliably absorb the occasional larger SRTM tile-boundary step. + +`altitude_source` is now written into the activity detail JSON at extract time +(`writer.py`), making the hysteresis endpoint source-aware for all newly uploaded +activities. + +### Tests + +- **`tests/test_dem.py`** (new) — 21 tests covering `_moving_average`, + `_median_filter`, `_hysteresis_gain_loss`, and `recalculate_elevation_hysteresis` + at the file level (no network, no extract pipeline) +- **`tests/test_edit_server.py`** (new) — 11 `TestClient` API tests for both + `/recalculate-elevation/hysteresis` and `/recalculate-elevation/dem` endpoints, + covering happy path, error codes (404/422/503), path-traversal rejection, and + on-disk JSON patching +- `httpx` added as a dev dependency (required by FastAPI `TestClient`) + +--- + +## [0.1.0-dev] — 2026-04-20 + +### Improvement — Elevation gain accuracy (hysteresis accumulation) + +The previous algorithm accumulated every positive elevation delta between +consecutive track points, counting GPS jitter and barometric quantization +noise as real climbing. This consistently overestimated gain — in extreme +cases by 100% on flat coastal routes. + +The new algorithm uses **hysteresis dead-band accumulation**: elevation is +only committed when it changes by more than a source-specific threshold from +the last committed value. GPS noise is suppressed without losing real climbs. + +- **`bincio/extract/models.py`** — `ParsedActivity` gains an `altitude_source` + field (`"barometric"` / `"gps"` / `"unknown"`) +- **`bincio/extract/parsers/fit.py`** — detects whether any record frame used + `enhanced_altitude` (barometric altimeter) vs `altitude` (GPS-derived) and + sets `altitude_source` accordingly +- **`bincio/extract/parsers/gpx.py`**, **`tcx.py`** — both set + `altitude_source = "gps"` +- **`bincio/extract/metrics.py`** — `_elevation()` replaced with hysteresis + accumulator; thresholds: **5 m** for barometric, **10 m** for GPS/unknown +- **`tests/test_metrics.py`** — 5 new parametric tests: flat GPS noise + suppression, barometric vs GPS threshold difference, real climb approximation, + unknown-treated-as-gps invariant + +### New feature — On-demand elevation recalculation from the edit drawer + +Two new buttons in the activity edit drawer fix inaccurate elevation stats +without re-uploading the file: + +**📐 Recalculate (hysteresis)** — re-applies source-aware hysteresis +accumulation to the original recorded elevation. Fast, offline, no network +required. Best for barometric altimeters (Karoo 2, Garmin with +`enhanced_altitude`, Wahoo) that were extracted before the noise-filtering +improvement. + +**⛰ Recalculate (DEM)** — replaces GPS altitude with SRTM terrain data, then +re-applies hysteresis. Best for GPS-only devices where the recorded altitude +is noisy. + +DEM pipeline (revised after discovering that a naive 5 m threshold produced +results worse than no correction on some activities): +1. Subsample GPS track to one point per 10 s +2. Query Open-Elevation API in batches of 512 +3. Linearly interpolate back to the full 1 Hz series +4. Apply a **45 s sliding median filter** to suppress SRTM tile-boundary + steps (occur every ~7 s at cycling speed; were accumulating through 5 m + threshold and inflating gain by 50 %+) +5. Apply **10 m hysteresis** to the smoothed series +6. Back up original `elevation_m` as `elevation_m_original` in the timeseries + on the first DEM run (never overwrites an existing backup) + +- **`bincio/extract/dem.py`** (new) — `lookup_elevations()`, + `recalculate_elevation()` (DEM + median + 10 m hysteresis), + `recalculate_elevation_hysteresis()` (offline, reads `elevation_m_original` + if available, uses 5 m/10 m source-aware threshold) +- **`POST /api/activity/{id}/recalculate-elevation/dem`** and + **`POST /api/activity/{id}/recalculate-elevation/hysteresis`** — on both + `bincio serve` (auth-gated, triggers `merge_one` + rebuild) and + `bincio edit` (no auth) +- **`bincio serve --dem-url URL`** / **`bincio edit --dem-url URL`** — override + the default DEM endpoint (also read from `DEM_URL` env var) +- Default DEM endpoint: **`https://api.open-elevation.com`** — works out of + the box with no configuration +- **`GET /api/me`** response gains `dem_configured: bool` +- **`EditDrawer.svelte`** — two side-by-side buttons with individual spinners, + shows `↑ Xm ↓ Ym` on success or inline error + +--- + +## [0.1.0-dev] — 2026-04-16 + +### New feature — Self-service user settings page + +- **`site/src/pages/settings/index.astro`** — new `/settings/` page with three sections: + - **Account** — display name editor, storage quota view (uploaded activities + originals size) + - **Integrations** — per-user Strava client ID/secret (replaces instance-level credentials for + multi-user deployments); saved to `settings` table via `PATCH /api/me` + - **Danger zone** — two separate destructive actions: + - **Delete originals** — removes `{user_dir}/originals/` without touching activities + - **Delete all activities** — wipes all activities, edits, GeoJSON, and `_merged/`; triggers rebuild + - Nav visibility toggles — user can hide any combination of Feed / Stats / Athlete tabs from + their navigation; preference saved to `settings` table and applied in `Base.astro` + +### New feature — Upload overwrite option + +- **`POST /api/upload`** — new `overwrite: bool` form field; when true, an existing activity + with the same ID is replaced rather than returning 409. UI checkbox added to the upload modal. + +### New feature — Admin tools + +- **Ghost user detection** — `/admin/` now marks users whose handle has a data directory but + no entry in the `users` table (e.g. manually created dirs, or users deleted from DB); shown + with a "ghost" badge +- **Delete directory button** — admin can delete a user's entire data directory without + touching the DB entry; useful for cleaning up ghost dirs or corrupted accounts +- **Delete all activities** (`DELETE /api/admin/users/{handle}/activities`) — wipes + `activities/`, `edits/`, `_merged/`, and `index.json` for a handle, then triggers a rebuild; + admin page shows a confirmation `` before firing +- **"Admin" nav link** — visible in the top-right for admins only + +### New feature — Password reset (admin-generated one-time code) + +No email infrastructure required. Flow: + +1. Admin visits `/admin/` → clicks "Reset pwd" → a 24-hour code appears inline (click to copy) +2. Admin sends it out-of-band (Signal, Telegram, etc.) +3. User goes to `/reset-password/`, enters handle + code + new password + +- `POST /api/admin/users/{handle}/reset-password-code` (admin) → `{code, expires_in_hours: 24}` +- `POST /api/auth/reset-password` (public) → body `{handle, code, password}` +- `reset_codes` table in `instance.db`; generating a new code invalidates prior unused codes; + used codes kept for audit + +### New feature — Re-extract from Strava originals + +- **`POST /api/admin/reextract`** — re-runs the extract pipeline over all + `{user_dir}/originals/strava/*.json` files without hitting the Strava API again; + streams progress via SSE; useful after pipeline improvements +- Runs as a subprocess to avoid OOM (`malloc_trim` + `gc.collect` every 50 activities); + processes in batches of 100 to bound peak RSS + +### New feature — Community page + +- **`/community/` tab** — sortable table of all registered users: display name, handle, + member since, invited by; replaces the earlier inline community section on the about page + +### New feature — Streaming upload progress + +- **`POST /api/upload`** now returns `text/event-stream` instead of JSON +- Per-file progress events: `↓ 3/47 (6%) — morning_ride.fit` +- Final `done` event: `"12 added, 35 duplicates"` +- Vite proxy configured to not buffer the stream + +### Bug fixes + +- **`elevation_gain_m` null for modern Garmin FIT files** — session message `total_ascent` + field now read as fallback when per-point elevation gain is zero +- **Map flash on activity detail** — map container height set before `fitBounds` to prevent + a zero-height frame during load +- **Absolute `track_url` / `detail_url` paths** — `ActivityDetail` and `loadActivity` now + handle both relative and absolute paths in BAS JSON +- **Corrupted time streams causing OOM** — `metrics.py` guards against non-monotonic or + pathologically large time arrays before allocating the 1 Hz dense array +- **Merge race condition** — `merge_all` wipe + rewrite is now guarded; concurrent upload + triggers can no longer interleave a `shutil.rmtree` with a write from another request +- **Temp ZIP leak** — upload temp files now written to `/tmp/` and always deleted in a + `finally` block; a startup hook auto-cleans any leftovers +- **`bincio init` always overwrites `private`** — fixed to preserve existing value when + `index.json` already exists +- **Auth wall flash** — `Base.astro` now sets the auth state synchronously from a cookie + hint before the `fetch('/api/me')` resolves, eliminating the visible flash +- **Single-user redirect loop** — `index.astro` no longer redirects to `/u/{handle}/` on + private (multi-user) instances +- **Theme-aware Plot tooltips** — forced black text on white background; was rendering + grey-on-white (unreadable in light mode) and white-on-dark (unreadable in dark mode) +- **Theme-aware chart axis colors** — axis labels and tick marks now use the correct + foreground color in both light and dark themes +- **TS type annotation in `define:vars` script** — removed; Astro injects `define:vars` + blocks as plain JS, not TypeScript +- **Image refs with spaces/parens in filenames** — local image references in markdown + descriptions are now stripped before rendering to avoid broken inline `` tags + +--- + +## [0.1.0-dev] — 2026-04-10 + +### New feature — Per-instance user limit + +Operators can now cap the maximum number of registered users on an instance. + +- **`bincio/serve/db.py`** + - New `settings` table (key/value, upsert-safe via `ON CONFLICT DO UPDATE`). + - `count_users(db)` — returns total number of rows in `users`. + - `get_setting(db, key)` / `set_setting(db, key, value)` — generic persistent settings store. + +- **`bincio/serve/server.py`** — `POST /api/register` now reads the `max_users` setting; if + set to N > 0 and the current user count is already ≥ N, registration is rejected with + HTTP 403 and a clear message. Imports `count_users` and `get_setting`. + +- **`bincio/serve/init_cmd.py`** — new `--max-users N` flag (default 0 = unlimited). Saves + the value to the `settings` table via `set_setting`. Printed in the init summary. + +- **`bincio/serve/cli.py`** — new `--max-users N` flag on `bincio serve`. Writes to the DB + on startup (lets operators change the limit without re-running `bincio init`). Startup + banner now shows `Users: max N` or `Users: unlimited`. + +--- + +### New feature — Original file storage option (upload & Strava sync) + +Users can now choose whether to keep their source files on the server after processing. +Keeping originals allows reprocessing if the pipeline improves; discarding them is the +privacy-conscious choice. Previously, uploaded files were always deleted after processing. + +- **`bincio/serve/db.py`** — `store_originals` is stored as a settings key. `bincio init` + writes `store_originals=true` on first run. + +- **`bincio/serve/server.py`** — `POST /api/upload` accepts a new `store_original: bool` + form field. On success, if true, the staged file is moved to `{user_dir}/originals/` + instead of being deleted. `GET /api/me` now includes `store_originals_default: bool` + (read from the instance setting) so the frontend can pre-populate the checkbox. + `POST /api/strava/sync` checks the `store_originals` instance setting; if true, creates + `{user_dir}/originals/strava/` and passes it as `originals_dir` to `run_strava_sync`. + +- **`bincio/edit/server.py`** — `POST /api/upload` gains the same `store_original` form + field with identical behaviour (originals stored in `{data_dir}/originals/`). + +- **`bincio/edit/ops.py`** — `run_strava_sync` gains an `originals_dir: Optional[Path]` + parameter, passed through to `ingest.strava_sync`. + +- **`bincio/extract/ingest.py`** — `strava_sync` gains `originals_dir: Optional[Path]`. + When set, saves `{"meta": …, "streams": …}` as JSON to + `originals_dir/{activity_id}.json` before processing each activity. This preserves the + raw Strava API response for future reprocessing without needing another API call. + +- **`bincio/serve/init_cmd.py`** — sets `store_originals=true` in the settings table on + first init (skipped if the key already exists, so re-running init doesn't override + an operator's choice). + +- **`site/src/layouts/Base.astro`** — upload modal file view gains a "Keep original file on + server" checkbox. Defaults to unchecked; pre-checked after login if the instance setting + is `true` (read from `store_originals_default` in the `/api/me` response). The checkbox + value is sent as the `store_original` form field. + +- **`bincio/serve/server.py`** and **`bincio/edit/server.py`** — `Form` added to the + FastAPI imports (was missing, causing a startup `NameError`). + +--- + +### New feature — About page (multilingual) + +New static `/about/` page explaining the project, with a Ko-fi donation button, data +storage disclaimer, and early-software caveats. Available in four languages. + +- **`site/src/pages/about/index.astro`** — English +- **`site/src/pages/about/it/index.astro`** — Italian +- **`site/src/pages/about/es/index.astro`** — Spanish +- **`site/src/pages/about/ca/index.astro`** — Catalan + +All four pages share the same structure: +- Language switcher (EN / IT / ES / CA) in the top-right corner. +- Ko-fi donation button (`https://ko-fi.com/brutsalvadi`) at the top. +- **Community stats section** — fetches `GET /api/stats` on load; shown only in + multi-user mode (silently hidden in single-user mode where the endpoint doesn't exist). + Displays total member count and an indented invitation tree: each row shows display name, + `@handle`, membership duration (days / months), and either "founder" or "invited by @X". + UI labels are fully translated per language. +- Sections: What is this · Your data on this server · Early-stage software · Disclaimer · + Open source. +- All pages use `public={true}` so they bypass the instance auth wall. + +"About" link added to the main nav bar (visible when not on a public page). +The upload modal's "Keep original file" checkbox links to `/about/` for context. + +--- + +### New feature — Community stats API + +- **`bincio/serve/db.py`** — `get_member_tree(db)` joins `users` with `invites` (on + `used_by`) to reconstruct the invitation graph. Returns a list ordered oldest-first with + `handle`, `display_name`, `created_at`, and `invited_by` (inviter handle or `None` for + the founder/admin). + +- **`bincio/serve/server.py`** — new public `GET /api/stats` endpoint (no auth required). + Returns `user_count` and a `members` array where each entry includes `handle`, + `display_name`, `member_since` (Unix timestamp), `member_for_days`, and `invited_by`. + +--- + +### Fix — `bincio dev` now watches data directory for live re-merge + +Previously, editing a sidecar or running `bincio extract` while `bincio dev` was running +required a manual restart to pick up changes. Now a background watcher thread re-merges +automatically. + +- **`bincio/dev.py`** — new `_watch_data(data)` function, started as a daemon thread + alongside `bincio serve`. Uses `watchfiles` (already bundled with `uvicorn[standard]`, + no new dependency) for OS-level file event watching — no polling. + - Watches every `{user_dir}/edits/` and `{user_dir}/activities/` directory. + - On any change, identifies which users were affected and calls `merge_all(user_dir)` + for each. + - Skips churn files written by merge itself (`.timeseries.json`, `.geojson`, + `index.json`) to avoid re-triggering. + - Prints `↺ {handle}: merged` on each successful re-merge; warns on failure. + - Astro dev picks up the result automatically since `public/data` is a symlink into + the live data directory. + +--- + +### Tests + +- **`tests/test_server_imports.py`** (new) — smoke tests that import `bincio.serve.server` + and `bincio.edit.server` at module level, catching `NameError`, missing imports, and + syntax errors before they reach the runtime. Also asserts that key routes (`/api/me`, + `/api/upload`, `/api/strava/sync`, `/api/register`, `/api/activity/{activity_id}`) are + registered on each app. + +--- + +## [0.1.0-dev] — 2026-04-06 ### New feature — Strava sync from UI @@ -37,7 +373,7 @@ bincio edit --strava-client-id YOUR_ID --strava-client-secret YOUR_SECRET `site/.env`, regardless of whether the edit server is running. This is intentional — the env var is the "edit mode enabled" flag. Remove it from `.env` to hide the button. -## [Unreleased] — 2026-04-01 +## [0.1.0-dev] — 2026-04-01 ### Security fixes (second-pass audit) diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index feb1c2b..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,265 +0,0 @@ -# BincioActivity — Context for Claude - -## What this project is - -BincioActivity is a federated, open-source, self-hosted activity stats platform -(think personal Strava). Two-stage pipeline: - -1. **`bincio extract`** (Python): GPX/FIT/TCX → BAS JSON data store -2. **`bincio render`** (Astro/Node): BAS data store → static website - -The BAS (BincioActivity Schema) JSON files are the federation protocol. -Anyone can publish their data as BAS JSON and others can include it. - -## Key design decisions - -- **Unified data layout** — single-user and multi-user share the same structure: activities always live in `{data-root}/{handle}/`. The only difference is the presence of `instance.db` (auth). No mode switching, no migration. -- **No database, no server** — everything is static files; multi-user VPS mode adds SQLite auth only -- **Python with uv** for the extract stage -- **Astro + Svelte + Tailwind + MapLibre GL + Observable Plot** for the site -- **Haversine** (not geopy) for distance calculations (10x faster) -- **Worker initializer pattern** for ProcessPoolExecutor — large shared data - (strava_lookup dict, known_hashes frozenset) is sent once per worker via - `initializer=`, not once per task -- **BAS activity IDs** always use UTC with Z suffix for URL safety -- **TCX files** from Garmin use both `http://` and `https://` namespace URIs — - parser handles both -- **Shard manifest for multi-user** — no activity data duplication; root `index.json` - lists user shard URLs; browser resolves all shards concurrently; same mechanism - handles yearly pagination and remote federation -- **Iterative RDP** implemented inline in `simplify.py` — no `rdp` PyPI package - (not available as a pure-Python wheel for Pyodide) - -## Your data - -- Source: `~/your-activity-data/` - - `activities/` — Strava export (GPX, FIT, TCX, all with .gz variants) - - Any subdirectories with FIT files from Garmin/Karoo devices - - `activities.csv` — Strava metadata (names, descriptions, gear) -- Extracted output: `~/bincio_data/` (or `/tmp/bincio_test/` for testing) - -Configure input paths in `extract_config.yaml`. - -## Project structure - -``` -bincio/ Python package - extract/ - models.py DataPoint, ParsedActivity, LapData - parsers/ GPX, FIT, TCX parsers + factory - sport.py sport name normalisation - metrics.py haversine-based stats computation (single pass) - timeseries.py downsample to 1Hz, build BAS timeseries object - simplify.py RDP track simplification → GeoJSON (iterative, no rdp dep) - dedup.py exact (hash) + near-duplicate detection - strava_csv.py Strava activities.csv importer - writer.py BAS JSON + GeoJSON writer - config.py extract_config.yaml loader - cli.py `bincio extract` CLI - render/ - cli.py `bincio render` CLI (symlinks data, runs astro build/dev) - merge.py sidecar edit overlay (produces _merged/) - edit/ - cli.py `bincio edit` CLI (single-user local only) - server.py FastAPI write API for the edit drawer - serve/ - cli.py `bincio serve` CLI (multi-user VPS) - server.py FastAPI: auth, user mgmt, write API (auth-gated) - db.py SQLite data layer (users, sessions, invites) - init_cmd.py `bincio init` CLI: bootstrap instance.db + admin user -schema/ - bas-v1.schema.json JSON Schema for BAS -SCHEMA.md Human-readable BAS spec -site/ Astro project - src/ - layouts/Base.astro Reads instancePrivate from index.json; injects auth wall - pages/ - index.astro Activity feed (loads index.json client-side) - activity/[id].astro Single activity (SSG, loads detail JSON client-side) - activity/local/ IDB-only activities (converted locally via Pyodide) - stats/index.astro Heatmap + year totals - u/[handle].astro Per-user profile pages (multi-user) - login/index.astro Login form (public page) - register/index.astro Registration with invite code (public page) - invites/index.astro Invite management - convert/index.astro Local file conversion via Pyodide (browser-only) - components/ - ActivityFeed.svelte Card grid, sport filter, pagination - ActivityDetail.svelte Map + stats + charts + photo gallery - ActivityMap.svelte MapLibre GL (gradient track, linked hover dot) - ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence tabs) - StatsView.svelte Yearly heatmap + totals - EditDrawer.svelte Slide-in edit panel (visible when PUBLIC_EDIT_URL set) - LocalActivityDetail.svelte Detail view for IDB-only (locally converted) activities - lib/ - types.ts BAS TypeScript types - format.ts formatDistance, formatDuration, sportIcon, etc. - localstore.ts IndexedDB store for locally converted activities - dataloader.ts Fetches index.json, resolves shards recursively -``` - -## How to run - -```bash -# Single-user (no login) -cd ~/src/bincio_activity -uv run bincio extract --output /tmp/bincio_test # → /tmp/bincio_test/{handle}/ -uv run bincio dev --data-dir /tmp/bincio_test -# → http://localhost:4321/u/{handle}/ - -# Multi-user (with login) -uv run bincio init --data-dir /tmp/bincio_test --handle dave -uv run bincio extract --output /tmp/bincio_test # → /tmp/bincio_test/dave/ -uv run bincio dev --data-dir /tmp/bincio_test -# → http://localhost:4321 (login required) - -# bincio dev does everything: merges sidecars, writes manifest, -# symlinks public/data, starts bincio serve (if instance.db exists), -# starts astro dev. Ctrl+C stops all. - -# Tests -uv run pytest -``` - -## MapLibre GL + Vite/Astro — known gotchas - -Learnt the hard way during debugging (March 2026): - -- **`maplibregl.workerUrl = ...` is the v3 API and silently no-ops in v4+.** - The v5 API is `maplibregl.setWorkerUrl(url)`, but you don't need it at all in a - normal Vite environment — MapLibre handles the blob worker automatically. - -- **`optimizeDeps: { exclude: ['maplibre-gl'] }` breaks tile loading.** - It prevents Vite from converting MapLibre's UMD bundle to ESM. The UMD bundle - uses AMD `define()` internally; served raw, the tile worker blob fails silently → - black map, no tiles. The correct setting is `include: ['maplibre-gl']`. - -- **`build.target: 'es2022'` (and `optimizeDeps.esbuildOptions.target`) is required.** - MapLibre's dependencies use ES2022 class field syntax. If esbuild downgrades it, - helpers like `__publicField` aren't available inside the serialised worker blob - scope → tile loading fails. This is a known upstream issue (maplibre-gl-js #6680). - -- **Use static imports, not dynamic `await import('maplibre-gl')`, when possible.** - With `client:only="svelte"` in Astro, SSR never runs for the component so there is - no `window is not defined` risk. Static import lets Vite pre-bundle correctly. - -- **Use `client:only="svelte"` (not `client:load`) for the activity detail page.** - `client:load` does SSR + hydration; complex interactive components with MapLibre - can hit hydration mismatch issues. `client:only` mounts fresh on the client only. - -- **MapLibre v5 requires explicit `center` and `zoom` in the Map constructor.** - v4 silently defaulted to `center: [0,0], zoom: 0`. v5 leaves internal projection - state undefined → `Cannot read properties of undefined (reading 'lng')` crashes - on any operation that touches coordinates (markers, resize, render). Always pass - `center` and `zoom` even if you plan to `fitBounds` later. - -- **MapLibre v5 requires `setLngLat()` on markers before `.addTo(map)`.** - v4 tolerated markers without coordinates. v5 calls `Marker._update()` inside - `addTo()`, which needs valid lngLat → same `'lng'` crash. Set a dummy `[0, 0]` - if the real position arrives later (e.g. hover markers). - -## Observable Plot — known gotchas - -- **Curve names are hyphenated, not camelCase.** - Use `"monotone-x"`, not `"monotoneX"`. Plot uses its own curve name registry - (not raw d3 identifiers). Wrong names throw `unknown curve` at runtime. - -The working `astro.config.mjs` Vite section: -```js -vite: { - optimizeDeps: { - include: ['maplibre-gl'], - esbuildOptions: { target: 'es2022' }, - }, - build: { target: 'es2022' }, -}, -``` - -## Activity sidecar edits — design spec - -Users edit activities via **sidecar markdown files** in the data dir. -No database, no server — consistent with the project's static-files-only philosophy. - -### File naming - -``` -~/bincio_data/ - activities/{id}.json ← immutable extract output - edits/{id}.md ← user edits (sidecar) - edits/images/{id}/ ← uploaded photos - _merged/ ← render-time merge output (gitignored-style) -``` - -### Sidecar format - -```markdown ---- -title: "Epic climb up Monte Grappa" -sport: cycling -hide_stats: [cadence] -highlight: true -private: false -gear: "Trek Domane" ---- - -Rode with friends. Legs felt great after the rest week... -``` - -### Editing UX: drawer in Astro + `bincio edit` write API - -- `bincio edit --data-dir ~/bincio_data` starts a FastAPI server on port 4041 -- Set `PUBLIC_EDIT_URL=http://localhost:4041` in `site/.env` to enable the edit button -- Clicking Edit on any activity detail page opens a slide-in drawer -- Saving writes the sidecar and triggers `merge_all()` automatically -- `bincio render` always runs `merge_all()` before build/serve and symlinks `public/data` → `_merged/` - -### `PUBLIC_EDIT_URL` as feature flag - -- **Unset** → no Edit button, normal static site -- **Set** → edit drawer enabled; lives in `site/.env` (gitignored) - -## Multi-user VPS architecture - -`bincio serve` is a FastAPI app that owns auth and write ops. nginx proxies `/api/*` to it; static files are served by nginx directly. The Vite dev server replicates this proxy for local testing. - -Key facts: -- Session cookie: `bincio_session`, httpOnly, SameSite=Lax, 30-day max-age -- Rate limiting: 10 login attempts / 15 min / IP (in-memory, resets on restart) -- Invite limits: admins unlimited, regular users 3 each (`_MAX_USER_INVITES` in `db.py`) -- Instance privacy: `instance.private=true` in root `index.json` → `Base.astro` injects a - `fetch('/api/me')` auth wall; `/login/` and `/register/` have `public={true}` to skip it -- Incremental rebuild: `POST /api/activity/{id}` triggers `bincio render --handle {user}` - as a fire-and-forget subprocess (only if `--site-dir` was passed to `bincio serve`) -- Write API in `bincio serve` delegates to `bincio.edit.server._apply_sidecar_edit`; the - Strava sync delegates to `bincio.edit.server.strava_sync` with a temporary data_dir swap - -## Known issues / next steps - -- `bincio render --watch` mode not yet implemented -- Activity IDs in older test data may use `+0000` format (pre-fix); re-run extract to get `Z` format -- Some activities appear with both untitled and titled IDs (near-dedup timing race) -- Remote federation (remote shard URLs in root manifest) is parsed but not yet displayed with attribution in the UI -- The `site/.env` file is gitignored — copy from `site/.env.example` - -## What "good" looks like (not yet done) - -- [ ] Friends/federation pages in site (remote shard attribution) -- [ ] Personal records page -- [ ] Activity search / full-text filter in feed -- [ ] GitHub Actions template for auto-publish -- [ ] Karoo/Garmin Connect importers beyond Strava -- [ ] `bincio render --watch` incremental rebuild on sidecar/data changes -- [ ] Highlight badge in activity feed cards -- [x] `bincio.render.merge` — sidecar parser, `_merged/` output, private filter, highlight sort -- [x] `bincio edit` FastAPI write API (GET/POST activity, image upload/delete, triggers merge) -- [x] `EditDrawer.svelte` — slide-in edit UI in the Astro site -- [x] `PUBLIC_EDIT_URL` feature flag -- [x] Markdown rendering in activity description with image path rewriting -- [x] Photo gallery with lightbox on activity detail page -- [x] `bincio serve` — multi-user VPS server (auth, invites, write API) -- [x] `bincio init` — instance bootstrap (SQLite, admin user, root manifest) -- [x] Login, register, invites pages -- [x] Per-user profile pages (`/u/{handle}/`) -- [x] Instance privacy (auth wall, private-by-default) -- [x] Shard-based combined feed (no duplication, concurrent resolution) -- [x] Local file conversion via Pyodide (`/convert/` page, IDB storage) diff --git a/README.md b/README.md index 6160ab4..a2cb2f5 100644 --- a/README.md +++ b/README.md @@ -203,7 +203,7 @@ Privacy is enforced at extract time. A `private` activity never enters `index.js `index.json` is everything the feed page needs — no extra fetches until you open an activity. `{id}.json` contains the full timeseries (elevation, speed, HR, cadence, power at 1 Hz) for charts and the detail map. Both are human-readable and editable with any text editor. -See [SCHEMA.md](SCHEMA.md) for the full specification. +See [SCHEMA.md](docs/schema.md) for the full specification. --- @@ -293,7 +293,6 @@ bincio/ Python package server.py FastAPI write API (activity edits, image + file upload) schema/ bas-v1.schema.json JSON Schema for BAS format -SCHEMA.md Human-readable BAS specification site/ Astro project src/ pages/ diff --git a/bincio/cli.py b/bincio/cli.py index 3d73efc..017833a 100644 --- a/bincio/cli.py +++ b/bincio/cli.py @@ -18,6 +18,7 @@ from bincio.import_.cli import import_group # noqa: E402 from bincio.serve.init_cmd import init # noqa: E402 from bincio.serve.cli import serve # noqa: E402 from bincio.dev import dev # noqa: E402 +from bincio.reextract_cmd import reextract_originals # noqa: E402 main.add_command(extract) main.add_command(render) @@ -26,3 +27,4 @@ main.add_command(import_group) main.add_command(init) main.add_command(serve) main.add_command(dev) +main.add_command(reextract_originals) diff --git a/bincio/dev.py b/bincio/dev.py index 24b5292..b40d441 100644 --- a/bincio/dev.py +++ b/bincio/dev.py @@ -86,6 +86,62 @@ def _start_serve(data: Path, api_port: int, site: Path) -> None: server.run() +def _watch_data(data: Path) -> None: + """Watch the data directory for sidecar/activity changes and re-merge. + + Monitors every user's edits/ and activities/ subdirectories. When any file + changes (new activity extracted, sidecar saved), re-runs merge_all for that + user so the _merged/ symlink tree stays current. Astro dev picks up the + result automatically because public/data is a symlink into the live data dir. + + Uses watchfiles (bundled with uvicorn[standard]) for efficient OS-level + file watching — no polling. + """ + from watchfiles import watch, Change + + watch_paths = [] + for user_dir in _user_dirs(data): + for sub in ("edits", "activities"): + p = user_dir / sub + p.mkdir(exist_ok=True) + watch_paths.append(p) + + if not watch_paths: + return + + console.print(f" [dim]Watching {len(watch_paths)} director{'y' if len(watch_paths) == 1 else 'ies'} for changes…[/dim]") + + # Build a map from path prefix → user dir for targeted merge + prefix_to_user: dict[str, Path] = {} + for user_dir in _user_dirs(data): + for sub in ("edits", "activities"): + prefix_to_user[str(user_dir / sub)] = user_dir + + for changes in watch(*watch_paths, yield_on_timeout=False): + # Find which users were affected + affected: set[Path] = set() + for change_type, path in changes: + # Skip timeseries / geojson / index churn written by merge itself + if any(path.endswith(s) for s in (".timeseries.json", ".geojson", "index.json")): + continue + for prefix, user_dir in prefix_to_user.items(): + if path.startswith(prefix): + affected.add(user_dir) + break + + if not affected: + continue + + for user_dir in affected: + handle = user_dir.name + try: + from bincio.render.merge import merge_all + merge_all(user_dir) + console.print(f" [dim]↺ {handle}: merged[/dim]") + except Exception as exc: + console.print(f" [yellow]⚠ {handle}: merge failed — {exc}[/yellow]") + + @click.command("dev") @click.option("--data-dir", default=None, help="BAS data directory (must contain instance.db)") @click.option("--site-dir", default=None, help="Astro project directory (default: ./site)") @@ -144,6 +200,10 @@ def dev( t = threading.Thread(target=_start_serve, args=(data, api_port, site), daemon=True) t.start() + # Watch data dir for sidecar/activity changes → auto-merge + watcher = threading.Thread(target=_watch_data, args=(data,), daemon=True) + watcher.start() + # Build env for astro dev env = { **os.environ, diff --git a/bincio/edit/cli.py b/bincio/edit/cli.py index afe0d62..f28e615 100644 --- a/bincio/edit/cli.py +++ b/bincio/edit/cli.py @@ -24,6 +24,8 @@ console = Console() help="Strava API client ID (enables Strava sync in the UI). Also reads STRAVA_CLIENT_ID env var.") @click.option("--strava-client-secret", default=None, envvar="STRAVA_CLIENT_SECRET", help="Strava API client secret. Also reads STRAVA_CLIENT_SECRET env var.") +@click.option("--dem-url", default=None, envvar="DEM_URL", + help="Base URL of an Open-Elevation-compatible API (default: https://api.open-elevation.com).") def edit( data_dir: Optional[str], port: int, @@ -31,6 +33,7 @@ def edit( config_path: Optional[str], strava_client_id: Optional[str], strava_client_secret: Optional[str], + dem_url: Optional[str], ) -> None: """Start a local web UI for editing activity sidecar files. @@ -69,11 +72,13 @@ def edit( srv.site_url = site_url srv.strava_client_id = strava_client_id or "" srv.strava_client_secret = strava_client_secret or "" + srv.dem_url = dem_url or "" if strava_client_id: console.print(f"Strava sync: [green]enabled[/green] (client {strava_client_id})") else: console.print("Strava sync: [yellow]disabled[/yellow] (pass --strava-client-id to enable)") + console.print(f"DEM: [cyan]{srv.dem_url}[/cyan]") uvicorn.run(srv.app, host="127.0.0.1", port=port, log_level="warning") diff --git a/bincio/edit/ops.py b/bincio/edit/ops.py index fddddb5..50a1f16 100644 --- a/bincio/edit/ops.py +++ b/bincio/edit/ops.py @@ -9,7 +9,7 @@ from __future__ import annotations import json import re from pathlib import Path -from typing import Any +from typing import Any, Optional # ── Shared constants (imported by edit/server.py and serve/server.py) ───────── @@ -58,13 +58,20 @@ def apply_sidecar_edit(activity_id: str, payload: dict[str, Any], data_dir: Path merge_one(data_dir, activity_id) -def run_strava_sync(data_dir: Path, client_id: str, client_secret: str) -> dict[str, Any]: +def run_strava_sync( + data_dir: Path, + client_id: str, + client_secret: str, + originals_dir: Optional[Path] = None, +) -> dict[str, Any]: """Fetch new Strava activities and write them into data_dir. Args: data_dir: Per-user data directory. client_id: Strava OAuth client ID. client_secret: Strava OAuth client secret. + originals_dir: If set, raw Strava API data (meta + streams) is saved here + as JSON files for potential future reprocessing. Returns: Dict with keys: ok, imported, skipped, error_count, errors. @@ -75,7 +82,7 @@ def run_strava_sync(data_dir: Path, client_id: str, client_secret: str) -> dict[ from bincio.extract.ingest import strava_sync as _strava_sync from bincio.render.merge import merge_all - result = _strava_sync(data_dir, client_id, client_secret) + result = _strava_sync(data_dir, client_id, client_secret, originals_dir=originals_dir) if result["imported"]: merge_all(data_dir) diff --git a/bincio/edit/server.py b/bincio/edit/server.py index 7e7b516..2f25567 100644 --- a/bincio/edit/server.py +++ b/bincio/edit/server.py @@ -3,14 +3,15 @@ from __future__ import annotations import json +import secrets import shutil from pathlib import Path from typing import Any -from fastapi import FastAPI, File, HTTPException, Request, UploadFile +from fastapi import FastAPI, File, Form, HTTPException, Request, UploadFile from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.gzip import GZipMiddleware -from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse +from fastapi.responses import HTMLResponse, JSONResponse, RedirectResponse, StreamingResponse from bincio.edit.ops import SPORTS, STAT_PANELS, VALID_ACTIVITY_ID @@ -19,6 +20,10 @@ data_dir: Path | None = None site_url: str = "http://localhost:4321" strava_client_id: str = "" strava_client_secret: str = "" +dem_url: str = "https://api.open-elevation.com" # Open-Elevation-compatible API base URL + +# In-memory CSRF state tokens for OAuth flows (token → True); cleared after use +_oauth_states: set[str] = set() app = FastAPI(title="BincioActivity Edit Server", docs_url=None, redoc_url=None) @@ -38,6 +43,21 @@ def _check_id(activity_id: str) -> str: return activity_id +_ALLOWED_IMAGE_TYPES = {"image/jpeg", "image/png", "image/webp", "image/gif"} +_MAX_IMAGE_BYTES = 10 * 1024 * 1024 # 10 MB + + +def _unique_image_name(directory: Path, filename: str) -> str: + """Return a filename that does not collide with existing files in directory.""" + stem, suffix = Path(filename).stem, Path(filename).suffix + candidate = filename + counter = 1 + while (directory / candidate).exists(): + candidate = f"{stem}_{counter}{suffix}" + counter += 1 + return candidate + + # ── HTML UI ─────────────────────────────────────────────────────────────────── _HTML = """\ @@ -164,7 +184,7 @@ textarea { resize: vertical; min-height: 140px; } Highlight in feed @@ -388,7 +408,7 @@ async def get_activity(activity_id: str) -> JSONResponse: "gear": fm.get("gear", detail.get("gear") or ""), "description": body or fm.get("description") or detail.get("description") or "", "highlight": fm.get("highlight", detail.get("custom", {}).get("highlight", False)), - "private": fm.get("private", detail.get("privacy") == "private"), + "private": fm.get("private", detail.get("privacy") in ("private", "unlisted")), "hide_stats": fm.get("hide_stats", detail.get("custom", {}).get("hide_stats", [])), "images": images, }) @@ -408,6 +428,45 @@ async def save_activity(activity_id: str, payload: dict[str, Any]) -> JSONRespon return JSONResponse({"ok": True, "sidecar": str(sidecar_path)}) +@app.post("/api/activity/{activity_id}/recalculate-elevation/dem") +async def recalculate_elevation_dem_endpoint(activity_id: str) -> JSONResponse: + """Replace GPS altitude with DEM terrain elevation and recompute gain/loss. + + Requires --dem-url to be set when starting bincio edit. + """ + if not dem_url: + raise HTTPException(503, "DEM URL not configured.") + dd = _get_data_dir() + _check_id(activity_id) + try: + from bincio.extract.dem import recalculate_elevation + from bincio.render.merge import merge_one + result = recalculate_elevation(dd, activity_id, dem_url) + merge_one(dd, activity_id) + return JSONResponse(result) + except FileNotFoundError as e: + raise HTTPException(404, str(e)) + except ValueError as e: + raise HTTPException(422, str(e)) + + +@app.post("/api/activity/{activity_id}/recalculate-elevation/hysteresis") +async def recalculate_elevation_hysteresis_endpoint(activity_id: str) -> JSONResponse: + """Recompute gain/loss from original recorded elevation using source-aware hysteresis.""" + dd = _get_data_dir() + _check_id(activity_id) + try: + from bincio.extract.dem import recalculate_elevation_hysteresis + from bincio.render.merge import merge_one + result = recalculate_elevation_hysteresis(dd, activity_id) + merge_one(dd, activity_id) + return JSONResponse(result) + except FileNotFoundError as e: + raise HTTPException(404, str(e)) + except ValueError as e: + raise HTTPException(422, str(e)) + + @app.post("/api/activity/{activity_id}/images") async def upload_image(activity_id: str, file: UploadFile = File(...)) -> JSONResponse: dd = _get_data_dir() @@ -419,14 +478,15 @@ async def upload_image(activity_id: str, file: UploadFile = File(...)) -> JSONRe images_dir = dd / "edits" / "images" / activity_id images_dir.mkdir(parents=True, exist_ok=True) - safe_name = Path(file.filename).name - # Only allow image content types ct = file.content_type or "" - if not ct.startswith("image/"): - raise HTTPException(400, f"Only image files are accepted (got {ct})") - dest = images_dir / safe_name - dest.write_bytes(await file.read()) - return JSONResponse({"ok": True, "filename": dest.name}) + if ct not in _ALLOWED_IMAGE_TYPES: + raise HTTPException(400, "Only JPEG, PNG, WebP, or GIF images are accepted") + contents = await file.read() + if len(contents) > _MAX_IMAGE_BYTES: + raise HTTPException(413, f"Image too large (max {_MAX_IMAGE_BYTES // (1024 * 1024)} MB)") + safe_name = _unique_image_name(images_dir, Path(file.filename).name) + (images_dir / safe_name).write_bytes(contents) + return JSONResponse({"ok": True, "filename": safe_name}) @app.get("/api/athlete") @@ -517,63 +577,115 @@ def _file_suffix(name: str) -> str: @app.post("/api/upload") -async def upload_activity(file: UploadFile = File(...)) -> JSONResponse: - """Accept a FIT/GPX/TCX file, extract it, update index.json, and re-merge.""" +async def upload_activity( + files: list[UploadFile] = File(...), + store_original: bool = Form(False), +) -> StreamingResponse: + """Accept FIT/GPX/TCX files and/or activities.csv; stream SSE progress while processing. + + activities.csv (Strava export format) can be included in the batch to: + - Enrich activity files in the same batch (matched by filename) + - Retroactively update sidecars for existing activities (matched by strava_id) + """ + from bincio.extract.ingest import ingest_parsed + from bincio.extract.parsers.factory import parse_file + from bincio.extract.writer import make_activity_id + from bincio.render.merge import merge_all + dd = _get_data_dir() - - name = Path(file.filename or "upload.fit").name # strip any path components - suffix = _file_suffix(name) - if suffix not in _SUPPORTED_SUFFIXES: - raise HTTPException(400, f"Unsupported file type '{Path(name).suffix}'. Expected FIT, GPX, or TCX.") - - _MAX_UPLOAD_BYTES = 50 * 1024 * 1024 # 50 MB - contents = await file.read() - if len(contents) > _MAX_UPLOAD_BYTES: - raise HTTPException(413, f"File too large ({len(contents)} bytes). Maximum is 50 MB.") - staging = dd / "_uploads" staging.mkdir(exist_ok=True) - staged = staging / name - staged.write_bytes(contents) - try: - from bincio.extract.metrics import compute - from bincio.extract.parsers.factory import parse_file - from bincio.extract.writer import build_summary, make_activity_id, write_activity, write_index + # Read all files into memory now (async), then process synchronously in the generator + csv_bytes_list: list[bytes] = [] + activity_items: list[tuple[str, bytes]] = [] - activity = parse_file(staged) - metrics = compute(activity) - activity_id = make_activity_id(activity) - - existing_json = dd / "activities" / f"{activity_id}.json" - if existing_json.exists(): - raise HTTPException(409, f"Activity already exists: {activity_id}") - - write_activity(activity, metrics, dd, privacy="public", rdp_epsilon=0.0001) - summary = build_summary(activity, metrics, activity_id, "public") - - # Read current index to preserve owner + existing summaries - index_path = dd / "index.json" - if index_path.exists(): - index_data = json.loads(index_path.read_text(encoding="utf-8")) + for f in files: + fname = Path(f.filename or "").name + raw = await f.read() + if fname.lower().endswith(".csv"): + csv_bytes_list.append(raw) else: - index_data = {"owner": {"handle": "unknown"}, "activities": []} - owner = index_data.get("owner", {}) - existing = {s["id"]: s for s in index_data.get("activities", [])} - existing[activity_id] = summary - write_index(list(existing.values()), dd, owner) + activity_items.append((fname, raw)) - from bincio.render.merge import merge_all - merge_all(dd) + # Build metadata from the first CSV found (activities.csv from Strava export) + metadata = None + if csv_bytes_list: + from bincio.extract.strava_csv import StravaMetadata + import tempfile + with tempfile.NamedTemporaryFile(suffix=".csv", delete=False) as tmp: + tmp.write(csv_bytes_list[0]) + tmp_path = Path(tmp.name) + try: + metadata = StravaMetadata(tmp_path) + finally: + tmp_path.unlink(missing_ok=True) - except HTTPException: - raise - except Exception as exc: - raise HTTPException(422, f"Failed to process activity file: {type(exc).__name__}") - finally: - staged.unlink(missing_ok=True) + total_files = len(activity_items) - return JSONResponse({"ok": True, "id": activity_id}) + def event_stream(): + added = 0 + duplicates = 0 + errors = 0 + any_added = False + + for n, (name, contents) in enumerate(activity_items, 1): + suffix = _file_suffix(name) + if suffix not in _SUPPORTED_SUFFIXES: + errors += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'error', 'detail': 'unsupported type'})}\n\n" + continue + + if len(contents) > _MAX_UPLOAD_BYTES: + errors += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'error', 'detail': 'file too large'})}\n\n" + continue + + staged = staging / name + staged.write_bytes(contents) + kept = False + try: + activity = parse_file(staged) + if metadata is not None: + metadata.enrich(name, activity) + activity_id = make_activity_id(activity) + if (dd / "activities" / f"{activity_id}.json").exists(): + duplicates += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'duplicate'})}\n\n" + continue + ingest_parsed(activity, dd, privacy="public") + if store_original: + originals_dir = dd / "originals" + originals_dir.mkdir(exist_ok=True) + staged.rename(originals_dir / name) + kept = True + added += 1 + any_added = True + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'imported'})}\n\n" + except Exception: + errors += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'error'})}\n\n" + finally: + if not kept: + staged.unlink(missing_ok=True) + + csv_updates = 0 + if metadata is not None: + from bincio.extract.strava_csv import apply_csv_to_data_dir + csv_updates = apply_csv_to_data_dir(dd, metadata) + if csv_updates: + yield f"data: {json.dumps({'type': 'csv', 'updates': csv_updates})}\n\n" + + if any_added or csv_updates: + merge_all(dd) + + yield f"data: {json.dumps({'type': 'done', 'added': added, 'csv_updates': csv_updates, 'duplicates': duplicates, 'errors': errors})}\n\n" + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) @app.post("/api/import-bas") @@ -667,16 +779,21 @@ async def strava_auth_url(request: Request) -> JSONResponse: """Return the Strava OAuth URL the browser should open.""" if not strava_client_id: raise HTTPException(400, "Strava client ID not configured. Pass --strava-client-id to bincio edit.") + state = secrets.token_urlsafe(16) + _oauth_states.add(state) redirect_uri = str(request.url_for("strava_callback")) from bincio.extract.strava_api import auth_url - return JSONResponse({"url": auth_url(strava_client_id, redirect_uri)}) + return JSONResponse({"url": auth_url(strava_client_id, redirect_uri, state=state)}) @app.get("/api/strava/callback", name="strava_callback") -async def strava_callback(code: str = "", error: str = "") -> RedirectResponse: +async def strava_callback(code: str = "", error: str = "", state: str = "") -> RedirectResponse: """Strava OAuth callback — exchange code for token then redirect to the site.""" if error or not code: return RedirectResponse(f"{site_url}?strava=error") + if state not in _oauth_states: + return RedirectResponse(f"{site_url}?strava=error") + _oauth_states.discard(state) if not strava_client_id or not strava_client_secret: return RedirectResponse(f"{site_url}?strava=error") dd = _get_data_dir() @@ -747,3 +864,51 @@ async def strava_reset(request: Request) -> JSONResponse: token["last_sync_at"] = last_ts save_token(dd, token) return JSONResponse({"ok": True, "mode": "soft", "last_sync_at": last_ts}) + + +@app.post("/api/upload/strava-zip") +async def upload_strava_zip( + file: UploadFile = File(...), + private: str = Form(default="false"), +) -> StreamingResponse: + """Accept a Strava bulk export ZIP and stream SSE progress while processing. + + The ZIP is written to a temp file, processed activity-by-activity, then deleted. + Originals are never kept — the UI informs the user of this upfront. + """ + if not file.filename or not file.filename.lower().endswith(".zip"): + raise HTTPException(400, "Please upload a .zip file") + + privacy = "private" if private.lower() in ("true", "1", "yes") else "public" + + dd = _get_data_dir() + import tempfile + tmp = tempfile.NamedTemporaryFile(suffix=".zip", delete=False, dir=dd) + zip_path = Path(tmp.name) + try: + while chunk := await file.read(1024 * 1024): # 1 MB chunks + tmp.write(chunk) + finally: + tmp.close() + + from bincio.extract.strava_zip import strava_zip_iter + from bincio.render.merge import merge_all + + def event_stream(): + any_imported = False + try: + for event in strava_zip_iter(zip_path, dd, privacy=privacy): + yield f"data: {json.dumps(event)}\n\n" + if event.get("type") == "progress" and event.get("status") == "imported": + any_imported = True + if event.get("type") == "done" and any_imported: + merge_all(dd) + except Exception as exc: + zip_path.unlink(missing_ok=True) + yield f"data: {json.dumps({'type': 'error', 'message': str(exc)})}\n\n" + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) diff --git a/bincio/extract/cli.py b/bincio/extract/cli.py index b95ec37..6ddf179 100644 --- a/bincio/extract/cli.py +++ b/bincio/extract/cli.py @@ -171,7 +171,7 @@ def extract( dedup = DedupIndex(output_dir=cfg.output_dir) known_hashes: frozenset = frozenset(dedup._by_hash.keys()) - n_workers = workers or os.cpu_count() or 4 + n_workers = workers or cfg.workers or os.cpu_count() or 4 console.print(f"Using [bold]{n_workers}[/bold] worker processes.") owner = {"handle": cfg.owner_handle, "display_name": cfg.owner_display_name} diff --git a/bincio/extract/config.py b/bincio/extract/config.py index 1c8c6a1..baa72d6 100644 --- a/bincio/extract/config.py +++ b/bincio/extract/config.py @@ -51,6 +51,7 @@ class ExtractConfig: track: TrackConfig = field(default_factory=TrackConfig) classifier: ClassifierConfig = field(default_factory=ClassifierConfig) incremental: bool = True + workers: Optional[int] = None # None → use CPU count owner_handle: str = "me" owner_display_name: str = "Me" athlete: AthleteConfig | None = None @@ -109,6 +110,7 @@ def load_config(path: Path) -> ExtractConfig: track=track, classifier=classifier, incremental=raw.get("incremental", True), + workers=raw.get("workers"), owner_handle=owner.get("handle", "me"), owner_display_name=owner.get("display_name", "Me"), athlete=athlete, diff --git a/bincio/extract/dem.py b/bincio/extract/dem.py new file mode 100644 index 0000000..3f89e61 --- /dev/null +++ b/bincio/extract/dem.py @@ -0,0 +1,382 @@ +"""DEM (Digital Elevation Model) lookup and elevation recalculation. + +Queries any Open-Elevation-compatible HTTP API to replace noisy GPS altitude +with terrain altitude, then re-applies hysteresis-based accumulation. + +Compatible APIs: + - https://api.open-elevation.com (free, SRTM, accepts large batches) + - https://api.opentopodata.org/v1/srtm30m (more reliable, max 100 pts/req) + +Pass the base URL (without path) to bincio serve/edit via --dem-url. +""" + +from __future__ import annotations + +import json +import statistics +import urllib.error +import urllib.request +from pathlib import Path +from typing import Optional + +# Sample one GPS point per N seconds when building the DEM query. +# SRTM30 resolution is ~30 m; at 30 km/h cycling that's ~3 s per tile — +# sampling every 10 s is more than enough. +_DEFAULT_SAMPLE_INTERVAL_S = 10 + +# Maximum locations per API request. Open-Elevation supports ~512 per POST. +_DEFAULT_BATCH_SIZE = 512 + +# Hysteresis threshold after DEM correction. +# SRTM30 at 1 Hz produces tile-boundary steps of ~1–3 m every few seconds. +# A 5 m threshold barely filters them; 10 m suppresses them reliably. +_DEM_HYSTERESIS_M = 10.0 + +# Median filter window (seconds / samples at 1 Hz) applied to DEM-interpolated +# series before hysteresis. 45 s smooths SRTM tile steps while keeping real +# climbs (typical cycling ramp > 100 m over > 2 min). +_MEDIAN_WINDOW_S = 60 + +# Moving-average window (seconds) applied to the 1 Hz elevation series before +# hysteresis in the on-demand recalculation. Pre-smoothing lets us use a +# much lower dead-band (capturing real small climbs) while still suppressing +# GPS jitter and barometric quantization noise. +_MA_WINDOW_S = 30 + + +def _moving_average(values: list[float], window: int) -> list[float]: + """Apply a centred sliding-window moving average to *values*. + + Edge handling: window shrinks symmetrically at both ends (same effective + behaviour as scipy's 'nearest' / numpy's 'reflect' mode). + """ + half = window // 2 + n = len(values) + out: list[float] = [] + cumsum = [0.0] * (n + 1) + for i, v in enumerate(values): + cumsum[i + 1] = cumsum[i] + v + for i in range(n): + lo = max(0, i - half) + hi = min(n, i + half + 1) + out.append((cumsum[hi] - cumsum[lo]) / (hi - lo)) + return out + + +def _median_filter(values: list[float], window: int) -> list[float]: + """Apply a sliding-window median filter to *values*. + + The window is centred on each sample; edges are handled by shrinking the + window symmetrically (same as scipy's 'reflect' / 'nearest' default). + """ + half = window // 2 + n = len(values) + out: list[float] = [] + for i in range(n): + lo = max(0, i - half) + hi = min(n, i + half + 1) + out.append(statistics.median(values[lo:hi])) + return out + + +def lookup_elevations( + latlons: list[tuple[float, float]], + api_url: str, + batch_size: int = _DEFAULT_BATCH_SIZE, + timeout_s: int = 30, +) -> list[Optional[float]]: + """Query a DEM API for terrain elevation at the given (lat, lon) pairs. + + Uses the Open-Elevation API format:: + + POST {api_url}/api/v1/lookup + {"locations": [{"latitude": lat, "longitude": lon}, ...]} + + Returns a list the same length as *latlons*. Elements are ``None`` + wherever the API returned no data (network error, ocean, etc.). + """ + if not latlons: + return [] + + results: list[Optional[float]] = [None] * len(latlons) + url = f"{api_url.rstrip('/')}/api/v1/lookup" + + for start in range(0, len(latlons), batch_size): + batch = latlons[start : start + batch_size] + payload = json.dumps( + {"locations": [{"latitude": lat, "longitude": lon} for lat, lon in batch]} + ).encode("utf-8") + req = urllib.request.Request( + url, + data=payload, + headers={"Content-Type": "application/json", "Accept": "application/json"}, + method="POST", + ) + try: + with urllib.request.urlopen(req, timeout=timeout_s) as resp: + data = json.loads(resp.read().decode("utf-8")) + for i, item in enumerate(data.get("results", [])): + elev = item.get("elevation") + if elev is not None: + results[start + i] = float(elev) + except (urllib.error.URLError, json.JSONDecodeError, KeyError, ValueError): + pass # leave this batch as None; caller checks overall coverage + + return results + + +def _hysteresis_gain_loss( + elevations: list[float], threshold_m: float +) -> tuple[float, float]: + """Compute elevation gain and loss using a hysteresis dead-band. + + Only commits a new elevation level when it differs from the last committed + value by at least *threshold_m*. Returns (gain, loss) in metres, both + as positive numbers. + """ + gain = loss = 0.0 + committed = elevations[0] + for e in elevations[1:]: + diff = e - committed + if abs(diff) >= threshold_m: + if diff > 0: + gain += diff + else: + loss += abs(diff) + committed = e + return gain, loss + + +def recalculate_elevation( + user_dir: Path, + activity_id: str, + dem_url: str, + sample_interval_s: int = _DEFAULT_SAMPLE_INTERVAL_S, +) -> dict: + """Replace GPS elevation with DEM terrain elevation and recompute gain/loss. + + Algorithm + --------- + 1. Read the activity's 1 Hz timeseries for lat / lon / t arrays. + 2. Subsample GPS points every *sample_interval_s* seconds. + 3. Query the DEM API for those points (batched). + 4. Linearly interpolate DEM elevation back to every GPS-valid second. + 5. Apply a 45 s median filter to smooth SRTM tile-boundary noise. + 6. Apply :data:`_DEM_HYSTERESIS_M` (10 m) hysteresis to compute gain/loss. + 7. Preserve the original elevation as ``elevation_m_original`` in the + timeseries (only on the first DEM run — never overwrites a prior backup). + 8. Write the smoothed DEM elevation array as ``elevation_m``. + 9. Patch ``elevation_gain_m`` / ``elevation_loss_m`` in the detail JSON. + 10. Patch ``elevation_gain_m`` in ``index.json`` (summary entry for feed). + + Returns + ------- + dict with keys ``elevation_gain_m``, ``elevation_loss_m``, + ``points_queried`` (DEM responses received). + + Raises + ------ + FileNotFoundError + Activity detail or timeseries file not found. + ValueError + Activity has no GPS coordinates, or the DEM API returned too few results. + """ + acts_dir = user_dir / "activities" + json_path = acts_dir / f"{activity_id}.json" + ts_path = acts_dir / f"{activity_id}.timeseries.json" + + if not json_path.exists(): + raise FileNotFoundError(f"Activity not found: {activity_id}") + if not ts_path.exists(): + raise ValueError("Activity has no timeseries data") + + ts = json.loads(ts_path.read_text(encoding="utf-8")) + lat_arr: list[Optional[float]] = ts.get("lat") or [] + lon_arr: list[Optional[float]] = ts.get("lon") or [] + t_arr: list[int] = ts.get("t") or [] + + if not lat_arr or not lon_arr: + raise ValueError( + "Activity has no GPS coordinates " + "(privacy=no_gps or data recorded without GPS)" + ) + + n = len(t_arr) + + # ── 1. Subsample GPS-valid indices ──────────────────────────────────────── + gps_idx = [ + i for i in range(n) + if lat_arr[i] is not None and lon_arr[i] is not None + ] + if len(gps_idx) < 2: + raise ValueError("Too few GPS points for DEM lookup") + + sampled_idx = gps_idx[::sample_interval_s] + if gps_idx[-1] not in sampled_idx: + sampled_idx.append(gps_idx[-1]) # always include the last point + + # ── 2. Query DEM ────────────────────────────────────────────────────────── + latlons = [(float(lat_arr[i]), float(lon_arr[i])) for i in sampled_idx] # type: ignore[arg-type] + dem_elev = lookup_elevations(latlons, dem_url) + + # Build (t, elevation) pairs for valid DEM responses only + valid_pairs: list[tuple[int, float]] = [ + (t_arr[sampled_idx[k]], dem_elev[k]) + for k in range(len(sampled_idx)) + if dem_elev[k] is not None + ] + n_queried = len(valid_pairs) + if n_queried < 2: + raise ValueError( + f"DEM API returned too few results " + f"({n_queried} of {len(sampled_idx)} points). " + f"Check the DEM URL: {dem_url}" + ) + + # ── 3. Linear interpolation → full 1 Hz series ─────────────────────────── + new_ele: list[Optional[float]] = [None] * n + j = 0 + for i in gps_idx: + t = t_arr[i] + while j + 1 < len(valid_pairs) - 1 and valid_pairs[j + 1][0] <= t: + j += 1 + t0, e0 = valid_pairs[j] + if j + 1 < len(valid_pairs): + t1, e1 = valid_pairs[j + 1] + frac = max(0.0, min(1.0, (t - t0) / (t1 - t0))) if t1 > t0 else 0.0 + new_ele[i] = round(e0 + frac * (e1 - e0), 1) + else: + new_ele[i] = round(e0, 1) + + # ── 4. Median filter to suppress SRTM tile-boundary steps ──────────────── + valid_indices = [i for i, e in enumerate(new_ele) if e is not None] + valid_eles_raw = [new_ele[i] for i in valid_indices] # type: ignore[misc] + smoothed = _median_filter(valid_eles_raw, _MEDIAN_WINDOW_S) # type: ignore[arg-type] + + # Write smoothed values back into new_ele + for idx, e in zip(valid_indices, smoothed): + new_ele[idx] = round(e, 1) + + # ── 5. Hysteresis accumulation on smoothed series ───────────────────────── + smoothed_valid = [e for e in new_ele if e is not None] + gain, loss = _hysteresis_gain_loss(smoothed_valid, _DEM_HYSTERESIS_M) # type: ignore[arg-type] + + gain_r = round(gain, 1) + loss_r = round(loss, 1) + + # ── 6. Preserve original elevation (only if not already backed up) ──────── + if "elevation_m_original" not in ts: + ts["elevation_m_original"] = ts.get("elevation_m") + + # ── 7. Write timeseries ─────────────────────────────────────────────────── + ts["elevation_m"] = new_ele + ts_path.write_text(json.dumps(ts, indent=2, ensure_ascii=False), encoding="utf-8") + + # ── 8. Patch activity detail JSON ───────────────────────────────────────── + detail = json.loads(json_path.read_text(encoding="utf-8")) + detail["elevation_gain_m"] = gain_r + detail["elevation_loss_m"] = loss_r + json_path.write_text(json.dumps(detail, indent=2, ensure_ascii=False), encoding="utf-8") + + # ── 9. Patch index.json summary ─────────────────────────────────────────── + index_path = user_dir / "index.json" + if index_path.exists(): + index = json.loads(index_path.read_text(encoding="utf-8")) + for s in index.get("activities", []): + if s.get("id") == activity_id: + s["elevation_gain_m"] = gain_r + break + index_path.write_text( + json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8" + ) + + return { + "elevation_gain_m": gain_r, + "elevation_loss_m": loss_r, + "points_queried": n_queried, + } + + +def recalculate_elevation_hysteresis(user_dir: Path, activity_id: str) -> dict: + """Recompute elevation gain/loss from the original recorded elevation data. + + Algorithm + --------- + 1. Read ``elevation_m_original`` (backup from a prior DEM run) if present, + otherwise read ``elevation_m`` from the timeseries. + 2. Apply a :data:`_MA_WINDOW_S` (30 s) moving average to smooth out + barometric quantization steps and GPS jitter. + 3. Apply a low dead-band threshold to the smoothed series: + - **1 m** for barometric altimeters (FIT files with ``enhanced_altitude``) + - **3 m** for GPS-derived altitude (GPX, TCX, FIT without enhanced_altitude) + + The 30 s pre-smoothing makes the low thresholds safe: after averaging, + 0.2 m barometric quantization noise and short-period GPS jitter are + suppressed below the threshold, while real terrain changes (which persist + across the window) are preserved. + + The elevation array in the timeseries is **not** modified — only the + summary stats in the detail JSON and ``index.json`` are patched. + + ``altitude_source`` is read from the detail JSON (written by the extractor + for activities recorded after this field was added). For older activities + it falls back to ``"unknown"`` → 3 m GPS threshold. + + Returns + ------- + dict with keys ``elevation_gain_m``, ``elevation_loss_m``, + ``threshold_m``, ``altitude_source``. + """ + acts_dir = user_dir / "activities" + json_path = acts_dir / f"{activity_id}.json" + ts_path = acts_dir / f"{activity_id}.timeseries.json" + + if not json_path.exists(): + raise FileNotFoundError(f"Activity not found: {activity_id}") + if not ts_path.exists(): + raise ValueError("Activity has no timeseries data") + + ts = json.loads(ts_path.read_text(encoding="utf-8")) + + # Prefer the pre-DEM backup; fall back to the current elevation array + ele_arr: list[Optional[float]] = ( + ts.get("elevation_m_original") or ts.get("elevation_m") or [] + ) + elevations = [e for e in ele_arr if e is not None] + if len(elevations) < 2: + raise ValueError("Not enough elevation data to compute gain/loss") + + # Determine source-aware threshold + detail = json.loads(json_path.read_text(encoding="utf-8")) + altitude_source = detail.get("altitude_source", "unknown") + threshold = 1.0 if altitude_source == "barometric" else 3.0 + + # Pre-smooth to suppress noise, then accumulate with low dead-band + smoothed = _moving_average(elevations, _MA_WINDOW_S) + gain, loss = _hysteresis_gain_loss(smoothed, threshold) + gain_r = round(gain, 1) + loss_r = round(loss, 1) + + # Patch detail JSON + detail["elevation_gain_m"] = gain_r + detail["elevation_loss_m"] = loss_r + json_path.write_text(json.dumps(detail, indent=2, ensure_ascii=False), encoding="utf-8") + + # Patch index.json summary + index_path = user_dir / "index.json" + if index_path.exists(): + index = json.loads(index_path.read_text(encoding="utf-8")) + for s in index.get("activities", []): + if s.get("id") == activity_id: + s["elevation_gain_m"] = gain_r + break + index_path.write_text( + json.dumps(index, indent=2, ensure_ascii=False), encoding="utf-8" + ) + + return { + "elevation_gain_m": gain_r, + "elevation_loss_m": loss_r, + "threshold_m": threshold, + "altitude_source": altitude_source, + } diff --git a/bincio/extract/garmin_api.py b/bincio/extract/garmin_api.py new file mode 100644 index 0000000..0087389 --- /dev/null +++ b/bincio/extract/garmin_api.py @@ -0,0 +1,225 @@ +"""Garmin Connect credential storage and client factory. + +Credential storage layout +───────────────────────── + {data_dir.parent}/.garmin_key ← Fernet key (outside nginx webroot, chmod 600) + {user_dir}/garmin_creds.json ← encrypted email + password + {user_dir}/garmin_session/ ← garth OAuth token directory (plain JSON, short-lived) + +Security model +────────────── +- The Fernet key lives one directory above the data root, which nginx does NOT serve. + For a standard VPS install: data_dir = /var/bincio/data/ → key at /var/bincio/.garmin_key. +- Credentials are encrypted with that key before being written to disk. +- The garth session directory holds OAuth tokens (not the user's password). + These expire independently and are refreshed automatically by the library. +- If the session expires and re-login is needed, the stored credentials are decrypted + and used automatically — the user does not need to re-enter them. + +DISCLAIMER +────────── +This module uses the unofficial `garminconnect` library. +See docs/garmin_connect_disclaimer.md before shipping this feature to users. +""" + +from __future__ import annotations + +import json +import os +import stat +from pathlib import Path +from typing import Optional + +# ── Constants ───────────────────────────────────────────────────────────────── + +_CREDS_FILE = "garmin_creds.json" +_SESSION_DIR = "garmin_session" +_KEY_FILENAME = ".garmin_key" + + +class GarminError(Exception): + pass + + +# ── Encryption key management ───────────────────────────────────────────────── + +def _key_path(data_dir: Path) -> Path: + """Return the path to the Fernet key file (one level above data_dir).""" + return data_dir.parent / _KEY_FILENAME + + +def _get_or_create_key(data_dir: Path) -> bytes: + """Load the Fernet key, creating and locking it down on first use.""" + from cryptography.fernet import Fernet + + kp = _key_path(data_dir) + if kp.exists(): + return kp.read_bytes().strip() + + key = Fernet.generate_key() + kp.parent.mkdir(parents=True, exist_ok=True) + kp.write_bytes(key) + kp.chmod(stat.S_IRUSR | stat.S_IWUSR) # 0o600 — owner read/write only + return key + + +def _fernet(data_dir: Path): + from cryptography.fernet import Fernet + return Fernet(_get_or_create_key(data_dir)) + + +# ── Credential encryption helpers ───────────────────────────────────────────── + +def _encrypt(data_dir: Path, value: str) -> str: + return _fernet(data_dir).encrypt(value.encode()).decode() + + +def _decrypt(data_dir: Path, token: str) -> str: + try: + return _fernet(data_dir).decrypt(token.encode()).decode() + except Exception as exc: + raise GarminError("Failed to decrypt Garmin credentials — key may have changed") from exc + + +# ── Credential CRUD ─────────────────────────────────────────────────────────── + +def has_credentials(user_dir: Path) -> bool: + return (user_dir / _CREDS_FILE).exists() + + +def save_credentials(data_dir: Path, user_dir: Path, email: str, password: str) -> None: + """Encrypt and persist the user's Garmin email + password.""" + payload = { + "email": _encrypt(data_dir, email), + "password": _encrypt(data_dir, password), + } + creds_path = user_dir / _CREDS_FILE + creds_path.write_text(json.dumps(payload, indent=2)) + creds_path.chmod(stat.S_IRUSR | stat.S_IWUSR) # 0o600 + + +def load_credentials(data_dir: Path, user_dir: Path) -> tuple[str, str]: + """Return (email, password) decrypted from disk.""" + creds_path = user_dir / _CREDS_FILE + if not creds_path.exists(): + raise GarminError("No Garmin credentials stored for this user") + try: + raw = json.loads(creds_path.read_text()) + except Exception as exc: + raise GarminError("Garmin credentials file is corrupt") from exc + return _decrypt(data_dir, raw["email"]), _decrypt(data_dir, raw["password"]) + + +def delete_credentials(user_dir: Path) -> None: + """Remove stored credentials and session (disconnect).""" + creds_path = user_dir / _CREDS_FILE + if creds_path.exists(): + creds_path.unlink() + + session_dir = user_dir / _SESSION_DIR + if session_dir.exists(): + import shutil + shutil.rmtree(session_dir) + + +# ── Session management (garth OAuth tokens) ─────────────────────────────────── + +def _session_dir(user_dir: Path) -> Path: + d = user_dir / _SESSION_DIR + d.mkdir(exist_ok=True) + return d + + +def _save_session(user_dir: Path, client) -> None: + """Persist garth OAuth tokens so the next sync skips re-login.""" + try: + client.garth.dump(str(_session_dir(user_dir))) + except Exception: + pass # session save is best-effort + + +def _load_session(user_dir: Path, client) -> bool: + """Try to restore a saved garth session. Returns True on success.""" + sd = user_dir / _SESSION_DIR + if not sd.exists(): + return False + try: + client.garth.load(str(sd)) + return True + except Exception: + return False + + +# ── Client factory ──────────────────────────────────────────────────────────── + +def get_client(data_dir: Path, user_dir: Path): + """Return a logged-in Garmin client. + + Strategy: + 1. Try to resume a saved garth session (fast, no network round-trip). + 2. If that fails or the session has expired, re-login using the stored + (encrypted) credentials. + 3. Persist the refreshed session for next time. + + Raises GarminError if credentials are missing or login fails. + """ + try: + import garminconnect + except ImportError as exc: + raise GarminError( + "garminconnect is not installed. " + "Run: uv sync --extra garmin" + ) from exc + + client = garminconnect.Garmin() + + # Try cached session first + if _load_session(user_dir, client): + try: + client.garth.refresh_oauth2() # renew access token if needed + _save_session(user_dir, client) # persist refreshed token + return client + except Exception: + pass # session is dead — fall through to full re-login + + # Full login with stored credentials + email, password = load_credentials(data_dir, user_dir) + try: + client = garminconnect.Garmin(email=email, password=password) + client.login() + except Exception as exc: + raise GarminError(f"Garmin login failed: {exc}") from exc + + _save_session(user_dir, client) + return client + + +def test_login(data_dir: Path, user_dir: Path, email: str, password: str) -> dict: + """Attempt a login with the supplied credentials (does not save them). + + Returns a dict with display_name and full_name on success. + Raises GarminError on failure. + """ + try: + import garminconnect + except ImportError as exc: + raise GarminError("garminconnect is not installed") from exc + + try: + client = garminconnect.Garmin(email=email, password=password) + client.login() + except Exception as exc: + raise GarminError(f"Login failed: {exc}") from exc + + try: + profile = client.get_profile_user_summary() + display = profile.get("displayName", email) + full = f"{profile.get('firstName', '')} {profile.get('lastName', '')}".strip() + except Exception: + display, full = email, "" + + # Credentials are valid — save them and the session + save_credentials(data_dir, user_dir, email, password) + _save_session(user_dir, client) + + return {"display_name": display, "full_name": full} diff --git a/bincio/extract/garmin_sync.py b/bincio/extract/garmin_sync.py new file mode 100644 index 0000000..395c886 --- /dev/null +++ b/bincio/extract/garmin_sync.py @@ -0,0 +1,196 @@ +"""Garmin Connect incremental sync — generator-based, mirrors strava_sync_iter. + +Sync state is stored in {user_dir}/garmin_sync.json: + { + "last_sync_at": "2026-04-12" ← date of last successful sync (YYYY-MM-DD) + } + +We query Garmin for all activities from (last_sync_at - 1 day) to today, +then skip any that already exist (FileExistsError from ingest_parsed). +The -1 day buffer catches activities that were saved to Garmin slightly +after their recorded end time crosses midnight. + +Each yielded dict has a ``type`` key: + - ``"fetching"`` — about to contact Garmin + - ``"progress"`` — one activity processed; keys: n, total, name, status, garmin_id + - ``"done"`` — final summary; keys: imported, skipped, error_count, errors + - ``"error"`` — fatal error; key: message +""" + +from __future__ import annotations + +import io +import json +import zipfile +from datetime import datetime, timedelta, timezone +from pathlib import Path +from typing import Generator + +_SYNC_FILE = "garmin_sync.json" + + +# ── Sync state helpers ──────────────────────────────────────────────────────── + +def _load_sync_state(user_dir: Path) -> dict: + p = user_dir / _SYNC_FILE + if not p.exists(): + return {} + try: + return json.loads(p.read_text()) + except Exception: + return {} + + +def _save_sync_state(user_dir: Path, state: dict) -> None: + (user_dir / _SYNC_FILE).write_text(json.dumps(state, indent=2)) + + +# ── FIT extraction from ZIP ─────────────────────────────────────────────────── + +def _extract_fit(zip_bytes: bytes) -> tuple[bytes, str]: + """Return (fit_bytes, filename) from a Garmin activity ZIP. + + Garmin always packages the original FIT as the first .fit entry. + Raises ValueError if no FIT file is found. + """ + with zipfile.ZipFile(io.BytesIO(zip_bytes)) as zf: + fit_names = [n for n in zf.namelist() if n.lower().endswith(".fit")] + if not fit_names: + raise ValueError(f"No FIT file in archive. Contents: {zf.namelist()}") + name = fit_names[0] + return zf.read(name), name + + +# ── Main generator ──────────────────────────────────────────────────────────── + +def garmin_sync_iter( + data_dir: Path, + user_dir: Path, +) -> Generator[dict, None, None]: + """Fetch new activities from Garmin Connect and ingest them. + + Args: + data_dir: Root data directory (used for encryption key lookup). + user_dir: Per-user directory (contains activities/, garmin_creds.json, etc.). + """ + from bincio.extract.garmin_api import GarminError, get_client + from bincio.extract.ingest import ingest_parsed + from bincio.extract.parsers.fit import FitParser + + # ── Login ────────────────────────────────────────────────────────────────── + try: + client = get_client(data_dir, user_dir) + except GarminError as exc: + yield {"type": "error", "message": str(exc)} + return + + yield {"type": "fetching"} + + # ── Determine date range ─────────────────────────────────────────────────── + state = _load_sync_state(user_dir) + last = state.get("last_sync_at") + + if last: + # Start one day before last sync to catch edge cases around midnight + start_dt = datetime.fromisoformat(last) - timedelta(days=1) + else: + # First sync: import everything Garmin has + start_dt = datetime(2000, 1, 1) + + start_date = start_dt.strftime("%Y-%m-%d") + end_date = datetime.now().strftime("%Y-%m-%d") + + # ── Fetch activity list ──────────────────────────────────────────────────── + try: + activities = client.get_activities_by_date( + startdate=start_date, + enddate=end_date, + ) + except Exception as exc: + yield {"type": "error", "message": f"Failed to fetch activity list: {exc}"} + return + + total = len(activities) + imported = 0 + skipped = 0 + errors: list[str] = [] + parser = FitParser() + + # ── Process each activity ────────────────────────────────────────────────── + for n, meta in enumerate(activities, 1): + garmin_id = meta.get("activityId") + name = meta.get("activityName") or "Untitled" + + try: + # Download original FIT (wrapped in a ZIP by Garmin) + try: + zip_bytes = client.download_activity( + garmin_id, + dl_fmt=client.ActivityDownloadFormat.ORIGINAL, + ) + except Exception as exc: + raise RuntimeError(f"Download failed: {exc}") from exc + + try: + fit_bytes, fit_name = _extract_fit(zip_bytes) + except Exception as exc: + raise RuntimeError(f"ZIP extraction failed: {exc}") from exc + + # Parse FIT — pass a dummy Path so the parser has a filename for + # any format-detection logic; raw bytes are the actual data. + fake_path = Path(fit_name) + try: + parsed = parser.parse(fake_path, fit_bytes) + except Exception as exc: + raise RuntimeError(f"FIT parse error: {exc}") from exc + + # Ingest — raises FileExistsError if already present (dedup) + ingest_parsed(parsed, user_dir) + imported += 1 + yield { + "type": "progress", + "n": n, "total": total, "name": name, + "status": "imported", + "garmin_id": garmin_id, + } + + except FileExistsError: + skipped += 1 + yield { + "type": "progress", + "n": n, "total": total, "name": name, + "status": "skipped", + "garmin_id": garmin_id, + } + + except Exception as exc: + errors.append(f"{garmin_id} ({name}): {type(exc).__name__}: {exc}") + yield { + "type": "progress", + "n": n, "total": total, "name": name, + "status": "error", + "garmin_id": garmin_id, + } + + # ── Persist sync state ───────────────────────────────────────────────────── + state["last_sync_at"] = datetime.now(timezone.utc).strftime("%Y-%m-%d") + _save_sync_state(user_dir, state) + + yield { + "type": "done", + "imported": imported, + "skipped": skipped, + "error_count": len(errors), + "errors": errors[:5], + } + + +def run_garmin_sync(data_dir: Path, user_dir: Path) -> dict: + """Blocking wrapper around garmin_sync_iter for non-SSE callers.""" + result: dict = {} + for event in garmin_sync_iter(data_dir, user_dir): + if event["type"] == "done": + result = event + elif event["type"] == "error": + raise RuntimeError(event["message"]) + return result diff --git a/bincio/extract/ingest.py b/bincio/extract/ingest.py index 84406eb..de44618 100644 --- a/bincio/extract/ingest.py +++ b/bincio/extract/ingest.py @@ -10,7 +10,6 @@ from __future__ import annotations import json from pathlib import Path from typing import Any, Optional - from bincio.extract.models import ParsedActivity @@ -47,8 +46,9 @@ def ingest_parsed( raise FileExistsError(f"Activity already exists: {activity_id}") metrics = compute(parsed) - write_activity(parsed, metrics, data_dir, privacy=privacy, rdp_epsilon=rdp_epsilon) - summary = build_summary(parsed, metrics, activity_id, privacy) + effective_privacy = parsed.privacy if parsed.privacy is not None else privacy + write_activity(parsed, metrics, data_dir, privacy=effective_privacy, rdp_epsilon=rdp_epsilon) + summary = build_summary(parsed, metrics, activity_id, effective_privacy) index_path = data_dir / "index.json" if index_path.exists(): @@ -60,26 +60,36 @@ def ingest_parsed( summaries[activity_id] = summary write_index(list(summaries.values()), data_dir, owner) + # Rebuild athlete.json with updated MMP curves and records. + # Preserve any manually-set fields (max_hr, ftp_w, zones, etc.) from the existing file. + from bincio.extract.writer import write_athlete_json + _COMPUTED = {"bas_version", "generated_at", "power_curve", "records", "best_climbs"} + athlete_config: dict[str, Any] = {} + athlete_path = data_dir / "athlete.json" + if athlete_path.exists(): + try: + existing = json.loads(athlete_path.read_text(encoding="utf-8")) + athlete_config = {k: v for k, v in existing.items() if k not in _COMPUTED} + except Exception: + pass + write_athlete_json(list(summaries.values()), data_dir, athlete_config) + return activity_id -def strava_sync( +def strava_sync_iter( data_dir: Path, client_id: str, client_secret: str, -) -> dict[str, Any]: - """Fetch new Strava activities and ingest them into data_dir. + originals_dir: Optional[Path] = None, +): + """Generator version of strava_sync — yields progress dicts, then a final summary. - Args: - data_dir: Per-user data directory. - client_id: Strava OAuth client ID. - client_secret: Strava OAuth client secret. - - Returns: - Dict with keys: ok, imported, skipped, error_count, errors. - - Raises: - RuntimeError: If Strava credentials are missing or API calls fail. + Each yielded dict has a ``type`` key: + - ``"fetching"`` — about to fetch the activity list from Strava + - ``"progress"`` — one activity processed; keys: n, total, name, status ("imported"|"skipped"|"error") + - ``"done"`` — final summary; keys: imported, skipped, error_count, errors + - ``"error"`` — fatal error before processing started; key: message """ import time @@ -95,43 +105,82 @@ def strava_sync( from bincio.extract.writer import make_activity_id if not client_id or not client_secret: - raise RuntimeError("Strava not configured (missing client_id or client_secret)") + yield {"type": "error", "message": "Strava not configured"} + return try: token = ensure_fresh(data_dir, client_id, client_secret) except StravaError as e: - raise RuntimeError(str(e)) from e + yield {"type": "error", "message": str(e)} + return + + yield {"type": "fetching"} after: Optional[int] = token.get("last_sync_at") try: activities = fetch_activities(token["access_token"], after=after) except StravaError as e: - raise RuntimeError(str(e)) from e + yield {"type": "error", "message": str(e)} + return + total = len(activities) imported = 0 skipped = 0 errors: list[str] = [] - for meta in activities: + for n, meta in enumerate(activities, 1): + name = meta.get("name", "Untitled") try: activity_id = make_activity_id(strava_meta_to_partial(meta)) if (data_dir / "activities" / f"{activity_id}.json").exists(): skipped += 1 + yield {"type": "progress", "n": n, "total": total, "name": name, "status": "skipped"} continue streams = fetch_streams(token["access_token"], meta["id"]) + if originals_dir is not None: + orig_path = originals_dir / f"{activity_id}.json" + orig_path.write_text( + json.dumps({"meta": meta, "streams": streams}, indent=2), + encoding="utf-8", + ) parsed = strava_to_parsed(meta, streams) ingest_parsed(parsed, data_dir, privacy="public", rdp_epsilon=0.0001) imported += 1 + yield {"type": "progress", "n": n, "total": total, "name": name, "status": "imported"} except Exception as exc: errors.append(f"{meta.get('id')}: {type(exc).__name__}") + yield {"type": "progress", "n": n, "total": total, "name": name, "status": "error"} token["last_sync_at"] = int(time.time()) save_token(data_dir, token) - return { - "ok": True, + yield { + "type": "done", "imported": imported, "skipped": skipped, "error_count": len(errors), "errors": errors[:5], } + + +def strava_sync( + data_dir: Path, + client_id: str, + client_secret: str, + originals_dir: Optional[Path] = None, +) -> dict[str, Any]: + """Fetch new Strava activities and ingest them into data_dir. + + Returns: + Dict with keys: ok, imported, skipped, error_count, errors. + + Raises: + RuntimeError: If Strava credentials are missing or API calls fail. + """ + result: dict[str, Any] = {} + for event in strava_sync_iter(data_dir, client_id, client_secret, originals_dir): + if event["type"] == "error": + raise RuntimeError(event["message"]) + if event["type"] == "done": + result = event + return {"ok": True, **{k: v for k, v in result.items() if k != "type"}} diff --git a/bincio/extract/metrics.py b/bincio/extract/metrics.py index 216db4e..cc0072a 100644 --- a/bincio/extract/metrics.py +++ b/bincio/extract/metrics.py @@ -70,7 +70,7 @@ def compute(activity: ParsedActivity) -> ComputedMetrics: duration_s = _duration(pts) distance_m, moving_time_s, avg_speed_kmh, max_speed_kmh = _gps_stats(pts) - gain, loss = _elevation(pts) + gain, loss = _elevation(pts, activity.altitude_source) avg_hr, max_hr = _hr_stats(pts) avg_cad = _avg_nonnull([p.cadence_rpm for p in pts]) avg_pow = _avg_nonnull([p.power_w for p in pts]) @@ -131,6 +131,10 @@ def compute_mmp(pts: list[DataPoint], started_at: datetime) -> Optional[list[lis t_min = min(sparse) t_max = max(sparse) + # Guard against corrupted time data (e.g. absolute Unix timestamps stored as + # elapsed offsets, which can make t_max astronomically large and OOM the process). + if t_max - t_min > 7 * 24 * 3600: # > 1 week → corrupted stream + return None power_1hz: list[int] = [sparse.get(t, 0) for t in range(t_min, t_max + 1)] n = len(power_1hz) @@ -190,6 +194,10 @@ def compute_best_efforts( t_min = min(sparse_speed) t_max = max(sparse_speed) + # Guard against corrupted time data (e.g. absolute Unix timestamps stored as + # elapsed offsets, which can make t_max astronomically large and OOM the process). + if t_max - t_min > 7 * 24 * 3600: # > 1 week → corrupted stream + return None, None speed_1hz: list[float] = [sparse_speed.get(t, 0.0) for t in range(t_min, t_max + 1)] ele_1hz: list[Optional[float]] = [sparse_ele.get(t) for t in range(t_min, t_max + 1)] @@ -339,17 +347,40 @@ def _duration(pts: list[DataPoint]) -> Optional[int]: return int((pts[-1].timestamp - pts[0].timestamp).total_seconds()) -def _elevation(pts: list[DataPoint]) -> tuple[Optional[float], Optional[float]]: +# Hysteresis thresholds per altitude source. +# Only commit a new elevation when it differs from the last committed value by +# at least this amount, filtering out GPS noise and barometric quantization steps. +_ELEVATION_THRESHOLD: dict[str, float] = { + "barometric": 5.0, # barometric altimeter: smaller steps are real + "gps": 10.0, # GPS altitude: noisier, needs wider dead-band + "unknown": 10.0, # treat unknown as GPS to be conservative +} + + +def _elevation( + pts: list[DataPoint], + altitude_source: str = "unknown", +) -> tuple[Optional[float], Optional[float]]: + """Hysteresis-based elevation accumulation. + + Only commits a new elevation when it differs from the last committed value + by at least the source-specific threshold, filtering GPS jitter and + barometric quantization noise that would otherwise inflate the gain figure. + """ elevations = [p.elevation_m for p in pts if p.elevation_m is not None] if len(elevations) < 2: return None, None + threshold = _ELEVATION_THRESHOLD.get(altitude_source, 10.0) gain = loss = 0.0 - for a, b in zip(elevations, elevations[1:]): - diff = b - a - if diff > 0: - gain += diff - else: - loss += diff + committed = elevations[0] + for e in elevations[1:]: + diff = e - committed + if abs(diff) >= threshold: + if diff > 0: + gain += diff + else: + loss += diff + committed = e return gain, loss diff --git a/bincio/extract/models.py b/bincio/extract/models.py index 253bb38..8ea3004 100644 --- a/bincio/extract/models.py +++ b/bincio/extract/models.py @@ -55,4 +55,9 @@ class ParsedActivity: description: Optional[str] = None gear: Optional[str] = None strava_id: Optional[str] = None + privacy: Optional[str] = None # "public", "private", or None (caller decides) laps: list[LapData] = field(default_factory=list) + # "barometric" = device has a barometric altimeter (FIT enhanced_altitude present) + # "gps" = altitude derived from GPS triangulation (GPX, TCX, FIT altitude-only) + # "unknown" = source not detected (treated as gps for threshold purposes) + altitude_source: str = "unknown" diff --git a/bincio/extract/parsers/base.py b/bincio/extract/parsers/base.py index c68f424..543b95e 100644 --- a/bincio/extract/parsers/base.py +++ b/bincio/extract/parsers/base.py @@ -27,8 +27,14 @@ class BaseParser(ABC): raw_bytes is the original file content (used for hashing). decompressed_bytes is what parsers should actually parse. + + Gzip is handled both by extension (.gz) and by magic bytes (0x1f 0x8b), + so files that are gzip-compressed but named without .gz still parse correctly. """ raw = path.read_bytes() - if path.suffix == ".gz": - return raw, gzip.decompress(raw) + if path.suffix == ".gz" or raw[:2] == b'\x1f\x8b': + try: + return raw, gzip.decompress(raw) + except Exception: + pass # not actually gzip despite the magic bytes — fall through return raw, raw diff --git a/bincio/extract/parsers/fit.py b/bincio/extract/parsers/fit.py index 6fa596a..c723d4a 100644 --- a/bincio/extract/parsers/fit.py +++ b/bincio/extract/parsers/fit.py @@ -20,6 +20,8 @@ class FitParser: sub_sport: str | None = None device: str | None = None + has_baro_alt = False # True if any record used enhanced_altitude + with fitdecode.FitReader(io.BytesIO(raw_bytes)) as fit: for frame in fit: if not isinstance(frame, fitdecode.FitDataMessage): @@ -57,12 +59,19 @@ class FitParser: lat = _semicircles_to_deg(_get(frame, "position_lat")) lon = _semicircles_to_deg(_get(frame, "position_long")) speed_raw = _get(frame, "speed") # m/s + # enhanced_altitude is written by barometric altimeters (most + # modern Garmins). Fall back to GPS-derived altitude if absent. + _alt = _get(frame, "enhanced_altitude") + if _alt is not None: + has_baro_alt = True + else: + _alt = _get(frame, "altitude") dp = DataPoint( timestamp=ts, lat=lat, lon=lon, - elevation_m=_get(frame, "altitude"), + elevation_m=_alt, hr_bpm=_get(frame, "heart_rate"), cadence_rpm=_get(frame, "cadence"), speed_kmh=speed_raw * 3.6 if speed_raw is not None else None, @@ -95,6 +104,8 @@ class FitParser: if not points: raise ValueError(f"No record messages found in {path.name}") + altitude_source = "barometric" if has_baro_alt else "gps" + return ParsedActivity( points=points, sport=sport, @@ -104,6 +115,7 @@ class FitParser: laps=laps, source_file=path.name, source_hash="", + altitude_source=altitude_source, ) diff --git a/bincio/extract/parsers/gpx.py b/bincio/extract/parsers/gpx.py index 9208e87..e17449c 100644 --- a/bincio/extract/parsers/gpx.py +++ b/bincio/extract/parsers/gpx.py @@ -53,6 +53,7 @@ class GpxParser(BaseParser): started_at=started_at, source_file=path.name, source_hash="", # set by factory + altitude_source="gps", ) diff --git a/bincio/extract/parsers/tcx.py b/bincio/extract/parsers/tcx.py index 5d52c2e..1c60f2b 100644 --- a/bincio/extract/parsers/tcx.py +++ b/bincio/extract/parsers/tcx.py @@ -83,6 +83,7 @@ class TcxParser: started_at=points[0].timestamp, source_file=path.name, source_hash="", + altitude_source="gps", ) diff --git a/bincio/extract/strava_api.py b/bincio/extract/strava_api.py index 1367c2d..4675ae9 100644 --- a/bincio/extract/strava_api.py +++ b/bincio/extract/strava_api.py @@ -37,16 +37,18 @@ class StravaError(Exception): # ── OAuth helpers ────────────────────────────────────────────────────────────── -def auth_url(client_id: str, redirect_uri: str) -> str: +def auth_url(client_id: str, redirect_uri: str, state: str = "") -> str: """Return the Strava OAuth authorization URL.""" - params = urllib.parse.urlencode({ + params: dict[str, str] = { "client_id": client_id, "redirect_uri": redirect_uri, "response_type": "code", "scope": "activity:read_all", "approval_prompt": "auto", - }) - return f"{_AUTH_URL}?{params}" + } + if state: + params["state"] = state + return f"{_AUTH_URL}?{urllib.parse.urlencode(params)}" def exchange_code(client_id: str, client_secret: str, code: str) -> dict: @@ -199,6 +201,10 @@ def strava_to_parsed(meta: dict, streams: dict) -> ParsedActivity: source = f"strava:{meta['id']}" source_hash = "sha256:" + hashlib.sha256(source.encode()).hexdigest() + # Map Strava visibility to BAS privacy: only_me → unlisted, everything else → public + visibility = meta.get("visibility") or "" + is_private = meta.get("private", False) or visibility == "only_me" + return ParsedActivity( points=points, sport=normalise_sport(meta.get("sport_type") or meta.get("type") or ""), @@ -208,4 +214,5 @@ def strava_to_parsed(meta: dict, streams: dict) -> ParsedActivity: title=meta.get("name") or None, description=meta.get("description") or None, strava_id=str(meta["id"]), + privacy="unlisted" if is_private else "public", ) diff --git a/bincio/extract/strava_csv.py b/bincio/extract/strava_csv.py index 472d847..a77f0b0 100644 --- a/bincio/extract/strava_csv.py +++ b/bincio/extract/strava_csv.py @@ -1,14 +1,14 @@ """Import metadata from Strava's activities.csv bulk export. Strava export columns we care about: - Activity ID, Activity Date, Activity Name, Activity Type, - Activity Description, Filename + Activity ID, Activity Date, Activity Name, Activity Description, Filename """ import csv +import json import re from pathlib import Path -from typing import Optional +from typing import Iterator, Optional _STRAVA_DATE_FMTS = ( @@ -18,10 +18,11 @@ _STRAVA_DATE_FMTS = ( class StravaMetadata: - """Maps original filename → Strava metadata.""" + """Maps original filename → Strava metadata, with secondary strava_id index.""" def __init__(self, csv_path: Path) -> None: self._by_filename: dict[str, dict] = {} + self._by_strava_id: dict[str, dict] = {} self._load(csv_path) def _load(self, path: Path) -> None: @@ -29,16 +30,21 @@ class StravaMetadata: reader = csv.DictReader(f) for row in reader: filename = row.get("Filename", "").strip() - if not filename: - continue - # Strava stores paths like "activities/12345.fit.gz" - basename = Path(filename).name - self._by_filename[basename] = row + if filename: + basename = Path(filename).name + self._by_filename[basename] = row + strava_id = row.get("Activity ID", "").strip() + if strava_id: + self._by_strava_id[strava_id] = row def lookup(self, source_file: str) -> Optional[dict]: """Return the Strava CSV row for a given source filename, or None.""" return self._by_filename.get(source_file) + def lookup_by_strava_id(self, strava_id: str) -> Optional[dict]: + """Return the Strava CSV row for a given Strava activity ID, or None.""" + return self._by_strava_id.get(str(strava_id)) + def enrich(self, source_file: str, activity: object) -> None: """Mutate a ParsedActivity with Strava metadata if found.""" row = self.lookup(source_file) @@ -53,3 +59,97 @@ class StravaMetadata: if not activity.strava_id and row.get("Activity ID"): # type: ignore[attr-defined] activity.strava_id = row["Activity ID"].strip() # type: ignore[attr-defined] + + +# ── Retroactive sidecar update ──────────────────────────────────────────────── + +def _parse_sidecar(path: Path) -> tuple[dict, str]: + """Return (frontmatter_dict, body) from a sidecar .md file.""" + import re as _re + import yaml + text = path.read_text(encoding="utf-8") + if text.startswith("---"): + parts = _re.split(r"^---[ \t]*$", text, maxsplit=2, flags=_re.MULTILINE) + if len(parts) >= 3: + fm = yaml.safe_load(parts[1]) or {} + return fm, parts[2].strip() + return {}, text.strip() + + +def _write_sidecar(path: Path, fm: dict, body: str) -> None: + import yaml + path.parent.mkdir(parents=True, exist_ok=True) + fm_text = yaml.safe_dump(fm, default_flow_style=False, allow_unicode=True).strip() + content = f"---\n{fm_text}\n---\n" + if body: + content += f"\n{body}\n" + path.write_text(content, encoding="utf-8") + + +def _update_sidecar_from_row(sidecar_path: Path, row: dict) -> bool: + """Create or update a sidecar with CSV title/description. + + Only fills fields that are not already set in the sidecar. + Returns True if anything changed. + """ + title = row.get("Activity Name", "").strip() + description = row.get("Activity Description", "").strip() + if not title and not description: + return False + + fm, body = _parse_sidecar(sidecar_path) if sidecar_path.exists() else ({}, "") + + changed = False + if title and "title" not in fm: + fm["title"] = title + changed = True + if description and not body: + body = description + changed = True + + if not changed: + return False + + _write_sidecar(sidecar_path, fm, body) + return True + + +def apply_csv_to_data_dir(data_dir: Path, metadata: StravaMetadata) -> int: + """Retroactively apply CSV metadata to existing activities via sidecars. + + Scans all activity JSONs in data_dir/activities/. For each activity that + has a strava_id, looks up the corresponding CSV row and creates/updates + the sidecar in data_dir/edits/ with any missing title or description. + + Only writes fields not already present in the sidecar — manual edits are + never overwritten. + + Returns the count of activities whose sidecars were created or updated. + """ + activities_dir = data_dir / "activities" + edits_dir = data_dir / "edits" + + if not activities_dir.exists(): + return 0 + + updated = 0 + for json_path in sorted(activities_dir.glob("*.json")): + try: + detail = json.loads(json_path.read_text(encoding="utf-8")) + except Exception: + continue + + strava_id = detail.get("strava_id") + if not strava_id: + continue + + row = metadata.lookup_by_strava_id(str(strava_id)) + if row is None: + continue + + activity_id = json_path.stem + sidecar_path = edits_dir / f"{activity_id}.md" + if _update_sidecar_from_row(sidecar_path, row): + updated += 1 + + return updated diff --git a/bincio/extract/strava_zip.py b/bincio/extract/strava_zip.py new file mode 100644 index 0000000..8eda4d2 --- /dev/null +++ b/bincio/extract/strava_zip.py @@ -0,0 +1,148 @@ +"""Process a Strava bulk export ZIP file into a BAS data store. + +The ZIP (downloaded from strava.com/athlete/delete_your_account or the data export +page) contains: + activities/ ← GPX, FIT, TCX files (plain or .gz variants) + activities.csv ← metadata (title, description, gear, strava ID) + bikes.csv / shoes.csv / … (ignored here) + +Processing strategy: stream one activity at a time to keep disk usage low. +The ZIP is never fully extracted; each activity file is extracted to a temp path, +parsed, ingested, then immediately deleted. The ZIP itself is deleted once done. +""" + +from __future__ import annotations + +import io +import json +import tempfile +import zipfile +from pathlib import Path +from typing import Generator, Optional + + +# File extensions recognised as activity files inside the ZIP. +_ACTIVITY_SUFFIXES = {".gpx", ".fit", ".tcx", ".gpx.gz", ".fit.gz", ".tcx.gz"} + + +def _is_activity_file(name: str) -> bool: + n = name.lower() + return any(n.endswith(s) for s in _ACTIVITY_SUFFIXES) + + +def strava_zip_iter( + zip_path: Path, + data_dir: Path, + originals_dir: Optional[Path] = None, + privacy: str = "public", +) -> Generator[dict, None, None]: + """Process a Strava export ZIP, yielding SSE-style progress dicts. + + Event types: + {"type": "validating"} + {"type": "error", "message": str} + {"type": "extracting_csv"} + {"type": "progress", "n": int, "total": int, "name": str, "status": "imported"|"skipped"|"error"} + {"type": "done", "imported": int, "skipped": int, "error_count": int, "errors": list[str]} + + The zip_path file is deleted after processing regardless of success/failure. + """ + from bincio.extract.ingest import ingest_parsed + from bincio.extract.parsers.factory import parse_file + from bincio.extract.strava_csv import StravaMetadata + + yield {"type": "validating"} + + try: + zf = zipfile.ZipFile(zip_path, "r") + except zipfile.BadZipFile as e: + zip_path.unlink(missing_ok=True) + yield {"type": "error", "message": f"Not a valid ZIP file: {e}"} + return + + try: + names = zf.namelist() + + # Validate structure + has_csv = "activities.csv" in names + activity_files = [n for n in names if n.startswith("activities/") and _is_activity_file(n)] + + if not has_csv: + yield {"type": "error", "message": "This doesn't look like a Strava export: activities.csv not found"} + return + if not activity_files: + yield {"type": "error", "message": "No activity files found in activities/ folder"} + return + + # Load activities.csv into memory (it's small — ~700 KB) + yield {"type": "extracting_csv"} + csv_bytes = zf.read("activities.csv") + with tempfile.NamedTemporaryFile(suffix=".csv", delete=False) as tmp_csv: + tmp_csv.write(csv_bytes) + tmp_csv_path = Path(tmp_csv.name) + try: + metadata = StravaMetadata(tmp_csv_path) + finally: + tmp_csv_path.unlink(missing_ok=True) + + total = len(activity_files) + imported = 0 + skipped = 0 + errors: list[str] = [] + + for n, zip_entry in enumerate(activity_files, 1): + entry_name = Path(zip_entry).name # e.g. "12345678.fit.gz" + # Title from metadata if available; fall back to filename stem + meta_row = metadata.lookup(entry_name) + display_name = (meta_row or {}).get("Activity Name", "").strip() or entry_name + + # Determine activity ID from entry to check for duplicates before extracting + # (can't do this without parsing, so we extract to a small temp file) + suffix = "".join(Path(entry_name).suffixes) # ".fit.gz" or ".gpx" etc. + tmp_path: Optional[Path] = None + try: + with tempfile.NamedTemporaryFile(suffix=suffix, delete=False, dir=data_dir) as tmp: + tmp.write(zf.read(zip_entry)) + tmp_path = Path(tmp.name) + + parsed = parse_file(tmp_path) + + # Enrich with CSV metadata + if meta_row: + if not parsed.title and meta_row.get("Activity Name"): + parsed.title = meta_row["Activity Name"].strip() + if not parsed.description and meta_row.get("Activity Description"): + parsed.description = meta_row["Activity Description"].strip() + if not parsed.strava_id and meta_row.get("Activity ID"): + parsed.strava_id = meta_row["Activity ID"].strip() + + if originals_dir is not None: + import shutil + orig_dest = originals_dir / entry_name + shutil.copy2(tmp_path, orig_dest) + + ingest_parsed(parsed, data_dir, privacy=privacy) + imported += 1 + yield {"type": "progress", "n": n, "total": total, "name": display_name, "status": "imported"} + + except FileExistsError: + skipped += 1 + yield {"type": "progress", "n": n, "total": total, "name": display_name, "status": "skipped"} + except Exception as exc: + errors.append(f"{entry_name}: {type(exc).__name__}") + yield {"type": "progress", "n": n, "total": total, "name": display_name, "status": "error"} + finally: + if tmp_path is not None: + tmp_path.unlink(missing_ok=True) + + finally: + zf.close() + zip_path.unlink(missing_ok=True) + + yield { + "type": "done", + "imported": imported, + "skipped": skipped, + "error_count": len(errors), + "errors": errors[:5], + } diff --git a/bincio/extract/timeseries.py b/bincio/extract/timeseries.py index 6b6a0b7..dc164b8 100644 --- a/bincio/extract/timeseries.py +++ b/bincio/extract/timeseries.py @@ -14,13 +14,14 @@ def build_timeseries( ) -> dict: """Return the BAS `timeseries` object. - privacy='no_gps' or 'private' → lat/lon set to null. + privacy='no_gps' → lat/lon set to null. All other privacy levels + (including 'unlisted') retain GPS in the timeseries. Downsamples so at most one point per second is emitted. """ if not points: return {"t": []} - include_gps = privacy not in ("no_gps", "private") + include_gps = privacy not in ("no_gps", "private") # "private" = legacy alias for "unlisted" # Downsample: keep at most one point per second sampled: list[DataPoint] = [] diff --git a/bincio/extract/writer.py b/bincio/extract/writer.py index 8648bc6..c862a97 100644 --- a/bincio/extract/writer.py +++ b/bincio/extract/writer.py @@ -48,7 +48,10 @@ def write_activity( acts_dir.mkdir(parents=True, exist_ok=True) source = _infer_source(activity) - has_gps = metrics.bbox is not None and privacy not in ("no_gps", "private") + # "unlisted" activities keep their GPS track (not in the public feed, but the + # URL is not secret — same model as the detail JSON). Only "no_gps" suppresses + # the track. "private" is the legacy alias for "unlisted". + has_gps = metrics.bbox is not None and privacy not in ("no_gps",) # Build timeseries once — written to a separate file to keep detail JSON small. # Treat an empty timeseries (no points) as None so no file is created. @@ -90,6 +93,7 @@ def write_activity( "source": source, "source_file": activity.source_file, "source_hash": activity.source_hash, + "altitude_source": activity.altitude_source, "strava_id": activity.strava_id, "duplicate_of": duplicate_of, "privacy": privacy, @@ -220,7 +224,7 @@ def build_summary( privacy: str = "public", ) -> dict: """Build the Activity Summary object for index.json.""" - has_gps = metrics.bbox is not None and privacy not in ("no_gps", "private") + has_gps = metrics.bbox is not None and privacy not in ("no_gps",) return { "id": activity_id, "title": activity.title or _auto_title(activity), diff --git a/bincio/reextract_cmd.py b/bincio/reextract_cmd.py new file mode 100644 index 0000000..8c68231 --- /dev/null +++ b/bincio/reextract_cmd.py @@ -0,0 +1,147 @@ +"""bincio reextract-originals — re-extract activities from stored Strava originals.""" + +from __future__ import annotations + +import ctypes +import gc +import json +import sys +from pathlib import Path + +import click + + +def _emit(obj: dict) -> None: + """Write a JSON progress line to stdout (flushed immediately).""" + print(json.dumps(obj), flush=True) + + +# On Linux, malloc_trim(0) returns freed arenas to the OS, keeping RSS low. +# CPython's allocator otherwise holds onto freed memory indefinitely. +try: + _libc = ctypes.CDLL("libc.so.6") + def _trim_heap() -> None: + _libc.malloc_trim(0) +except Exception: + def _trim_heap() -> None: # type: ignore[misc] + pass + +_GC_EVERY = 50 # call gc.collect() + malloc_trim every N activities + + +@click.command("reextract-originals") +@click.option("--data-dir", required=True, type=click.Path(), help="BAS data directory") +@click.option("--handle", required=True, help="User handle to re-extract for") +@click.option("--force", is_flag=True, default=False, help="Re-extract even if activity JSON already exists") +@click.option("--offset", default=0, type=int, help="Skip first N originals (for batch processing)") +@click.option("--limit", default=0, type=int, help="Process at most N originals then stop (0 = all)") +def reextract_originals(data_dir: str, handle: str, force: bool, offset: int, limit: int) -> None: + """Re-extract activities from stored Strava originals (originals/strava/*.json). + + Prints one JSON object per line to stdout for streaming progress: + {"type": "status", "message": "..."} + {"type": "progress", "n": 1, "total": 2015, "name": "...", "status": "imported"|"skipped"|"error", ["detail": "..."]} + {"type": "done", "imported": N, "skipped": N, "errors": N} + {"type": "error", "message": "..."} + """ + from bincio.extract.strava_api import strava_to_parsed + from bincio.extract.metrics import compute as compute_metrics + from bincio.extract.writer import ( + build_summary, make_activity_id, write_activity, write_index, + ) + from bincio.render.merge import merge_all + + dd = Path(data_dir).expanduser().resolve() + user_dir = dd / handle + originals_dir = user_dir / "originals" / "strava" + + if not originals_dir.exists(): + _emit({"type": "error", "message": f"No Strava originals directory at {originals_dir}"}) + sys.exit(1) + + all_files = sorted(originals_dir.glob("*.json")) + if not all_files: + _emit({"type": "error", "message": "No Strava originals found"}) + sys.exit(1) + + # Apply offset/limit for batch processing + batch = all_files[offset:] if not limit else all_files[offset: offset + limit] + total_all = len(all_files) + total = len(batch) + original_files = batch + + _emit({"type": "status", "message": ( + f"Batch {offset + 1}–{offset + total} of {total_all}, starting extraction…" + if offset or limit else + f"Found {total_all} originals, starting extraction…" + )}) + + # Load existing index to get owner info and existing summaries + index_path = user_dir / "index.json" + try: + existing_index = json.loads(index_path.read_text(encoding="utf-8")) if index_path.exists() else {} + except Exception: + existing_index = {} + owner = existing_index.get("owner", {"handle": handle}) + summaries: dict[str, dict] = {s["id"]: s for s in existing_index.get("activities", [])} + + imported = skipped = errors = 0 + + for n, orig_path in enumerate(original_files, 1): + try: + raw = json.loads(orig_path.read_text(encoding="utf-8")) + meta = raw.get("meta", {}) + streams = raw.get("streams", {}) + name = meta.get("name", orig_path.stem) + + parsed = strava_to_parsed(meta, streams) + activity_id = make_activity_id(parsed) + + if not force and (user_dir / "activities" / f"{activity_id}.json").exists(): + skipped += 1 + _emit({"type": "progress", "n": n, "total": total, "name": name, "status": "skipped"}) + else: + metrics = compute_metrics(parsed) + ep = parsed.privacy if parsed.privacy is not None else "public" + write_activity(parsed, metrics, user_dir, privacy=ep, rdp_epsilon=0.0001) + summaries[activity_id] = build_summary(parsed, metrics, activity_id, ep) + imported += 1 + _emit({"type": "progress", "n": n, "total": total, "name": name, "status": "imported"}) + + # Explicitly free large objects; also free the raw JSON dict and streams + raw = meta = streams = None # type: ignore[assignment] + try: + del parsed, metrics + except NameError: + pass + + except Exception as exc: + errors += 1 + _emit({"type": "progress", "n": n, "total": total, "name": orig_path.stem, + "status": "error", "detail": str(exc)}) + + # Periodically reclaim freed memory from CPython's allocator arena + if n % _GC_EVERY == 0: + gc.collect() + _trim_heap() + + # Final cleanup before the index write (which loads all summaries at once) + gc.collect() + _trim_heap() + + if imported > 0: + _emit({"type": "status", "message": "Writing index…"}) + try: + write_index(list(summaries.values()), user_dir, owner) + except Exception as exc: + _emit({"type": "error", "message": f"write_index failed: {exc}"}) + sys.exit(1) + + _emit({"type": "status", "message": "Running merge…"}) + try: + merge_all(user_dir) + except Exception as exc: + _emit({"type": "error", "message": f"merge_all failed: {exc}"}) + sys.exit(1) + + _emit({"type": "done", "imported": imported, "skipped": skipped, "errors": errors}) diff --git a/bincio/render/cli.py b/bincio/render/cli.py index e045f7e..b77917b 100644 --- a/bincio/render/cli.py +++ b/bincio/render/cli.py @@ -133,6 +133,11 @@ def _write_root_manifest(data: Path) -> None: root.write_text(json.dumps(manifest, indent=2)) console.print(f"Root manifest updated: [cyan]{len(users)}[/cyan] user shard(s)") + if len(users) > 1: + from bincio.render.merge import write_combined_feed + n = write_combined_feed(data) + console.print(f"Combined feed: [cyan]{n}[/cyan] activities across all users") + def _link_data(site: Path, data: Path) -> None: """Symlink site/public/data → data root (each user has their own _merged/).""" @@ -168,6 +173,8 @@ def _link_data(site: Path, data: Path) -> None: help="Deploy after build. Currently supports: github.") @click.option("--handle", default=None, help="(Multi-user) Incrementally re-merge one user's shard only.") +@click.option("--no-build", "no_build", is_flag=True, + help="Skip the Astro build step (just merge sidecars and update manifests).") def render( config_path: Optional[str], data_dir: Optional[str], @@ -176,6 +183,7 @@ def render( serve: bool, deploy: Optional[str], handle: Optional[str], + no_build: bool, ) -> None: """Build (or serve) the BincioActivity static site from a BAS data store.""" @@ -185,9 +193,14 @@ def render( console.print(f"Site: [cyan]{site}[/cyan]") console.print(f"Data: [cyan]{data}[/cyan]") - _ensure_npm(site) _merge_edits(data, handle=handle) _write_root_manifest(data) + + if no_build: + console.print("[green]Data updated.[/green] Skipping Astro build (--no-build).") + return + + _ensure_npm(site) _link_data(site, data) env = {**os.environ, "BINCIO_DATA_DIR": str(data)} diff --git a/bincio/render/merge.py b/bincio/render/merge.py index 8064776..2386791 100644 --- a/bincio/render/merge.py +++ b/bincio/render/merge.py @@ -49,7 +49,7 @@ def apply_sidecar(detail: dict, fm: dict, body: str) -> dict: if "highlight" in fm: d["custom"]["highlight"] = bool(fm["highlight"]) if "private" in fm: - d["privacy"] = "private" if fm["private"] else detail.get("privacy", "public") + d["privacy"] = "unlisted" if fm["private"] else detail.get("privacy", "public") if "hide_stats" in fm: d["custom"]["hide_stats"] = [str(s) for s in (fm["hide_stats"] or [])] @@ -69,7 +69,7 @@ def _apply_sidecar_summary(summary: dict, fm: dict) -> dict: if "highlight" in fm: s["custom"]["highlight"] = bool(fm["highlight"]) if "private" in fm: - s["privacy"] = "private" if fm["private"] else summary.get("privacy", "public") + s["privacy"] = "unlisted" if fm["private"] else summary.get("privacy", "public") return s @@ -152,12 +152,10 @@ def merge_one(data_dir: Path, activity_id: str) -> None: s = _apply_sidecar_summary(s, fm) activities.append(s) - activities = [a for a in activities if a.get("privacy") != "private"] activities.sort(key=lambda a: a.get("started_at", ""), reverse=True) activities.sort(key=lambda a: 0 if a.get("custom", {}).get("highlight") else 1) - index["activities"] = activities - (merged_dir / "index.json").write_text(json.dumps(index, indent=2, ensure_ascii=False)) + _write_year_shards(merged_dir, activities, index) def merge_all(data_dir: Path) -> int: @@ -261,18 +259,142 @@ def merge_all(data_dir: Path) -> int: s = _apply_sidecar_summary(s, fm) activities.append(s) - # Drop private activities from the published feed - activities = [a for a in activities if a.get("privacy") != "private"] - - # Sort: newest first, then bring highlighted activities to the top + # "unlisted" (and legacy "private") activities are kept in the index so + # the owner can reach them by direct URL; the feed UI filters them out + # for non-owners client-side. + # Sort: newest first, then bring highlighted activities to the top. activities.sort(key=lambda a: a.get("started_at", ""), reverse=True) activities.sort(key=lambda a: 0 if a.get("custom", {}).get("highlight") else 1) - index["activities"] = activities - (merged_dir / "index.json").write_text( - json.dumps(index, indent=2, ensure_ascii=False) - ) - elif (merged_dir / "index.json").exists(): - (merged_dir / "index.json").unlink() + _write_year_shards(merged_dir, activities, index) + else: + # Remove any stale year shard files if the source index disappeared + for f in merged_dir.glob("index-*.json"): + f.unlink() + if (merged_dir / "index.json").exists(): + (merged_dir / "index.json").unlink() return len(sidecars) + + +# Fields only needed for athlete.json aggregation at extract time — they add +# bulk to every summary entry but are never read by the feed UI. +_FEED_STRIP = {"best_efforts", "best_climb_m", "source"} + + +def _write_year_shards(merged_dir: Path, activities: list[dict], index_meta: dict) -> None: + """Split activities by year and write index-{year}.json shards. + + Replaces merged_dir/index.json with a shard manifest so the feed can + load only the most-recent year on first paint and fetch older years lazily. + """ + from collections import defaultdict + + # Remove stale year shard files from previous runs + for f in merged_dir.glob("index-*.json"): + f.unlink() + + by_year: dict[str, list[dict]] = defaultdict(list) + for a in activities: + year = (a.get("started_at") or "")[:4] or "unknown" + # Strip aggregation-only fields to keep shard files small + slim = {k: v for k, v in a.items() if k not in _FEED_STRIP} + by_year[year].append(slim) + + years = sorted(by_year.keys(), reverse=True) # newest first + shards = [] + for year in years: + shard_doc = { + **{k: v for k, v in index_meta.items() if k not in ("activities", "shards")}, + "shards": [], + "activities": by_year[year], + } + fname = f"index-{year}.json" + (merged_dir / fname).write_text(json.dumps(shard_doc, indent=2, ensure_ascii=False)) + shards.append({"url": fname, "year": int(year) if year.isdigit() else 0, + "count": len(by_year[year])}) + + root_doc = { + **{k: v for k, v in index_meta.items() if k not in ("activities", "shards")}, + "shards": shards, + "activities": [], + } + (merged_dir / "index.json").write_text(json.dumps(root_doc, indent=2, ensure_ascii=False)) + + +FEED_PAGE_SIZE = 50 + +# Extra fields stripped from the combined feed — preview_coords is the biggest +# contributor (~24% of shard size) but the feed cards need it for thumbnails, +# so we keep it. mmp is never displayed in feed cards. +_COMBINED_FEED_STRIP = _FEED_STRIP | {"mmp"} + + +def write_combined_feed(data_dir: Path) -> int: + """Build data_dir/feed.json — the N most recent activities across all users. + + The global feed page loads this single file instead of resolving 20+ user + shards recursively. Returns the number of activities written. + """ + user_dirs = sorted( + p for p in data_dir.iterdir() + if p.is_dir() and (p / "activities").exists() + ) + + all_activities: list[dict] = [] + for user_dir in user_dirs: + handle = user_dir.name + merged = user_dir / "_merged" + index_path = merged / "index.json" if merged.exists() else user_dir / "index.json" + if not index_path.exists(): + continue + + index = json.loads(index_path.read_text(encoding="utf-8")) + shards = index.get("shards", []) + activities = index.get("activities", []) + + if shards: + year_shards = [s for s in shards if re.match(r"index-\d{4}\.json$", s.get("url", ""))] + base = index_path.parent + for shard in year_shards: + shard_path = base / shard["url"] + if shard_path.exists(): + shard_data = json.loads(shard_path.read_text(encoding="utf-8")) + for a in shard_data.get("activities", []): + a_tagged = {**a, "handle": handle} + detail_url = a_tagged.get("detail_url", "") + if detail_url and not detail_url.startswith("http") and not detail_url.startswith("/"): + merged_rel = f"{handle}/_merged/" if merged.exists() else f"{handle}/" + a_tagged["detail_url"] = merged_rel + detail_url + track_url = a_tagged.get("track_url", "") + if track_url and not track_url.startswith("http") and not track_url.startswith("/"): + merged_rel = f"{handle}/_merged/" if merged.exists() else f"{handle}/" + a_tagged["track_url"] = merged_rel + track_url + all_activities.append(a_tagged) + else: + for a in activities: + all_activities.append({**a, "handle": handle}) + + all_activities.sort(key=lambda a: a.get("started_at", ""), reverse=True) + + # Remove stale feed pages + for f in data_dir.glob("feed*.json"): + f.unlink() + + if not all_activities: + return 0 + + pages = [all_activities[i:i + FEED_PAGE_SIZE] for i in range(0, len(all_activities), FEED_PAGE_SIZE)] + for page_num, page in enumerate(pages): + slim = [{k: v for k, v in a.items() if k not in _COMBINED_FEED_STRIP} for a in page] + fname = "feed.json" if page_num == 0 else f"feed-{page_num + 1}.json" + doc = { + "bas_version": "1.0", + "page": page_num + 1, + "total_pages": len(pages), + "total_activities": len(all_activities), + "activities": slim, + } + (data_dir / fname).write_text(json.dumps(doc, indent=2, ensure_ascii=False)) + + return len(all_activities) diff --git a/bincio/serve/cli.py b/bincio/serve/cli.py index 085ce0b..478cc1b 100644 --- a/bincio/serve/cli.py +++ b/bincio/serve/cli.py @@ -19,9 +19,13 @@ console = Console() @click.option("--strava-client-id", default=None, envvar="STRAVA_CLIENT_ID", help="Strava OAuth client ID (enables per-user Strava sync)") @click.option("--strava-client-secret", default=None, envvar="STRAVA_CLIENT_SECRET", help="Strava OAuth client secret") @click.option("--max-users", default=None, type=int, help="Override max users for this instance (0 = unlimited; updates the DB setting)") +@click.option("--public-url", default=None, envvar="PUBLIC_URL", help="Public base URL (e.g. https://yourdomain.com). Required for Strava OAuth to work behind a reverse proxy.") +@click.option("--webroot", default=None, type=click.Path(), help="Nginx webroot (e.g. /var/www/bincio). When set, uploads trigger a full Astro build + rsync so new activity pages are immediately accessible without a git push.") +@click.option("--dem-url", default=None, envvar="DEM_URL", help="Base URL of an Open-Elevation-compatible API (default: https://api.open-elevation.com).") def serve(data_dir: str, site_dir: Optional[str], host: str, port: int, strava_client_id: Optional[str], strava_client_secret: Optional[str], - max_users: Optional[int]) -> None: + max_users: Optional[int], public_url: Optional[str], + webroot: Optional[str], dem_url: Optional[str]) -> None: """Start the bincio multi-user application server. Handles auth, user management, and write operations. @@ -51,6 +55,12 @@ def serve(data_dir: str, site_dir: Optional[str], host: str, port: int, srv.strava_client_id = strava_client_id if strava_client_secret: srv.strava_client_secret = strava_client_secret + if public_url: + srv.public_url = public_url + if webroot and site_dir: + srv.webroot = Path(webroot).expanduser().resolve() + if dem_url: + srv.dem_url = dem_url db = open_db(dd) current_limit = get_setting(db, "max_users") @@ -60,11 +70,21 @@ def serve(data_dir: str, site_dir: Optional[str], host: str, port: int, console.print(f" Data: [cyan]{dd}[/cyan]") if srv.site_dir: console.print(f" Site: [cyan]{srv.site_dir}[/cyan]") + if srv.webroot: + console.print(f" Web: [cyan]{srv.webroot}[/cyan] (auto-rebuild on upload)") console.print(f" URL: [cyan]http://{host}:{port}[/cyan]") if current_limit and int(current_limit) > 0: console.print(f" Users: [yellow]max {current_limit}[/yellow]") else: console.print(f" Users: [dim]unlimited[/dim]") + console.print(f" DEM: [cyan]{srv.dem_url}[/cyan]") console.print() - uvicorn.run(srv.app, host=host, port=port, log_level="info") + log_config = uvicorn.config.LOGGING_CONFIG.copy() + # Make bincio.serve logger emit at INFO through uvicorn's handler + log_config["loggers"]["bincio.serve"] = { + "handlers": ["default"], + "level": "INFO", + "propagate": False, + } + uvicorn.run(srv.app, host=host, port=port, log_level="info", log_config=log_config) diff --git a/bincio/serve/db.py b/bincio/serve/db.py index d939ad4..96f450e 100644 --- a/bincio/serve/db.py +++ b/bincio/serve/db.py @@ -45,17 +45,36 @@ CREATE TABLE IF NOT EXISTS invites ( used_at INTEGER ); +CREATE TABLE IF NOT EXISTS reset_codes ( + code TEXT PRIMARY KEY, + handle TEXT NOT NULL REFERENCES users(handle) ON DELETE CASCADE, + created_by TEXT NOT NULL REFERENCES users(handle) ON DELETE CASCADE, + created_at INTEGER NOT NULL, + expires_at INTEGER NOT NULL, + used_at INTEGER +); + CREATE TABLE IF NOT EXISTS settings ( key TEXT PRIMARY KEY, value TEXT NOT NULL ); +CREATE TABLE IF NOT EXISTS user_prefs ( + handle TEXT NOT NULL REFERENCES users(handle) ON DELETE CASCADE, + key TEXT NOT NULL, + value TEXT NOT NULL, + PRIMARY KEY (handle, key) +); + CREATE INDEX IF NOT EXISTS sessions_handle ON sessions(handle); CREATE INDEX IF NOT EXISTS invites_created_by ON invites(created_by); +CREATE INDEX IF NOT EXISTS reset_codes_handle ON reset_codes(handle); +CREATE INDEX IF NOT EXISTS user_prefs_handle ON user_prefs(handle); """ _SESSION_DAYS = 30 _INVITE_LENGTH = 8 +_RESET_CODE_TTL_S = 24 * 3600 # 24 hours # ── Data classes ────────────────────────────────────────────────────────────── @@ -143,6 +162,13 @@ def authenticate(db: sqlite3.Connection, handle: str, password: str) -> Optional ) +def change_password(db: sqlite3.Connection, handle: str, new_password: str) -> None: + """Replace the password hash for a user.""" + new_hash = bcrypt.hashpw(new_password.encode(), bcrypt.gensalt()).decode() + db.execute("UPDATE users SET password_hash = ? WHERE handle = ?", (new_hash, handle)) + db.commit() + + def list_users(db: sqlite3.Connection) -> list[User]: rows = db.execute("SELECT * FROM users ORDER BY created_at").fetchall() return [User(handle=r["handle"], display_name=r["display_name"], @@ -154,6 +180,33 @@ def delete_user(db: sqlite3.Connection, handle: str) -> None: db.commit() +def get_member_tree(db: sqlite3.Connection) -> list[dict]: + """Return users with their inviter handle and join timestamp. + + Each entry: {handle, display_name, created_at, invited_by (handle or None)}. + Ordered oldest-first so callers can build the tree top-down. + """ + users = {r["handle"]: r for r in db.execute( + "SELECT handle, display_name, created_at FROM users ORDER BY created_at" + ).fetchall()} + # Map invitee → inviter from the used invites + invited_by: dict[str, str] = {} + for row in db.execute( + "SELECT created_by, used_by FROM invites WHERE used_by IS NOT NULL" + ).fetchall(): + invited_by[row["used_by"]] = row["created_by"] + + return [ + { + "handle": r["handle"], + "display_name": r["display_name"], + "created_at": r["created_at"], + "invited_by": invited_by.get(r["handle"]), + } + for r in users.values() + ] + + def count_users(db: sqlite3.Connection) -> int: """Return the total number of registered users.""" row = db.execute("SELECT COUNT(*) FROM users").fetchone() @@ -290,3 +343,84 @@ def get_invite(db: sqlite3.Connection, code: str) -> Optional[Invite]: created_at=row["created_at"], used_at=row["used_at"], ) + + +# ── Password reset codes ────────────────────────────────────────────────────── + +def create_reset_code(db: sqlite3.Connection, handle: str, created_by: str) -> str: + """Generate a password reset code for a user (admin only, out-of-band delivery). + + Any previous unused codes for this handle are invalidated first. + Returns the new code. + """ + now = int(time.time()) + # Invalidate existing unused codes for this handle + db.execute( + "DELETE FROM reset_codes WHERE handle = ? AND used_at IS NULL", + (handle,), + ) + code = secrets.token_urlsafe(_INVITE_LENGTH)[:_INVITE_LENGTH].upper() + db.execute( + "INSERT INTO reset_codes (code, handle, created_by, created_at, expires_at) " + "VALUES (?, ?, ?, ?, ?)", + (code, handle, created_by, now, now + _RESET_CODE_TTL_S), + ) + db.commit() + return code + + +# ── User preferences ───────────────────────────────────────────────────────── + +def get_user_prefs(db: sqlite3.Connection, handle: str) -> dict[str, str]: + """Return all preferences for a user as a plain dict.""" + rows = db.execute( + "SELECT key, value FROM user_prefs WHERE handle = ?", (handle,) + ).fetchall() + return {r["key"]: r["value"] for r in rows} + + +def set_user_pref(db: sqlite3.Connection, handle: str, key: str, value: str) -> None: + db.execute( + "INSERT INTO user_prefs (handle, key, value) VALUES (?, ?, ?) " + "ON CONFLICT(handle, key) DO UPDATE SET value = excluded.value", + (handle, key, value), + ) + db.commit() + + +def set_user_prefs(db: sqlite3.Connection, handle: str, prefs: dict[str, str]) -> None: + """Bulk-upsert multiple preferences for a user.""" + for key, value in prefs.items(): + db.execute( + "INSERT INTO user_prefs (handle, key, value) VALUES (?, ?, ?) " + "ON CONFLICT(handle, key) DO UPDATE SET value = excluded.value", + (handle, key, value), + ) + db.commit() + + +def use_reset_code(db: sqlite3.Connection, code: str, handle: str) -> bool: + """Validate a reset code for the given handle and mark it used. + + Returns False if the code is invalid, already used, expired, or + belongs to a different handle. + """ + now = int(time.time()) + row = db.execute( + "SELECT handle, expires_at, used_at FROM reset_codes WHERE code = ?", + (code,), + ).fetchone() + if not row: + return False + if row["handle"] != handle: + return False + if row["used_at"] is not None: + return False + if row["expires_at"] < now: + return False + db.execute( + "UPDATE reset_codes SET used_at = ? WHERE code = ?", + (now, code), + ) + db.commit() + return True diff --git a/bincio/serve/init_cmd.py b/bincio/serve/init_cmd.py index 5122857..7c6fc8e 100644 --- a/bincio/serve/init_cmd.py +++ b/bincio/serve/init_cmd.py @@ -24,7 +24,7 @@ def init(data_dir: str, handle: str, password: str, display_name: str, name: str Creates the SQLite database, the admin user, the per-user data directory, and prints a first invite code. Safe to re-run — skips steps already done. """ - from bincio.serve.db import create_invite, create_user, get_user, open_db, set_setting + from bincio.serve.db import create_invite, create_user, get_user, open_db, set_setting, get_setting dd = Path(data_dir).expanduser().resolve() dd.mkdir(parents=True, exist_ok=True) @@ -55,7 +55,19 @@ def init(data_dir: str, handle: str, password: str, display_name: str, name: str from datetime import datetime, timezone root_index = dd / "index.json" - if not root_index.exists(): + if root_index.exists(): + # Preserve existing manifest but always enforce private: True for a multi-user instance. + manifest = json.loads(root_index.read_text()) + instance = manifest.setdefault("instance", {}) + if not instance.get("private"): + instance["private"] = True + if name: + instance["name"] = name + root_index.write_text(json.dumps(manifest, indent=2)) + console.print(" [green]✓[/green] root index.json updated (private: true)") + else: + console.print(" [yellow]·[/yellow] root index.json already private — skipping") + else: manifest = { "bas_version": "1.0", "instance": {"name": name or "BincioActivity", "private": True}, @@ -65,8 +77,6 @@ def init(data_dir: str, handle: str, password: str, display_name: str, name: str } root_index.write_text(json.dumps(manifest, indent=2)) console.print(" [green]✓[/green] root index.json manifest written") - else: - console.print(" [yellow]·[/yellow] root index.json already exists — skipping") # ── User limit ──────────────────────────────────────────────────────────── if max_users > 0: @@ -75,6 +85,11 @@ def init(data_dir: str, handle: str, password: str, display_name: str, name: str else: console.print(" [dim]·[/dim] no user limit (unlimited)") + # ── Original file storage default ───────────────────────────────────────── + if get_setting(db, "store_originals") is None: + set_setting(db, "store_originals", "true") + console.print(" [green]✓[/green] store_originals = true (users can override per upload)") + # ── First invite code ───────────────────────────────────────────────────── code = create_invite(db, handle) diff --git a/bincio/serve/server.py b/bincio/serve/server.py index 3e6e202..d69ec15 100644 --- a/bincio/serve/server.py +++ b/bincio/serve/server.py @@ -9,13 +9,21 @@ Run via `bincio serve` CLI command. from __future__ import annotations import json +import logging import re +import secrets +import shutil import subprocess +import threading import time +import uuid from pathlib import Path from typing import Any, Optional -from fastapi import Cookie, FastAPI, File, HTTPException, Request, Response, UploadFile +log = logging.getLogger("bincio.serve") + +from fastapi import Cookie, FastAPI, File, Form, HTTPException, Request, Response, UploadFile +from fastapi.responses import RedirectResponse, StreamingResponse from fastapi.middleware.cors import CORSMiddleware from fastapi.middleware.gzip import GZipMiddleware from fastapi.responses import JSONResponse @@ -29,21 +37,128 @@ from bincio.serve.db import ( create_user, delete_session, get_invite, + get_member_tree, get_session, get_setting, get_user, + get_user_prefs, + set_user_prefs, list_invites, list_users, open_db, use_invite, ) +from pydantic import BaseModel, Field + +# ── Pydantic request/response models ───────────────────────────────────────── + + +class LoginRequest(BaseModel): + handle: str = Field(..., description="User handle (username)") + password: str = Field(..., description="User password") + + +class LoginResponse(BaseModel): + ok: bool = Field(True, description="Success flag") + handle: str = Field(..., description="User handle") + display_name: str = Field(..., description="User's display name") + + +class ResetPasswordRequest(BaseModel): + handle: str = Field(..., description="User handle") + code: str = Field(..., description="Reset code (24 hours valid)") + password: str = Field(..., description="New password (min 8 chars)") + + +class RegisterRequest(BaseModel): + code: str = Field(..., description="Invite code") + handle: str = Field(..., description="Desired username (lowercase, 1-30 chars)") + password: str = Field(..., description="Password (min 8 characters)") + display_name: str = Field(default="", description="Full name (optional, defaults to handle)") + + +class RegisterResponse(BaseModel): + ok: bool = Field(True, description="Success flag") + handle: str = Field(..., description="New user's handle") + + +class CurrentUserResponse(BaseModel): + handle: str = Field(..., description="User handle") + display_name: str = Field(..., description="User's display name") + is_admin: bool = Field(..., description="Whether user is an admin") + store_originals_default: bool = Field( + default=True, + description="Instance-wide default for storing original files" + ) + + +class ActivityEditRequest(BaseModel): + title: str | None = Field(default=None, description="Activity title") + description: str | None = Field(default=None, description="Activity description (markdown)") + sport: str | None = Field(default=None, description="Sport type") + private: bool | None = Field(default=None, description="Hide from public feed") + highlight: bool | None = Field(default=None, description="Mark as favorite") + gear: str | None = Field(default=None, description="Gear used (e.g., 'Trek Domane')") + + +class ActivityEditResponse(BaseModel): + ok: bool = Field(True, description="Success flag") + + +class ResetPasswordCodeResponse(BaseModel): + ok: bool = Field(True, description="Success flag") + code: str = Field(..., description="One-time reset code") + expires_in_hours: int = Field(24, description="Code validity period in hours") + + +class GenericResponse(BaseModel): + ok: bool = Field(True, description="Success flag") + + +# ── Active job tracker ─────────────────────────────────────────────────────── +# Tracks in-progress upload/processing jobs so admins can see what's running. +# Jobs are added when a streaming upload starts and removed when it finishes. + +_jobs_lock = threading.Lock() +_active_jobs: dict[str, dict] = {} + + +def _job_start(user_handle: str, total_files: int) -> str: + job_id = uuid.uuid4().hex[:8] + with _jobs_lock: + _active_jobs[job_id] = { + "id": job_id, + "user": user_handle, + "started_at": int(time.time()), + "total": total_files, + "done": 0, + "current": "", + } + return job_id + + +def _job_update(job_id: str, done: int, current: str) -> None: + with _jobs_lock: + if job_id in _active_jobs: + _active_jobs[job_id]["done"] = done + _active_jobs[job_id]["current"] = current + + +def _job_finish(job_id: str) -> None: + with _jobs_lock: + _active_jobs.pop(job_id, None) + + # ── Globals (set by CLI before uvicorn starts) ──────────────────────────────── data_dir: Path | None = None site_dir: Path | None = None # for post-write rebuild trigger +webroot: Path | None = None # nginx webroot — when set, trigger full rebuild + rsync strava_client_id: str = "" strava_client_secret: str = "" +public_url: str = "" # e.g. "https://yourdomain.com" — used for OAuth redirect URIs +dem_url: str = "https://api.open-elevation.com" # Open-Elevation-compatible API base URL _db = None # sqlite3.Connection, opened lazily @@ -54,6 +169,29 @@ def _get_db(): return _db +_STRAVA_CREDS_FILE = "strava_credentials.json" + + +def _strava_creds(handle: str) -> tuple[str, str]: + """Return (client_id, client_secret) for a user. + + Per-user credentials stored in {user_dir}/strava_credentials.json take + precedence over the global instance-level strava_client_id/secret. + Returns ("", "") when neither is configured. + """ + creds_path = _get_data_dir() / handle / _STRAVA_CREDS_FILE + if creds_path.exists(): + try: + d = json.loads(creds_path.read_text(encoding="utf-8")) + cid = str(d.get("client_id", "")).strip() + csec = str(d.get("client_secret", "")).strip() + if cid and csec: + return cid, csec + except Exception: + pass + return strava_client_id, strava_client_secret + + def _get_data_dir() -> Path: if data_dir is None: raise HTTPException(500, "Server not configured") @@ -62,7 +200,20 @@ def _get_data_dir() -> Path: # ── App ─────────────────────────────────────────────────────────────────────── -app = FastAPI(title="BincioActivity Serve", docs_url=None, redoc_url=None) +app = FastAPI(title="BincioActivity Serve") + + +@app.on_event("startup") +async def _cleanup_orphaned_tmp_zips() -> None: + """Remove tmp*.zip files left in user data dirs by the pre-fix upload handler.""" + import glob as _glob + data_dir = _get_data_dir() + for p in _glob.glob(str(data_dir / "*" / "tmp*.zip")): + try: + Path(p).unlink() + except Exception: + pass + app.add_middleware(GZipMiddleware, minimum_size=1024) app.add_middleware( @@ -141,44 +292,161 @@ def _set_session_cookie(response: Response, token: str) -> None: ) +# ── Image upload constants ──────────────────────────────────────────────────── + +_ALLOWED_IMAGE_TYPES = {"image/jpeg", "image/png", "image/webp", "image/gif"} +_MAX_IMAGE_BYTES = 10 * 1024 * 1024 # 10 MB + + +def _unique_image_name(directory: Path, filename: str) -> str: + """Return a filename that does not collide with existing files in directory.""" + stem, suffix = Path(filename).stem, Path(filename).suffix + candidate = filename + counter = 1 + while (directory / candidate).exists(): + candidate = f"{stem}_{counter}{suffix}" + counter += 1 + return candidate + + # ── Post-write rebuild ──────────────────────────────────────────────────────── +# Serialises concurrent rebuilds — only one full build runs at a time. +# A second upload that arrives while a build is in progress will queue and +# run after the first finishes, picking up all data written in between. +_rebuild_lock = threading.Lock() + + def _trigger_rebuild(handle: str) -> None: - """Asynchronously re-merge one user's shard and rewrite the root manifest.""" + """Asynchronously re-merge and optionally rebuild + rsync the site. + + - Without --webroot: fast path — merges sidecars + rewrites root manifest + (~1 s). New activity pages require the nginx try_files fallback to work. + - With --webroot: full Astro build + rsync to the nginx webroot (~30–60 s, + serialised). New activity pages are immediately accessible. + """ if site_dir is None: return - subprocess.Popen( - ["uv", "run", "bincio", "render", - "--data-dir", str(data_dir), - "--site-dir", str(site_dir), - "--handle", handle], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, - ) + if not _VALID_HANDLE.match(handle): + return # safety: never pass untrusted strings to subprocess + + uv = shutil.which("uv") or str(Path.home() / ".local" / "bin" / "uv") + _data_dir = str(data_dir) + _site_dir = str(site_dir) + _webroot = str(webroot) if webroot else None + _handle = handle + + def _run() -> None: + try: + if _webroot is None: + # Fast: only update data, skip Astro build. + # Serialised with the same lock: merge_all wipes and recreates + # _merged/activities/ — concurrent runs would corrupt each other. + log.info("rebuild[%s]: merge-only (no webroot)", _handle) + with _rebuild_lock: + result = subprocess.run( + [uv, "run", "bincio", "render", + "--data-dir", _data_dir, + "--site-dir", _site_dir, + "--handle", _handle, + "--no-build"], + capture_output=True, + text=True, + ) + if result.returncode != 0: + log.error("rebuild[%s]: merge failed (rc=%d):\n%s\n%s", + _handle, result.returncode, result.stdout, result.stderr) + else: + log.info("rebuild[%s]: merge done", _handle) + else: + # Full build + rsync — serialised so concurrent uploads don't race + log.info("rebuild[%s]: full build + rsync to %s", _handle, _webroot) + with _rebuild_lock: + result = subprocess.run( + [uv, "run", "bincio", "render", + "--data-dir", _data_dir, + "--site-dir", _site_dir, + "--handle", _handle], + capture_output=True, + text=True, + ) + if result.returncode != 0: + log.error("rebuild[%s]: build failed (rc=%d):\n%s\n%s", + _handle, result.returncode, result.stdout, result.stderr) + else: + log.info("rebuild[%s]: build done, rsyncing", _handle) + # Prune dist/data/ before rsync: Astro resolves the + # public/data symlink and copies all activity JSON into + # dist/, but nginx already serves /data/ directly from + # the live data dir — rsyncing it would duplicate GBs. + dist_data = Path(_site_dir) / "dist" / "data" + if dist_data.exists(): + shutil.rmtree(dist_data) + rsync = subprocess.run( + ["rsync", "-a", "--delete", "--exclude=data/", + f"{_site_dir}/dist/", _webroot + "/"], + capture_output=True, + text=True, + ) + if rsync.returncode != 0: + log.error("rebuild[%s]: rsync failed (rc=%d):\n%s\n%s", + _handle, rsync.returncode, rsync.stdout, rsync.stderr) + else: + log.info("rebuild[%s]: rsync done", _handle) + except Exception: + log.exception("rebuild[%s]: unexpected error", _handle) + + threading.Thread(target=_run, daemon=True).start() # ── Auth endpoints ──────────────────────────────────────────────────────────── -@app.get("/api/me") +@app.get("/api/me", response_model=CurrentUserResponse) async def me(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: user = _current_user(bincio_session) if not user: raise HTTPException(404, "Not authenticated") + store_orig = get_setting(_get_db(), "store_originals") return JSONResponse({ "handle": user.handle, "display_name": user.display_name, "is_admin": user.is_admin, + "store_originals_default": store_orig != "false", + "dem_configured": bool(dem_url), }) -@app.post("/api/auth/login") -async def login(request: Request) -> JSONResponse: +@app.get("/api/stats") +async def stats() -> JSONResponse: + """Public endpoint: member count, join dates, and invitation tree.""" + import time as _time + now = int(_time.time()) + members = get_member_tree(_get_db()) + return JSONResponse({ + "user_count": len(members), + "members": [ + { + "handle": m["handle"], + "display_name": m["display_name"], + "member_since": m["created_at"], + "member_for_days": (now - m["created_at"]) // 86400, + "invited_by": m["invited_by"], + } + for m in members + ], + }) + + +@app.post("/api/auth/login", response_model=LoginResponse) +async def login( + login_req: LoginRequest, + request: Request, +) -> JSONResponse: ip = request.client.host if request.client else "unknown" _check_rate_limit(ip, _login_attempts, _LOGIN_RATE_LIMIT, "Too many login attempts. Try again later.") - body = await request.json() - handle = body.get("handle", "").strip().lower() - password = body.get("password", "") + handle = login_req.handle.strip().lower() + password = login_req.password user = authenticate(_get_db(), handle, password) if not user: @@ -190,7 +458,7 @@ async def login(request: Request) -> JSONResponse: return resp -@app.post("/api/auth/logout") +@app.post("/api/auth/logout", response_model=GenericResponse) async def logout(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: if bincio_session: delete_session(_get_db(), bincio_session) @@ -199,18 +467,36 @@ async def logout(bincio_session: Optional[str] = Cookie(default=None)) -> JSONRe return resp +@app.post("/api/auth/reset-password", response_model=GenericResponse) +async def reset_password(reset_req: ResetPasswordRequest) -> JSONResponse: + """Validate a reset code and set a new password. Public endpoint.""" + from bincio.serve.db import use_reset_code, change_password + handle = reset_req.handle.strip().lower() + code = reset_req.code.strip().upper() + new_pw = reset_req.password + if len(new_pw) < 8: + raise HTTPException(400, "Password must be at least 8 characters") + db = _get_db() + if not use_reset_code(db, code, handle): + raise HTTPException(400, "Invalid or expired reset code") + change_password(db, handle, new_pw) + return JSONResponse({"ok": True}) + + # ── Registration ────────────────────────────────────────────────────────────── -@app.post("/api/register") -async def register(request: Request) -> JSONResponse: +@app.post("/api/register", response_model=RegisterResponse) +async def register( + register_req: RegisterRequest, + request: Request, +) -> JSONResponse: ip = request.client.host if request.client else "unknown" _check_rate_limit(ip, _register_attempts, _REGISTER_RATE_LIMIT, "Too many registration attempts. Try again later.") - body = await request.json() - code = body.get("code", "").strip().upper() - handle = body.get("handle", "").strip().lower() - password = body.get("password", "") - display = body.get("display_name", "").strip() or handle + code = register_req.code.strip().upper() + handle = register_req.handle.strip().lower() + password = register_req.password + display = register_req.display_name.strip() or handle if not _VALID_HANDLE.match(handle): raise HTTPException(400, "Invalid handle (lowercase letters, numbers, _ - only; max 30 chars)") @@ -234,14 +520,24 @@ async def register(request: Request) -> JSONResponse: # Create per-user directories dd = _get_data_dir() - (dd / handle / "activities").mkdir(parents=True, exist_ok=True) - (dd / handle / "edits").mkdir(parents=True, exist_ok=True) + user_dir = dd / handle + (user_dir / "activities").mkdir(parents=True, exist_ok=True) + (user_dir / "edits").mkdir(parents=True, exist_ok=True) + + # Write an empty index.json so the shard URL resolves immediately, + # even before the user uploads any activities. + from bincio.extract.writer import write_index + index_path = user_dir / "index.json" + if not index_path.exists(): + write_index([], user_dir, {"handle": handle, "display_name": display or handle}) # Update root manifest so the new user's shard is discoverable immediately - # (Astro dev re-evaluates getStaticPaths() on each request from this file) from bincio.render.cli import _write_root_manifest _write_root_manifest(dd) + # Rebuild site so the new user's profile pages exist immediately + _trigger_rebuild(handle) + token = create_session(_get_db(), handle) resp = JSONResponse({"ok": True, "handle": handle}) _set_session_cookie(resp, token) @@ -287,6 +583,649 @@ async def admin_users(bincio_session: Optional[str] = Cookie(default=None)) -> J } for u in users]) +@app.get("/api/admin/jobs") +async def admin_jobs(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Return currently active upload/processing jobs. Admin only.""" + _require_admin(bincio_session) + with _jobs_lock: + jobs = list(_active_jobs.values()) + return JSONResponse(jobs) + + +@app.get("/api/admin/disk") +async def admin_disk(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Per-user disk usage breakdown. Admin only.""" + _require_admin(bincio_session) + import shutil + + data_dir = _get_data_dir() + + def _mb(path: Path) -> float: + if not path.exists(): + return 0.0 + # Use lstat to count symlink entries (few bytes each) rather than following + # the link to the target — prevents _merged/ from double-counting activities/. + total = sum(f.lstat().st_size for f in path.rglob("*") if f.is_file() or f.is_symlink()) + return round(total / 1_048_576, 1) + + def _count(path: Path, pattern: str = "*") -> int: + if not path.exists(): + return 0 + return sum(1 for f in path.glob(pattern) if f.is_file()) + + db = _get_db() + from bincio.serve.db import get_user as _get_user + users = [] + for user_dir in sorted(data_dir.iterdir()): + if not user_dir.is_dir() or user_dir.name.startswith("_"): + continue + # leaked tmp zips + leaked = [f for f in user_dir.glob("tmp*.zip") if f.is_file()] + users.append({ + "handle": user_dir.name, + "in_db": _get_user(db, user_dir.name) is not None, + "total_mb": _mb(user_dir), + "activities_mb": _mb(user_dir / "activities"), + "activities_count": _count(user_dir / "activities", "*.json"), + "merged_mb": _mb(user_dir / "_merged"), + "originals_mb": _mb(user_dir / "originals"), + "originals_strava_mb": _mb(user_dir / "originals" / "strava"), + "images_mb": _mb(user_dir / "edits" / "images"), + "leaked_zips_mb": round(sum(f.stat().st_size for f in leaked) / 1_048_576, 1), + "leaked_zips_count": len(leaked), + }) + + disk = shutil.disk_usage("/") + return JSONResponse({ + "disk": { + "total_gb": round(disk.total / 1_073_741_824, 1), + "used_gb": round(disk.used / 1_073_741_824, 1), + "free_gb": round(disk.free / 1_073_741_824, 1), + "percent": round(disk.used / disk.total * 100, 1), + }, + "users": users, + }) + + +@app.post("/api/admin/users/{handle}/reset-password-code", response_model=ResetPasswordCodeResponse) +async def admin_reset_password_code( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Generate a one-time password reset code for a user. Admin only.""" + from bincio.serve.db import create_reset_code + admin = _require_admin(bincio_session) + db = _get_db() + if not get_user(db, handle): + raise HTTPException(404, f"User '{handle}' not found") + code = create_reset_code(db, handle, admin.handle) + return JSONResponse({"ok": True, "code": code, "expires_in_hours": 24}) + + +@app.post("/api/admin/users/{handle}/rebuild") +async def admin_rebuild( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Trigger a merge_all + site rebuild for a user. Admin only.""" + _require_admin(bincio_session) + user_dir = _get_data_dir() / handle + if not user_dir.is_dir(): + raise HTTPException(404, f"No data directory for user '{handle}'") + _trigger_rebuild(handle) + return JSONResponse({"ok": True}) + + +@app.post("/api/admin/users/{handle}/rebuild-sync") +async def admin_rebuild_sync( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Run merge+rebuild synchronously and return full output. Admin only. + + Unlike /rebuild (fire-and-forget), this blocks until done and returns stdout/stderr. + Use for debugging when you need to see what went wrong. + """ + _require_admin(bincio_session) + user_dir = _get_data_dir() / handle + if not user_dir.is_dir(): + raise HTTPException(404, f"No data directory for user '{handle}'") + if site_dir is None: + raise HTTPException(503, "Server has no --site-dir configured; rebuild not available") + + uv = shutil.which("uv") or str(Path.home() / ".local" / "bin" / "uv") + cmd = [uv, "run", "bincio", "render", + "--data-dir", str(data_dir), + "--site-dir", str(site_dir), + "--handle", handle, + "--no-build"] + if webroot: + cmd = [uv, "run", "bincio", "render", + "--data-dir", str(data_dir), + "--site-dir", str(site_dir), + "--handle", handle] + + result = subprocess.run(cmd, capture_output=True, text=True, timeout=300) + resp: dict[str, Any] = { + "ok": result.returncode == 0, + "returncode": result.returncode, + "stdout": result.stdout, + "stderr": result.stderr, + } + + if result.returncode == 0 and webroot: + dist_data = site_dir / "dist" / "data" + if dist_data.exists(): + shutil.rmtree(dist_data) + rsync = subprocess.run( + ["rsync", "-a", "--delete", "--exclude=data/", + f"{site_dir}/dist/", str(webroot) + "/"], + capture_output=True, text=True, timeout=120, + ) + resp["rsync_returncode"] = rsync.returncode + resp["rsync_stdout"] = rsync.stdout + resp["rsync_stderr"] = rsync.stderr + resp["ok"] = rsync.returncode == 0 + + return JSONResponse(resp) + + +@app.post("/api/admin/users/{handle}/reextract-originals") +async def admin_reextract_originals( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> StreamingResponse: + """Re-extract activities from stored Strava originals without hitting the API. + + Spawns `bincio reextract-originals` as a subprocess so heavy memory use + is isolated from the server process. Streams its JSON-lines output as SSE. + Triggers a full rebuild on completion. + """ + import asyncio + _require_admin(bincio_session) + user_dir = _get_data_dir() / handle + originals_dir = user_dir / "originals" / "strava" + if not originals_dir.exists(): + raise HTTPException(404, f"No Strava originals directory for '{handle}'") + + # Use the bincio script from the same venv bin dir as the running Python. + # This is reliable in systemd environments where PATH may not include uv. + import sys as _sys + bincio_exe = str(Path(_sys.executable).parent / "bincio") + data_dir = str(_get_data_dir()) + + # Count originals so we can split into memory-safe batches. + total_originals = len(list(originals_dir.glob("*.json"))) + # Each activity can briefly peak at ~10–30 MB; 100 per batch keeps RSS + # well under 3 GB even on a cheap VPS. + _BATCH = 100 + log.info("reextract[%s]: %d originals, batch size %d, via %s", + handle, total_originals, _BATCH, bincio_exe) + + async def event_stream(): + total_imported = total_skipped = total_errors = 0 + offset = 0 + + while offset < total_originals: + limit = min(_BATCH, total_originals - offset) + proc = await asyncio.create_subprocess_exec( + bincio_exe, "reextract-originals", + "--data-dir", data_dir, + "--handle", handle, + "--offset", str(offset), + "--limit", str(limit), + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.PIPE, + ) + assert proc.stdout is not None + + async for raw_line in proc.stdout: + line = raw_line.decode(errors="replace").strip() + if not line: + continue + yield f"data: {line}\n\n" + try: + evt = json.loads(line) + if evt.get("type") == "done": + total_imported += evt.get("imported", 0) + total_skipped += evt.get("skipped", 0) + total_errors += evt.get("errors", 0) + except Exception: + pass + + await proc.wait() + if proc.returncode != 0: + stderr_out = await proc.stderr.read() if proc.stderr else b"" + log.error("reextract[%s]: batch offset=%d exited %d — stderr: %s", + handle, offset, proc.returncode, + stderr_out.decode(errors="replace")[:500]) + yield f"data: {json.dumps({'type': 'error', 'message': f'Batch {offset}–{offset+limit} exited with code {proc.returncode}'})}\n\n" + return # stop on batch failure + + offset += limit + + # All batches complete + log.info("reextract[%s]: all batches done — imported=%d skipped=%d errors=%d; triggering rebuild", + handle, total_imported, total_skipped, total_errors) + _trigger_rebuild(handle) + yield f"data: {json.dumps({'type': 'done', 'imported': total_imported, 'skipped': total_skipped, 'errors': total_errors})}\n\n" + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) + + +@app.get("/api/admin/users/{handle}/diag") +async def admin_diag( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Return a diagnostic snapshot of a user's data directory. Admin only.""" + _require_admin(bincio_session) + user_dir = _get_data_dir() / handle + if not user_dir.is_dir(): + raise HTTPException(404, f"No data directory for user '{handle}'") + + def _count(path: Path, glob: str = "*") -> int: + return sum(1 for f in path.glob(glob) if f.is_file()) if path.exists() else 0 + + def _size_mb(path: Path) -> float: + if not path.exists(): + return 0.0 + return sum(f.stat().st_size for f in path.rglob("*") if f.is_file()) / 1_048_576 + + activities_dir = user_dir / "activities" + merged_dir = user_dir / "_merged" + originals_dir = user_dir / "originals" + uploads_dir = user_dir / "_uploads" + + merged_index = merged_dir / "index.json" + root_index = user_dir / "index.json" + + merged_activity_count: int | None = None + if merged_index.exists(): + try: + idx = json.loads(merged_index.read_text()) + merged_activity_count = len(idx.get("activities", [])) + except Exception: + merged_activity_count = -1 + + root_activity_count: int | None = None + if root_index.exists(): + try: + idx = json.loads(root_index.read_text()) + root_activity_count = len(idx.get("activities", [])) + except Exception: + root_activity_count = -1 + + # Peek at a few filenames in activities/ to understand the actual state + acts_sample: list[str] = [] + acts_symlinks = 0 + if activities_dir.exists(): + for f in sorted(activities_dir.iterdir())[:10]: + acts_sample.append(f.name + (" → symlink" if f.is_symlink() else "")) + if f.is_symlink(): + acts_symlinks += 1 + + # Check _merged/activities/ separately + merged_acts_dir = merged_dir / "activities" + merged_acts_json = _count(merged_acts_dir, "*.json") + merged_acts_geojson = _count(merged_acts_dir, "*.geojson") + + # List pending files + pending_files: list[str] = [] + if uploads_dir.exists(): + pending_files = [f.name for f in uploads_dir.iterdir() if f.is_file()] + + return JSONResponse({ + "handle": handle, + "user_dir": str(user_dir), + "activities": { + "json_files": _count(activities_dir, "*.json"), + "geojson_files": _count(activities_dir, "*.geojson"), + "size_mb": round(_size_mb(activities_dir), 2), + "sample": acts_sample, + "symlink_count": acts_symlinks, + }, + "originals": { + "exists": originals_dir.exists(), + "size_mb": round(_size_mb(originals_dir), 2), + "strava_originals": _count(originals_dir / "strava", "*.json") if (originals_dir / "strava").exists() else 0, + }, + "merged": { + "exists": merged_dir.exists(), + "activity_count_in_index": merged_activity_count, + "size_mb": round(_size_mb(merged_dir), 2), + "activities_json": merged_acts_json, + "activities_geojson": merged_acts_geojson, + }, + "root_index": { + "exists": root_index.exists(), + "activity_count": root_activity_count, + }, + "pending_uploads": len(pending_files), + "pending_files": pending_files, + "dedup_cache_exists": (user_dir / ".bincio_cache.json").exists(), + "athlete_json_exists": (user_dir / "athlete.json").exists(), + }) + + +def _wipe_user_activities(user_dir: Path) -> int: + """Delete all extracted activity files and caches for a user. + + Removes activities/ (JSON + GeoJSON + timeseries), edits/, originals/, + _merged/, index.json, athlete.json, and the dedup cache. + Leaves the user directory itself intact (account remains in the DB). + Returns the number of files deleted. + """ + import shutil + deleted = 0 + + for subdir in ("activities", "edits", "originals"): + d = user_dir / subdir + if d.exists(): + for f in d.rglob("*"): + if f.is_file(): + deleted += 1 + shutil.rmtree(d) + + for name in ("_merged", ): + d = user_dir / name + if d.exists(): + shutil.rmtree(d) + + for name in ("index.json", "athlete.json", ".bincio_cache.json"): + f = user_dir / name + if f.exists(): + f.unlink() + deleted += 1 + + return deleted + + +@app.delete("/api/admin/users/{handle}/activities") +async def admin_delete_activities( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Delete all activity data for a user and wipe the merged cache.""" + _require_admin(bincio_session) + user_dir = _get_data_dir() / handle + if not user_dir.is_dir(): + raise HTTPException(404, f"No data directory for user '{handle}'") + + deleted = _wipe_user_activities(user_dir) + _trigger_rebuild(handle) + return JSONResponse({"ok": True, "deleted": deleted}) + + +@app.delete("/api/admin/users/{handle}/directory") +async def admin_delete_user_directory( + handle: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Delete the entire user directory from disk (for ghost users not in the DB). + + Refuses if the handle exists as an account in the database — use + DELETE /api/admin/users/{handle}/activities for registered users. + """ + import shutil + _require_admin(bincio_session) + db = _get_db() + from bincio.serve.db import get_user as _get_user + if _get_user(db, handle) is not None: + raise HTTPException( + 400, + f"User '{handle}' is still in the database. Remove the account first, " + "or use 'Reset data' to wipe only activity files.", + ) + user_dir = _get_data_dir() / handle + if not user_dir.is_dir(): + raise HTTPException(404, f"No directory for '{handle}'") + shutil.rmtree(user_dir) + # Rebuild root manifest so the ghost shard disappears from the site + from bincio.render.cli import _write_root_manifest + try: + _write_root_manifest(_get_data_dir()) + except Exception: + pass + return JSONResponse({"ok": True}) + + + +# ── Self-service user settings ──────────────────────────────────────────────── + +@app.get("/api/me/storage") +async def me_storage(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Return per-category disk usage for the logged-in user.""" + user = _require_user(bincio_session) + dd = _get_data_dir() / user.handle + + def _mb(path: Path) -> float: + if not path.exists(): + return 0.0 + total = sum(f.lstat().st_size for f in path.rglob("*") if f.is_file() or f.is_symlink()) + return round(total / 1_048_576, 2) + + def _count(path: Path, pattern: str = "*") -> int: + if not path.exists(): + return 0 + return sum(1 for f in path.glob(pattern) if f.is_file()) + + activities_mb = _mb(dd / "activities") + originals_mb = _mb(dd / "originals") + strava_mb = _mb(dd / "originals" / "strava") + images_mb = _mb(dd / "edits" / "images") + total_mb = _mb(dd) + + return JSONResponse({ + "total_mb": total_mb, + "activities_mb": activities_mb, + "activities_count": _count(dd / "activities", "*.json"), + "originals_mb": originals_mb, + "strava_originals_mb": strava_mb, + "strava_originals_count": _count(dd / "originals" / "strava", "*.json"), + "images_mb": images_mb, + }) + + +@app.delete("/api/me/originals") +async def me_delete_originals(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Delete the user's originals/ directory (frees space after re-extraction).""" + user = _require_user(bincio_session) + originals = _get_data_dir() / user.handle / "originals" + if not originals.exists(): + return JSONResponse({"ok": True, "freed_mb": 0.0}) + + freed = round( + sum(f.stat().st_size for f in originals.rglob("*") if f.is_file()) / 1_048_576, 2 + ) + shutil.rmtree(originals) + return JSONResponse({"ok": True, "freed_mb": freed}) + + +@app.delete("/api/me/activities") +async def me_delete_activities( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Wipe all extracted activity data (activities/, edits/, _merged/, index/athlete JSON). + + Requires the user's current password in the request body for confirmation. + """ + user = _require_user(bincio_session) + body = await request.json() + password = body.get("password", "") + if not authenticate(_get_db(), user.handle, password): + raise HTTPException(401, "Wrong password") + + user_dir = _get_data_dir() / user.handle + deleted = _wipe_user_activities(user_dir) + _trigger_rebuild(user.handle) + return JSONResponse({"ok": True, "deleted": deleted}) + + +@app.delete("/api/me") +async def me_delete_account( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Delete the account and all data permanently. + + Requires the user's current password. Deletes the DB row, all sessions, + and the entire user data directory. The root shard manifest is updated. + """ + user = _require_user(bincio_session) + body = await request.json() + password = body.get("password", "") + if not authenticate(_get_db(), user.handle, password): + raise HTTPException(401, "Wrong password") + + # Wipe data directory + user_dir = _get_data_dir() / user.handle + if user_dir.is_dir(): + shutil.rmtree(user_dir) + + # Remove from DB (cascades to sessions, invites, reset_codes) + from bincio.serve.db import delete_user as _delete_user + _delete_user(_get_db(), user.handle) + + # Update root manifest so the shard disappears + from bincio.render.cli import _write_root_manifest + try: + _write_root_manifest(_get_data_dir()) + except Exception: + pass + + resp = JSONResponse({"ok": True}) + resp.delete_cookie(_SESSION_COOKIE) + return resp + + +@app.put("/api/me/display-name") +async def me_update_display_name( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Update the logged-in user's display name.""" + user = _require_user(bincio_session) + body = await request.json() + display_name = str(body.get("display_name", "")).strip() + if len(display_name) > 60: + raise HTTPException(400, "Display name too long (max 60 characters)") + db = _get_db() + db.execute("UPDATE users SET display_name = ? WHERE handle = ?", (display_name, user.handle)) + db.commit() + return JSONResponse({"ok": True, "display_name": display_name}) + + +@app.get("/api/me/prefs") +async def me_get_prefs(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Return all user preferences as a key→value dict.""" + user = _require_user(bincio_session) + return JSONResponse(get_user_prefs(_get_db(), user.handle)) + + +@app.put("/api/me/prefs") +async def me_set_prefs( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Upsert one or more user preferences. Body: {key: value, ...} (all strings).""" + user = _require_user(bincio_session) + body = await request.json() + if not isinstance(body, dict): + raise HTTPException(400, "Body must be a JSON object") + # Coerce all values to strings; ignore unknown keys silently + prefs = {str(k): str(v) for k, v in body.items()} + set_user_prefs(_get_db(), user.handle, prefs) + return JSONResponse({"ok": True}) + + +@app.get("/api/me/strava-credentials") +async def me_get_strava_credentials(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Return whether per-user Strava credentials are configured (never returns the secret).""" + user = _require_user(bincio_session) + creds_path = _get_data_dir() / user.handle / _STRAVA_CREDS_FILE + has_user_creds = False + client_id_hint = "" + if creds_path.exists(): + try: + d = json.loads(creds_path.read_text(encoding="utf-8")) + cid = str(d.get("client_id", "")).strip() + csec = str(d.get("client_secret", "")).strip() + if cid and csec: + has_user_creds = True + client_id_hint = cid + except Exception: + pass + return JSONResponse({ + "has_user_creds": has_user_creds, + "client_id": client_id_hint, + "instance_configured": bool(strava_client_id), + }) + + +@app.put("/api/me/strava-credentials") +async def me_set_strava_credentials( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Save per-user Strava credentials. Body: {client_id, client_secret}.""" + user = _require_user(bincio_session) + body = await request.json() + cid = str(body.get("client_id", "")).strip() + csec = str(body.get("client_secret", "")).strip() + if not cid: + raise HTTPException(400, "client_id is required") + creds_path = _get_data_dir() / user.handle / _STRAVA_CREDS_FILE + # If client_secret is omitted, preserve existing secret (if any) + if not csec: + if creds_path.exists(): + try: + existing = json.loads(creds_path.read_text(encoding="utf-8")) + csec = str(existing.get("client_secret", "")).strip() + except Exception: + pass + if not csec: + raise HTTPException(400, "client_secret is required (no existing secret to preserve)") + creds_path.write_text( + json.dumps({"client_id": cid, "client_secret": csec}, indent=2), + encoding="utf-8", + ) + return JSONResponse({"ok": True}) + + +@app.delete("/api/me/strava-credentials") +async def me_delete_strava_credentials(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Remove per-user Strava credentials (falls back to instance credentials).""" + user = _require_user(bincio_session) + creds_path = _get_data_dir() / user.handle / _STRAVA_CREDS_FILE + creds_path.unlink(missing_ok=True) + return JSONResponse({"ok": True}) + + +@app.put("/api/me/password") +async def me_change_password( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Change the logged-in user's password. Requires current password.""" + from bincio.serve.db import change_password as _change_password + user = _require_user(bincio_session) + body = await request.json() + current = body.get("current_password", "") + new_pw = body.get("new_password", "") + if not authenticate(_get_db(), user.handle, current): + raise HTTPException(401, "Current password is wrong") + if len(new_pw) < 8: + raise HTTPException(400, "New password must be at least 8 characters") + _change_password(_get_db(), user.handle, new_pw) + return JSONResponse({"ok": True}) + + # ── Write API (ported from bincio edit, auth-gated) ─────────────────────────── def _user_data_dir(handle: str) -> Path: @@ -312,13 +1251,17 @@ async def get_activity( user = _require_user(bincio_session) _check_id(activity_id) path = _require_owns(activity_id, user) - return JSONResponse(json.loads(path.read_text())) + detail = json.loads(path.read_text()) + # Normalise for EditDrawer: add `private` bool so the drawer works regardless + # of whether the raw JSON uses the old "private" or the new "unlisted" value. + detail["private"] = detail.get("privacy") in ("private", "unlisted") + return JSONResponse(detail) -@app.post("/api/activity/{activity_id}") +@app.post("/api/activity/{activity_id}", response_model=ActivityEditResponse) async def post_activity( activity_id: str, - request: Request, + edit_req: ActivityEditRequest, bincio_session: Optional[str] = Cookie(default=None), ) -> JSONResponse: user = _require_user(bincio_session) @@ -329,9 +1272,123 @@ async def post_activity( raise HTTPException(404, "Activity not found") from bincio.edit.ops import apply_sidecar_edit - body = await request.json() + body = edit_req.model_dump(exclude_none=True) + # apply_sidecar_edit already calls merge_one internally — no full rebuild needed. apply_sidecar_edit(activity_id, body, dd) + return JSONResponse({"ok": True}) + + +@app.post("/api/activity/{activity_id}/recalculate-elevation/dem") +async def recalculate_elevation_dem_endpoint( + activity_id: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Replace GPS altitude with DEM terrain elevation and recompute gain/loss. + + Requires --dem-url to be set when starting bincio serve. + """ + user = _require_user(bincio_session) + _check_id(activity_id) + if not dem_url: + raise HTTPException(503, "DEM URL not configured.") + dd = _get_data_dir() / user.handle + if not (dd / "activities" / f"{activity_id}.json").exists(): + raise HTTPException(404, "Activity not found") + try: + from bincio.extract.dem import recalculate_elevation + from bincio.render.merge import merge_one + result = recalculate_elevation(dd, activity_id, dem_url) + merge_one(dd, activity_id) + _trigger_rebuild(user.handle) + return JSONResponse(result) + except FileNotFoundError as e: + raise HTTPException(404, str(e)) + except ValueError as e: + raise HTTPException(422, str(e)) + + +@app.post("/api/activity/{activity_id}/recalculate-elevation/hysteresis") +async def recalculate_elevation_hysteresis_endpoint( + activity_id: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Recompute gain/loss from original recorded elevation using source-aware hysteresis.""" + user = _require_user(bincio_session) + _check_id(activity_id) + dd = _get_data_dir() / user.handle + if not (dd / "activities" / f"{activity_id}.json").exists(): + raise HTTPException(404, "Activity not found") + try: + from bincio.extract.dem import recalculate_elevation_hysteresis + from bincio.render.merge import merge_one + result = recalculate_elevation_hysteresis(dd, activity_id) + merge_one(dd, activity_id) + _trigger_rebuild(user.handle) + return JSONResponse(result) + except FileNotFoundError as e: + raise HTTPException(404, str(e)) + except ValueError as e: + raise HTTPException(422, str(e)) + + +@app.delete("/api/activity/{activity_id}", response_model=GenericResponse) +async def delete_activity( + activity_id: str, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Delete a single activity and all associated files for the logged-in user.""" + user = _require_user(bincio_session) + _check_id(activity_id) + dd = _get_data_dir() / user.handle + acts_dir = dd / "activities" + + json_path = acts_dir / f"{activity_id}.json" + if not json_path.exists(): + raise HTTPException(404, "Activity not found") + + import shutil + + # Remove the source files (activities dir) + for suffix in (".json", ".geojson", ".timeseries.json"): + p = acts_dir / f"{activity_id}{suffix}" + p.unlink(missing_ok=True) + + # Remove sidecar edit and images + sidecar = dd / "edits" / f"{activity_id}.md" + sidecar.unlink(missing_ok=True) + images_dir = dd / "edits" / "images" / activity_id + if images_dir.exists(): + shutil.rmtree(images_dir) + + # Remove from the extract-level flat index so merge_all doesn't re-add + # the summary even though the detail file is gone. + index_path = dd / "index.json" + if index_path.exists(): + try: + idx = json.loads(index_path.read_text(encoding="utf-8")) + idx["activities"] = [a for a in idx.get("activities", []) if a.get("id") != activity_id] + index_path.write_text(json.dumps(idx, indent=2, ensure_ascii=False)) + except Exception: + pass # corrupt index — merge_all will clean up on next run + + # Remove from dedup cache so the file can be re-uploaded if needed + cache_path = dd / ".bincio_cache.json" + if cache_path.exists(): + try: + cache = json.loads(cache_path.read_text(encoding="utf-8")) + if isinstance(cache, dict) and "activities" in cache: + cache["activities"] = [ + a for a in cache["activities"] if a.get("id") != activity_id + ] + cache_path.write_text(json.dumps(cache, indent=2, ensure_ascii=False)) + except Exception: + pass # corrupt cache — leave it; next extract will rebuild + + # Full merge needed: activity removed from index + from bincio.render.merge import merge_all + merge_all(dd) _trigger_rebuild(user.handle) + return JSONResponse({"ok": True}) @@ -362,13 +1419,17 @@ async def upload_image( if not file.filename: raise HTTPException(400, "No filename") ct = file.content_type or "" - if not ct.startswith("image/"): - raise HTTPException(400, f"Only image files accepted (got {ct})") + if ct not in _ALLOWED_IMAGE_TYPES: + raise HTTPException(400, "Only JPEG, PNG, WebP, or GIF images are accepted") + contents = await file.read() + if len(contents) > _MAX_IMAGE_BYTES: + raise HTTPException(413, f"Image too large (max {_MAX_IMAGE_BYTES // (1024*1024)} MB)") images_dir = dd / "edits" / "images" / activity_id images_dir.mkdir(parents=True, exist_ok=True) - safe_name = Path(file.filename).name - (images_dir / safe_name).write_bytes(await file.read()) - _trigger_rebuild(user.handle) + safe_name = _unique_image_name(images_dir, Path(file.filename).name) + (images_dir / safe_name).write_bytes(contents) + from bincio.render.merge import merge_one + merge_one(dd, activity_id) return JSONResponse({"ok": True, "filename": safe_name}) @@ -388,7 +1449,8 @@ async def delete_image( target.unlink() if target.parent.exists() and not any(target.parent.iterdir()): shutil.rmtree(target.parent) - _trigger_rebuild(user.handle) + from bincio.render.merge import merge_one + merge_one(dd, activity_id) return JSONResponse({"ok": True}) @@ -397,9 +1459,9 @@ async def get_athlete(bincio_session: Optional[str] = Cookie(default=None)) -> J user = _require_user(bincio_session) dd = _get_data_dir() / user.handle athlete_path = dd / "athlete.json" - if not athlete_path.exists(): - raise HTTPException(404, "athlete.json not found — run bincio extract first") - data = json.loads(athlete_path.read_text(encoding="utf-8")) + data: dict = {} + if athlete_path.exists(): + data = json.loads(athlete_path.read_text(encoding="utf-8")) # Layer edits/athlete.yaml on top edits_path = dd / "edits" / "athlete.yaml" if edits_path.exists(): @@ -411,14 +1473,7 @@ async def get_athlete(bincio_session: Optional[str] = Cookie(default=None)) -> J data[k] = edits[k] except Exception: pass - return JSONResponse({ - "max_hr": data.get("max_hr"), - "ftp_w": data.get("ftp_w"), - "hr_zones": data.get("hr_zones"), - "power_zones": data.get("power_zones"), - "seasons": data.get("seasons", []), - "gear": data.get("gear", {}), - }) + return JSONResponse(data) @app.post("/api/athlete") @@ -428,8 +1483,14 @@ async def save_athlete( ) -> JSONResponse: user = _require_user(bincio_session) dd = _get_data_dir() / user.handle - if not (dd / "athlete.json").exists(): - raise HTTPException(404, "athlete.json not found — run bincio extract first") + athlete_path = dd / "athlete.json" + if not athlete_path.exists(): + from datetime import datetime, timezone + athlete_path.write_text(json.dumps({ + "bas_version": "1.0", + "generated_at": datetime.now(timezone.utc).isoformat(), + "power_curve": {}, + }), encoding="utf-8") payload = await request.json() edits_dir = dd / "edits" edits_dir.mkdir(exist_ok=True) @@ -463,57 +1524,585 @@ async def save_athlete( _SUPPORTED_SUFFIXES = {".fit", ".gpx", ".tcx", ".fit.gz", ".gpx.gz", ".tcx.gz"} +def _file_suffix(name: str) -> str: + """Return the effective suffix, including .gz double-extension.""" + p = Path(name.lower()) + if p.suffix == ".gz": + return p.stem.rsplit(".", 1)[-1].join([".", ".gz"]) if "." in p.stem else ".gz" + return p.suffix + + @app.post("/api/upload") async def upload_activity( + files: list[UploadFile] = File(...), + store_original: bool = Form(False), + overwrite: bool = Form(False), + bincio_session: Optional[str] = Cookie(default=None), +) -> StreamingResponse: + """Accept FIT/GPX/TCX files and/or activities.csv; stream SSE progress while processing. + + activities.csv (Strava export format) can be included in the batch to: + - Enrich activity files in the same batch (matched by filename) + - Retroactively update sidecars for existing activities (matched by strava_id) + + SSE events: + {"type": "progress", "n": N, "total": T, "name": "...", "status": "imported"|"overwritten"|"duplicate"|"error"} + {"type": "csv", "updates": N} -- only when CSV was included + {"type": "done", "added": N, "csv_updates": N, "duplicates": N, "overwritten": N, "errors": N} + """ + from bincio.extract.ingest import ingest_parsed + from bincio.extract.parsers.factory import parse_file + from bincio.extract.writer import make_activity_id + from bincio.render.merge import merge_all + + user = _require_user(bincio_session) + dd = _get_data_dir() / user.handle + staging = dd / "_uploads" + staging.mkdir(exist_ok=True) + + # Read all files into memory now (async), then process synchronously in the generator + csv_bytes_list: list[bytes] = [] + activity_items: list[tuple[str, bytes]] = [] # (original_filename, bytes) + + for f in files: + fname = Path(f.filename or "").name + raw = await f.read() + if fname.lower().endswith(".csv"): + csv_bytes_list.append(raw) + else: + activity_items.append((fname, raw)) + + # Build metadata from the first CSV + metadata = None + if csv_bytes_list: + from bincio.extract.strava_csv import StravaMetadata + import tempfile + with tempfile.NamedTemporaryFile(suffix=".csv", delete=False) as tmp: + tmp.write(csv_bytes_list[0]) + tmp_path = Path(tmp.name) + try: + metadata = StravaMetadata(tmp_path) + finally: + tmp_path.unlink(missing_ok=True) + + total_files = len(activity_items) + job_id = _job_start(user.handle, total_files) if total_files > 0 else None + + def event_stream(): + added = 0 + overwritten = 0 + duplicates = 0 + errors = 0 + any_added = False + + for n, (name, contents) in enumerate(activity_items, 1): + if job_id: + _job_update(job_id, n - 1, name) + + suffix = _file_suffix(name) + if suffix not in _SUPPORTED_SUFFIXES: + errors += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'error', 'detail': 'unsupported type'})}\n\n" + continue + + if len(contents) > 50 * 1024 * 1024: + errors += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'error', 'detail': 'file too large'})}\n\n" + continue + + staged = staging / name + staged.write_bytes(contents) + kept = False + try: + activity = parse_file(staged) + if metadata is not None: + metadata.enrich(name, activity) + activity_id = make_activity_id(activity) + was_overwrite = False + if (dd / "activities" / f"{activity_id}.json").exists(): + if not overwrite: + duplicates += 1 + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'duplicate'})}\n\n" + continue + # Overwrite: delete existing files before re-ingesting. + for ext in (".json", ".geojson", ".timeseries.json"): + (dd / "activities" / f"{activity_id}{ext}").unlink(missing_ok=True) + # Remove stale summary from index so ingest_parsed writes a clean one + index_path = dd / "index.json" + if index_path.exists(): + idx = json.loads(index_path.read_text(encoding="utf-8")) + idx["activities"] = [a for a in idx.get("activities", []) if a.get("id") != activity_id] + index_path.write_text(json.dumps(idx, indent=2, ensure_ascii=False)) + # Remove from dedup hash cache so the new file isn't blocked + cache_path = dd / ".bincio_cache.json" + if cache_path.exists(): + try: + cache = json.loads(cache_path.read_text(encoding="utf-8")) + cache.pop(activity_id, None) + cache_path.write_text(json.dumps(cache, ensure_ascii=False)) + except Exception: + pass + # Remove merged copies (merge_all will regenerate them after ingest) + merged_acts = dd / "_merged" / "activities" + if merged_acts.exists(): + for ext in (".json", ".geojson", ".timeseries.json"): + p = merged_acts / f"{activity_id}{ext}" + if p.exists() or p.is_symlink(): + p.unlink(missing_ok=True) + was_overwrite = True + ingest_parsed(activity, dd, privacy="public") + if store_original: + originals_dir = dd / "originals" + originals_dir.mkdir(exist_ok=True) + staged.rename(originals_dir / name) + kept = True + if was_overwrite: + overwritten += 1 + else: + added += 1 + any_added = True + status = 'overwritten' if was_overwrite else 'imported' + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': status})}\n\n" + except Exception as exc: + errors += 1 + log.error("upload[%s]: failed to process %s: %s", user.handle, name, exc, exc_info=True) + yield f"data: {json.dumps({'type': 'progress', 'n': n, 'total': total_files, 'name': name, 'status': 'error', 'detail': str(exc)})}\n\n" + finally: + if not kept: + staged.unlink(missing_ok=True) + + # Retroactively apply CSV metadata to existing activities + csv_updates = 0 + if metadata is not None: + from bincio.extract.strava_csv import apply_csv_to_data_dir + csv_updates = apply_csv_to_data_dir(dd, metadata) + if csv_updates: + yield f"data: {json.dumps({'type': 'csv', 'updates': csv_updates})}\n\n" + + if any_added or csv_updates: + merge_all(dd) + if any_added: + _trigger_rebuild(user.handle) + + yield f"data: {json.dumps({'type': 'done', 'added': added, 'overwritten': overwritten, 'csv_updates': csv_updates, 'duplicates': duplicates, 'errors': errors})}\n\n" + + if job_id: + _job_finish(job_id) + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) + + +@app.post("/api/upload/strava-zip") +async def upload_strava_zip( file: UploadFile = File(...), + private: str = Form(default="false"), + bincio_session: Optional[str] = Cookie(default=None), +) -> StreamingResponse: + """Accept a Strava bulk export ZIP and stream SSE progress while processing. + + The ZIP is written to a temp file, processed activity-by-activity, then deleted. + Originals are never kept — the UI informs the user of this upfront. + """ + user = _require_user(bincio_session) + if not file.filename or not file.filename.lower().endswith(".zip"): + raise HTTPException(400, "Please upload a .zip file") + + privacy = "unlisted" if private.lower() in ("true", "1", "yes") else "public" + + dd = _get_data_dir() / user.handle + import tempfile + tmp = tempfile.NamedTemporaryFile(suffix=".zip", delete=False) + zip_path = Path(tmp.name) + try: + while chunk := await file.read(1024 * 1024): # 1 MB chunks + tmp.write(chunk) + finally: + tmp.close() + + from bincio.extract.strava_zip import strava_zip_iter + from bincio.render.merge import merge_all + + log.info("strava-zip[%s]: received %s, privacy=%s", user.handle, file.filename, privacy) + + def event_stream(): + any_imported = False + imported_count = 0 + error_count = 0 + try: + for event in strava_zip_iter(zip_path, dd, privacy=privacy): + yield f"data: {json.dumps(event)}\n\n" + if event.get("type") == "progress": + status = event.get("status") + if status == "imported": + any_imported = True + imported_count += 1 + elif status == "error": + error_count += 1 + log.warning("strava-zip[%s]: error on %s: %s", + user.handle, event.get("name"), event.get("detail", "")) + if event.get("type") == "done": + log.info("strava-zip[%s]: done — imported=%d errors=%d", + user.handle, imported_count, error_count) + if any_imported: + merge_all(dd) + _trigger_rebuild(user.handle) + except Exception as exc: + log.error("strava-zip[%s]: fatal error: %s", user.handle, exc, exc_info=True) + yield f"data: {json.dumps({'type': 'error', 'message': str(exc)})}\n\n" + finally: + zip_path.unlink(missing_ok=True) + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) + + +# ── Feedback ────────────────────────────────────────────────────────────────── + +_FEEDBACK_IMAGE_SUFFIXES = {".jpg", ".jpeg", ".png", ".gif", ".webp", ".heic"} +_FEEDBACK_MAX_IMAGES = 3 +_FEEDBACK_MAX_IMAGE_BYTES = 2 * 1024 * 1024 # 2 MB + + +@app.post("/api/feedback") +async def submit_feedback( + text: str = Form(""), + images: list[UploadFile] = File(default=[]), bincio_session: Optional[str] = Cookie(default=None), ) -> JSONResponse: user = _require_user(bincio_session) + + text = text.strip() + if not text and not any(f.filename for f in images): + raise HTTPException(400, "Feedback must include text or at least one image") + if len(images) > _FEEDBACK_MAX_IMAGES: + raise HTTPException(400, f"Maximum {_FEEDBACK_MAX_IMAGES} images per submission") + + feedback_dir = _get_data_dir() / "_feedback" + feedback_dir.mkdir(exist_ok=True) + images_dir = feedback_dir / user.handle + images_dir.mkdir(exist_ok=True) + + now = int(time.time()) + submission_id = f"{now}_{secrets.token_hex(4)}" + saved_images: list[str] = [] + + for img in images: + if not img.filename: + continue + suffix = Path(img.filename).suffix.lower() + if suffix not in _FEEDBACK_IMAGE_SUFFIXES: + raise HTTPException(400, f"Unsupported image type '{suffix}'") + contents = await img.read() + if len(contents) > _FEEDBACK_MAX_IMAGE_BYTES: + raise HTTPException(413, f"Image '{img.filename}' exceeds 2 MB limit") + safe_name = f"{submission_id}_{Path(img.filename).name}" + (images_dir / safe_name).write_bytes(contents) + saved_images.append(safe_name) + + from datetime import datetime, timezone + entry = { + "id": submission_id, + "handle": user.handle, + "submitted_at": datetime.now(timezone.utc).isoformat(), + "text": text, + "images": saved_images, + } + + log_file = feedback_dir / f"{user.handle}.json" + existing: list[dict] = [] + if log_file.exists(): + try: + existing = json.loads(log_file.read_text()) + except Exception: + existing = [] + existing.append(entry) + log_file.write_text(json.dumps(existing, indent=2)) + + return JSONResponse({"ok": True, "id": submission_id}) + + +# ── Strava ──────────────────────────────────────────────────────────────────── + +_strava_oauth_states: set[str] = set() + + +@app.get("/api/strava/status") +async def strava_status(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + user = _require_user(bincio_session) + cid, _ = _strava_creds(user.handle) + if not cid: + return JSONResponse({"configured": False, "connected": False, "last_sync": None}) dd = _get_data_dir() / user.handle - name = Path(file.filename or "upload.fit").name - p = Path(name.lower()) - suffix = (p.stem.rsplit(".", 1)[-1].join([".", ".gz"]) if "." in p.stem else ".gz") if p.suffix == ".gz" else p.suffix - if suffix not in _SUPPORTED_SUFFIXES: - raise HTTPException(400, f"Unsupported file type '{suffix}'") - contents = await file.read() - if len(contents) > 50 * 1024 * 1024: - raise HTTPException(413, "File too large (max 50 MB)") - staging = dd / "_uploads" - staging.mkdir(exist_ok=True) - staged = staging / name - staged.write_bytes(contents) + from bincio.extract.strava_api import load_token + token = load_token(dd) + return JSONResponse({ + "configured": True, + "connected": token is not None, + "last_sync": token.get("last_sync_at") if token else None, + }) + + +@app.post("/api/strava/reset") +async def strava_reset(request: Request, bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Reset last_sync_at so the next sync re-fetches from a chosen point. + + mode=soft — set to the started_at of the most recent activity on disk + (next sync only fetches activities newer than the last known one) + mode=hard — clear last_sync_at entirely + (next sync re-downloads full Strava history, skipping existing files) + """ + user = _require_user(bincio_session) + dd = _get_data_dir() / user.handle + from bincio.extract.strava_api import load_token, save_token + token = load_token(dd) + if token is None: + raise HTTPException(400, "Not connected to Strava") + + body = await request.json() + mode = body.get("mode", "soft") + + if mode == "hard": + token.pop("last_sync_at", None) + save_token(dd, token) + return JSONResponse({"ok": True, "mode": "hard", "last_sync_at": None}) + + # soft: find the most recent started_at across the user's merged index + from datetime import datetime, timezone + last_ts: int | None = None + for index_path in [dd / "_merged" / "index.json", dd / "index.json"]: + if not index_path.exists(): + continue + try: + index_data = json.loads(index_path.read_text(encoding="utf-8")) + started_ats = [ + a.get("started_at") for a in index_data.get("activities", []) + if a.get("started_at") + ] + if started_ats: + latest = max(started_ats) + dt = datetime.fromisoformat(latest.replace("Z", "+00:00")) + last_ts = int(dt.astimezone(timezone.utc).timestamp()) + break + except Exception: + continue + + if last_ts is None: + token.pop("last_sync_at", None) + else: + token["last_sync_at"] = last_ts + save_token(dd, token) + return JSONResponse({"ok": True, "mode": "soft", "last_sync_at": last_ts}) + + +@app.get("/api/strava/auth-url") +async def strava_auth_url(request: Request, bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + user = _require_user(bincio_session) + cid, _ = _strava_creds(user.handle) + if not cid: + raise HTTPException(400, "Strava client ID not configured on this server") + state = secrets.token_urlsafe(16) + _strava_oauth_states.add(state) + if public_url: + redirect_uri = public_url.rstrip("/") + "/api/strava/callback" + else: + redirect_uri = str(request.url_for("strava_callback")) + from bincio.extract.strava_api import auth_url + return JSONResponse({"url": auth_url(cid, redirect_uri, state=state)}) + + +@app.get("/api/strava/callback", name="strava_callback") +async def strava_callback( + request: Request, + code: str = "", + error: str = "", + state: str = "", + bincio_session: Optional[str] = Cookie(default=None), +) -> RedirectResponse: + site_origin = public_url.rstrip("/") if public_url else str(request.base_url).rstrip("/") + if error or not code: + return RedirectResponse(f"{site_origin}/?strava=error") + if state not in _strava_oauth_states: + return RedirectResponse(f"{site_origin}/?strava=error") + _strava_oauth_states.discard(state) + user = _current_user(bincio_session) + if not user: + return RedirectResponse(f"{site_origin}/?strava=error") + cid, csec = _strava_creds(user.handle) + if not cid or not csec: + return RedirectResponse(f"{site_origin}/?strava=error") + dd = _get_data_dir() / user.handle + from bincio.extract.strava_api import StravaError, exchange_code, save_token try: - from bincio.extract.ingest import ingest_parsed - from bincio.extract.parsers.factory import parse_file - activity = parse_file(staged) - activity_id_check = dd / "activities" / f"{activity.source_file}.json" - from bincio.extract.writer import make_activity_id - activity_id = make_activity_id(activity) - if (dd / "activities" / f"{activity_id}.json").exists(): - raise HTTPException(409, f"Activity already exists: {activity_id}") - ingest_parsed(activity, dd, privacy="public") - from bincio.render.merge import merge_all - merge_all(dd) - _trigger_rebuild(user.handle) - except HTTPException: - raise - except Exception as exc: - raise HTTPException(422, f"Failed to process activity: {type(exc).__name__}: {exc}") - finally: - staged.unlink(missing_ok=True) - return JSONResponse({"ok": True, "id": activity_id}) + token = exchange_code(cid, csec, code) + except StravaError: + return RedirectResponse(f"{site_origin}/?strava=error") + save_token(dd, token) + return RedirectResponse(f"{site_origin}/?strava=connected") + + +@app.get("/api/strava/sync/stream") +async def serve_strava_sync_stream(bincio_session: Optional[str] = Cookie(default=None)) -> StreamingResponse: + """SSE endpoint — streams per-activity progress then a final summary event.""" + user = _require_user(bincio_session) + cid, csec = _strava_creds(user.handle) + if not cid or not csec: + raise HTTPException(400, "Strava not configured on this server") + dd = _get_data_dir() / user.handle + store_orig_setting = get_setting(_get_db(), "store_originals") + store_orig = store_orig_setting == "true" + originals_dir = (dd / "originals" / "strava") if store_orig else None + if originals_dir: + originals_dir.mkdir(parents=True, exist_ok=True) + + from bincio.extract.ingest import strava_sync_iter + + def event_stream(): + try: + for event in strava_sync_iter(dd, cid, csec, originals_dir): + if event["type"] == "done": + _trigger_rebuild(user.handle) # start before client closes connection + yield f"data: {json.dumps(event)}\n\n" + except Exception as exc: + yield f"data: {json.dumps({'type': 'error', 'message': str(exc)})}\n\n" + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) @app.post("/api/strava/sync") async def serve_strava_sync(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: user = _require_user(bincio_session) - if not strava_client_id or not strava_client_secret: + cid, csec = _strava_creds(user.handle) + if not cid or not csec: raise HTTPException(400, "Strava not configured on this server") dd = _get_data_dir() / user.handle + store_orig_setting = get_setting(_get_db(), "store_originals") + store_orig = store_orig_setting == "true" + originals_dir = (dd / "originals" / "strava") if store_orig else None + if originals_dir: + originals_dir.mkdir(parents=True, exist_ok=True) from bincio.edit.ops import run_strava_sync try: - result = run_strava_sync(dd, strava_client_id, strava_client_secret) + result = run_strava_sync(dd, cid, csec, originals_dir=originals_dir) except RuntimeError as e: raise HTTPException(502, str(e)) _trigger_rebuild(user.handle) return JSONResponse(result) + + +# ── Garmin Connect endpoints ────────────────────────────────────────────────── + +def _garmin_user_message(exc: Exception) -> str: + """Return a human-friendly error message for common Garmin login failures.""" + msg = str(exc) + fallback = ( + " In the meantime, you can export your activities from Garmin Connect " + "(garmin.com → Activities → Export) or Garmin Express as FIT files " + "and upload them directly." + ) + if "429" in msg or "rate limit" in msg.lower(): + return ( + "Garmin is rate-limiting this server's IP address (HTTP 429). " + "Wait a few hours and try again." + fallback + ) + if "403" in msg: + return ( + "Cloudflare is blocking the login request (HTTP 403). " + "This is a known upstream issue — try again later or update garminconnect " + "(uv sync --extra garmin)." + fallback + ) + if "GARMIN Authentication Application" in msg or "unexpected title" in msg.lower(): + return ( + "Garmin's login page returned a CAPTCHA or MFA challenge that " + "cannot be completed automatically. Try again later, or disable " + "two-factor authentication on your Garmin account." + fallback + ) + return f"Login failed: {exc}" + fallback + +@app.get("/api/garmin/status") +async def garmin_status(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Return whether Garmin credentials are stored for the current user.""" + user = _require_user(bincio_session) + dd = _get_data_dir() / user.handle + from bincio.extract.garmin_api import has_credentials + from bincio.extract.garmin_sync import _load_sync_state + connected = has_credentials(dd) + last_sync = None + if connected: + state = _load_sync_state(dd) + last_sync = state.get("last_sync_at") + return JSONResponse({"connected": connected, "last_sync": last_sync}) + + +@app.post("/api/garmin/connect") +async def garmin_connect( + request: Request, + bincio_session: Optional[str] = Cookie(default=None), +) -> JSONResponse: + """Test Garmin login with the supplied credentials and save them on success.""" + user = _require_user(bincio_session) + body = await request.json() + email = (body.get("email") or "").strip() + password = body.get("password") or "" + if not email or not password: + raise HTTPException(400, "email and password are required") + + data_dir = _get_data_dir() + user_dir = data_dir / user.handle + from bincio.extract.garmin_api import GarminError, test_login + try: + info = test_login(data_dir, user_dir, email, password) + except GarminError as exc: + raise HTTPException(400, _garmin_user_message(exc)) + return JSONResponse({"ok": True, **info}) + + +@app.post("/api/garmin/disconnect") +async def garmin_disconnect(bincio_session: Optional[str] = Cookie(default=None)) -> JSONResponse: + """Remove stored Garmin credentials and session for the current user.""" + user = _require_user(bincio_session) + dd = _get_data_dir() / user.handle + from bincio.extract.garmin_api import delete_credentials + delete_credentials(dd) + return JSONResponse({"ok": True}) + + +@app.get("/api/garmin/sync/stream") +async def garmin_sync_stream(bincio_session: Optional[str] = Cookie(default=None)) -> StreamingResponse: + """SSE endpoint — streams per-activity Garmin sync progress.""" + user = _require_user(bincio_session) + data_dir = _get_data_dir() + user_dir = data_dir / user.handle + + from bincio.extract.garmin_api import GarminError, has_credentials + if not has_credentials(user_dir): + raise HTTPException(400, "No Garmin credentials stored — connect first") + + from bincio.extract.garmin_sync import garmin_sync_iter + + def event_stream(): + try: + for event in garmin_sync_iter(data_dir, user_dir): + if event["type"] == "done": + _trigger_rebuild(user.handle) + yield f"data: {json.dumps(event)}\n\n" + except GarminError as exc: + yield f"data: {json.dumps({'type': 'error', 'message': _garmin_user_message(exc)})}\n\n" + except Exception as exc: + yield f"data: {json.dumps({'type': 'error', 'message': _garmin_user_message(exc)})}\n\n" + + return StreamingResponse( + event_stream(), + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) diff --git a/docs/admin-guide.md b/docs/admin-guide.md new file mode 100644 index 0000000..f72588d --- /dev/null +++ b/docs/admin-guide.md @@ -0,0 +1,306 @@ +# Administrator Guide + +This guide covers everything needed to deploy and maintain a multi-user BincioActivity instance. + +## Before You Start + +**[Multi-user Deployment](deployment/multi-user.md)** has the complete step-by-step instructions. This guide focuses on day-to-day admin tasks once the instance is running. + +## Initializing an Instance + +```bash +uv sync --extra serve + +uv run bincio init \ + --data-dir /var/bincio \ + --handle your_admin_handle \ + --display-name "Your Name" \ + --name "Instance Name" +``` + +You'll be prompted for a password. This creates: + +- `/var/bincio/instance.db` — SQLite database (users, sessions, invites, reset codes) +- `/var/bincio/index.json` — root shard manifest (`"private": true` by default) +- Your admin user account +- A first invite code + +`bincio init` is idempotent — safe to re-run. + +Optional flags: + +- `--max-users N` — limit total registered users (0 or omitted = unlimited) +- `--store-originals false` — don't keep uploaded source files (defaults to true) + +## Inviting Users + +### Generate an invite code (as admin) + +From the web UI at `/invites/` (requires login as admin), or via CLI: + +```bash +uv run python -c " +from pathlib import Path +from bincio.serve.db import open_db, create_invite +db = open_db(Path('/var/bincio')) +code = create_invite(db, 'your_handle') +print(f'https://yourdomain.com/register/?code={code}') +" +``` + +### Invite limits + +- **Admins:** unlimited invites +- **Regular users:** 3 invites each (configurable in `bincio/serve/db.py` as `_MAX_USER_INVITES`) + +### Share the invite link + +Send the registration link to the user: + +``` +https://yourdomain.com/register/?code=ABCD1234 +``` + +They create their own handle and password. After registration, they can: +- Upload activity files (GPX, FIT, TCX) +- Sync from Strava +- Edit activity titles, descriptions, photos +- Control privacy per activity + +## Password Reset + +BincioActivity has no email system. Password resets work via **admin-generated one-time codes**. + +### Reset a user's password (as admin) + +1. Open `/admin/` in the web UI (must be logged in as admin) +2. Find the user and click **Reset password** +3. A code appears (monospace, click to copy) +4. Send the code out-of-band (Signal, Telegram, WhatsApp, etc.) + +The code is valid for **24 hours**. Users reset their password at `/reset-password/` by entering: + +- Their **handle** +- The **code** +- Their **new password** + +### Reset code API (CLI) + +To generate a reset code programmatically: + +```bash +uv run python -c " +from pathlib import Path +from bincio.serve.db import open_db, create_reset_code +db = open_db(Path('/var/bincio')) +code, expires_in_hours = create_reset_code(db, 'user_handle', 'your_handle') +print(f'Code: {code} (expires in {expires_in_hours} hours)') +" +``` + +## Monitoring Active Jobs + +The `/api/admin/jobs` endpoint (admin-only) shows which uploads/syncs are in progress: + +```bash +curl -b "bincio_session=$(cat /tmp/session.txt)" http://localhost:4041/api/admin/jobs +``` + +Returns: + +```json +[ + { + "id": "a1b2c3d4", + "user": "alice", + "started_at": 1712345678, + "total": 50, + "done": 23, + "current": "activity_2026-03-15_120000Z.fit" + } +] +``` + +## Triggering Rebuilds + +`bincio serve` can trigger incremental rebuilds when you pass `--site-dir`: + +```bash +uv run bincio serve \ + --data-dir /var/bincio \ + --site-dir /var/www/bincio/src/site +``` + +After any write operation (edit, upload, Strava sync), the affected user's shard is rebuilt automatically and the static site is updated. + +To manually rebuild a single user's shard: + +```bash +uv run bincio render \ + --data-dir /var/bincio \ + --handle alice +``` + +To rebuild everything (slow): + +```bash +uv run bincio render --data-dir /var/bincio +``` + +## Instance Settings + +Settings are stored in `instance.db` and control instance-wide behavior: + +| Setting | Default | Controls | +|---------|---------|----------| +| `max_users` | unlimited | Maximum registered users allowed | +| `store_originals` | `true` | Keep uploaded source files and Strava sync data | + +Read/set settings via CLI: + +```bash +uv run python -c " +from pathlib import Path +from bincio.serve.db import open_db, get_setting, set_setting +db = open_db(Path('/var/bincio')) +print(get_setting(db, 'max_users')) +set_setting(db, 'max_users', 100) +db.commit() +" +``` + +Or check the database directly: + +```bash +sqlite3 /var/bincio/instance.db +> SELECT key, value FROM settings; +``` + +## Instance Privacy + +By default, new instances are **private** — only authenticated users can view anything. Edit the root `index.json` to toggle: + +```json +{ + "private": false, + "shards": [...] +} +``` + +- **`"private": true`** — all pages (except login/register) require authentication +- **`"private": false`** — public access to all activities; individual activities can still be marked private via the `private` flag in sidecars + +After any change, run `bincio render` to apply it: + +```bash +uv run bincio render --data-dir /var/bincio +``` + +## Data Directory Layout + +``` +/var/bincio/ + instance.db ← SQLite: users, sessions, invites, reset codes + index.json ← root shard manifest + {handle}/ + index.json ← user's BAS feed (activities list) + _merged/ ← sidecar-merged output (served to browser) + activities/ ← extracted activity JSON files + {id}.json + ... + edits/ ← user-made sidecar edits + {id}.md + images/{id}/ + athlete.json ← profile (from Strava or manual) + strava_token.json ← OAuth token (if synced from Strava) + originals/ ← source files (if store_originals=true) + _feedback/ ← user feedback submissions + {handle}.json + {handle}/ + {timestamp}_{id}_{filename} +``` + +## Database Schema + +`instance.db` contains: + +- **`users`** — handle, password hash, display_name, is_admin, created_at +- **`sessions`** — session_id, handle, created_at, expires_at +- **`invites`** — code, created_by, created_at, used_by, used_at +- **`reset_codes`** — code, handle, created_by, created_at, expires_at, used_at +- **`settings`** — key, value (instance config) +- **`user_preferences`** — handle, key, value (per-user settings) + +Query the database directly: + +```bash +sqlite3 /var/bincio/instance.db ".tables" +sqlite3 /var/bincio/instance.db "SELECT handle, is_admin FROM users;" +``` + +## API Endpoints for Admins + +The `/api/admin/*` endpoints require authentication and admin privileges: + +- `GET /api/admin/users` — List all users +- `POST /api/admin/users/{handle}/reset-password-code` — Generate a reset code +- `GET /api/admin/jobs` — Show active uploads/syncs +- `GET /api/stats` — Community stats (public) + +See [API Reference](reference/api.md) for full details. + +### Explore the API with Swagger UI + +When `bincio serve` is running, visit `/api/docs` to see an interactive Swagger UI. You can: +- Browse all endpoints with their parameters and response types +- Try out requests directly (if you're logged in as admin) +- See live examples of request/response bodies + +ReDoc (another API documentation format) is also available at `/api/redoc` with a different UI. + +## Running as a systemd service + +See [Multi-user Deployment](deployment/multi-user.md#step-5--start-bincio-serve) for the systemd unit file. Key points: + +- Set `User=bincio` (unprivileged user) +- Set `WorkingDirectory` to the repo root +- Use `--site-dir` to enable incremental rebuilds +- Restart policy: `Restart=on-failure` + +Monitor with: + +```bash +systemctl status bincio +journalctl -u bincio -f +``` + +## Troubleshooting + +### Activities not appearing after upload + +1. Check if the job is still running: `GET /api/admin/jobs` +2. Check logs: `journalctl -u bincio -f` +3. If `store_originals=true`, verify the source file is readable in `{handle}/originals/` +4. Re-trigger the merge: `uv run bincio render --data-dir /var/bincio --handle alice` + +### Database locked + +If you see "database is locked": + +1. Verify no other `bincio` processes are running: `ps aux | grep bincio` +2. Kill any stuck processes: `pkill -f 'uv run bincio'` +3. Restart the service: `systemctl restart bincio` + +### High memory usage + +The first rebuild on a large instance can be memory-intensive. Consider: + +- Running `bincio render` during off-hours +- Rebuilding one user at a time: `uv run bincio render --data-dir /var/bincio --handle alice` +- Increasing swap or upgrading the machine + +## See also + +- [Multi-user Deployment](deployment/multi-user.md) — complete step-by-step setup +- [Single-user Deployment](deployment/single-user.md) — if you're hosting a read-only site +- [API Reference](reference/api.md) — all HTTP endpoints diff --git a/docs/architecture.md b/docs/architecture.md index d90a3e3..d8974ea 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -21,7 +21,7 @@ GPX / FIT / TCX files Any static host (GitHub Pages, Netlify, VPS, USB stick, …) ``` -The BAS data store is the contract between the two stages. Any tool in any language can produce BAS-compliant JSON. See [SCHEMA.md](../SCHEMA.md) for the format. +The BAS data store is the contract between the two stages. Any tool in any language can produce BAS-compliant JSON. See [SCHEMA.md](schema.md) for the format. --- diff --git a/docs/architecture.mmd b/docs/architecture.mmd new file mode 100644 index 0000000..4abca73 --- /dev/null +++ b/docs/architecture.mmd @@ -0,0 +1,274 @@ +graph LR + + subgraph API + subgraph api_activity["activity"] + api__api_activity__activity_id_["GET /api/activity/{activity_id}"] + api__api_activity__activity_id_["POST /api/activity/{activity_id}"] + api__api_activity__activity_id_["DELETE /api/activity/{activity_id}"] + api__api_activity__activity_id__images["GET /api/activity/{activity_id}/images"] + api__api_activity__activity_id__images["POST /api/activity/{activity_id}/images"] + api__api_activity__activity_id__images__filename_["DELETE /api/activity/{activity_id}/images/{filename}"] + end + subgraph api_admin["admin"] + api__api_admin_users["GET /api/admin/users"] + api__api_admin_jobs["GET /api/admin/jobs"] + api__api_admin_disk["GET /api/admin/disk"] + api__api_admin_users__handle__reset_password_code["POST /api/admin/users/{handle}/reset-password-code"] + api__api_admin_users__handle__rebuild["POST /api/admin/users/{handle}/rebuild"] + api__api_admin_users__handle__activities["DELETE /api/admin/users/{handle}/activities"] + end + subgraph api_athlete["athlete"] + api__api_athlete["GET /api/athlete"] + api__api_athlete["POST /api/athlete"] + end + subgraph api_auth["auth"] + api__api_auth_login["POST /api/auth/login"] + api__api_auth_logout["POST /api/auth/logout"] + api__api_auth_reset_password["POST /api/auth/reset-password"] + end + subgraph api_feedback["feedback"] + api__api_feedback["POST /api/feedback"] + end + subgraph api_garmin["garmin"] + api__api_garmin_status["GET /api/garmin/status"] + api__api_garmin_connect["POST /api/garmin/connect"] + api__api_garmin_disconnect["POST /api/garmin/disconnect"] + api__api_garmin_sync_stream["GET /api/garmin/sync/stream"] + end + subgraph api_invites["invites"] + api__api_invites["GET /api/invites"] + api__api_invites["POST /api/invites"] + end + subgraph api_me["me"] + api__api_me["GET /api/me"] + end + subgraph api_register["register"] + api__api_register["POST /api/register"] + end + subgraph api_stats["stats"] + api__api_stats["GET /api/stats"] + end + subgraph api_strava["strava"] + api__api_strava_status["GET /api/strava/status"] + api__api_strava_reset["POST /api/strava/reset"] + api__api_strava_auth_url["GET /api/strava/auth-url"] + api__api_strava_callback["GET /api/strava/callback"] + api__api_strava_sync_stream["GET /api/strava/sync/stream"] + api__api_strava_sync["POST /api/strava/sync"] + end + subgraph api_upload["upload"] + api__api_upload["POST /api/upload"] + api__api_upload_strava_zip["POST /api/upload/strava-zip"] + end + end + + subgraph Pages + site_src_pages_about_ca_index_astro["pages/about/ca/"] + site_src_pages_about_es_index_astro["pages/about/es/"] + site_src_pages_about_index_astro["pages/about/"] + site_src_pages_about_it_index_astro["pages/about/it/"] + site_src_pages_activity__id__astro["pages/activity/[id].astro"] + site_src_pages_activity_index_astro["pages/activity/"] + site_src_pages_activity_local_index_astro["pages/activity/local/"] + site_src_pages_admin_index_astro["pages/admin/"] + site_src_pages_athlete_index_astro["pages/athlete/"] + site_src_pages_community_index_astro["pages/community/"] + site_src_pages_convert_index_astro["pages/convert/"] + site_src_pages_feedback_index_astro["pages/feedback/"] + site_src_pages_index_astro["pages/"] + site_src_pages_invites_index_astro["pages/invites/"] + site_src_pages_login_index_astro["pages/login/"] + site_src_pages_record_index_astro["pages/record/"] + site_src_pages_register_index_astro["pages/register/"] + site_src_pages_reset_password_index_astro["pages/reset-password/"] + site_src_pages_stats_index_astro["pages/stats/"] + site_src_pages_u__handle__athlete_index_astro["pages/u/[handle]/athlete/"] + site_src_pages_u__handle__index_astro["pages/u/[handle]/"] + site_src_pages_u__handle__stats_index_astro["pages/u/[handle]/stats/"] + end + + subgraph Components + site_src_components_ActivityCharts_svelte["components/ActivityCharts.svelte"] + site_src_components_ActivityDetail_svelte["components/ActivityDetail.svelte"] + site_src_components_ActivityDetailLoader_svelte["components/ActivityDetailLoader.svelte"] + site_src_components_ActivityFeed_svelte["components/ActivityFeed.svelte"] + site_src_components_ActivityMap_svelte["components/ActivityMap.svelte"] + site_src_components_AthleteDrawer_svelte["components/AthleteDrawer.svelte"] + site_src_components_AthleteView_svelte["components/AthleteView.svelte"] + site_src_components_CommunityView_svelte["components/CommunityView.svelte"] + site_src_components_EditDrawer_svelte["components/EditDrawer.svelte"] + site_src_components_LocalActivityDetail_svelte["components/LocalActivityDetail.svelte"] + site_src_components_MmpChart_svelte["components/MmpChart.svelte"] + site_src_components_RecordsView_svelte["components/RecordsView.svelte"] + site_src_components_StatsView_svelte["components/StatsView.svelte"] + end + + subgraph Python + subgraph py_edit["edit"] + bincio_edit_cli_py["cli"] + bincio_edit_ops_py["ops"] + bincio_edit_server_py["server"] + end + subgraph py_extract["extract"] + bincio_extract_cli_py["cli"] + bincio_extract_config_py["config"] + bincio_extract_dedup_py["dedup"] + bincio_extract_garmin_api_py["garmin_api"] + bincio_extract_garmin_sync_py["garmin_sync"] + bincio_extract_ingest_py["ingest"] + bincio_extract_metrics_py["metrics"] + bincio_extract_models_py["models"] + bincio_extract_parsers_base_py["base"] + bincio_extract_parsers_factory_py["factory"] + bincio_extract_parsers_fit_py["fit"] + bincio_extract_parsers_gpx_py["gpx"] + bincio_extract_parsers_tcx_py["tcx"] + bincio_extract_simplify_py["simplify"] + bincio_extract_sport_py["sport"] + bincio_extract_strava_api_py["strava_api"] + bincio_extract_strava_csv_py["strava_csv"] + bincio_extract_strava_zip_py["strava_zip"] + bincio_extract_timeseries_py["timeseries"] + bincio_extract_writer_py["writer"] + end + subgraph py_import_["import_"] + bincio_import__cli_py["cli"] + bincio_import__strava_py["strava"] + end + subgraph py_render["render"] + bincio_render_cli_py["cli"] + bincio_render_merge_py["merge"] + end + subgraph py_root["root"] + bincio_cli_py["cli"] + bincio_dev_py["dev"] + end + subgraph py_serve["serve"] + bincio_serve_cli_py["cli"] + bincio_serve_db_py["db"] + bincio_serve_init_cmd_py["init_cmd"] + bincio_serve_server_py["server"] + end + end + + site_src_components_EditDrawer_svelte -->|fetch| api__api_activity_ + site_src_components_AthleteDrawer_svelte -->|fetch| api__api_athlete + site_src_components_AthleteView_svelte -->|fetch| api__api_athlete + site_src_components_AthleteView_svelte -->|fetch| api__api_athlete__ + site_src_layouts_Base_astro -->|fetch| api__api_me + site_src_layouts_Base_astro -->|fetch| api__api_admin_jobs + site_src_layouts_Base_astro -->|fetch| api__api_auth_logout + site_src_layouts_Base_astro -->|fetch| api__api_admin_jobs__ + site_src_layouts_Base_astro -->|fetch| api__api_auth_logout__ + site_src_layouts_Base_astro -->|fetch| api__api_upload + site_src_layouts_Base_astro -->|fetch| api__api_strava_status + site_src_layouts_Base_astro -->|fetch| api__api_strava_auth_url + site_src_layouts_Base_astro -->|fetch| api__api_strava_sync_stream + site_src_layouts_Base_astro -->|fetch| api__api_strava_reset + site_src_layouts_Base_astro -->|fetch| api__api_upload_strava_zip + site_src_layouts_Base_astro -->|fetch| api__api_garmin_status + site_src_layouts_Base_astro -->|fetch| api__api_garmin_connect + site_src_layouts_Base_astro -->|fetch| api__api_garmin_sync_stream + site_src_layouts_Base_astro -->|fetch| api__api_garmin_disconnect + site_src_pages_admin_index_astro -->|fetch| api__api_admin_disk + site_src_pages_admin_index_astro -->|fetch| api__api_admin_users___h__rebuild + site_src_pages_admin_index_astro -->|fetch| api__api_admin_users___h__reset_password_code + site_src_pages_admin_index_astro -->|fetch| api__api_admin_users___pendingHandle__activities + site_src_pages_admin_index_astro -->|fetch| api__api_admin_disk__ + site_src_pages_admin_index_astro -->|fetch| api__api_admin_users_ + site_src_pages_about_index_astro -->|fetch| api__api_me + site_src_pages_about_index_astro -->|fetch| api__api_stats + site_src_pages_about_index_astro -->|fetch| api__api_stats___ + site_src_pages_feedback_index_astro -->|fetch| api__api_feedback + site_src_pages_feedback_index_astro -->|fetch| api__api_me + site_src_pages_feedback_index_astro -->|fetch| api__api_feedback__ + site_src_pages_feedback_index_astro -->|fetch| api__api_me__ + site_src_pages_register_index_astro -->|fetch| api__api_register + site_src_pages_reset_password_index_astro -->|fetch| api__api_auth_reset_password + site_src_pages_invites_index_astro -->|fetch| api__api_invites + site_src_pages_invites_index_astro -->|fetch| api__api_invites__ + site_src_pages_login_index_astro -->|fetch| api__api_auth_login + site_src_pages_convert_index_astro -->|fetch| api__api_import_bas + site_src_pages_about_it_index_astro -->|fetch| api__api_me + site_src_pages_about_it_index_astro -->|fetch| api__api_stats + site_src_pages_about_it_index_astro -->|fetch| api__api_stats___ + site_src_pages_about_ca_index_astro -->|fetch| api__api_me + site_src_pages_about_ca_index_astro -->|fetch| api__api_stats + site_src_pages_about_ca_index_astro -->|fetch| api__api_stats___ + site_src_pages_about_es_index_astro -->|fetch| api__api_me + site_src_pages_about_es_index_astro -->|fetch| api__api_stats + site_src_pages_about_es_index_astro -->|fetch| api__api_stats___ + site_src_components_ActivityDetail_svelte --> site_src_components_ActivityMap_svelte + site_src_components_ActivityDetail_svelte --> site_src_components_ActivityCharts_svelte + site_src_components_ActivityDetail_svelte --> site_src_components_EditDrawer_svelte + site_src_components_ActivityDetailLoader_svelte --> site_src_components_ActivityDetail_svelte + site_src_components_AthleteView_svelte --> site_src_components_MmpChart_svelte + site_src_components_AthleteView_svelte --> site_src_components_RecordsView_svelte + site_src_components_AthleteView_svelte --> site_src_components_AthleteDrawer_svelte + site_src_components_LocalActivityDetail_svelte --> site_src_components_ActivityDetail_svelte + site_src_pages_index_astro --> site_src_components_ActivityFeed_svelte + site_src_pages_index_astro --> site_src_layouts_Base_astro + site_src_pages_record_index_astro --> site_src_layouts_Base_astro + site_src_pages_activity__id__astro --> site_src_components_ActivityDetail_svelte + site_src_pages_activity__id__astro --> site_src_layouts_Base_astro + site_src_pages_activity_index_astro --> site_src_components_ActivityDetailLoader_svelte + site_src_pages_activity_index_astro --> site_src_layouts_Base_astro + site_src_pages_admin_index_astro --> site_src_layouts_Base_astro + site_src_pages_about_index_astro --> site_src_layouts_Base_astro + site_src_pages_feedback_index_astro --> site_src_layouts_Base_astro + site_src_pages_register_index_astro --> site_src_layouts_Base_astro + site_src_pages_reset_password_index_astro --> site_src_layouts_Base_astro + site_src_pages_community_index_astro --> site_src_components_CommunityView_svelte + site_src_pages_community_index_astro --> site_src_layouts_Base_astro + site_src_pages_invites_index_astro --> site_src_layouts_Base_astro + site_src_pages_login_index_astro --> site_src_layouts_Base_astro + site_src_pages_convert_index_astro --> site_src_layouts_Base_astro + site_src_pages_activity_local_index_astro --> site_src_components_LocalActivityDetail_svelte + site_src_pages_activity_local_index_astro --> site_src_layouts_Base_astro + site_src_pages_u__handle__index_astro --> site_src_components_ActivityFeed_svelte + site_src_pages_u__handle__index_astro --> site_src_layouts_Base_astro + site_src_pages_u__handle__athlete_index_astro --> site_src_components_AthleteView_svelte + site_src_pages_u__handle__athlete_index_astro --> site_src_layouts_Base_astro + site_src_pages_u__handle__stats_index_astro --> site_src_components_StatsView_svelte + site_src_pages_u__handle__stats_index_astro --> site_src_layouts_Base_astro + site_src_pages_about_it_index_astro --> site_src_layouts_Base_astro + site_src_pages_about_ca_index_astro --> site_src_layouts_Base_astro + site_src_pages_about_es_index_astro --> site_src_layouts_Base_astro + bincio_cli_py --> bincio_edit_cli_py + bincio_cli_py --> bincio_extract_cli_py + bincio_cli_py --> bincio_render_cli_py + bincio_cli_py --> bincio_serve_cli_py + bincio_cli_py --> bincio_import__cli_py + bincio_cli_py --> bincio_serve_init_cmd_py + bincio_cli_py --> bincio_dev_py + bincio_import__strava_py --> bincio_extract_models_py + bincio_import__strava_py --> bincio_extract_sport_py + bincio_edit_server_py --> bincio_edit_ops_py + bincio_extract_simplify_py --> bincio_extract_models_py + bincio_extract_metrics_py --> bincio_extract_models_py + bincio_extract_ingest_py --> bincio_extract_models_py + bincio_extract_strava_api_py --> bincio_extract_models_py + bincio_extract_strava_api_py --> bincio_extract_sport_py + bincio_extract_cli_py --> bincio_extract_parsers_factory_py + bincio_extract_cli_py --> bincio_extract_config_py + bincio_extract_cli_py --> bincio_extract_dedup_py + bincio_extract_writer_py --> bincio_extract_metrics_py + bincio_extract_writer_py --> bincio_extract_simplify_py + bincio_extract_writer_py --> bincio_extract_timeseries_py + bincio_extract_writer_py --> bincio_extract_models_py + bincio_extract_timeseries_py --> bincio_extract_models_py + bincio_serve_server_py --> bincio_serve_db_py + bincio_serve_server_py --> bincio_edit_ops_py + bincio_extract_parsers_tcx_py --> bincio_extract_models_py + bincio_extract_parsers_tcx_py --> bincio_extract_sport_py + bincio_extract_parsers_fit_py --> bincio_extract_models_py + bincio_extract_parsers_fit_py --> bincio_extract_sport_py + bincio_extract_parsers_gpx_py --> bincio_extract_sport_py + bincio_extract_parsers_gpx_py --> bincio_extract_models_py + bincio_extract_parsers_gpx_py --> bincio_extract_parsers_base_py + bincio_extract_parsers_factory_py --> bincio_extract_parsers_gpx_py + bincio_extract_parsers_factory_py --> bincio_extract_models_py + bincio_extract_parsers_factory_py --> bincio_extract_parsers_tcx_py + bincio_extract_parsers_factory_py --> bincio_extract_parsers_base_py + bincio_extract_parsers_factory_py --> bincio_extract_parsers_fit_py + bincio_extract_parsers_base_py --> bincio_extract_models_py \ No newline at end of file diff --git a/docs/deployment/multi-user.md b/docs/deployment/multi-user.md index 3ddae04..373d139 100644 --- a/docs/deployment/multi-user.md +++ b/docs/deployment/multi-user.md @@ -204,4 +204,4 @@ The browser fetches and merges remote shards concurrently. Remote activities app - [CLI reference — bincio serve](../reference/cli.md#bincio-serve) - [CLI reference — bincio dev](../reference/cli.md#bincio-dev) - [API reference](../reference/api.md) -- [BAS schema — instance manifest](../../SCHEMA.md#instance-manifest) +- [BAS schema — instance manifest](../schema.md#instance-manifest) diff --git a/docs/deployment/vps.md b/docs/deployment/vps.md new file mode 100644 index 0000000..f4fbff8 --- /dev/null +++ b/docs/deployment/vps.md @@ -0,0 +1,410 @@ +# VPS deployment guide + +Concrete setup for a Debian VPS running a private multi-user bincio instance. +Code is deployed directly from your laptop via `git push` — no GitHub required. + +## Assumptions + +- Bare Debian 12 VPS with root SSH access +- You own a domain pointed at the VPS +- You have Strava API credentials +- Up to ~30 users + +--- + +## 1. Install system dependencies + +```bash +apt update && apt upgrade -y +apt install -y git curl nginx certbot python3-certbot-nginx sqlite3 rsync +``` + +**Node.js 20 LTS** (the Debian package is too old): +```bash +curl -fsSL https://deb.nodesource.com/setup_20.x | bash - +apt install -y nodejs +``` + +**uv** (manages Python and all Python deps): +```bash +curl -LsSf https://astral.sh/uv/install.sh | sh +# add to PATH: +echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc +source ~/.bashrc +``` + +--- + +## 2. Set up the code directory + +```bash +mkdir -p /opt/bincio +git init --bare /opt/bincio-repo.git +``` + +Create the post-receive hook at `/opt/bincio-repo.git/hooks/post-receive`: + +```bash +#!/bin/bash +set -e + +REPO=/opt/bincio-repo.git +DEPLOY=/opt/bincio +DATA=/var/bincio/data + +while read oldrev newrev refname; do + echo "--- Checking out $refname ---" + git --work-tree=$DEPLOY --git-dir=$REPO checkout -f $newrev + + echo "--- Syncing Python deps ---" + cd $DEPLOY + ~/.local/bin/uv sync --extra serve --extra strava --extra garmin + + echo "--- Syncing JS deps ---" + cd $DEPLOY/site + npm install --silent + + echo "--- Building site ---" + cd $DEPLOY + ~/.local/bin/uv run bincio render --data-dir $DATA --site-dir $DEPLOY/site + + echo "--- Pruning dist/data (nginx serves /data/ directly from $DATA) ---" + rm -rf $DEPLOY/site/dist/data + + echo "--- Copying dist to webroot ---" + rsync -a --delete --exclude=data/ $DEPLOY/site/dist/ /var/www/bincio/ + + echo "--- Restarting API ---" + systemctl restart bincio || echo "WARNING: bincio service restart failed — check journalctl -u bincio" + + echo "--- Done ---" +done +``` + +```bash +chmod +x /opt/bincio-repo.git/hooks/post-receive +mkdir -p /var/www/bincio /var/bincio/data /var/bincio/sources +``` + +--- + +## 3. systemd service + +The hook restarts the `bincio` service on every deploy, so it must exist before the first push. + +Create `/etc/bincio/secrets.env`: + +```bash +mkdir -p /etc/bincio +chmod 700 /etc/bincio +cat > /etc/bincio/secrets.env <:/opt/bincio-repo.git +``` + +Push your code: + +```bash +git push vps main +``` + +The hook checks out the code, installs deps, and builds the site. +Subsequent pushes (including unpublished branches) work the same way: + +```bash +git push vps mobile_app # deploy any branch directly +``` + +--- + +## 5. Initialise the instance + +```bash +cd /opt/bincio + +uv run bincio init \ + --data-dir /var/bincio/data \ + --handle dave \ + --display-name "Dave" \ + --name "My Bincio" +# prompted for password; prints a first invite code +``` + +Enable the edit/upload UI (this env var is read at build time and is gitignored, so it must be set on the server): + +```bash +echo "PUBLIC_EDIT_ENABLED=true" > /opt/bincio/site/.env +``` + +Set the user cap: + +```bash +sqlite3 /var/bincio/data/instance.db \ + "INSERT INTO settings VALUES ('max_users', '30');" +``` + +--- + +## 6. Prepare your own activities + +Source files (raw GPX/FIT) live separately from the BAS output: + +``` +/var/bincio/sources/dave/ ← raw activity files, rsync'd from laptop +/var/bincio/data/dave/ ← BAS JSON output (bincio extract writes here) +``` + +Configure `/opt/bincio/extract_config.yaml` on the server to point to your +source dir: + +```yaml +sources: + - path: /var/bincio/sources/dave/activities + type: strava_export + - path: /var/bincio/sources/dave/activities.csv + type: strava_csv + +output: + dir: /var/bincio/data + +workers: 2 # cap extract parallelism on the VPS (default: all CPUs) +``` + +Sync and extract (run from your laptop or SSH in): + +```bash +# push raw files from laptop +rsync -avz ~/your-activity-data/ root@:/var/bincio/sources/dave/ + +# extract on server +ssh root@ "cd /opt/bincio && uv run bincio extract" + +# rebuild site +ssh root@ "cd /opt/bincio && \ + uv run bincio render --data-dir /var/bincio/data --site-dir site && \ + rm -rf site/dist/data && \ + rsync -a --delete --exclude=data/ site/dist/ /var/www/bincio/" +``` + +--- + +## 7. nginx + +Create `/etc/nginx/sites-available/bincio`: + +```nginx +server { + listen 80; + server_name yourdomain.com; + + root /var/www/bincio; + index index.html; + + client_max_body_size 2G; # Strava export ZIPs can exceed 1 GB + client_body_timeout 300s; # allow slow uploads without nginx dropping the connection + + # API → bincio serve + location /api/ { + proxy_pass http://127.0.0.1:4041; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_read_timeout 120s; # Strava sync can be slow + } + + # Data files served live from disk — bypasses the build/rsync cycle + # so uploads and merges are visible immediately without a site rebuild. + # + # IMPORTANT: because nginx owns /data/ here, the post-receive hook must + # delete dist/data/ before rsyncing to the webroot. Otherwise astro build + # copies all activity JSON (GBs) into dist/ and rsync duplicates it again. + # The hook already does this; manual rebuilds must do the same. + location /data/ { + alias /var/bincio/data/; + add_header Cache-Control "no-cache, must-revalidate"; + } + + # Activity detail pages: fall back to the dynamic shell for activities uploaded + # after the last site build (avoids 404 while waiting for a rebuild). + location /activity/ { + try_files $uri $uri/ /activity/index.html; + } + + # Per-user profile pages: fall back to the home page while the background + # rebuild (triggered automatically on registration) completes. + location /u/ { + try_files $uri $uri/ /index.html; + } + + # Static files + location / { + try_files $uri $uri/ $uri.html =404; + } +} +``` + +```bash +# disable the default nginx welcome page +rm /etc/nginx/sites-enabled/default +ln -s /etc/nginx/sites-available/bincio /etc/nginx/sites-enabled/ +nginx -t && systemctl reload nginx +``` + +### Enable gzip compression + +The default `nginx.conf` has gzip on but `gzip_types` commented out, so only +HTML is compressed. Activity index shards are JSON and compress ~90% — enable +the full list: + +```bash +# In /etc/nginx/nginx.conf, uncomment the gzip block: +gzip_vary on; +gzip_proxied any; +gzip_comp_level 6; +gzip_buffers 16 8k; +gzip_http_version 1.1; +gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript; +``` + +```bash +nginx -t && systemctl reload nginx +``` + +You can verify the site is served correctly by hitting the IP directly: +`http:///` — you should see the bincio activity feed, not the nginx welcome page. + +--- + +## 8. SSL + +SSL requires the domain to be pointing at the VPS first. In your DNS provider, add: + +``` +Type: A +Name: @ +Value: +TTL: 300 +``` + +Verify propagation before running certbot: + +```bash +dig yourdomain.com A +short # must return your VPS IP +``` + +Then: + +```bash +certbot --nginx -d yourdomain.com +# certbot edits the nginx config and sets up automatic renewal +``` + +--- + +## 9. Invite users + +After `bincio init` prints the first invite code, you can generate more from +the browser at `/u/{handle}/athlete/` → **Invites** button (visible only to +the page owner), or directly via the CLI: + +```bash +sqlite3 /var/bincio/data/instance.db \ + "INSERT INTO invites (code, created_by, created_at) \ + VALUES (upper(hex(randomblob(4))), 'dave', unixepoch());" +``` + +Share the link: `https://yourdomain.com/register/?code=XXXXXXXX` + +Each new user uploads their activities via the **+** button in the top nav +(supports bulk GPX/FIT/TCX drop). They can later connect Strava for +incremental sync from the same modal. + +--- + +## Reading user feedback + +Users can submit feedback from the **Feedback** link in the nav (visible when logged in). +Submissions are stored as JSON on the server: + +``` +/var/bincio/data/_feedback/ + {handle}.json ← one file per user, array of submissions + {handle}/ ← attached images +``` + +To read all feedback: + +```bash +cat /var/bincio/data/_feedback/*.json | python3 -m json.tool +``` + +Per-user only: + +```bash +cat /var/bincio/data/_feedback/pres.json | python3 -m json.tool +``` + +--- + +## Day-to-day operations + +| Task | Command | +|------|---------| +| Deploy code update | `git push vps main` (from laptop) | +| Sync your raw files | `rsync -avz ~/your-activity-data/ root@:/var/bincio/sources/dave/` | +| Re-extract after sync | `ssh root@ "cd /opt/bincio && uv run bincio extract"` then push again to rebuild | +| View API logs | `journalctl -u bincio -f` | +| Restart API | `systemctl restart bincio` | +| Check nginx logs | `tail -f /var/log/nginx/error.log` | +| Renew SSL (auto) | `certbot renew --dry-run` | + +--- + +## See also + +- [Multi-user architecture](multi-user.md) +- [CLI reference](../reference/cli.md) +- [API reference](../reference/api.md) \ No newline at end of file diff --git a/docs/developer-guide.md b/docs/developer-guide.md new file mode 100644 index 0000000..8273931 --- /dev/null +++ b/docs/developer-guide.md @@ -0,0 +1,307 @@ +# Developer Guide + +This guide is for developers contributing to BincioActivity. + +## Prerequisites + +- **Python 3.12+** with [uv](https://docs.astral.sh/uv/) +- **Node 20+** with npm +- **Git** + +## Local Setup + +```bash +git clone https://github.com/brutsalvadi/bincio-activity.git +cd bincio-activity + +# Install Python dependencies +uv sync + +# Install optional extras for multi-user development +uv sync --extra serve --extra edit + +# Install Node dependencies (for the site) +cd site && npm install && cd .. +``` + +## Running Locally + +### Single-user (fastest for testing extract logic) + +```bash +# Configure where to find your test activities +cp extract_config.example.yaml extract_config.yaml +$EDITOR extract_config.yaml # set input.dirs and output.dir + +# Extract activities +uv run bincio extract + +# Start the dev server (no login, no API server) +uv run bincio dev --data-dir ~/bincio_data +# → http://localhost:4321/u/{handle}/ +``` + +### Multi-user (for testing auth, write API, admin features) + +```bash +# Create a test instance with an admin user +uv run bincio init --data-dir /tmp/bincio_test --handle testadmin + +# Extract activities +uv run bincio extract --output /tmp/bincio_test + +# Start everything (bincio serve + astro dev) +uv run bincio dev --data-dir /tmp/bincio_test +# → http://localhost:4321 (login with testadmin/{password}) +``` + +Ctrl+C stops both servers. + +## Running Tests + +```bash +# All tests +uv run pytest + +# Specific test file +uv run pytest tests/extract/test_parsers.py + +# Specific test function +uv run pytest tests/extract/test_parsers.py::test_gpx_parser + +# With verbose output +uv run pytest -vv + +# With coverage report +uv run pytest --cov=bincio +``` + +Tests are in `tests/` and use pytest + fixtures for DRY test data. + +## Project Structure + +``` +bincio/ + extract/ Python package for GPX/FIT/TCX parsing + models.py DataPoint, ParsedActivity, LapData + parsers/ GPX, FIT, TCX parsers + factory + sport.py Sport name normalization + metrics.py Haversine-based stats (distance, elevation) + timeseries.py 1Hz downsampling → BAS timeseries object + simplify.py RDP track simplification (no external deps) + dedup.py Exact + fuzzy duplicate detection + strava_csv.py Strava activities.csv importer + writer.py BAS JSON + GeoJSON output + config.py extract_config.yaml loader + cli.py `bincio extract` command + render/ + cli.py `bincio render` command + merge.py Sidecar edit overlay (produces _merged/) + edit/ + cli.py `bincio edit` FastAPI server + server.py Edit API endpoints + serve/ + cli.py `bincio serve` command + server.py Multi-user FastAPI server (auth, invites, admin) + db.py SQLite data layer + init_cmd.py `bincio init` bootstrap + shared/ (if needed) + +site/ Astro + Svelte + Tailwind frontend + src/ + layouts/ Base.astro (auth wall, nav) + pages/ Routes (activity feed, detail, login, etc.) + components/ Svelte components (maps, charts, edit drawer) + lib/ TypeScript utilities (types, format, dataloader) + +tests/ pytest test suite + extract/ + render/ + serve/ + fixtures/ Shared test data +``` + +## Key Concepts + +### BAS (BincioActivity Schema) + +Activity data flows as **BAS JSON** files in `{user}/activities/`. The format is specified in [SCHEMA.md](schema.md). + +Key files: + +- `{id}.json` — activity metadata + timeseries +- `_merged/` symlink — sidecar edits overlaid on activities +- `edits/{id}.md` — user-created sidecar (optional) + +### Shard model + +Multi-user instances use a **shard manifest** (root `index.json`) that lists per-user shards. The browser fetches all shards concurrently and merges them. This allows: + +- Federation (remote shard URLs) +- Yearly pagination +- No data duplication + +### Extract pipeline + +``` +GPX/FIT/TCX files + ↓ (parse) +ParsedActivity + ↓ (calculate metrics) +BAS Activity JSON + ↓ (downsample to 1Hz) +Timeseries + ↓ (simplify with RDP) +GeoJSON + ↓ (write) +activities/{id}.json + activities/{id}.geojson +``` + +### Render pipeline + +``` +{user}/ + activities/*.json (extracted) + edits/*.md (user sidecars) + ↓ (merge_all) +_merged/ + index.json (sidecar edits applied) + activities/{id}.json + {id}.geojson + ↓ (astro build) +site/dist/ +``` + +Editing does not require re-extraction. + +## Making Changes + +### Adding a new endpoint + +1. Add a route in `bincio/serve/server.py` (or `bincio/edit/server.py` for single-user) +2. Add Pydantic models for request/response if needed +3. Add tests in `tests/serve/` +4. Update `docs/reference/api.md` with the new endpoint +5. If admin-only, protect it with `await _require_admin(bincio_session)` + +### Adding a parser for a new format + +1. Create `bincio/extract/parsers/myformat.py` +2. Implement a parser class with `parse(file_path: Path) -> ParsedActivity` +3. Register it in `bincio/extract/parsers/__init__.py` +4. Add tests in `tests/extract/test_parsers.py` + +### Modifying BAS schema + +1. Edit `schema/bas-v1.schema.json` (JSON Schema) +2. Update `SCHEMA.md` (human-readable spec) +3. Update TypeScript types in `site/src/lib/types.ts` +4. Add a migration if the change is breaking + +### Frontend changes + +**Svelte components** are in `site/src/components/`. Key ones: + +- `ActivityFeed.svelte` — activity grid + filters +- `ActivityDetail.svelte` — activity page (maps, charts, photos) +- `EditDrawer.svelte` — sidecar editor + +Use `uv run bincio dev` to test changes live. The site hot-reloads on file changes. + +## Code Style + +- **Python:** PEP 8, type hints where possible +- **JavaScript/TypeScript:** ESLint + Prettier (configured in `site/`) +- **Svelte:** No self-closing non-void tags; interactive divs need `role` + keyboard handler + +## Git Workflow + +1. Create a branch: `git checkout -b feature/my-feature` +2. Make changes and test locally +3. Commit: `git commit -m "Clear, specific commit message"` +4. Push: `git push origin feature/my-feature` +5. Open a pull request + +**Commit message style:** + +- Imperative mood ("add feature", not "added feature") +- Reference issues if relevant: "fix #123" +- First line ≤ 50 characters +- Blank line, then detailed explanation if needed + +## Performance Considerations + +### Extract speed + +- **ProcessPoolExecutor with initializer** — large data (Strava lookups, hash sets) is sent once per worker, not per task +- **Haversine** — 10x faster than geopy for distance calculations +- **Lazy parsing** — FIT files decoded only once per task + +### Render speed + +- **RDP simplification** — custom implementation (no external wheels for Pyodide) +- **Gzip compression** — activity JSON and geojson are served gzipped +- **Concurrent shard fetch** — browser loads all shards in parallel + +### Frontend + +- **MapLibre GL v5** — requires explicit center/zoom and workarounds +- **Observable Plot** — use hyphenated curve names (e.g. `"monotone-x"`) +- **Client-only for complex components** — use `client:only="svelte"` for activity detail to avoid hydration mismatches + +## Debugging + +### Python + +```bash +# Interactive debugger +uv run python -m pdb -m bincio.extract.cli + +# Or use breakpoint() in code +breakpoint() +uv run bincio extract +``` + +### TypeScript + +Check your editor's TypeScript integration. The site has strict `tsconfig.json`. + +### Frontend + +- Open DevTools (F12) +- Check the Network tab for API calls +- Check Console for client-side errors + +### Database + +```bash +# Inspect the SQLite database directly +sqlite3 /tmp/bincio_test/instance.db +> SELECT * FROM users; +``` + +## Documentation + +- User-facing docs go in `docs/` +- API docs are auto-generated from FastAPI routes (and should be typed with Pydantic models) +- Code comments should explain *why*, not *what* + +## Known Issues & Limitations + +See the [GitHub repository](https://github.com/brutsalvadi/bincio-activity) for known issues and planned features. + +## Contributing + +Contributions are welcome! Please: + +1. Check existing issues/PRs so you're not duplicating work +2. Open an issue first for large changes +3. Include tests for new features +4. Update docs (user guide, API ref, or developer guide) +5. Follow the code style guidelines + +## See also + +- [Architecture](architecture.md) — system design and data flow +- [BAS Schema](schema.md) — activity data format +- [API Reference](reference/api.md) — all HTTP endpoints diff --git a/docs/elevation.md b/docs/elevation.md new file mode 100644 index 0000000..706e02e --- /dev/null +++ b/docs/elevation.md @@ -0,0 +1,344 @@ +# Elevation gain calculation — problem analysis and roadmap + +## The problem + +Bincio's current algorithm naively accumulates every positive elevation delta between +consecutive track points. This **always overestimates** real climbing because it treats +sensor noise as genuine ascent. The overestimation ranges from insignificant on long +mountain rides to catastrophic on flat routes where 100% of the reported gain is noise. + +--- + +## Current algorithm (`metrics.py:_elevation`) + +```python +def _elevation(pts): + elevations = [p.elevation_m for p in pts if p.elevation_m is not None] + gain = loss = 0.0 + for a, b in zip(elevations, elevations[1:]): + diff = b - a + if diff > 0: + gain += diff + else: + loss += diff + return gain, loss +``` + +Every positive step — including 0.1m GPS jitter, barometric quantization steps of +0.2m, and random-walk noise — is counted as climbing. There is no filtering, +smoothing, or minimum-step threshold of any kind. + +--- + +## Root causes of overestimation + +### 1. GPS-derived altitude noise + +GPS units measure altitude from satellite triangulation. This is inherently less +accurate than horizontal positioning: typical GPS altitude error is ±5–15m, and the +error follows a correlated random walk. On a flat route, the track oscillates above +and below the true elevation, and the positive half of those oscillations accumulates +as phantom climbing. + +**Characteristic signature:** elevation range far smaller than reported gain; nearly +100% of deltas are sub-1m; median step size is 0.0m. + +### 2. Barometric altimeter quantization + +Devices with a barometric sensor report higher-quality data, but they still apply +internal smoothing and quantise the output to fixed increments (commonly 0.2m or +0.4m). The device holds the reading steady for several seconds, then steps to the +next quantised value. Small-but-real oscillations at the quantisation boundary +(e.g. hovering between 128.2m and 128.4m while essentially flat) produce repeated +tiny up/down steps that accumulate. + +**Characteristic signature:** many repeated identical elevations (device holding); +most transitions are 0.0m, 0.2m or 0.4m; significant fraction of gain from sub-1m +steps even on a real climb. + +### 3. High sampling rate amplifying both effects + +At 1 Hz, both GPS and barometric sensors produce more noise steps per meter of +real climbing than at lower rates. Downsampling to 1 Hz (as the timeseries writer +does) does not eliminate noise already present in the source data. + +--- + +## Case studies + +### Activity 1 — diego_p, 2026-04-11T051441Z (Wahoo ELEMNT, GPX) + +- **URL:** https://bincio.org/activity/2026-04-11T051441Z/ +- **Bincio reports:** 353.6m gain +- **Correct estimate:** ~150–160m (Wahoo device's own reading, which applies internal + thresholding) +- **Excess:** ~200m (56% overestimate) + +| Metric | Value | +|--------|-------| +| Points | 15,721 | +| Elevation range | −10.6m to +5.4m (16m total span) | +| Median \|delta\| | 0.000m | +| Zero-change steps | 12,358 (79%) | +| Sub-0.5m steps | 3,339 (21%) | +| Steps ≥ 1m | 0 (0%) | +| Gain from sub-1m steps | 353.6m (100% of total) | + +**Diagnosis:** This is a flat coastal route. The GPS altitude range is only 16m. +Every single metre of reported gain is sub-1m GPS jitter — no real climbing is +recorded at all. Even a 1m threshold would produce exactly 0m gain, which is wrong +in the other direction (the route may have minor real undulation). The Wahoo device's +own algorithm uses internal hysteresis to report ~153m. + +### Activity 2 — m4xw3ll__, 2026-04-14T161945Z (Bryton Rider, FIT) + +- **URL:** http://your-instance/activity/2026-04-14T161945Z/ +- **Bincio reports:** 1285.2m gain +- **Correct estimate:** ~885m (Strava / device reading) +- **Excess:** ~400m (45% overestimate) + +| Metric | Value | +|--------|-------| +| Points | 6,583 | +| Elevation range | 0.0m to 454.0m (454m total span) | +| Median \|delta\| | 0.000m | +| Zero-change steps | 4,077 (62%) | +| Sub-0.5m steps | 769 (12%) | +| 0.5–1m steps | 647 (10%) | +| 1–2m steps | 921 (14%) | +| 2m+ steps | 168 (3%) | +| Gain from sub-1m steps | 484.0m (38% of total) | + +**Diagnosis:** Real climbing exists (0–454m) but 38% of the reported gain comes from +sub-1m barometric quantization noise. The Bryton records elevation at ≈0.2m +increments. At quantization boundaries the device oscillates producing repeated +tiny up/down steps. A simple 1m threshold gives 801m (10% below truth); 2m gives +only 221m (too aggressive). Pure threshold-based filtering doesn't work well here. + +--- + +## Alternative algorithms + +### A. Simple threshold + +Only count a step if it exceeds `min_step_m`: + +```python +gain += diff if diff >= min_step_m else 0 +``` + +**Pros:** trivial to implement, zero overhead. +**Cons:** flat/hiking routes with gradual slopes produce many steps < threshold +that together represent real climbing. A 2m threshold already loses 30% of real +gain on the Bryton activity. Requires per-device tuning that is impractical. + +--- + +### B. Hysteresis / dead-band accumulation + +Track a "committed" elevation. Only commit a new elevation when it differs from +the last committed value by more than `threshold_m`. Accumulate from committed to +committed only. + +```python +def _elevation_hysteresis(elevations, threshold_m=10.0): + gain = loss = 0.0 + committed = elevations[0] + for e in elevations[1:]: + diff = e - committed + if abs(diff) >= threshold_m: + if diff > 0: + gain += diff + else: + loss += abs(diff) + committed = e + return gain, loss +``` + +**Pros:** naturally handles both GPS drift and barometric quantization; used by +Strava (proprietary variant), RideWithGPS (10m default), and GPSies (5m). +**Cons:** threshold choice is critical and device-dependent. On a genuine 8m climb +followed by descent, a 10m threshold records zero. Needs to be lower for cycling +than hiking (slopes are smoother, sensors better). + +**Results on our case studies with threshold=10m:** +- Wahoo flat (correct ~153m): would likely produce 0–30m. Fixes the gross overcount + but may undercount real minor undulation. +- Bryton climb (correct ~885m): would need evaluation on the raw data. + +--- + +### C. Moving-average pre-smoothing + +Apply a sliding-window mean or Gaussian blur to the elevation series, then +accumulate naively. + +```python +import statistics + +def smooth(elevations, window=30): + half = window // 2 + out = [] + for i, e in enumerate(elevations): + lo, hi = max(0, i - half), min(len(elevations), i + half + 1) + out.append(statistics.mean(elevations[lo:hi])) + return out + +gain, loss = _elevation(smooth(elevations, window=30)) +``` + +**Pros:** easy to implement; smoothing removes high-frequency noise while preserving +long-wavelength terrain. +**Cons:** loses real short climbs (e.g. a 20m ramp over 20 seconds is averaged to +near-flat). Window size needs tuning per sample rate. Edge effects at start/end. + +--- + +### D. Savitzky-Golay filter + +A polynomial least-squares smoothing filter that better preserves peaks and +troughs than a simple moving average. Available in `scipy.signal.savgol_filter` +(scipy is already an indirect dependency via numpy, which is used nowhere critical — +but adding scipy is a dependency choice). + +**Pros:** better terrain shape preservation than moving average; standard in +scientific signal processing. +**Cons:** requires scipy; harder to implement without it; window/order tuning still +required. + +--- + +### E. Kalman filter (device-class-aware) + +A Kalman filter can be tuned with separate process noise and measurement noise +parameters for GPS vs barometric data. This is what high-end cycling computers do +internally. + +**Pros:** theoretically optimal; can be parameterised per device class. +**Cons:** significantly more complex; requires knowing the device class (GPS-only vs +barometric); still requires parameter tuning. + +--- + +### F. Source-aware strategy + +Use different algorithms depending on the file type and whether the device reported +enhanced (barometric) altitude: + +- **FIT file with `enhanced_altitude` field**: barometric data, use hysteresis 5m +- **FIT file with GPS altitude only**: treat as GPS, use hysteresis 10–15m or + discard altitude entirely and use a DEM lookup +- **GPX with `` tag**: assume GPS unless `` contains barometric + fields; use hysteresis 10–15m +- **Strava-enriched data**: Strava's API provides corrected `altitude` arrays; use + as-is with hysteresis 2m to catch quantization + +--- + +## What Strava/Garmin/others do + +| Platform | Method | +|---|---| +| Strava | Proprietary; replaces raw altitude with DEM-corrected data for GPS-only devices; applies internal smoothing before accumulation | +| Garmin Connect | Uses enhanced\_altitude (barometric), applies Kalman filter on-device; Connect re-applies server-side smoothing | +| Wahoo | On-device hysteresis (≈3m threshold); the GPX file contains already-smoothed altitude | +| RideWithGPS | 10m hysteresis by default, configurable | +| Komoot | DEM correction + smoothing | +| TrainingPeaks | Configurable threshold (5m default) | + +Strava's approach of DEM (Digital Elevation Model) correction is the gold standard +for GPS-only tracks: replace the noisy GPS altitude entirely with the ground truth +from a 30m-resolution DEM such as SRTM. This requires an additional data source +(e.g. the Open-Elevation API or a locally hosted SRTM tile set) but completely +eliminates GPS altitude noise. + +--- + +## Recommended fix + +Given the two failure modes observed: + +### Short term — ✅ Implemented (2026-04-20) + +**Hysteresis accumulation** with source-aware thresholds, applied at extract time: + +| Source | Threshold | +|---|---| +| FIT with `enhanced_altitude` (barometric) | 5 m | +| FIT with GPS altitude | 10 m | +| GPX | 10 m | +| TCX | 10 m | + +`ParsedActivity.altitude_source` is set by each parser (`"barometric"` / `"gps"` / +`"unknown"`). `_elevation()` in `metrics.py` selects the threshold from this value. + +New activities extracted after this change benefit automatically. Existing activities +require re-extraction from source files. + +### Medium term — ✅ Implemented (2026-04-20) + +**Two on-demand recalculation options** in the activity edit drawer: + +#### Option 1 — Hysteresis (fast, offline) + +Re-applies the same source-aware hysteresis accumulation as the extract +pipeline directly to the recorded elevation, with no network calls. + +- Uses `elevation_m_original` from the timeseries (the backup saved on the + first DEM run) if present; otherwise uses the current `elevation_m`. +- Threshold: **5 m** for barometric sources, **10 m** for GPS. +- Does not modify the elevation array in the timeseries — only patches + `elevation_gain_m` / `elevation_loss_m`. +- Best for: devices with a barometric altimeter (e.g. Karoo 2, Garmin with + `enhanced_altitude`) where the recorded elevation is already accurate but + was extracted before the hysteresis fix was deployed. + +#### Option 2 — DEM terrain correction (SRTM30, requires network) + +Replaces the recorded GPS altitude with terrain data from an +Open-Elevation-compatible API (SRTM30, ~30 m resolution): + +1. GPS track subsampled to one point per 10 s to minimise API calls. +2. Terrain elevation fetched via `POST https://api.open-elevation.com/api/v1/lookup` + in batches of 512. +3. DEM elevation linearly interpolated back to the full 1 Hz series. +4. **45 s sliding median filter** applied to suppress SRTM tile-boundary + steps (these occur every ~7 s at cycling speed and accumulate as phantom + gain through a naive threshold). +5. **10 m hysteresis** applied to the smoothed series. +6. Original elevation backed up as `elevation_m_original` in the timeseries + (only on the first DEM run — never overwrites an existing backup). +7. Timeseries and activity JSON patched in place; chart and stats update. + +Best for: GPS-only devices (no barometric sensor) where the recorded +altitude is noisy and the DEM terrain is a better ground truth. + +> **Why median + 10 m, not 5 m?** SRTM30 at 1 Hz produces step changes at +> tile boundaries of 1–3 m every few seconds. A 5 m threshold lets most of +> these through; they accumulate and can inflate the result by 50 %+. The +> 45 s median smooths the steps before the dead-band sees them; 10 m catches +> any residual outliers. + +Implementation: `bincio/extract/dem.py` — `lookup_elevations()`, +`recalculate_elevation()`, `recalculate_elevation_hysteresis()`. +API endpoints: `POST /api/activity/{id}/recalculate-elevation/dem` and +`POST /api/activity/{id}/recalculate-elevation/hysteresis` on both servers. +Default DEM endpoint: `https://api.open-elevation.com`; override with +`--dem-url` or `DEM_URL` env var. + +--- + +## Implementation status + +| File | Status | +|---|---| +| `bincio/extract/models.py` | ✅ `altitude_source` field added | +| `bincio/extract/parsers/fit.py` | ✅ detects `enhanced_altitude` vs GPS | +| `bincio/extract/parsers/gpx.py` | ✅ sets `altitude_source = "gps"` | +| `bincio/extract/parsers/tcx.py` | ✅ sets `altitude_source = "gps"` | +| `bincio/extract/metrics.py` | ✅ hysteresis `_elevation()` with source-aware threshold | +| `bincio/extract/dem.py` | ✅ `lookup_elevations()` + `recalculate_elevation()` (median+10m) + `recalculate_elevation_hysteresis()` | +| `bincio/serve/server.py` | ✅ `POST /api/activity/{id}/recalculate-elevation/{dem\|hysteresis}` | +| `bincio/edit/server.py` | ✅ same endpoints (single-user) | +| `site/src/components/EditDrawer.svelte` | ✅ two buttons: "Recalculate (hysteresis)" + "Recalculate (DEM)" | +| `tests/test_metrics.py` | ✅ 5 parametric tests | diff --git a/docs/garmin_connect_disclaimer.md b/docs/garmin_connect_disclaimer.md new file mode 100644 index 0000000..5a00a0c --- /dev/null +++ b/docs/garmin_connect_disclaimer.md @@ -0,0 +1,109 @@ +# Garmin Connect Sync — Disclaimer + +**This feature uses an unofficial, community-maintained library to access Garmin Connect. +It is not affiliated with, endorsed by, or supported by Garmin Ltd. or its subsidiaries.** + +--- + +## What this feature does + +When you enable Garmin Connect sync, BincioActivity will: + +1. Ask for your Garmin Connect **email address and password** +2. Store those credentials on the server, encrypted at rest +3. Use them to log in to Garmin Connect on your behalf and download your activity files (FIT format) +4. Import those activities into your BincioActivity account + +--- + +## What you need to know before enabling this + +### Your credentials are stored on the server + +Unlike Strava (which uses OAuth — you authorize without sharing your password), +Garmin Connect has no official third-party API. This feature works by logging in +as you, using your actual email and password. + +This means: + +- The server operator has technical access to your stored credentials +- You are trusting both the software and the person running the server +- Only enable this on a server you control or run by someone you fully trust + +### This uses an unofficial API + +Garmin does not provide a public developer API for activity data. +This feature relies on a reverse-engineered interface that: + +- May break without notice when Garmin changes their systems +- Is not covered by any Garmin service agreement or SLA +- May violate Garmin Connect's Terms of Service + +BincioActivity takes no responsibility for account restrictions or bans +that may result from using this feature. + +### Cloudflare bot protection and rate limiting + +Garmin's login page (`sso.garmin.com`) is protected by Cloudflare, which +periodically blocks automated login attempts. When this happens, the sync +feature will fail at the login step with a "Login failed" error — even if +your credentials are correct. + +The underlying `garth` library tries three login strategies in sequence. +A blocked session typically looks like this in the server logs: + +``` +mobile+cffi returned 429: Mobile login returned 429 — IP rate limited by Garmin +mobile+requests failed: Mobile login failed (non-JSON): HTTP 403 +widget+cffi failed: Widget login: unexpected title 'GARMIN Authentication Application' +``` + +What each error means: +- **429** — Garmin is rate-limiting the server's IP address +- **403** — Cloudflare is blocking the request outright +- **unexpected title 'GARMIN Authentication Application'** — the login flow hit a + CAPTCHA or MFA challenge page that the library cannot handle automatically + +This is an upstream issue outside BincioActivity's control. The underlying +`garminconnect`/`garth` library usually releases a fix within days to weeks. +The workaround is to update those packages on the server: + +```bash +uv sync --extra garmin +``` + +If login consistently fails despite updating, check the +[garminconnect issue tracker](https://github.com/cyberjunky/python-garminconnect/issues) +for the current status. + +### Two-factor authentication (2FA) + +If your Garmin account has 2FA enabled, this feature may not work or may +require additional steps. Garmin has changed their authentication flow +several times; compatibility depends on the current state of the underlying library. + +### Rate limits + +Garmin does not publish API rate limits. Syncing too frequently or importing +large volumes of activities may result in temporary or permanent IP blocks. +BincioActivity applies conservative limits, but cannot guarantee uninterrupted access. + +--- + +## How to revoke access + +BincioActivity does not hold an OAuth token that can be revoked from Garmin's settings. +To stop BincioActivity from accessing your Garmin account: + +1. Delete your stored credentials from BincioActivity (Settings → Garmin Connect → Disconnect) +2. **Change your Garmin Connect password** — this is the only way to guarantee that + no previously stored credentials can be used + +--- + +## Recommendation + +If you have concerns about credential storage, consider the alternative: +export your activities from Garmin Connect or Garmin Express as FIT files +and upload them directly to BincioActivity. This requires no credentials +and is always available. diff --git a/docs/getting-started.md b/docs/getting-started.md index 6f2d7b2..28062a3 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -117,4 +117,4 @@ In multi-user mode the edit UI is always available via `bincio serve` — no ext - [Single-user deployment](deployment/single-user.md) — GitHub Pages, Netlify, VPS - [Multi-user deployment](deployment/multi-user.md) — VPS with nginx, inviting users - [CLI reference](reference/cli.md) — all commands and options -- [BAS schema](../SCHEMA.md) — the data format and federation protocol +- [BAS schema](schema.md) — the data format and federation protocol diff --git a/docs/graph.html b/docs/graph.html new file mode 100644 index 0000000..14c6d20 --- /dev/null +++ b/docs/graph.html @@ -0,0 +1,1370 @@ + + + + +Bincio — architecture graph + + + + +
+

Bincio architecture

+
+ + + + + + + + +
+ + +
+
+
+ + + + diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..6a2736b --- /dev/null +++ b/docs/index.md @@ -0,0 +1,38 @@ +# BincioActivity Documentation + +Welcome to BincioActivity — a federated, self-hosted activity stats platform. This documentation is organized by audience: + +## For Users + +**[Getting Started](getting-started.md)** — Extract your activities from Strava/Garmin, set up a local site, and deploy it. + +**[User Guide](user-guide.md)** — Upload activities, sync from Strava, edit titles/descriptions, manage photos, control privacy, configure your profile. + +## For Administrators + +**[Admin Guide](admin-guide.md)** — Deploy a multi-user instance, manage users, reset passwords, monitor rebuild status. + +**[Multi-user Deployment](deployment/multi-user.md)** — Step-by-step setup with nginx, systemd, and multi-user architecture. + +**[Single-user Deployment](deployment/single-user.md)** — Deploy as a read-only static site or with a local edit server. + +## For Developers + +**[Developer Guide](developer-guide.md)** — Local setup, how to run tests, architecture overview, how to contribute. + +**[Architecture](architecture.md)** — BAS data format, shard model, federation protocol, federation design. + +**[API Reference](reference/api.md)** — HTTP endpoints, request/response formats, authentication, rate limits. + +**[CLI Reference](reference/cli.md)** — All bincio commands and options. + +## Quick Links + +- [GitHub repo](https://github.com/brutsalvadi/bincio-activity) +- [BAS Schema](schema.md) — The data format specification +- [Architecture diagram](architecture.mmd) (Mermaid diagram) +- Live Swagger UI at `/api/docs` (when server is running) + +--- + +**Status:** This is early-stage, self-hosted software. See the [GitHub repo](https://github.com/brutsalvadi/bincio-activity) for known issues and planned features. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index f737a1d..3219f36 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -116,6 +116,7 @@ uv run bincio edit [OPTIONS] | `--port PORT` | `4041` | Bind port | | `--strava-client-id ID` | from config | Strava OAuth client ID | | `--strava-client-secret SECRET` | from config | Strava OAuth client secret | +| `--dem-url URL` | `https://api.open-elevation.com` | Open-Elevation-compatible API for the "Recalculate elevation" button (also `DEM_URL` env var) | Set `PUBLIC_EDIT_URL=http://localhost:4041` in `site/.env` to enable the Edit button and Upload ↑ button in the site. @@ -164,6 +165,7 @@ uv run bincio serve [OPTIONS] | `--site-dir DIR` | — | Astro site dir — enables post-write incremental rebuilds | | `--host HOST` | `127.0.0.1` | Bind address (keep on localhost; nginx proxies from outside) | | `--port PORT` | `4041` | Bind port | +| `--dem-url URL` | `https://api.open-elevation.com` | Open-Elevation-compatible API for the "Recalculate elevation" button (also `DEM_URL` env var) | Requires `bincio init` to have been run first. Handles auth, user management, and write operations. nginx is responsible for serving static files and proxying `/api/*` to this server. diff --git a/SCHEMA.md b/docs/schema.md similarity index 93% rename from SCHEMA.md rename to docs/schema.md index 0960341..3802199 100644 --- a/SCHEMA.md +++ b/docs/schema.md @@ -124,7 +124,7 @@ needed to render an activity card in a feed — no timeseries, no full track. | `avg_cadence_rpm` | integer\|null | no | Average cadence (rpm for cycling, spm for running). | | `avg_power_w` | integer\|null | no | Average power in watts. | | `source` | string\|null | no | Origin of data. See **Source values**. | -| `privacy` | string | yes | One of: `public`, `blur_start`, `no_gps`, `private`. | +| `privacy` | string | yes | One of: `public`, `blur_start`, `no_gps`, `unlisted`. (`private` is a deprecated alias for `unlisted`.) | | `mmp` | array\|null | no | Mean Maximal Power curve — `[[duration_s, avg_watts], ...]`. | | `best_efforts` | array\|null | no | Best efforts by distance — `[[distance_km, time_s], ...]`. | | `best_climb_m` | number\|null | no | Best single climb in metres (Kadane's algorithm). | @@ -165,12 +165,21 @@ timestamp alone is sufficient: `2024-06-01T073012Z`. ### Privacy levels -| Level | GPS track published | Timeseries lat/lon | Stats in index | +| Level | GPS track published | Timeseries lat/lon | Shown in feed | |---|---|---|---| -| `public` | Full track | Included | Yes | -| `blur_start` | First/last 200 m removed | Trimmed | Yes | -| `no_gps` | Not published | Not included | Yes | -| `private` | Not published | Not included | No (not in index at all) | +| `public` | Full track | Included | Yes — everyone | +| `blur_start` | First/last 200 m removed | Trimmed | Yes — everyone | +| `no_gps` | Not published | Not included | Yes — everyone | +| `unlisted` | Full track | Included | No — owner only (via direct URL) | +| `private` | *(deprecated alias for `unlisted`)* | Included | No — owner only | + +**`unlisted`** activities are not shown in the public feed but are fully accessible +by direct URL — the GPS track, timeseries, and detail JSON are all served as normal +static files. This is "security by obscurity": knowing the URL is sufficient to +access the activity. If you need true data exclusion, use `no_gps` for GPS removal +while keeping stats public, or delete the activity entirely. + +The legacy `private` value is accepted everywhere `unlisted` is valid. --- diff --git a/docs/user-guide.md b/docs/user-guide.md new file mode 100644 index 0000000..5c59210 --- /dev/null +++ b/docs/user-guide.md @@ -0,0 +1,198 @@ +# User Guide + +This guide covers everything you can do as a BincioActivity user. + +## Getting Your Account + +Your instance administrator sends you a registration link: + +``` +https://yourdomain.com/register/?code=ABCD1234 +``` + +Click it and create: + +- **Handle** — your username in URLs (lowercase letters, numbers, `_`, `-`; 1–30 chars) +- **Password** — at least 8 characters +- **Display name** — your full name (shown on your profile page) + +You're now logged in and ready to upload activities! + +## Uploading Activities + +Click **Upload** to add activities from files. + +### Supported formats + +- **GPX** — GPS Exchange format (most common) +- **FIT** — Garmin's native format +- **TCX** — Training Center XML +- **Compressed files** — `.gz` variants of any format above + +### Using Strava export + +If you exported activities from Strava, you likely have a folder like: + +``` +activities/ + ├── 2026-03-15_morning_run.gpx + ├── 2026-03-14_evening_ride.fit + └── ... +``` + +Just drag the whole `activities/` folder into the upload box, or select multiple files at once. + +### Upload options + +- **Store original files** — keep the source GPX/FIT/TCX file on the server (checked by default; you can uncheck per upload) +- **Skip duplicates** — the system detects exact duplicates automatically + +After upload, the server extracts GPS tracks, calculates distance/elevation/time, and generates your activity feed. You can keep uploading — the system deduplicates by file hash. + +## Syncing from Strava + +If your instance supports Strava sync, click **Sync from Strava** in the upload modal. + +1. Authorize BincioActivity to read your Strava data +2. Select which activities to import +3. The server fetches GPS and metrics from Strava and stores them + +Your OAuth token is stored securely on the server. You can revoke access at any time in [Strava Settings](https://www.strava.com/settings/apps). + +## Editing Activities + +Click **Edit** on any activity to: + +- **Change the title** — rename the activity +- **Add a description** — write notes or a story (supports markdown and embedded images) +- **Upload photos** — add photos taken during the activity +- **Choose sport type** — cycling, running, hiking, etc. +- **Assign gear** — tag the bike/shoes/watch used +- **Set privacy** — hide the activity from the public feed +- **Highlight** — mark your favorite activities + +Changes save instantly. The site rebuilds in the background. + +### Recalculating elevation + +If an activity shows an unrealistic elevation gain, the edit drawer has two buttons: + +**📐 Recalculate (hysteresis)** — recomputes gain and loss from the original recorded +elevation using a noise-filtering dead-band algorithm. Fast and offline — no network +call. Best for devices with a barometric altimeter (Garmin, Karoo, Wahoo) whose +elevation data is accurate but was extracted before the noise-filtering was improved. + +**⛰ Recalculate (DEM)** — replaces the recorded GPS altitude with SRTM terrain data +from the [Open-Elevation API](https://open-elevation.com) and recomputes gain and +loss. The elevation chart and summary stats both update. Best for GPS-only devices +(no barometric sensor) where the recorded altitude is noisy. + +> **Note:** Both corrections require a GPS track (activities marked *No GPS* cannot be +> corrected). The DEM option uses ~30 m resolution terrain data; very short or indoor +> activities see little improvement from DEM correction. + +### Photo gallery + +Upload photos for an activity. They appear in a lightbox on the activity detail page. The server stores them in your data directory. + +### Markdown in descriptions + +Descriptions support basic markdown: + +```markdown +# Title +**bold** _italic_ `code` + +- bullet list +- another item + +[link](https://example.com) + +![image name](image.jpg) +``` + +Images are stored in `edits/images/{id}/` and paths are rewritten automatically. + +## Privacy Control + +Each activity has a privacy setting: + +- **Public** (`public: true`) — visible to all logged-in users in the feed +- **Unlisted** (`private: true`) — not shown in the feed, but accessible by direct URL (for sharing) +- **No GPS** (remove GPS track) — hides the map but keeps distance/time stats + +Your instance admin can also make the whole instance public or private. + +### Deleting an activity + +You can't delete an activity directly, but you can: + +- Mark it **private** to hide it from the feed +- Edit the sidecar manually in `{data-root}/edits/{id}.md` and delete the file + +## Your Profile + +Click your name in the top-right to view your profile at `/u/{handle}/`. It shows: + +- Your display name +- All your public activities (organized by year) +- Summary stats (total distance, time, elevation) + +## Account Settings + +Click your name → **Settings** to: + +- **Change password** — update your account password +- **View your handle** — the username used in URLs +- **See your data** — information about what's stored on the server + +If you forget your password, ask your instance administrator to generate a reset code. + +## Feedback + +Found a bug or want to suggest a feature? Click **Feedback** at the bottom of any page to submit a message and optional screenshots. The admin team can see all feedback submissions. + +## Local Activity Conversion + +If your instance has the `/convert/` page enabled, you can: + +1. Upload a GPX/FIT/TCX file **locally in your browser** (no server upload) +2. The file is processed in JavaScript (powered by Pyodide, Python in the browser) +3. You see the activity preview immediately +4. You can then save it to your local browser storage (IndexedDB) or upload it to the server + +This is useful for testing or converting files without uploading them first. + +## Offline Activity Storage (experimental) + +Activities converted locally are stored in your browser's **IndexedDB** (local storage). They: + +- Don't upload to the server +- Persist across browser sessions +- Can be deleted from settings + +This is useful for activities you don't want to publish yet, or for testing before uploading. + +## Frequently Asked Questions + +**Can I download my data?** +Your instance's complete activity feed is at `/u/{handle}/index.json` (the BAS format). You can also ask the admin to copy your data directory directly. + +**Can I transfer activities between instances?** +Yes! Copy the `{handle}/activities/` and `{handle}/edits/` directories to another instance. The system uses content hashing, so you can merge multiple instances. + +**What formats does my activity support?** +BincioActivity extracts GPS tracks, distance, elevation, moving time, average speed, heart rate, power, cadence, and temperature (if available in the source file). + +**Can I share my activities with someone outside my instance?** +Mark activities as **unlisted** (`private: true`). Anyone with the direct URL can view them, even if they're not logged in. + +**How do I delete my account?** +Ask your instance administrator. They can delete your user record from `instance.db`, which removes you from the login system. Your activity data remains for audit, but can be deleted from disk if you request it. + +## See also + +- [Getting Started](getting-started.md) — initial setup +- [API Reference](reference/api.md) — technical details about how data flows +- [BAS Schema](schema.md) — the activity JSON format +- [Admin Guide](admin-guide.md) — for instance admins diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..b443f55 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,66 @@ +site_name: BincioActivity +site_description: Federated, open-source, self-hosted activity stats platform +site_author: Davide Brugali +repo_url: https://github.com/brutsalvadi/bincio-activity +repo_name: brutsalvadi/bincio-activity +edit_uri: edit/main/docs/ + +docs_dir: docs +site_dir: mkdocs-site + +theme: + name: material + palette: + - scheme: light + primary: blue + accent: blue + toggle: + icon: material/lightbulb-outline + name: Switch to dark mode + - scheme: slate + primary: blue + accent: blue + toggle: + icon: material/lightbulb + name: Switch to light mode + features: + - navigation.tabs + - navigation.tabs.sticky + - navigation.top + - search.suggest + - search.highlight + - content.code.copy + - content.code.annotate + +plugins: + - search + +markdown_extensions: + - admonition + - pymdownx.details + - pymdownx.superfences + - pymdownx.highlight: + use_pygments: true + - pymdownx.inlinehilite + - pymdownx.tabbed: + alternate_style: true + - toc: + permalink: true + +nav: + - Home: index.md + - Getting Started: getting-started.md + - User Guide: user-guide.md + - Admin Guide: admin-guide.md + - Deployment: + - Single-user: deployment/single-user.md + - Multi-user: deployment/multi-user.md + - VPS: deployment/vps.md + - Developer: + - Developer Guide: developer-guide.md + - Architecture: architecture.md + - Reference: + - API: reference/api.md + - CLI: reference/cli.md + - Schema: schema.md + - Garmin Disclaimer: garmin_connect_disclaimer.md diff --git a/publish.sh b/publish.sh deleted file mode 100755 index 814b3d9..0000000 --- a/publish.sh +++ /dev/null @@ -1,82 +0,0 @@ -#!/usr/bin/env bash -set -euo pipefail - -REMOTE="github-public" -BRANCH="main" -LOCAL_DIR="$(cd "$(dirname "$0")" && pwd)" -PUBLISH_DIR="${LOCAL_DIR}/publish" -MANIFEST="${PUBLISH_DIR}/manifest" -DRY_RUN=false - -for arg in "$@"; do - case "$arg" in - --dry-run) DRY_RUN=true ;; - *) echo "Unknown argument: $arg"; exit 1 ;; - esac -done - -if ! git -C "$LOCAL_DIR" remote get-url "$REMOTE" &>/dev/null; then - echo "ERROR: remote '${REMOTE}' not found." - echo " git remote add ${REMOTE} https://github.com/brutsalvadi/bincio-activity.git" - exit 1 -fi -if [[ -n "$(git -C "$LOCAL_DIR" status --porcelain --untracked-files=no)" ]]; then - echo "ERROR: uncommitted changes. Commit or stash first." - exit 1 -fi -if [[ ! -f "$MANIFEST" ]]; then - echo "ERROR: manifest not found at ${MANIFEST}" - exit 1 -fi - -# Collect file list AND stage into temp dir before touching git state. -# Both must happen here — `git rm -rf .` removes the manifest itself, -# so it can't be re-read during the orphan branch step. -STAGING="$(mktemp -d)" -trap 'rm -rf "$STAGING"' EXIT -FILES=() - -while IFS= read -r relpath || [[ -n "$relpath" ]]; do - [[ -z "$relpath" || "$relpath" == \#* ]] && continue - override="${PUBLISH_DIR}/${relpath}" - original="${LOCAL_DIR}/${relpath}" - dest="${STAGING}/${relpath}" - mkdir -p "$(dirname "$dest")" - if [[ -f "$override" ]]; then - cp "$override" "$dest" - elif [[ -f "$original" ]]; then - cp "$original" "$dest" - else - echo "ERROR: '${relpath}' in manifest but not found (no override, no original)" - exit 1 - fi - FILES+=("$relpath") -done < "$MANIFEST" - -echo "Files to be published:" -printf ' %s\n' "${FILES[@]}" - -if $DRY_RUN; then - echo "" - echo "Dry run complete. No changes made." - exit 0 -fi - -# Create orphan branch, wipe working tree, restore only manifest files -git -C "$LOCAL_DIR" checkout --orphan _public_tmp -git -C "$LOCAL_DIR" rm -rf . --quiet -cp -r "${STAGING}/." "${LOCAL_DIR}/" - -TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)" -# Add only manifest files — never picks up untracked files outside the manifest -for relpath in "${FILES[@]}"; do - git -C "$LOCAL_DIR" add -- "$relpath" -done -git -C "$LOCAL_DIR" commit -m "Published ${TIMESTAMP}" -git -C "$LOCAL_DIR" push --force "$REMOTE" "HEAD:${BRANCH}" - -git -C "$LOCAL_DIR" checkout main -git -C "$LOCAL_DIR" branch -D _public_tmp - -echo "" -echo "Done: $(git -C "$LOCAL_DIR" remote get-url "$REMOTE")" diff --git a/publish/CLAUDE.md b/publish/CLAUDE.md deleted file mode 100644 index 683fec3..0000000 --- a/publish/CLAUDE.md +++ /dev/null @@ -1,223 +0,0 @@ -# BincioActivity — Context for Claude - -## What this project is - -BincioActivity is a federated, open-source, self-hosted activity stats platform -(think personal Strava). Two-stage pipeline: - -1. **`bincio extract`** (Python): GPX/FIT/TCX → BAS JSON data store -2. **`bincio render`** (Astro/Node): BAS data store → static website - -The BAS (BincioActivity Schema) JSON files are the federation protocol. -Anyone can publish their data as BAS JSON and others can include it. - -## Key design decisions - -- **No database, no server** — everything is static files -- **Python with uv** for the extract stage -- **Astro + Svelte + Tailwind + MapLibre GL + Observable Plot** for the site -- **Haversine** (not geopy) for distance calculations (10x faster) -- **Worker initializer pattern** for ProcessPoolExecutor — large shared data - (strava_lookup dict, known_hashes frozenset) is sent once per worker via - `initializer=`, not once per task -- **BAS activity IDs** always use UTC with Z suffix for URL safety -- **TCX files** from Garmin use both `http://` and `https://` namespace URIs — - parser handles both - -## Your data - -- Source: `~/your-activity-data/` - - `activities/` — Strava export (GPX, FIT, TCX, all with .gz variants) - - Any subdirectories with FIT files from Garmin/Karoo devices - - `activities.csv` — Strava metadata (names, descriptions, gear) -- Extracted output: `~/bincio_data/` (or `/tmp/bincio_test/` for testing) - -Configure input paths in `extract_config.yaml`. - -## Project structure - -``` -bincio/ Python package - extract/ - models.py DataPoint, ParsedActivity, LapData - parsers/ GPX, FIT, TCX parsers + factory - sport.py sport name normalisation - metrics.py haversine-based stats computation (single pass) - timeseries.py downsample to 1Hz, build BAS timeseries object - simplify.py RDP track simplification → GeoJSON - dedup.py exact (hash) + near-duplicate detection - strava_csv.py Strava activities.csv importer - writer.py BAS JSON + GeoJSON writer - config.py extract_config.yaml loader - cli.py `bincio extract` CLI - render/ - cli.py `bincio render` CLI (symlinks data, runs astro build/dev) - merge.py sidecar edit overlay (produces _merged/) - edit/ - cli.py `bincio edit` CLI - server.py FastAPI write API for the edit drawer -schema/ - bas-v1.schema.json JSON Schema for BAS -SCHEMA.md Human-readable BAS spec -site/ Astro project - src/ - layouts/Base.astro - pages/ - index.astro Activity feed (loads index.json client-side) - activity/[id].astro Single activity (SSG, loads detail JSON client-side) - stats/index.astro Heatmap + year totals - components/ - ActivityFeed.svelte Card grid, sport filter, pagination - ActivityDetail.svelte Map + stats + charts + photo gallery - ActivityMap.svelte MapLibre GL (gradient track, linked hover dot) - ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence tabs) - StatsView.svelte Yearly heatmap + totals - EditDrawer.svelte Slide-in edit panel (visible when PUBLIC_EDIT_URL set) - lib/ - types.ts BAS TypeScript types - format.ts formatDistance, formatDuration, sportIcon, etc. -``` - -## How to run - -```bash -# Extract -cd ~/src/bincio_activity -uv run bincio extract --input ~/your-activity-data/activities --output /tmp/bincio_test - -# Site dev server -cd site -ln -sf /tmp/bincio_test/_merged public/data # point at merged output -cp .env.example .env && $EDITOR .env # set BINCIO_DATA_DIR -npm run dev - -# Edit server (optional — enables edit drawer in the site) -uv run bincio edit --data-dir /tmp/bincio_test -# set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env - -# Tests -uv run pytest -``` - -## MapLibre GL + Vite/Astro — known gotchas - -Learnt the hard way during debugging (March 2026): - -- **`maplibregl.workerUrl = ...` is the v3 API and silently no-ops in v4+.** - The v5 API is `maplibregl.setWorkerUrl(url)`, but you don't need it at all in a - normal Vite environment — MapLibre handles the blob worker automatically. - -- **`optimizeDeps: { exclude: ['maplibre-gl'] }` breaks tile loading.** - It prevents Vite from converting MapLibre's UMD bundle to ESM. The UMD bundle - uses AMD `define()` internally; served raw, the tile worker blob fails silently → - black map, no tiles. The correct setting is `include: ['maplibre-gl']`. - -- **`build.target: 'es2022'` (and `optimizeDeps.esbuildOptions.target`) is required.** - MapLibre's dependencies use ES2022 class field syntax. If esbuild downgrades it, - helpers like `__publicField` aren't available inside the serialised worker blob - scope → tile loading fails. This is a known upstream issue (maplibre-gl-js #6680). - -- **Use static imports, not dynamic `await import('maplibre-gl')`, when possible.** - With `client:only="svelte"` in Astro, SSR never runs for the component so there is - no `window is not defined` risk. Static import lets Vite pre-bundle correctly. - -- **Use `client:only="svelte"` (not `client:load`) for the activity detail page.** - `client:load` does SSR + hydration; complex interactive components with MapLibre - can hit hydration mismatch issues. `client:only` mounts fresh on the client only. - -- **MapLibre v5 requires explicit `center` and `zoom` in the Map constructor.** - v4 silently defaulted to `center: [0,0], zoom: 0`. v5 leaves internal projection - state undefined → `Cannot read properties of undefined (reading 'lng')` crashes - on any operation that touches coordinates (markers, resize, render). Always pass - `center` and `zoom` even if you plan to `fitBounds` later. - -- **MapLibre v5 requires `setLngLat()` on markers before `.addTo(map)`.** - v4 tolerated markers without coordinates. v5 calls `Marker._update()` inside - `addTo()`, which needs valid lngLat → same `'lng'` crash. Set a dummy `[0, 0]` - if the real position arrives later (e.g. hover markers). - -## Observable Plot — known gotchas - -- **Curve names are hyphenated, not camelCase.** - Use `"monotone-x"`, not `"monotoneX"`. Plot uses its own curve name registry - (not raw d3 identifiers). Wrong names throw `unknown curve` at runtime. - -The working `astro.config.mjs` Vite section: -```js -vite: { - optimizeDeps: { - include: ['maplibre-gl'], - esbuildOptions: { target: 'es2022' }, - }, - build: { target: 'es2022' }, -}, -``` - -## Activity sidecar edits — design spec - -Users edit activities via **sidecar markdown files** in the data dir. -No database, no server — consistent with the project's static-files-only philosophy. - -### File naming - -``` -~/bincio_data/ - activities/{id}.json ← immutable extract output - edits/{id}.md ← user edits (sidecar) - edits/images/{id}/ ← uploaded photos - _merged/ ← render-time merge output (gitignored-style) -``` - -### Sidecar format - -```markdown ---- -title: "Epic climb up Monte Grappa" -sport: cycling -hide_stats: [cadence] -highlight: true -private: false -gear: "Trek Domane" ---- - -Rode with friends. Legs felt great after the rest week... -``` - -### Editing UX: drawer in Astro + `bincio edit` write API - -- `bincio edit --data-dir ~/bincio_data` starts a FastAPI server on port 4041 -- Set `PUBLIC_EDIT_URL=http://localhost:4041` in `site/.env` to enable the edit button -- Clicking Edit on any activity detail page opens a slide-in drawer -- Saving writes the sidecar and triggers `merge_all()` automatically -- `bincio render` always runs `merge_all()` before build/serve and symlinks `public/data` → `_merged/` - -### `PUBLIC_EDIT_URL` as feature flag - -- **Unset** → no Edit button, normal static site -- **Set** → edit drawer enabled; lives in `site/.env` (gitignored) - -## Known issues / next steps - -- `bincio render` Python CLI is functional but `--watch` mode not yet implemented -- Activity IDs in older test data may use `+0000` format (pre-fix); re-run extract to get `Z` format -- Some activities appear with both untitled and titled IDs (near-dedup timing race) -- Federation (remote data sources) not yet implemented in site -- Friends pages (`/friends/{handle}/`) not yet implemented -- The `site/.env` file is gitignored — copy from `site/.env.example` - -## What "good" looks like (not yet done) - -- [ ] `bincio render` Python CLI wraps `astro build` properly -- [ ] Friends/federation pages in site -- [ ] Personal records page -- [ ] Activity search / full-text filter in feed -- [ ] GitHub Actions template for auto-publish -- [ ] Karoo/Garmin Connect importers beyond Strava -- [x] `bincio.render.merge` — sidecar parser, `_merged/` output, private filter, highlight sort -- [x] `bincio edit` FastAPI write API (GET/POST activity, image upload/delete, triggers merge) -- [x] `EditDrawer.svelte` — slide-in edit UI in the Astro site -- [x] `PUBLIC_EDIT_URL` feature flag -- [x] Markdown rendering in activity description with image path rewriting -- [x] Photo gallery with lightbox on activity detail page -- [ ] `bincio render --watch` incremental rebuild on sidecar/data changes -- [ ] Highlight badge in activity feed cards diff --git a/publish/extract_config.example.yaml b/publish/extract_config.example.yaml deleted file mode 100644 index a02d6f2..0000000 --- a/publish/extract_config.example.yaml +++ /dev/null @@ -1,31 +0,0 @@ -owner: - handle: yourname - display_name: Your Name - -input: - dirs: - - ~/Activities/gpx - - ~/Activities/fit - # Strava bulk export metadata — provides names, descriptions, gear - # metadata_csv: ~/strava_export/activities.csv - -output: - dir: ~/bincio_data - -default_privacy: public - -sensors: - heart_rate: true - cadence: true - temperature: true - power: true - -track: - simplify: rdp - rdp_epsilon: 0.0001 # ~11m at equator - timeseries_hz: 1 # 1 sample/second max - -classifier: - enabled: false # ML activity type classifier (requires scikit-learn extra) - -incremental: true # skip files whose hash hasn't changed since last run diff --git a/publish/manifest b/publish/manifest deleted file mode 100644 index 9e127b4..0000000 --- a/publish/manifest +++ /dev/null @@ -1,72 +0,0 @@ -# BincioActivity — public release manifest -# One relative path per line. -# If publish/ exists, that sanitized version is used instead of the original. - -.gitignore -.python-version -CHANGELOG.md -CHEATSHEET.md -CLAUDE.md -README.md -SCHEMA.md -pyproject.toml -extract_config.example.yaml -schema/bas-v1.schema.json -bincio/__init__.py -bincio/cli.py -bincio/edit/__init__.py -bincio/edit/cli.py -bincio/edit/server.py -bincio/extract/__init__.py -bincio/extract/cli.py -bincio/extract/config.py -bincio/extract/dedup.py -bincio/extract/metrics.py -bincio/extract/models.py -bincio/extract/parsers/__init__.py -bincio/extract/parsers/base.py -bincio/extract/parsers/factory.py -bincio/extract/parsers/fit.py -bincio/extract/parsers/gpx.py -bincio/extract/parsers/tcx.py -bincio/extract/simplify.py -bincio/extract/sport.py -bincio/extract/strava_csv.py -bincio/extract/timeseries.py -bincio/extract/writer.py -bincio/import_/__init__.py -bincio/import_/cli.py -bincio/import_/strava.py -bincio/render/__init__.py -bincio/render/cli.py -bincio/render/merge.py -publish.sh -publish/CLAUDE.md -publish/extract_config.example.yaml -publish/manifest -site/.env.example -site/astro.config.mjs -site/package.json -site/tailwind.config.mjs -site/tsconfig.json -site/src/components/ActivityCharts.svelte -site/src/components/ActivityDetail.svelte -site/src/components/ActivityFeed.svelte -site/src/components/ActivityMap.svelte -site/src/components/AthleteDrawer.svelte -site/src/components/AthleteView.svelte -site/src/components/EditDrawer.svelte -site/src/components/MmpChart.svelte -site/src/components/RecordsView.svelte -site/src/components/StatsView.svelte -site/src/layouts/Base.astro -site/src/lib/format.ts -site/src/lib/types.ts -site/src/pages/activity/[id].astro -site/src/pages/athlete/index.astro -site/src/pages/index.astro -site/src/pages/stats/index.astro -tests/__init__.py -tests/test_merge.py -tests/test_sport.py -tests/test_writer.py diff --git a/pyproject.toml b/pyproject.toml index 34004a5..f9ee2d1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -44,6 +44,10 @@ serve = [ strava = [ "requests>=2.32", ] +garmin = [ + "garminconnect>=0.2", + "cryptography>=42.0", +] dev = [ "pytest>=9.0", "pytest-cov>=5.0", @@ -52,6 +56,9 @@ dev = [ "types-pyyaml", "types-jsonschema", ] +docs = [ + "mkdocs-material>=9.5", +] [project.scripts] bincio = "bincio.cli:main" @@ -64,6 +71,13 @@ dev = [ "mypy>=1.11", "types-pyyaml", "types-jsonschema", + "mkdocs-material>=9.5", + # serve/edit extras pulled in so test_db.py and test_server_imports.py pass in CI + "fastapi>=0.110", + "uvicorn[standard]>=0.29", + "python-multipart>=0.0.9", + "bcrypt>=4.1", + "httpx>=0.28.1", ] [tool.ruff] diff --git a/scripts/bulk_private.py b/scripts/bulk_private.py new file mode 100644 index 0000000..7a33941 --- /dev/null +++ b/scripts/bulk_private.py @@ -0,0 +1,108 @@ +#!/usr/bin/env python3 +""" +Bulk-set activities matching a title pattern to private by writing sidecar files. + +Usage: + uv run python scripts/bulk_private.py --data-dir /var/bincio/data/brut --match "morning walk" "afternoon walk" + + --dry-run Print what would be changed without writing anything. + --handle Subdirectory name (if data-dir is the root, not the user dir). +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +from pathlib import Path + +import yaml + + +def parse_sidecar(path: Path) -> tuple[dict, str]: + text = path.read_text(encoding="utf-8") + if text.startswith("---"): + parts = re.split(r"^---[ \t]*$", text, maxsplit=2, flags=re.MULTILINE) + if len(parts) >= 3: + fm = yaml.safe_load(parts[1]) or {} + return fm, parts[2].strip() + return {}, text.strip() + + +def write_sidecar(path: Path, fm: dict, body: str) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + content = "---\n" + yaml.dump(fm, allow_unicode=True, default_flow_style=False) + "---\n" + if body: + content += "\n" + body + "\n" + path.write_text(content, encoding="utf-8") + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("--data-dir", required=True, help="User data directory (e.g. /var/bincio/data/brut)") + ap.add_argument("--handle", default=None, help="Handle subdir if data-dir is the instance root") + ap.add_argument("--match", nargs="+", required=True, help="Title patterns to match (case-insensitive substring)") + ap.add_argument("--dry-run", action="store_true", help="Print changes without writing") + args = ap.parse_args() + + data_dir = Path(args.data_dir) + if args.handle: + data_dir = data_dir / args.handle + + index_path = data_dir / "index.json" + if not index_path.exists(): + sys.exit(f"ERROR: index.json not found at {index_path}") + + index = json.loads(index_path.read_text(encoding="utf-8")) + activities = index.get("activities", []) + + patterns = [p.lower() for p in args.match] + + matched = [ + a for a in activities + if any(pat in (a.get("title") or "").lower() for pat in patterns) + ] + + if not matched: + print("No activities matched.") + return + + print(f"Found {len(matched)} matching activities:") + edits_dir = data_dir / "edits" + changed = 0 + + for act in matched: + aid = act["id"] + title = act.get("title", "(no title)") + date = act.get("started_at", "")[:10] + sidecar_path = edits_dir / f"{aid}.md" + + # Load existing sidecar if present + if sidecar_path.exists(): + fm, body = parse_sidecar(sidecar_path) + else: + fm, body = {}, "" + + if fm.get("private") is True: + print(f" [already private] {date} {title}") + continue + + print(f" {'[DRY RUN] ' if args.dry_run else ''}→ private {date} {title}") + if not args.dry_run: + fm["private"] = True + write_sidecar(sidecar_path, fm, body) + changed += 1 + + if args.dry_run: + print("\nDry run — nothing written. Re-run without --dry-run to apply.") + else: + print(f"\n{changed} sidecar(s) written.") + if changed: + print("Running merge_all …") + from bincio.render.merge import merge_all + n = merge_all(data_dir) + print(f"merge_all done ({n} sidecar(s) applied).") + + +if __name__ == "__main__": + main() diff --git a/scripts/disk_report.sh b/scripts/disk_report.sh new file mode 100644 index 0000000..a7ce70a --- /dev/null +++ b/scripts/disk_report.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# Bincio VPS disk usage report +# Run on the VPS: bash scripts/disk_report.sh +# Or remotely: ssh root@ 'bash -s' < scripts/disk_report.sh + +DATA=/var/bincio/data +SITE=/var/bincio/site # adjust if your site build lives elsewhere + +hr() { echo; echo "── $* ──────────────────────────────────────"; } + +hr "DISK OVERVIEW" +df -h / | tail -1 | awk '{printf "Used: %s / %s (%s full)\n", $3, $2, $5}' + +hr "BINCIO ROOT" +du -sh /var/bincio/ 2>/dev/null + +hr "DATA ROOT: $DATA" +du -sh "$DATA" 2>/dev/null + +hr "PER-USER BREAKDOWN" +for user_dir in "$DATA"/*/; do + handle=$(basename "$user_dir") + [[ "$handle" == _* ]] && continue # skip _feedback etc. + + total=$(du -sh "$user_dir" 2>/dev/null | cut -f1) + + act=$(du -sh "$user_dir/activities" 2>/dev/null | cut -f1 || echo "—") + merged=$(du -sh "$user_dir/_merged" 2>/dev/null | cut -f1 || echo "—") + edits=$(du -sh "$user_dir/edits" 2>/dev/null | cut -f1 || echo "—") + images=$(du -sh "$user_dir/edits/images" 2>/dev/null | cut -f1 || echo "—") + orig=$(du -sh "$user_dir/originals" 2>/dev/null | cut -f1 || echo "—") + orig_strava=$(du -sh "$user_dir/originals/strava" 2>/dev/null | cut -f1 || echo "—") + orig_fit=$(du -sh "$user_dir/originals" 2>/dev/null) # will count below by extension + + n_act=$(find "$user_dir/activities" -name "*.json" 2>/dev/null | wc -l | tr -d ' ') + n_orig=$(find "$user_dir/originals" -type f 2>/dev/null | wc -l | tr -d ' ') + n_strava=$(find "$user_dir/originals/strava" -name "*.json" 2>/dev/null | wc -l | tr -d ' ') + + echo "" + echo " @$handle (total: $total)" + echo " activities/ $act ($n_act JSON files)" + echo " _merged/ $merged" + echo " edits/ $edits (images: $images)" + echo " originals/ $orig ($n_orig files)" + echo " strava/ $orig_strava ($n_strava JSON)" +done + +hr "FEEDBACK" +du -sh "$DATA/_feedback" 2>/dev/null || echo " (none)" + +hr "SITE BUILD" +du -sh "$SITE" 2>/dev/null || echo " (not found at $SITE)" + +hr "LOGS" +journalctl --disk-usage 2>/dev/null || echo " (journalctl unavailable)" + +hr "LARGEST FILES IN DATA (top 20)" +find "$DATA" -type f -printf '%s\t%p\n' 2>/dev/null \ + | sort -rn | head -20 \ + | awk '{ + size=$1; path=$2; + if (size >= 1048576) printf "%6.1f MB %s\n", size/1048576, path; + else if (size >= 1024) printf "%6.1f KB %s\n", size/1024, path; + else printf "%6d B %s\n", size, path; + }' + +hr "EXTENSION BREAKDOWN IN originals/" +find "$DATA" -path "*/originals/*" -type f 2>/dev/null \ + | sed 's/.*\.//' | sort | uniq -c | sort -rn \ + | awk '{printf " %6d .%s\n", $1, $2}' + +echo diff --git a/scripts/gen_graph.py b/scripts/gen_graph.py new file mode 100644 index 0000000..262d0ca --- /dev/null +++ b/scripts/gen_graph.py @@ -0,0 +1,544 @@ +#!/usr/bin/env python3 +"""Generate architecture graphs for the bincio codebase. + +Outputs: + docs/architecture.mmd — Mermaid source (embeddable in markdown / GitHub) + docs/graph.html — interactive vis.js graph (open in a browser) + +Usage: + uv run python scripts/gen_graph.py + # or just: + python scripts/gen_graph.py +""" + +import json +import re +from pathlib import Path + +ROOT = Path(__file__).parent.parent +SITE_SRC = ROOT / "site" / "src" +DOCS = ROOT / "docs" +DOCS.mkdir(exist_ok=True) + +# ── helpers ─────────────────────────────────────────────────────────────────── + +def read(path: Path) -> str: + try: + return path.read_text(encoding="utf-8") + except Exception: + return "" + + +def short(path: Path, base: Path) -> str: + """Return a short display label for a file path.""" + try: + rel = path.relative_to(base) + except ValueError: + rel = path + parts = rel.parts + # Drop leading site/src/ or bincio/ + if parts[:2] == ("site", "src"): + parts = parts[2:] + elif parts[:1] == ("bincio",): + parts = parts[1:] + name = "/".join(parts) + # Strip index.astro → parent dir + if name.endswith("/index.astro"): + name = name[: -len("/index.astro")] + "/" + return name + + +# ── 1. API routes from server.py ────────────────────────────────────────────── + +def extract_routes(server_path: Path) -> list[dict]: + """Parse @app.{method}("/api/...") decorators.""" + text = read(server_path) + routes = [] + for m in re.finditer( + r'@app\.(get|post|put|patch|delete)\("(/api/[^"]+)"', + text, + re.MULTILINE, + ): + method, path = m.group(1).upper(), m.group(2) + # Find the function name on the next non-blank line + tail = text[m.end():] + fn_m = re.search(r"async def (\w+)", tail[:200]) + fn = fn_m.group(1) if fn_m else "?" + routes.append({"method": method, "path": path, "fn": fn}) + return routes + + +# ── 2. Frontend → API edges ─────────────────────────────────────────────────── + +_FETCH_RE = re.compile(r"""fetch\(\s*[`'"](/api/[^`'"]+)[`'"]""") +_INTERP_RE = re.compile(r"""`[^`]*/api/([^`$\s{]+)""") # template literals + + +def extract_api_calls(file_path: Path) -> list[str]: + """Return all /api/... paths referenced by a frontend file.""" + text = read(file_path) + found = [] + for m in _FETCH_RE.finditer(text): + found.append(m.group(1).split("?")[0]) # strip query string + # Template literals: `/api/admin/users/${h}/rebuild` → /api/admin/users/{h}/rebuild + for m in _INTERP_RE.finditer(text): + raw = "/api/" + m.group(1) + normalised = re.sub(r"\$\{[^}]+\}", "{x}", raw) + found.append(normalised) + return found + + +def normalise_route(path: str, routes: list[dict]) -> str | None: + """Match a raw path like /api/admin/users/brut/rebuild to a known route pattern.""" + for r in routes: + pattern = re.sub(r"\{[^}]+\}", r"[^/]+", re.escape(r["path"])) + "$" + if re.match(pattern, path): + return r["path"] + return path # keep as-is if not matched + + +# ── 3. Component imports (Svelte / Astro) ───────────────────────────────────── + +_IMPORT_SVELTE_RE = re.compile( + r"""import\s+\w+\s+from\s+['"]([^'"]+\.svelte)['"]""" +) +_IMPORT_ASTRO_RE = re.compile( + r"""import\s+\w+\s+from\s+['"]([^'"]+\.astro)['"]""" +) + + +def extract_component_imports(file_path: Path) -> list[Path]: + text = read(file_path) + results = [] + for pattern in (_IMPORT_SVELTE_RE, _IMPORT_ASTRO_RE): + for m in pattern.finditer(text): + ref = m.group(1) + target = (file_path.parent / ref).resolve() + if target.exists(): + results.append(target) + return results + + +# ── 4. Python module imports ────────────────────────────────────────────────── + +_PY_FROM_RE = re.compile(r"^from (bincio\.\S+) import", re.MULTILINE) +_PY_IMP_RE = re.compile(r"^import (bincio\.\S+)", re.MULTILINE) + + +def extract_py_imports(file_path: Path, py_files: list[Path]) -> list[Path]: + text = read(file_path) + modules = set() + for m in _PY_FROM_RE.finditer(text): + modules.add(m.group(1)) + for m in _PY_IMP_RE.finditer(text): + modules.add(m.group(1)) + + results = [] + for mod in modules: + # bincio.serve.db → bincio/serve/db.py + candidate = ROOT / Path(*mod.split(".")).with_suffix(".py") + if candidate.exists() and candidate != file_path: + results.append(candidate) + return results + + +# ── 5. Collect all data ─────────────────────────────────────────────────────── + +def collect() -> dict: + server_path = ROOT / "bincio" / "serve" / "server.py" + routes = extract_routes(server_path) + + # Frontend files + fe_files = list(SITE_SRC.rglob("*.svelte")) + list(SITE_SRC.rglob("*.astro")) + + # Python files (bincio package only) + py_files = [ + p for p in (ROOT / "bincio").rglob("*.py") + if "__pycache__" not in str(p) and p.name != "__init__.py" + ] + + # --- edges: page/component → API endpoint + api_edges = [] # (source_file, route_path) + for f in fe_files: + calls = extract_api_calls(f) + for call in calls: + norm = normalise_route(call, routes) + api_edges.append((f, norm)) + + # --- edges: component imports + comp_edges = [] # (importer_file, imported_file) + for f in fe_files: + for dep in extract_component_imports(f): + comp_edges.append((f, dep)) + + # --- edges: python imports + py_edges = [] # (importer_file, imported_file) + for f in py_files: + for dep in extract_py_imports(f, py_files): + py_edges.append((f, dep)) + + return { + "routes": routes, + "fe_files": fe_files, + "py_files": py_files, + "api_edges": api_edges, + "comp_edges": comp_edges, + "py_edges": py_edges, + } + + +# ── 6. Mermaid output ───────────────────────────────────────────────────────── + +def to_node_id(path: Path) -> str: + return re.sub(r"[^a-zA-Z0-9]", "_", str(path.relative_to(ROOT))) + + +def write_mermaid(data: dict) -> Path: + lines = ["graph LR", ""] + + routes = data["routes"] + + # Subgraph: API endpoints grouped by domain + domains: dict[str, list[dict]] = {} + for r in routes: + parts = r["path"].strip("/").split("/") + domain = parts[1] if len(parts) > 1 else "other" + domains.setdefault(domain, []).append(r) + + lines.append(" subgraph API") + for domain, rs in sorted(domains.items()): + lines.append(f" subgraph api_{domain}[\"{domain}\"]") + for r in rs: + nid = "api_" + re.sub(r"[^a-zA-Z0-9]", "_", r["path"]) + lines.append(f' {nid}["{r["method"]} {r["path"]}"]') + lines.append(" end") + lines.append(" end") + lines.append("") + + # Subgraph: pages + pages = [f for f in data["fe_files"] if "/pages/" in str(f)] + lines.append(" subgraph Pages") + for f in sorted(pages): + nid = to_node_id(f) + label = short(f, ROOT) + lines.append(f' {nid}["{label}"]') + lines.append(" end") + lines.append("") + + # Subgraph: components + comps = [f for f in data["fe_files"] if "/components/" in str(f)] + lines.append(" subgraph Components") + for f in sorted(comps): + nid = to_node_id(f) + label = short(f, ROOT) + lines.append(f' {nid}["{label}"]') + lines.append(" end") + lines.append("") + + # Subgraph: Python modules + py_groups: dict[str, list[Path]] = {} + for f in data["py_files"]: + rel = f.relative_to(ROOT / "bincio") + group = rel.parts[0] if len(rel.parts) > 1 else "root" + py_groups.setdefault(group, []).append(f) + + lines.append(" subgraph Python") + for group, files in sorted(py_groups.items()): + lines.append(f' subgraph py_{group}["{group}"]') + for f in sorted(files): + nid = to_node_id(f) + lines.append(f' {nid}["{f.stem}"]') + lines.append(" end") + lines.append(" end") + lines.append("") + + # Edges: page/component → API + seen = set() + for src, route_path in data["api_edges"]: + src_nid = to_node_id(src) + dst_nid = "api_" + re.sub(r"[^a-zA-Z0-9]", "_", route_path) + edge = f" {src_nid} -->|fetch| {dst_nid}" + if edge not in seen: + lines.append(edge) + seen.add(edge) + + # Edges: component imports + seen_comp = set() + for src, dst in data["comp_edges"]: + src_nid = to_node_id(src) + dst_nid = to_node_id(dst) + edge = f" {src_nid} --> {dst_nid}" + if edge not in seen_comp: + lines.append(edge) + seen_comp.add(edge) + + # Edges: python imports + seen_py = set() + for src, dst in data["py_edges"]: + src_nid = to_node_id(src) + dst_nid = to_node_id(dst) + edge = f" {src_nid} --> {dst_nid}" + if edge not in seen_py: + lines.append(edge) + seen_py.add(edge) + + out = DOCS / "architecture.mmd" + out.write_text("\n".join(lines), encoding="utf-8") + return out + + +# ── 7. vis.js HTML output ───────────────────────────────────────────────────── + +def write_visjs(data: dict) -> Path: + nodes: list[dict] = [] + edges: list[dict] = [] + node_ids: dict[str, int] = {} + + def add_node(key: str, label: str, group: str, title: str = "") -> int: + if key in node_ids: + return node_ids[key] + nid = len(nodes) + node_ids[key] = nid + nodes.append({"id": nid, "label": label, "group": group, "title": title or label}) + return nid + + def add_edge(src_key: str, dst_key: str, label: str = "") -> None: + if src_key not in node_ids or dst_key not in node_ids: + return + e: dict = {"from": node_ids[src_key], "to": node_ids[dst_key], "arrows": "to"} + if label: + e["label"] = label + edges.append(e) + + # API endpoint nodes + for r in data["routes"]: + key = f"api:{r['path']}" + label = f"{r['method']}\n{r['path']}" + add_node(key, label, "api", f"{r['method']} {r['path']} → {r['fn']}()") + + # Frontend file nodes + for f in data["fe_files"]: + key = str(f) + label = f.name.replace("/index.astro", "/").replace("index.astro", f.parent.name + "/") + is_page = "/pages/" in str(f) + is_layout = "/layouts/" in str(f) + group = "page" if is_page else ("layout" if is_layout else "component") + title = short(f, ROOT) + add_node(key, label, group, title) + + # Python module nodes + for f in data["py_files"]: + key = str(f) + rel = f.relative_to(ROOT / "bincio") + group = "py_" + rel.parts[0] if len(rel.parts) > 1 else "py_root" + add_node(key, f.stem, group, str(f.relative_to(ROOT))) + + # Edges: page/component → API + seen = set() + for src, route_path in data["api_edges"]: + src_key = str(src) + dst_key = f"api:{route_path}" + k = (src_key, dst_key) + if k not in seen: + seen.add(k) + add_edge(src_key, dst_key, "fetch") + + # Edges: component imports + seen_comp = set() + for src, dst in data["comp_edges"]: + k = (str(src), str(dst)) + if k not in seen_comp: + seen_comp.add(k) + add_edge(str(src), str(dst)) + + # Edges: python imports + seen_py = set() + for src, dst in data["py_edges"]: + k = (str(src), str(dst)) + if k not in seen_py: + seen_py.add(k) + add_edge(str(src), str(dst)) + + # Group colours for legend + groups = { + "api": {"color": {"background": "#f59e0b", "border": "#d97706"}, "font": {"color": "#000"}}, + "page": {"color": {"background": "#3b82f6", "border": "#2563eb"}, "font": {"color": "#fff"}}, + "component": {"color": {"background": "#8b5cf6", "border": "#7c3aed"}, "font": {"color": "#fff"}}, + "layout": {"color": {"background": "#06b6d4", "border": "#0891b2"}, "font": {"color": "#000"}}, + "py_extract": {"color": {"background": "#22c55e", "border": "#16a34a"}, "font": {"color": "#000"}}, + "py_render": {"color": {"background": "#84cc16", "border": "#65a30d"}, "font": {"color": "#000"}}, + "py_serve": {"color": {"background": "#ef4444", "border": "#dc2626"}, "font": {"color": "#fff"}}, + "py_edit": {"color": {"background": "#f97316", "border": "#ea580c"}, "font": {"color": "#fff"}}, + "py_root": {"color": {"background": "#6b7280", "border": "#4b5563"}, "font": {"color": "#fff"}}, + } + + html = f""" + + + +Bincio — architecture graph + + + + +
+

Bincio architecture

+
+ + + + + + + + +
+ + +
+
+
+ + + + +""" + + out = DOCS / "graph.html" + out.write_text(html, encoding="utf-8") + return out + + +# ── main ────────────────────────────────────────────────────────────────────── + +if __name__ == "__main__": + print("Collecting codebase graph data…") + data = collect() + + r = len(data["routes"]) + f = len(data["fe_files"]) + p = len(data["py_files"]) + ae = len(data["api_edges"]) + ce = len(data["comp_edges"]) + pe = len(data["py_edges"]) + print(f" {r} API routes | {f} frontend files | {p} Python modules") + print(f" {ae} API call edges | {ce} component import edges | {pe} Python import edges") + + mmd = write_mermaid(data) + print(f"\nMermaid → {mmd.relative_to(ROOT)}") + + html = write_visjs(data) + print(f"vis.js → {html.relative_to(ROOT)}") + print("\nOpen docs/graph.html in a browser to explore interactively.") diff --git a/scripts/pull_feedback.sh b/scripts/pull_feedback.sh new file mode 100755 index 0000000..346ec8e --- /dev/null +++ b/scripts/pull_feedback.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Pull user feedback from the VPS into ./feedback/ locally. +# Usage: bash scripts/pull_feedback.sh + +set -e +VPS=${1:?Usage: $0 user@host} +REMOTE=/var/bincio/data/_feedback +LOCAL=$(dirname "$0")/../feedback + +mkdir -p "$LOCAL" + +echo "Syncing feedback from $VPS:$REMOTE → $LOCAL" +rsync -avz --progress "${VPS}:${REMOTE}/" "$LOCAL/" + +echo "" +echo "=== Feedback summary ===" +for f in "$LOCAL"/*.json; do + [[ -f "$f" ]] || continue + handle=$(basename "$f" .json) + count=$(python3 -c "import json,sys; d=json.load(open('$f')); print(len(d) if isinstance(d, list) else 1)" 2>/dev/null || echo "?") + echo " @$handle: $count submission(s)" +done diff --git a/scripts/rebuild.sh b/scripts/rebuild.sh new file mode 100755 index 0000000..c52a34e --- /dev/null +++ b/scripts/rebuild.sh @@ -0,0 +1,26 @@ +#!/bin/bash +set -e + +REPO=/opt/bincio-repo.git +DEPLOY=/opt/bincio +DATA=/var/bincio/data + +echo "--- Syncing Python deps ---" +cd $DEPLOY +~/.local/bin/uv sync --extra serve --extra strava --extra garmin + +echo "--- Syncing JS deps ---" +cd $DEPLOY/site +npm install --silent + +echo "--- Building site ---" +cd $DEPLOY +~/.local/bin/uv run bincio render --data-dir $DATA --site-dir $DEPLOY/site + +echo "--- Copying dist to webroot ---" +rsync -a --delete $DEPLOY/site/dist/ /var/www/bincio/ + +echo "--- Restarting API ---" +systemctl restart bincio || echo "WARNING: bincio service restart failed — check journalctl -u bincio" + +echo "--- Done ---" \ No newline at end of file diff --git a/site/astro.config.mjs b/site/astro.config.mjs index d2c0249..6bd511f 100644 --- a/site/astro.config.mjs +++ b/site/astro.config.mjs @@ -25,6 +25,28 @@ export default defineConfig({ // In production nginx handles this — same pattern, no code change needed. server: { proxy: { + // Both /api/upload and /api/upload/strava-zip return SSE streams in response + // to POST requests. Vite's default proxy buffers the full body before forwarding, + // which breaks streaming and causes EPIPE on long uploads. + // selfHandleResponse + manual pipe sends chunks as they arrive. + '/api/upload': { + target: serveTarget, + changeOrigin: true, + selfHandleResponse: true, + configure: (proxy) => { + proxy.on('proxyRes', (proxyRes, req, res) => { + res.writeHead(proxyRes.statusCode ?? 200, proxyRes.headers); + proxyRes.pipe(res, { end: true }); + }); + proxy.on('error', (err, _req, res) => { + if (err.code === 'EPIPE' || err.code === 'ECONNRESET') return; + if (!res.headersSent) { + res.writeHead(502); + res.end('proxy error'); + } + }); + }, + }, '/api': { target: serveTarget, changeOrigin: true, diff --git a/site/package.json b/site/package.json index 264bc1a..8853b0c 100644 --- a/site/package.json +++ b/site/package.json @@ -22,7 +22,7 @@ "@capacitor/filesystem": "^8.1.2", "@capacitor/geolocation": "^8.2.0", "@capacitor/ios": "^8.3.0", - "@observablehq/plot": "^0.6.0", + "@observablehq/plot": "0.6.17", "@types/dompurify": "^3.0.5", "astro": "^5.0.0", "dompurify": "^3.3.3", diff --git a/site/src/components/ActivityCharts.svelte b/site/src/components/ActivityCharts.svelte index 6f07c5d..d7541ac 100644 --- a/site/src/components/ActivityCharts.svelte +++ b/site/src/components/ActivityCharts.svelte @@ -21,15 +21,20 @@ let chartEl: HTMLDivElement; let chart: SVGElement | null = null; - // Cumulative distance in km, integrated from speed_kmh + // Cumulative distance in km, integrated from speed_kmh. + // Speeds > 150 km/h are treated as 0 (GPS glitch guard) — otherwise a single + // 1-second spike at 220 km/h pushes all subsequent points ~60 m too far right + // on the distance axis and stretches the chart out of proportion. $: dist_km = (() => { if (!timeseries.speed_kmh.some(v => v != null)) return null; - const d: (number | null)[] = [0]; + const d: number[] = [0]; for (let i = 1; i < timeseries.t.length; i++) { const v = timeseries.speed_kmh[i]; const dt = timeseries.t[i] - timeseries.t[i - 1]; const prev = d[i - 1]; - d.push(v != null && prev != null ? prev + v * dt / 3600 : prev); + // Clamp to 150 km/h; treat null or out-of-range as 0 movement + const vSafe = (v != null && v > 0 && v <= 150) ? v : 0; + d.push(prev + vSafe * dt / 3600); } return d; })(); @@ -79,6 +84,15 @@ $: dataMin = metricValues.length ? Math.min(...metricValues) : 0; $: dataMax = metricValues.length ? Math.max(...metricValues) : 100; + // Explicit y domain for the line chart. + // We compute this once from all data and pass it explicitly to Plot so that + // switching x-axis mode (time ↔ distance) never changes the y range — Observable + // Plot auto-infers different domains when the x-channel changes because it only + // considers plottable points, but we want the scale to stay anchored to the + // full dataset. areaY extends down to 0, so include 0 in the minimum. + $: lineDomainMin = Math.min(0, dataMin); + $: lineDomainMax = dataMax; + // Range handles — reset whenever the metric or chart type changes let trimMin = 0; let trimMax = 100; @@ -121,9 +135,27 @@ // Reset when switching away from a zone-capable metric or leaving histogram $: if (!canAlignZones) alignZones = false; + // ── Theme-aware colours ────────────────────────────────────────────────── + function getThemeColors() { + const isDark = document.documentElement.getAttribute('data-theme') !== 'light'; + return { + axis: isDark ? '#71717a' : '#52525b', // zinc-500 / zinc-600 + rule: isDark ? 'rgba(255,255,255,0.25)' : 'rgba(0,0,0,0.2)', + tooltipFg: isDark ? '#ffffff' : '#18181b', + tooltipBg: isDark ? '#09090b' : '#ffffff', // text outline backing + ruleY: isDark ? '#3f3f46' : '#d4d4d8', // baseline rule + }; + } + // ── Rendering ──────────────────────────────────────────────────────────── - onMount(() => { renderChart(); }); - onDestroy(() => { chart?.remove(); chart = null; }); + let themeObserver: MutationObserver | null = null; + + onMount(() => { + renderChart(); + themeObserver = new MutationObserver(() => renderChart()); + themeObserver.observe(document.documentElement, { attributes: true, attributeFilter: ['data-theme'] }); + }); + onDestroy(() => { chart?.remove(); chart = null; themeObserver?.disconnect(); }); $: if (chartEl) { activeTab; xMode; chartType; histData; histThresholds; alignZones; @@ -162,24 +194,33 @@ function renderLine(w: number, h: number, yKey: string, yLabel: string, color: string) { const x = xMode === 'distance' ? 'dist_km' : 't'; + const tc = getThemeColors(); const marks: any[] = []; + // monotone-x requires strictly increasing x. In time mode t is always + // strictly increasing. In distance mode, stopped segments produce many + // consecutive points with identical dist_km, which causes NaN Bézier + // control points and visual artifacts — use linear instead. + const curve = xMode === 'distance' ? 'linear' : 'monotone-x'; + if (activeTab === 'cadence') { - marks.push(Plot.lineY(data, { x, y: yKey, stroke: color, strokeWidth: 1.5, curve: 'monotone-x' })); + marks.push(Plot.lineY(data, { x, y: yKey, stroke: color, strokeWidth: 1.5, curve })); } else { marks.push( - Plot.areaY(data, { x, y: yKey, fill: color, fillOpacity: 0.15, curve: 'monotone-x' }), - Plot.lineY(data, { x, y: yKey, stroke: color, strokeWidth: 1.5, curve: 'monotone-x' }), + Plot.areaY(data, { x, y: yKey, fill: color, fillOpacity: 0.15, curve }), + Plot.lineY(data, { x, y: yKey, stroke: color, strokeWidth: 1.5, curve }), ); } marks.push( - Plot.ruleX(data, Plot.pointerX({ x, stroke: 'rgba(255,255,255,0.3)', strokeWidth: 1, strokeDasharray: '4,4' })), - Plot.dot(data, Plot.pointerX({ x, y: yKey, r: 4, fill: color, stroke: 'white', strokeWidth: 1.5 })), + Plot.ruleX(data, Plot.pointerX({ x, stroke: tc.rule, strokeWidth: 1, strokeDasharray: '4,4' })), + Plot.dot(data, Plot.pointerX({ x, y: yKey, r: 4, fill: color, stroke: tc.tooltipBg, strokeWidth: 1.5 })), Plot.text(data, Plot.pointerX({ x, y: yKey, text: (d: any) => d[yKey] != null ? `${Math.round(d[yKey])}` : '', - dy: -12, fill: 'white', fontSize: 11, fontWeight: '600', + dy: -12, + fill: tc.tooltipFg, stroke: tc.tooltipBg, strokeWidth: 3, + fontSize: 11, fontWeight: '600', })), ); @@ -193,9 +234,9 @@ return Plot.plot({ width: w, height: h, marginLeft: 48, marginBottom: 32, - style: { background: 'transparent', color: 'var(--text-4, #a1a1aa)', fontSize: '11px' }, + style: { background: 'transparent', color: tc.axis, fontSize: '11px' }, x: { label: null, tickFormat: xTickFormat, grid: false, ticks: 6 }, - y: { label: yLabel, grid: true, tickCount: 4 }, + y: { label: yLabel, grid: true, tickCount: 4, domain: [lineDomainMin, lineDomainMax] }, marks, }); } @@ -204,6 +245,7 @@ const yTickFormat = (v: number) => v >= 60 ? `${Math.round(v / 60)}m` : `${v}s`; const rawZones = activeTab === 'hr' ? athlete?.hr_zones : activeTab === 'power' ? athlete?.power_zones : null; const zoneColors = activeTab === 'hr' ? HR_ZONE_COLORS : PWR_ZONE_COLORS; + const tc = getThemeColors(); // ── Zone-aligned: one colored bar per zone ────────────────────────────── if (alignZones && rawZones?.length) { @@ -224,7 +266,7 @@ return Plot.plot({ width: w, height: h, marginLeft: 48, marginBottom: 32, - style: { background: 'transparent', color: 'var(--text-4, #a1a1aa)', fontSize: '11px' }, + style: { background: 'transparent', color: tc.axis, fontSize: '11px' }, x: { label: yLabel, grid: false, domain: [clampedZones[0][0], clampedZones[clampedZones.length - 1][1]] }, y: { label: 'Time', grid: true, tickCount: 4, tickFormat: yTickFormat }, marks: [ @@ -240,7 +282,7 @@ fontSize: 10, fontWeight: '600', dy: -8, }), - Plot.ruleY([0], { stroke: '#52525b' }), + Plot.ruleY([0], { stroke: tc.ruleY }), ], }); } @@ -251,7 +293,7 @@ { y: 'count' }, { x: yKey, fill: color, fillOpacity: 0.7, thresholds: histThresholds }, )), - Plot.ruleY([0], { stroke: '#52525b' }), + Plot.ruleY([0], { stroke: tc.ruleY }), ]; if (rawZones?.length) { @@ -282,7 +324,7 @@ return Plot.plot({ width: w, height: h, marginLeft: 48, marginBottom: 32, - style: { background: 'transparent', color: 'var(--text-4, #a1a1aa)', fontSize: '11px' }, + style: { background: 'transparent', color: tc.axis, fontSize: '11px' }, x: { label: yLabel, grid: false, ticks: 6, domain: [trimMin, trimMax] }, y: { label: 'Time', grid: true, tickCount: 4, tickFormat: yTickFormat }, marks, @@ -347,7 +389,7 @@ -
+
{#if chartType === 'histogram'} diff --git a/site/src/components/ActivityDetail.svelte b/site/src/components/ActivityDetail.svelte index 18bae3d..38c98b8 100644 --- a/site/src/components/ActivityDetail.svelte +++ b/site/src/components/ActivityDetail.svelte @@ -54,7 +54,7 @@ } $: trackUrl = activity.track_url - ? (activity.track_url.startsWith('http') ? activity.track_url : `${base}data/${activity.track_url}`) + ? (activity.track_url.startsWith('http') || activity.track_url.startsWith('/') ? activity.track_url : `${base}data/${activity.track_url}`) : null; $: color = sportColor(activity.sport); @@ -70,22 +70,31 @@ $: rawDescription = localDescription || detail?.description || ''; $: descriptionHtml = (() => { if (!rawDescription) return ''; + // Strip local image refs before marked sees them. marked only parses ![alt](url) as an + // image when the URL has no spaces — filenames like "WhatsApp Image 2026.jpg" are left + // as literal text instead. The lazy .*? anchored to the image extension handles filenames + // with spaces and nested parens (e.g. "file(2).jpg") correctly. + const stripped = rawDescription + .replace(/!\[[^\]]*\]\((?!https?:\/\/|\/|data:).*?\.(?:jpe?g|png|gif|webp|bmp|avif|heic)\)/gi, '') + .trim(); + if (!stripped) return ''; const renderer = new marked.Renderer(); - // Local relative images are always shown in the gallery — suppress inline rendering + // Any remaining remote images render inline; local ones (shouldn't exist after strip) are suppressed renderer.image = ({ href, title, text }) => { const isLocal = href && !href.startsWith('http') && !href.startsWith('/') && !href.startsWith('data:'); if (isLocal) return ''; const titleAttr = title ? ` title="${title}"` : ''; return `${text}`; }; - return DOMPurify.sanitize(marked(rawDescription, { renderer }) as string); + return DOMPurify.sanitize(marked(stripped, { renderer }) as string); })(); // Derive image dir from detail_url so multi-user paths resolve correctly. - // "dave/_merged/activities/foo.json" → "/data/dave/_merged/activities/images/{id}/" + // Relative: "dave/_merged/activities/foo.json" → "/data/dave/_merged/activities/images/{id}/" + // Absolute: "/data/dave/_merged/activities/foo.json" → "/data/dave/_merged/activities/images/{id}/" $: imageBase = (() => { const du = activity.detail_url ?? ''; - const dir = du.startsWith('http') + const dir = du.startsWith('http') || du.startsWith('/') ? du.substring(0, du.lastIndexOf('/') + 1) : du.includes('/') ? `${base}data/${du.substring(0, du.lastIndexOf('/') + 1)}` @@ -112,7 +121,7 @@ {#if editOpen && editEnabled} - editOpen = false} /> + editOpen = false} on:deleted={() => { window.location.href = base; }} /> {/if} @@ -244,6 +253,7 @@ {trackUrl} {timeseries} bbox={detail?.bbox ?? null} + initialCoords={activity.preview_coords} accentColor={color} bind:hoveredIdx /> diff --git a/site/src/components/ActivityDetailLoader.svelte b/site/src/components/ActivityDetailLoader.svelte new file mode 100644 index 0000000..91cd3f9 --- /dev/null +++ b/site/src/components/ActivityDetailLoader.svelte @@ -0,0 +1,123 @@ + + +{#if loading} +

Loading activity…

+{:else if notFound} +
+

Activity not found.

+

It may still be processing — try refreshing in a moment.

+ ← Back to feed +
+{:else if activity} + +{/if} diff --git a/site/src/components/ActivityFeed.svelte b/site/src/components/ActivityFeed.svelte index ffc330e..77f4e56 100644 --- a/site/src/components/ActivityFeed.svelte +++ b/site/src/components/ActivityFeed.svelte @@ -1,8 +1,8 @@ + +
+
+

Community

+

What everyone's been up to — together.

+
+ + {#if loading} +

Loading…

+ {:else if error} +

{error}

+ {:else} + + +
+ {#each PERIODS as p} + + {/each} +
+ + + {#if totals.users > 0} +
+ {#each [ + { label: 'Activities', value: totals.count.toLocaleString() }, + { label: 'Distance', value: formatDistance(totals.distance_m) }, + { label: 'Elevation', value: `${Math.round(totals.elevation_m / 1000).toLocaleString()} km↑` }, + { label: 'Time', value: formatDuration(totals.duration_s) }, + ] as item} +
+
{item.value}
+
{item.label}
+
+ {/each} +
+ {/if} + + + {#if totals.users === 0} +

No public activities in this period yet.

+ {:else} +
+ + + + + + + + + + + + + + + {#each sorted as u, i} + + + + + + + + + + + {/each} + +
# + + + + + + + + + +
{i + 1} + + {u.display_name} + + @{u.handle} + {u.count}{u.distance_m > 0 ? formatDistance(u.distance_m) : '—'}{u.elevation_m > 0 ? `${Math.round(u.elevation_m).toLocaleString()} m` : '—'}{u.duration_s > 0 ? formatDuration(u.duration_s) : '—'}
+
+ {/if} + + {/if} +
diff --git a/site/src/components/EditDrawer.svelte b/site/src/components/EditDrawer.svelte index 9397030..985cef0 100644 --- a/site/src/components/EditDrawer.svelte +++ b/site/src/components/EditDrawer.svelte @@ -5,7 +5,7 @@ export let activityId: string; export let editUrl: string; - const dispatch = createEventDispatcher<{ saved: { title: string; description: string }; close: void }>(); + const dispatch = createEventDispatcher<{ saved: { title: string; description: string }; close: void; deleted: void }>(); const SPORTS: Sport[] = ['cycling', 'running', 'hiking', 'walking', 'swimming', 'skiing', 'other']; const STAT_PANELS = [ @@ -21,6 +21,13 @@ let saving = false; let saveStatus = ''; let saveOk = false; + let confirmDelete = false; + let deleting = false; + + // Elevation recalculation + let recalculating: '' | 'dem' | 'hysteresis' = ''; + let recalcStatus = ''; + let recalcOk = false; // Form state let title = ''; @@ -49,9 +56,12 @@ title = d.title ?? ''; sport = d.sport ?? 'cycling'; gear = d.gear ?? ''; - description = d.description ?? ''; + // Strip any auto-inserted image markdown refs — images are tracked via custom.images + description = (d.description ?? '').replace(/!\[[^\]]*\]\([^)]+\)\n?/g, '').trim(); highlight = d.highlight ?? false; - isPrivate = d.private ?? false; + // d.private is a bool (from the API); d.privacy is the raw field on older + // endpoints. Accept either so the drawer works with both serve and edit servers. + isPrivate = d.private ?? (d.privacy === 'unlisted' || d.privacy === 'private') ?? false; hideStats = d.hide_stats ?? []; images = d.images ?? []; } catch (e: any) { @@ -93,9 +103,6 @@ if (res.ok) { const d = await res.json(); if (!images.includes(d.filename)) images = [...images, d.filename]; - // Insert markdown reference at cursor or end - const ref = `\n![${d.filename.replace(/\.[^.]+$/, '')}](${d.filename})`; - description = description.trimEnd() + ref; } } } catch (e: any) { @@ -109,9 +116,6 @@ async function deleteImage(filename: string) { await fetch(`${api}/images/${encodeURIComponent(filename)}`, { method: 'DELETE' }); images = images.filter(f => f !== filename); - // Remove the markdown reference — escape filename before using in regex - const escaped = filename.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - description = description.replace(new RegExp(`!\\[[^\\]]*\\]\\(${escaped}\\)`, 'g'), '').trim(); } function toggleStat(key: string) { @@ -120,6 +124,42 @@ : [...hideStats, key]; } + async function recalculateElevation(method: 'dem' | 'hysteresis') { + recalculating = method; + recalcStatus = ''; + recalcOk = false; + try { + const res = await fetch(`${api}/recalculate-elevation/${method}`, { method: 'POST' }); + const d = await res.json(); + if (!res.ok) throw new Error(d.detail ?? await res.text()); + recalcOk = true; + const gain = d.elevation_gain_m != null ? `↑ ${Math.round(d.elevation_gain_m)} m` : ''; + const loss = d.elevation_loss_m != null ? `↓ ${Math.round(d.elevation_loss_m)} m` : ''; + recalcStatus = [gain, loss].filter(Boolean).join(' '); + } catch (e: any) { + recalcStatus = e.message; + recalcOk = false; + } finally { + recalculating = ''; + } + } + + async function deleteActivity() { + if (!confirmDelete) { confirmDelete = true; return; } + deleting = true; + try { + const res = await fetch(api, { method: 'DELETE' }); + if (!res.ok) throw new Error(await res.text()); + dispatch('deleted'); + } catch (e: any) { + saveStatus = `Delete failed: ${e.message}`; + saveOk = false; + confirmDelete = false; + } finally { + deleting = false; + } + } + load(); @@ -249,6 +289,36 @@ + +
+

Elevation

+
+ + +
+ {#if recalcStatus} +

+ {recalcStatus} +

+ {/if} +
+
{/if} @@ -284,11 +354,26 @@
+ {#if saveStatus} {saveStatus} diff --git a/site/src/components/MmpChart.svelte b/site/src/components/MmpChart.svelte index 4a9efcb..cbdb4bf 100644 --- a/site/src/components/MmpChart.svelte +++ b/site/src/components/MmpChart.svelte @@ -82,6 +82,10 @@ $: colorMap = Object.fromEntries(selectedKeys.map((k, i) => [k, curveColor(k, i)])); + function getAxisColor() { + return document.documentElement.getAttribute('data-theme') === 'light' ? '#52525b' : '#a1a1aa'; + } + function renderChart(data: typeof plotData, cmap: typeof colorMap) { if (!chartEl) return; chartEl.innerHTML = ''; @@ -95,16 +99,18 @@ height: 320, marginLeft: 52, marginBottom: 40, - style: { background: 'transparent', color: '#e4e4e7' }, + style: { background: 'transparent', color: getAxisColor() }, x: { type: 'log', label: 'Duration', tickFormat: (d: number) => formatDuration(d), grid: true, + domain: [data[0]?.d ?? 1, Math.max(7200, ...data.map(d => d.d))], }, y: { label: 'Avg power (W)', grid: true, + zero: true, }, color: { domain: selectedKeys, @@ -158,7 +164,9 @@ onMount(() => { const ro = new ResizeObserver(() => renderChart(currentPlotData, currentColorMap)); ro.observe(chartEl); - return () => ro.disconnect(); + const mo = new MutationObserver(() => renderChart(currentPlotData, currentColorMap)); + mo.observe(document.documentElement, { attributes: true, attributeFilter: ['data-theme'] }); + return () => { ro.disconnect(); mo.disconnect(); }; }); // ── Toggle helpers ───────────────────────────────────────────────────────── @@ -179,6 +187,11 @@ ]; + +
{#each allRangeKeys as key, i} diff --git a/site/src/components/StatsView.svelte b/site/src/components/StatsView.svelte index 72a9de0..5c9386a 100644 --- a/site/src/components/StatsView.svelte +++ b/site/src/components/StatsView.svelte @@ -1,7 +1,7 @@ diff --git a/site/src/pages/about/es/index.astro b/site/src/pages/about/es/index.astro new file mode 100644 index 0000000..01bb4b7 --- /dev/null +++ b/site/src/pages/about/es/index.astro @@ -0,0 +1,238 @@ +--- +import Base from '../../../layouts/Base.astro'; +const baseUrl = import.meta.env.BASE_URL ?? '/'; +const labels = { + community: 'Comunidad', + members: 'miembro', + members_pl: 'miembros', + day: 'día', + days: 'días', + invited_by: 'invitado por', + founder: 'fundador', +}; +--- + +
+
+

Acerca de BincioActivity

+
+ EN + IT + ES + CA +
+
+

Seguimiento de actividades open-source y autoalojado

+
+ + ☕ Apoya en Ko-fi + + +
+ +
+ + + +
+

¿Qué es esto?

+

+ BincioActivity es una plataforma gratuita y de código abierto para registrar tus + actividades al aire libre: ciclismo, running, senderismo y más. Está diseñada para + ser autoalojada: tú (o alguien de confianza) gestionas el servidor, y tus datos + permanecen bajo tu control. +

+

+ Las actividades se almacenan en un formato JSON abierto llamado BAS (BincioActivity Schema), + diseñado para ser legible y portable. La plataforma no tiene analíticas ocultas, + no incluye publicidad y no comparte datos con terceros. +

+
+ +
+

Registro e invitaciones

+

+ Esta instancia es solo por invitación. Para registrarte necesitas un enlace de + invitación de un miembro existente — cada enlace es de un solo uso y está vinculado + a un código único. +

+

+ Una vez que tengas una cuenta, puedes generar hasta 3 enlaces de invitación para + compartir con personas de confianza. Gestiona tus invitaciones desde la página de invitaciones + (requiere inicio de sesión). +

+
+ +
+

Tus datos en este servidor

+

+ Cuando subes un archivo FIT, GPX o TCX, el servidor lo convierte al formato BAS. + Por defecto, el archivo fuente original también se guarda en la carpeta + originals/ de tu cuenta. + Puedes desactivar esta opción en el momento de la subida desmarcando + "Conservar el archivo original en el servidor". +

+

+ Se recomienda conservar los originales durante estas primeras etapas del proyecto: + si la cadena de procesamiento mejora (mejor suavizado de elevación, cálculo de velocidad, + detección de vueltas, etc.) podrás volver a importar tus archivos para aprovechar los + cambios. Si elegiste no conservar los originales, tendrías que subir los archivos + de nuevo manualmente. +

+

+ Al sincronizar con Strava, los datos brutos obtenidos de la API de Strava también + pueden almacenarse localmente. Esto lo controla una configuración global del servidor + establecida por el operador. +

+
+ +
+

Software en fase temprana

+

+ BincioActivity está en desarrollo activo. El formato de datos, la cadena de procesamiento + y la API del servidor pueden cambiar entre versiones. Los cambios incompatibles son + posibles, especialmente en esta etapa. Cuando ocurran, volver a importar los archivos + originales es la forma más segura de actualizar tus datos. +

+

+ No existe ninguna garantía de disponibilidad, integridad de datos ni compatibilidad + futura para ninguna versión en particular. Usa este software bajo tu propia + responsabilidad y mantén tus propias copias de seguridad de los datos importantes. +

+
+ +
+

Descargo de responsabilidad

+

+ BincioActivity se proporciona "tal cual", sin + garantía de ningún tipo. Los autores y operadores del servidor no aceptan ninguna + responsabilidad por: +

+
    +
  • Pérdida, corrupción o acceso no autorizado a tus datos de actividad
    • +
  • Datos expuestos por una mala configuración del servidor o la infraestructura
    • +
  • Inexactitudes en las estadísticas calculadas (distancia, elevación, frecuencia cardíaca, etc.)
    • +
  • Cualquier consecuencia derivada de actuar sobre la información mostrada por esta aplicación
    • +
+

+ Eres responsable de proteger tu cuenta con una contraseña segura, de revisar qué + datos compartes y de realizar tus propias copias de seguridad. Los datos de GPS y + salud pueden ser sensibles — reflexiona sobre qué subes y quién puede verlo. +

+
+ +
+

Código abierto

+

+ BincioActivity es software de código abierto. Eres libre de inspeccionar el código, + alojar tu propia instancia y contribuir con mejoras. +

+
+ +
+
+ + + diff --git a/site/src/pages/about/index.astro b/site/src/pages/about/index.astro new file mode 100644 index 0000000..2b43f9d --- /dev/null +++ b/site/src/pages/about/index.astro @@ -0,0 +1,246 @@ +--- +import Base from '../../layouts/Base.astro'; +const baseUrl = import.meta.env.BASE_URL ?? '/'; +const labels = { + community: 'Community', + members: 'member', + members_pl: 'members', + day: 'day', + days: 'days', + invited_by: 'invited by', + founder: 'founder', + loading: 'Loading…', +}; +--- + +
+
+

About BincioActivity

+
+ EN + IT + ES + CA +
+
+

Open-source, self-hosted activity tracking

+
+ + ☕ Support on Ko-fi + + +
+ +
+ + + + +
+

What is this?

+

+ BincioActivity is a free, open-source platform for tracking your outdoor activities — + cycling, running, hiking, and more. It is designed to be self-hosted: you (or someone + you trust) run the server, and your data stays under your control. +

+

+ Activities are stored in an open JSON format called BAS (BincioActivity Schema), + which is designed to be readable and portable. The platform has no hidden analytics, + no advertising, and no third-party data sharing. +

+
+ +
+

Joining & invitations

+

+ This instance is invite-only. To join, you need an invite link from an existing + member — each link is single-use and tied to a unique code. +

+

+ Once you have an account, you can generate up to 3 invite links to + share with people you trust. You can manage your invites from the invites page + (requires login). +

+
+ +
+

Your data on this server

+

+ When you upload a FIT, GPX, or TCX file, the server converts it to BAS format. + By default the original source file is also kept in your account's + originals/ folder. + You can opt out of this at upload time by unchecking "Keep original file on server". +

+

+ Keeping originals is recommended during these early stages of the project: if the + processing pipeline improves (better elevation smoothing, speed calculation, lap + detection, etc.) you can re-import your files to take advantage of the changes. + If you chose not to keep originals, you would need to upload the files again manually. +

+

+ When syncing from Strava, the raw activity data fetched from the Strava API can + similarly be stored locally. This is controlled by an instance-wide setting + configured by the server operator. +

+
+ +
+

Early-stage software

+

+ BincioActivity is under active development. The data format, processing pipeline, + and server API may change between versions. Breaking changes are possible, especially + at this stage. When they occur, re-importing your original files is the safest way + to bring your data up to date. +

+

+ There is no guarantee of uptime, data integrity, or forward compatibility for + any particular version. Use this software at your own risk, and keep your own + backups of important data. +

+
+ +
+

Disclaimer

+

+ BincioActivity is provided "as is", without + warranty of any kind. The authors and server operators accept no responsibility for: +

+
    +
  • Loss, corruption, or unauthorised access to your activity data
    • +
  • Data exposed through misconfiguration of the server or infrastructure
    • +
  • Inaccuracies in computed statistics (distance, elevation, heart rate, etc.)
    • +
  • Any consequences of acting on information displayed by this application
    • +
+

+ You are responsible for securing your account with a strong password, reviewing + what data you share, and making your own backups. GPS and health data can be + sensitive — think carefully about what you upload and who can see it. +

+
+ +
+

Open source

+

+ BincioActivity is open-source software. You are free to inspect the code, + self-host your own instance, and contribute improvements. +

+
+ +
+
+ + + diff --git a/site/src/pages/about/it/index.astro b/site/src/pages/about/it/index.astro new file mode 100644 index 0000000..1dce80f --- /dev/null +++ b/site/src/pages/about/it/index.astro @@ -0,0 +1,238 @@ +--- +import Base from '../../../layouts/Base.astro'; +const baseUrl = import.meta.env.BASE_URL ?? '/'; +const labels = { + community: 'Comunità', + members: 'membro', + members_pl: 'membri', + day: 'giorno', + days: 'giorni', + invited_by: 'invitato da', + founder: 'fondatore', +}; +--- + +
+
+

Informazioni su BincioActivity

+
+ EN + IT + ES + CA +
+
+

Tracciamento attività open-source e self-hosted

+
+ + ☕ Supporta su Ko-fi + + +
+ +
+ + + +
+

Cos'è?

+

+ BincioActivity è una piattaforma gratuita e open-source per tracciare le tue attività + all'aperto — ciclismo, corsa, escursionismo e altro. È progettata per essere + self-hosted: tu (o qualcuno di cui ti fidi) gestisci il server, e i tuoi dati + rimangono sotto il tuo controllo. +

+

+ Le attività vengono salvate in un formato JSON aperto chiamato BAS (BincioActivity Schema), + progettato per essere leggibile e portabile. La piattaforma non ha analytics nascosti, + nessuna pubblicità e nessuna condivisione di dati con terze parti. +

+
+ +
+

Iscrizione e inviti

+

+ Questa istanza è accessibile solo su invito. Per registrarti hai bisogno di un link + di invito da parte di un membro già registrato — ogni link è monouso e associato a + un codice univoco. +

+

+ Una volta registrato, puoi generare fino a 3 link di invito da + condividere con persone di fiducia. Gestisci i tuoi inviti dalla pagina inviti + (richiede il login). +

+
+ +
+

I tuoi dati su questo server

+

+ Quando carichi un file FIT, GPX o TCX, il server lo converte nel formato BAS. + Di default, il file sorgente originale viene conservato nella cartella + originals/ del tuo account. + Puoi disattivare questa opzione al momento del caricamento deselezionando + "Mantieni il file originale sul server". +

+

+ Conservare i file originali è consigliato in questa fase iniziale del progetto: se la + pipeline di elaborazione migliorasse (migliore smoothing del dislivello, calcolo della + velocità, rilevamento dei giri, ecc.) potrai reimportare i file per beneficiare delle + modifiche. Se hai scelto di non conservare gli originali, dovrai ricaricare i file + manualmente. +

+

+ Durante la sincronizzazione con Strava, i dati grezzi dell'attività ottenuti dall'API + Strava possono essere conservati localmente. Questo è controllato da un'impostazione + a livello di istanza configurata dall'operatore del server. +

+
+ +
+

Software in fase iniziale

+

+ BincioActivity è in sviluppo attivo. Il formato dei dati, la pipeline di elaborazione + e le API del server potrebbero cambiare tra una versione e l'altra. Modifiche + incompatibili sono possibili, soprattutto in questa fase. Quando si verificano, + reimportare i file originali è il modo più sicuro per aggiornare i propri dati. +

+

+ Non vi è alcuna garanzia di uptime, integrità dei dati o compatibilità futura per + nessuna versione specifica. Usa questo software a tuo rischio e pericolo, e conserva + sempre i tuoi backup dei dati importanti. +

+
+ +
+

Limitazione di responsabilità

+

+ BincioActivity è fornito "così com'è", senza + garanzie di alcun tipo. Gli autori e gli operatori del server non si assumono alcuna + responsabilità per: +

+
    +
  • Perdita, corruzione o accesso non autorizzato ai tuoi dati di attività
    • +
  • Dati esposti a causa di una configurazione errata del server o dell'infrastruttura
    • +
  • Imprecisioni nelle statistiche calcolate (distanza, dislivello, frequenza cardiaca, ecc.)
    • +
  • Qualsiasi conseguenza derivante dall'utilizzo delle informazioni visualizzate dall'applicazione
    • +
+

+ Sei responsabile di proteggere il tuo account con una password robusta, di verificare + quali dati condividi e di eseguire i tuoi backup. I dati GPS e sanitari possono essere + sensibili — rifletti attentamente su cosa carichi e su chi può vederlo. +

+
+ +
+

Open source

+

+ BincioActivity è software open-source. Sei libero di esaminare il codice, + ospitare la tua istanza e contribuire con miglioramenti. +

+
+ +
+
+ + + diff --git a/site/src/pages/activity/[id].astro b/site/src/pages/activity/[id].astro index 273c53d..cffc987 100644 --- a/site/src/pages/activity/[id].astro +++ b/site/src/pages/activity/[id].astro @@ -6,6 +6,15 @@ import ActivityDetail from '../../components/ActivityDetail.svelte'; import type { BASIndex, ActivitySummary, AthleteZones } from '../../lib/types'; export async function getStaticPaths() { + // Activity pages are not pre-built — all /activity/{id}/ URLs are served + // by the activity/index.html shell via nginx try_files and loaded dynamically + // by ActivityDetailLoader. Pre-building thousands of pages at build time + // exhausts server memory. The shell handles public, unlisted, and local + // activities identically with no loss of functionality. + return []; + + // Dead code below — kept for reference only. + /* eslint-disable no-unreachable */ try { const candidates = [ process.env.BINCIO_DATA_DIR, @@ -52,7 +61,7 @@ export async function getStaticPaths() { // Build the map from the index first const byId = new Map( activities - .filter(a => a.privacy !== 'private' && a.id) + .filter(a => a.privacy !== 'private' && a.privacy !== 'unlisted' && a.id) .map(a => [a.id, { activity: a, athlete }]) ); @@ -80,7 +89,7 @@ export async function getStaticPaths() { if (byId.has(id)) continue; // already covered by the index try { const detail = JSON.parse(readFileSync(join(actsDir, file), 'utf-8')); - if (detail.privacy === 'private') continue; + if (detail.privacy === 'private' || detail.privacy === 'unlisted') continue; // Build a minimal ActivitySummary from the detail file const a: ActivitySummary = { id, @@ -119,6 +128,7 @@ export async function getStaticPaths() { } catch { return []; } + /* eslint-enable no-unreachable */ } const { activity, athlete } = Astro.props as { activity: ActivitySummary; athlete: AthleteZones | null }; diff --git a/site/src/pages/activity/index.astro b/site/src/pages/activity/index.astro new file mode 100644 index 0000000..453a95d --- /dev/null +++ b/site/src/pages/activity/index.astro @@ -0,0 +1,7 @@ +--- +import Base from '../../layouts/Base.astro'; +import ActivityDetailLoader from '../../components/ActivityDetailLoader.svelte'; +--- + + + diff --git a/site/src/pages/admin/index.astro b/site/src/pages/admin/index.astro new file mode 100644 index 0000000..25124ff --- /dev/null +++ b/site/src/pages/admin/index.astro @@ -0,0 +1,397 @@ +--- +import Base from '../../layouts/Base.astro'; +--- + +
+

Admin

+ + +
+

Loading disk info…

+
+ + +

Users

+
+ + + + + + + + + + + + + + + +
HandleTotalActivitiesOriginalsMergedImages
Loading…
+
+ + + +
+

Re-extract from Strava originals —

+ +
+
+
+ + + + +
+

Data directory snapshot —

+ +
+ 

+    
+
+    
+    
+      
Reset all data for ?

+      
Removes all activities, originals, edits, and images. The account is kept. This cannot be undone.

+      

+        
+        
+      

+    
+  

+
+
+
diff --git a/site/src/pages/athlete/index.astro b/site/src/pages/athlete/index.astro
index 58865d5..3f56408 100644
--- a/site/src/pages/athlete/index.astro
+++ b/site/src/pages/athlete/index.astro
@@ -13,5 +13,5 @@ const handle = shards[0]?.handle ?? null;
     window.location.replace(base + 'u/' + handle + '/athlete/');
   
 ) : (
-  
No data found. Run bincio extract first.

+  
No data found. Upload activities to get started.

 )}
diff --git a/site/src/pages/community/index.astro b/site/src/pages/community/index.astro
new file mode 100644
index 0000000..c7a2d72
--- /dev/null
+++ b/site/src/pages/community/index.astro
@@ -0,0 +1,7 @@
+---
+import Base from '../../layouts/Base.astro';
+import CommunityView from '../../components/CommunityView.svelte';
+---
+
+  
+
diff --git a/site/src/pages/feedback/index.astro b/site/src/pages/feedback/index.astro
new file mode 100644
index 0000000..df401b0
--- /dev/null
+++ b/site/src/pages/feedback/index.astro
@@ -0,0 +1,162 @@
+---
+import Base from '../../layouts/Base.astro';
+---
+
+  

+    
Send feedback

+    
Report a bug, suggest a feature, or share anything useful. Plain text only — no account details needed.

+
+    

+
+      

+        
+      

+
+      
+      

+        
Attach up to 3 screenshots (max 2 MB each)

+        

+          Drop images or click to browse
+          
+        

+        

+      

+
+      
+
+      
+
+    

+
+    
+  

+
+
+
diff --git a/site/src/pages/index.astro b/site/src/pages/index.astro
index eac5563..2adc560 100644
--- a/site/src/pages/index.astro
+++ b/site/src/pages/index.astro
@@ -1,11 +1,11 @@
 ---
 import Base from '../layouts/Base.astro';
 import ActivityFeed from '../components/ActivityFeed.svelte';
-import { readShardHandles } from '../lib/manifest';
+import { readShardHandles, isInstancePrivate } from '../lib/manifest';
 
 const base = import.meta.env.BASE_URL;
 const shards = readShardHandles();
-const isSingleUser = shards.length === 1;
+const isSingleUser = shards.length === 1 && !isInstancePrivate();
 const singleHandle = isSingleUser ? shards[0].handle : null;
 ---
 {isSingleUser ? (
diff --git a/site/src/pages/invites/index.astro b/site/src/pages/invites/index.astro
index 0262565..f8e76c2 100644
--- a/site/src/pages/invites/index.astro
+++ b/site/src/pages/invites/index.astro
@@ -47,6 +47,17 @@ import Base from '../../layouts/Base.astro';
     return li;
   }
 
+  function fallbackCopy(text: string, done: () => void) {
+    const ta = document.createElement('textarea');
+    ta.value = text;
+    ta.style.cssText = 'position:fixed;opacity:0;top:0;left:0';
+    document.body.appendChild(ta);
+    ta.focus();
+    ta.select();
+    try { document.execCommand('copy'); done(); } catch (_) {}
+    document.body.removeChild(ta);
+  }
+
   async function loadInvites() {
     try {
       const r = await fetch('/api/invites', { credentials: 'include' });
@@ -66,9 +77,16 @@ import Base from '../../layouts/Base.astro';
       // Copy link buttons
       listEl.querySelectorAll('.copy-btn').forEach(btn => {
         btn.addEventListener('click', () => {
-          navigator.clipboard.writeText((btn as HTMLElement).dataset.link ?? '');
-          btn.textContent = 'Copied!';
-          setTimeout(() => { btn.textContent = 'Copy link'; }, 2000);
+          const text = (btn as HTMLElement).dataset.link ?? '';
+          const done = () => {
+            btn.textContent = 'Copied!';
+            setTimeout(() => { btn.textContent = 'Copy link'; }, 2000);
+          };
+          if (navigator.clipboard) {
+            navigator.clipboard.writeText(text).then(done).catch(() => fallbackCopy(text, done));
+          } else {
+            fallbackCopy(text, done);
+          }
         });
       });
     } catch (e: any) {
diff --git a/site/src/pages/login/index.astro b/site/src/pages/login/index.astro
index 9db2e98..2779abf 100644
--- a/site/src/pages/login/index.astro
+++ b/site/src/pages/login/index.astro
@@ -4,6 +4,9 @@ const editUrl = import.meta.env.PUBLIC_EDIT_URL ?? '';
 ---
 
   

+    

+      mangia
bevi
stai calmo
non strappare
+    

     
Sign in

 
     

@@ -30,6 +33,9 @@ const editUrl = import.meta.env.PUBLIC_EDIT_URL ?? '';
       

         Have an invite? Create account
       

+      

+        Forgot password?
+      

     )}
   

 
diff --git a/site/src/pages/reset-password/index.astro b/site/src/pages/reset-password/index.astro
new file mode 100644
index 0000000..1379dad
--- /dev/null
+++ b/site/src/pages/reset-password/index.astro
@@ -0,0 +1,87 @@
+---
+import Base from '../../layouts/Base.astro';
+---
+
+  

+    
Reset password

+    
Enter the reset code you received from the admin.

+    
Don't have a code? Contact the instance admin — they can generate one for you from the admin panel. Codes expire after 24 hours.

+
+    
+      

+        
+        
+      

+      

+        
+        
+      

+      

+        
+        
+        
At least 8 characters

+      

+      
+      
+      
+    
+
+    

+      Back to sign in
+    

+  

+
+
+
diff --git a/site/src/pages/settings/index.astro b/site/src/pages/settings/index.astro
new file mode 100644
index 0000000..7f039b9
--- /dev/null
+++ b/site/src/pages/settings/index.astro
@@ -0,0 +1,518 @@
+---
+import Base from '../../layouts/Base.astro';
+---
+
+  

+    
Settings

+
+    
+    

+      
Storage

+      
Loading…

+      
+    

+
+    
+    

+      
Profile

+      

+        

+          
+          
+        

+        
+      

+      
+    

+
+    
+    

+      
Password

+      

+        

+          
+          
+        

+        

+          
+          
+          
At least 8 characters

+        

+        
+        
+      

+    

+
+    
+    

+      
Navigation

+      
Hide items from the top nav bar. Affects only your view.

+      

+        
+        
+        
+      

+      
+    

+
+    
+    

+      
Strava API credentials

+      
Loading…

+      

+        

+          
+          
+        

+        

+          
+          
+        

+        
+        

+          
+          
+        

+      

+    

+
+    
+    

+      
Danger zone

+
+      
+      

+        
Delete original files

+        
Removes the raw source files kept for reprocessing (originals/). Your extracted activities, edits, and photos are not affected.

+        
+        
+      

+
+      
+      

+        
Delete all activity data

+        
Wipes all extracted activities, edits, and photos. Your account and original files are kept. Cannot be undone.

+        
+      

+
+      

+        
Delete account

+        
Permanently deletes your account and all data. Cannot be undone.

+        
+      

+    

+  

+
+  
+  
+    

+    

+    

+      
+      
+    

+    
+    

+      
+      
+    

+  
+
+
+
diff --git a/site/src/pages/stats/index.astro b/site/src/pages/stats/index.astro
index b2ceda2..65bcf96 100644
--- a/site/src/pages/stats/index.astro
+++ b/site/src/pages/stats/index.astro
@@ -14,5 +14,5 @@ const handle = shards[0]?.handle ?? null;
     window.location.replace(base + 'u/' + handle + '/stats/');
   
 ) : (
-  
No data found. Run bincio extract first.

+  
No data found. Upload activities to get started.

 )}
diff --git a/site/src/pages/u/[handle]/athlete/index.astro b/site/src/pages/u/[handle]/athlete/index.astro
index 137186d..705f7bf 100644
--- a/site/src/pages/u/[handle]/athlete/index.astro
+++ b/site/src/pages/u/[handle]/athlete/index.astro
@@ -20,7 +20,7 @@ const indexUrl = `${mergedBase}index.json`;
 const athleteUrl = `${mergedBase}athlete.json`;
 ---
 
-  

+  

     

       

         
@{handle}

diff --git a/site/src/pages/u/[handle]/index.astro b/site/src/pages/u/[handle]/index.astro
index 2193f97..a5ea76e 100644
--- a/site/src/pages/u/[handle]/index.astro
+++ b/site/src/pages/u/[handle]/index.astro
@@ -20,7 +20,7 @@ const { handle, shardUrl } = Astro.props as { handle: string; shardUrl: string }
 const base = import.meta.env.BASE_URL;
 ---
 
-  

+  

     

       
@{handle}