add publish infrastructure and update docs for public release

- publish/manifest: explicit file allowlist for public repo - publish/CLAUDE.md: sanitized version (no personal data paths) - publish/extract_config.example.yaml: generic paths and owner - publish.sh: orphan-branch publish script (single squashed commit) - site/.env.example: documents BINCIO_DATA_DIR and PUBLIC_EDIT_URL - README.md: updated pipeline diagram, quick start, project layout - CHEATSHEET.md: added bincio render and bincio edit sections, sidecar format reference, updated daily workflow
2026-03-29 16:07:06 +02:00
parent 547105550b
commit 6fa7e2bea3
7 changed files with 460 additions and 23 deletions
@@ -6,8 +6,8 @@
 # 1. Drop new .fit / .gpx / .tcx files into your input dir, then:
 bincio extract

-# 2. Rebuild the site
-cd site && npm run build
+# 2. Rebuild the site (merges any sidecar edits, then builds)
+bincio render

 # 3. Done — copy site/dist/ to your host
 ```
@@ -29,24 +29,53 @@ To force a full re-extract: `rm -rf ~/bincio_data && bincio extract`

 ---

-## Site
+## Render

 ```bash
+bincio render                            # merge edits + production build → site/dist/
+bincio render --serve                    # merge edits + dev server → http://localhost:4321
+bincio render --data-dir ~/bincio_data   # explicit data dir
+```
+
+`bincio render` always runs `merge_all()` first (applies sidecar edits, produces `_merged/`),
+then symlinks `site/public/data` → `_merged/` and runs the Astro build or dev server.
+
+```bash
+# Direct npm (skips merge step — use for quick site-only iteration)
 cd site
-
-# Symlink data (do once)
-ln -sf ~/bincio_data public/data
-
-# Dev server with hot reload
-npm run dev                              # → http://localhost:4321
-
-# Production build
-npm run build                            # → site/dist/
-
-# Preview production build locally
+npm run dev
+npm run build
 npm run preview
 ```

+## Edit
+
+```bash
+# Start the edit server (port 4041 by default)
+bincio edit --data-dir ~/bincio_data
+
+# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env to enable the Edit button
+# Then browse to any activity and click Edit — a drawer opens in the same page
+```
+
+Saves write a sidecar `.md` to `~/bincio_data/edits/{id}.md` and immediately
+trigger a merge. Refresh the page to see the updated content.
+
+### Sidecar format
+
+```markdown
+---
+title: "Renamed title"
+sport: cycling
+gear: "Trek Domane"
+highlight: true          # sort to top of feed
+private: false           # true = hidden from feed
+hide_stats: [cadence]    # suppress stat panels
+---
+
+Description in **markdown**. Images go in the gallery — drag & drop in the Edit drawer.
+```
+
 ---

 ## Python / tests
@@ -128,7 +157,9 @@ Mapping lives in `bincio/extract/sport.py`.

 ## Patching activities (manual fixes)

-When re-running extract isn't practical, patch the JSON directly:
+Prefer the Edit drawer for title/sport/description/photo changes — it writes a sidecar
+and keeps extracted data pristine. For bulk fixes or fields not exposed in the UI,
+patch the JSON directly:

 ```bash
 # Fix sport for a single activity
@@ -194,8 +225,11 @@ print(len(others), 'total')
 | File | Purpose |
 |---|---|
 | `extract_config.yaml` | Main config (input dirs, output dir, privacy) |
+| `site/.env` | Site env vars (`BINCIO_DATA_DIR`, `PUBLIC_EDIT_URL`) — copy from `.env.example` |
 | `SCHEMA.md` | BAS format specification |
 | `CLAUDE.md` | Dev notes, gotchas, design decisions |
+| `bincio/render/merge.py` | Sidecar overlay logic — `parse_sidecar`, `merge_all` |
+| `bincio/edit/server.py` | FastAPI edit API — GET/POST activity, image upload |
 | `bincio/extract/sport.py` | Sport name normalisation + mapping |
 | `bincio/extract/metrics.py` | Distance, speed, HR, elevation computation |
 | `bincio/extract/parsers/fit.py` | FIT file parser |
@@ -19,9 +19,10 @@ GPX / FIT / TCX files
        │
        ▼
  ~/bincio_data/          ← BAS data store. Human-readable JSON + GeoJSON.
+   edits/*.md             ← Optional sidecar edits (titles, descriptions, photos).
        │
        ▼
-  bincio render / npm     ← Astro build. Reads JSON, writes static HTML/JS/CSS.
+  bincio render           ← Merges sidecars → _merged/. Runs Astro build.
        │
        ▼
  site/dist/              ← Drop anywhere. Open index.html. Done.
@@ -46,17 +47,19 @@ bincio extract

 # 4. Build the site (requires Node >= 20)
 cd site && npm install
-ln -sf ~/bincio_data public/data
-npm run build
+cp .env.example .env        # configure BINCIO_DATA_DIR
+bincio render               # merges edits + runs astro build
 # → open site/dist/index.html
 ```

 For live development with hot reload:
 ```bash
-cd site
-ln -sf ~/bincio_data public/data
-npm run dev
+bincio render --serve       # merges edits, links data, starts astro dev
 # → http://localhost:4321
+
+# Optional: enable the activity edit UI
+bincio edit                 # starts edit server on http://localhost:4041
+# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
 ```

 ---
@@ -207,7 +210,11 @@ bincio/                 Python package
    strava_csv.py       Strava activities.csv reader
    writer.py           BAS JSON + GeoJSON writer
  render/
-    cli.py              `bincio render` stub
+    cli.py              `bincio render` — merge + astro build/serve
+    merge.py            sidecar edit overlay (produces _merged/)
+  edit/
+    cli.py              `bincio edit` — local edit server
+    server.py           FastAPI write API for the edit drawer
 schema/
  bas-v1.schema.json    JSON Schema for BAS format
 SCHEMA.md               Human-readable BAS specification
@@ -219,10 +226,11 @@ site/                   Astro project
      stats/index.astro         Yearly heatmaps + totals
    components/
      ActivityFeed.svelte        Card grid, sport filter, pagination
-      ActivityDetail.svelte      Map + stats + charts
+      ActivityDetail.svelte      Map + stats + charts + photo gallery
      ActivityMap.svelte         MapLibre GL map
      ActivityCharts.svelte      Observable Plot charts
      StatsView.svelte           Heatmap, percentile scaling, sport filter
+      EditDrawer.svelte          Slide-in activity editor
    lib/
      types.ts                  BAS TypeScript types
      format.ts                 Formatting helpers
@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+REMOTE="github-public"
+BRANCH="main"
+LOCAL_DIR="$(cd "$(dirname "$0")" && pwd)"
+PUBLISH_DIR="${LOCAL_DIR}/publish"
+MANIFEST="${PUBLISH_DIR}/manifest"
+DRY_RUN=false
+
+for arg in "$@"; do
+    case "$arg" in
+        --dry-run) DRY_RUN=true ;;
+        *) echo "Unknown argument: $arg"; exit 1 ;;
+    esac
+done
+
+if ! git -C "$LOCAL_DIR" remote get-url "$REMOTE" &>/dev/null; then
+    echo "ERROR: remote '${REMOTE}' not found."
+    echo "  git remote add ${REMOTE} https://github.com/brutsalvadi/bincio-activity.git"
+    exit 1
+fi
+if [[ -n "$(git -C "$LOCAL_DIR" status --porcelain)" ]]; then
+    echo "ERROR: uncommitted changes. Commit or stash first."
+    exit 1
+fi
+if [[ ! -f "$MANIFEST" ]]; then
+    echo "ERROR: manifest not found at ${MANIFEST}"
+    exit 1
+fi
+
+STAGING="$(mktemp -d)"
+trap 'rm -rf "$STAGING"' EXIT
+
+while IFS= read -r relpath || [[ -n "$relpath" ]]; do
+    [[ -z "$relpath" || "$relpath" == \#* ]] && continue
+    override="${PUBLISH_DIR}/${relpath}"
+    original="${LOCAL_DIR}/${relpath}"
+    dest="${STAGING}/${relpath}"
+    mkdir -p "$(dirname "$dest")"
+    if [[ -f "$override" ]]; then
+        cp "$override" "$dest"
+    elif [[ -f "$original" ]]; then
+        cp "$original" "$dest"
+    else
+        echo "ERROR: '${relpath}' in manifest but not found (no override, no original)"
+        exit 1
+    fi
+done < "$MANIFEST"
+
+echo "Files to be published:"
+find "$STAGING" -type f | sed "s|${STAGING}/||" | sort
+
+if $DRY_RUN; then
+    echo ""
+    echo "Dry run complete. No changes made."
+    exit 0
+fi
+
+git -C "$LOCAL_DIR" checkout --orphan _public_tmp
+git -C "$LOCAL_DIR" rm -rf . --quiet
+cp -r "${STAGING}/." "${LOCAL_DIR}/"
+
+TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
+git -C "$LOCAL_DIR" add -A
+git -C "$LOCAL_DIR" commit -m "Published ${TIMESTAMP}"
+git -C "$LOCAL_DIR" push --force "$REMOTE" "HEAD:${BRANCH}"
+
+git -C "$LOCAL_DIR" checkout main
+git -C "$LOCAL_DIR" branch -D _public_tmp
+
+echo ""
+echo "Done: $(git -C "$LOCAL_DIR" remote get-url "$REMOTE")"
@@ -0,0 +1,223 @@
+# BincioActivity — Context for Claude
+
+## What this project is
+
+BincioActivity is a federated, open-source, self-hosted activity stats platform
+(think personal Strava). Two-stage pipeline:
+
+1. **`bincio extract`** (Python): GPX/FIT/TCX → BAS JSON data store
+2. **`bincio render`** (Astro/Node): BAS data store → static website
+
+The BAS (BincioActivity Schema) JSON files are the federation protocol.
+Anyone can publish their data as BAS JSON and others can include it.
+
+## Key design decisions
+
+- **No database, no server** — everything is static files
+- **Python with uv** for the extract stage
+- **Astro + Svelte + Tailwind + MapLibre GL + Observable Plot** for the site
+- **Haversine** (not geopy) for distance calculations (10x faster)
+- **Worker initializer pattern** for ProcessPoolExecutor — large shared data
+  (strava_lookup dict, known_hashes frozenset) is sent once per worker via
+  `initializer=`, not once per task
+- **BAS activity IDs** always use UTC with Z suffix for URL safety
+- **TCX files** from Garmin use both `http://` and `https://` namespace URIs —
+  parser handles both
+
+## Your data
+
+- Source: `~/your-activity-data/`
+  - `activities/` — Strava export (GPX, FIT, TCX, all with .gz variants)
+  - Any subdirectories with FIT files from Garmin/Karoo devices
+  - `activities.csv` — Strava metadata (names, descriptions, gear)
+- Extracted output: `~/bincio_data/` (or `/tmp/bincio_test/` for testing)
+
+Configure input paths in `extract_config.yaml`.
+
+## Project structure
+
+```
+bincio/                    Python package
+  extract/
+    models.py              DataPoint, ParsedActivity, LapData
+    parsers/               GPX, FIT, TCX parsers + factory
+    sport.py               sport name normalisation
+    metrics.py             haversine-based stats computation (single pass)
+    timeseries.py          downsample to 1Hz, build BAS timeseries object
+    simplify.py            RDP track simplification → GeoJSON
+    dedup.py               exact (hash) + near-duplicate detection
+    strava_csv.py          Strava activities.csv importer
+    writer.py              BAS JSON + GeoJSON writer
+    config.py              extract_config.yaml loader
+    cli.py                 `bincio extract` CLI
+  render/
+    cli.py                 `bincio render` CLI (symlinks data, runs astro build/dev)
+    merge.py               sidecar edit overlay (produces _merged/)
+  edit/
+    cli.py                 `bincio edit` CLI
+    server.py              FastAPI write API for the edit drawer
+schema/
+  bas-v1.schema.json       JSON Schema for BAS
+SCHEMA.md                  Human-readable BAS spec
+site/                      Astro project
+  src/
+    layouts/Base.astro
+    pages/
+      index.astro           Activity feed (loads index.json client-side)
+      activity/[id].astro   Single activity (SSG, loads detail JSON client-side)
+      stats/index.astro     Heatmap + year totals
+    components/
+      ActivityFeed.svelte   Card grid, sport filter, pagination
+      ActivityDetail.svelte Map + stats + charts + photo gallery
+      ActivityMap.svelte    MapLibre GL (gradient track, linked hover dot)
+      ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence tabs)
+      StatsView.svelte      Yearly heatmap + totals
+      EditDrawer.svelte     Slide-in edit panel (visible when PUBLIC_EDIT_URL set)
+    lib/
+      types.ts              BAS TypeScript types
+      format.ts             formatDistance, formatDuration, sportIcon, etc.
+```
+
+## How to run
+
+```bash
+# Extract
+cd ~/src/bincio_activity
+uv run bincio extract --input ~/your-activity-data/activities --output /tmp/bincio_test
+
+# Site dev server
+cd site
+ln -sf /tmp/bincio_test/_merged public/data   # point at merged output
+cp .env.example .env && $EDITOR .env           # set BINCIO_DATA_DIR
+npm run dev
+
+# Edit server (optional — enables edit drawer in the site)
+uv run bincio edit --data-dir /tmp/bincio_test
+# set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
+
+# Tests
+uv run pytest
+```
+
+## MapLibre GL + Vite/Astro — known gotchas
+
+Learnt the hard way during debugging (March 2026):
+
+- **`maplibregl.workerUrl = ...` is the v3 API and silently no-ops in v4+.**
+  The v5 API is `maplibregl.setWorkerUrl(url)`, but you don't need it at all in a
+  normal Vite environment — MapLibre handles the blob worker automatically.
+
+- **`optimizeDeps: { exclude: ['maplibre-gl'] }` breaks tile loading.**
+  It prevents Vite from converting MapLibre's UMD bundle to ESM. The UMD bundle
+  uses AMD `define()` internally; served raw, the tile worker blob fails silently →
+  black map, no tiles. The correct setting is `include: ['maplibre-gl']`.
+
+- **`build.target: 'es2022'` (and `optimizeDeps.esbuildOptions.target`) is required.**
+  MapLibre's dependencies use ES2022 class field syntax. If esbuild downgrades it,
+  helpers like `__publicField` aren't available inside the serialised worker blob
+  scope → tile loading fails. This is a known upstream issue (maplibre-gl-js #6680).
+
+- **Use static imports, not dynamic `await import('maplibre-gl')`, when possible.**
+  With `client:only="svelte"` in Astro, SSR never runs for the component so there is
+  no `window is not defined` risk. Static import lets Vite pre-bundle correctly.
+
+- **Use `client:only="svelte"` (not `client:load`) for the activity detail page.**
+  `client:load` does SSR + hydration; complex interactive components with MapLibre
+  can hit hydration mismatch issues. `client:only` mounts fresh on the client only.
+
+- **MapLibre v5 requires explicit `center` and `zoom` in the Map constructor.**
+  v4 silently defaulted to `center: [0,0], zoom: 0`. v5 leaves internal projection
+  state undefined → `Cannot read properties of undefined (reading 'lng')` crashes
+  on any operation that touches coordinates (markers, resize, render). Always pass
+  `center` and `zoom` even if you plan to `fitBounds` later.
+
+- **MapLibre v5 requires `setLngLat()` on markers before `.addTo(map)`.**
+  v4 tolerated markers without coordinates. v5 calls `Marker._update()` inside
+  `addTo()`, which needs valid lngLat → same `'lng'` crash. Set a dummy `[0, 0]`
+  if the real position arrives later (e.g. hover markers).
+
+## Observable Plot — known gotchas
+
+- **Curve names are hyphenated, not camelCase.**
+  Use `"monotone-x"`, not `"monotoneX"`. Plot uses its own curve name registry
+  (not raw d3 identifiers). Wrong names throw `unknown curve` at runtime.
+
+The working `astro.config.mjs` Vite section:
+```js
+vite: {
+  optimizeDeps: {
+    include: ['maplibre-gl'],
+    esbuildOptions: { target: 'es2022' },
+  },
+  build: { target: 'es2022' },
+},
+```
+
+## Activity sidecar edits — design spec
+
+Users edit activities via **sidecar markdown files** in the data dir.
+No database, no server — consistent with the project's static-files-only philosophy.
+
+### File naming
+
+```
+~/bincio_data/
+  activities/{id}.json    ← immutable extract output
+  edits/{id}.md           ← user edits (sidecar)
+  edits/images/{id}/      ← uploaded photos
+  _merged/                ← render-time merge output (gitignored-style)
+```
+
+### Sidecar format
+
+```markdown
+---
+title: "Epic climb up Monte Grappa"
+sport: cycling
+hide_stats: [cadence]
+highlight: true
+private: false
+gear: "Trek Domane"
+---
+
+Rode with friends. Legs felt great after the rest week...
+```
+
+### Editing UX: drawer in Astro + `bincio edit` write API
+
+- `bincio edit --data-dir ~/bincio_data` starts a FastAPI server on port 4041
+- Set `PUBLIC_EDIT_URL=http://localhost:4041` in `site/.env` to enable the edit button
+- Clicking Edit on any activity detail page opens a slide-in drawer
+- Saving writes the sidecar and triggers `merge_all()` automatically
+- `bincio render` always runs `merge_all()` before build/serve and symlinks `public/data` → `_merged/`
+
+### `PUBLIC_EDIT_URL` as feature flag
+
+- **Unset** → no Edit button, normal static site
+- **Set** → edit drawer enabled; lives in `site/.env` (gitignored)
+
+## Known issues / next steps
+
+- `bincio render` Python CLI is functional but `--watch` mode not yet implemented
+- Activity IDs in older test data may use `+0000` format (pre-fix); re-run extract to get `Z` format
+- Some activities appear with both untitled and titled IDs (near-dedup timing race)
+- Federation (remote data sources) not yet implemented in site
+- Friends pages (`/friends/{handle}/`) not yet implemented
+- The `site/.env` file is gitignored — copy from `site/.env.example`
+
+## What "good" looks like (not yet done)
+
+- [ ] `bincio render` Python CLI wraps `astro build` properly
+- [ ] Friends/federation pages in site
+- [ ] Personal records page
+- [ ] Activity search / full-text filter in feed
+- [ ] GitHub Actions template for auto-publish
+- [ ] Karoo/Garmin Connect importers beyond Strava
+- [x] `bincio.render.merge` — sidecar parser, `_merged/` output, private filter, highlight sort
+- [x] `bincio edit` FastAPI write API (GET/POST activity, image upload/delete, triggers merge)
+- [x] `EditDrawer.svelte` — slide-in edit UI in the Astro site
+- [x] `PUBLIC_EDIT_URL` feature flag
+- [x] Markdown rendering in activity description with image path rewriting
+- [x] Photo gallery with lightbox on activity detail page
+- [ ] `bincio render --watch` incremental rebuild on sidecar/data changes
+- [ ] Highlight badge in activity feed cards
@@ -0,0 +1,31 @@
+owner:
+  handle: yourname
+  display_name: Your Name
+
+input:
+  dirs:
+    - ~/Activities/gpx
+    - ~/Activities/fit
+  # Strava bulk export metadata — provides names, descriptions, gear
+  # metadata_csv: ~/strava_export/activities.csv
+
+output:
+  dir: ~/bincio_data
+
+default_privacy: public
+
+sensors:
+  heart_rate: true
+  cadence: true
+  temperature: true
+  power: true
+
+track:
+  simplify: rdp
+  rdp_epsilon: 0.0001   # ~11m at equator
+  timeseries_hz: 1       # 1 sample/second max
+
+classifier:
+  enabled: false          # ML activity type classifier (requires scikit-learn extra)
+
+incremental: true         # skip files whose hash hasn't changed since last run
@@ -0,0 +1,59 @@
+# BincioActivity — public release manifest
+# One relative path per line.
+# If publish/<path> exists, that sanitized version is used instead of the original.
+
+.gitignore
+.python-version
+CHEATSHEET.md
+CLAUDE.md
+README.md
+SCHEMA.md
+pyproject.toml
+extract_config.example.yaml
+schema/bas-v1.schema.json
+bincio/__init__.py
+bincio/cli.py
+bincio/edit/__init__.py
+bincio/edit/cli.py
+bincio/edit/server.py
+bincio/extract/__init__.py
+bincio/extract/cli.py
+bincio/extract/config.py
+bincio/extract/dedup.py
+bincio/extract/metrics.py
+bincio/extract/models.py
+bincio/extract/parsers/__init__.py
+bincio/extract/parsers/base.py
+bincio/extract/parsers/factory.py
+bincio/extract/parsers/fit.py
+bincio/extract/parsers/gpx.py
+bincio/extract/parsers/tcx.py
+bincio/extract/simplify.py
+bincio/extract/sport.py
+bincio/extract/strava_csv.py
+bincio/extract/timeseries.py
+bincio/extract/writer.py
+bincio/render/__init__.py
+bincio/render/cli.py
+bincio/render/merge.py
+site/.env.example
+site/astro.config.mjs
+site/package.json
+site/tailwind.config.mjs
+site/tsconfig.json
+site/src/components/ActivityCharts.svelte
+site/src/components/ActivityDetail.svelte
+site/src/components/ActivityFeed.svelte
+site/src/components/ActivityMap.svelte
+site/src/components/EditDrawer.svelte
+site/src/components/StatsView.svelte
+site/src/layouts/Base.astro
+site/src/lib/format.ts
+site/src/lib/types.ts
+site/src/pages/activity/[id].astro
+site/src/pages/index.astro
+site/src/pages/stats/index.astro
+tests/__init__.py
+tests/test_merge.py
+tests/test_sport.py
+tests/test_writer.py
@@ -0,0 +1,9 @@
+# Copy to .env and fill in your values.
+
+# Path to your BAS data store (output of `bincio extract`)
+BINCIO_DATA_DIR=~/bincio_data
+
+# Optional: URL of a running `bincio edit` server.
+# When set, an Edit button appears on activity detail pages.
+# Leave unset (or remove) for production / public deployments.
+# PUBLIC_EDIT_URL=http://localhost:4041