add publish infrastructure and update docs for public release

- publish/manifest: explicit file allowlist for public repo
- publish/CLAUDE.md: sanitized version (no personal data paths)
- publish/extract_config.example.yaml: generic paths and owner
- publish.sh: orphan-branch publish script (single squashed commit)
- site/.env.example: documents BINCIO_DATA_DIR and PUBLIC_EDIT_URL
- README.md: updated pipeline diagram, quick start, project layout
- CHEATSHEET.md: added bincio render and bincio edit sections,
  sidecar format reference, updated daily workflow
This commit is contained in:
Davide Scaini
2026-03-29 16:07:06 +02:00
parent 547105550b
commit 6fa7e2bea3
7 changed files with 460 additions and 23 deletions
+49 -15
View File
@@ -6,8 +6,8 @@
# 1. Drop new .fit / .gpx / .tcx files into your input dir, then: # 1. Drop new .fit / .gpx / .tcx files into your input dir, then:
bincio extract bincio extract
# 2. Rebuild the site # 2. Rebuild the site (merges any sidecar edits, then builds)
cd site && npm run build bincio render
# 3. Done — copy site/dist/ to your host # 3. Done — copy site/dist/ to your host
``` ```
@@ -29,24 +29,53 @@ To force a full re-extract: `rm -rf ~/bincio_data && bincio extract`
--- ---
## Site ## Render
```bash ```bash
bincio render # merge edits + production build → site/dist/
bincio render --serve # merge edits + dev server → http://localhost:4321
bincio render --data-dir ~/bincio_data # explicit data dir
```
`bincio render` always runs `merge_all()` first (applies sidecar edits, produces `_merged/`),
then symlinks `site/public/data``_merged/` and runs the Astro build or dev server.
```bash
# Direct npm (skips merge step — use for quick site-only iteration)
cd site cd site
npm run dev
# Symlink data (do once) npm run build
ln -sf ~/bincio_data public/data
# Dev server with hot reload
npm run dev # → http://localhost:4321
# Production build
npm run build # → site/dist/
# Preview production build locally
npm run preview npm run preview
``` ```
## Edit
```bash
# Start the edit server (port 4041 by default)
bincio edit --data-dir ~/bincio_data
# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env to enable the Edit button
# Then browse to any activity and click Edit — a drawer opens in the same page
```
Saves write a sidecar `.md` to `~/bincio_data/edits/{id}.md` and immediately
trigger a merge. Refresh the page to see the updated content.
### Sidecar format
```markdown
---
title: "Renamed title"
sport: cycling
gear: "Trek Domane"
highlight: true # sort to top of feed
private: false # true = hidden from feed
hide_stats: [cadence] # suppress stat panels
---
Description in **markdown**. Images go in the gallery — drag & drop in the Edit drawer.
```
--- ---
## Python / tests ## Python / tests
@@ -128,7 +157,9 @@ Mapping lives in `bincio/extract/sport.py`.
## Patching activities (manual fixes) ## Patching activities (manual fixes)
When re-running extract isn't practical, patch the JSON directly: Prefer the Edit drawer for title/sport/description/photo changes — it writes a sidecar
and keeps extracted data pristine. For bulk fixes or fields not exposed in the UI,
patch the JSON directly:
```bash ```bash
# Fix sport for a single activity # Fix sport for a single activity
@@ -194,8 +225,11 @@ print(len(others), 'total')
| File | Purpose | | File | Purpose |
|---|---| |---|---|
| `extract_config.yaml` | Main config (input dirs, output dir, privacy) | | `extract_config.yaml` | Main config (input dirs, output dir, privacy) |
| `site/.env` | Site env vars (`BINCIO_DATA_DIR`, `PUBLIC_EDIT_URL`) — copy from `.env.example` |
| `SCHEMA.md` | BAS format specification | | `SCHEMA.md` | BAS format specification |
| `CLAUDE.md` | Dev notes, gotchas, design decisions | | `CLAUDE.md` | Dev notes, gotchas, design decisions |
| `bincio/render/merge.py` | Sidecar overlay logic — `parse_sidecar`, `merge_all` |
| `bincio/edit/server.py` | FastAPI edit API — GET/POST activity, image upload |
| `bincio/extract/sport.py` | Sport name normalisation + mapping | | `bincio/extract/sport.py` | Sport name normalisation + mapping |
| `bincio/extract/metrics.py` | Distance, speed, HR, elevation computation | | `bincio/extract/metrics.py` | Distance, speed, HR, elevation computation |
| `bincio/extract/parsers/fit.py` | FIT file parser | | `bincio/extract/parsers/fit.py` | FIT file parser |
+16 -8
View File
@@ -19,9 +19,10 @@ GPX / FIT / TCX files
~/bincio_data/ ← BAS data store. Human-readable JSON + GeoJSON. ~/bincio_data/ ← BAS data store. Human-readable JSON + GeoJSON.
edits/*.md ← Optional sidecar edits (titles, descriptions, photos).
bincio render / npmAstro build. Reads JSON, writes static HTML/JS/CSS. bincio render Merges sidecars → _merged/. Runs Astro build.
site/dist/ ← Drop anywhere. Open index.html. Done. site/dist/ ← Drop anywhere. Open index.html. Done.
@@ -46,17 +47,19 @@ bincio extract
# 4. Build the site (requires Node >= 20) # 4. Build the site (requires Node >= 20)
cd site && npm install cd site && npm install
ln -sf ~/bincio_data public/data cp .env.example .env # configure BINCIO_DATA_DIR
npm run build bincio render # merges edits + runs astro build
# → open site/dist/index.html # → open site/dist/index.html
``` ```
For live development with hot reload: For live development with hot reload:
```bash ```bash
cd site bincio render --serve # merges edits, links data, starts astro dev
ln -sf ~/bincio_data public/data
npm run dev
# → http://localhost:4321 # → http://localhost:4321
# Optional: enable the activity edit UI
bincio edit # starts edit server on http://localhost:4041
# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
``` ```
--- ---
@@ -207,7 +210,11 @@ bincio/ Python package
strava_csv.py Strava activities.csv reader strava_csv.py Strava activities.csv reader
writer.py BAS JSON + GeoJSON writer writer.py BAS JSON + GeoJSON writer
render/ render/
cli.py `bincio render` stub cli.py `bincio render` — merge + astro build/serve
merge.py sidecar edit overlay (produces _merged/)
edit/
cli.py `bincio edit` — local edit server
server.py FastAPI write API for the edit drawer
schema/ schema/
bas-v1.schema.json JSON Schema for BAS format bas-v1.schema.json JSON Schema for BAS format
SCHEMA.md Human-readable BAS specification SCHEMA.md Human-readable BAS specification
@@ -219,10 +226,11 @@ site/ Astro project
stats/index.astro Yearly heatmaps + totals stats/index.astro Yearly heatmaps + totals
components/ components/
ActivityFeed.svelte Card grid, sport filter, pagination ActivityFeed.svelte Card grid, sport filter, pagination
ActivityDetail.svelte Map + stats + charts ActivityDetail.svelte Map + stats + charts + photo gallery
ActivityMap.svelte MapLibre GL map ActivityMap.svelte MapLibre GL map
ActivityCharts.svelte Observable Plot charts ActivityCharts.svelte Observable Plot charts
StatsView.svelte Heatmap, percentile scaling, sport filter StatsView.svelte Heatmap, percentile scaling, sport filter
EditDrawer.svelte Slide-in activity editor
lib/ lib/
types.ts BAS TypeScript types types.ts BAS TypeScript types
format.ts Formatting helpers format.ts Formatting helpers
Executable
+73
View File
@@ -0,0 +1,73 @@
#!/usr/bin/env bash
set -euo pipefail
REMOTE="github-public"
BRANCH="main"
LOCAL_DIR="$(cd "$(dirname "$0")" && pwd)"
PUBLISH_DIR="${LOCAL_DIR}/publish"
MANIFEST="${PUBLISH_DIR}/manifest"
DRY_RUN=false
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN=true ;;
*) echo "Unknown argument: $arg"; exit 1 ;;
esac
done
if ! git -C "$LOCAL_DIR" remote get-url "$REMOTE" &>/dev/null; then
echo "ERROR: remote '${REMOTE}' not found."
echo " git remote add ${REMOTE} https://github.com/brutsalvadi/bincio-activity.git"
exit 1
fi
if [[ -n "$(git -C "$LOCAL_DIR" status --porcelain)" ]]; then
echo "ERROR: uncommitted changes. Commit or stash first."
exit 1
fi
if [[ ! -f "$MANIFEST" ]]; then
echo "ERROR: manifest not found at ${MANIFEST}"
exit 1
fi
STAGING="$(mktemp -d)"
trap 'rm -rf "$STAGING"' EXIT
while IFS= read -r relpath || [[ -n "$relpath" ]]; do
[[ -z "$relpath" || "$relpath" == \#* ]] && continue
override="${PUBLISH_DIR}/${relpath}"
original="${LOCAL_DIR}/${relpath}"
dest="${STAGING}/${relpath}"
mkdir -p "$(dirname "$dest")"
if [[ -f "$override" ]]; then
cp "$override" "$dest"
elif [[ -f "$original" ]]; then
cp "$original" "$dest"
else
echo "ERROR: '${relpath}' in manifest but not found (no override, no original)"
exit 1
fi
done < "$MANIFEST"
echo "Files to be published:"
find "$STAGING" -type f | sed "s|${STAGING}/||" | sort
if $DRY_RUN; then
echo ""
echo "Dry run complete. No changes made."
exit 0
fi
git -C "$LOCAL_DIR" checkout --orphan _public_tmp
git -C "$LOCAL_DIR" rm -rf . --quiet
cp -r "${STAGING}/." "${LOCAL_DIR}/"
TIMESTAMP="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
git -C "$LOCAL_DIR" add -A
git -C "$LOCAL_DIR" commit -m "Published ${TIMESTAMP}"
git -C "$LOCAL_DIR" push --force "$REMOTE" "HEAD:${BRANCH}"
git -C "$LOCAL_DIR" checkout main
git -C "$LOCAL_DIR" branch -D _public_tmp
echo ""
echo "Done: $(git -C "$LOCAL_DIR" remote get-url "$REMOTE")"
+223
View File
@@ -0,0 +1,223 @@
# BincioActivity — Context for Claude
## What this project is
BincioActivity is a federated, open-source, self-hosted activity stats platform
(think personal Strava). Two-stage pipeline:
1. **`bincio extract`** (Python): GPX/FIT/TCX → BAS JSON data store
2. **`bincio render`** (Astro/Node): BAS data store → static website
The BAS (BincioActivity Schema) JSON files are the federation protocol.
Anyone can publish their data as BAS JSON and others can include it.
## Key design decisions
- **No database, no server** — everything is static files
- **Python with uv** for the extract stage
- **Astro + Svelte + Tailwind + MapLibre GL + Observable Plot** for the site
- **Haversine** (not geopy) for distance calculations (10x faster)
- **Worker initializer pattern** for ProcessPoolExecutor — large shared data
(strava_lookup dict, known_hashes frozenset) is sent once per worker via
`initializer=`, not once per task
- **BAS activity IDs** always use UTC with Z suffix for URL safety
- **TCX files** from Garmin use both `http://` and `https://` namespace URIs —
parser handles both
## Your data
- Source: `~/your-activity-data/`
- `activities/` — Strava export (GPX, FIT, TCX, all with .gz variants)
- Any subdirectories with FIT files from Garmin/Karoo devices
- `activities.csv` — Strava metadata (names, descriptions, gear)
- Extracted output: `~/bincio_data/` (or `/tmp/bincio_test/` for testing)
Configure input paths in `extract_config.yaml`.
## Project structure
```
bincio/ Python package
extract/
models.py DataPoint, ParsedActivity, LapData
parsers/ GPX, FIT, TCX parsers + factory
sport.py sport name normalisation
metrics.py haversine-based stats computation (single pass)
timeseries.py downsample to 1Hz, build BAS timeseries object
simplify.py RDP track simplification → GeoJSON
dedup.py exact (hash) + near-duplicate detection
strava_csv.py Strava activities.csv importer
writer.py BAS JSON + GeoJSON writer
config.py extract_config.yaml loader
cli.py `bincio extract` CLI
render/
cli.py `bincio render` CLI (symlinks data, runs astro build/dev)
merge.py sidecar edit overlay (produces _merged/)
edit/
cli.py `bincio edit` CLI
server.py FastAPI write API for the edit drawer
schema/
bas-v1.schema.json JSON Schema for BAS
SCHEMA.md Human-readable BAS spec
site/ Astro project
src/
layouts/Base.astro
pages/
index.astro Activity feed (loads index.json client-side)
activity/[id].astro Single activity (SSG, loads detail JSON client-side)
stats/index.astro Heatmap + year totals
components/
ActivityFeed.svelte Card grid, sport filter, pagination
ActivityDetail.svelte Map + stats + charts + photo gallery
ActivityMap.svelte MapLibre GL (gradient track, linked hover dot)
ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence tabs)
StatsView.svelte Yearly heatmap + totals
EditDrawer.svelte Slide-in edit panel (visible when PUBLIC_EDIT_URL set)
lib/
types.ts BAS TypeScript types
format.ts formatDistance, formatDuration, sportIcon, etc.
```
## How to run
```bash
# Extract
cd ~/src/bincio_activity
uv run bincio extract --input ~/your-activity-data/activities --output /tmp/bincio_test
# Site dev server
cd site
ln -sf /tmp/bincio_test/_merged public/data # point at merged output
cp .env.example .env && $EDITOR .env # set BINCIO_DATA_DIR
npm run dev
# Edit server (optional — enables edit drawer in the site)
uv run bincio edit --data-dir /tmp/bincio_test
# set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
# Tests
uv run pytest
```
## MapLibre GL + Vite/Astro — known gotchas
Learnt the hard way during debugging (March 2026):
- **`maplibregl.workerUrl = ...` is the v3 API and silently no-ops in v4+.**
The v5 API is `maplibregl.setWorkerUrl(url)`, but you don't need it at all in a
normal Vite environment — MapLibre handles the blob worker automatically.
- **`optimizeDeps: { exclude: ['maplibre-gl'] }` breaks tile loading.**
It prevents Vite from converting MapLibre's UMD bundle to ESM. The UMD bundle
uses AMD `define()` internally; served raw, the tile worker blob fails silently →
black map, no tiles. The correct setting is `include: ['maplibre-gl']`.
- **`build.target: 'es2022'` (and `optimizeDeps.esbuildOptions.target`) is required.**
MapLibre's dependencies use ES2022 class field syntax. If esbuild downgrades it,
helpers like `__publicField` aren't available inside the serialised worker blob
scope → tile loading fails. This is a known upstream issue (maplibre-gl-js #6680).
- **Use static imports, not dynamic `await import('maplibre-gl')`, when possible.**
With `client:only="svelte"` in Astro, SSR never runs for the component so there is
no `window is not defined` risk. Static import lets Vite pre-bundle correctly.
- **Use `client:only="svelte"` (not `client:load`) for the activity detail page.**
`client:load` does SSR + hydration; complex interactive components with MapLibre
can hit hydration mismatch issues. `client:only` mounts fresh on the client only.
- **MapLibre v5 requires explicit `center` and `zoom` in the Map constructor.**
v4 silently defaulted to `center: [0,0], zoom: 0`. v5 leaves internal projection
state undefined → `Cannot read properties of undefined (reading 'lng')` crashes
on any operation that touches coordinates (markers, resize, render). Always pass
`center` and `zoom` even if you plan to `fitBounds` later.
- **MapLibre v5 requires `setLngLat()` on markers before `.addTo(map)`.**
v4 tolerated markers without coordinates. v5 calls `Marker._update()` inside
`addTo()`, which needs valid lngLat → same `'lng'` crash. Set a dummy `[0, 0]`
if the real position arrives later (e.g. hover markers).
## Observable Plot — known gotchas
- **Curve names are hyphenated, not camelCase.**
Use `"monotone-x"`, not `"monotoneX"`. Plot uses its own curve name registry
(not raw d3 identifiers). Wrong names throw `unknown curve` at runtime.
The working `astro.config.mjs` Vite section:
```js
vite: {
optimizeDeps: {
include: ['maplibre-gl'],
esbuildOptions: { target: 'es2022' },
},
build: { target: 'es2022' },
},
```
## Activity sidecar edits — design spec
Users edit activities via **sidecar markdown files** in the data dir.
No database, no server — consistent with the project's static-files-only philosophy.
### File naming
```
~/bincio_data/
activities/{id}.json ← immutable extract output
edits/{id}.md ← user edits (sidecar)
edits/images/{id}/ ← uploaded photos
_merged/ ← render-time merge output (gitignored-style)
```
### Sidecar format
```markdown
---
title: "Epic climb up Monte Grappa"
sport: cycling
hide_stats: [cadence]
highlight: true
private: false
gear: "Trek Domane"
---
Rode with friends. Legs felt great after the rest week...
```
### Editing UX: drawer in Astro + `bincio edit` write API
- `bincio edit --data-dir ~/bincio_data` starts a FastAPI server on port 4041
- Set `PUBLIC_EDIT_URL=http://localhost:4041` in `site/.env` to enable the edit button
- Clicking Edit on any activity detail page opens a slide-in drawer
- Saving writes the sidecar and triggers `merge_all()` automatically
- `bincio render` always runs `merge_all()` before build/serve and symlinks `public/data``_merged/`
### `PUBLIC_EDIT_URL` as feature flag
- **Unset** → no Edit button, normal static site
- **Set** → edit drawer enabled; lives in `site/.env` (gitignored)
## Known issues / next steps
- `bincio render` Python CLI is functional but `--watch` mode not yet implemented
- Activity IDs in older test data may use `+0000` format (pre-fix); re-run extract to get `Z` format
- Some activities appear with both untitled and titled IDs (near-dedup timing race)
- Federation (remote data sources) not yet implemented in site
- Friends pages (`/friends/{handle}/`) not yet implemented
- The `site/.env` file is gitignored — copy from `site/.env.example`
## What "good" looks like (not yet done)
- [ ] `bincio render` Python CLI wraps `astro build` properly
- [ ] Friends/federation pages in site
- [ ] Personal records page
- [ ] Activity search / full-text filter in feed
- [ ] GitHub Actions template for auto-publish
- [ ] Karoo/Garmin Connect importers beyond Strava
- [x] `bincio.render.merge` — sidecar parser, `_merged/` output, private filter, highlight sort
- [x] `bincio edit` FastAPI write API (GET/POST activity, image upload/delete, triggers merge)
- [x] `EditDrawer.svelte` — slide-in edit UI in the Astro site
- [x] `PUBLIC_EDIT_URL` feature flag
- [x] Markdown rendering in activity description with image path rewriting
- [x] Photo gallery with lightbox on activity detail page
- [ ] `bincio render --watch` incremental rebuild on sidecar/data changes
- [ ] Highlight badge in activity feed cards
+31
View File
@@ -0,0 +1,31 @@
owner:
handle: yourname
display_name: Your Name
input:
dirs:
- ~/Activities/gpx
- ~/Activities/fit
# Strava bulk export metadata — provides names, descriptions, gear
# metadata_csv: ~/strava_export/activities.csv
output:
dir: ~/bincio_data
default_privacy: public
sensors:
heart_rate: true
cadence: true
temperature: true
power: true
track:
simplify: rdp
rdp_epsilon: 0.0001 # ~11m at equator
timeseries_hz: 1 # 1 sample/second max
classifier:
enabled: false # ML activity type classifier (requires scikit-learn extra)
incremental: true # skip files whose hash hasn't changed since last run
+59
View File
@@ -0,0 +1,59 @@
# BincioActivity — public release manifest
# One relative path per line.
# If publish/<path> exists, that sanitized version is used instead of the original.
.gitignore
.python-version
CHEATSHEET.md
CLAUDE.md
README.md
SCHEMA.md
pyproject.toml
extract_config.example.yaml
schema/bas-v1.schema.json
bincio/__init__.py
bincio/cli.py
bincio/edit/__init__.py
bincio/edit/cli.py
bincio/edit/server.py
bincio/extract/__init__.py
bincio/extract/cli.py
bincio/extract/config.py
bincio/extract/dedup.py
bincio/extract/metrics.py
bincio/extract/models.py
bincio/extract/parsers/__init__.py
bincio/extract/parsers/base.py
bincio/extract/parsers/factory.py
bincio/extract/parsers/fit.py
bincio/extract/parsers/gpx.py
bincio/extract/parsers/tcx.py
bincio/extract/simplify.py
bincio/extract/sport.py
bincio/extract/strava_csv.py
bincio/extract/timeseries.py
bincio/extract/writer.py
bincio/render/__init__.py
bincio/render/cli.py
bincio/render/merge.py
site/.env.example
site/astro.config.mjs
site/package.json
site/tailwind.config.mjs
site/tsconfig.json
site/src/components/ActivityCharts.svelte
site/src/components/ActivityDetail.svelte
site/src/components/ActivityFeed.svelte
site/src/components/ActivityMap.svelte
site/src/components/EditDrawer.svelte
site/src/components/StatsView.svelte
site/src/layouts/Base.astro
site/src/lib/format.ts
site/src/lib/types.ts
site/src/pages/activity/[id].astro
site/src/pages/index.astro
site/src/pages/stats/index.astro
tests/__init__.py
tests/test_merge.py
tests/test_sport.py
tests/test_writer.py
+9
View File
@@ -0,0 +1,9 @@
# Copy to .env and fill in your values.
# Path to your BAS data store (output of `bincio extract`)
BINCIO_DATA_DIR=~/bincio_data
# Optional: URL of a running `bincio edit` server.
# When set, an Edit button appears on activity detail pages.
# Leave unset (or remove) for production / public deployments.
# PUBLIC_EDIT_URL=http://localhost:4041