chore: update changelog, remove stale files, scrub VPS IP

- CHANGELOG.md: add [Unreleased] 2026-04-16 section covering settings
  page, admin tools, password reset, re-extract, community page, SSE
  upload progress, and all bug fixes since 2026-04-10
- Remove docs-proposal.md (internal planning doc, not user-facing)
- Remove publish/ directory (leftover artefacts from publish.sh, not
  meant to be tracked)
- scripts/pull_feedback.sh: replace hardcoded default VPS IP with a
  required positional argument to avoid leaking server address
- docs/squash-for-github.md: document the squash-for-github commit
  strategy for future reference
This commit is contained in:
Davide Scaini
2026-04-16 18:09:32 +02:00
parent a78f6ee3bd
commit c68dfa9057
7 changed files with 175 additions and 441 deletions
+95
View File
@@ -1,5 +1,100 @@
# Changelog # Changelog
## [Unreleased] — 2026-04-16
### New feature — Self-service user settings page
- **`site/src/pages/settings/index.astro`** — new `/settings/` page with three sections:
- **Account** — display name editor, storage quota view (uploaded activities + originals size)
- **Integrations** — per-user Strava client ID/secret (replaces instance-level credentials for
multi-user deployments); saved to `settings` table via `PATCH /api/me`
- **Danger zone** — two separate destructive actions:
- **Delete originals** — removes `{user_dir}/originals/` without touching activities
- **Delete all activities** — wipes all activities, edits, GeoJSON, and `_merged/`; triggers rebuild
- Nav visibility toggles — user can hide any combination of Feed / Stats / Athlete tabs from
their navigation; preference saved to `settings` table and applied in `Base.astro`
### New feature — Upload overwrite option
- **`POST /api/upload`** — new `overwrite: bool` form field; when true, an existing activity
with the same ID is replaced rather than returning 409. UI checkbox added to the upload modal.
### New feature — Admin tools
- **Ghost user detection** — `/admin/` now marks users whose handle has a data directory but
no entry in the `users` table (e.g. manually created dirs, or users deleted from DB); shown
with a "ghost" badge
- **Delete directory button** — admin can delete a user's entire data directory without
touching the DB entry; useful for cleaning up ghost dirs or corrupted accounts
- **Delete all activities** (`DELETE /api/admin/users/{handle}/activities`) — wipes
`activities/`, `edits/`, `_merged/`, and `index.json` for a handle, then triggers a rebuild;
admin page shows a confirmation `<dialog>` before firing
- **"Admin" nav link** — visible in the top-right for admins only
### New feature — Password reset (admin-generated one-time code)
No email infrastructure required. Flow:
1. Admin visits `/admin/` → clicks "Reset pwd" → a 24-hour code appears inline (click to copy)
2. Admin sends it out-of-band (Signal, Telegram, etc.)
3. User goes to `/reset-password/`, enters handle + code + new password
- `POST /api/admin/users/{handle}/reset-password-code` (admin) → `{code, expires_in_hours: 24}`
- `POST /api/auth/reset-password` (public) → body `{handle, code, password}`
- `reset_codes` table in `instance.db`; generating a new code invalidates prior unused codes;
used codes kept for audit
### New feature — Re-extract from Strava originals
- **`POST /api/admin/reextract`** — re-runs the extract pipeline over all
`{user_dir}/originals/strava/*.json` files without hitting the Strava API again;
streams progress via SSE; useful after pipeline improvements
- Runs as a subprocess to avoid OOM (`malloc_trim` + `gc.collect` every 50 activities);
processes in batches of 100 to bound peak RSS
### New feature — Community page
- **`/community/` tab** — sortable table of all registered users: display name, handle,
member since, invited by; replaces the earlier inline community section on the about page
### New feature — Streaming upload progress
- **`POST /api/upload`** now returns `text/event-stream` instead of JSON
- Per-file progress events: `↓ 3/47 (6%) — morning_ride.fit`
- Final `done` event: `"12 added, 35 duplicates"`
- Vite proxy configured to not buffer the stream
### Bug fixes
- **`elevation_gain_m` null for modern Garmin FIT files** — session message `total_ascent`
field now read as fallback when per-point elevation gain is zero
- **Map flash on activity detail** — map container height set before `fitBounds` to prevent
a zero-height frame during load
- **Absolute `track_url` / `detail_url` paths** — `ActivityDetail` and `loadActivity` now
handle both relative and absolute paths in BAS JSON
- **Corrupted time streams causing OOM** — `metrics.py` guards against non-monotonic or
pathologically large time arrays before allocating the 1 Hz dense array
- **Merge race condition** — `merge_all` wipe + rewrite is now guarded; concurrent upload
triggers can no longer interleave a `shutil.rmtree` with a write from another request
- **Temp ZIP leak** — upload temp files now written to `/tmp/` and always deleted in a
`finally` block; a startup hook auto-cleans any leftovers
- **`bincio init` always overwrites `private`** — fixed to preserve existing value when
`index.json` already exists
- **Auth wall flash** — `Base.astro` now sets the auth state synchronously from a cookie
hint before the `fetch('/api/me')` resolves, eliminating the visible flash
- **Single-user redirect loop** — `index.astro` no longer redirects to `/u/{handle}/` on
private (multi-user) instances
- **Theme-aware Plot tooltips** — forced black text on white background; was rendering
grey-on-white (unreadable in light mode) and white-on-dark (unreadable in dark mode)
- **Theme-aware chart axis colors** — axis labels and tick marks now use the correct
foreground color in both light and dark themes
- **TS type annotation in `define:vars` script** — removed; Astro injects `define:vars`
blocks as plain JS, not TypeScript
- **Image refs with spaces/parens in filenames** — local image references in markdown
descriptions are now stripped before rendering to avoid broken inline `<img>` tags
---
## [Unreleased] — 2026-04-10 ## [Unreleased] — 2026-04-10
### New feature — Per-instance user limit ### New feature — Per-instance user limit
-113
View File
@@ -1,113 +0,0 @@
# Documentation proposal
## Problem
The project has no user-facing or developer-facing docs. Knowledge lives in `CLAUDE.md`
(written for AI context, not humans), scattered inline comments, and the code itself.
As the feature surface grows and more users join, we need:
- A guide for **users** (how to upload, sync, edit, manage privacy)
- A guide for **admins** (how to run an instance, manage users, reset passwords)
- An **API reference** (what endpoints exist, what they expect, what they return)
- A **developer guide** (how to run locally, architecture, how to contribute)
---
## Proposed structure
```
docs/
index.md Overview and quick links
user-guide.md End-user: upload, sync, edit, privacy, settings
admin-guide.md Admin: deploy, init, invite users, reset passwords, rebuild
api.md API reference (hand-written, augmented by OpenAPI)
architecture.md BAS schema, data flow, shard model, federation
developer-guide.md Local setup, how to run tests, how to contribute
```
`CLAUDE.md` stays as-is — it is AI context, not user docs. The two serve different
audiences and should not be merged.
---
## API documentation strategy
FastAPI auto-generates an OpenAPI 3.1 spec from the route decorators. It is already
served at `/api/docs` (Swagger UI) and `/api/redoc` (ReDoc) when the server is running.
Right now the auto-docs are sparse because:
- Most endpoints return bare `JSONResponse` instead of typed Pydantic response models
- Endpoint docstrings are minimal or absent
- Request bodies are raw `request.json()` instead of Pydantic models
### Recommended approach: two-layer docs
**Layer 1 — machine-readable (OpenAPI, auto-generated)**
Incrementally add Pydantic request/response models to the endpoints that matter most
(auth, activity CRUD, admin actions). FastAPI will pick them up automatically and the
Swagger UI becomes usable. No extra tooling needed.
Priority endpoints to type first:
- `POST /api/auth/login` / `logout` / `reset-password`
- `POST /api/register`
- `GET /api/me`
- `GET|POST /api/activity/{id}`
- `DELETE /api/activity/{id}`
- `POST /api/admin/users/{handle}/reset-password-code`
- `GET|POST /api/me/preferences` (once built)
**Layer 2 — human-readable (`docs/api.md`)**
A hand-written reference that groups endpoints by domain (auth, activities, admin,
sync), explains the overall auth model (cookie-based, httpOnly), rate limiting, and
covers things OpenAPI can't express well (SSE streams, error semantics, side effects
like rebuild triggers).
The OpenAPI spec and the hand-written doc are complementary, not duplicates:
OpenAPI is precise and machine-readable; `api.md` gives context and explains *why*.
---
## Tooling options
| Option | Pros | Cons |
|--------|------|------|
| Plain markdown in `docs/` | Zero tooling, lives in repo, renders on GitHub | No search, no versioning, no sidebar nav |
| MkDocs + Material theme | Beautiful, search, auto-nav from folder structure, can embed OpenAPI via plugins | Needs Python dep + build step; another thing to deploy |
| Docusaurus | Great for open-source projects, versioning, i18n | Node toolchain, heavier |
| VitePress | Fast, Vite-based (already in the stack), markdown + Vue | Still a separate site to host |
| Just the Swagger UI at `/api/docs` | Auto-generated, always up-to-date | Only covers the API, not user/admin/architecture |
**Recommendation:** Start with plain markdown in `docs/` — no build step, always
available, no new infrastructure. If the project goes public or the user base grows,
migrate to MkDocs Material (one `mkdocs.yml` + `pip install mkdocs-material`).
For the API specifically: enable the Swagger UI on the live server (currently it may
be disabled in production) so admins can explore it directly at `/api/docs`.
---
## Enabling Swagger UI in production
By default FastAPI serves `/docs` and `/redoc`. In `bincio serve`, the FastAPI app is
created with:
```python
app = FastAPI(docs_url=None, redoc_url=None) # check current value
```
For a private instance (auth-walled), it is safe to expose `/api/docs` — add a note
in `admin-guide.md` that it exists. Alternatively, serve it only when an env var is set.
---
## Suggested first milestone
1. Create `docs/` with `index.md`, `admin-guide.md`, `api.md`
2. `admin-guide.md`: deploy, init, invite, password reset, rebuild, reset data
3. `api.md`: auth endpoints + activity endpoints, hand-written
4. Enable Swagger UI on the server (or at least document that it exists at `/api/docs`)
5. Add Pydantic models to the 8 priority endpoints above
Everything else (user guide, architecture, developer guide, MkDocs) is second milestone.
+78
View File
@@ -0,0 +1,78 @@
# squash-for-github branch strategy
`squash-for-github` is a curated public-facing branch. It has its own orphan
history (unrelated to `main`) and grows by appending one large squash commit
each time you want to publish a batch of work.
## When to use
Whenever `main` has accumulated enough work worth publishing — typically after a
meaningful feature set or before tagging a release.
## How it works
`squash-for-github` and `main` have completely unrelated histories (different
root commits). Because of this, `git merge --squash` won't work. Instead, use
`git commit-tree` to create a new commit that carries **main's file tree** but
is **parented to the current squash-for-github tip**.
## Steps
1. **Collect commit messages** to write the summary:
```bash
git log --oneline main ^squash-for-github
```
2. **Switch to the branch:**
```bash
git checkout squash-for-github
```
3. **Create the squash commit** (replace the message with your summary):
```bash
NEW=$(git commit-tree main^{tree} -p HEAD -m "feat: your summary here")
git reset --hard $NEW
```
Or as a one-liner with a heredoc for a multi-line message:
```bash
git reset --hard $(git commit-tree main^{tree} -p HEAD -m "$(cat <<'EOF'
feat: short title
- bullet one
- bullet two
EOF
)")
```
4. **Verify:**
```bash
git log --oneline squash-for-github | head -5
```
5. **Push** when ready:
```bash
git push origin squash-for-github
# or force-push if you've rewritten history on the remote:
git push --force origin squash-for-github
```
6. **Return to main:**
```bash
git checkout main
```
## Why not `git merge --squash`?
The two branches share no common ancestor, so git refuses with
`fatal: refusing to merge unrelated histories`. `git commit-tree` bypasses this
by directly constructing the commit object: it takes the tree (file snapshot)
from `main`, sets the parent to the current `squash-for-github` tip, and
attaches your custom message — no merge machinery needed.
-223
View File
@@ -1,223 +0,0 @@
# BincioActivity — Context for Claude
## What this project is
BincioActivity is a federated, open-source, self-hosted activity stats platform
(think personal Strava). Two-stage pipeline:
1. **`bincio extract`** (Python): GPX/FIT/TCX → BAS JSON data store
2. **`bincio render`** (Astro/Node): BAS data store → static website
The BAS (BincioActivity Schema) JSON files are the federation protocol.
Anyone can publish their data as BAS JSON and others can include it.
## Key design decisions
- **No database, no server** — everything is static files
- **Python with uv** for the extract stage
- **Astro + Svelte + Tailwind + MapLibre GL + Observable Plot** for the site
- **Haversine** (not geopy) for distance calculations (10x faster)
- **Worker initializer pattern** for ProcessPoolExecutor — large shared data
(strava_lookup dict, known_hashes frozenset) is sent once per worker via
`initializer=`, not once per task
- **BAS activity IDs** always use UTC with Z suffix for URL safety
- **TCX files** from Garmin use both `http://` and `https://` namespace URIs —
parser handles both
## Your data
- Source: `~/your-activity-data/`
- `activities/` — Strava export (GPX, FIT, TCX, all with .gz variants)
- Any subdirectories with FIT files from Garmin/Karoo devices
- `activities.csv` — Strava metadata (names, descriptions, gear)
- Extracted output: `~/bincio_data/` (or `/tmp/bincio_test/` for testing)
Configure input paths in `extract_config.yaml`.
## Project structure
```
bincio/ Python package
extract/
models.py DataPoint, ParsedActivity, LapData
parsers/ GPX, FIT, TCX parsers + factory
sport.py sport name normalisation
metrics.py haversine-based stats computation (single pass)
timeseries.py downsample to 1Hz, build BAS timeseries object
simplify.py RDP track simplification → GeoJSON
dedup.py exact (hash) + near-duplicate detection
strava_csv.py Strava activities.csv importer
writer.py BAS JSON + GeoJSON writer
config.py extract_config.yaml loader
cli.py `bincio extract` CLI
render/
cli.py `bincio render` CLI (symlinks data, runs astro build/dev)
merge.py sidecar edit overlay (produces _merged/)
edit/
cli.py `bincio edit` CLI
server.py FastAPI write API for the edit drawer
schema/
bas-v1.schema.json JSON Schema for BAS
SCHEMA.md Human-readable BAS spec
site/ Astro project
src/
layouts/Base.astro
pages/
index.astro Activity feed (loads index.json client-side)
activity/[id].astro Single activity (SSG, loads detail JSON client-side)
stats/index.astro Heatmap + year totals
components/
ActivityFeed.svelte Card grid, sport filter, pagination
ActivityDetail.svelte Map + stats + charts + photo gallery
ActivityMap.svelte MapLibre GL (gradient track, linked hover dot)
ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence tabs)
StatsView.svelte Yearly heatmap + totals
EditDrawer.svelte Slide-in edit panel (visible when PUBLIC_EDIT_URL set)
lib/
types.ts BAS TypeScript types
format.ts formatDistance, formatDuration, sportIcon, etc.
```
## How to run
```bash
# Extract
cd ~/src/bincio_activity
uv run bincio extract --input ~/your-activity-data/activities --output /tmp/bincio_test
# Site dev server
cd site
ln -sf /tmp/bincio_test/_merged public/data # point at merged output
cp .env.example .env && $EDITOR .env # set BINCIO_DATA_DIR
npm run dev
# Edit server (optional — enables edit drawer in the site)
uv run bincio edit --data-dir /tmp/bincio_test
# set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
# Tests
uv run pytest
```
## MapLibre GL + Vite/Astro — known gotchas
Learnt the hard way during debugging (March 2026):
- **`maplibregl.workerUrl = ...` is the v3 API and silently no-ops in v4+.**
The v5 API is `maplibregl.setWorkerUrl(url)`, but you don't need it at all in a
normal Vite environment — MapLibre handles the blob worker automatically.
- **`optimizeDeps: { exclude: ['maplibre-gl'] }` breaks tile loading.**
It prevents Vite from converting MapLibre's UMD bundle to ESM. The UMD bundle
uses AMD `define()` internally; served raw, the tile worker blob fails silently →
black map, no tiles. The correct setting is `include: ['maplibre-gl']`.
- **`build.target: 'es2022'` (and `optimizeDeps.esbuildOptions.target`) is required.**
MapLibre's dependencies use ES2022 class field syntax. If esbuild downgrades it,
helpers like `__publicField` aren't available inside the serialised worker blob
scope → tile loading fails. This is a known upstream issue (maplibre-gl-js #6680).
- **Use static imports, not dynamic `await import('maplibre-gl')`, when possible.**
With `client:only="svelte"` in Astro, SSR never runs for the component so there is
no `window is not defined` risk. Static import lets Vite pre-bundle correctly.
- **Use `client:only="svelte"` (not `client:load`) for the activity detail page.**
`client:load` does SSR + hydration; complex interactive components with MapLibre
can hit hydration mismatch issues. `client:only` mounts fresh on the client only.
- **MapLibre v5 requires explicit `center` and `zoom` in the Map constructor.**
v4 silently defaulted to `center: [0,0], zoom: 0`. v5 leaves internal projection
state undefined → `Cannot read properties of undefined (reading 'lng')` crashes
on any operation that touches coordinates (markers, resize, render). Always pass
`center` and `zoom` even if you plan to `fitBounds` later.
- **MapLibre v5 requires `setLngLat()` on markers before `.addTo(map)`.**
v4 tolerated markers without coordinates. v5 calls `Marker._update()` inside
`addTo()`, which needs valid lngLat → same `'lng'` crash. Set a dummy `[0, 0]`
if the real position arrives later (e.g. hover markers).
## Observable Plot — known gotchas
- **Curve names are hyphenated, not camelCase.**
Use `"monotone-x"`, not `"monotoneX"`. Plot uses its own curve name registry
(not raw d3 identifiers). Wrong names throw `unknown curve` at runtime.
The working `astro.config.mjs` Vite section:
```js
vite: {
optimizeDeps: {
include: ['maplibre-gl'],
esbuildOptions: { target: 'es2022' },
},
build: { target: 'es2022' },
},
```
## Activity sidecar edits — design spec
Users edit activities via **sidecar markdown files** in the data dir.
No database, no server — consistent with the project's static-files-only philosophy.
### File naming
```
~/bincio_data/
activities/{id}.json ← immutable extract output
edits/{id}.md ← user edits (sidecar)
edits/images/{id}/ ← uploaded photos
_merged/ ← render-time merge output (gitignored-style)
```
### Sidecar format
```markdown
---
title: "Epic climb up Monte Grappa"
sport: cycling
hide_stats: [cadence]
highlight: true
private: false
gear: "Trek Domane"
---
Rode with friends. Legs felt great after the rest week...
```
### Editing UX: drawer in Astro + `bincio edit` write API
- `bincio edit --data-dir ~/bincio_data` starts a FastAPI server on port 4041
- Set `PUBLIC_EDIT_URL=http://localhost:4041` in `site/.env` to enable the edit button
- Clicking Edit on any activity detail page opens a slide-in drawer
- Saving writes the sidecar and triggers `merge_all()` automatically
- `bincio render` always runs `merge_all()` before build/serve and symlinks `public/data``_merged/`
### `PUBLIC_EDIT_URL` as feature flag
- **Unset** → no Edit button, normal static site
- **Set** → edit drawer enabled; lives in `site/.env` (gitignored)
## Known issues / next steps
- `bincio render` Python CLI is functional but `--watch` mode not yet implemented
- Activity IDs in older test data may use `+0000` format (pre-fix); re-run extract to get `Z` format
- Some activities appear with both untitled and titled IDs (near-dedup timing race)
- Federation (remote data sources) not yet implemented in site
- Friends pages (`/friends/{handle}/`) not yet implemented
- The `site/.env` file is gitignored — copy from `site/.env.example`
## What "good" looks like (not yet done)
- [ ] `bincio render` Python CLI wraps `astro build` properly
- [ ] Friends/federation pages in site
- [ ] Personal records page
- [ ] Activity search / full-text filter in feed
- [ ] GitHub Actions template for auto-publish
- [ ] Karoo/Garmin Connect importers beyond Strava
- [x] `bincio.render.merge` — sidecar parser, `_merged/` output, private filter, highlight sort
- [x] `bincio edit` FastAPI write API (GET/POST activity, image upload/delete, triggers merge)
- [x] `EditDrawer.svelte` — slide-in edit UI in the Astro site
- [x] `PUBLIC_EDIT_URL` feature flag
- [x] Markdown rendering in activity description with image path rewriting
- [x] Photo gallery with lightbox on activity detail page
- [ ] `bincio render --watch` incremental rebuild on sidecar/data changes
- [ ] Highlight badge in activity feed cards
-31
View File
@@ -1,31 +0,0 @@
owner:
handle: yourname
display_name: Your Name
input:
dirs:
- ~/Activities/gpx
- ~/Activities/fit
# Strava bulk export metadata — provides names, descriptions, gear
# metadata_csv: ~/strava_export/activities.csv
output:
dir: ~/bincio_data
default_privacy: public
sensors:
heart_rate: true
cadence: true
temperature: true
power: true
track:
simplify: rdp
rdp_epsilon: 0.0001 # ~11m at equator
timeseries_hz: 1 # 1 sample/second max
classifier:
enabled: false # ML activity type classifier (requires scikit-learn extra)
incremental: true # skip files whose hash hasn't changed since last run
-72
View File
@@ -1,72 +0,0 @@
# BincioActivity — public release manifest
# One relative path per line.
# If publish/<path> exists, that sanitized version is used instead of the original.
.gitignore
.python-version
CHANGELOG.md
CHEATSHEET.md
CLAUDE.md
README.md
SCHEMA.md
pyproject.toml
extract_config.example.yaml
schema/bas-v1.schema.json
bincio/__init__.py
bincio/cli.py
bincio/edit/__init__.py
bincio/edit/cli.py
bincio/edit/server.py
bincio/extract/__init__.py
bincio/extract/cli.py
bincio/extract/config.py
bincio/extract/dedup.py
bincio/extract/metrics.py
bincio/extract/models.py
bincio/extract/parsers/__init__.py
bincio/extract/parsers/base.py
bincio/extract/parsers/factory.py
bincio/extract/parsers/fit.py
bincio/extract/parsers/gpx.py
bincio/extract/parsers/tcx.py
bincio/extract/simplify.py
bincio/extract/sport.py
bincio/extract/strava_csv.py
bincio/extract/timeseries.py
bincio/extract/writer.py
bincio/import_/__init__.py
bincio/import_/cli.py
bincio/import_/strava.py
bincio/render/__init__.py
bincio/render/cli.py
bincio/render/merge.py
publish.sh
publish/CLAUDE.md
publish/extract_config.example.yaml
publish/manifest
site/.env.example
site/astro.config.mjs
site/package.json
site/tailwind.config.mjs
site/tsconfig.json
site/src/components/ActivityCharts.svelte
site/src/components/ActivityDetail.svelte
site/src/components/ActivityFeed.svelte
site/src/components/ActivityMap.svelte
site/src/components/AthleteDrawer.svelte
site/src/components/AthleteView.svelte
site/src/components/EditDrawer.svelte
site/src/components/MmpChart.svelte
site/src/components/RecordsView.svelte
site/src/components/StatsView.svelte
site/src/layouts/Base.astro
site/src/lib/format.ts
site/src/lib/types.ts
site/src/pages/activity/[id].astro
site/src/pages/athlete/index.astro
site/src/pages/index.astro
site/src/pages/stats/index.astro
tests/__init__.py
tests/test_merge.py
tests/test_sport.py
tests/test_writer.py
+2 -2
View File
@@ -1,9 +1,9 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Pull user feedback from the VPS into ./feedback/ locally. # Pull user feedback from the VPS into ./feedback/ locally.
# Usage: bash scripts/pull_feedback.sh [vps-host] (default: root@95.216.55.151) # Usage: bash scripts/pull_feedback.sh <user@host>
set -e set -e
VPS=${1:-root@95.216.55.151} VPS=${1:?Usage: $0 user@host}
REMOTE=/var/bincio/data/_feedback REMOTE=/var/bincio/data/_feedback
LOCAL=$(dirname "$0")/../feedback LOCAL=$(dirname "$0")/../feedback