improve configs, update docs

This commit is contained in:
Davide Scaini
2026-03-30 13:30:43 +02:00
parent a65923c3e0
commit d806072546
14 changed files with 799 additions and 109 deletions
+4
View File
@@ -24,5 +24,9 @@ site/.astro/
bincio_data/ bincio_data/
*.bincio_cache.json *.bincio_cache.json
# Local config / secrets (never commit)
.env
extract_config.yaml
# OS # OS
.DS_Store .DS_Store
+27
View File
@@ -2,6 +2,33 @@
## [Unreleased] — 2026-03-30 ## [Unreleased] — 2026-03-30
### Data ingestion
- **`bincio import strava`** — OAuth2 Strava importer (`bincio/import_/strava.py` + `bincio/import_/cli.py`)
- One-shot local OAuth2 callback server (port 8976); opens browser, receives code, exchanges for tokens
- Tokens saved to `~/.config/bincio/strava.json`; auto-refreshed on expiry (6h TTL)
- Fetches paginated activity list with `after=` timestamp for efficient incremental runs
- Per activity: `GET /activities/{id}/streams``_strava_to_parsed()``compute()``write_activity()`
- `_patch_from_summary()`: fills `None` metrics from Strava summary when sensors are missing (manual entries, indoor rides)
- Sync state persisted in `data_dir/_strava_sync.json` (imported IDs + last sync timestamp)
- Rate limit tracking via `X-RateLimit-Usage`; warns at 85% of 15-min window; auto-retries on 429
- Credentials read from (in order): CLI flags → env vars → `extract_config.yaml` under `import.strava`
- Install: `uv sync --extra strava`
- **Web file upload** — `POST /api/upload` in `bincio/edit/server.py`
- Accepts FIT/GPX/TCX (`.gz` variants too); 409 if activity already exists
- Runs full extract pipeline inline: `parse_file()``compute()``write_activity()``merge_all()`
- Staged to `data_dir/_uploads/` during processing; cleaned up in `finally`
- `↑` button in site nav, gated behind `PUBLIC_EDIT_URL`; drag-and-drop modal; auto-redirects on success
- **`extract_config.yaml` is now gitignored** — safe to store credentials under `import.strava`
- `StravaConfig` dataclass added to `bincio/extract/config.py`; parsed from `import.strava:` block
- `extract_config.example.yaml` is the tracked template
- **Theme-aware heatmap** (`StatsView.svelte`) — `applyIntensity()` now lerps from the correct
background colour in both dark (zinc-800 `#27272a`) and light (zinc-200 `#e4e4e7`) modes;
`emptyColor` and `baseRgb` reactive to `data-theme` via `MutationObserver`
### Athlete page ### Athlete page
- **`/athlete` page** — three-tab layout: Power Curve · Records · Profile - **`/athlete` page** — three-tab layout: Power Curve · Records · Profile
+59 -7
View File
@@ -3,13 +3,16 @@
## Daily workflow ## Daily workflow
```bash ```bash
# 1. Drop new .fit / .gpx / .tcx files into your input dir, then: # Option A — local files (Karoo / Garmin / Wahoo)
uv run bincio extract uv run bincio extract # processes new/changed files, skips unchanged
# 2. Rebuild the site (merges any sidecar edits, then builds) # Option B — pull from Strava (incremental; credentials in extract_config.yaml)
uv run bincio import strava # fetches only activities since last sync
# Rebuild the site (merges any sidecar edits, then builds)
uv run bincio render uv run bincio render
# 3. Done — copy site/dist/ to your host # Done — copy site/dist/ to your host
``` ```
--- ---
@@ -29,6 +32,47 @@ To force a full re-extract: `rm -rf ~/bincio_data && uv run bincio extract`
--- ---
## Import from Strava
```bash
# Install (one-time)
uv sync --extra strava
# Add credentials to extract_config.yaml (gitignored — safe for secrets):
# import:
# strava:
# client_id: 12345
# client_secret: your_secret
# First run — opens browser for OAuth, then imports all activities:
uv run bincio import strava
# Subsequent runs are incremental (only fetches since last sync):
uv run bincio import strava
# Other options:
uv run bincio import strava --since 2025-01-01 # explicit date cutoff
uv run bincio import strava --reauth # force new OAuth flow
uv run bincio import strava --output ~/other_dir # override output dir
```
Credentials resolution order:
1. `--client-id` / `--client-secret` flags
2. `STRAVA_CLIENT_ID` / `STRAVA_CLIENT_SECRET` env vars
3. `import.strava.client_id` / `client_secret` in `extract_config.yaml`
Tokens saved to `~/.config/bincio/strava.json` and auto-refreshed (6h TTL).
Sync state (imported IDs + last sync timestamp) in `data_dir/_strava_sync.json`.
---
## File upload (web UI)
When `PUBLIC_EDIT_URL` is set in `site/.env`, a `↑` button appears in the nav.
Drag a FIT/GPX/TCX onto the modal → the activity is extracted and appears immediately.
---
## Render ## Render
```bash ```bash
@@ -111,6 +155,8 @@ IDs are stable — safe to use in bookmarks and links.
## extract_config.yaml — key fields ## extract_config.yaml — key fields
This file is **gitignored** — copy from `extract_config.example.yaml` and add your credentials safely.
```yaml ```yaml
owner: owner:
handle: yourname handle: yourname
@@ -130,6 +176,11 @@ track:
rdp_epsilon: 0.0001 # GPS simplification — larger = fewer points rdp_epsilon: 0.0001 # GPS simplification — larger = fewer points
timeseries_hz: 1 # samples/sec in stored JSON (1 = 1 Hz) timeseries_hz: 1 # samples/sec in stored JSON (1 = 1 Hz)
import:
strava:
client_id: 12345 # from strava.com/settings/api
client_secret: abc # Authorization Callback Domain must be: localhost
athlete: athlete:
max_hr: 182 # used for context; zones below are authoritative max_hr: 182 # used for context; zones below are authoritative
ftp_w: 280 # functional threshold power in watts ftp_w: 280 # functional threshold power in watts
@@ -249,12 +300,13 @@ print(len(others), 'total')
| File | Purpose | | File | Purpose |
|---|---| |---|---|
| `extract_config.yaml` | Main config (input dirs, output dir, privacy) | | `extract_config.yaml` | Main config input dirs, output dir, athlete zones, Strava credentials. **Gitignored.** Copy from `.example`. |
| `site/.env` | Site env vars (`BINCIO_DATA_DIR`, `PUBLIC_EDIT_URL`) — copy from `.env.example` | | `site/.env` | Site env vars (`BINCIO_DATA_DIR`, `PUBLIC_EDIT_URL`) — copy from `site/.env.example`. Gitignored. |
| `SCHEMA.md` | BAS format specification | | `SCHEMA.md` | BAS format specification |
| `CLAUDE.md` | Dev notes, gotchas, design decisions | | `CLAUDE.md` | Dev notes, gotchas, design decisions |
| `bincio/render/merge.py` | Sidecar overlay logic — `parse_sidecar`, `merge_all` | | `bincio/render/merge.py` | Sidecar overlay logic — `parse_sidecar`, `merge_all` |
| `bincio/edit/server.py` | FastAPI edit API — GET/POST activity, image upload | | `bincio/edit/server.py` | FastAPI edit API — GET/POST activity, image upload, file upload (`POST /api/upload`) |
| `bincio/import_/strava.py` | Strava OAuth2 client + stream → BAS conversion |
| `bincio/extract/sport.py` | Sport name normalisation + mapping | | `bincio/extract/sport.py` | Sport name normalisation + mapping |
| `bincio/extract/metrics.py` | Distance, speed, HR, elevation computation | | `bincio/extract/metrics.py` | Distance, speed, HR, elevation computation |
| `bincio/extract/parsers/fit.py` | FIT file parser | | `bincio/extract/parsers/fit.py` | FIT file parser |
+106 -80
View File
@@ -51,24 +51,32 @@ bincio/ Python package
writer.py BAS JSON + GeoJSON writer writer.py BAS JSON + GeoJSON writer
config.py extract_config.yaml loader config.py extract_config.yaml loader
cli.py `bincio extract` CLI cli.py `bincio extract` CLI
import_/
strava.py Strava API importer (OAuth2, streams → BAS JSON)
cli.py `bincio import strava` CLI
render/ render/
cli.py `bincio render` CLI (symlinks data, runs astro build/dev) cli.py `bincio render` CLI (symlinks data, runs astro build/dev)
edit/
server.py FastAPI write API (activity edits, image upload, file upload)
cli.py `bincio edit` CLI
schema/ schema/
bas-v1.schema.json JSON Schema for BAS bas-v1.schema.json JSON Schema for BAS
SCHEMA.md Human-readable BAS spec SCHEMA.md Human-readable BAS spec
site/ Astro project site/ Astro project
src/ src/
layouts/Base.astro layouts/Base.astro Nav (upload button + theme toggle), theme CSS vars
pages/ pages/
index.astro Activity feed (loads index.json client-side) index.astro Activity feed (loads index.json client-side)
activity/[id].astro Single activity (SSG, loads detail JSON client-side) activity/[id].astro Single activity (SSG, loads detail JSON client-side)
stats/index.astro Heatmap + year totals stats/index.astro Heatmap + year totals
athlete/index.astro MMP curve + athlete profile (planned)
components/ components/
ActivityFeed.svelte Card grid, sport filter, pagination ActivityFeed.svelte Card grid, sport filter, pagination
ActivityDetail.svelte Map + stats + charts wrapper ActivityDetail.svelte Map + stats + charts wrapper
ActivityMap.svelte MapLibre GL (gradient track, linked hover dot) ActivityMap.svelte MapLibre GL (gradient track, linked hover dot)
ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence/power tabs) ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence/power tabs)
StatsView.svelte Yearly heatmap + totals StatsView.svelte Yearly heatmap + click-to-pin tooltip
EditDrawer.svelte Slide-in activity editor
lib/ lib/
types.ts BAS TypeScript types types.ts BAS TypeScript types
format.ts formatDistance, formatDuration, sportIcon, etc. format.ts formatDistance, formatDuration, sportIcon, etc.
@@ -77,14 +85,23 @@ site/ Astro project
## How to run ## How to run
```bash ```bash
# Extract # Extract from local files
cd ~/src/bincio_activity cd ~/src/bincio_activity
uv run bincio extract --input ~/src/cycling_data_davide/activities --output /tmp/bincio_test uv run bincio extract --input ~/src/cycling_data_davide/activities --output /tmp/bincio_test
# Import from Strava (credentials in extract_config.yaml under import.strava)
uv sync --extra strava
uv run bincio import strava # first run opens browser for OAuth
uv run bincio import strava # subsequent runs are incremental
# Site dev server # Site dev server
cd site cd site
ln -sf /tmp/bincio_test public/data # symlink data ln -sf /tmp/bincio_test public/data # symlink data
BINCIO_DATA_DIR=/tmp/bincio_test npm run dev npm run dev
# Edit server (enables drawer + file upload in the site)
uv run bincio edit --data-dir ~/bincio_data # port 4041
# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
# Tests # Tests
uv run pytest uv run pytest
@@ -489,116 +506,125 @@ to power-having activities, pull their `mmp` arrays, take element-wise max per s
7. `AthleteDrawer.svelte` — zones + gear editing form 7. `AthleteDrawer.svelte` — zones + gear editing form
8. Season config in `extract_config.yaml` / `edits/athlete.yaml` 8. Season config in `extract_config.yaml` / `edits/athlete.yaml`
## Data ingestion — design plan ## Data ingestion
How activity data gets into BincioActivity. Three orthogonal vectors. How activity data gets into BincioActivity. Three orthogonal vectors.
### Vector 1 — Web file upload (extends existing edit server) ### Vector 1 — Web file upload
The lowest-friction path: drag a FIT/GPX/TCX file onto the site, it appears immediately. Drag a FIT/GPX/TCX onto the site while the edit server is running — activity
appears immediately.
**Backend** (`bincio/edit/server.py`):
``` ```
POST /api/upload multipart FIT/GPX/TCX POST /api/upload multipart FIT/GPX/TCX (also .gz variants)
→ saves to staging dir → stages file to data_dir/_uploads/<name>
bincio extract on just that file parse_file() → compute() → write_activity() → build_summary()
merge_all() 409 if activity already exists (same timestamp = same ID)
returns { id, redirect: "/activity/{id}/" } updates index.json + merge_all()
→ returns { ok: true, id: "2024-05-15T103000Z" }
→ cleans up staged file in finally block
``` ```
An "Upload activity" button in the nav (gated behind `PUBLIC_EDIT_URL` like the edit drawer). **Frontend** (`site/src/layouts/Base.astro`):
No CLI needed. Preserves static-site output — the server only exists in local editing mode. - `↑` button in nav right cluster, only rendered when `PUBLIC_EDIT_URL` is set
- Modal with drag-and-drop zone + click-to-browse
- Auto-redirects to `/activity/{id}/` on success
- Escape / backdrop click closes modal
### Vector 2 — Platform importers ### Vector 2 — `bincio import strava` ✓
#### `bincio import strava` — pull your Strava history **`bincio/import_/strava.py`** + **`bincio/import_/cli.py`**
Install: `uv sync --extra strava`
```bash ```bash
bincio import strava \ # First run (full sync — opens browser):
--client-id 12345 \ bincio import strava --client-id 12345 --client-secret abc --output ~/bincio_data
--client-secret abc... \
--output ~/bincio_data \ # Subsequent runs (incremental — picks up from last sync automatically):
--since 2024-01-01 # optional, default: all-time bincio import strava --client-id 12345 --client-secret abc
# Explicit date range:
bincio import strava --client-id 12345 --client-secret abc --since 2025-01-01
# Force re-auth (rotate credentials or re-authorize):
bincio import strava --client-id 12345 --client-secret abc --reauth
# Credentials via env vars (good for scripts):
export STRAVA_CLIENT_ID=12345
export STRAVA_CLIENT_SECRET=abc
bincio import strava --output ~/bincio_data
``` ```
**How Strava API access works:** **Getting Strava API credentials (~2 minutes, no approval needed):**
1. Go to `strava.com/settings/api`
Every Strava user can register an API app at `strava.com/settings/api` — no review, 2. Create an application — name and website can be anything; set
no approval, no fees. Fill in a name, website (`localhost` is fine), and callback **Authorization Callback Domain** to `localhost`
domain (`localhost` for local use). You instantly get a Client ID and Client Secret. 3. Paste Client ID and Client Secret into `extract_config.yaml`:
```yaml
import:
strava:
client_id: 12345
client_secret: your_secret_here
```
`extract_config.yaml` is gitignored — safe to store credentials there.
Strava's "developer" label is misleading: formal review is only required for Strava's "developer" label is misleading: formal review is only required for
commercial apps used by *other* people. For a self-hosted personal tool, each user commercial apps that authenticate *other users*. For a personal self-hosted tool
brings their own credentials and authenticates their own account. Rate limits are you authenticate your own account — no review, no fees.
generous for personal use: **100 requests / 15 min, 1000 / day**. Rate limits: **100 req / 15 min, 1000 / day** (generous for personal use).
The importer: **How the importer works:**
1. Opens a local OAuth2 callback server (like `gh auth login`)
2. Pops a browser to `strava.com/oauth/authorize?scope=activity:read_all`
3. User clicks Authorize → callback receives the code → exchanges for tokens
4. Tokens saved to `~/.config/bincio/strava_tokens.json`
5. Fetches paginated activity list → for each, fetches streams (lat/lng, time,
altitude, HR, cadence, power, velocity) → converts to BAS JSON
6. Idempotent: existing IDs (matched by Strava activity ID embedded in BAS metadata)
are skipped. Safe to re-run for incremental sync.
Strava streams give the same data as FIT files at ~1 Hz (GPS, power meter, HR strap). *OAuth dance (first run):*
- Starts a one-shot local HTTP server on port 8976
- Opens `strava.com/oauth/authorize?scope=activity:read_all` in the browser
- Receives the authorization code at `/callback`
- Exchanges code for access + refresh tokens
- Saves to `~/.config/bincio/strava.json` (keyed by client_id)
- Subsequent runs load saved tokens and refresh silently when expired (6h TTL)
#### `bincio import garmin` — Garmin Connect *Sync loop:*
- Reads `data_dir/_strava_sync.json` for set of already-imported Strava IDs
and timestamp of last sync
- Uses Strava `after=<unix_ts>` parameter for server-side filtering (efficient —
no need to scan all pages on incremental runs; 1-hour overlap to catch late saves)
- Per activity: `GET /activities/{id}/streams` → `_strava_to_parsed()` →
`compute()` → `_patch_from_summary()` → `write_activity()` → `build_summary()`
- Writes updated `index.json` + `_strava_sync.json`
- Calls `merge_all()` if `edits/` directory exists
No official public API. Options: *Conversion details:*
- **`garminconnect` Python library** — unofficial but widely used (same approach as - `sport_type` (or `type`) → `normalise_sport()` — same mapping as FIT/GPX
tapiriik, garmin-connect-export). Works with email/password or session cookies. - Streams: `time` (s since start) + `latlng` + `altitude` + `heartrate` +
- **FIT file sync** — Garmin Express / Tapiriik sync FIT files to a local folder; `cadence` + `watts` + `velocity_smooth` (m/s → km/h) → `DataPoint` list
`bincio extract` picks them up normally. Simplest. - `source_hash`: `sha256("strava:{id}")` — stable, not file-content-based
- `_patch_from_summary()`: fills `None` metric fields (distance, duration,
elevation, HR, power) from the Strava activity summary for activities with
missing/sparse sensors or manual entries
- Rate limit: warns at 85% of 15-min window; auto-retries with 60s sleep on 429
#### Watch mode — for ongoing device sync ### Vector 3 — Platform watch mode (planned)
```bash ```bash
bincio extract --watch ~/Dropbox/Garmin/Activities --output ~/bincio_data bincio extract --watch ~/Dropbox/Garmin/Activities --output ~/bincio_data
``` ```
Watches a directory for new FIT/GPX/TCX files (using `watchfiles` or `inotify`). Directory watcher (`watchfiles` / `inotify`) for ongoing FIT sync from Karoo,
New file dropped → auto-extract → `merge_all()` → site reflects it on next reload. Garmin, Wahoo. New file → auto-extract → merge. Not yet implemented.
Zero friction for users who already sync files from Garmin/Karoo/Wahoo to a folder
via Dropbox, Syncthing, or Garmin Express.
#### Strava webhook — real-time push (advanced) ### Vector 4 — Federation (planned)
```bash
bincio edit --data-dir ~/bincio_data --webhook-strava
```
Registers a Strava webhook subscription. When you finish a ride, Strava pushes a
notification → server fetches streams → extract → merge. Requires a **publicly
accessible URL** (works with Tailscale, a VPS, or ngrok). Not needed for most
self-hosters; polling via `bincio import strava --since yesterday` is simpler.
### Vector 3 — Federation (pull remote BAS feeds)
The cleanest "data in from the web" path for the self-hosted model:
anyone who publishes a `index.json` at a public URL is a data source.
```yaml ```yaml
# extract_config.yaml # extract_config.yaml
sources: sources:
- url: https://alice.example.com/data/index.json - url: https://alice.example.com/data/index.json
handle: alice handle: alice
- url: https://bob.example.com/data/index.json
handle: bob
``` ```
`bincio render` fetches remote index files at build time, merges them into the `bincio render` fetches remote BAS index files at build time. No API keys.
site. No API keys, no OAuth. Local `.md` sidecars can annotate remote activities. Local `.md` sidecars can annotate remote activities.
Not yet implemented — see friends/federation items in the checklist below.
### Implementation priority
1. **Web file upload** — trivial to build, highest immediate UX value
2. **`bincio import strava`** — covers historical migration and incremental sync;
most cyclists already have years of data there
3. **Watch mode** — covers ongoing FIT-file-based workflows (Karoo, Garmin)
4. **Garmin Connect importer** — second most common platform
5. **Federation** — longer term; enables the "personal Strava" social layer
--- ---
@@ -626,7 +652,7 @@ Not yet implemented — see friends/federation items in the checklist below.
- [ ] Map thumbnail in activity cards (SVG path from GeoJSON) - [ ] Map thumbnail in activity cards (SVG path from GeoJSON)
- [ ] GitHub Actions template for auto-publish - [ ] GitHub Actions template for auto-publish
- [x] **Ingestion: web file upload** — `POST /api/upload` in edit server, drag-and-drop in nav - [x] **Ingestion: web file upload** — `POST /api/upload` in edit server, drag-and-drop in nav
- [ ] **Ingestion: `bincio import strava`** — OAuth2 + streams API, idempotent incremental sync - [x] **Ingestion: `bincio import strava`** — OAuth2 + streams API, idempotent incremental sync
- [ ] **Ingestion: `bincio extract --watch`** — directory watcher for ongoing FIT sync - [ ] **Ingestion: `bincio extract --watch`** — directory watcher for ongoing FIT sync
- [ ] **Ingestion: `bincio import garmin`** — garminconnect library or FIT folder sync - [ ] **Ingestion: `bincio import garmin`** — garminconnect library or FIT folder sync
- [ ] **Ingestion: federation** — `sources:` in config, remote BAS index pull at render time - [ ] **Ingestion: federation** — `sources:` in config, remote BAS index pull at render time
+33 -9
View File
@@ -12,13 +12,14 @@ BincioActivity is a self-hosted, federated activity stats platform. You point it
## How it works ## How it works
``` ```
GPX / FIT / TCX files GPX / FIT / TCX files Strava API
▼ ▼
bincio extract bincio import strava ← Pull from Strava, or upload via browser ↑
│ │
└────────────┬───────────┘
bincio extract ← Python CLI. Reads files, writes plain JSON. ~/bincio_data/ ← BAS data store. Plain JSON + GeoJSON.
~/bincio_data/ ← BAS data store. Human-readable JSON + GeoJSON.
edits/*.md ← Optional sidecar edits (titles, descriptions, photos). edits/*.md ← Optional sidecar edits (titles, descriptions, photos).
@@ -43,10 +44,16 @@ uv sync # installs the bincio package + all dependencies
# 2. Configure # 2. Configure
cp extract_config.example.yaml extract_config.yaml cp extract_config.example.yaml extract_config.yaml
$EDITOR extract_config.yaml # set input dirs, output dir, your name $EDITOR extract_config.yaml # set input dirs, output dir, your name
# extract_config.yaml is gitignored — safe to add credentials here
# 3. Extract activities → BAS JSON # 3a. Extract from local files
uv run bincio extract uv run bincio extract
# 3b. Or import from Strava
uv sync --extra strava
# Add credentials to extract_config.yaml under import.strava, then:
bincio import strava # opens browser on first run
# 4. Build the site (requires Node >= 20) # 4. Build the site (requires Node >= 20)
cd site && npm install && cd .. cd site && npm install && cd ..
cp site/.env.example site/.env # configure BINCIO_DATA_DIR cp site/.env.example site/.env # configure BINCIO_DATA_DIR
@@ -59,10 +66,11 @@ For live development with hot reload:
uv run bincio render --serve # merges edits, links data, starts astro dev uv run bincio render --serve # merges edits, links data, starts astro dev
# → http://localhost:4321 # → http://localhost:4321
# Optional: enable the activity edit UI # Optional: enable the activity edit UI + file upload
uv sync --extra edit # install FastAPI + uvicorn (one-time) uv sync --extra edit # install FastAPI + uvicorn (one-time)
uv run bincio edit # starts edit server on http://localhost:4041 uv run bincio edit # starts edit server on http://localhost:4041
# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env # Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
# → Edit button and ↑ Upload button appear in the site nav
``` ```
--- ---
@@ -112,6 +120,9 @@ uv sync # install / update dependencies
### `extract_config.yaml` ### `extract_config.yaml`
This is the single configuration file for the Python side of BincioActivity.
It is **gitignored** — safe to store credentials here. Copy from `extract_config.example.yaml`.
```yaml ```yaml
owner: owner:
handle: yourname handle: yourname
@@ -134,6 +145,13 @@ track:
incremental: true # skip files whose hash hasn't changed incremental: true # skip files whose hash hasn't changed
# Strava API credentials — from strava.com/settings/api
# Authorization Callback Domain must be set to: localhost
import:
strava:
client_id: 12345
client_secret: your_client_secret_here
# Optional: athlete profile for zone overlays on HR/power charts # Optional: athlete profile for zone overlays on HR/power charts
athlete: athlete:
max_hr: 182 max_hr: 182
@@ -211,6 +229,8 @@ At build time the renderer fetches their public data and renders it under `/frie
| Layer | Technology | | Layer | Technology |
|---|---| |---|---|
| Extract | Python 3.12, click, fitdecode, gpxpy, lxml | | Extract | Python 3.12, click, fitdecode, gpxpy, lxml |
| Strava import | requests (optional extra: `uv sync --extra strava`) |
| Edit server | FastAPI + uvicorn (optional extra: `uv sync --extra edit`) |
| Site framework | Astro 4 (static output) | | Site framework | Astro 4 (static output) |
| UI components | Svelte 5 | | UI components | Svelte 5 |
| Styling | Tailwind CSS v3 | | Styling | Tailwind CSS v3 |
@@ -235,12 +255,16 @@ bincio/ Python package
dedup.py hash-based + near-duplicate detection dedup.py hash-based + near-duplicate detection
strava_csv.py Strava activities.csv reader strava_csv.py Strava activities.csv reader
writer.py BAS JSON + GeoJSON writer writer.py BAS JSON + GeoJSON writer
config.py extract_config.yaml loader (includes import.strava)
import_/
strava.py Strava OAuth2 + streams → BAS JSON
cli.py `bincio import strava` entry point
render/ render/
cli.py `bincio render` — merge + astro build/serve cli.py `bincio render` — merge + astro build/serve
merge.py sidecar edit overlay (produces _merged/) merge.py sidecar edit overlay (produces _merged/)
edit/ edit/
cli.py `bincio edit` — local edit server cli.py `bincio edit` — local edit server
server.py FastAPI write API for the edit drawer server.py FastAPI write API (activity edits, image + file upload)
schema/ schema/
bas-v1.schema.json JSON Schema for BAS format bas-v1.schema.json JSON Schema for BAS format
SCHEMA.md Human-readable BAS specification SCHEMA.md Human-readable BAS specification
+3 -1
View File
@@ -1,4 +1,4 @@
"""Top-level CLI entry point: `bincio extract` and `bincio render`.""" """Top-level CLI entry point."""
import click import click
@@ -14,7 +14,9 @@ def main() -> None:
from bincio.extract.cli import extract # noqa: E402 from bincio.extract.cli import extract # noqa: E402
from bincio.render.cli import render # noqa: E402 from bincio.render.cli import render # noqa: E402
from bincio.edit.cli import edit # noqa: E402 from bincio.edit.cli import edit # noqa: E402
from bincio.import_.cli import import_group # noqa: E402
main.add_command(extract) main.add_command(extract)
main.add_command(render) main.add_command(render)
main.add_command(edit) main.add_command(edit)
main.add_command(import_group)
+14
View File
@@ -27,6 +27,12 @@ class ClassifierConfig:
enabled: bool = False # off by default; opt-in enabled: bool = False # off by default; opt-in
@dataclass
class StravaConfig:
client_id: str = ""
client_secret: str = ""
@dataclass @dataclass
class AthleteConfig: class AthleteConfig:
max_hr: int | None = None max_hr: int | None = None
@@ -48,6 +54,7 @@ class ExtractConfig:
owner_handle: str = "me" owner_handle: str = "me"
owner_display_name: str = "Me" owner_display_name: str = "Me"
athlete: AthleteConfig | None = None athlete: AthleteConfig | None = None
strava: StravaConfig | None = None
def load_config(path: Path) -> ExtractConfig: def load_config(path: Path) -> ExtractConfig:
@@ -87,6 +94,12 @@ def load_config(path: Path) -> ExtractConfig:
power_zones=ath_raw.get("power_zones"), power_zones=ath_raw.get("power_zones"),
) if ath_raw else None ) if ath_raw else None
strava_raw = (raw.get("import") or {}).get("strava") or {}
strava = StravaConfig(
client_id=str(strava_raw["client_id"]) if strava_raw.get("client_id") else "",
client_secret=str(strava_raw["client_secret"]) if strava_raw.get("client_secret") else "",
) if strava_raw else None
return ExtractConfig( return ExtractConfig(
input_dirs=dirs, input_dirs=dirs,
output_dir=out, output_dir=out,
@@ -99,6 +112,7 @@ def load_config(path: Path) -> ExtractConfig:
owner_handle=owner.get("handle", "me"), owner_handle=owner.get("handle", "me"),
owner_display_name=owner.get("display_name", "Me"), owner_display_name=owner.get("display_name", "Me"),
athlete=athlete, athlete=athlete,
strava=strava,
) )
+1
View File
@@ -0,0 +1 @@
"""BincioActivity importers — pull data from external platforms."""
+134
View File
@@ -0,0 +1,134 @@
"""bincio import — CLI command group for external platform importers."""
from __future__ import annotations
from pathlib import Path
from typing import Optional
import click
from rich.console import Console
console = Console()
@click.group("import")
def import_group() -> None:
"""Import activities from external platforms (Strava, Garmin, …)."""
@import_group.command("strava")
@click.option("--client-id", default=None, envvar="STRAVA_CLIENT_ID",
help="Strava API client ID. Falls back to import.strava.client_id in extract_config.yaml.")
@click.option("--client-secret", default=None, envvar="STRAVA_CLIENT_SECRET",
help="Strava API client secret. Falls back to import.strava.client_secret in extract_config.yaml.")
@click.option("--output", "output_dir", default=None,
help="BAS data store directory (default: from config or ~/bincio_data).")
@click.option("--config", "config_path", default=None,
help="Path to extract_config.yaml (default: ./extract_config.yaml).")
@click.option("--since", default=None, metavar="YYYY-MM-DD",
help="Only import activities after this date (default: incremental from last sync).")
@click.option("--reauth", is_flag=True, default=False,
help="Force re-authorization even if valid tokens exist.")
def strava_cmd(
client_id: Optional[str],
client_secret: Optional[str],
output_dir: Optional[str],
config_path: Optional[str],
since: Optional[str],
reauth: bool,
) -> None:
"""Import activities from Strava.
Credentials are resolved in this order:
1. --client-id / --client-secret flags
2. STRAVA_CLIENT_ID / STRAVA_CLIENT_SECRET environment variables
3. import.strava.client_id / client_secret in extract_config.yaml
Tokens are saved to ~/.config/bincio/strava.json and refreshed automatically.
\b
How to get API credentials (takes ~2 minutes, no approval needed):
1. Go to strava.com/settings/api
2. Create an application (name/website can be anything;
Authorization Callback Domain: localhost)
3. Copy the Client ID and Client Secret into extract_config.yaml:
\b
import:
strava:
client_id: 12345
client_secret: your_secret_here
\b
Examples:
bincio import strava # uses extract_config.yaml
bincio import strava --since 2025-01-01 # only activities from 2025
bincio import strava --reauth # force fresh OAuth flow
"""
try:
import requests # noqa: F401
except ImportError:
raise click.ClickException(
"requests is required for the Strava importer.\n"
"Install with: uv sync --extra strava"
)
from bincio.import_.strava import StravaClient, TOKENS_FILE, sync as strava_sync
# Load config to get credentials + output dir if not given on CLI
cfg = _load_config(config_path)
# Resolve credentials: CLI flag > env var (already consumed by click) > config file
if not client_id and cfg and cfg.strava:
client_id = cfg.strava.client_id or None
if not client_secret and cfg and cfg.strava:
client_secret = cfg.strava.client_secret or None
if not client_id or not client_secret:
raise click.UsageError(
"Strava client ID and secret are required.\n"
"Add them to extract_config.yaml under import.strava, or pass --client-id/--client-secret."
)
out = _resolve_output(output_dir, cfg)
console.print(f"Output dir: [cyan]{out}[/cyan]")
if reauth and TOKENS_FILE.exists():
TOKENS_FILE.unlink()
console.print("Removed saved tokens — starting fresh OAuth flow.")
client = StravaClient(client_id, client_secret, console)
client.authenticate()
since_dt = None
if since:
from datetime import datetime, timezone
try:
since_dt = datetime.strptime(since, "%Y-%m-%d").replace(tzinfo=timezone.utc)
except ValueError:
raise click.BadParameter(f"Expected YYYY-MM-DD, got {since!r}", param_hint="--since")
strava_sync(client, out, since_dt, console)
def _load_config(config_path: Optional[str]):
"""Load extract_config.yaml if available; return None if not found."""
from bincio.extract.config import load_config
candidates = []
if config_path:
candidates.append(Path(config_path))
candidates.append(Path("extract_config.yaml"))
for p in candidates:
if p.exists():
return load_config(p)
return None
def _resolve_output(explicit: Optional[str], cfg) -> Path:
if explicit:
return Path(explicit).expanduser().resolve()
if cfg and cfg.output_dir:
return cfg.output_dir
default = Path.home() / "bincio_data"
console.print(f"[yellow]No output dir specified; using [cyan]{default}[/cyan]")
return default
+388
View File
@@ -0,0 +1,388 @@
"""Strava API importer for BincioActivity.
Converts Strava activities + streams into BAS JSON using the same extract
pipeline (ParsedActivity → compute() → write_activity()) as local files.
OAuth tokens are stored in ~/.config/bincio/strava.json and refreshed
automatically. No server needed — the OAuth dance uses a one-shot local
callback server (same pattern as `gh auth login`).
"""
from __future__ import annotations
import dataclasses
import hashlib
import json
import secrets
import time
import webbrowser
from datetime import datetime, timedelta, timezone
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
from typing import Any
from urllib.parse import parse_qs, urlencode, urlparse
from rich.console import Console
from rich.progress import BarColumn, MofNCompleteColumn, Progress, TextColumn, TimeElapsedColumn
from bincio.extract.models import DataPoint, ParsedActivity
from bincio.extract.sport import normalise_sport
STRAVA_AUTH_URL = "https://www.strava.com/oauth/authorize"
STRAVA_TOKEN_URL = "https://www.strava.com/oauth/token"
STRAVA_API_BASE = "https://www.strava.com/api/v3"
TOKENS_FILE = Path.home() / ".config" / "bincio" / "strava.json"
SYNC_FILE = "_strava_sync.json" # lives in output_dir
CALLBACK_PORT = 8976
STREAM_KEYS = "time,latlng,altitude,heartrate,cadence,watts,velocity_smooth"
# ── API client ────────────────────────────────────────────────────────────────
class StravaClient:
def __init__(self, client_id: str, client_secret: str, console: Console) -> None:
self.client_id = client_id
self.client_secret = client_secret
self._console = console
self._tokens: dict = {}
self._15min_used = 0
self._daily_used = 0
# ── auth ──────────────────────────────────────────────────────────────────
def authenticate(self) -> None:
"""Load saved tokens (refreshing if needed) or run the OAuth dance."""
if TOKENS_FILE.exists():
saved = json.loads(TOKENS_FILE.read_text(encoding="utf-8"))
if saved.get("client_id") == self.client_id:
self._tokens = saved
self._ensure_fresh()
self._console.print("[green]✓[/green] Authenticated via saved tokens.")
return
self._oauth_dance()
def _ensure_fresh(self) -> None:
if time.time() > self._tokens.get("expires_at", 0) - 60:
self._refresh()
def _refresh(self) -> None:
import requests
r = requests.post(STRAVA_TOKEN_URL, data={
"client_id": self.client_id,
"client_secret": self.client_secret,
"grant_type": "refresh_token",
"refresh_token": self._tokens["refresh_token"],
}, timeout=30)
r.raise_for_status()
self._tokens.update(r.json())
self._tokens["client_id"] = self.client_id
self._save_tokens()
def _oauth_dance(self) -> None:
"""Open browser for OAuth2 authorization, receive callback."""
import requests
state = secrets.token_urlsafe(16)
code_holder: dict[str, str] = {}
class _Handler(BaseHTTPRequestHandler):
def do_GET(self) -> None:
qs = parse_qs(urlparse(self.path).query)
if qs.get("state", [None])[0] == state:
code_holder["code"] = qs.get("code", [None])[0] or ""
self.send_response(200)
self.end_headers()
self.wfile.write(
b"<html><body style='font-family:sans-serif;padding:2rem'>"
b"<h2>Authorized! You can close this tab.</h2></body></html>"
)
def log_message(self, *_: Any) -> None:
pass
server = HTTPServer(("127.0.0.1", CALLBACK_PORT), _Handler)
params = urlencode({
"client_id": self.client_id,
"redirect_uri": f"http://localhost:{CALLBACK_PORT}/callback",
"response_type": "code",
"scope": "activity:read_all",
"state": state,
})
url = f"{STRAVA_AUTH_URL}?{params}"
self._console.print(f"Opening browser for Strava authorization…")
self._console.print(f"If nothing opens, visit: [cyan]{url}[/cyan]")
webbrowser.open(url)
server.handle_request() # blocks until one request received
server.server_close()
code = code_holder.get("code")
if not code:
raise RuntimeError("Authorization failed — no code received from Strava.")
r = requests.post(STRAVA_TOKEN_URL, data={
"client_id": self.client_id,
"client_secret": self.client_secret,
"code": code,
"grant_type": "authorization_code",
}, timeout=30)
r.raise_for_status()
self._tokens = r.json()
self._tokens["client_id"] = self.client_id
self._save_tokens()
self._console.print("[green]✓[/green] Authorized!")
def _save_tokens(self) -> None:
TOKENS_FILE.parent.mkdir(parents=True, exist_ok=True)
TOKENS_FILE.write_text(json.dumps(self._tokens, indent=2), encoding="utf-8")
# ── HTTP ──────────────────────────────────────────────────────────────────
def _get(self, path: str, **params: Any) -> Any:
import requests as req
self._ensure_fresh()
headers = {"Authorization": f"Bearer {self._tokens['access_token']}"}
while True:
r = req.get(f"{STRAVA_API_BASE}{path}", params=params, headers=headers, timeout=30)
# Track rate limits
usage = r.headers.get("X-RateLimit-Usage", "")
if usage:
parts = usage.split(",")
if len(parts) == 2:
self._15min_used = int(parts[0])
self._daily_used = int(parts[1])
if r.status_code == 429:
self._console.print("[yellow]Rate limit reached, waiting 60 s…[/yellow]")
time.sleep(60)
continue
r.raise_for_status()
limit_hdr = r.headers.get("X-RateLimit-Limit", "")
if limit_hdr:
lparts = limit_hdr.split(",")
if len(lparts) == 2:
l15 = int(lparts[0])
if self._15min_used > int(l15 * 0.85):
self._console.print(
f"[yellow]Warning:[/yellow] {self._15min_used}/{l15} requests used this 15-min window."
)
return r.json()
# ── API calls ─────────────────────────────────────────────────────────────
def get_activities(self, after: int | None = None, per_page: int = 200) -> list[dict]:
"""Fetch full paginated activity list. `after` is a Unix timestamp."""
activities: list[dict] = []
page = 1
while True:
params: dict[str, Any] = {"per_page": per_page, "page": page}
if after is not None:
params["after"] = after
batch: list[dict] = self._get("/athlete/activities", **params)
if not batch:
break
activities.extend(batch)
if len(batch) < per_page:
break
page += 1
return activities
def get_streams(self, activity_id: int) -> dict[str, list]:
"""Return {stream_type: [values...]}. Empty dict on any failure."""
try:
data: dict = self._get(
f"/activities/{activity_id}/streams",
keys=STREAM_KEYS,
key_by_type="true",
)
return {k: v["data"] for k, v in data.items() if isinstance(v, dict) and "data" in v}
except Exception:
return {}
# ── conversion ────────────────────────────────────────────────────────────────
def _strava_to_parsed(act: dict, streams: dict[str, list]) -> ParsedActivity:
"""Build a ParsedActivity from a Strava activity dict + its streams."""
started_at = datetime.fromisoformat(act["start_date"].replace("Z", "+00:00"))
sport = normalise_sport(act.get("sport_type") or act.get("type") or "")
times = streams.get("time", []) # seconds since start
latlngs = streams.get("latlng", []) # [[lat, lon], ...]
altitudes = streams.get("altitude", []) # metres
heartrates = streams.get("heartrate", []) # bpm
cadences = streams.get("cadence", []) # rpm
watts = streams.get("watts", []) # W
velocities = streams.get("velocity_smooth", []) # m/s
points: list[DataPoint] = []
for i, t in enumerate(times):
ll = latlngs[i] if i < len(latlngs) else None
points.append(DataPoint(
timestamp = started_at + timedelta(seconds=int(t)),
lat = float(ll[0]) if ll else None,
lon = float(ll[1]) if ll else None,
elevation_m = float(altitudes[i]) if i < len(altitudes) else None,
hr_bpm = int(heartrates[i]) if i < len(heartrates) else None,
cadence_rpm = int(cadences[i]) if i < len(cadences) else None,
power_w = int(watts[i]) if i < len(watts) else None,
speed_kmh = float(velocities[i]) * 3.6 if i < len(velocities) else None,
))
strava_id = str(act["id"])
source_hash = "sha256:" + hashlib.sha256(f"strava:{strava_id}".encode()).hexdigest()
return ParsedActivity(
points = points,
sport = sport,
started_at = started_at,
source_file = f"strava_{strava_id}",
source_hash = source_hash,
title = act.get("name") or None,
strava_id = strava_id,
)
def _patch_from_summary(metrics: Any, act: dict) -> Any:
"""Fill None metric fields using Strava activity summary values.
Useful for activities without streams (manual entries, indoor rides with
no sensors) where compute() returns _empty().
"""
patches: dict[str, Any] = {}
if metrics.distance_m is None and act.get("distance"):
patches["distance_m"] = float(act["distance"])
if metrics.moving_time_s is None and act.get("moving_time"):
patches["moving_time_s"] = int(act["moving_time"])
if metrics.duration_s is None and act.get("elapsed_time"):
patches["duration_s"] = int(act["elapsed_time"])
if metrics.elevation_gain_m is None and act.get("total_elevation_gain"):
patches["elevation_gain_m"] = float(act["total_elevation_gain"])
if metrics.avg_hr_bpm is None and act.get("average_heartrate"):
patches["avg_hr_bpm"] = int(act["average_heartrate"])
if metrics.max_hr_bpm is None and act.get("max_heartrate"):
patches["max_hr_bpm"] = int(act["max_heartrate"])
if metrics.avg_power_w is None and act.get("average_watts"):
patches["avg_power_w"] = int(act["average_watts"])
if metrics.avg_cadence_rpm is None and act.get("average_cadence"):
patches["avg_cadence_rpm"] = int(act["average_cadence"])
if metrics.avg_speed_kmh is None and act.get("average_speed"):
patches["avg_speed_kmh"] = float(act["average_speed"]) * 3.6
return dataclasses.replace(metrics, **patches) if patches else metrics
# ── main sync ─────────────────────────────────────────────────────────────────
def sync(
client: StravaClient,
output_dir: Path,
since: datetime | None,
console: Console,
) -> None:
"""Fetch new Strava activities and write BAS JSON files.
Idempotent: already-imported Strava IDs (tracked in _strava_sync.json)
are skipped. `since` overrides the auto-detected checkpoint.
"""
from bincio.extract.metrics import compute
from bincio.extract.writer import build_summary, make_activity_id, write_activity, write_index
output_dir.mkdir(parents=True, exist_ok=True)
# ── load sync state ───────────────────────────────────────────────────────
sync_path = output_dir / SYNC_FILE
sync_state: dict = json.loads(sync_path.read_text(encoding="utf-8")) if sync_path.exists() else {}
imported_ids: set[str] = set(sync_state.get("imported_ids", []))
# ── determine `after` timestamp ───────────────────────────────────────────
after_ts: int | None = None
if since:
after_ts = int(since.timestamp())
elif sync_state.get("last_sync"):
# 1-hour overlap to catch delayed Strava saves
last = datetime.fromisoformat(sync_state["last_sync"])
after_ts = int((last - timedelta(hours=1)).timestamp())
# else: full sync (first run)
# ── fetch activity list ───────────────────────────────────────────────────
since_label = f" since {since.date()}" if since else (" (incremental)" if after_ts else " (full sync)")
console.print(f"Fetching Strava activity list{since_label}")
all_acts = client.get_activities(after=after_ts)
new_acts = [a for a in all_acts if str(a["id"]) not in imported_ids]
console.print(
f"Found [bold]{len(new_acts)}[/bold] new activities "
f"([bold]{len(all_acts) - len(new_acts)}[/bold] already imported)."
)
if not new_acts:
console.print("[green]All up to date.[/green]")
return
# ── load existing index ───────────────────────────────────────────────────
index_path = output_dir / "index.json"
if index_path.exists():
index_data = json.loads(index_path.read_text(encoding="utf-8"))
else:
index_data = {"owner": {"handle": "strava_user"}, "activities": []}
owner = index_data.get("owner", {})
summaries: dict[str, dict] = {s["id"]: s for s in index_data.get("activities", [])}
# ── import loop ───────────────────────────────────────────────────────────
errors: list[tuple[str, str]] = []
imported = 0
with Progress(
TextColumn("[progress.description]{task.description}"),
BarColumn(),
MofNCompleteColumn(),
TimeElapsedColumn(),
console=console,
) as progress:
task = progress.add_task("Importing…", total=len(new_acts))
for act in new_acts:
progress.advance(task)
strava_id = str(act["id"])
try:
streams = client.get_streams(act["id"])
parsed = _strava_to_parsed(act, streams)
metrics = compute(parsed)
metrics = _patch_from_summary(metrics, act)
act_id = make_activity_id(parsed)
write_activity(parsed, metrics, output_dir, privacy="public")
summaries[act_id] = build_summary(parsed, metrics, act_id, "public")
imported_ids.add(strava_id)
imported += 1
except Exception as exc:
errors.append((strava_id, str(exc)))
# ── write index + sync state ──────────────────────────────────────────────
write_index(list(summaries.values()), output_dir, owner)
sync_state["imported_ids"] = sorted(imported_ids)
sync_state["last_sync"] = datetime.now(timezone.utc).isoformat()
sync_path.write_text(json.dumps(sync_state, indent=2), encoding="utf-8")
# Trigger merge if sidecar edits directory exists
if (output_dir / "edits").exists():
from bincio.render.merge import merge_all
merge_all(output_dir)
console.print(
f"\n[green]Done.[/green] "
f"Imported [bold]{imported}[/bold] activities, "
f"errors [bold]{len(errors)}[/bold]."
)
if errors:
console.print("\n[red]Errors:[/red]")
for sid, msg in errors[:20]:
console.print(f" Strava {sid}: {msg}")
if len(errors) > 20:
console.print(f" … and {len(errors) - 20} more.")
+10
View File
@@ -31,6 +31,16 @@ classifier:
incremental: true # skip files whose hash hasn't changed since last run incremental: true # skip files whose hash hasn't changed since last run
# ── Platform importers ─────────────────────────────────────────────────────────
# Credentials for `bincio import strava`.
# Get them from strava.com/settings/api (2 minutes, no approval needed).
# Authorization Callback Domain must be set to: localhost
# import:
# strava:
# client_id: 12345
# client_secret: your_client_secret_here
# ── Athlete zones ───────────────────────────────────────────────────────────────
# athlete: # athlete:
# max_hr: 182 # used to derive default HR zone display # max_hr: 182 # used to derive default HR zone display
# ftp_w: 280 # functional threshold power in watts # ftp_w: 280 # functional threshold power in watts
+5
View File
@@ -31,6 +31,11 @@ classifier:
incremental: true # skip files whose hash hasn't changed since last run incremental: true # skip files whose hash hasn't changed since last run
import:
strava:
client_id: # paste your Client ID from strava.com/settings/api
client_secret: # paste your Client Secret
athlete: athlete:
max_hr: 190 max_hr: 190
ftp_w: 210 ftp_w: 210
+3
View File
@@ -37,6 +37,9 @@ edit = [
"uvicorn[standard]>=0.29", "uvicorn[standard]>=0.29",
"python-multipart>=0.0.9", "python-multipart>=0.0.9",
] ]
strava = [
"requests>=2.32",
]
dev = [ dev = [
"pytest>=9.0", "pytest>=9.0",
"pytest-cov>=5.0", "pytest-cov>=5.0",
+1 -1
View File
@@ -4,6 +4,6 @@
BINCIO_DATA_DIR=~/bincio_data BINCIO_DATA_DIR=~/bincio_data
# Optional: URL of a running `bincio edit` server. # Optional: URL of a running `bincio edit` server.
# When set, an Edit button appears on activity detail pages. # When set, an Edit button (and Upload ↑ button) appear in the site.
# Leave unset (or remove) for production / public deployments. # Leave unset (or remove) for production / public deployments.
# PUBLIC_EDIT_URL=http://localhost:4041 # PUBLIC_EDIT_URL=http://localhost:4041