improve configs, update docs
This commit is contained in:
Davide Scaini
@@ -51,24 +51,32 @@ bincio/ Python package
|
||||
writer.py BAS JSON + GeoJSON writer
|
||||
config.py extract_config.yaml loader
|
||||
cli.py `bincio extract` CLI
|
||||
import_/
|
||||
strava.py Strava API importer (OAuth2, streams → BAS JSON)
|
||||
cli.py `bincio import strava` CLI
|
||||
render/
|
||||
cli.py `bincio render` CLI (symlinks data, runs astro build/dev)
|
||||
edit/
|
||||
server.py FastAPI write API (activity edits, image upload, file upload)
|
||||
cli.py `bincio edit` CLI
|
||||
schema/
|
||||
bas-v1.schema.json JSON Schema for BAS
|
||||
SCHEMA.md Human-readable BAS spec
|
||||
site/ Astro project
|
||||
src/
|
||||
layouts/Base.astro
|
||||
layouts/Base.astro Nav (upload button + theme toggle), theme CSS vars
|
||||
pages/
|
||||
index.astro Activity feed (loads index.json client-side)
|
||||
activity/[id].astro Single activity (SSG, loads detail JSON client-side)
|
||||
stats/index.astro Heatmap + year totals
|
||||
athlete/index.astro MMP curve + athlete profile (planned)
|
||||
components/
|
||||
ActivityFeed.svelte Card grid, sport filter, pagination
|
||||
ActivityDetail.svelte Map + stats + charts wrapper
|
||||
ActivityMap.svelte MapLibre GL (gradient track, linked hover dot)
|
||||
ActivityCharts.svelte Observable Plot (elevation/speed/HR/cadence/power tabs)
|
||||
StatsView.svelte Yearly heatmap + totals
|
||||
StatsView.svelte Yearly heatmap + click-to-pin tooltip
|
||||
EditDrawer.svelte Slide-in activity editor
|
||||
lib/
|
||||
types.ts BAS TypeScript types
|
||||
format.ts formatDistance, formatDuration, sportIcon, etc.
|
||||
@@ -77,14 +85,23 @@ site/ Astro project
|
||||
## How to run
|
||||
|
||||
```bash
|
||||
# Extract
|
||||
# Extract from local files
|
||||
cd ~/src/bincio_activity
|
||||
uv run bincio extract --input ~/src/cycling_data_davide/activities --output /tmp/bincio_test
|
||||
|
||||
# Import from Strava (credentials in extract_config.yaml under import.strava)
|
||||
uv sync --extra strava
|
||||
uv run bincio import strava # first run opens browser for OAuth
|
||||
uv run bincio import strava # subsequent runs are incremental
|
||||
|
||||
# Site dev server
|
||||
cd site
|
||||
ln -sf /tmp/bincio_test public/data # symlink data
|
||||
BINCIO_DATA_DIR=/tmp/bincio_test npm run dev
|
||||
npm run dev
|
||||
|
||||
# Edit server (enables drawer + file upload in the site)
|
||||
uv run bincio edit --data-dir ~/bincio_data # port 4041
|
||||
# Set PUBLIC_EDIT_URL=http://localhost:4041 in site/.env
|
||||
|
||||
# Tests
|
||||
uv run pytest
|
||||
@@ -489,116 +506,125 @@ to power-having activities, pull their `mmp` arrays, take element-wise max per s
|
||||
7. `AthleteDrawer.svelte` — zones + gear editing form
|
||||
8. Season config in `extract_config.yaml` / `edits/athlete.yaml`
|
||||
|
||||
## Data ingestion — design plan
|
||||
## Data ingestion
|
||||
|
||||
How activity data gets into BincioActivity. Three orthogonal vectors.
|
||||
|
||||
### Vector 1 — Web file upload (extends existing edit server)
|
||||
### Vector 1 — Web file upload ✓
|
||||
|
||||
The lowest-friction path: drag a FIT/GPX/TCX file onto the site, it appears immediately.
|
||||
Drag a FIT/GPX/TCX onto the site while the edit server is running — activity
|
||||
appears immediately.
|
||||
|
||||
**Backend** (`bincio/edit/server.py`):
|
||||
```
|
||||
POST /api/upload multipart FIT/GPX/TCX
|
||||
→ saves to staging dir
|
||||
→ bincio extract on just that file
|
||||
→ merge_all()
|
||||
→ returns { id, redirect: "/activity/{id}/" }
|
||||
POST /api/upload multipart FIT/GPX/TCX (also .gz variants)
|
||||
→ stages file to data_dir/_uploads/<name>
|
||||
→ parse_file() → compute() → write_activity() → build_summary()
|
||||
→ 409 if activity already exists (same timestamp = same ID)
|
||||
→ updates index.json + merge_all()
|
||||
→ returns { ok: true, id: "2024-05-15T103000Z" }
|
||||
→ cleans up staged file in finally block
|
||||
```
|
||||
|
||||
An "Upload activity" button in the nav (gated behind `PUBLIC_EDIT_URL` like the edit drawer).
|
||||
No CLI needed. Preserves static-site output — the server only exists in local editing mode.
|
||||
**Frontend** (`site/src/layouts/Base.astro`):
|
||||
- `↑` button in nav right cluster, only rendered when `PUBLIC_EDIT_URL` is set
|
||||
- Modal with drag-and-drop zone + click-to-browse
|
||||
- Auto-redirects to `/activity/{id}/` on success
|
||||
- Escape / backdrop click closes modal
|
||||
|
||||
### Vector 2 — Platform importers
|
||||
### Vector 2 — `bincio import strava` ✓
|
||||
|
||||
#### `bincio import strava` — pull your Strava history
|
||||
**`bincio/import_/strava.py`** + **`bincio/import_/cli.py`**
|
||||
|
||||
Install: `uv sync --extra strava`
|
||||
|
||||
```bash
|
||||
bincio import strava \
|
||||
--client-id 12345 \
|
||||
--client-secret abc... \
|
||||
--output ~/bincio_data \
|
||||
--since 2024-01-01 # optional, default: all-time
|
||||
# First run (full sync — opens browser):
|
||||
bincio import strava --client-id 12345 --client-secret abc --output ~/bincio_data
|
||||
|
||||
# Subsequent runs (incremental — picks up from last sync automatically):
|
||||
bincio import strava --client-id 12345 --client-secret abc
|
||||
|
||||
# Explicit date range:
|
||||
bincio import strava --client-id 12345 --client-secret abc --since 2025-01-01
|
||||
|
||||
# Force re-auth (rotate credentials or re-authorize):
|
||||
bincio import strava --client-id 12345 --client-secret abc --reauth
|
||||
|
||||
# Credentials via env vars (good for scripts):
|
||||
export STRAVA_CLIENT_ID=12345
|
||||
export STRAVA_CLIENT_SECRET=abc
|
||||
bincio import strava --output ~/bincio_data
|
||||
```
|
||||
|
||||
**How Strava API access works:**
|
||||
|
||||
Every Strava user can register an API app at `strava.com/settings/api` — no review,
|
||||
no approval, no fees. Fill in a name, website (`localhost` is fine), and callback
|
||||
domain (`localhost` for local use). You instantly get a Client ID and Client Secret.
|
||||
**Getting Strava API credentials (~2 minutes, no approval needed):**
|
||||
1. Go to `strava.com/settings/api`
|
||||
2. Create an application — name and website can be anything; set
|
||||
**Authorization Callback Domain** to `localhost`
|
||||
3. Paste Client ID and Client Secret into `extract_config.yaml`:
|
||||
```yaml
|
||||
import:
|
||||
strava:
|
||||
client_id: 12345
|
||||
client_secret: your_secret_here
|
||||
```
|
||||
`extract_config.yaml` is gitignored — safe to store credentials there.
|
||||
|
||||
Strava's "developer" label is misleading: formal review is only required for
|
||||
commercial apps used by *other* people. For a self-hosted personal tool, each user
|
||||
brings their own credentials and authenticates their own account. Rate limits are
|
||||
generous for personal use: **100 requests / 15 min, 1000 / day**.
|
||||
commercial apps that authenticate *other users*. For a personal self-hosted tool
|
||||
you authenticate your own account — no review, no fees.
|
||||
Rate limits: **100 req / 15 min, 1000 / day** (generous for personal use).
|
||||
|
||||
The importer:
|
||||
1. Opens a local OAuth2 callback server (like `gh auth login`)
|
||||
2. Pops a browser to `strava.com/oauth/authorize?scope=activity:read_all`
|
||||
3. User clicks Authorize → callback receives the code → exchanges for tokens
|
||||
4. Tokens saved to `~/.config/bincio/strava_tokens.json`
|
||||
5. Fetches paginated activity list → for each, fetches streams (lat/lng, time,
|
||||
altitude, HR, cadence, power, velocity) → converts to BAS JSON
|
||||
6. Idempotent: existing IDs (matched by Strava activity ID embedded in BAS metadata)
|
||||
are skipped. Safe to re-run for incremental sync.
|
||||
**How the importer works:**
|
||||
|
||||
Strava streams give the same data as FIT files at ~1 Hz (GPS, power meter, HR strap).
|
||||
*OAuth dance (first run):*
|
||||
- Starts a one-shot local HTTP server on port 8976
|
||||
- Opens `strava.com/oauth/authorize?scope=activity:read_all` in the browser
|
||||
- Receives the authorization code at `/callback`
|
||||
- Exchanges code for access + refresh tokens
|
||||
- Saves to `~/.config/bincio/strava.json` (keyed by client_id)
|
||||
- Subsequent runs load saved tokens and refresh silently when expired (6h TTL)
|
||||
|
||||
#### `bincio import garmin` — Garmin Connect
|
||||
*Sync loop:*
|
||||
- Reads `data_dir/_strava_sync.json` for set of already-imported Strava IDs
|
||||
and timestamp of last sync
|
||||
- Uses Strava `after=<unix_ts>` parameter for server-side filtering (efficient —
|
||||
no need to scan all pages on incremental runs; 1-hour overlap to catch late saves)
|
||||
- Per activity: `GET /activities/{id}/streams` → `_strava_to_parsed()` →
|
||||
`compute()` → `_patch_from_summary()` → `write_activity()` → `build_summary()`
|
||||
- Writes updated `index.json` + `_strava_sync.json`
|
||||
- Calls `merge_all()` if `edits/` directory exists
|
||||
|
||||
No official public API. Options:
|
||||
- **`garminconnect` Python library** — unofficial but widely used (same approach as
|
||||
tapiriik, garmin-connect-export). Works with email/password or session cookies.
|
||||
- **FIT file sync** — Garmin Express / Tapiriik sync FIT files to a local folder;
|
||||
`bincio extract` picks them up normally. Simplest.
|
||||
*Conversion details:*
|
||||
- `sport_type` (or `type`) → `normalise_sport()` — same mapping as FIT/GPX
|
||||
- Streams: `time` (s since start) + `latlng` + `altitude` + `heartrate` +
|
||||
`cadence` + `watts` + `velocity_smooth` (m/s → km/h) → `DataPoint` list
|
||||
- `source_hash`: `sha256("strava:{id}")` — stable, not file-content-based
|
||||
- `_patch_from_summary()`: fills `None` metric fields (distance, duration,
|
||||
elevation, HR, power) from the Strava activity summary for activities with
|
||||
missing/sparse sensors or manual entries
|
||||
- Rate limit: warns at 85% of 15-min window; auto-retries with 60s sleep on 429
|
||||
|
||||
#### Watch mode — for ongoing device sync
|
||||
### Vector 3 — Platform watch mode (planned)
|
||||
|
||||
```bash
|
||||
bincio extract --watch ~/Dropbox/Garmin/Activities --output ~/bincio_data
|
||||
```
|
||||
|
||||
Watches a directory for new FIT/GPX/TCX files (using `watchfiles` or `inotify`).
|
||||
New file dropped → auto-extract → `merge_all()` → site reflects it on next reload.
|
||||
Zero friction for users who already sync files from Garmin/Karoo/Wahoo to a folder
|
||||
via Dropbox, Syncthing, or Garmin Express.
|
||||
Directory watcher (`watchfiles` / `inotify`) for ongoing FIT sync from Karoo,
|
||||
Garmin, Wahoo. New file → auto-extract → merge. Not yet implemented.
|
||||
|
||||
#### Strava webhook — real-time push (advanced)
|
||||
|
||||
```bash
|
||||
bincio edit --data-dir ~/bincio_data --webhook-strava
|
||||
```
|
||||
|
||||
Registers a Strava webhook subscription. When you finish a ride, Strava pushes a
|
||||
notification → server fetches streams → extract → merge. Requires a **publicly
|
||||
accessible URL** (works with Tailscale, a VPS, or ngrok). Not needed for most
|
||||
self-hosters; polling via `bincio import strava --since yesterday` is simpler.
|
||||
|
||||
### Vector 3 — Federation (pull remote BAS feeds)
|
||||
|
||||
The cleanest "data in from the web" path for the self-hosted model:
|
||||
anyone who publishes a `index.json` at a public URL is a data source.
|
||||
### Vector 4 — Federation (planned)
|
||||
|
||||
```yaml
|
||||
# extract_config.yaml
|
||||
sources:
|
||||
- url: https://alice.example.com/data/index.json
|
||||
handle: alice
|
||||
- url: https://bob.example.com/data/index.json
|
||||
handle: bob
|
||||
```
|
||||
|
||||
`bincio render` fetches remote index files at build time, merges them into the
|
||||
site. No API keys, no OAuth. Local `.md` sidecars can annotate remote activities.
|
||||
Not yet implemented — see friends/federation items in the checklist below.
|
||||
|
||||
### Implementation priority
|
||||
|
||||
1. **Web file upload** — trivial to build, highest immediate UX value
|
||||
2. **`bincio import strava`** — covers historical migration and incremental sync;
|
||||
most cyclists already have years of data there
|
||||
3. **Watch mode** — covers ongoing FIT-file-based workflows (Karoo, Garmin)
|
||||
4. **Garmin Connect importer** — second most common platform
|
||||
5. **Federation** — longer term; enables the "personal Strava" social layer
|
||||
`bincio render` fetches remote BAS index files at build time. No API keys.
|
||||
Local `.md` sidecars can annotate remote activities.
|
||||
|
||||
---
|
||||
|
||||
@@ -626,7 +652,7 @@ Not yet implemented — see friends/federation items in the checklist below.
|
||||
- [ ] Map thumbnail in activity cards (SVG path from GeoJSON)
|
||||
- [ ] GitHub Actions template for auto-publish
|
||||
- [x] **Ingestion: web file upload** — `POST /api/upload` in edit server, drag-and-drop in nav
|
||||
- [ ] **Ingestion: `bincio import strava`** — OAuth2 + streams API, idempotent incremental sync
|
||||
- [x] **Ingestion: `bincio import strava`** — OAuth2 + streams API, idempotent incremental sync
|
||||
- [ ] **Ingestion: `bincio extract --watch`** — directory watcher for ongoing FIT sync
|
||||
- [ ] **Ingestion: `bincio import garmin`** — garminconnect library or FIT folder sync
|
||||
- [ ] **Ingestion: federation** — `sources:` in config, remote BAS index pull at render time
|
||||
|
||||
Reference in New Issue
Block a user