bincio-activity/ARCHITECTURE.md

# BincioActivity — Architecture

## Overview

BincioActivity is a two-stage pipeline that turns raw activity files (GPX, FIT, TCX) into a self-hosted static website. There is no database and no application server — everything is files.

```
Raw files  ──►  BAS data store  ──►  Static site
           extract             render
           (Python)            (Astro/Node)
```

---

## Stage 1 — Extract

`bincio extract` reads your activity files and writes a **BAS (BincioActivity Schema)** data store: a directory of JSON files.

```mermaid
flowchart LR
    subgraph Input
        A[GPX files]
        B[FIT files]
        C[TCX files]
        D[activities.csv\nStrava metadata]
    end

    subgraph Extract ["bincio extract (Python)"]
        E[Parse]
        F[Compute metrics]
        G[Deduplicate]
        H[Write BAS JSON]
    end

    subgraph Output ["BAS data store  ~/bincio_data/"]
        I[activities/\n*.json *.geojson]
        J[index.json\nall summaries]
        K[athlete.json\nzones + records]
    end

    A & B & C --> E
    D --> E
    E --> F --> G --> H
    H --> I & J & K
```

The data store is **immutable extract output** — never edited directly.

---

## Stage 2 — Render

`bincio render` merges any user edits, then runs an Astro build to produce a static site.

```mermaid
flowchart LR
    subgraph DataStore ["BAS data store"]
        A[activities/*.json]
        B[edits/*.md\nsidecar files]
        C[edits/images/]
    end

    subgraph Render ["bincio render (Astro/Node)"]
        D[merge_all\napply sidecars]
        E[Astro build\nSSG]
    end

    subgraph Site ["Static site  dist/"]
        F[HTML pages]
        G[JS bundles]
        H[data/\nBAS JSON]
    end

    A & B & C --> D --> E --> F & G & H
```

The rendered site is **fully static**: no server needed to serve it. GitHub Pages, Netlify, nginx — all work.

---

## The edit flow

When `bincio edit` is running locally, an **Edit** button appears in the site. It opens a drawer that writes sidecar files without touching the immutable extract output.

```mermaid
sequenceDiagram
    actor User
    participant Site as Static site\n(Astro dev server)
    participant EditServer as bincio edit server\n(FastAPI, port 4041)
    participant DataStore as BAS data store

    User->>Site: clicks Edit on activity
    Site->>EditServer: GET /api/activity/{id}
    EditServer->>DataStore: reads .json + sidecar .md
    EditServer-->>Site: current values

    User->>Site: edits title / description / sport
    Site->>EditServer: POST /api/activity/{id}
    EditServer->>DataStore: writes edits/{id}.md
    EditServer->>DataStore: runs merge_all()
    EditServer-->>Site: ok

    Note over Site: feed reloads with merged data
```

The edit server is **never public-facing** — it only binds to `127.0.0.1` and is only enabled when `PUBLIC_EDIT_URL` is set in `site/.env`.

---

## Data sources

There are three ways activities enter the data store:

```mermaid
flowchart TD
    subgraph Sources
        A[📁 Local files\nGPX / FIT / TCX]
        B[🟠 Strava API\nOAuth sync]
        C[📱 Convert page\nPyodide in-browser]
    end

    subgraph EditServer ["bincio edit server\n(when running)"]
        D[POST /api/upload]
        E[POST /api/strava/sync]
    end

    subgraph CLI ["bincio extract CLI"]
        F[batch extract]
    end

    subgraph DataStore ["BAS data store"]
        G[activities/*.json]
        H[index.json]
    end

    A -->|bulk| F --> G & H
    A -->|single file via UI| D --> G & H
    B --> E --> G & H
    C -->|download JSON| A
    C -->|POST to edit server\nif configured| D
```

**Local files → CLI** is the primary path for bulk imports.
**Strava sync** and **file upload** go through the edit server for single activities.
**Convert page** runs the extract pipeline in-browser — output is either downloaded or sent to the edit server.

---

## The convert page

`/convert/` is a page in the static site that runs the full extract pipeline **inside the browser** using Pyodide (Python compiled to WebAssembly).

```mermaid
sequenceDiagram
    actor User as User\n(on phone or desktop)
    participant Page as /convert/ page\n(browser)
    participant Pyodide as Pyodide runtime\n(Python in WASM)
    participant EditServer as bincio edit server\n(optional)

    User->>Page: opens /convert/
    Page->>Pyodide: load Pyodide + packages\n(lxml, fitdecode, bincio wheel)
    Note over Page,Pyodide: ~10MB, cached after first visit

    User->>Page: selects GPX / FIT / TCX file
    Page->>Pyodide: write file to virtual FS\nrun parse → metrics → write
    Pyodide-->>Page: BAS JSON + GeoJSON strings

    alt Download
        Page->>User: download activity.json\n(+ activity.geojson if GPS)
    else Save to bincio
        Page->>EditServer: POST /api/upload
        EditServer-->>Page: {id}
        Page->>User: redirect to activity page
    end
```

**"Save to bincio"** only appears when the page is served from an instance with `PUBLIC_EDIT_URL` set — the same flag that enables the edit drawer. For anyone else, downloading the JSON is the output.

### Why the convert page belongs to the instance

The convert page is part of the static site build — it uses the same styling, the same Pyodide wheel, and (if enabled) the same edit server. It is a tool **for the people who use this instance**:

- The instance owner uses it to convert files on mobile without needing a computer
- (Future) other users of a multi-user instance use it to upload their own activities

It is not a standalone public tool (though technically anyone with the URL could use the download path, since Pyodide runs locally in their browser).

---

## Federation

Federation is how users follow each other across different bincio instances without any central server.

```mermaid
flowchart TB
    subgraph InstanceA ["Instance A — bincio.alice.com"]
        A1[BAS data store]
        A2[Static site]
        A1 --> A2
    end

    subgraph InstanceB ["Instance B — bincio.bob.com"]
        B1[BAS data store]
        B2[Static site]
        B1 --> B2
    end

    subgraph Browser ["Bob's browser"]
        C[Feed]
    end

    A2 -->|BAS JSON URL| C
    B2 -->|BAS JSON URL| C

    %% Bob adds Alice's index.json URL to his followed feeds.
    %% No accounts. No central server. Just a URL.
```

Federation is a planned feature — the data format (BAS JSON) is designed for it, but the site UI doesn't yet support adding followed feeds.

---

## Mobile app (Capacitor)

The existing Astro/Svelte site is wrapped in a **Capacitor** native shell to produce iOS and Android apps. No code is rewritten — Capacitor provides the native APIs that a plain browser PWA cannot.

### What Capacitor adds over a browser PWA

| Capability | Browser PWA | Capacitor app |
|---|---|---|
| Background GPS (iOS) | ✗ killed by OS | ✅ native entitlement |
| Background GPS (Android) | ⚠️ limited | ✅ foreground service |
| Filesystem access | ✗ sandboxed | ✅ full device storage |
| Install without App Store | ✅ (Android sideload) | iOS: App Store or TestFlight |

### Architecture

```mermaid
flowchart TD
    subgraph App ["Capacitor app (iOS / Android)"]
        subgraph WebView ["WebView — existing Astro/Svelte site"]
            A[/record/ — GPS recorder]
            B[/convert/ — Pyodide converter]
            C[/feed/, /activity/, etc.]
        end
        subgraph Native ["Native layer"]
            D[Geolocation plugin\nbackground GPS]
            E[Filesystem plugin\nread/write device storage]
        end
    end

    subgraph Cloud ["Cloud instance (optional)"]
        F[Static site]
        G[bincio edit server]
    end

    A -->|JS bridge| D
    B -->|JS bridge| E
    B -->|POST /api/import-bas| G
    C --> F
```

### Mobile workflows

**Workflow 1 — Record on phone, save to cloud**
```
Phone app: /record/  →  GPS recording (Capacitor Geolocation)
                     →  export GPX  →  /convert/
/convert/            →  Pyodide runs extract pipeline in WebView
                     →  POST /api/import-bas to cloud edit server
Cloud instance       →  saves activity, site updates
```

**Workflow 2 — Record on phone, save locally (offline)**
```
Phone app: /record/  →  GPS recording
                     →  export GPX  →  /convert/
/convert/            →  Pyodide converts in WebView
                     →  download BAS JSON to device storage
                        (upload to cloud later when online)
```

**Workflow 3 — Import from OsmAnd / Organic Maps**
```
OsmAnd / Organic Maps  →  exports GPX to device storage
Phone app: /convert/   →  user picks GPX file
                       →  Pyodide converts in WebView
                       →  save to cloud or download
```

**Workflow 4 — Fully offline on phone (future)**
```
Phone app: /record/  →  GPS recording (native Geolocation)
                     →  export GPX  →  /convert/
/convert/            →  Pyodide converts (loaded from local cache)
                     →  saveActivityLocally() writes to IndexedDB
Data loader          →  loadIndex() / loadActivity(id) read from IndexedDB
WebView              →  feed + activity pages populated from local data
                     →  edit drawer writes sidecars (merge logic in JS)
```
See "Fully offline — missing pieces" below for implementation status.

### Setup

```bash
cd site

# First time: initialise native projects
npx cap add android   # creates site/android/
npx cap add ios       # creates site/ios/ — requires macOS + Xcode

# Daily workflow
npm run cap:sync      # build Astro + sync to native projects
npm run cap:android   # build + open Android Studio
npm run cap:ios       # build + open Xcode
```

### The bincio Python wheel

The `/convert/` page loads the extract pipeline via **Pyodide** (Python compiled to WebAssembly). The pipeline is bundled as `site/public/bincio.whl` — a standard Python wheel that Pyodide loads via `micropip`.

To rebuild the wheel after changing the extract code:
```bash
# from the repo root
uv build --wheel
cp dist/bincio-*.whl site/public/bincio.whl
```

Pyodide loads these packages on first visit (cached by the browser after that):
- `lxml`, `pyyaml` — prebuilt Pyodide packages (~2 MB)
- `fitdecode`, `gpxpy`, `rdp` — pure Python, installed via micropip
- `bincio.whl` — our wheel, ~150 KB

Total cold-start download: ~10 MB. Subsequent visits: instant (all cached).

### Fully offline — missing pieces

#### 1. Data access abstraction *(do this first)*

The prior design asked "how do we serve `/data/*` to the WebView from local storage?" — but that question skips a cleaner solution: eliminate the `fetch('/data/*')` calls from the hot path entirely.

The Svelte components (`ActivityFeed`, `ActivityDetail`, etc.) currently call `fetch(...)` directly. Replacing those with a thin loader module (`loadIndex()`, `loadActivity(id)`) with two implementations decouples the data source from the UI:

```
src/lib/dataloader.ts
  loadIndex()      → fetch('/data/index.json')       [cloud build]
                   → idbGet('/data/local-index')     [app build]
  loadActivity(id) → fetch('/data/activities/...')   [cloud build]
                   → idbGet('/data/activities/...')  [app build]
```

The build variant is selected by an env var (`PUBLIC_DATA_MODE=cloud|local`) at Astro build time. No service worker required. No iOS WKWebView question. Useful regardless of offline support (testability, mocking, future federation transports).

**This is the first thing to build.** The service worker work already done (`sw.js`, `localstore.ts`) provides a working fallback and is useful for the browser, but the abstraction layer is cleaner and should land before investing more in the SW path.

Status: not yet implemented.

#### 2. Pyodide available offline

Currently loads from CDN (~10 MB) on first use. A service worker can cache the CDN assets on first online visit so all subsequent visits work offline. Standard PWA pattern, no app size increase, no code changes to the convert page.

Status: not yet implemented.

#### 3. Sidecar merge in JS

The edit drawer currently calls the edit server's `POST /api/activity/{id}` which triggers the Python `merge_all()`. For offline editing this needs to run locally. Port to JS (~150 lines) — **do not use Pyodide for this**.

Pyodide is lazy-loaded on `/convert/`. It is not warm just because the app is open. A user tapping Edit → change title → Save would trigger a full ~10 MB Pyodide cold start for a trivial one-line edit. The edit drawer must remain decoupled from the convert page's machinery.

Status: not yet implemented.

#### 4. If service workers fail on iOS

If testing in a Capacitor iOS WebView reveals that service workers are blocked, fall back to one of:

1. **Data access abstraction only** (§1 above) — if all `fetch('/data/*')` calls are gone, the SW question is moot.
2. **Native Swift/Kotlin micro-server** — a small HTTP server embedded in the native layer, serving from device storage. No third-party plugins.
3. **Re-run `cap sync` on import** — bundle updated static data files and resync. Crude but reliable for low-frequency imports.

`capacitor-nodejs` is not a fallback option. It embeds a full Node runtime, has spotty iOS support, and carries significant long-term maintenance cost.

#### 5. Open questions — data lifecycle on device

These don't need solutions today but will surface during implementation:

- **Storage limits:** iOS may evict IndexedDB under storage pressure. `@capacitor/filesystem` stores in the app's Documents directory (not evictable) — may be preferable for activity data.
- **Uninstall / reinstall:** IndexedDB is wiped on uninstall. Documents directory survives reinstall on iOS. Strategy TBD.
- **Sync conflicts:** if the same activity exists locally and on a cloud instance (e.g. uploaded via Strava, also recorded in-app), the merge strategy is undefined. Likely: server wins on pull, local wins on push, user resolves conflicts manually.

#### 6. Implementation order

| Step | Effort | Status |
|---|---|---|
| Data access abstraction (`dataloader.ts`) | Medium | Not started — **do first** |
| Sidecar merge in JS | Medium | Not started |
| Pyodide service worker cache | Medium | Not started |
| Test SW in Capacitor Android WebView | Low | Not started |
| Test SW on iOS | Low | Not started |
| Native micro-server (iOS fallback, if needed) | Hard | Contingency only |

---

### The `/api/import-bas` endpoint

The `/convert/` page sends pre-converted BAS JSON directly to the edit server, avoiding the need to re-parse a file on the server side. The endpoint:
1. Validates the `id` field
2. Writes `activities/{id}.json` (and `.geojson` if provided)
3. Rebuilds `index.json`
4. Runs `merge_all()`

---

## Deployment

```mermaid
flowchart LR
    subgraph Local ["Local machine"]
        A[bincio extract]
        B[bincio edit\noptional]
    end

    subgraph VPS ["VPS or static host"]
        C[Static site\ndist/]
        D[nginx]
    end

    subgraph Phone ["Phone"]
        E[OsmAnd /\nOrganic Maps]
        F[Browser:\n/convert/]
    end

    A -->|rsync / CI| C
    C --> D
    E -->|GPX export| F
    F -->|upload| B
    B --> A
```

The edit server is **always local** in the single-user setup. In a future multi-user deployment on a VPS, it would run as a service alongside nginx.