bincio-activity/ARCHITECTURE.md

# BincioActivity — Architecture

## Overview

BincioActivity is a two-stage pipeline that turns raw activity files (GPX, FIT, TCX) into a self-hosted static website. There is no database and no application server — everything is files.

```
Raw files  ──►  BAS data store  ──►  Static site
           extract             render
           (Python)            (Astro/Node)
```

---

## Stage 1 — Extract

`bincio extract` reads your activity files and writes a **BAS (BincioActivity Schema)** data store: a directory of JSON files.

```mermaid
flowchart LR
    subgraph Input
        A[GPX files]
        B[FIT files]
        C[TCX files]
        D[activities.csv\nStrava metadata]
    end

    subgraph Extract ["bincio extract (Python)"]
        E[Parse]
        F[Compute metrics]
        G[Deduplicate]
        H[Write BAS JSON]
    end

    subgraph Output ["BAS data store  ~/bincio_data/"]
        I[activities/\n*.json *.geojson]
        J[index.json\nall summaries]
        K[athlete.json\nzones + records]
    end

    A & B & C --> E
    D --> E
    E --> F --> G --> H
    H --> I & J & K
```

The data store is **immutable extract output** — never edited directly.

---

## Stage 2 — Render

`bincio render` merges any user edits, then runs an Astro build to produce a static site.

```mermaid
flowchart LR
    subgraph DataStore ["BAS data store"]
        A[activities/*.json]
        B[edits/*.md\nsidecar files]
        C[edits/images/]
    end

    subgraph Render ["bincio render (Astro/Node)"]
        D[merge_all\napply sidecars]
        E[Astro build\nSSG]
    end

    subgraph Site ["Static site  dist/"]
        F[HTML pages]
        G[JS bundles]
        H[data/\nBAS JSON]
    end

    A & B & C --> D --> E --> F & G & H
```

The rendered site is **fully static**: no server needed to serve it. GitHub Pages, Netlify, nginx — all work.

---

## The edit flow

When `bincio edit` is running locally, an **Edit** button appears in the site. It opens a drawer that writes sidecar files without touching the immutable extract output.

```mermaid
sequenceDiagram
    actor User
    participant Site as Static site\n(Astro dev server)
    participant EditServer as bincio edit server\n(FastAPI, port 4041)
    participant DataStore as BAS data store

    User->>Site: clicks Edit on activity
    Site->>EditServer: GET /api/activity/{id}
    EditServer->>DataStore: reads .json + sidecar .md
    EditServer-->>Site: current values

    User->>Site: edits title / description / sport
    Site->>EditServer: POST /api/activity/{id}
    EditServer->>DataStore: writes edits/{id}.md
    EditServer->>DataStore: runs merge_all()
    EditServer-->>Site: ok

    Note over Site: feed reloads with merged data
```

The edit server is **never public-facing** — it only binds to `127.0.0.1` and is only enabled when `PUBLIC_EDIT_URL` is set in `site/.env`.

---

## Data sources

There are three ways activities enter the data store:

```mermaid
flowchart TD
    subgraph Sources
        A[📁 Local files\nGPX / FIT / TCX]
        B[🟠 Strava API\nOAuth sync]
        C[📱 Convert page\nPyodide in-browser]
    end

    subgraph EditServer ["bincio edit server\n(when running)"]
        D[POST /api/upload]
        E[POST /api/strava/sync]
    end

    subgraph CLI ["bincio extract CLI"]
        F[batch extract]
    end

    subgraph DataStore ["BAS data store"]
        G[activities/*.json]
        H[index.json]
    end

    A -->|bulk| F --> G & H
    A -->|single file via UI| D --> G & H
    B --> E --> G & H
    C -->|download JSON| A
    C -->|POST to edit server\nif configured| D
```

**Local files → CLI** is the primary path for bulk imports.
**Strava sync** and **file upload** go through the edit server for single activities.
**Convert page** runs the extract pipeline in-browser — output is either downloaded or sent to the edit server.

---

## The convert page

`/convert/` is a page in the static site that runs the full extract pipeline **inside the browser** using Pyodide (Python compiled to WebAssembly).

```mermaid
sequenceDiagram
    actor User as User\n(on phone or desktop)
    participant Page as /convert/ page\n(browser)
    participant Pyodide as Pyodide runtime\n(Python in WASM)
    participant EditServer as bincio edit server\n(optional)

    User->>Page: opens /convert/
    Page->>Pyodide: load Pyodide + packages\n(lxml, fitdecode, bincio wheel)
    Note over Page,Pyodide: ~8MB, cached after first visit

    User->>Page: selects GPX / FIT / TCX file
    Page->>Pyodide: write file to virtual FS\nrun parse → metrics → write
    Pyodide-->>Page: BAS JSON + GeoJSON strings

    alt Download
        Page->>User: download activity.json\n(+ activity.geojson if GPS)
    else Save to bincio
        Page->>EditServer: POST /api/upload
        EditServer-->>Page: {id}
        Page->>User: redirect to activity page
    end
```

**"Save to bincio"** only appears when the page is served from an instance with `PUBLIC_EDIT_URL` set — the same flag that enables the edit drawer. For anyone else, downloading the JSON is the output.

### Why the convert page belongs to the instance

The convert page is part of the static site build — it uses the same styling, the same Pyodide wheel, and (if enabled) the same edit server. It is a tool **for the people who use this instance**:

- The instance owner uses it to convert files on mobile without needing a computer
- (Future) other users of a multi-user instance use it to upload their own activities

It is not a standalone public tool (though technically anyone with the URL could use the download path, since Pyodide runs locally in their browser).

---

## Federation

Federation is how users follow each other across different bincio instances without any central server.

```mermaid
flowchart TB
    subgraph InstanceA ["Instance A — bincio.alice.com"]
        A1[BAS data store]
        A2[Static site]
        A1 --> A2
    end

    subgraph InstanceB ["Instance B — bincio.bob.com"]
        B1[BAS data store]
        B2[Static site]
        B1 --> B2
    end

    subgraph Browser ["Bob's browser"]
        C[Feed]
    end

    A2 -->|BAS JSON URL| C
    B2 -->|BAS JSON URL| C

    Note1["Bob adds Alice's index.json URL\nto his followed feeds.\nNo accounts. No central server.\nJust a URL."]
```

Federation is a planned feature — the data format (BAS JSON) is designed for it, but the site UI doesn't yet support adding followed feeds.

---

## Mobile app (Capacitor)

The existing Astro/Svelte site is wrapped in a **Capacitor** native shell to produce iOS and Android apps. No code is rewritten — Capacitor provides the native APIs that a plain browser PWA cannot.

### What Capacitor adds over a browser PWA

| Capability | Browser PWA | Capacitor app |
|---|---|---|
| Background GPS (iOS) | ✗ killed by OS | ✅ native entitlement |
| Background GPS (Android) | ⚠️ limited | ✅ foreground service |
| Filesystem access | ✗ sandboxed | ✅ full device storage |
| Install without App Store | ✅ | iOS: App Store / TestFlight |
| Local HTTP server | ✗ | ✅ (future, via native plugin) |

### Architecture

```mermaid
flowchart TD
    subgraph App ["Capacitor app (iOS / Android)"]
        subgraph WebView ["WebView — existing Astro/Svelte site"]
            A[/record/ — GPS recorder]
            B[/convert/ — Pyodide converter]
            C[/feed/, /activity/, etc.]
        end
        subgraph Native ["Native layer"]
            D[Geolocation plugin\nbackground GPS]
            E[Filesystem plugin\nread/write device storage]
        end
    end

    subgraph Cloud ["Cloud instance (optional)"]
        F[Static site]
        G[bincio edit server]
    end

    A -->|JS bridge| D
    B -->|JS bridge| E
    B -->|POST /api/import-bas| G
    C --> F
```

### Mobile workflows

**Workflow 1 — Record on phone, save to cloud**
```
Phone app: /record/  →  GPS recording (Capacitor Geolocation)
                     →  export GPX  →  /convert/
/convert/            →  Pyodide runs extract pipeline in WebView
                     →  POST /api/import-bas to cloud edit server
Cloud instance       →  saves activity, site updates
```

**Workflow 2 — Record on phone, save locally (offline)**
```
Phone app: /record/  →  GPS recording
                     →  export GPX  →  /convert/
/convert/            →  Pyodide converts in WebView
                     →  download BAS JSON to device storage
                        (upload to cloud later when online)
```

**Workflow 3 — Import from OsmAnd / Organic Maps**
```
OsmAnd / Organic Maps  →  exports GPX to device storage
Phone app: /convert/   →  user picks GPX file
                       →  Pyodide converts in WebView
                       →  save to cloud or download
```

**Workflow 4 — Fully offline on phone (future)**
```
Phone app: /record/  →  GPS recording (native Geolocation)
                     →  export GPX  →  /convert/
/convert/            →  Pyodide converts (loaded from local cache)
                     →  writes BAS JSON to device storage (Filesystem plugin)
Local Node server    →  serves BAS JSON from device storage to WebView
WebView              →  feed + activity pages read from local server
                     →  edit drawer saves sidecars to device storage
```
See "Fully offline — missing pieces" below for implementation status.

### Setup

```bash
cd site

# First time: initialise native projects
npx cap add android   # creates site/android/
npx cap add ios       # creates site/ios/ — requires macOS + Xcode

# Daily workflow
npm run cap:sync      # build Astro + sync to native projects
npm run cap:android   # build + open Android Studio
npm run cap:ios       # build + open Xcode
```

### The bincio Python wheel

The `/convert/` page loads the extract pipeline via **Pyodide** (Python compiled to WebAssembly). The pipeline is bundled as `site/public/bincio.whl` — a standard Python wheel that Pyodide loads via `micropip`.

To rebuild the wheel after changing the extract code:
```bash
# from the repo root
uv build --wheel
cp dist/bincio-*.whl site/public/bincio.whl
```

Pyodide loads these packages on first visit (cached by the browser after that):
- `lxml`, `pyyaml` — prebuilt Pyodide packages (~2 MB)
- `fitdecode`, `gpxpy`, `rdp` — pure Python, installed via micropip
- `bincio.whl` — our wheel, ~150 KB

Total cold-start download: ~10 MB. Subsequent visits: instant (all cached).

### Fully offline — missing pieces

Five things need to be in place before Workflow 4 works end-to-end.

#### 1. Pyodide available offline

Currently loads from CDN on first use. Two options:

- **Service worker cache** *(recommended)* — on first online visit the service worker caches all Pyodide assets; subsequent visits work offline. Standard PWA pattern, no app size increase.
- **Bundle in app assets** — copy `pyodide/` into `site/public/` at build time. Adds ~25 MB to the app but works with zero network dependency. Use if targeting environments with no initial connectivity.

Status: not yet implemented.

#### 2. Write converted activities to device storage

`@capacitor/filesystem` is already installed. After Pyodide converts a file, instead of (or in addition to) POSTing to the edit server, the convert page should:
1. Write `{id}.json` and `{id}.geojson` to a local `activities/` directory via the Filesystem plugin
2. Append the summary to a local `index.json`

This is self-contained JS work — no native changes needed.

Status: not yet implemented.

#### 3. Serve local activity data to the WebView *(the core problem)*

The Astro site reads `/data/index.json` and `/data/activities/*.json` via `fetch()`. In the current setup these come from either bundled static assets (baked in at build time) or a remote server. There is no built-in mechanism for a WebView to dynamically serve locally-stored files at those URLs.

Two approaches:

**Option A — Service worker interception** *(web-only, no native code)*
A service worker intercepts `fetch('/data/*')` and responds with data from IndexedDB. No native plugin required. Limitation: iOS WKWebView has historically restricted service worker scope in Capacitor apps, though this has improved in recent iOS versions. Best option if cross-platform parity is not critical.

**Option B — Local HTTP server via `capacitor-nodejs`** *(recommended for iOS reliability)*
[`capacitor-nodejs`](https://github.com/hampoelz/Capacitor-NodeJS) embeds a Node.js runtime inside the Capacitor app. A small Express/Fastify server:
- Serves the static site assets (or proxies to the WebView asset loader)
- Serves BAS JSON from device storage at `/data/*`
- Handles the edit/save API (`/api/import-bas`, `/api/activity/{id}`, etc.)
The WebView points to `http://localhost:PORT/`. Reliable on both iOS and Android.

Status: not yet implemented. Option B is the recommended path.

#### 4. Port edit server API to local JS

Once Option B above is in place, the Node.js server needs to implement the same API surface as the current FastAPI edit server — at minimum:

| Endpoint | Purpose |
|---|---|
| `POST /api/import-bas` | Save converted activity to device storage |
| `GET /api/activity/{id}` | Read activity + sidecar |
| `POST /api/activity/{id}` | Write sidecar to device storage |
| `GET /data/index.json` | Serve merged index from device storage |
| `GET /data/activities/*` | Serve individual activity files |

The sidecar merge logic (`merge_all`) can either be reimplemented in JS (~150 lines) or invoked via Pyodide (already present in the app).

Strava sync and image upload require network access by definition — they are out of scope for offline mode.

Status: not yet implemented.

#### 5. Summary

| Step | Effort | Status |
|---|---|---|
| Pyodide service worker cache | Medium | Not started |
| Write to Filesystem after convert | Easy | Not started |
| Local Node.js server (`capacitor-nodejs`) | Hard | Not started |
| Port edit API to local JS | Medium | Not started |
| Sidecar merge in JS (or via Pyodide) | Medium | Not started |

The local Node.js server (step 3B) is the critical path item — it unblocks steps 4 and makes the feed dynamic. Steps 1 and 2 are independent and can be done in any order.

---

### The `/api/import-bas` endpoint

The `/convert/` page sends pre-converted BAS JSON directly to the edit server, avoiding the need to re-parse a file on the server side. The endpoint:
1. Validates the `id` field
2. Writes `activities/{id}.json` (and `.geojson` if provided)
3. Rebuilds `index.json`
4. Runs `merge_all()`

---

## Deployment

```mermaid
flowchart LR
    subgraph Local ["Local machine"]
        A[bincio extract]
        B[bincio edit\noptional]
    end

    subgraph VPS ["VPS or static host"]
        C[Static site\ndist/]
        D[nginx]
    end

    subgraph Phone ["Phone"]
        E[OsmAnd /\nOrganic Maps]
        F[Browser:\n/convert/]
    end

    A -->|rsync / CI| C
    C --> D
    E -->|GPX export| F
    F -->|upload| B
    B --> A
```

The edit server is **always local** in the single-user setup. In a future multi-user deployment on a VPS, it would run as a service alongside nginx.