feat(mobile/upload): upload_format setting + Option A local update from server response

Settings → Sync → Upload format: "Original file" (default) / "Extracted JSON".
- raw (default): reads original_path as base64, POSTs to /api/upload/raw; after
  success, overwrites local detail_json/timeseries_json/geojson/source_hash with
  the server's DEM-corrected extraction (Option A). Falls back to bas if the file
  is missing.
- bas: POSTs pre-extracted JSON to /api/upload/bas, faster, no DEM correction.

Switching modes is safe — the server deduplicates by activity id so a previous
raw upload will return status:"duplicate" on a subsequent bas attempt.
This commit is contained in:
Davide Scaini
2026-04-27 11:44:32 +02:00
parent 0d2176aef0
commit 93247d510f
3 changed files with 65 additions and 15 deletions
+10 -8
View File
@@ -673,6 +673,7 @@ CREATE TABLE settings (
| `api_token` | Bearer token for API auth (obtained via Connect, never stored in plaintext long-term) | Both | | `api_token` | Bearer token for API auth (obtained via Connect, never stored in plaintext long-term) | Both |
| `sync_mode` | `"summaries"` (default) or `"full"` — controls whether geojson+timeseries are downloaded during sync | Both | | `sync_mode` | `"summaries"` (default) or `"full"` — controls whether geojson+timeseries are downloaded during sync | Both |
| `sync_upload` | `"true"` or `"false"` — whether to push local activities during sync | Both | | `sync_upload` | `"true"` or `"false"` — whether to push local activities during sync | Both |
| `upload_format` | `"raw"` (default) or `"bas"` — whether to upload the original FIT/GPX/TCX file (server re-extracts with DEM) or the pre-extracted JSON | Both |
| `auto_import_path` | Directory to watch for new FIT files | **Android only** | | `auto_import_path` | Directory to watch for new FIT files | **Android only** |
--- ---
@@ -700,17 +701,18 @@ Implemented in `mobile/db/sync.ts` → `uploadFeed()` / `uploadLocalActivities()
1. **Reconcile** against the server: fetch `GET /api/feed` and compare its activity IDs against local rows where `synced_at IS NOT NULL`. Any local activity that is marked as synced but absent from the server (e.g. server was wiped) has its `synced_at` cleared so it re-enters the upload queue. This is best-effort — if the feed fetch fails, upload proceeds with whatever is currently queued. 1. **Reconcile** against the server: fetch `GET /api/feed` and compare its activity IDs against local rows where `synced_at IS NOT NULL`. Any local activity that is marked as synced but absent from the server (e.g. server was wiped) has its `synced_at` cleared so it re-enters the upload queue. This is best-effort — if the feed fetch fails, upload proceeds with whatever is currently queued.
2. Query `activities WHERE origin = 'local' AND synced_at IS NULL`. 2. Query `activities WHERE origin = 'local' AND synced_at IS NULL`.
3. For each: parse `detail_json` from the DB row and construct `{ id: row.id, ...detail }`. 3. For each, choose the upload path based on the `upload_format` setting (default `'raw'`):
4. `POST {instance_url}/api/upload/bas` with body `{ activity, timeseries?, geojson? }`. - **`raw` (default):** if `original_path` is set and the file still exists on disk, read it as base64 and `POST {instance_url}/api/upload/raw { filename, base64 }`. The server re-extracts the file with DEM elevation correction and returns `{ id, detail, timeseries, geojson, source_hash }`. After a successful upload, the local row's `detail_json`, `timeseries_json`, `geojson`, and `source_hash` are updated with the server's better data (Option A). Falls back to `bas` if the file is missing.
5. On 200 (including `{ status: "duplicate" }`): set `synced_at = unixepoch()`. On error: log to console, count as failed, continue with the next activity. - **`bas`:** `POST {instance_url}/api/upload/bas { activity, timeseries?, geojson? }` with the pre-extracted JSON from the DB. Faster, but no DEM elevation correction.
4. On 200 (including `{ status: "duplicate" }`): set `synced_at = unixepoch()`. On error: log to console, count as failed, continue with next activity.
The UI shows live progress ("Uploading N / M…") during the batch and reports failures separately ("X uploaded, Y failed"). The UI shows live progress ("Uploading N / M…") during the batch and reports failures separately ("X uploaded, Y failed").
The server endpoint (`bincio/serve/server.py` → `POST /api/upload/bas`) accepts **Upload format setting (`upload_format`):** controls whether to prefer the original raw file or the pre-extracted JSON. `raw` is the default and is recommended — it produces DEM-corrected elevation data on the server and back-fills the local copy. Switching between modes is safe: the server deduplicates by activity id, so switching from `raw` to `bas` just results in the server returning `{ status: "duplicate" }` (HTTP 200) and the client marking the activity as synced.
pre-extracted BAS JSON rather than raw FIT/GPX/TCX. It writes the activity file,
**updates `user_dir/index.json`** with a summary entry (so `merge_all` can include **Server endpoint for BAS JSON** (`POST /api/upload/bas`): accepts pre-extracted BAS JSON, writes the activity file, updates `user_dir/index.json` with a summary entry (so `merge_all` can include the activity in year shards and the browser feed), writes geojson and timeseries if provided, then calls `merge_all()` + `write_combined_feed()`.
the activity in year shards and the browser feed), writes geojson and timeseries if
provided, then calls `merge_all()` + `write_combined_feed()`. **Server endpoint for raw files** (`POST /api/upload/raw`): accepts a base64-encoded FIT/GPX/TCX file, runs full server-side extraction with DEM correction, stores the result, updates the index, calls `merge_all()` + `write_combined_feed()`, and returns the full extracted data to the client.
> **Bug that was fixed:** earlier versions of both `/api/upload/bas` and `/api/upload/raw` > **Bug that was fixed:** earlier versions of both `/api/upload/bas` and `/api/upload/raw`
> wrote activity files to disk but never updated `user_dir/index.json`. Since `merge_all` > wrote activity files to disk but never updated `user_dir/index.json`. Since `merge_all`
+14
View File
@@ -17,12 +17,14 @@ export default function SettingsScreen() {
const storedToken = useSetting('api_token'); const storedToken = useSetting('api_token');
const storedSyncMode = (useSetting('sync_mode') ?? 'summaries') as 'summaries' | 'full'; const storedSyncMode = (useSetting('sync_mode') ?? 'summaries') as 'summaries' | 'full';
const storedSyncUpload = useSetting('sync_upload') === 'true'; const storedSyncUpload = useSetting('sync_upload') === 'true';
const storedUploadFormat = (useSetting('upload_format') ?? 'raw') as 'raw' | 'bas';
const [instanceUrl, setInstanceUrl] = useState(storedUrl); const [instanceUrl, setInstanceUrl] = useState(storedUrl);
const [handle, setHandle] = useState(storedHandle); const [handle, setHandle] = useState(storedHandle);
const [autoPath, setAutoPath] = useState(storedPath); const [autoPath, setAutoPath] = useState(storedPath);
const [syncMode, setSyncMode] = useState(storedSyncMode); const [syncMode, setSyncMode] = useState(storedSyncMode);
const [syncUpload, setSyncUpload] = useState(storedSyncUpload); const [syncUpload, setSyncUpload] = useState(storedSyncUpload);
const [uploadFormat, setUploadFormat] = useState(storedUploadFormat);
const [saved, setSaved] = useState(false); const [saved, setSaved] = useState(false);
const theme = useTheme(); const theme = useTheme();
const { paletteKey: palette, setPaletteOverride } = usePaletteControl(); const { paletteKey: palette, setPaletteOverride } = usePaletteControl();
@@ -213,6 +215,18 @@ export default function SettingsScreen() {
? 'Local activities are uploaded to the instance during sync.' ? 'Local activities are uploaded to the instance during sync.'
: 'Local activities stay on device only.'} : 'Local activities stay on device only.'}
</Text> </Text>
<Text style={[styles.subLabel, { borderTopWidth: 1, borderTopColor: '#27272a', paddingTop: 12 }]}>Upload format</Text>
<View style={styles.modeRow}>
<ModeButton label="Original file" active={uploadFormat === 'raw'} accent={theme.accent} dim={theme.dim}
onPress={() => { setUploadFormat('raw'); setSetting(db, 'upload_format', 'raw'); }} />
<ModeButton label="Extracted JSON" active={uploadFormat === 'bas'} accent={theme.accent} dim={theme.dim}
onPress={() => { setUploadFormat('bas'); setSetting(db, 'upload_format', 'bas'); }} />
</View>
<Text style={styles.hint}>
{uploadFormat === 'raw'
? 'Uploads the original FIT/GPX/TCX file. The server re-extracts it with DEM elevation correction and updates your local copy.'
: 'Uploads the pre-extracted JSON. Faster, but no DEM elevation correction.'}
</Text>
</Section> </Section>
<Section title="Palette"> <Section title="Palette">
+37 -3
View File
@@ -170,6 +170,7 @@ async function uploadLocalActivities(
FROM activities WHERE origin = 'local' AND synced_at IS NULL`, FROM activities WHERE origin = 'local' AND synced_at IS NULL`,
); );
const preferRaw = (await getSetting(db, 'upload_format') ?? 'raw') === 'raw';
const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' }; const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' };
let uploaded = 0; let uploaded = 0;
let failed = 0; let failed = 0;
@@ -182,9 +183,10 @@ async function uploadLocalActivities(
try { try {
let resp: Response; let resp: Response;
// Prefer raw upload when the original FIT/GPX/TCX file is still on disk. // When preferRaw is set and the original file is still on disk, send the raw
// The server re-extracts it with DEM elevation correction, producing better data. // bytes to /api/upload/raw so the server re-extracts with DEM elevation correction.
const useRaw = row.original_path !== null && const useRaw = preferRaw &&
row.original_path !== null &&
(await FileSystem.getInfoAsync(row.original_path)).exists; (await FileSystem.getInfoAsync(row.original_path)).exists;
if (useRaw) { if (useRaw) {
@@ -211,6 +213,38 @@ async function uploadLocalActivities(
if (resp.ok) { if (resp.ok) {
await db.runAsync(`UPDATE activities SET synced_at = ? WHERE id = ?`, [now, row.id]); await db.runAsync(`UPDATE activities SET synced_at = ? WHERE id = ?`, [now, row.id]);
// Option A: after a raw upload, update local detail/timeseries/geojson with the
// server's DEM-corrected extraction so the app shows better elevation data.
if (useRaw) {
try {
const data = await resp.json() as {
id: string;
detail: object;
timeseries: object | null;
geojson: object | null;
source_hash: string;
};
if (data.id === row.id) {
await db.runAsync(
`UPDATE activities
SET detail_json = ?,
timeseries_json = COALESCE(?, timeseries_json),
geojson = COALESCE(?, geojson),
source_hash = ?
WHERE id = ?`,
[
JSON.stringify(data.detail),
data.timeseries ? JSON.stringify(data.timeseries) : null,
data.geojson ? JSON.stringify(data.geojson) : null,
data.source_hash,
row.id,
],
);
}
} catch {
// Non-fatal: synced_at is already set, local data stays as-is
}
}
uploaded++; uploaded++;
} else { } else {
console.warn(`upload ${row.id}: HTTP ${resp.status}`); console.warn(`upload ${row.id}: HTTP ${resp.status}`);