feat(mobile/upload): upload_format setting + Option A local update from server response
Settings → Sync → Upload format: "Original file" (default) / "Extracted JSON". - raw (default): reads original_path as base64, POSTs to /api/upload/raw; after success, overwrites local detail_json/timeseries_json/geojson/source_hash with the server's DEM-corrected extraction (Option A). Falls back to bas if the file is missing. - bas: POSTs pre-extracted JSON to /api/upload/bas, faster, no DEM correction. Switching modes is safe — the server deduplicates by activity id so a previous raw upload will return status:"duplicate" on a subsequent bas attempt.
This commit is contained in:
+10
-8
@@ -673,6 +673,7 @@ CREATE TABLE settings (
|
|||||||
| `api_token` | Bearer token for API auth (obtained via Connect, never stored in plaintext long-term) | Both |
|
| `api_token` | Bearer token for API auth (obtained via Connect, never stored in plaintext long-term) | Both |
|
||||||
| `sync_mode` | `"summaries"` (default) or `"full"` — controls whether geojson+timeseries are downloaded during sync | Both |
|
| `sync_mode` | `"summaries"` (default) or `"full"` — controls whether geojson+timeseries are downloaded during sync | Both |
|
||||||
| `sync_upload` | `"true"` or `"false"` — whether to push local activities during sync | Both |
|
| `sync_upload` | `"true"` or `"false"` — whether to push local activities during sync | Both |
|
||||||
|
| `upload_format` | `"raw"` (default) or `"bas"` — whether to upload the original FIT/GPX/TCX file (server re-extracts with DEM) or the pre-extracted JSON | Both |
|
||||||
| `auto_import_path` | Directory to watch for new FIT files | **Android only** |
|
| `auto_import_path` | Directory to watch for new FIT files | **Android only** |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -700,17 +701,18 @@ Implemented in `mobile/db/sync.ts` → `uploadFeed()` / `uploadLocalActivities()
|
|||||||
|
|
||||||
1. **Reconcile** against the server: fetch `GET /api/feed` and compare its activity IDs against local rows where `synced_at IS NOT NULL`. Any local activity that is marked as synced but absent from the server (e.g. server was wiped) has its `synced_at` cleared so it re-enters the upload queue. This is best-effort — if the feed fetch fails, upload proceeds with whatever is currently queued.
|
1. **Reconcile** against the server: fetch `GET /api/feed` and compare its activity IDs against local rows where `synced_at IS NOT NULL`. Any local activity that is marked as synced but absent from the server (e.g. server was wiped) has its `synced_at` cleared so it re-enters the upload queue. This is best-effort — if the feed fetch fails, upload proceeds with whatever is currently queued.
|
||||||
2. Query `activities WHERE origin = 'local' AND synced_at IS NULL`.
|
2. Query `activities WHERE origin = 'local' AND synced_at IS NULL`.
|
||||||
3. For each: parse `detail_json` from the DB row and construct `{ id: row.id, ...detail }`.
|
3. For each, choose the upload path based on the `upload_format` setting (default `'raw'`):
|
||||||
4. `POST {instance_url}/api/upload/bas` with body `{ activity, timeseries?, geojson? }`.
|
- **`raw` (default):** if `original_path` is set and the file still exists on disk, read it as base64 and `POST {instance_url}/api/upload/raw { filename, base64 }`. The server re-extracts the file with DEM elevation correction and returns `{ id, detail, timeseries, geojson, source_hash }`. After a successful upload, the local row's `detail_json`, `timeseries_json`, `geojson`, and `source_hash` are updated with the server's better data (Option A). Falls back to `bas` if the file is missing.
|
||||||
5. On 200 (including `{ status: "duplicate" }`): set `synced_at = unixepoch()`. On error: log to console, count as failed, continue with the next activity.
|
- **`bas`:** `POST {instance_url}/api/upload/bas { activity, timeseries?, geojson? }` with the pre-extracted JSON from the DB. Faster, but no DEM elevation correction.
|
||||||
|
4. On 200 (including `{ status: "duplicate" }`): set `synced_at = unixepoch()`. On error: log to console, count as failed, continue with next activity.
|
||||||
|
|
||||||
The UI shows live progress ("Uploading N / M…") during the batch and reports failures separately ("X uploaded, Y failed").
|
The UI shows live progress ("Uploading N / M…") during the batch and reports failures separately ("X uploaded, Y failed").
|
||||||
|
|
||||||
The server endpoint (`bincio/serve/server.py` → `POST /api/upload/bas`) accepts
|
**Upload format setting (`upload_format`):** controls whether to prefer the original raw file or the pre-extracted JSON. `raw` is the default and is recommended — it produces DEM-corrected elevation data on the server and back-fills the local copy. Switching between modes is safe: the server deduplicates by activity id, so switching from `raw` to `bas` just results in the server returning `{ status: "duplicate" }` (HTTP 200) and the client marking the activity as synced.
|
||||||
pre-extracted BAS JSON rather than raw FIT/GPX/TCX. It writes the activity file,
|
|
||||||
**updates `user_dir/index.json`** with a summary entry (so `merge_all` can include
|
**Server endpoint for BAS JSON** (`POST /api/upload/bas`): accepts pre-extracted BAS JSON, writes the activity file, updates `user_dir/index.json` with a summary entry (so `merge_all` can include the activity in year shards and the browser feed), writes geojson and timeseries if provided, then calls `merge_all()` + `write_combined_feed()`.
|
||||||
the activity in year shards and the browser feed), writes geojson and timeseries if
|
|
||||||
provided, then calls `merge_all()` + `write_combined_feed()`.
|
**Server endpoint for raw files** (`POST /api/upload/raw`): accepts a base64-encoded FIT/GPX/TCX file, runs full server-side extraction with DEM correction, stores the result, updates the index, calls `merge_all()` + `write_combined_feed()`, and returns the full extracted data to the client.
|
||||||
|
|
||||||
> **Bug that was fixed:** earlier versions of both `/api/upload/bas` and `/api/upload/raw`
|
> **Bug that was fixed:** earlier versions of both `/api/upload/bas` and `/api/upload/raw`
|
||||||
> wrote activity files to disk but never updated `user_dir/index.json`. Since `merge_all`
|
> wrote activity files to disk but never updated `user_dir/index.json`. Since `merge_all`
|
||||||
|
|||||||
@@ -17,12 +17,14 @@ export default function SettingsScreen() {
|
|||||||
const storedToken = useSetting('api_token');
|
const storedToken = useSetting('api_token');
|
||||||
const storedSyncMode = (useSetting('sync_mode') ?? 'summaries') as 'summaries' | 'full';
|
const storedSyncMode = (useSetting('sync_mode') ?? 'summaries') as 'summaries' | 'full';
|
||||||
const storedSyncUpload = useSetting('sync_upload') === 'true';
|
const storedSyncUpload = useSetting('sync_upload') === 'true';
|
||||||
|
const storedUploadFormat = (useSetting('upload_format') ?? 'raw') as 'raw' | 'bas';
|
||||||
|
|
||||||
const [instanceUrl, setInstanceUrl] = useState(storedUrl);
|
const [instanceUrl, setInstanceUrl] = useState(storedUrl);
|
||||||
const [handle, setHandle] = useState(storedHandle);
|
const [handle, setHandle] = useState(storedHandle);
|
||||||
const [autoPath, setAutoPath] = useState(storedPath);
|
const [autoPath, setAutoPath] = useState(storedPath);
|
||||||
const [syncMode, setSyncMode] = useState(storedSyncMode);
|
const [syncMode, setSyncMode] = useState(storedSyncMode);
|
||||||
const [syncUpload, setSyncUpload] = useState(storedSyncUpload);
|
const [syncUpload, setSyncUpload] = useState(storedSyncUpload);
|
||||||
|
const [uploadFormat, setUploadFormat] = useState(storedUploadFormat);
|
||||||
const [saved, setSaved] = useState(false);
|
const [saved, setSaved] = useState(false);
|
||||||
const theme = useTheme();
|
const theme = useTheme();
|
||||||
const { paletteKey: palette, setPaletteOverride } = usePaletteControl();
|
const { paletteKey: palette, setPaletteOverride } = usePaletteControl();
|
||||||
@@ -213,6 +215,18 @@ export default function SettingsScreen() {
|
|||||||
? 'Local activities are uploaded to the instance during sync.'
|
? 'Local activities are uploaded to the instance during sync.'
|
||||||
: 'Local activities stay on device only.'}
|
: 'Local activities stay on device only.'}
|
||||||
</Text>
|
</Text>
|
||||||
|
<Text style={[styles.subLabel, { borderTopWidth: 1, borderTopColor: '#27272a', paddingTop: 12 }]}>Upload format</Text>
|
||||||
|
<View style={styles.modeRow}>
|
||||||
|
<ModeButton label="Original file" active={uploadFormat === 'raw'} accent={theme.accent} dim={theme.dim}
|
||||||
|
onPress={() => { setUploadFormat('raw'); setSetting(db, 'upload_format', 'raw'); }} />
|
||||||
|
<ModeButton label="Extracted JSON" active={uploadFormat === 'bas'} accent={theme.accent} dim={theme.dim}
|
||||||
|
onPress={() => { setUploadFormat('bas'); setSetting(db, 'upload_format', 'bas'); }} />
|
||||||
|
</View>
|
||||||
|
<Text style={styles.hint}>
|
||||||
|
{uploadFormat === 'raw'
|
||||||
|
? 'Uploads the original FIT/GPX/TCX file. The server re-extracts it with DEM elevation correction and updates your local copy.'
|
||||||
|
: 'Uploads the pre-extracted JSON. Faster, but no DEM elevation correction.'}
|
||||||
|
</Text>
|
||||||
</Section>
|
</Section>
|
||||||
|
|
||||||
<Section title="Palette">
|
<Section title="Palette">
|
||||||
|
|||||||
+37
-3
@@ -170,6 +170,7 @@ async function uploadLocalActivities(
|
|||||||
FROM activities WHERE origin = 'local' AND synced_at IS NULL`,
|
FROM activities WHERE origin = 'local' AND synced_at IS NULL`,
|
||||||
);
|
);
|
||||||
|
|
||||||
|
const preferRaw = (await getSetting(db, 'upload_format') ?? 'raw') === 'raw';
|
||||||
const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' };
|
const headers = { Authorization: `Bearer ${token}`, 'Content-Type': 'application/json' };
|
||||||
let uploaded = 0;
|
let uploaded = 0;
|
||||||
let failed = 0;
|
let failed = 0;
|
||||||
@@ -182,9 +183,10 @@ async function uploadLocalActivities(
|
|||||||
try {
|
try {
|
||||||
let resp: Response;
|
let resp: Response;
|
||||||
|
|
||||||
// Prefer raw upload when the original FIT/GPX/TCX file is still on disk.
|
// When preferRaw is set and the original file is still on disk, send the raw
|
||||||
// The server re-extracts it with DEM elevation correction, producing better data.
|
// bytes to /api/upload/raw so the server re-extracts with DEM elevation correction.
|
||||||
const useRaw = row.original_path !== null &&
|
const useRaw = preferRaw &&
|
||||||
|
row.original_path !== null &&
|
||||||
(await FileSystem.getInfoAsync(row.original_path)).exists;
|
(await FileSystem.getInfoAsync(row.original_path)).exists;
|
||||||
|
|
||||||
if (useRaw) {
|
if (useRaw) {
|
||||||
@@ -211,6 +213,38 @@ async function uploadLocalActivities(
|
|||||||
|
|
||||||
if (resp.ok) {
|
if (resp.ok) {
|
||||||
await db.runAsync(`UPDATE activities SET synced_at = ? WHERE id = ?`, [now, row.id]);
|
await db.runAsync(`UPDATE activities SET synced_at = ? WHERE id = ?`, [now, row.id]);
|
||||||
|
// Option A: after a raw upload, update local detail/timeseries/geojson with the
|
||||||
|
// server's DEM-corrected extraction so the app shows better elevation data.
|
||||||
|
if (useRaw) {
|
||||||
|
try {
|
||||||
|
const data = await resp.json() as {
|
||||||
|
id: string;
|
||||||
|
detail: object;
|
||||||
|
timeseries: object | null;
|
||||||
|
geojson: object | null;
|
||||||
|
source_hash: string;
|
||||||
|
};
|
||||||
|
if (data.id === row.id) {
|
||||||
|
await db.runAsync(
|
||||||
|
`UPDATE activities
|
||||||
|
SET detail_json = ?,
|
||||||
|
timeseries_json = COALESCE(?, timeseries_json),
|
||||||
|
geojson = COALESCE(?, geojson),
|
||||||
|
source_hash = ?
|
||||||
|
WHERE id = ?`,
|
||||||
|
[
|
||||||
|
JSON.stringify(data.detail),
|
||||||
|
data.timeseries ? JSON.stringify(data.timeseries) : null,
|
||||||
|
data.geojson ? JSON.stringify(data.geojson) : null,
|
||||||
|
data.source_hash,
|
||||||
|
row.id,
|
||||||
|
],
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// Non-fatal: synced_at is already set, local data stays as-is
|
||||||
|
}
|
||||||
|
}
|
||||||
uploaded++;
|
uploaded++;
|
||||||
} else {
|
} else {
|
||||||
console.warn(`upload ${row.id}: HTTP ${resp.status}`);
|
console.warn(`upload ${row.id}: HTTP ${resp.status}`);
|
||||||
|
|||||||
Reference in New Issue
Block a user