fix(upload): prevent false 422s and EMFILE crash during bulk uploads
Four related issues made uploading 271+ activities unreliable: 1. merge_all/write_combined_feed were inside the extraction try/except — any merge race returned 422 even though the file was on disk, causing the mobile app to permanently mark the upload as failed. Fixed by moving them to a separate best-effort try/except after the extraction block. Switch to merge_one (single-activity symlink) instead of merge_all (full rebuild) so each upload is O(1) FS ops, not O(N). 2. The dev watcher fired merge_all for every activity .json write AND the upload endpoint also ran merge_all — O(N²) symlink operations during bulk uploads. Watcher now skips activities/*.json changes (upload endpoint handles those directly). 3. Vite/Chokidar followed the public/data symlink and opened a handle per activity file; constant merge rebuilds exhausted file descriptors and crashed the Astro dev server. Fixed with watch.ignored on public/data. 4. _write_year_shards and write_combined_feed used f.unlink() without missing_ok=True — concurrent callers racing the same file threw FileNotFoundError which propagated as a false extraction failure.
This commit is contained in:
+16
-7
@@ -596,9 +596,12 @@ async def upload_bas_activity(
|
||||
|
||||
_upsert_index_summary(user_dir, activity_id, activity, geojson_body)
|
||||
|
||||
from bincio.render.merge import merge_all, write_combined_feed
|
||||
merge_all(user_dir)
|
||||
write_combined_feed(_get_data_dir())
|
||||
try:
|
||||
from bincio.render.merge import merge_one, write_combined_feed
|
||||
merge_one(user_dir, activity_id)
|
||||
write_combined_feed(_get_data_dir())
|
||||
except Exception as exc:
|
||||
log.warning("upload/bas[%s]: merge/feed failed (non-fatal): %s", user.handle, exc)
|
||||
|
||||
log.info("upload/bas[%s]: imported %s", user.handle, activity_id)
|
||||
return JSONResponse({"ok": True, "id": activity_id, "status": "imported"})
|
||||
@@ -688,10 +691,6 @@ async def upload_raw_activity(
|
||||
|
||||
_upsert_index_summary(user_dir, act_id, detail, geojson)
|
||||
|
||||
from bincio.render.merge import merge_all, write_combined_feed
|
||||
merge_all(user_dir)
|
||||
write_combined_feed(_get_data_dir())
|
||||
|
||||
except Exception as exc:
|
||||
log.warning("upload/raw[%s]: extraction failed: %s", user.handle, exc)
|
||||
raise HTTPException(422, f"Could not extract activity: {exc}") from exc
|
||||
@@ -699,6 +698,16 @@ async def upload_raw_activity(
|
||||
tmp_in.unlink(missing_ok=True)
|
||||
shutil.rmtree(tmp_out, ignore_errors=True)
|
||||
|
||||
# Merge and update feed — best effort; a race or transient FS error here must
|
||||
# not turn a successful extraction into a 422 (the file is on disk; the mobile
|
||||
# would retry indefinitely and the activity would never be marked synced).
|
||||
try:
|
||||
from bincio.render.merge import merge_one, write_combined_feed
|
||||
merge_one(user_dir, act_id)
|
||||
write_combined_feed(_get_data_dir())
|
||||
except Exception as exc:
|
||||
log.warning("upload/raw[%s]: merge/feed failed (non-fatal): %s", user.handle, exc)
|
||||
|
||||
log.info("upload/raw[%s]: imported %s", user.handle, act_id)
|
||||
return JSONResponse({
|
||||
"ok": True,
|
||||
|
||||
Reference in New Issue
Block a user