add architecture graph generator and docs

scripts/gen_graph.py parses FastAPI routes, frontend fetch() calls,
component imports, and Python imports to auto-generate:
- docs/architecture.mmd: Mermaid diagram with API/Pages/Components/Python subgraphs
- docs/graph.html: standalone vis.js interactive graph (dark theme, group filters,
  search highlight, click-to-highlight connected nodes)

docs-proposal.md: proposal for a docs/ folder structure, API documentation strategy,
and tooling recommendations (plain markdown → MkDocs Material).
This commit is contained in:
Davide Scaini
2026-04-14 22:45:03 +02:00
parent 9419bd0c20
commit a14cee8710
4 changed files with 2277 additions and 0 deletions
+113
View File
@@ -0,0 +1,113 @@
# Documentation proposal
## Problem
The project has no user-facing or developer-facing docs. Knowledge lives in `CLAUDE.md`
(written for AI context, not humans), scattered inline comments, and the code itself.
As the feature surface grows and more users join, we need:
- A guide for **users** (how to upload, sync, edit, manage privacy)
- A guide for **admins** (how to run an instance, manage users, reset passwords)
- An **API reference** (what endpoints exist, what they expect, what they return)
- A **developer guide** (how to run locally, architecture, how to contribute)
---
## Proposed structure
```
docs/
index.md Overview and quick links
user-guide.md End-user: upload, sync, edit, privacy, settings
admin-guide.md Admin: deploy, init, invite users, reset passwords, rebuild
api.md API reference (hand-written, augmented by OpenAPI)
architecture.md BAS schema, data flow, shard model, federation
developer-guide.md Local setup, how to run tests, how to contribute
```
`CLAUDE.md` stays as-is — it is AI context, not user docs. The two serve different
audiences and should not be merged.
---
## API documentation strategy
FastAPI auto-generates an OpenAPI 3.1 spec from the route decorators. It is already
served at `/api/docs` (Swagger UI) and `/api/redoc` (ReDoc) when the server is running.
Right now the auto-docs are sparse because:
- Most endpoints return bare `JSONResponse` instead of typed Pydantic response models
- Endpoint docstrings are minimal or absent
- Request bodies are raw `request.json()` instead of Pydantic models
### Recommended approach: two-layer docs
**Layer 1 — machine-readable (OpenAPI, auto-generated)**
Incrementally add Pydantic request/response models to the endpoints that matter most
(auth, activity CRUD, admin actions). FastAPI will pick them up automatically and the
Swagger UI becomes usable. No extra tooling needed.
Priority endpoints to type first:
- `POST /api/auth/login` / `logout` / `reset-password`
- `POST /api/register`
- `GET /api/me`
- `GET|POST /api/activity/{id}`
- `DELETE /api/activity/{id}`
- `POST /api/admin/users/{handle}/reset-password-code`
- `GET|POST /api/me/preferences` (once built)
**Layer 2 — human-readable (`docs/api.md`)**
A hand-written reference that groups endpoints by domain (auth, activities, admin,
sync), explains the overall auth model (cookie-based, httpOnly), rate limiting, and
covers things OpenAPI can't express well (SSE streams, error semantics, side effects
like rebuild triggers).
The OpenAPI spec and the hand-written doc are complementary, not duplicates:
OpenAPI is precise and machine-readable; `api.md` gives context and explains *why*.
---
## Tooling options
| Option | Pros | Cons |
|--------|------|------|
| Plain markdown in `docs/` | Zero tooling, lives in repo, renders on GitHub | No search, no versioning, no sidebar nav |
| MkDocs + Material theme | Beautiful, search, auto-nav from folder structure, can embed OpenAPI via plugins | Needs Python dep + build step; another thing to deploy |
| Docusaurus | Great for open-source projects, versioning, i18n | Node toolchain, heavier |
| VitePress | Fast, Vite-based (already in the stack), markdown + Vue | Still a separate site to host |
| Just the Swagger UI at `/api/docs` | Auto-generated, always up-to-date | Only covers the API, not user/admin/architecture |
**Recommendation:** Start with plain markdown in `docs/` — no build step, always
available, no new infrastructure. If the project goes public or the user base grows,
migrate to MkDocs Material (one `mkdocs.yml` + `pip install mkdocs-material`).
For the API specifically: enable the Swagger UI on the live server (currently it may
be disabled in production) so admins can explore it directly at `/api/docs`.
---
## Enabling Swagger UI in production
By default FastAPI serves `/docs` and `/redoc`. In `bincio serve`, the FastAPI app is
created with:
```python
app = FastAPI(docs_url=None, redoc_url=None) # check current value
```
For a private instance (auth-walled), it is safe to expose `/api/docs` — add a note
in `admin-guide.md` that it exists. Alternatively, serve it only when an env var is set.
---
## Suggested first milestone
1. Create `docs/` with `index.md`, `admin-guide.md`, `api.md`
2. `admin-guide.md`: deploy, init, invite, password reset, rebuild, reset data
3. `api.md`: auth endpoints + activity endpoints, hand-written
4. Enable Swagger UI on the server (or at least document that it exists at `/api/docs`)
5. Add Pydantic models to the 8 priority endpoints above
Everything else (user guide, architecture, developer guide, MkDocs) is second milestone.
+274
View File
@@ -0,0 +1,274 @@
graph LR
subgraph API
subgraph api_activity["activity"]
api__api_activity__activity_id_["GET /api/activity/{activity_id}"]
api__api_activity__activity_id_["POST /api/activity/{activity_id}"]
api__api_activity__activity_id_["DELETE /api/activity/{activity_id}"]
api__api_activity__activity_id__images["GET /api/activity/{activity_id}/images"]
api__api_activity__activity_id__images["POST /api/activity/{activity_id}/images"]
api__api_activity__activity_id__images__filename_["DELETE /api/activity/{activity_id}/images/{filename}"]
end
subgraph api_admin["admin"]
api__api_admin_users["GET /api/admin/users"]
api__api_admin_jobs["GET /api/admin/jobs"]
api__api_admin_disk["GET /api/admin/disk"]
api__api_admin_users__handle__reset_password_code["POST /api/admin/users/{handle}/reset-password-code"]
api__api_admin_users__handle__rebuild["POST /api/admin/users/{handle}/rebuild"]
api__api_admin_users__handle__activities["DELETE /api/admin/users/{handle}/activities"]
end
subgraph api_athlete["athlete"]
api__api_athlete["GET /api/athlete"]
api__api_athlete["POST /api/athlete"]
end
subgraph api_auth["auth"]
api__api_auth_login["POST /api/auth/login"]
api__api_auth_logout["POST /api/auth/logout"]
api__api_auth_reset_password["POST /api/auth/reset-password"]
end
subgraph api_feedback["feedback"]
api__api_feedback["POST /api/feedback"]
end
subgraph api_garmin["garmin"]
api__api_garmin_status["GET /api/garmin/status"]
api__api_garmin_connect["POST /api/garmin/connect"]
api__api_garmin_disconnect["POST /api/garmin/disconnect"]
api__api_garmin_sync_stream["GET /api/garmin/sync/stream"]
end
subgraph api_invites["invites"]
api__api_invites["GET /api/invites"]
api__api_invites["POST /api/invites"]
end
subgraph api_me["me"]
api__api_me["GET /api/me"]
end
subgraph api_register["register"]
api__api_register["POST /api/register"]
end
subgraph api_stats["stats"]
api__api_stats["GET /api/stats"]
end
subgraph api_strava["strava"]
api__api_strava_status["GET /api/strava/status"]
api__api_strava_reset["POST /api/strava/reset"]
api__api_strava_auth_url["GET /api/strava/auth-url"]
api__api_strava_callback["GET /api/strava/callback"]
api__api_strava_sync_stream["GET /api/strava/sync/stream"]
api__api_strava_sync["POST /api/strava/sync"]
end
subgraph api_upload["upload"]
api__api_upload["POST /api/upload"]
api__api_upload_strava_zip["POST /api/upload/strava-zip"]
end
end
subgraph Pages
site_src_pages_about_ca_index_astro["pages/about/ca/"]
site_src_pages_about_es_index_astro["pages/about/es/"]
site_src_pages_about_index_astro["pages/about/"]
site_src_pages_about_it_index_astro["pages/about/it/"]
site_src_pages_activity__id__astro["pages/activity/[id].astro"]
site_src_pages_activity_index_astro["pages/activity/"]
site_src_pages_activity_local_index_astro["pages/activity/local/"]
site_src_pages_admin_index_astro["pages/admin/"]
site_src_pages_athlete_index_astro["pages/athlete/"]
site_src_pages_community_index_astro["pages/community/"]
site_src_pages_convert_index_astro["pages/convert/"]
site_src_pages_feedback_index_astro["pages/feedback/"]
site_src_pages_index_astro["pages/"]
site_src_pages_invites_index_astro["pages/invites/"]
site_src_pages_login_index_astro["pages/login/"]
site_src_pages_record_index_astro["pages/record/"]
site_src_pages_register_index_astro["pages/register/"]
site_src_pages_reset_password_index_astro["pages/reset-password/"]
site_src_pages_stats_index_astro["pages/stats/"]
site_src_pages_u__handle__athlete_index_astro["pages/u/[handle]/athlete/"]
site_src_pages_u__handle__index_astro["pages/u/[handle]/"]
site_src_pages_u__handle__stats_index_astro["pages/u/[handle]/stats/"]
end
subgraph Components
site_src_components_ActivityCharts_svelte["components/ActivityCharts.svelte"]
site_src_components_ActivityDetail_svelte["components/ActivityDetail.svelte"]
site_src_components_ActivityDetailLoader_svelte["components/ActivityDetailLoader.svelte"]
site_src_components_ActivityFeed_svelte["components/ActivityFeed.svelte"]
site_src_components_ActivityMap_svelte["components/ActivityMap.svelte"]
site_src_components_AthleteDrawer_svelte["components/AthleteDrawer.svelte"]
site_src_components_AthleteView_svelte["components/AthleteView.svelte"]
site_src_components_CommunityView_svelte["components/CommunityView.svelte"]
site_src_components_EditDrawer_svelte["components/EditDrawer.svelte"]
site_src_components_LocalActivityDetail_svelte["components/LocalActivityDetail.svelte"]
site_src_components_MmpChart_svelte["components/MmpChart.svelte"]
site_src_components_RecordsView_svelte["components/RecordsView.svelte"]
site_src_components_StatsView_svelte["components/StatsView.svelte"]
end
subgraph Python
subgraph py_edit["edit"]
bincio_edit_cli_py["cli"]
bincio_edit_ops_py["ops"]
bincio_edit_server_py["server"]
end
subgraph py_extract["extract"]
bincio_extract_cli_py["cli"]
bincio_extract_config_py["config"]
bincio_extract_dedup_py["dedup"]
bincio_extract_garmin_api_py["garmin_api"]
bincio_extract_garmin_sync_py["garmin_sync"]
bincio_extract_ingest_py["ingest"]
bincio_extract_metrics_py["metrics"]
bincio_extract_models_py["models"]
bincio_extract_parsers_base_py["base"]
bincio_extract_parsers_factory_py["factory"]
bincio_extract_parsers_fit_py["fit"]
bincio_extract_parsers_gpx_py["gpx"]
bincio_extract_parsers_tcx_py["tcx"]
bincio_extract_simplify_py["simplify"]
bincio_extract_sport_py["sport"]
bincio_extract_strava_api_py["strava_api"]
bincio_extract_strava_csv_py["strava_csv"]
bincio_extract_strava_zip_py["strava_zip"]
bincio_extract_timeseries_py["timeseries"]
bincio_extract_writer_py["writer"]
end
subgraph py_import_["import_"]
bincio_import__cli_py["cli"]
bincio_import__strava_py["strava"]
end
subgraph py_render["render"]
bincio_render_cli_py["cli"]
bincio_render_merge_py["merge"]
end
subgraph py_root["root"]
bincio_cli_py["cli"]
bincio_dev_py["dev"]
end
subgraph py_serve["serve"]
bincio_serve_cli_py["cli"]
bincio_serve_db_py["db"]
bincio_serve_init_cmd_py["init_cmd"]
bincio_serve_server_py["server"]
end
end
site_src_components_EditDrawer_svelte -->|fetch| api__api_activity_
site_src_components_AthleteDrawer_svelte -->|fetch| api__api_athlete
site_src_components_AthleteView_svelte -->|fetch| api__api_athlete
site_src_components_AthleteView_svelte -->|fetch| api__api_athlete__
site_src_layouts_Base_astro -->|fetch| api__api_me
site_src_layouts_Base_astro -->|fetch| api__api_admin_jobs
site_src_layouts_Base_astro -->|fetch| api__api_auth_logout
site_src_layouts_Base_astro -->|fetch| api__api_admin_jobs__
site_src_layouts_Base_astro -->|fetch| api__api_auth_logout__
site_src_layouts_Base_astro -->|fetch| api__api_upload
site_src_layouts_Base_astro -->|fetch| api__api_strava_status
site_src_layouts_Base_astro -->|fetch| api__api_strava_auth_url
site_src_layouts_Base_astro -->|fetch| api__api_strava_sync_stream
site_src_layouts_Base_astro -->|fetch| api__api_strava_reset
site_src_layouts_Base_astro -->|fetch| api__api_upload_strava_zip
site_src_layouts_Base_astro -->|fetch| api__api_garmin_status
site_src_layouts_Base_astro -->|fetch| api__api_garmin_connect
site_src_layouts_Base_astro -->|fetch| api__api_garmin_sync_stream
site_src_layouts_Base_astro -->|fetch| api__api_garmin_disconnect
site_src_pages_admin_index_astro -->|fetch| api__api_admin_disk
site_src_pages_admin_index_astro -->|fetch| api__api_admin_users___h__rebuild
site_src_pages_admin_index_astro -->|fetch| api__api_admin_users___h__reset_password_code
site_src_pages_admin_index_astro -->|fetch| api__api_admin_users___pendingHandle__activities
site_src_pages_admin_index_astro -->|fetch| api__api_admin_disk__
site_src_pages_admin_index_astro -->|fetch| api__api_admin_users_
site_src_pages_about_index_astro -->|fetch| api__api_me
site_src_pages_about_index_astro -->|fetch| api__api_stats
site_src_pages_about_index_astro -->|fetch| api__api_stats___
site_src_pages_feedback_index_astro -->|fetch| api__api_feedback
site_src_pages_feedback_index_astro -->|fetch| api__api_me
site_src_pages_feedback_index_astro -->|fetch| api__api_feedback__
site_src_pages_feedback_index_astro -->|fetch| api__api_me__
site_src_pages_register_index_astro -->|fetch| api__api_register
site_src_pages_reset_password_index_astro -->|fetch| api__api_auth_reset_password
site_src_pages_invites_index_astro -->|fetch| api__api_invites
site_src_pages_invites_index_astro -->|fetch| api__api_invites__
site_src_pages_login_index_astro -->|fetch| api__api_auth_login
site_src_pages_convert_index_astro -->|fetch| api__api_import_bas
site_src_pages_about_it_index_astro -->|fetch| api__api_me
site_src_pages_about_it_index_astro -->|fetch| api__api_stats
site_src_pages_about_it_index_astro -->|fetch| api__api_stats___
site_src_pages_about_ca_index_astro -->|fetch| api__api_me
site_src_pages_about_ca_index_astro -->|fetch| api__api_stats
site_src_pages_about_ca_index_astro -->|fetch| api__api_stats___
site_src_pages_about_es_index_astro -->|fetch| api__api_me
site_src_pages_about_es_index_astro -->|fetch| api__api_stats
site_src_pages_about_es_index_astro -->|fetch| api__api_stats___
site_src_components_ActivityDetail_svelte --> site_src_components_ActivityMap_svelte
site_src_components_ActivityDetail_svelte --> site_src_components_ActivityCharts_svelte
site_src_components_ActivityDetail_svelte --> site_src_components_EditDrawer_svelte
site_src_components_ActivityDetailLoader_svelte --> site_src_components_ActivityDetail_svelte
site_src_components_AthleteView_svelte --> site_src_components_MmpChart_svelte
site_src_components_AthleteView_svelte --> site_src_components_RecordsView_svelte
site_src_components_AthleteView_svelte --> site_src_components_AthleteDrawer_svelte
site_src_components_LocalActivityDetail_svelte --> site_src_components_ActivityDetail_svelte
site_src_pages_index_astro --> site_src_components_ActivityFeed_svelte
site_src_pages_index_astro --> site_src_layouts_Base_astro
site_src_pages_record_index_astro --> site_src_layouts_Base_astro
site_src_pages_activity__id__astro --> site_src_components_ActivityDetail_svelte
site_src_pages_activity__id__astro --> site_src_layouts_Base_astro
site_src_pages_activity_index_astro --> site_src_components_ActivityDetailLoader_svelte
site_src_pages_activity_index_astro --> site_src_layouts_Base_astro
site_src_pages_admin_index_astro --> site_src_layouts_Base_astro
site_src_pages_about_index_astro --> site_src_layouts_Base_astro
site_src_pages_feedback_index_astro --> site_src_layouts_Base_astro
site_src_pages_register_index_astro --> site_src_layouts_Base_astro
site_src_pages_reset_password_index_astro --> site_src_layouts_Base_astro
site_src_pages_community_index_astro --> site_src_components_CommunityView_svelte
site_src_pages_community_index_astro --> site_src_layouts_Base_astro
site_src_pages_invites_index_astro --> site_src_layouts_Base_astro
site_src_pages_login_index_astro --> site_src_layouts_Base_astro
site_src_pages_convert_index_astro --> site_src_layouts_Base_astro
site_src_pages_activity_local_index_astro --> site_src_components_LocalActivityDetail_svelte
site_src_pages_activity_local_index_astro --> site_src_layouts_Base_astro
site_src_pages_u__handle__index_astro --> site_src_components_ActivityFeed_svelte
site_src_pages_u__handle__index_astro --> site_src_layouts_Base_astro
site_src_pages_u__handle__athlete_index_astro --> site_src_components_AthleteView_svelte
site_src_pages_u__handle__athlete_index_astro --> site_src_layouts_Base_astro
site_src_pages_u__handle__stats_index_astro --> site_src_components_StatsView_svelte
site_src_pages_u__handle__stats_index_astro --> site_src_layouts_Base_astro
site_src_pages_about_it_index_astro --> site_src_layouts_Base_astro
site_src_pages_about_ca_index_astro --> site_src_layouts_Base_astro
site_src_pages_about_es_index_astro --> site_src_layouts_Base_astro
bincio_cli_py --> bincio_import__cli_py
bincio_cli_py --> bincio_extract_cli_py
bincio_cli_py --> bincio_dev_py
bincio_cli_py --> bincio_edit_cli_py
bincio_cli_py --> bincio_serve_cli_py
bincio_cli_py --> bincio_render_cli_py
bincio_cli_py --> bincio_serve_init_cmd_py
bincio_import__strava_py --> bincio_extract_models_py
bincio_import__strava_py --> bincio_extract_sport_py
bincio_edit_server_py --> bincio_edit_ops_py
bincio_extract_simplify_py --> bincio_extract_models_py
bincio_extract_metrics_py --> bincio_extract_models_py
bincio_extract_ingest_py --> bincio_extract_models_py
bincio_extract_strava_api_py --> bincio_extract_models_py
bincio_extract_strava_api_py --> bincio_extract_sport_py
bincio_extract_cli_py --> bincio_extract_parsers_factory_py
bincio_extract_cli_py --> bincio_extract_config_py
bincio_extract_cli_py --> bincio_extract_dedup_py
bincio_extract_writer_py --> bincio_extract_models_py
bincio_extract_writer_py --> bincio_extract_metrics_py
bincio_extract_writer_py --> bincio_extract_timeseries_py
bincio_extract_writer_py --> bincio_extract_simplify_py
bincio_extract_timeseries_py --> bincio_extract_models_py
bincio_serve_server_py --> bincio_edit_ops_py
bincio_serve_server_py --> bincio_serve_db_py
bincio_extract_parsers_tcx_py --> bincio_extract_models_py
bincio_extract_parsers_tcx_py --> bincio_extract_sport_py
bincio_extract_parsers_fit_py --> bincio_extract_models_py
bincio_extract_parsers_fit_py --> bincio_extract_sport_py
bincio_extract_parsers_gpx_py --> bincio_extract_models_py
bincio_extract_parsers_gpx_py --> bincio_extract_parsers_base_py
bincio_extract_parsers_gpx_py --> bincio_extract_sport_py
bincio_extract_parsers_factory_py --> bincio_extract_models_py
bincio_extract_parsers_factory_py --> bincio_extract_parsers_gpx_py
bincio_extract_parsers_factory_py --> bincio_extract_parsers_base_py
bincio_extract_parsers_factory_py --> bincio_extract_parsers_fit_py
bincio_extract_parsers_factory_py --> bincio_extract_parsers_tcx_py
bincio_extract_parsers_base_py --> bincio_extract_models_py
+1358
View File
File diff suppressed because it is too large Load Diff
+532
View File
@@ -0,0 +1,532 @@
#!/usr/bin/env python3
"""Generate architecture graphs for the bincio codebase.
Outputs:
docs/architecture.mmd — Mermaid source (embeddable in markdown / GitHub)
docs/graph.html — interactive vis.js graph (open in a browser)
Usage:
uv run python scripts/gen_graph.py
# or just:
python scripts/gen_graph.py
"""
import json
import re
from pathlib import Path
ROOT = Path(__file__).parent.parent
SITE_SRC = ROOT / "site" / "src"
DOCS = ROOT / "docs"
DOCS.mkdir(exist_ok=True)
# ── helpers ───────────────────────────────────────────────────────────────────
def read(path: Path) -> str:
try:
return path.read_text(encoding="utf-8")
except Exception:
return ""
def short(path: Path, base: Path) -> str:
"""Return a short display label for a file path."""
try:
rel = path.relative_to(base)
except ValueError:
rel = path
parts = rel.parts
# Drop leading site/src/ or bincio/
if parts[:2] == ("site", "src"):
parts = parts[2:]
elif parts[:1] == ("bincio",):
parts = parts[1:]
name = "/".join(parts)
# Strip index.astro → parent dir
if name.endswith("/index.astro"):
name = name[: -len("/index.astro")] + "/"
return name
# ── 1. API routes from server.py ──────────────────────────────────────────────
def extract_routes(server_path: Path) -> list[dict]:
"""Parse @app.{method}("/api/...") decorators."""
text = read(server_path)
routes = []
for m in re.finditer(
r'@app\.(get|post|put|patch|delete)\("(/api/[^"]+)"',
text,
re.MULTILINE,
):
method, path = m.group(1).upper(), m.group(2)
# Find the function name on the next non-blank line
tail = text[m.end():]
fn_m = re.search(r"async def (\w+)", tail[:200])
fn = fn_m.group(1) if fn_m else "?"
routes.append({"method": method, "path": path, "fn": fn})
return routes
# ── 2. Frontend → API edges ───────────────────────────────────────────────────
_FETCH_RE = re.compile(r"""fetch\(\s*[`'"](/api/[^`'"]+)[`'"]""")
_INTERP_RE = re.compile(r"""`[^`]*/api/([^`$\s{]+)""") # template literals
def extract_api_calls(file_path: Path) -> list[str]:
"""Return all /api/... paths referenced by a frontend file."""
text = read(file_path)
found = []
for m in _FETCH_RE.finditer(text):
found.append(m.group(1).split("?")[0]) # strip query string
# Template literals: `/api/admin/users/${h}/rebuild` → /api/admin/users/{h}/rebuild
for m in _INTERP_RE.finditer(text):
raw = "/api/" + m.group(1)
normalised = re.sub(r"\$\{[^}]+\}", "{x}", raw)
found.append(normalised)
return found
def normalise_route(path: str, routes: list[dict]) -> str | None:
"""Match a raw path like /api/admin/users/brut/rebuild to a known route pattern."""
for r in routes:
pattern = re.sub(r"\{[^}]+\}", r"[^/]+", re.escape(r["path"])) + "$"
if re.match(pattern, path):
return r["path"]
return path # keep as-is if not matched
# ── 3. Component imports (Svelte / Astro) ─────────────────────────────────────
_IMPORT_SVELTE_RE = re.compile(
r"""import\s+\w+\s+from\s+['"]([^'"]+\.svelte)['"]"""
)
_IMPORT_ASTRO_RE = re.compile(
r"""import\s+\w+\s+from\s+['"]([^'"]+\.astro)['"]"""
)
def extract_component_imports(file_path: Path) -> list[Path]:
text = read(file_path)
results = []
for pattern in (_IMPORT_SVELTE_RE, _IMPORT_ASTRO_RE):
for m in pattern.finditer(text):
ref = m.group(1)
target = (file_path.parent / ref).resolve()
if target.exists():
results.append(target)
return results
# ── 4. Python module imports ──────────────────────────────────────────────────
_PY_FROM_RE = re.compile(r"^from (bincio\.\S+) import", re.MULTILINE)
_PY_IMP_RE = re.compile(r"^import (bincio\.\S+)", re.MULTILINE)
def extract_py_imports(file_path: Path, py_files: list[Path]) -> list[Path]:
text = read(file_path)
modules = set()
for m in _PY_FROM_RE.finditer(text):
modules.add(m.group(1))
for m in _PY_IMP_RE.finditer(text):
modules.add(m.group(1))
results = []
for mod in modules:
# bincio.serve.db → bincio/serve/db.py
candidate = ROOT / Path(*mod.split(".")).with_suffix(".py")
if candidate.exists() and candidate != file_path:
results.append(candidate)
return results
# ── 5. Collect all data ───────────────────────────────────────────────────────
def collect() -> dict:
server_path = ROOT / "bincio" / "serve" / "server.py"
routes = extract_routes(server_path)
# Frontend files
fe_files = list(SITE_SRC.rglob("*.svelte")) + list(SITE_SRC.rglob("*.astro"))
# Python files (bincio package only)
py_files = [
p for p in (ROOT / "bincio").rglob("*.py")
if "__pycache__" not in str(p) and p.name != "__init__.py"
]
# --- edges: page/component → API endpoint
api_edges = [] # (source_file, route_path)
for f in fe_files:
calls = extract_api_calls(f)
for call in calls:
norm = normalise_route(call, routes)
api_edges.append((f, norm))
# --- edges: component imports
comp_edges = [] # (importer_file, imported_file)
for f in fe_files:
for dep in extract_component_imports(f):
comp_edges.append((f, dep))
# --- edges: python imports
py_edges = [] # (importer_file, imported_file)
for f in py_files:
for dep in extract_py_imports(f, py_files):
py_edges.append((f, dep))
return {
"routes": routes,
"fe_files": fe_files,
"py_files": py_files,
"api_edges": api_edges,
"comp_edges": comp_edges,
"py_edges": py_edges,
}
# ── 6. Mermaid output ─────────────────────────────────────────────────────────
def to_node_id(path: Path) -> str:
return re.sub(r"[^a-zA-Z0-9]", "_", str(path.relative_to(ROOT)))
def write_mermaid(data: dict) -> Path:
lines = ["graph LR", ""]
routes = data["routes"]
# Subgraph: API endpoints grouped by domain
domains: dict[str, list[dict]] = {}
for r in routes:
parts = r["path"].strip("/").split("/")
domain = parts[1] if len(parts) > 1 else "other"
domains.setdefault(domain, []).append(r)
lines.append(" subgraph API")
for domain, rs in sorted(domains.items()):
lines.append(f" subgraph api_{domain}[\"{domain}\"]")
for r in rs:
nid = "api_" + re.sub(r"[^a-zA-Z0-9]", "_", r["path"])
lines.append(f' {nid}["{r["method"]} {r["path"]}"]')
lines.append(" end")
lines.append(" end")
lines.append("")
# Subgraph: pages
pages = [f for f in data["fe_files"] if "/pages/" in str(f)]
lines.append(" subgraph Pages")
for f in sorted(pages):
nid = to_node_id(f)
label = short(f, ROOT)
lines.append(f' {nid}["{label}"]')
lines.append(" end")
lines.append("")
# Subgraph: components
comps = [f for f in data["fe_files"] if "/components/" in str(f)]
lines.append(" subgraph Components")
for f in sorted(comps):
nid = to_node_id(f)
label = short(f, ROOT)
lines.append(f' {nid}["{label}"]')
lines.append(" end")
lines.append("")
# Subgraph: Python modules
py_groups: dict[str, list[Path]] = {}
for f in data["py_files"]:
rel = f.relative_to(ROOT / "bincio")
group = rel.parts[0] if len(rel.parts) > 1 else "root"
py_groups.setdefault(group, []).append(f)
lines.append(" subgraph Python")
for group, files in sorted(py_groups.items()):
lines.append(f' subgraph py_{group}["{group}"]')
for f in sorted(files):
nid = to_node_id(f)
lines.append(f' {nid}["{f.stem}"]')
lines.append(" end")
lines.append(" end")
lines.append("")
# Edges: page/component → API
seen = set()
for src, route_path in data["api_edges"]:
src_nid = to_node_id(src)
dst_nid = "api_" + re.sub(r"[^a-zA-Z0-9]", "_", route_path)
edge = f" {src_nid} -->|fetch| {dst_nid}"
if edge not in seen:
lines.append(edge)
seen.add(edge)
# Edges: component imports
seen_comp = set()
for src, dst in data["comp_edges"]:
src_nid = to_node_id(src)
dst_nid = to_node_id(dst)
edge = f" {src_nid} --> {dst_nid}"
if edge not in seen_comp:
lines.append(edge)
seen_comp.add(edge)
# Edges: python imports
seen_py = set()
for src, dst in data["py_edges"]:
src_nid = to_node_id(src)
dst_nid = to_node_id(dst)
edge = f" {src_nid} --> {dst_nid}"
if edge not in seen_py:
lines.append(edge)
seen_py.add(edge)
out = DOCS / "architecture.mmd"
out.write_text("\n".join(lines), encoding="utf-8")
return out
# ── 7. vis.js HTML output ─────────────────────────────────────────────────────
def write_visjs(data: dict) -> Path:
nodes: list[dict] = []
edges: list[dict] = []
node_ids: dict[str, int] = {}
def add_node(key: str, label: str, group: str, title: str = "") -> int:
if key in node_ids:
return node_ids[key]
nid = len(nodes)
node_ids[key] = nid
nodes.append({"id": nid, "label": label, "group": group, "title": title or label})
return nid
def add_edge(src_key: str, dst_key: str, label: str = "") -> None:
if src_key not in node_ids or dst_key not in node_ids:
return
e: dict = {"from": node_ids[src_key], "to": node_ids[dst_key], "arrows": "to"}
if label:
e["label"] = label
edges.append(e)
# API endpoint nodes
for r in data["routes"]:
key = f"api:{r['path']}"
label = f"{r['method']}\n{r['path']}"
add_node(key, label, "api", f"{r['method']} {r['path']}{r['fn']}()")
# Frontend file nodes
for f in data["fe_files"]:
key = str(f)
label = f.name.replace("/index.astro", "/").replace("index.astro", f.parent.name + "/")
is_page = "/pages/" in str(f)
is_layout = "/layouts/" in str(f)
group = "page" if is_page else ("layout" if is_layout else "component")
title = short(f, ROOT)
add_node(key, label, group, title)
# Python module nodes
for f in data["py_files"]:
key = str(f)
rel = f.relative_to(ROOT / "bincio")
group = "py_" + rel.parts[0] if len(rel.parts) > 1 else "py_root"
add_node(key, f.stem, group, str(f.relative_to(ROOT)))
# Edges: page/component → API
seen = set()
for src, route_path in data["api_edges"]:
src_key = str(src)
dst_key = f"api:{route_path}"
k = (src_key, dst_key)
if k not in seen:
seen.add(k)
add_edge(src_key, dst_key, "fetch")
# Edges: component imports
seen_comp = set()
for src, dst in data["comp_edges"]:
k = (str(src), str(dst))
if k not in seen_comp:
seen_comp.add(k)
add_edge(str(src), str(dst))
# Edges: python imports
seen_py = set()
for src, dst in data["py_edges"]:
k = (str(src), str(dst))
if k not in seen_py:
seen_py.add(k)
add_edge(str(src), str(dst))
# Group colours for legend
groups = {
"api": {"color": {"background": "#f59e0b", "border": "#d97706"}, "font": {"color": "#000"}},
"page": {"color": {"background": "#3b82f6", "border": "#2563eb"}, "font": {"color": "#fff"}},
"component": {"color": {"background": "#8b5cf6", "border": "#7c3aed"}, "font": {"color": "#fff"}},
"layout": {"color": {"background": "#06b6d4", "border": "#0891b2"}, "font": {"color": "#000"}},
"py_extract": {"color": {"background": "#22c55e", "border": "#16a34a"}, "font": {"color": "#000"}},
"py_render": {"color": {"background": "#84cc16", "border": "#65a30d"}, "font": {"color": "#000"}},
"py_serve": {"color": {"background": "#ef4444", "border": "#dc2626"}, "font": {"color": "#fff"}},
"py_edit": {"color": {"background": "#f97316", "border": "#ea580c"}, "font": {"color": "#fff"}},
"py_root": {"color": {"background": "#6b7280", "border": "#4b5563"}, "font": {"color": "#fff"}},
}
html = f"""<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<title>Bincio — architecture graph</title>
<script src="https://unpkg.com/vis-network@9.1.9/standalone/umd/vis-network.min.js"></script>
<style>
* {{ box-sizing: border-box; margin: 0; padding: 0; }}
body {{ background: #0f172a; color: #e2e8f0; font-family: system-ui, sans-serif; height: 100vh; display: flex; flex-direction: column; }}
#toolbar {{ display: flex; align-items: center; gap: 12px; padding: 10px 16px; background: #1e293b; border-bottom: 1px solid #334155; flex-shrink: 0; flex-wrap: wrap; }}
#toolbar h1 {{ font-size: 14px; font-weight: 600; color: #94a3b8; margin-right: 8px; }}
.filter-group {{ display: flex; gap: 6px; }}
.filter-group label {{ display: flex; align-items: center; gap: 4px; font-size: 12px; cursor: pointer; padding: 3px 8px; border-radius: 4px; border: 1px solid #334155; }}
.filter-group label:hover {{ background: #334155; }}
.dot {{ width: 10px; height: 10px; border-radius: 50%; display: inline-block; }}
#search {{ background: #0f172a; border: 1px solid #334155; color: #e2e8f0; padding: 4px 10px; border-radius: 6px; font-size: 12px; width: 180px; }}
#search::placeholder {{ color: #475569; }}
#info {{ margin-left: auto; font-size: 11px; color: #64748b; }}
#graph {{ flex: 1; }}
#tooltip {{ position: fixed; background: #1e293b; border: 1px solid #334155; border-radius: 6px; padding: 8px 12px; font-size: 12px; color: #e2e8f0; pointer-events: none; display: none; max-width: 320px; z-index: 100; }}
</style>
</head>
<body>
<div id="toolbar">
<h1>Bincio architecture</h1>
<div class="filter-group">
<label><input type="checkbox" data-group="api" checked> <span class="dot" style="background:#f59e0b"></span> API endpoints</label>
<label><input type="checkbox" data-group="page" checked> <span class="dot" style="background:#3b82f6"></span> Pages</label>
<label><input type="checkbox" data-group="component" checked> <span class="dot" style="background:#8b5cf6"></span> Components</label>
<label><input type="checkbox" data-group="layout" checked> <span class="dot" style="background:#06b6d4"></span> Layouts</label>
<label><input type="checkbox" data-group="py_extract" checked> <span class="dot" style="background:#22c55e"></span> extract</label>
<label><input type="checkbox" data-group="py_render" checked> <span class="dot" style="background:#84cc16"></span> render</label>
<label><input type="checkbox" data-group="py_serve" checked> <span class="dot" style="background:#ef4444"></span> serve</label>
<label><input type="checkbox" data-group="py_edit" checked> <span class="dot" style="background:#f97316"></span> edit</label>
</div>
<input id="search" type="text" placeholder="Search nodes…" />
<span id="info"></span>
</div>
<div id="graph"></div>
<div id="tooltip"></div>
<script>
const allNodes = {json.dumps(nodes, indent=2)};
const allEdges = {json.dumps(edges, indent=2)};
const groups = {json.dumps(groups, indent=2)};
const nodesDS = new vis.DataSet(allNodes);
const edgesDS = new vis.DataSet(allEdges);
const container = document.getElementById('graph');
const options = {{
nodes: {{
shape: 'box',
borderWidth: 1,
font: {{ size: 11, face: 'monospace' }},
margin: 6,
}},
edges: {{
smooth: {{ type: 'continuous' }},
color: {{ color: '#334155', highlight: '#60a5fa' }},
font: {{ size: 10, color: '#64748b', align: 'middle' }},
width: 1,
selectionWidth: 2,
}},
groups,
physics: {{
solver: 'forceAtlas2Based',
forceAtlas2Based: {{ gravitationalConstant: -40, springLength: 120 }},
stabilization: {{ iterations: 200 }},
}},
interaction: {{
hover: true,
tooltipDelay: 100,
navigationButtons: true,
keyboard: true,
}},
}};
const network = new vis.Network(container, {{ nodes: nodesDS, edges: edgesDS }}, options);
// Info count
document.getElementById('info').textContent =
`${{allNodes.length}} nodes · ${{allEdges.length}} edges`;
// Tooltip on hover
const tooltip = document.getElementById('tooltip');
network.on('hoverNode', params => {{
const node = nodesDS.get(params.node);
tooltip.textContent = node.title || node.label;
tooltip.style.display = 'block';
}});
network.on('blurNode', () => {{ tooltip.style.display = 'none'; }});
document.addEventListener('mousemove', e => {{
tooltip.style.left = (e.clientX + 14) + 'px';
tooltip.style.top = (e.clientY + 14) + 'px';
}});
// Highlight connected nodes on click
network.on('click', params => {{
if (!params.nodes.length) {{ network.unselectAll(); return; }}
const nid = params.nodes[0];
const connected = network.getConnectedNodes(nid);
network.selectNodes([nid, ...connected]);
}});
// Group visibility toggle
document.querySelectorAll('[data-group]').forEach(cb => {{
cb.addEventListener('change', () => {{
const group = cb.dataset.group;
const hidden = !cb.checked;
const toUpdate = allNodes
.filter(n => n.group === group)
.map(n => ({{ id: n.id, hidden }}));
nodesDS.update(toUpdate);
}});
}});
// Search / highlight
document.getElementById('search').addEventListener('input', e => {{
const q = e.target.value.trim().toLowerCase();
if (!q) {{ nodesDS.update(allNodes.map(n => ({{ id: n.id, opacity: 1 }}))); return; }}
const updates = allNodes.map(n => {{
const match = (n.label + n.title).toLowerCase().includes(q);
return {{ id: n.id, opacity: match ? 1 : 0.15 }};
}});
nodesDS.update(updates);
}});
</script>
</body>
</html>
"""
out = DOCS / "graph.html"
out.write_text(html, encoding="utf-8")
return out
# ── main ──────────────────────────────────────────────────────────────────────
if __name__ == "__main__":
print("Collecting codebase graph data…")
data = collect()
r = len(data["routes"])
f = len(data["fe_files"])
p = len(data["py_files"])
ae = len(data["api_edges"])
ce = len(data["comp_edges"])
pe = len(data["py_edges"])
print(f" {r} API routes | {f} frontend files | {p} Python modules")
print(f" {ae} API call edges | {ce} component import edges | {pe} Python import edges")
mmd = write_mermaid(data)
print(f"\nMermaid → {mmd.relative_to(ROOT)}")
html = write_visjs(data)
print(f"vis.js → {html.relative_to(ROOT)}")
print("\nOpen docs/graph.html in a browser to explore interactively.")