Track Jetson-PI multi-backend provider integration

[heiheiha798]

Purpose

This issue tracks review of the FlashRT multi-backend model-provider integration corresponding to Jetson-PI.

• FlashRT implementation branch: https://github.com/heiheiha798/FlashRT/tree/jetson-pi-link-check
• FlashRT branch tip: 28c3adf6c152609cb9379572b9916a53368b0420
• Companion Jetson-PI branch: flashrt-migration-merge in the private Jetson-PI repository
• Companion private Jetson-PI PR: https://github.com/diantoudedianshan/Jetson-PI/pull/1
• Jetson-PI reviewer/owner: @diantoudedianshan

The FlashRT branch is hosted on the contributor fork because the current contributor account has read-only access to the official FlashRT repository.

Scope

The branch introduces Jetson-PI as a provider-owned llama.cpp/GGML model provider while preserving the existing FlashRT CUDA Graph path. It includes Pi0/Pi0.5, text LLM, and multimodal LLM paths; staged runtime ports; repeatable decode; memory-domain and read-only DLPack integration; production CMake packaging; and tested CPU/CUDA/Vulkan/SYCL/OpenCL capability behavior.

Review status

Development and available-backend validation are complete. This work is pending review by @diantoudedianshan. The Jetson-PI owner will decide how the private implementation should be opened and integrated; this issue only tracks the completed development branch and review handoff.

The companion Jetson-PI PR is private and is accessible only to authorized repository members.

[LiangSu8899]

Super COOL!!!

[LiangSu8899]

@LiangSu8899 Verify interface completeness on branch after integration
#144
LiangSu8899/FlashRT-Nexus#12

[heiheiha798]

Update: the FlashRT-side integration is now up as a public PR — #148 (feat/jetson-pi-provider, rebased on origin/main).

Backend reference moved from the original private fork to the public repository PKU-SEC-Lab/Jetson-PI-Edge (Jetson-PI-flashrt branch). The switch is zero code changes on the FlashRT side — integration is path-parameterized via JETSON_PI_ROOT / JetsonPI_ROOT, no hardcoded git URLs.

Local CPU-only validation (conda gcc 13 toolchain): ctest -R "jetson_pi|llama_cpp" 8/8 passed; provider_c.so dlopen + dlsym resolves the FlashRT entry symbols and all Jetson-PI runtime deps; the Python frontend reaches jetson_pi_llm_open. GPU records (RTX 4090, CUDA/Vulkan/SYCL) are in docs/jetson_pi_validation_matrix.md.

This issue can be closed once #148 lands. Keeping it open for now so the review trail stays visible.

Paper: "Jetson-PI: Towards Onboard Real-Time Robot Control via Foresight-Aligned Asynchronous Inference" — https://arxiv.org/abs/2607.12659

[LiangSu8899]

Quick follow-up on this integration: after reviewing #148 against the runtime boundary that has since landed, I reworked the FlashRT side on a new branch and opened draft PR #157:

• Refactor Jetson-PI provider onto model runtime v1 #157
• branch: provider/model-runtime-v1-generic

The main reason for the rework is to avoid keeping a parallel model-runtime v2 now that frt_model_runtime_v1 has a generic selected-plan extension. The Jetson-PI implementation and backend ownership stay intact, but Pi0/LLM/MLLM now return the same v1 runtime used by other producers. Provider stages remain opaque, while existing CUDA Graph producers and the common runtime/kernel paths stay unchanged.

This also makes the backend boundary clearer: FlashRT publishes ports, identity, lifetime, and the selected stage plan; Jetson-PI continues to own GGML/llama.cpp execution and state. The matching generic adopter is being developed separately in Nexus, while the existing graph path is preserved:

• feat(model): adopt generic runtime stage plans LiangSu8899/FlashRT-Nexus#14

I kept #148 open so the original contributors can review the new boundary and confirm that the provider semantics were preserved. The detailed structure, build isolation, packaging, and validation checklist are all in #157, so I will keep this issue as the higher-level tracking thread.

[LiangSu8899]

and gonna test blackwell int4: #146