Search video by what was said, what appeared on screen, and recurring faces.
VidXP is a local-first video indexing and search engine distributed as a Python package. Use it from the command line, through its browser interface, or as an indexing and retrieval layer inside another application.
Finding one moment in a video should not require scrubbing through the entire timeline. VidXP builds a searchable index from three kinds of evidence:
- Dialogue: semantic search over timestamped WhisperX transcripts.
- Scenes: text-to-frame search using CLIP.
- Actors: groups similar detected faces and exports a highlighted video for a selected cluster.
After the required model weights are available, video processing and search run locally. VidXP also saves index state so an incomplete run is not mistaken for a searchable result.
| Capability | Available now | Result |
|---|---|---|
| Dialogue search | Transcription, word alignment, semantic phrase indexing | Matching video time |
| Scene search | Text search over sampled video frames | Matching frame and time |
| Actor grouping | Within-video face detection and clustering | Clustered detections and highlighted output video |
| Interfaces | Typer CLI, Streamlit browser interface, Python API | Interactive or programmatic use |
| Index management | Saved progress, ready/failed state, cancellation, isolated programmatic runs | Traceable and reusable indexes |
VidXP supports Python 3.10 through 3.13 and requires FFmpeg. See the
installation guide for the dlib compiler
requirements, source installation, model preparation, and troubleshooting.
Create and activate a virtual environment:
python -m venv venv# Windows
venv\Scripts\activate
# macOS/Linux
source venv/bin/activateInstall the command-line package:
python -m pip install vidxpInclude the browser interface:
python -m pip install "vidxp[frontend]"Confirm the installed package and its runtime dependencies:
vidxp --version
vidxp doctorThe first use of each capability downloads its model weights. Download the fixed dialogue, transcription, and scene models in advance with:
vidxp prepareBuild an index containing dialogue, scene, and actor information:
vidxp videoindex samplevideo.mp4Search the completed index:
vidxp dialogue "the bread just came out of the oven"
vidxp scene "a yellow taxi on a city street"
vidxp actor 1 samplevideo.mp4Index only selected capabilities or sample fewer visual frames:
vidxp videoindex samplevideo.mp4 --modalities scene --frame-stride 5--modalities accepts any combination of dialogue, scene, and actor.
Run vidxp --help or any command followed by --help for the complete command
reference.
Install the frontend extra and start:
vidxp-uiThe command starts a local Streamlit server and remains active until stopped. The interface can upload a video, start or cancel indexing, restore saved progress after a page reload, and search the capabilities available in the completed index.
The programmatic API supports isolated multi-video runs, supplied timestamped transcripts, resumable per-video checkpoints, and metadata-rich top-k results.
from vidxp.core import IndexConfig, VideoSource
from vidxp.core.runner import run_index
from vidxp.core.search import search_scene
config = IndexConfig(
dataset="my-library",
split="local",
run_id="demo",
enabled_modalities=("scene",),
frame_stride=5,
)
run_index(
[
VideoSource(video_id="video-1", path="videos/first.mp4"),
VideoSource(video_id="video-2", path="videos/second.mp4"),
],
config,
)
results = search_scene("a person enters a taxi", config=config, top_k=5)
for hit in results.hits:
print(hit.video_id, hit.start, hit.end, hit.score)The Python indexing and retrieval contract documents configuration, stored metadata, result fields, and run layout.
- The standard CLI and browser interface manage one local searchable video index at a time. The Python layer supports isolated multi-video runs.
- Actor clusters represent visually similar detected faces; VidXP does not automatically know or assign a person's name.
- Model weights are cached separately from the Python package and require additional disk space.
- CPU is the current default. Visual indexing processes every frame unless
--frame-strideis configured. - Search quality depends on the selected models, video domain, speech quality, and frame sampling.
VidXP is an evolving beta. The roadmap extends the current engine rather than replacing its working search paths.
| Area | Current foundation | Direction |
|---|---|---|
| Search results | Top result in the CLI; structured top-k Python results | Rich ranked results, metadata, previews, and filtering across interfaces |
| Temporal search | Frame and transcript-phrase timestamps | Better time ranges, scene boundaries, aggregation, and ranking |
| Video collections | One local CLI/UI index; isolated multi-video Python runs | User-facing persistent multi-video libraries and index management |
| Actor workflows | Face clustering and highlighted video export | Cluster browsing, labeling, actor search, and stronger tracking |
| Speaker context | Timestamped dialogue search | Active-speaker detection and links between speech and visible people |
| Product experience | CLI and browser indexing/search | Clearer progress, result navigation, recovery, and long-running job controls |
| Evaluation | DiDeMo and HiREST baselines | Combined and component benchmarks, beginning with a LongVALE pilot |
Roadmap items describe intended direction, not a release guarantee.
| Capability | Model |
|---|---|
| Dialogue embeddings | sentence-transformers/all-MiniLM-L6-v2 |
| Transcription | WhisperX large-v2 |
| Scene search | CLIP ViT-B/32 |
| Word alignment | WhisperX model selected for the detected language |
VidXP maintains the standard local CLI/UI index in chroma_data/. Starting a
new local indexing run replaces the previous or incomplete local index. Model
caches normally live outside this directory and outside the virtual environment.