Big Brain

Big Brain is a local-first AI powered document analysis platform. It allows users to index, analyze and query large sets of documents using natural language, while keeping full control over their data.

The project is designed as a modular product composed of a desktop application, an AI engine, and optional online services for authentication and subscriptions.

This repository is a monorepo that contains all components required to build, test and distribute the product.

Main features

• Local document ingestion and indexing
• Natural language querying
• Semantic analysis and summarization
• Offline-first desktop application
• Optional account and subscription system
• Windows executable distribution
• Full test coverage and quality tracking

Repository structure

• apps: End user applications and services
• ai: Core AI and document processing logic
• shared: Shared utilities and cross cutting concerns
• packaging: Build and distribution tooling
• tests: Integration, end to end and performance tests
• docs: Architecture, project management and quality documentation
• scripts: Developer tooling and automation

Each directory contains its own README with detailed explanations.

Tech stack

• Python for backend and AI
• Local database and search index
• Web based or embedded desktop frontend
• PyInstaller for Windows packaging

Development principles

• Security by design
• Local data ownership
• Clear separation of responsibilities
• Test driven development
• Industrial grade documentation

Getting started

Refer to the README files inside each directory to understand setup, development and testing workflows.

Development - Run locally

A quick developer workflow to run the desktop app and index documents locally.

Prerequisites

• Install Node.js and pnpm for the frontend.
• Install Python (3.10+) and create/activate a virtual environment or use conda; then install backend deps with pip install -r requirements/backend.txt.
• Install Ollama and make sure the ollama CLI is available in PATH.
• (Optional) Tauri prerequisites if building the desktop app.

Start services (order matters)

1. Start Ollama (LLM runtime):

   • ollama serve

2. Start the backend API (from apps/desktop/backend with the Python env active):

   • uvicorn app.main:app --host 127.0.0.1 --port 8000

3. Start the desktop frontend (from apps/desktop/frontend/apps/desktop):

   • pnpm install (first time)
   • pnpm tauri:dev

Index documents - Watched Folders (recommended)

The preferred way to index documents is through the watched folders system. Folders are managed via the API or the "Files" tab in the desktop app.

Add a folder to watch:

POST http://127.0.0.1:8000/folders
{"folder_path": "C:/path/to/your/docs", "recursive": true, "exclude_patterns": []}

Trigger a scan (returns a job_id):

POST http://127.0.0.1:8000/folders/{folder_id}/scan

Or use the CLI from the repository root:

python ai/indexing/create_embedding.py --scan-watched

You can also scan a single folder by id:

python ai/indexing/create_embedding.py --scan-folder-id 1

How it works:

• The scanner walks the folder recursively (respecting exclude patterns).
• Files are hashed (SHA-256). Unchanged files are skipped on subsequent scans.
• Moved/renamed files are detected by hash and their index entry is updated.
• Files that disappear are automatically deindexed from Qdrant.
• The folder registry is persisted in data/watched_folders.db (SQLite).

Index documents - Legacy (one-off)

To quickly ingest documents without the folder system:

• Put files into ai/data.
• Run from the repository root:

  python ai/indexing/create_embedding.py --data-dir ai/data
  

Notes

• Backend requirements live in requirements/backend.txt.
• Backend must be running at http://127.0.0.1:8000 for the frontend to connect.
• Qdrant must be running at http://localhost:6333 for indexing and querying.
• The watched-folder SQLite DB (data/watched_folders.db) is created automatically on first run.
• The vector index lives in Qdrant; the folder/document metadata lives in SQLite.

---

For component-specific setup and deeper details, see the README files inside each subfolder.