(Agent Banana) — Agentic Image Editing

An agentic image-editing pipeline that performs high-fidelity, localized edits on photos from natural-language instructions — grounding the target region, editing only that region, and blending it back in with no visible seams.

How it works

Edits follow an Image Layer Decomposition (ILD) process: locate → isolate → edit → recompose, wrapped in a ReAct-style Thought → Action → Observation agent loop.

1. Plan — the instruction is parsed into one or more edits (add / remove / replace / restyle / adjust), each classified as local or global and sequenced by an RL-style planner that reasons over prior turns in the session.
2. Ground — for each edit, a Gemini-based grounding advisor reasons about where the target likely is, then Florence-2 performs pixel-level localization on the original image. Candidates are ranked and refined against the target's profile (face accessory, watermark, global scene, etc.).
3. Act (ReAct loop) — a Gemini-driven executor sees the current image and bounding box, and at each step decides which tool to call — expand crop, generate edit, blend, retry — chaining each attempt on the previous result rather than starting over.
4. Critic — a VLM critic (Gemini, Ollama, or Hugging Face-hosted vision model) compares the untouched original against the edited result and verifies the instruction was actually fulfilled.
5. Quality gate — a seam detector (boundary gradient discontinuity + cross-boundary color shift) and a quality judge check the blend; if the seam is visible, the edit retries with a wider taper before being accepted.

Sessions persist across turns, so follow-up instructions ("now make it darker") fold in prior context automatically.

Interfaces

• Web Studio — a self-contained chat-style UI (served by the built-in HTTP server) for uploading an image and issuing edits conversationally, with live streaming of each agent step.

• HTTP API:

  Method	Path	Description
  POST	/api/edit	Run a full edit, return the final result
  POST	/api/edit-stream	Same, streamed as SSE — one event per agent step
  POST	/api/ground	Manual mode step 1 — parse instruction, return grounded bounding boxes for user confirmation
  POST	/api/edit-manual-stream	Manual mode step 2 — run the edit using user-confirmed bboxes (SSE)
  POST	/api/recompose	Re-blend a preview crop back into the source image
  GET	/health	Liveness probe

• CLI — agent-banana-edit --image path.jpg --instruction "..." runs the pipeline on a local file and writes the full artifact bundle (source, per-step previews/overlays/edits, final image, JSON report) to disk.

Tech stack

• Gemini (gemini-3.1-flash-image-preview, aka "Nano Banana") — image generation/editing and the ReAct orchestrator/critic
• Florence-2 (florence-community/Florence-2-base, via transformers/torch) — open-vocabulary object grounding
• Pure-Python http.server — no web framework dependency, keeps the Space lightweight

Project structure

IEA/
├── app.py                        # Hugging Face Spaces entry point
├── src/agent_banana/
│   ├── server.py                  # HTTP server, SSE endpoints, embedded web Studio UI
│   ├── cli.py                     # agent-banana-edit CLI
│   ├── pipeline.py                 # AgentBananaApp — orchestrates the full ILD pipeline
│   ├── planning.py                 # instruction parsing, RL-style multi-edit planner
│   ├── targeting.py                 # target classification, fallback/refined bboxes
│   ├── vlm_localizer.py             # Florence-2 grounding wrapper
│   ├── llm_grounding_advisor.py     # Gemini spatial-reasoning advisor
│   ├── react_executor.py            # Thought → Action → Observation edit loop
│   ├── tool_registry.py             # named tools available to the ReAct executor
│   ├── nano_banana.py                # Gemini image-generation client
│   ├── vlm_critic.py                 # semantic verification of edit results
│   ├── quality.py / seam_detector.py # blend quality + seam detection
│   ├── memory.py                     # session store + context folding across turns
│   └── models.py / vision.py / config.py
├── tests/test_agent_banana.py
├── Dockerfile
└── pyproject.toml

Setup

git clone https://github.com/vignesh1507/IEA.git
cd IEA
pip install -r requirements.txt
pip install -e .

Set GEMINI_API_KEY in your environment (or .env file / Space secrets). Optional env vars:

GEMINI_API_KEY=
CRITIC_PROVIDER=gemini        # gemini | huggingface | ollama
HF_API_TOKEN=                 # if CRITIC_PROVIDER=huggingface
HF_CRITIC_MODEL=meta-llama/Llama-3.2-11B-Vision-Instruct
OLLAMA_CRITIC_MODEL=llama3.2-vision

Run the Studio server

agent-banana-serve --host 0.0.0.0 --port 7860

Run from the CLI

agent-banana-edit --image photo.jpg --instruction "remove the watermark in the corner"

Deployment

Ships as a Docker image sized for Hugging Face Spaces (sdk: docker, port 7860). Build and run directly:

docker build -t moleculyst .
docker run -p 7860:7860 -e GEMINI_API_KEY=your-key moleculyst

License

MIT