Skip to content

Repository files navigation

EDA Pipeline with Agentic Workflow

An intelligent, production-ready EDA (Exploratory Data Analysis) pipeline powered by LangGraph, Groq, and open-source tools. Features a complete suite of specialized agents with interactive visualizations, export capabilities, and human-in-the-loop interaction.

πŸ†• What's New in v3.3

πŸ“Š Excel File Support (NEW!)

  • βœ… Excel Upload: Support for .xlsx and .xls files in addition to CSV
  • βœ… Automatic Detection: First sheet is automatically loaded and analyzed
  • βœ… Seamless Integration: Excel files work with all agents and workflows
  • βœ… Same Performance: Intelligent backend selection (in-memory vs sampled) for Excel files

What's New in v3.2

🚦 Human-in-the-Loop Approval Gates (NEW!)

  • βœ… Agent Approval Gates: Review and approve each agent before proceeding to the next
  • βœ… 4 Decision Options: Approve, Retry, Skip, or Stop at each agent
  • βœ… Decision History: Track all approval decisions throughout workflow
  • βœ… 3 New Workflows: Quick Analysis, Deep Dive, and ML Prep with Approval Gates
  • βœ… Detailed Review UI: See confidence scores, issues found, and recommendations
  • βœ… Agent-Specific Details: Tailored detail views for each agent type
  • βœ… Thumbnail Gallery: Quick preview of visualization plots with expand-to-full-size option

πŸ‘‰ Quick Start Guide | Full Documentation

Why Approval Gates Show Summary, Not Full Results?

The approval gate design follows a two-phase workflow for optimal user experience:

Phase 1: Approval Gate (Quick Review)

  • Purpose: Fast decision-making - approve, retry, skip, or stop
  • Shows: Summary metrics, key findings, reasoning, and thumbnail previews
  • Why compact?
    • ⚑ Speed: Quick review enables fast decisions (30-60 seconds per agent)
    • πŸ“ Size: With 8+ plots and detailed tables, gates would become unwieldy
    • 🎯 Focus: Summary view keeps attention on decision-relevant information
    • πŸ”„ Workflow: Detailed analysis happens after approval, not during

Phase 2: Results Tab (Deep Analysis)

  • Purpose: Comprehensive exploration of agent findings
  • Shows: All plots in full size, complete tables, interactive visualizations, export options
  • When: After workflow completes or individual agent runs
  • Where: Navigate to tabbed interface (Profile, Quality, Visualizations, etc.)

This separation ensures approval gates remain fast decision points while full results provide thorough analysis capabilities.

Enhanced Transformation System (v3.1)

  • βœ… Multi-Transformation Selection: Select and apply multiple transformations at once with checkboxes
  • βœ… Complete CSV Export: Apply transformations to full dataset and export as CSV
  • βœ… Column Change Visualization: See exactly which columns are added/removed during transformations
  • βœ… One-Hot Encoding Preview: Visual mapping showing how categorical columns transform to binary columns
  • βœ… Progress Tracking: Real-time progress bar when applying transformations to large datasets
  • βœ… Quick Actions: "Select All High Priority" for instant data cleaning

Fixes & Improvements

  • πŸ› Fixed transformation preview showing same results in before/after
  • πŸ› Fixed DatasetHandle backend access for large datasets
  • πŸ› Fixed CSV export not working
  • πŸ”§ Added support for all 7 transformation types (encoding, scaling, imputation, etc.)
  • πŸ”§ Enhanced error reporting with detailed tracebacks
  • πŸ”§ Smart column comparison (only shows common columns to prevent errors)

Features

Core Capabilities

  • 7 Specialized Agents: ProfileAgent, QualityAgent, TransformAgent, VisualizationAgent, FeatureAgent, StatAgent, TimeSeriesAgent
  • Interactive Chat Interface: Real-time Streamlit UI with progress tracking
  • Explainable AI: Every agent explains WHY and WHAT impact their findings have
  • Persistent State: LangGraph checkpoints for pause/resume workflows
  • Hybrid Scale: Handles small datasets in-memory, large datasets with intelligent sampling
  • Flexible Workflows: Pre-defined pipelines + individual agent execution

Advanced Features

  • Human-in-the-Loop Control: Agent approval gates for reviewing and approving each step
  • Progress Tracking: Real-time visual workflow progress with status indicators and ETA
  • Quality Visualizations: Interactive charts for missing values, outliers, duplicates, and data quality metrics
  • Before/After Comparison: Side-by-side transformation previews with delta metrics and impact analysis
  • Multi-Transformation Selection: Select and apply multiple transformations at once with checkboxes
  • Column Change Visualization: See exactly which columns are added/removed during transformations
  • Export System: Professional HTML reports, JSON data exports, and transformed CSV outputs with full dataset support
  • ML Preparation: Automated feature engineering and data preparation for machine learning
  • Decision Tracking: Complete audit trail of human approval decisions

Architecture

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Streamlit UI   β”‚  ← Interactive interface with progress tracking
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
         β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  LangGraph      β”‚  ← State machine orchestration
β”‚  Orchestration  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜
         β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚                      Specialized Agents (7)                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Profile  β”‚ Quality  β”‚Transform β”‚Visualize β”‚ Feature  β”‚ Stat β”‚ Time  β”‚
β”‚  Agent   β”‚  Agent   β”‚  Agent   β”‚  Agent   β”‚  Agent   β”‚Agent β”‚Series β”‚
β”‚          β”‚          β”‚          β”‚          β”‚          β”‚      β”‚Agent  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”˜
         β”‚
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β–Όβ”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Data Layer    β”‚  ← Pandas/DuckDB with intelligent switching
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

LangGraph Workflow Orchestration

The EDA Pipeline uses LangGraph for state machine-based workflow orchestration, enabling:

  • Stateful execution with checkpoint/resume capabilities
  • Human-in-the-loop approval gates with conditional routing
  • Context propagation between agents
  • Explainable decisions with reasoning logs

State Management

The workflow maintains a comprehensive EDAState that tracks:

EDAState:
  # Dataset metadata
  - dataset_path, dataset_mode, dataset_size
  - dataset_rows, dataset_cols
  
  # Workflow control
  - workflow_type: "quick_profile" | "deep_clean" | "feature_engineering"
  - current_step, completed_steps
  - pending_approval, approval_context
  
  # Analysis results
  - profile_results, quality_results
  - feature_results, stat_results
  - visualizations[]
  
  # Explainability
  - reasoning_log[] (timestamp, agent, reasoning, impact, confidence)
  
  # Human interaction
  - user_decisions[] (approved/rejected/modified)
  - user_messages[]
  
  # Transformations
  - pending_transformations[]
  - applied_transformations[]

Workflow Nodes

Analysis Nodes:

  1. profile - ProfileAgent analyzes dataset structure and statistics
  2. quality_check - QualityAgent detects missing values, duplicates, outliers
  3. transform_proposal - TransformAgent proposes cleaning transformations
  4. visualization - VisualizationAgent generates charts
  5. feature_analysis - FeatureAgent analyzes correlations and features
  6. statistical_analysis - StatAgent performs hypothesis testing

Control Nodes:

  • human_review_[step] - Interrupt points for human approval
  • Conditional routing based on user decisions (approve/retry/skip/stop)

Workflow Execution Flow

START
  β”‚
  β–Ό
[Profile]
  β”‚
  β–Ό
[Human Review] ───retry──> [Profile]
  β”‚
  β”‚ approved
  β–Ό
[Quality Check]
  β”‚
  β–Ό
[Human Review] ───retry──> [Quality Check]
  β”‚
  β”‚ approved
  β–Ό
[Transform Proposal]
  β”‚
  β–Ό
[Human Review] ───retry──> [Transform Proposal]
  β”‚
  β”‚ approved
  β–Ό
END

Conditional Routing

After each human review, the workflow routes based on user decision:

# Route after profile review
if decision == "approved":
    β†’ continue to quality_check
elif decision == "rejected":
    β†’ retry profile node
else:
    β†’ end workflow

# Similar routing for quality_check and transform_proposal

Checkpoint & Resume

Persistent State:

  • Workflow state is saved at each node
  • Can pause and resume from any point
  • State includes all analysis results and user decisions

Resume Example:

workflow = EDAWorkflow()
state = workflow.get_state(thread_id="session_123")
# ... user makes decision ...
workflow.update_state(thread_id, {"user_decisions": [decision]})
workflow.resume(thread_id)

Pre-defined Workflows

Quick Profile (5 mins)

  • Steps: profile β†’ quality_check β†’ basic_viz
  • Interrupts: after_profile
  • Use case: Fast dataset health check

Deep Clean (15 mins)

  • Steps: profile β†’ quality β†’ outlier β†’ missing β†’ transform
  • Interrupts: after_quality, before_transform
  • Use case: Thorough data cleaning

Feature Engineering (20 mins)

  • Steps: profile β†’ correlation β†’ importance β†’ interactions β†’ engineer
  • Interrupts: after_correlation, before_engineer
  • Use case: ML preparation and feature discovery

Context Propagation

Each agent receives context from previous steps:

context = {
    "profile_results": state["profile_results"],
    "quality_results": state["quality_results"],
    "feature_results": state["feature_results"]
}

# Agents use this context to make informed decisions
response = agent.analyze(dataset_handle, context)

Explainability

Every agent action is logged with reasoning:

reasoning_log = {
    "timestamp": "2026-07-23T10:30:00",
    "agent": "QualityAgent",
    "action": "quality_assessment",
    "reasoning": "Found 15% missing values in Age column...",
    "impact": "Missing values may bias analysis...",
    "confidence": 0.92
}

Benefits of LangGraph Orchestration

βœ… Stateful Execution - Pause and resume at any point
βœ… Human Control - Review and approve each step
βœ… Context Aware - Agents share insights
βœ… Explainable - Full reasoning trail
βœ… Flexible - Easy to add/modify workflows
βœ… Scalable - Handle complex multi-agent pipelines

Quick Start

1. Installation

# Clone the repository
git clone <repository-url>
cd test

# Install dependencies
pip install -r requirements.txt

2. Configuration

# Copy environment template
cp .env.example .env

# Edit .env and add your GROQ_API_KEY
# Get your key from: https://console.groq.com/

Example .env:

GROQ_API_KEY=your_groq_api_key_here
LANGCHAIN_TRACING_V2=false

3. Run the Application

streamlit run src/ui/app.py

The app will open in your browser at http://localhost:8501

4. Usage

  1. Upload Dataset: Click "Browse files" in the sidebar and upload a CSV or Excel file (.csv, .xlsx, .xls)
  2. View Quick Stats: See immediate dataset overview (rows, columns, size)
  3. Choose Analysis:
    • Quick Analysis: Run all 6 agents sequentially
    • Individual Agent: Select and run specific agents
    • Deep Dive: Comprehensive analysis with detailed insights
    • ML Preparation: Prepare data for machine learning workflows
  4. Enable Approval Gates (Optional - NEW! 🚦):
    • β˜‘οΈ Check "Enable Approval Gates" to review each agent before continuing
    • Uncheck for automatic execution
  5. View Results: Explore interactive tabs for each agent's analysis
  6. Transform Data:
    • β˜‘οΈ Select multiple transformations
    • Preview combined effect
    • Apply to full dataset
    • Export transformed CSV
  7. Export: Generate HTML reports, JSON data, or transformed CSV files

Example: Transform Titanic Dataset

1. Upload titanic_train.csv (891 rows, 12 columns)
2. Run "Quick Analysis"
3. Go to "Transform" tab
4. Click "β˜‘οΈ Select All High Priority"
   β†’ 3 transformations selected
5. Click "Preview Selected (3)"
   β†’ See: Sex becomes Sex_male & Sex_female
   β†’ See: Missing Age values filled with median (28)
   β†’ See: Cabin column removed (77% missing)
6. Click "Apply 3 transformations to Full Dataset"
   β†’ Progress: Loading β†’ Applying β†’ Saving (100%)
7. Go to "Export" tab
8. Check "Transformed CSV"
9. Export Now
10. Download: transformed_dataset_20260720.csv (891 rows, 13 columns)
    β†’ Ready for ML! πŸš€

Project Structure

β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ agents/              # 7 Specialized EDA agents
β”‚   β”‚   β”œβ”€β”€ profile.py       # Dataset profiling and statistics
β”‚   β”‚   β”œβ”€β”€ quality.py       # Data quality assessment
β”‚   β”‚   β”œβ”€β”€ transform.py     # Data cleaning and transformation
β”‚   β”‚   β”œβ”€β”€ visualization.py # Chart generation and visual analysis
β”‚   β”‚   β”œβ”€β”€ feature.py       # Feature engineering and analysis
β”‚   β”‚   β”œβ”€β”€ stat.py          # Statistical testing and analysis
β”‚   β”‚   └── time_series_agent.py  # Time series analysis (NEW!)
β”‚   β”œβ”€β”€ data/                # DatasetHandle and backend management
β”‚   β”‚   └── dataset_handle.py
β”‚   β”œβ”€β”€ graph/               # LangGraph workflow definitions
β”‚   β”œβ”€β”€ ui/                  # Streamlit interface
β”‚   β”‚   β”œβ”€β”€ app.py           # Main application
β”‚   β”‚   └── components/      # Reusable UI components
β”‚   β”‚       └── progress_tracker.py
β”‚   └── utils/               # Helper functions and utilities
β”‚       β”œβ”€β”€ export.py        # Export manager for HTML/JSON/CSV
β”‚       └── helpers.py
β”œβ”€β”€ tests/                   # Test and demo files
β”‚   β”œβ”€β”€ test_*.py            # Unit and integration tests
β”‚   β”œβ”€β”€ demo_*.py            # Interactive demonstrations
β”‚   β”‚   └── demo_time_series_agent.py  # Time series demo
β”‚   └── README.md            # Test documentation
β”œβ”€β”€ data/
β”‚   β”œβ”€β”€ uploads/             # Uploaded datasets
β”‚   β”œβ”€β”€ exports/             # Generated reports and exports
β”‚   β”œβ”€β”€ artifacts/           # Generated plots and visualizations
β”‚   └── checkpoints/         # LangGraph state persistence
β”œβ”€β”€ docs/                    # Detailed documentation
β”‚   β”œβ”€β”€ presentations/       # PowerPoint slides and diagrams
β”‚   β”‚   β”œβ”€β”€ scripts/         # Scripts to generate materials
β”‚   β”‚   β”œβ”€β”€ assets/          # Architecture diagrams (PNG)
β”‚   β”‚   └── *.pptx           # Generated presentations
β”‚   β”œβ”€β”€ TIME_SERIES_AGENT_PHASE1.md     # Time series implementation
β”‚   β”œβ”€β”€ TIME_SERIES_AGENT_SUMMARY.md    # Time series overview
β”‚   β”œβ”€β”€ TIME_SERIES_UI_INTEGRATION.md   # UI integration guide
β”‚   β”œβ”€β”€ PROGRESS_TRACKER.md
β”‚   β”œβ”€β”€ QUALITY_VISUALIZATION.md
β”‚   β”œβ”€β”€ BEFORE_AFTER_COMPARISON.md
β”‚   β”œβ”€β”€ EXPORT_FUNCTIONALITY.md
β”‚   └── UI_UX_ENHANCEMENTS_SUMMARY.md
β”œβ”€β”€ requirements.txt
β”œβ”€β”€ .env.example
└── README.md

Available Agents

1. ProfileAgent

  • Dataset shape and structure
  • Column types and distributions
  • Memory usage analysis
  • Basic statistics per column

2. QualityAgent

  • Missing value detection and patterns
  • Duplicate row identification
  • Outlier detection (IQR method)
  • Data quality scoring
  • Interactive quality visualizations

3. TransformAgent

  • Automated data cleaning proposals
  • Missing value imputation strategies
  • Outlier handling (capping, removal)
  • Data type conversions
  • Categorical encoding (one-hot, label)
  • Numeric scaling (standard, min-max)
  • Before/after comparison views
  • Multi-selection: Apply multiple transformations at once
  • Full dataset application: Apply to entire dataset with progress tracking
  • CSV export: Save transformed data for external use

4. VisualizationAgent

  • Automatic chart generation
  • Distribution plots
  • Correlation heatmaps
  • Trend analysis
  • Interactive Plotly visualizations

5. FeatureAgent

  • Feature importance analysis
  • Correlation analysis
  • Feature engineering suggestions
  • ML-ready feature preparation

6. StatAgent

  • Statistical hypothesis testing
  • Distribution analysis (normality tests)
  • Comparative statistics
  • Confidence intervals

7. TimeSeriesAgent (NEW! πŸ•’)

  • Temporal profiling (frequency, gaps, duplicates)
  • Trend & seasonality detection (STL decomposition)
  • Stationarity testing (ADF test)
  • Time series visualizations
  • Forecasting insights and recommendations
  • Auto-detection: Automatically triggered for datetime columns
  • UI Integration: Available in individual agent dropdown and results tab
  • Phase 1: Core analysis (temporal profiling, decomposition, stationarity) βœ… Complete

Transformation Workflow

The TransformAgent offers a complete data transformation pipeline:

1. Review Proposals

  • Agent analyzes your data and proposes transformations
  • Organized by priority (High, Medium, Low)
  • Each proposal includes reasoning and impact

2. Multi-Selection

  • β˜‘οΈ Check boxes to select multiple transformations
  • Quick Actions:
    • "Select All High Priority" - Instant data cleaning
    • "Preview Selected (N)" - See combined effect
    • "Deselect All" - Clear selections

3. Preview Transformations

  • See before/after comparison with sample data
  • View removed columns and new columns side-by-side
  • See exact value mappings (e.g., 'male' β†’ [0,1])
  • Understand the combined effect of all selected transformations

4. Apply to Full Dataset

  • Click "Apply N transformations to Full Dataset"
  • Progress tracking shows: Loading β†’ Applying β†’ Saving
  • Warning for large datasets (memory usage)
  • Preview transformed data inline

5. Export Transformed CSV

  • Go to Export section
  • Check "Transformed CSV" (now enabled)
  • Download your transformed dataset
  • Use in Excel, Python, R, ML tools, etc.

Example Workflow

1. Load Titanic dataset
2. Review 8 transformation proposals
3. β˜‘οΈ Select: "One-hot encode Sex", "Impute Age", "Drop Cabin"
4. Preview β†’ See 'Sex' becomes 'Sex_male' and 'Sex_female'
5. Apply 3 transformations β†’ 891 rows processed
6. Export β†’ Download transformed_dataset.csv
7. Use in your ML pipeline! πŸš€

Available Workflows

Quick Analysis (All Agents)

  • Runs all 6 agents sequentially
  • Comprehensive dataset overview
  • ~5-10 minutes for typical datasets
  • Best for: First-time analysis, complete understanding
  • NEW: Available with Approval Gates for human review

Deep Dive Workflow

  • Thorough analysis (5 agents: Profile, Quality, Viz, Feature, Stat)
  • Detailed quality assessment
  • Advanced statistical tests
  • Best for: Critical datasets, production data
  • NEW: Available with Approval Gates for step-by-step review

ML Preparation

  • Feature engineering focus (4 agents: Profile, Quality, Feature, Transform)
  • Correlation analysis
  • Feature selection recommendations
  • Training-ready data export
  • Best for: Machine learning projects
  • NEW: Available with Approval Gates for controlled ML prep

Individual Agent Execution

  • Run any single agent on-demand
  • Fast, targeted analysis
  • Best for: Specific questions, iterative exploration

🚦 NEW: Workflows with Approval Gates

All workflows now available with Human-in-the-Loop approval gates:

  • Pause after each agent for human review
  • See confidence scores, issues, and recommendations
  • 4 decision options: Approve, Retry, Skip, or Stop
  • Full decision history tracking
  • Best for: Critical analysis, compliance, learning

πŸ‘‰ Get Started with Approval Gates

Export Features

Generate professional outputs in multiple formats:

HTML Reports

  • Beautiful, interactive reports
  • Embedded visualizations (Plotly charts)
  • Confidence scores and reasoning
  • Table of contents navigation
  • Shareable with stakeholders

JSON Data

  • Complete analysis results
  • Structured, hierarchical format
  • Easy integration with other tools
  • API-ready format

Transformed CSV

  • Cleaned and transformed dataset
  • Apply multiple transformations: encoding, imputation, scaling, etc.
  • Full dataset processing (not just samples)
  • Progress tracking for large datasets
  • Ready for downstream processing (Excel, Python, R, ML tools)
  • Includes all selected TransformAgent changes

Export Location: data/exports/

Naming Convention: [custom_name_]<type>_<timestamp>.<ext>

See docs/EXPORT_FUNCTIONALITY.md for detailed documentation.

UI Components & Visualizations

Progress Tracking

  • Visual workflow stepper
  • Real-time status updates (pending β†’ running β†’ completed)
  • Individual step timing with ETA
  • Color-coded indicators

Quality Dashboard

  • Missing value heatmaps
  • Outlier detection box plots
  • Duplicate analysis gauge charts
  • Overall quality score
  • Interactive drill-down

Before/After Comparison

  • Side-by-side data preview
  • Delta metrics (Missing, Duplicates, Outliers)
  • Distribution comparison charts
  • Statistical comparison tables
  • Impact severity indicators
  • Column transformation mapping: See exactly how columns change
  • Removed vs New columns: Side-by-side view of added/removed columns
  • One-hot encoding visualization: Grouped display of encoded columns
  • Value-by-value mapping: See how original values transform to new values

Tech Stack

Category Technology Purpose
Orchestration LangGraph State machine workflow management
LLM Groq (Llama 3.1) Fast LLM inference for agent reasoning
UI Streamlit Interactive web interface
Data Processing Pandas, DuckDB In-memory and large dataset handling
Visualization Plotly, Seaborn, Matplotlib Interactive and static charts
Statistics SciPy, Statsmodels, Scikit-learn Statistical analysis and ML prep
Persistence SQLite State checkpointing

Configuration & Settings

Environment Variables

  • GROQ_API_KEY (required): Your Groq API key
  • LANGCHAIN_TRACING_V2 (optional): Enable LangSmith tracing
  • LANGCHAIN_API_KEY (optional): LangSmith API key
  • LANGCHAIN_PROJECT (optional): Project name for tracing

In-App Settings

  • Show Reasoning: Display agent's decision-making process
  • Show Confidence: Show confidence scores for findings
  • Analysis Type: Choose workflow type
  • Agent Selection: Pick individual agents to run

Requirements

  • Python 3.9+
  • 4GB RAM minimum (8GB recommended for large datasets)
  • Internet connection (for Groq API)
  • Modern web browser

Best Practices

For Small Datasets (< 10k rows)

  • Use "Quick Analysis" for complete overview
  • All processing happens in-memory
  • Fast execution (~2-5 minutes)

For Large Datasets (> 100k rows)

  • System automatically switches to DuckDB backend
  • Intelligent sampling for visualization
  • Enable progress tracking to monitor long-running tasks

For Production Use

  1. Run Deep Dive Workflow for thorough analysis
  2. Export HTML reports for documentation
  3. Export JSON for integration with other systems
  4. Use transformed CSV for downstream processing
  5. Review confidence scores and reasoning before acting on recommendations

For ML Projects

  1. Use ML Preparation workflow
  2. Review FeatureAgent recommendations
  3. Check StatAgent for distribution assumptions
  4. Export transformed CSV for model training
  5. Use QualityAgent to ensure data cleanliness

Troubleshooting

"No API key found"

  • Ensure .env file exists in project root
  • Verify GROQ_API_KEY is set correctly
  • Restart Streamlit after changing .env

Dataset Upload Issues

  • Ensure CSV is properly formatted
  • Check file encoding (UTF-8 recommended)
  • Verify no corrupted rows
  • Maximum recommended size: 500MB

Slow Performance

  • Large datasets trigger automatic DuckDB backend
  • Reduce sample size for visualizations
  • Run individual agents instead of complete analysis
  • Close other browser tabs to free memory

Export Failures

  • Ensure data/exports/ directory exists
  • Check disk space availability
  • Verify write permissions
  • For CSV export: Must click "Apply to Full Dataset" first
  • Check that transformed dataset shows "βœ… Ready (N rows)"

Transformation Issues

  • CSV checkbox disabled: Apply transformations to full dataset first
  • Columns not showing: Check the "Removed vs New Columns" section in preview
  • One-hot encoding not visible: Look for new columns like ColumnName_value1, ColumnName_value2
  • DatasetHandle error: Restart the app if backend connection issues occur
  • Large dataset slow: Normal - check progress bar for status

Documentation

Detailed documentation available in docs/:

Presentations & Marketing Materials

  • presentations/: PowerPoint presentations and architecture diagrams
    • Ready-to-use presentation slides (18 slides)
    • High-resolution architecture diagram (PNG, 300 DPI)
    • Scripts to regenerate materials
    • See docs/presentations/README.md for details

Core Features

  • PROGRESS_TRACKER.md: Progress tracking component guide
  • QUALITY_VISUALIZATION.md: Quality visualization system
  • EXPORT_FUNCTIONALITY.md: Complete export system documentation
  • UI_UX_ENHANCEMENTS_SUMMARY.md: All UI/UX improvements overview

Transformation Features

  • TRANSFORMATION_PREVIEW_FIX.md: Technical details of transformation preview fix
  • HOW_TO_SEE_NEW_COLUMNS.md: Guide to viewing transformed columns
  • UI_LAYOUT_DIAGRAM.md: Visual layout of transformation UI
  • TRANSFORMATION_PREVIEW_VISUAL_GUIDE.md: Before/after comparison guide
  • CSV_EXPORT_FEATURE.md: Complete CSV export documentation
  • EXPORT_CSV_QUICK_GUIDE.md: Quick start guide for CSV export
  • CSV_EXPORT_FIX.md: Technical fix for DatasetHandle backend
  • MULTI_TRANSFORMATION_SELECTION.md: Multi-selection feature guide

Contributing

Contributions welcome! Areas for improvement:

  • Additional agent types (time series, text analysis)
  • More export formats (PDF, Excel)
  • Advanced visualizations
  • Performance optimizations
  • Custom workflow builders

License

[Add your license here]

Acknowledgments

Built with:


Current Version: 3.3.1
Last Updated: July 2026

Recent Updates

  • v3.3.1 (July 2026): πŸ•’ TimeSeriesAgent Phase 1 - temporal profiling, trend/seasonality detection, stationarity testing
  • v3.3 (July 2026): πŸ“Š Excel file support (.xlsx, .xls), automatic sheet detection, seamless integration
  • v3.2 (July 2026): 🚦 Human-in-the-Loop approval gates, decision tracking, step-by-step agent review
  • v3.1 (July 2026): Multi-transformation selection, CSV export, column visualization
  • v3.0 (June 2026): Complete agent suite, progress tracking, quality visualizations

For questions, issues, or feature requests, please open an issue on the repository.

About

An end-to-end AI product that handles the entire machine learning pipeline, from EDA to model development

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages