An intelligent, production-ready EDA (Exploratory Data Analysis) pipeline powered by LangGraph, Groq, and open-source tools. Features a complete suite of specialized agents with interactive visualizations, export capabilities, and human-in-the-loop interaction.
- β
Excel Upload: Support for
.xlsxand.xlsfiles in addition to CSV - β Automatic Detection: First sheet is automatically loaded and analyzed
- β Seamless Integration: Excel files work with all agents and workflows
- β Same Performance: Intelligent backend selection (in-memory vs sampled) for Excel files
- β Agent Approval Gates: Review and approve each agent before proceeding to the next
- β 4 Decision Options: Approve, Retry, Skip, or Stop at each agent
- β Decision History: Track all approval decisions throughout workflow
- β 3 New Workflows: Quick Analysis, Deep Dive, and ML Prep with Approval Gates
- β Detailed Review UI: See confidence scores, issues found, and recommendations
- β Agent-Specific Details: Tailored detail views for each agent type
- β Thumbnail Gallery: Quick preview of visualization plots with expand-to-full-size option
π Quick Start Guide | Full Documentation
The approval gate design follows a two-phase workflow for optimal user experience:
Phase 1: Approval Gate (Quick Review)
- Purpose: Fast decision-making - approve, retry, skip, or stop
- Shows: Summary metrics, key findings, reasoning, and thumbnail previews
- Why compact?
- β‘ Speed: Quick review enables fast decisions (30-60 seconds per agent)
- π Size: With 8+ plots and detailed tables, gates would become unwieldy
- π― Focus: Summary view keeps attention on decision-relevant information
- π Workflow: Detailed analysis happens after approval, not during
Phase 2: Results Tab (Deep Analysis)
- Purpose: Comprehensive exploration of agent findings
- Shows: All plots in full size, complete tables, interactive visualizations, export options
- When: After workflow completes or individual agent runs
- Where: Navigate to tabbed interface (Profile, Quality, Visualizations, etc.)
This separation ensures approval gates remain fast decision points while full results provide thorough analysis capabilities.
- β Multi-Transformation Selection: Select and apply multiple transformations at once with checkboxes
- β Complete CSV Export: Apply transformations to full dataset and export as CSV
- β Column Change Visualization: See exactly which columns are added/removed during transformations
- β One-Hot Encoding Preview: Visual mapping showing how categorical columns transform to binary columns
- β Progress Tracking: Real-time progress bar when applying transformations to large datasets
- β Quick Actions: "Select All High Priority" for instant data cleaning
- π Fixed transformation preview showing same results in before/after
- π Fixed DatasetHandle backend access for large datasets
- π Fixed CSV export not working
- π§ Added support for all 7 transformation types (encoding, scaling, imputation, etc.)
- π§ Enhanced error reporting with detailed tracebacks
- π§ Smart column comparison (only shows common columns to prevent errors)
- 7 Specialized Agents: ProfileAgent, QualityAgent, TransformAgent, VisualizationAgent, FeatureAgent, StatAgent, TimeSeriesAgent
- Interactive Chat Interface: Real-time Streamlit UI with progress tracking
- Explainable AI: Every agent explains WHY and WHAT impact their findings have
- Persistent State: LangGraph checkpoints for pause/resume workflows
- Hybrid Scale: Handles small datasets in-memory, large datasets with intelligent sampling
- Flexible Workflows: Pre-defined pipelines + individual agent execution
- Human-in-the-Loop Control: Agent approval gates for reviewing and approving each step
- Progress Tracking: Real-time visual workflow progress with status indicators and ETA
- Quality Visualizations: Interactive charts for missing values, outliers, duplicates, and data quality metrics
- Before/After Comparison: Side-by-side transformation previews with delta metrics and impact analysis
- Multi-Transformation Selection: Select and apply multiple transformations at once with checkboxes
- Column Change Visualization: See exactly which columns are added/removed during transformations
- Export System: Professional HTML reports, JSON data exports, and transformed CSV outputs with full dataset support
- ML Preparation: Automated feature engineering and data preparation for machine learning
- Decision Tracking: Complete audit trail of human approval decisions
βββββββββββββββββββ
β Streamlit UI β β Interactive interface with progress tracking
ββββββββββ¬βββββββββ
β
ββββββββββΌβββββββββ
β LangGraph β β State machine orchestration
β Orchestration β
ββββββββββ¬βββββββββ
β
ββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Specialized Agents (7) β
ββββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββββββ¬βββββββ¬ββββββββ€
β Profile β Quality βTransform βVisualize β Feature β Stat β Time β
β Agent β Agent β Agent β Agent β Agent βAgent βSeries β
β β β β β β βAgent β
ββββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββββββ΄βββββββ΄ββββββββ
β
ββββββββββΌβββββββββ
β Data Layer β β Pandas/DuckDB with intelligent switching
βββββββββββββββββββ
The EDA Pipeline uses LangGraph for state machine-based workflow orchestration, enabling:
- Stateful execution with checkpoint/resume capabilities
- Human-in-the-loop approval gates with conditional routing
- Context propagation between agents
- Explainable decisions with reasoning logs
The workflow maintains a comprehensive EDAState that tracks:
EDAState:
# Dataset metadata
- dataset_path, dataset_mode, dataset_size
- dataset_rows, dataset_cols
# Workflow control
- workflow_type: "quick_profile" | "deep_clean" | "feature_engineering"
- current_step, completed_steps
- pending_approval, approval_context
# Analysis results
- profile_results, quality_results
- feature_results, stat_results
- visualizations[]
# Explainability
- reasoning_log[] (timestamp, agent, reasoning, impact, confidence)
# Human interaction
- user_decisions[] (approved/rejected/modified)
- user_messages[]
# Transformations
- pending_transformations[]
- applied_transformations[]Analysis Nodes:
- profile - ProfileAgent analyzes dataset structure and statistics
- quality_check - QualityAgent detects missing values, duplicates, outliers
- transform_proposal - TransformAgent proposes cleaning transformations
- visualization - VisualizationAgent generates charts
- feature_analysis - FeatureAgent analyzes correlations and features
- statistical_analysis - StatAgent performs hypothesis testing
Control Nodes:
- human_review_[step] - Interrupt points for human approval
- Conditional routing based on user decisions (approve/retry/skip/stop)
START
β
βΌ
[Profile]
β
βΌ
[Human Review] βββretryββ> [Profile]
β
β approved
βΌ
[Quality Check]
β
βΌ
[Human Review] βββretryββ> [Quality Check]
β
β approved
βΌ
[Transform Proposal]
β
βΌ
[Human Review] βββretryββ> [Transform Proposal]
β
β approved
βΌ
END
After each human review, the workflow routes based on user decision:
# Route after profile review
if decision == "approved":
β continue to quality_check
elif decision == "rejected":
β retry profile node
else:
β end workflow
# Similar routing for quality_check and transform_proposalPersistent State:
- Workflow state is saved at each node
- Can pause and resume from any point
- State includes all analysis results and user decisions
Resume Example:
workflow = EDAWorkflow()
state = workflow.get_state(thread_id="session_123")
# ... user makes decision ...
workflow.update_state(thread_id, {"user_decisions": [decision]})
workflow.resume(thread_id)Quick Profile (5 mins)
- Steps: profile β quality_check β basic_viz
- Interrupts: after_profile
- Use case: Fast dataset health check
Deep Clean (15 mins)
- Steps: profile β quality β outlier β missing β transform
- Interrupts: after_quality, before_transform
- Use case: Thorough data cleaning
Feature Engineering (20 mins)
- Steps: profile β correlation β importance β interactions β engineer
- Interrupts: after_correlation, before_engineer
- Use case: ML preparation and feature discovery
Each agent receives context from previous steps:
context = {
"profile_results": state["profile_results"],
"quality_results": state["quality_results"],
"feature_results": state["feature_results"]
}
# Agents use this context to make informed decisions
response = agent.analyze(dataset_handle, context)Every agent action is logged with reasoning:
reasoning_log = {
"timestamp": "2026-07-23T10:30:00",
"agent": "QualityAgent",
"action": "quality_assessment",
"reasoning": "Found 15% missing values in Age column...",
"impact": "Missing values may bias analysis...",
"confidence": 0.92
}β
Stateful Execution - Pause and resume at any point
β
Human Control - Review and approve each step
β
Context Aware - Agents share insights
β
Explainable - Full reasoning trail
β
Flexible - Easy to add/modify workflows
β
Scalable - Handle complex multi-agent pipelines
# Clone the repository
git clone <repository-url>
cd test
# Install dependencies
pip install -r requirements.txt# Copy environment template
cp .env.example .env
# Edit .env and add your GROQ_API_KEY
# Get your key from: https://console.groq.com/Example .env:
GROQ_API_KEY=your_groq_api_key_here
LANGCHAIN_TRACING_V2=falsestreamlit run src/ui/app.pyThe app will open in your browser at http://localhost:8501
- Upload Dataset: Click "Browse files" in the sidebar and upload a CSV or Excel file (
.csv,.xlsx,.xls) - View Quick Stats: See immediate dataset overview (rows, columns, size)
- Choose Analysis:
- Quick Analysis: Run all 6 agents sequentially
- Individual Agent: Select and run specific agents
- Deep Dive: Comprehensive analysis with detailed insights
- ML Preparation: Prepare data for machine learning workflows
- Enable Approval Gates (Optional - NEW! π¦):
- βοΈ Check "Enable Approval Gates" to review each agent before continuing
- Uncheck for automatic execution
- View Results: Explore interactive tabs for each agent's analysis
- Transform Data:
- βοΈ Select multiple transformations
- Preview combined effect
- Apply to full dataset
- Export transformed CSV
- Export: Generate HTML reports, JSON data, or transformed CSV files
1. Upload titanic_train.csv (891 rows, 12 columns)
2. Run "Quick Analysis"
3. Go to "Transform" tab
4. Click "βοΈ Select All High Priority"
β 3 transformations selected
5. Click "Preview Selected (3)"
β See: Sex becomes Sex_male & Sex_female
β See: Missing Age values filled with median (28)
β See: Cabin column removed (77% missing)
6. Click "Apply 3 transformations to Full Dataset"
β Progress: Loading β Applying β Saving (100%)
7. Go to "Export" tab
8. Check "Transformed CSV"
9. Export Now
10. Download: transformed_dataset_20260720.csv (891 rows, 13 columns)
β Ready for ML! π
βββ src/
β βββ agents/ # 7 Specialized EDA agents
β β βββ profile.py # Dataset profiling and statistics
β β βββ quality.py # Data quality assessment
β β βββ transform.py # Data cleaning and transformation
β β βββ visualization.py # Chart generation and visual analysis
β β βββ feature.py # Feature engineering and analysis
β β βββ stat.py # Statistical testing and analysis
β β βββ time_series_agent.py # Time series analysis (NEW!)
β βββ data/ # DatasetHandle and backend management
β β βββ dataset_handle.py
β βββ graph/ # LangGraph workflow definitions
β βββ ui/ # Streamlit interface
β β βββ app.py # Main application
β β βββ components/ # Reusable UI components
β β βββ progress_tracker.py
β βββ utils/ # Helper functions and utilities
β βββ export.py # Export manager for HTML/JSON/CSV
β βββ helpers.py
βββ tests/ # Test and demo files
β βββ test_*.py # Unit and integration tests
β βββ demo_*.py # Interactive demonstrations
β β βββ demo_time_series_agent.py # Time series demo
β βββ README.md # Test documentation
βββ data/
β βββ uploads/ # Uploaded datasets
β βββ exports/ # Generated reports and exports
β βββ artifacts/ # Generated plots and visualizations
β βββ checkpoints/ # LangGraph state persistence
βββ docs/ # Detailed documentation
β βββ presentations/ # PowerPoint slides and diagrams
β β βββ scripts/ # Scripts to generate materials
β β βββ assets/ # Architecture diagrams (PNG)
β β βββ *.pptx # Generated presentations
β βββ TIME_SERIES_AGENT_PHASE1.md # Time series implementation
β βββ TIME_SERIES_AGENT_SUMMARY.md # Time series overview
β βββ TIME_SERIES_UI_INTEGRATION.md # UI integration guide
β βββ PROGRESS_TRACKER.md
β βββ QUALITY_VISUALIZATION.md
β βββ BEFORE_AFTER_COMPARISON.md
β βββ EXPORT_FUNCTIONALITY.md
β βββ UI_UX_ENHANCEMENTS_SUMMARY.md
βββ requirements.txt
βββ .env.example
βββ README.md
- Dataset shape and structure
- Column types and distributions
- Memory usage analysis
- Basic statistics per column
- Missing value detection and patterns
- Duplicate row identification
- Outlier detection (IQR method)
- Data quality scoring
- Interactive quality visualizations
- Automated data cleaning proposals
- Missing value imputation strategies
- Outlier handling (capping, removal)
- Data type conversions
- Categorical encoding (one-hot, label)
- Numeric scaling (standard, min-max)
- Before/after comparison views
- Multi-selection: Apply multiple transformations at once
- Full dataset application: Apply to entire dataset with progress tracking
- CSV export: Save transformed data for external use
- Automatic chart generation
- Distribution plots
- Correlation heatmaps
- Trend analysis
- Interactive Plotly visualizations
- Feature importance analysis
- Correlation analysis
- Feature engineering suggestions
- ML-ready feature preparation
- Statistical hypothesis testing
- Distribution analysis (normality tests)
- Comparative statistics
- Confidence intervals
- Temporal profiling (frequency, gaps, duplicates)
- Trend & seasonality detection (STL decomposition)
- Stationarity testing (ADF test)
- Time series visualizations
- Forecasting insights and recommendations
- Auto-detection: Automatically triggered for datetime columns
- UI Integration: Available in individual agent dropdown and results tab
- Phase 1: Core analysis (temporal profiling, decomposition, stationarity) β Complete
The TransformAgent offers a complete data transformation pipeline:
- Agent analyzes your data and proposes transformations
- Organized by priority (High, Medium, Low)
- Each proposal includes reasoning and impact
- βοΈ Check boxes to select multiple transformations
- Quick Actions:
- "Select All High Priority" - Instant data cleaning
- "Preview Selected (N)" - See combined effect
- "Deselect All" - Clear selections
- See before/after comparison with sample data
- View removed columns and new columns side-by-side
- See exact value mappings (e.g., 'male' β [0,1])
- Understand the combined effect of all selected transformations
- Click "Apply N transformations to Full Dataset"
- Progress tracking shows: Loading β Applying β Saving
- Warning for large datasets (memory usage)
- Preview transformed data inline
- Go to Export section
- Check "Transformed CSV" (now enabled)
- Download your transformed dataset
- Use in Excel, Python, R, ML tools, etc.
1. Load Titanic dataset
2. Review 8 transformation proposals
3. βοΈ Select: "One-hot encode Sex", "Impute Age", "Drop Cabin"
4. Preview β See 'Sex' becomes 'Sex_male' and 'Sex_female'
5. Apply 3 transformations β 891 rows processed
6. Export β Download transformed_dataset.csv
7. Use in your ML pipeline! π
- Runs all 6 agents sequentially
- Comprehensive dataset overview
- ~5-10 minutes for typical datasets
- Best for: First-time analysis, complete understanding
- NEW: Available with Approval Gates for human review
- Thorough analysis (5 agents: Profile, Quality, Viz, Feature, Stat)
- Detailed quality assessment
- Advanced statistical tests
- Best for: Critical datasets, production data
- NEW: Available with Approval Gates for step-by-step review
- Feature engineering focus (4 agents: Profile, Quality, Feature, Transform)
- Correlation analysis
- Feature selection recommendations
- Training-ready data export
- Best for: Machine learning projects
- NEW: Available with Approval Gates for controlled ML prep
- Run any single agent on-demand
- Fast, targeted analysis
- Best for: Specific questions, iterative exploration
All workflows now available with Human-in-the-Loop approval gates:
- Pause after each agent for human review
- See confidence scores, issues, and recommendations
- 4 decision options: Approve, Retry, Skip, or Stop
- Full decision history tracking
- Best for: Critical analysis, compliance, learning
π Get Started with Approval Gates
Generate professional outputs in multiple formats:
- Beautiful, interactive reports
- Embedded visualizations (Plotly charts)
- Confidence scores and reasoning
- Table of contents navigation
- Shareable with stakeholders
- Complete analysis results
- Structured, hierarchical format
- Easy integration with other tools
- API-ready format
- Cleaned and transformed dataset
- Apply multiple transformations: encoding, imputation, scaling, etc.
- Full dataset processing (not just samples)
- Progress tracking for large datasets
- Ready for downstream processing (Excel, Python, R, ML tools)
- Includes all selected TransformAgent changes
Export Location: data/exports/
Naming Convention: [custom_name_]<type>_<timestamp>.<ext>
See docs/EXPORT_FUNCTIONALITY.md for detailed documentation.
- Visual workflow stepper
- Real-time status updates (pending β running β completed)
- Individual step timing with ETA
- Color-coded indicators
- Missing value heatmaps
- Outlier detection box plots
- Duplicate analysis gauge charts
- Overall quality score
- Interactive drill-down
- Side-by-side data preview
- Delta metrics (Missing, Duplicates, Outliers)
- Distribution comparison charts
- Statistical comparison tables
- Impact severity indicators
- Column transformation mapping: See exactly how columns change
- Removed vs New columns: Side-by-side view of added/removed columns
- One-hot encoding visualization: Grouped display of encoded columns
- Value-by-value mapping: See how original values transform to new values
| Category | Technology | Purpose |
|---|---|---|
| Orchestration | LangGraph | State machine workflow management |
| LLM | Groq (Llama 3.1) | Fast LLM inference for agent reasoning |
| UI | Streamlit | Interactive web interface |
| Data Processing | Pandas, DuckDB | In-memory and large dataset handling |
| Visualization | Plotly, Seaborn, Matplotlib | Interactive and static charts |
| Statistics | SciPy, Statsmodels, Scikit-learn | Statistical analysis and ML prep |
| Persistence | SQLite | State checkpointing |
GROQ_API_KEY(required): Your Groq API keyLANGCHAIN_TRACING_V2(optional): Enable LangSmith tracingLANGCHAIN_API_KEY(optional): LangSmith API keyLANGCHAIN_PROJECT(optional): Project name for tracing
- Show Reasoning: Display agent's decision-making process
- Show Confidence: Show confidence scores for findings
- Analysis Type: Choose workflow type
- Agent Selection: Pick individual agents to run
- Python 3.9+
- 4GB RAM minimum (8GB recommended for large datasets)
- Internet connection (for Groq API)
- Modern web browser
- Use "Quick Analysis" for complete overview
- All processing happens in-memory
- Fast execution (~2-5 minutes)
- System automatically switches to DuckDB backend
- Intelligent sampling for visualization
- Enable progress tracking to monitor long-running tasks
- Run Deep Dive Workflow for thorough analysis
- Export HTML reports for documentation
- Export JSON for integration with other systems
- Use transformed CSV for downstream processing
- Review confidence scores and reasoning before acting on recommendations
- Use ML Preparation workflow
- Review FeatureAgent recommendations
- Check StatAgent for distribution assumptions
- Export transformed CSV for model training
- Use QualityAgent to ensure data cleanliness
- Ensure
.envfile exists in project root - Verify
GROQ_API_KEYis set correctly - Restart Streamlit after changing
.env
- Ensure CSV is properly formatted
- Check file encoding (UTF-8 recommended)
- Verify no corrupted rows
- Maximum recommended size: 500MB
- Large datasets trigger automatic DuckDB backend
- Reduce sample size for visualizations
- Run individual agents instead of complete analysis
- Close other browser tabs to free memory
- Ensure
data/exports/directory exists - Check disk space availability
- Verify write permissions
- For CSV export: Must click "Apply to Full Dataset" first
- Check that transformed dataset shows "β Ready (N rows)"
- CSV checkbox disabled: Apply transformations to full dataset first
- Columns not showing: Check the "Removed vs New Columns" section in preview
- One-hot encoding not visible: Look for new columns like
ColumnName_value1,ColumnName_value2 - DatasetHandle error: Restart the app if backend connection issues occur
- Large dataset slow: Normal - check progress bar for status
Detailed documentation available in docs/:
- presentations/: PowerPoint presentations and architecture diagrams
- Ready-to-use presentation slides (18 slides)
- High-resolution architecture diagram (PNG, 300 DPI)
- Scripts to regenerate materials
- See
docs/presentations/README.mdfor details
- PROGRESS_TRACKER.md: Progress tracking component guide
- QUALITY_VISUALIZATION.md: Quality visualization system
- EXPORT_FUNCTIONALITY.md: Complete export system documentation
- UI_UX_ENHANCEMENTS_SUMMARY.md: All UI/UX improvements overview
- TRANSFORMATION_PREVIEW_FIX.md: Technical details of transformation preview fix
- HOW_TO_SEE_NEW_COLUMNS.md: Guide to viewing transformed columns
- UI_LAYOUT_DIAGRAM.md: Visual layout of transformation UI
- TRANSFORMATION_PREVIEW_VISUAL_GUIDE.md: Before/after comparison guide
- CSV_EXPORT_FEATURE.md: Complete CSV export documentation
- EXPORT_CSV_QUICK_GUIDE.md: Quick start guide for CSV export
- CSV_EXPORT_FIX.md: Technical fix for DatasetHandle backend
- MULTI_TRANSFORMATION_SELECTION.md: Multi-selection feature guide
Contributions welcome! Areas for improvement:
- Additional agent types (time series, text analysis)
- More export formats (PDF, Excel)
- Advanced visualizations
- Performance optimizations
- Custom workflow builders
[Add your license here]
Built with:
- LangChain - LLM orchestration
- LangGraph - Graph-based workflows
- Groq - Ultra-fast LLM inference
- Streamlit - Interactive web UI
- Plotly - Interactive visualizations
Current Version: 3.3.1
Last Updated: July 2026
- v3.3.1 (July 2026): π TimeSeriesAgent Phase 1 - temporal profiling, trend/seasonality detection, stationarity testing
- v3.3 (July 2026): π Excel file support (.xlsx, .xls), automatic sheet detection, seamless integration
- v3.2 (July 2026): π¦ Human-in-the-Loop approval gates, decision tracking, step-by-step agent review
- v3.1 (July 2026): Multi-transformation selection, CSV export, column visualization
- v3.0 (June 2026): Complete agent suite, progress tracking, quality visualizations
For questions, issues, or feature requests, please open an issue on the repository.