Similify 🚀

Similarity, Simplified.

AI-powered duplicate news detection system built using NLP and TF-IDF similarity scoring.
Similify helps detect highly similar or duplicate articles instantly with accurate similarity scores.

---

📌 Overview

Similify is an intelligent content similarity detection system designed to:

• Detect duplicate or near-duplicate news articles
• Calculate similarity scores using TF-IDF + Cosine Similarity
• Help publishers maintain originality
• Prevent repeated content publishing
• Improve editorial quality control

The system exposes a FastAPI backend and can be integrated with web or mobile applications.

---

🧠 How It Works

1. Clean and preprocess text
2. Convert text into TF-IDF vectors
3. Compare new article with stored articles
4. Calculate cosine similarity score
5. Flag as duplicate if similarity ≥ threshold

---

🏗️ Beckend Project Architecture

Similify/
│
├── app.py                  # FastAPI application
├── clean_data.csv          # Preprocessed articles dataset
├── tfidf_matrix.pkl        # Stored TF-IDF matrix
├── vectorizer.pkl          # Saved TF-IDF vectorizer
├── utils.py                # Cleaning + similarity functions
├── requirements.txt        # Dependencies
└── README.md

---

⚙️ Tech Stack

• Python
• FastAPI
• Scikit-learn
• Pandas
• NumPy
• TF-IDF Vectorizer
• Cosine Similarity
• Uvicorn
• Ngrok (for public API testing)

---

✨ Features

• 🔎 Real-time article similarity detection
• 📊 Similarity score percentage output
• 🚨 Duplicate threshold control (default: 0.80 / 0.85)
• 🧹 Custom NLP cleaning pipeline
• ⚡ FastAPI REST API
• 🌍 Ngrok public endpoint support
• 📁 CSV-based dataset support
• 🔁 TF-IDF pre-trained model loading

---

📈 Industry Use Cases

• 📰 News Agencies (duplicate detection)
• 🏢 Content Publishing Platforms
• 📚 Academic Plagiarism Checking
• 🛒 Product Description Similarity Check
• 🤖 AI Content Validation Systems
• 🧾 Media Monitoring Systems

---

💼 Business Benefits

• Prevents duplicate content publishing
• Saves editorial review time
• Improves content credibility
• Maintains SEO quality
• Scalable to millions of documents
• Lightweight and cost-effective solution

---

📱 Running the React Native App (Similify Mobile)

Similify also includes a React Native mobile application that connects to the FastAPI backend for real-time duplicate detection.

---

🛠️ Prerequisites

Before running the mobile app, make sure you have installed:

• Node.js (LTS version recommended)
• npm or yarn
• Android Studio (for Android development)
• Xcode (Mac only – for iOS development)
• React Native CLI or Expo CLI (depending on your setup)

Verify installation:

node -v
npm -v

---

📦 Install Dependencies

Navigate to the mobile app directory:

cd similify-app
npm install

---

▶️ Start Metro Bundler

npx react-native start

---

📱 Run on Android

Make sure Android Emulator is running OR a physical device is connected.

npx react-native run-android

---

🍎 Run on iOS (Mac Only)

npx react-native run-ios

---

🌍 Connecting Mobile App to Backend

Update the API base URL inside your project:

const BASE_URL = "http://127.0.0.1:8000";

⚠️ Important Notes

If using Android Emulator, use:

const BASE_URL = "http://10.0.2.2:8000";

If using a real physical device, use your system’s local IP address:

const BASE_URL = "http://192.168.X.X:8000";

To find your local IP:

ipconfig   # Windows
ifconfig   # macOS/Linux

---

🌍 Using Ngrok for Public Testing

Start backend:

uvicorn app:app --reload

Start ngrok:

ngrok http 8000

Copy the generated HTTPS URL and update:

const BASE_URL = "https://your-ngrok-url.ngrok.io";

---

🧪 Troubleshooting

Clear Metro Cache

npx react-native start --reset-cache

Reinstall Node Modules

rm -rf node_modules
npm install

🚀 Backend Installation & Setup

1️⃣ Clone Repository

git clone https://github.com/shabbir0000/Similify_Model.git

2️⃣ Create Virtual Environment

python -m venv venv
source venv/bin/activate   # macOS/Linux
venv\Scripts\activate      # Windows

3️⃣ Install Dependencies

pip install -r requirements.txt

---

▶️ Running the API

uvicorn app:app --reload

Server will start at:

http://127.0.0.1:8000

---

🌍 Run with Ngrok (Public URL)

Install ngrok and run:

ngrok http 8000

You will get a public HTTPS URL like:

https://abcd1234.ngrok.io

Use this URL in your frontend/mobile app.

---

📡 API Endpoint

POST /check-duplicate

Request Body

{
  "text": "string",
  "top_n": 5
}

Response

{
  "is_duplicate": true,
  "results": [
    {
      "article_id": 101,
      "title": "Sample News Title",
      "similarity_score": 0.87,
      "url": "https://example.com"
    }
  ]
}

---

🎯 Duplicate Threshold

Default threshold: 0.80

You can modify it inside:

duplicate_threshold = 0.80

Recommended range:

• 0.75 → Loose matching
• 0.80 → Balanced (Recommended)
• 0.85 → Strict duplicate detection

---

🔮 Future Improvements

• Semantic similarity using Transformers (BERT)
• Elasticsearch integration
• MongoDB support
• Real-time streaming comparison
• Admin dashboard
• Similarity heatmap visualization

---

👨‍💻 Author

Muhammad Shabbir
AI Engineer | NLP & AI Automation Specialist
IBM Certified AI Professional