FlyRank BE Task 3 is a polite web scraper built for the FlyRank Backend AI Engineering internship (Week 5 — "The Polite Scraper"). It implements the full data-gathering pipeline — fetch → parse → extract → clean → structure — against a public practice site, wrapped in a professionalism layer that respects robots.txt, rate-limits itself, and identifies as a real bot rather than hiding behind a browser's User-Agent.
FlyRank BE Task 3 follows a Layered Pipeline Architecture, separating the scraping pipeline itself from the API that serves the resulting dataset.
| Layer | Responsibility |
|---|---|
| Scraper | Fetch → robots.txt check → parse → extract → clean → structure |
| Scripts | One-off CLI runner that executes the full scrape and saves the dataset |
| Controllers | Serves saved records and stats; exposes a capped live-scrape demo |
| Middleware | Logging, rate limiting, error handling, security |
| Data | Committed, structured JSON dataset (data/books.json) |
| Utils | Response formatting |
The API follows REST conventions with consistent JSON envelopes for every response and standard HTTP status codes. Query params drive search, filtering, and rating thresholds over the saved dataset — the same shape the scraper itself produces.
| Method | Endpoint | Description |
|---|---|---|
| GET | /records |
List saved records — filterable by search, inStock, minRating |
| GET | /records/:index |
Fetch a single saved record by index |
| GET | /stats |
Total, in-stock/out-of-stock counts, average price, average rating |
| GET | /scrape/run |
Runs a live, capped scrape (1–2 pages) for demo purposes — not persisted |
| Capability | Description |
|---|---|
| robots.txt Compliance | Checked before every single page fetch, not just once |
| Self-Throttling | Fixed delay between requests instead of firing concurrently |
| Bot Identification | Custom, descriptive User-Agent naming the bot and its purpose |
| Clean Extraction | Title, price, stock status, star rating, and image URL normalized per item |
| Structured Output | Committed JSON dataset, one clean record per product |
| Live Demo Endpoint | Capped live re-scrape for demonstration, without touching the saved dataset |
| Security Headers | Helmet-based HTTP header hardening |
| CORS Control | Cross-origin requests enabled |
| Rate Limiting | IP-based request throttling with custom JSON error responses |
| Centralized Error Handling | Single source of truth for error formatting and logging |
| Structured Logging | Per-request method, path, status, and latency logging |
| Graceful Shutdown | Clean process termination on SIGINT/SIGTERM with forced timeout |
| Automated Testing | Jest + Supertest coverage for the parser and every API endpoint |
| Live Demo UI | Interactive glass/3D interface to trigger and view real responses |
CLI: npm run scrape
↓
robots.txt check (per page)
↓
Polite HTTP fetch (delayed, identified)
↓
HTML parsed with Cheerio
↓
Fields extracted & cleaned
↓
Structured records saved to data/books.json
↓
API serves saved dataset (search / filter / stats)
| Layer | Technology |
|---|---|
| Runtime | Node.js |
| Framework | Express.js 5 |
| Scraping | Axios, Cheerio, robots-parser |
| Security | Helmet, CORS |
| Performance | Compression |
| Reliability | express-rate-limit |
| Testing | Jest, Supertest |
npm testTests run against local fixtures and the parser directly — no live network calls required.
Every page fetch is preceded by a fresh robots.txt check against that page's URL — not just checked once at startup. Requests are spaced out with a fixed delay rather than fired concurrently, and the scraper identifies itself with a descriptive User-Agent rather than impersonating a browser. The target site, books.toscrape.com, is built specifically for scraping practice and permits this kind of use.
https://fly-rank-be-task3.vercel.app/
Muhammad Ashhadullah Zaheer
LinkedIn: https://www.linkedin.com/in/muhammad-ashhadullah-zaheer-41194a340/