feat: noindex questions whose default project matches bot-only keywords

[SylvainChevalier]

Closes #4423.

Adds a noindex/nofollow robots meta directive to the individual question page when the default project's name (case-insensitively) contains any of:

• futureeval
• AI Forecasting Benchmark Tournament
• minibench

Tournament pages and notebooks remain indexable.

Generated with Claude Code

Summary by CodeRabbit

• SEO
  • Prevented search engines from indexing selected default project pages while still allowing existing links to be followed.
  • Preserved existing page titles, descriptions, and social sharing metadata for all question pages.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5f05a805-a4a2-47c9-9ff0-b09df2498fff

📥 Commits

Reviewing files that changed from the base of the PR and between 90742b1 and 84760fd.

📒 Files selected for processing (1)

• front_end/src/app/(main)/questions/[id]/[[...slug]]/page.tsx

---

📝 Walkthrough

Walkthrough

Question metadata now identifies configured project keywords case-insensitively and conditionally returns robots: { index: false, follow: false }; other metadata remains unchanged.

Changes

Question noindex metadata

Layer / File(s)	Summary
Keyword-based robots metadata front_end/src/app/(main)/questions/[id]/[[...slug]]/page.tsx	Adds default project keywords, derives the project name, and conditionally disables indexing in generated robots metadata.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Suggested reviewers: ncarazon

Poem

I’m a bunny guarding pages bright,
Keywords guide the robots’ flight.
Some may wander, some may stay,
Search bots hop along their way. 🐇

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly states the noindex behavior for bot-only question pages.
Linked Issues check	✅ Passed	The change noindexes question pages whose default project matches the specified FutureEval/AIB and MiniBench keywords, matching issue #4423.
Out of Scope Changes check	✅ Passed	The diff only adds keyword-based robots metadata handling for question pages and stays within the stated indexing scope.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/issue-4423-tell-google-to-stop-indexing

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

🚀 Preview Environment

Your preview environment is ready!

Resource	Details
🌐 Preview URL	https://metaculus-pr-5024-claude-issue-4423-tell-google-preview.mtcl.cc
📦 Docker Image	ghcr.io/metaculus/metaculus:claude-issue-4423-tell-google-to-stop-indexing-84760fd
🗄️ PostgreSQL	NeonDB branch preview/pr-5024-claude-issue-4423-tell-google
⚡ Redis	Fly Redis mtc-redis-pr-5024-claude-issue-4423-tell-google

Details

• Commit: 84760fdae44121f48bb8162a9fb08acb75ae796c
• Branch: claude/issue-4423-tell-google-to-stop-indexing
• Fly App: metaculus-pr-5024-claude-issue-4423-tell-google

---

ℹ️ Preview Environment Info

Isolation:

• PostgreSQL and Redis are fully isolated from production
• Each PR gets its own database branch and Redis instance
• Changes pushed to this PR will trigger a new deployment

Limitations:

• Background workers and cron jobs are not deployed in preview environments
• If you need to test background jobs, use Heroku staging environments

Cleanup:

• This preview will be automatically destroyed when the PR is closed

[hlbmtc]

@SylvainChevalier Should we exclude futureeval though? I remember we had a semi-related discussion with christian -- https://metaculus.slack.com/archives/C01Q9AQBVHB/p1784232763121349?thread_ts=1784230348.457349&cid=C01Q9AQBVHB