Firecrawl Research Index

An index for agents pushing the frontier of AI/ML research

Developer Tools

Artificial Intelligence

GitHub

Visit WebsiteSee on Product HuntGithub ⧉Twitter ⧉

I don't do ML research, I build production agents, but the core pain here is universal: agents quietly acting on stale or misranked sources and nobody noticing until it bites. Pairing each paper with the code that implements it, refreshed daily, is the clever bit. Curious whether you'll extend that same "source + the thing that implements it, kept current" idea beyond arXiv to general docs/APIs, because that's exactly where my agents drift. Nice launch.

The indexing isn’t the hard part, it’s the finding of what papers make earlier papers obsolete.

Or, not only that, it’s the understanding that many papers (even recent ones) are irrelevant. You’ll get agents touting results, only to read the paper and see that the conclusions are based on GPT 4o, which for many purposes has more in common with the Apollo guidance module than current models trained with reinforcement learning.

Okay fine, that’s a bit too hyperbolic, but really: the changes in training paradigms just make a huge corpus of research not very applicable to current LLMs, and that’s a point that models skip right over.

This is useful. For research agents, the part I’d want surfaced is provenance per downstream step: which paper, which repo or issue, which claim, and what changed in the experiment because of it.

Does /search/research return enough citation structure for an agent to keep that trail, or is that on the harness?

3M+ arXiv papers plus GitHub artifacts all in one index refreshed daily is seriously impressive. The recall benchmark results are pretty convincing too. I'm curious — does the index also cover papers from conferences like NeurIPS or ICML, or is it purely arXiv-based right now?

Getting state-of-the-art recall on arXivQA is legitimately hard. The tricky bit isn't crawling the PDFs. It's parsing structured content from LaTeX source vs. the rendered PDF without losing math notation and figure references. We've spent time on similar extraction challenges when pulling structured data from dense technical documents. What does your indexing pipeline use for equation and table extraction from arXiv source tarballs?

This is cool. How do you decide which GitHub repos qualify as top research artifacts?

Hey Eric! IT's truly impressive. 3M+ papers is more than enough to make researches that actually matter. Love to see you helping on this and wish you all the best!

@ericciarla congrats! Will definitely try it