Clarify benchmark task definition and labeling rubric in README

[rafaelsideguide]

Several reviewers assumed truth_text represents full-page content. In reality, contractors labeled snippets from rendered pages: ~100 words of important_content and ~10 words of not_important_content. This needs to be explicit in the README to avoid misinterpretation of drift and scoring.

TODO:

• Improve docs https://github.com/firecrawl/scrape-evals/blob/main/datasets/README.md

[williambrach]

Hi @rafaelsideguide,

Thanks for opening the issue. I can create a PR to update the READMEs, but I think it would be better to have a conversation to verify first.

Methodology & Goals

• Explicitly define the input/output expectation: Is this testing Raw HTML -> Text or Cleaned HTML -> Markdown? (This is crucial as playwright_api extracts raw text while the firecrawl_api is utilising markdown).
• Coverage and Quality explanation what they means and achieve
• split into two separate datasets?

Dataset Integrity

• Address the data provenance concerns raised in issue [Discussion] Benchmark Reproducibility #1
• Add more in depth explanation for lie_text field and truth_text field
• New section for "Data Collection": Detail the human labeling rubric used to distinguish "Main Content" from "Boilerplate."

Metrics & Scoring

• [] Explain metrics, why these metrics, why they are useful, edge cases, limitations

Is there something else that would be nice to add?

[FrancescoSaverioZuppichini]

@williambrach the code is assuming markdown as input but convert it to text ->

scrape-evals/evals/analysis/quality_analyzer.py

Line 21 in a94764f

text = md

thus for the first point, there shouldn't be any issue

[williambrach]

@FrancescoSaverioZuppichini Yes, but then the baselines compared against Firecrawl are at a disadvantage because they are not utilizing a cleaning step (I am referring to Playwright specifically). A more relevant comparison (apples to apples) would be Raw HTML from Firecrawl, or Playwright combined with a cleaning lib (like Readability or Trafilatura).