[feature suggestion] instead of using random datasets, it should use real datasets

[asb]

Almost all benchmark configurations set max_model_len or for TensorRT --max_seq_len, which controls the maximum supported length of request (inclusive of the prompt and any generated output). It is typically set to ISL + OSL + tiny_margin (where tiny_margin may be 20 or 200 tokens). This is also done in generate_sweep_configs.py. These options of course impact memory allocation and so maximum achievable batch size.

It's understandable for a benchmark to be showing something approaching the peak obtainable results, but tuning the inference engine to the precise benchmark workload in this way seems to be going against the idea that "We want server configs to reflect real world deployments as much as possible" and the goal "to provide benchmarks that both emulate real world applications as much as possible and reflect the continuous pace of software innovation.". I struggle to think of a real world application of DeepSeek that would be able to run with a ~2k maximum sequence length for instance.

[cquil11]

Yes, a good point and something we have discussed. This is sort of solved by no longer using random datasets and instead using real datasets. Something we are discussing.

[asb]

Using real datasets might address the issue, depending on whether the datasets cover a sufficiently large variety of ISLs (and expected OSLs) that make tuning to a specific length in a way that that could never be done in a real-world deployment is ineffective.

I think it would be better to set some explicit rules for benchmark participants on this kind of tuning (which had been suggested in my original title), and this may be necessary even if real datasets are used. The retitling of this issue to "feature suggestion] instead of using random datasets, it should use real datasets" doesn't really reflect my suggestion.