[BFCL] How should we calculate the overall accuracy for V2-Live dataset?

[HuanzhiMao]

Would love to hear the thoughts from the community on this matter:

BFCL V2 • Live dataset features a diverse set of 2,251 question-function-answer pairs. It comprises of 258 simple, 7 multiple, 16 parallel, 24 parallel multiple, 875 irrelevance detection, and 41 relevance detection entries.
When determining the ast_summary score for the Live leaderboard (with only live categories), we take the weighted average of the four AST categories (simple, multiple. parallel. parallel multiple). This is because we want the summary score to reflect the real-life composition. If people tend to do more multiple, then the multiple category should carry more weight.

The question is: How should we calculate the overall_accuracy for the BFCL V2 • Live dataset?

Currently, it is calculated as the weighted average of all Live categories (simple, multiple. parallel. parallel multiple, irrelevance, relevance). However, because the irrelevance category accounts for about half of the total entries, it would "unfairly" benefit the model that tends to get parsing/decoding errors. For some weak models that don't follow the formatting instructions (and thus the model handler cannot correctly decode their response), they would get almost 100% in the irrelevance category and that means at least half points in the overall_accuracy field. This is not ideal because the number of entries for irrelevance and relevance do not accurately reflect the real-life distributions, but rather, picked by us. So weight by the entry count seems unfair.

[akshita-sukhlecha]

The point on irrelevance category unfairly improving the overall score is valid.

One way we can make overall_accuracy more realistic is by having five categories: simple, multiple, parallel, parallel multiple, effective_relevance.
And effective_relevance can be the harmonic mean of irrelevance and relevance (just like F1 score)

[leocnj]

For Live_Acc, can we still follow Non-Live_Acc to just include AST scores?

Today, I used BFCL v3 to measure an instruct model (before fine-tuning to booster its function calling performance) and obtained the following results.

Metric	Value
Non-Live_Acc	2.56
Multi_Turn_Acc	0
Live_Acc	32.31
Relevance	41.18
Irrelevance	83.96

For Live case, scores on simple, multiple, parallel, parallel multiple are close to zero. However, due to including Relevance related scores when computing Live_Acc, we will end up on 32.31, too far from the score on both Non-Live and Multi_turn.

[HuanzhiMao]

For Live_Acc, can we still follow Non-Live_Acc to just include AST scores?

Today, I used BFCL v3 to measure an instruct model (before fine-tuning to booster its function calling performance) and obtained the following results.

Metric Value
Non-Live_Acc 2.56
Multi_Turn_Acc 0
Live_Acc 32.31
Relevance 41.18
Irrelevance 83.96
For Live case, scores on simple, multiple, parallel, parallel multiple are close to zero. However, due to including Relevance related scores when computing Live_Acc, we will end up on 32.31, too far from the score on both Non-Live and Multi_turn.

Valid point @leocnj . Could you raise an issue to help keep track? We will address that in the v4 release.