My app crashes after exceeding the 4096 token context window of Apple Foundation Model

[jadams777]

Questions:

• Is there a way to increase the context window size?
• Is there a programmatic way to prevent exceeding the available context window? This would prevent crashes.

[grabbou]

Yeah, this is known limitations by Apple. I should’ve provided a safety mechanism around context overflow. Will look into it when back next week. For now, I’d suggest counting tokens manually.

[jadams777]

Thank you. Codex estimates three characters per token, but says it's just a guess.

[grabbou]

Yup, we don’t know how Apple tokenizes input so all we can do at this point is rough estimation too.

[duochizhacai]

@grabbou any good new？