The future of TTS and Media Overlays

[rickj]

Section

https://www.w3.org/TR/epub-rs-34/#sec-media-overlays

Describe the problem

With these things in mind:

• The timeline for 3.4 implementation and adoption
• The increasing advance of AI voicing for TTS
• The reluctance of specific markets to support media overlays (thinking non-trade, academic as an example)

Is the 3.4 release an opportunity to introduce a broader topic for what is currently Media Overlays, and suggest an approach in the spec to TTS as a whole that could be AI voicing with highlighting OR media overlays?

Describe the fix or new feature you propose

No response

[mattgarrish]

Are you asking for auto-synchronizing of TTS and text without using media overlays, @rickj?

It's possible with media overlays to synchronize text highlighting with tts playback by omitting an audio element, though I don't know that it has much support.

[sueneu]

@rickj I have seen TTS synchronized with text highlighting in epubs made by CircularFlo and read through Colibrio.

I am also interested in the future of TTS in Ebooks. I agree that AI voicing has the potential for advancing TTS. This could be an efficient and inexpensive way for publishers to add audio to their titles, making more books accessible to readers who want/need it.

Have you seen TTS support among commercial reading systems? Like Apple, Kindle, etc. I thought it was mostly available in reading systems made for specific uses, like textbooks, as you mentioned.

[llemeurfr]

@sueneu, I don't know what a commercial reading system is. Thorium Reader — a wide-usage reading system — provides TTS with text "highlights" in multiple variants (associated with different user preferences).
Text tokenisation and speech synthesis are programmed at the reading system level. Speech synthesis may be provided by the operating system (system voices) or by local or remote AI voices, depending on available APIs and budget.
I'm having trouble understanding how the EPUB standard can help the reading system do this. The only help can come from the embedding of SSML in the HTML content (but there are issues with the support of SSML by speech synthesis APIs).
Is it a pure EPUB standardization issue, or related to best practices? We can discuss this.

[sueneu]

@llemeurfr I did struggle with how to talk about reading systems that are in general use like Apple Books and Kindle, perhaps there is a better word than commercial

Thorium Reader — a wide-usage reading system — provides TTS with text "highlights" in multiple variants (associated with different user preferences)

This is wonderful! Looking forward to doing tests with it.

I'm having trouble understanding how the EPUB standard can help the reading system do this.

If publishers/creators want to use TTS, it would be helpful for them to understand how to do so. I know my own customers are very interested in having finer control over TTS than we’ve been able to achieve. A best practices document could be the place for this.

Could the EPUB standard be more specific about how reading systems are implementing TTS? For instance, recommending support for SSML? The TTS technology seems likely to get better and less expensive with AI. Its use would increase. Standardization would be helpful.

Do you see that as out of scope for the EPUB Standard? — I’m still learning the scope of the Specifications

[mattgarrish]

Standardization would be helpful.

We have https://www.w3.org/TR/epub-tts-10/

All that material got bumped to a note because there's no direct implementation of it in reading systems. (There was also an attempt to add a new set of SSML attributes to HTML that seems to have stalled.)

But like @llemeurfr I don't yet follow what we'd be standardizing or where. I can't imagine we want anything like media overlays for this, or even have to make the author do anything at all. But is it realistic that reading systems would all implement automatic text and tts highlighting the same way, which is probably the only way we could write a standard for how to implement the feature in reading systems. Or is this not a standardization request but more along the lines of an implementation note to describe how it could work to feed interest?

[sueneu]

@mattgarrish

no direct implementation of it in reading systems

So that means reading systems don’t implement this themselves, but use APIs like @llemeurfr mentioned? Pardon my lack of understanding here : )

even have to make the author do anything at all

One Use case for SSML Pronunciation
When there are single words or names in a book that would be pronounced differently than default of the main language.
For instance, someone refers to their “abuela” in the middle of a sentence that is otherwise in English. The English voice will mispronounce the Spanish word. If the ebook maker marks up “abuela” as a span and uses CSS to indicate that the language is Spanish, the reading system will switch AI voices. This is jarring to the listener. And inaccurate. Using SSML pronunciation would allow TTS to pronounce “abuela” without switching voices mid-sentence.

[mattgarrish]

So that means reading systems don’t implement this themselves, but use APIs

I honestly don't remember the details anymore, but there was a case in Japan of PLS lexicons being used as a dictionary of terms. I think the information was converted into a form the reading system could use, so the lexicons themselves weren't being used except as an intermediary. There aren't web APIs for using PLS that I know of, though.

even have to make the author do anything at all
One Use case for SSML Pronunciation

Right, what I was getting at was defining a markup language like SMIL and forcing the author to define the synchronization points and have markup in the content documents to synch to. It would be really cool if you could pick the synchronization level you prefer - word, sentence, paragraph - rather than live with only what the publisher produced. An intelligent reading system could potentially do that for you.

The technologies for pronunciation and voice control will always have their place, but it's not that they've never been developed. The roadblock has been a lack of implementation.

[murata2makoto]

I think that we should try to work together with 1EdTech QTI WG. They have very similar interest. Rick, how do you feel?

[rickj]

My intent in raising this topic is to help provide clarity around a space that has broadened considerably since the original migration of the Media Overlays foundations from DAISY into the EPUB specification. With the EPUB 3.4 addition of a dedicated “Text-to-Speech” section under Media Overlays processing, it may be time to reconsider how these two concepts are presented within the specification.

Specifically, should Text-to-Speech (TTS) continue to exist as a subsection of Media Overlays—or has it matured into a parallel technology that warrants equal treatment under a broader conceptual umbrella (for example, “Aural Renditions” or “Synchronized Narration”)?

TTS appears across multiple EPUB specifications:

• Media Overlays 3.3 – Reading Systems SHOULD use TTS when an audio sibling is missing, making this the only normative TTS requirement.
• EPUB 3 Text-to-Speech Enhancements 1.0 (WG Note) – Provides author-level guidance on pronunciation (SSML phonemes, lexicons, CSS Speech).
• Core EPUB 3.3 – Lists “aural renditions of content (text-to-speech)” as part of EPUB’s global language support.
• Reading Systems 3.3 – Recognizes TTS-driven playback, including audio-only reading systems.
• Accessibility 1.1 – Establishes structured content and semantics as a foundation for effective TTS rendering.

Currently, TTS is presented as a subordinate feature of Media Overlays, implying it is merely a fallback mechanism. However, TTS now functions as a primary reading mode in many systems—serving accessibility, personalization, and adaptive rendering use cases that go well beyond fallback behavior.

This structure creates:

• Conceptual ambiguity between synchronized narration (Media Overlays) and synthetic narration (TTS).
• Fragmentation of references across specs, making author and implementer guidance harder to locate.

We should use the opportunity of 3.4 to reframe TTS and Media Overlays as peer mechanisms under a new conceptual umbrella (e.g., “Aural Renditions” or “Speech Rendering”) within the EPUB specifications. This reorganization would:

1. Clarify hierarchy and conceptual boundaries.
2. Improve discoverability of TTS-related guidance.
3. Support unified accessibility and aural rendering goals.
4. Create space for future developments in generative or adaptive speech synthesis.

[rickj]

I fed the above into ChatGPT, and it suggested:

4. Audio and Speech Presentation
   4.1 Media Overlays (synchronized pre-recorded audio)
   4.2 Text-to-Speech (synthetic audio rendering)
   4.3 Shared processing and accessibility considerations

[GeorgeKerscher]

The importance of "read aloud" in reading systems must be emphasised. There is a lot we can do in best practices and possibly in the specification.

Making read aloud clearly separate from media overlays and the TTS screen readers provide is something I totally support.

[mattgarrish]

With the EPUB 3.4 addition of a dedicated “Text-to-Speech” section under Media Overlays processing, it may be time to reconsider how these two concepts are presented within the specification.

For the record, there's actually nothing new in media overlays in 3.4. The section on using TTS has been in the spec since 3.0.1.

The TTS note was a collation of various sections that were scattered around the specifications prior to 3.3. We did that because there's no practical support for the technologies. Our SSML attribute creation went nowhere, PLS was promising but has no implementation, and CSS Speech has been around seemingly forever with only limited support in Safari.

I'd be all for starting over with more useful guidance than recommending a bunch of non-working technologies, but we'll need to figure out what that means first.