<Glazblog/>

TL;DR: the title says it all, and it's available there.

Eons ago, at a time BlueGriffon was only a Wysiwyg editor for the Web, my friend Mohamed Zergaoui asked why I was not turning BlueGriffon into an EPUB editor... I had been observing the electronic book market since the early days of Cytale and its Cybook but I was not involved into it on a daily basis. That seemed not only an excellent idea, but also a fairly workable one. EPUB is based on flavors of HTML so I would not have to reinvent the wheel.

I started diving into the EPUB specs the very same day, EPUB 2.0.1 (released in 2009) at that time. I immediately discovered a technology that was not far away from the Web but that was also clearly not the Web. In particular, I immediately saw that two crucial features were missing: it was impossible to aggregate a set of Web pages into a EPUB book through a trivial zip, and it was impossible to unzip a EPUB book and make it trivially readable inside a Web browser even with graceful degradation.

When the IDPF started working on EPUB 3.0 (with its 3.0.1 revision) and 3.1, I said this was coming too fast, and that the lack of Test Suites with interoperable implementations as we often have in W3C exit criteria was a critical issue. More importantly, the market was, in my opinion, not ready to absorb so quickly two major and one minor revisions of EPUB given the huge cost on both publishing chains and existing ebook bases. I also thought - and said - the EPUB 3.x specifications were suffering from clear technical issues, including the two missing features quoted above.

Today, times have changed and the Standards Committee that oversaw the future of EPUB, the IDPF, has now merged with the World Wide Web Consortium (W3C). As Jeff Jaffe, CEO of the W3C, said at that time,

Working together, Publishing@W3C will bring exciting new capabilities and features to the future of publishing, authoring and reading using Web technologies

Since the beginning of 2017, and with a steep acceleration during spring 2017, the Publishing@W3C activity has restarted work on the EPUB 3.x line and the future EPUB 4 line, creating a EPUB 3 Community Group (CG) for the former and a Publishing Working Group (WG) for the latter. If I had some reservations about the work division between these two entities, the whole thing seemed to be a very good idea. In fact, I started advocating for the merger between IDPF and W3C back in 2012, at a moment only a handful of people were willing to listen. It seemed to me that Publishing was a underrated first-class user of Web technologies and EPUB's growth was suffering from two critical ailments:

1. IDPF members were not at W3C so they could not confront their technical choices to browser vendors and the Web industry. It also meant they were inventing new solutions in a silo without bringing them to W3C standardization tables and too often without even knowing if the rendering engine vendors would implement them.
2. on another hand, W3C members had too little knowledge of the Publishing activity, that was historically quite skeptical about the Web... Working Groups at W3C were lacking ebook expertise and were therefore designing things without having ebooks in mind.

I was then particularly happy when the merger I advocated for was announced.

As I recently wrote on Medium, I am not any more. I am not convinced by the current approach implemented by Publishing@W3C on many counts:

• the organization of the Publishing@W3C activity, with a Publishing Business Group (BG) formally ruling (see Process section, second paragraph) the EPUB3 CG and a Steering Committee (see Process section, first paragraph) recreated the former IDPF structure inside W3C.  The BG Charter even says that it « advises W3C on the direction of current and future publishing activity work » as if the IDPF and W3C did not merge and as if W3C was still only a Liaison. It also says « the initial members of the Steering Committee shall be the individuals who served on IDPF’s Board of Directors immediately prior to the effective date of the Combination of IDPF with W3C », maintaining the silo we wanted to eliminate.
• the EPUB3 Community Group faces a major technical challenge, recently highlighted by representatives of the Japanese Publishing Industry: EPUB 3.1 represents too much of a technical change compared to EPUB 3.0.1 and is not implementable at a reasonable cost in a reasonable timeframe for them. Since EPUB 3 is recommended by the Japanese Government as the official ebook format in Japan, that's a bit of a blocker for EPUB 3.1 and its successors. The EPUB3 CG is then actively discussing a potential rescindment of EPUB 3.1, an extraction of the good bits we want to preserve, and the release of a EPUB 3.0.2 specification based on 3.0.1 plus those good bits. In short, the EPUB 3.1 line, that saw important clarifying changes from 3.0.1, is dead.
• the Publishing Working Group is working on a collection of specifications known as Web Publications (WP), Packaged Web Publications (PWP), and EPUB 4. What these specifications represent is extremely complicated to describe. With a daily observation of the activities of the Working Group, I still can't firmly say what they're up to, even if I am already convinced that some technological choices (for instance JSON-LD for manifests) are highly questionable and do not « lead Publishing to its full Web potential », to paraphrase the famous W3C motto. It must also be said that the EPUB 3.1 hiatus in the EPUB3 CG shakes the EPUB 4 plan to the ground, since it's now extremely clear the ebook market is not ready at all to move to yet another EPUB version, potentially incompatible with EPUB 3.x (for the record, backwards-compatibility in the EPUB world is a myth).
• the original sins of EPUB quoted above, including the two missing major features quoted in the second paragraph of the present article, are a minor requirement only. Editability of EPUB, one of the greatest flaws of that ecosystem, is still not a first-class requirement if not a requirement at all. Convergence with the Web is severely encumbered by personal agendas and technical choices made by one implementation vendor for its own sake; the whole W3C process based on consensus is worked around not because there is no consensus (the WG minutes show consensus all the time) but mostly beacause the rendering engine vendors are still not in the loop and their potential crucial contributions are sadly missed. And they are not in the loop because they don't understand a strategy that seems decorrelated from the Web; the financial impact of any commitment to the Publishing@W3C is then a understandable no-go.
• the original design choices of EPUB, using painful-to-edit-or-render XML dialects, were also an original sin. We're about to make the same mistake, again and again, either retaining things that partly block the software ecosystem or imagining new silos that won't be editable nor grokable by a Web Browser. Simplicity, Web-centricity and mainstream implementations are not in sight.

Since the whole organization of Publishing @W3C is governed by the merger agreement between IDPF and W3C, I do not expect to change anyone's mind with the present article. I only felt the need to express my opinion, in both public and private fora. Unsurprisingly, the feedback to my private warnings was fairly negative. In short, it works as expected and I should stop spitting in the soup. Well, if that works as expected, the expectations were pretty low, sorry to say, and were not worth a merger between two Standard Bodies.

I have then decided to work on a different format for electronic books, called WebBook. A format strictly based on Web technologies and when I say "Web technologies", I mean the most basic ones: html, CSS, JavaScript, SVG and friends; the class of specifications all Web authors use and master on a daily basis. Not all details are decided or even ironed, the proposal is still a work in progress at this point, but I know where I want to go to.

I will of course happily accept all feedback. If people like my idea, great! If people disagree with it, too bad for me but fine! At least during the early moments of my proposal, and because my guts tell me my goals are A Good Thing™️, I'm running this as a Benevolent Dictator, not as a consensus-based effort. Convince me and your suggestions will make it in.

I have started from a list of requirements, something that was never done that way in the EPUB world:

1. one URL is enough to retrieve a remote WebBook instance, there is no need to download every resource composing that instance

2. the contents of a WebBook instance can be placed inside a Web site’s directory and are directly readable by a Web browser using the URL for that directory

3. the contents of a WebBook instance can be placed inside a local directory and are directly readable by a Web browser opening its index.html or index.xhtml topmost file

4. each individual resource in a WebBook instance, on a Web site or on a local disk, is directly readable by a Web browser

5. any html document can be used as content document inside a WebBook instance, without restriction

6. any stylesheet, replaced resource (images, audio, video, etc.) or additional resource useable by a html document (JavaScript, manifests, etc.) can be used inside the navigation document or the content documents of a WebBook instance, without restriction

7. the navigation document and the content documents inside a WebBook instance can be created and edited by any html editor

8. the metadata, table of contents contained in the navigation document of a WebBook instance can be created and edited by any html editor

9. the WebBook specification is backwards-compatible

10. the WebBook specification is forwards-compatible, at the potential cost of graceful degradation of some content

11. WebBook instances can be recognized without having to detect their MIME type

12. it’s possible to deliver electronic books in a form that is compatible with both WebBook and EPUB 3.0.1

I also made a strong design choice: the Level 1 of the specification will not be a fit-all-cases document. WebBook will start small, simple and extensible, and each use case will be evaluated individually, sequentially and will result in light extensions at a speed the Publishing industry can bear with. So don't tell me WebBook Level 1 doesn't support a given type of ebook or is not at parity level with EPUB 3.x. It's on purpose.

With that said, the WebBook Level 1 is available here and, again, I am happily accepting Issues and PR on github. You'll find in the spec references to:

• « Moby Dick » released as a WebBook instance
• « Moby Dick » released as a EPUB3-compatible WebBook instance
• a script usable with Node.js to automagically convert a EPUB3 package into a EPUB3-compatible WebBook

My EPUB Editor BlueGriffon is already modified to deal with WebBook. The next public version will allow users to create EPUB3-compatible WebBooks.

I hope this proposal will show stakeholders of the Publishing@W3C activity another path to greater convergence with the Web is possible. Should this proposal be considered by them, I will of course happily contribute to the debate, and hopefully the solution.

• MathBird add-on to Thunderbird: insert MathML expressions into your messages. Available from here.
• JSCSSP, the CSS parser in JavaScript that does not use RegExps...
• BlueGriffon, the Open-Source Wysiwyg cross-platform Web editor based on Gecko
• commercial add-ons to BlueGriffon
• BlueGriffon EPUB3 Validator, an add-on to Firefox
• BlueGriffon EPUB Edition, the only native EPUB2/EPUB3 Wysiwyg cross-platform editor on the market (as of march 2013)
• FreeRecord, an add-on to Firefox to program the video recorder in your Freebox and much more
• Folding Tags library for CodeMirror
• a Firefox add-on adding a floating Table of Contents to W3C specs

Retired software section...

• NVu, once the leading Open-Source Wysiwyg cross-platform Web editor based on Gecko
• CSSize, a batch tool to transform (think XSLT) HTML documents based on CSS-like syntax.
• Meuf, one of the very first (right after Nathanael Borenstein's ANDREW, AFAIK...) Mail User Agents to fully implement MIME, multiple charsets, inline audio and video, X-Faces and more.
• HTTPtool, a file transfer app (ftp-alike) over HTTP GET and PUT
• a unofficial patch for the CERN v3.0 httpd server adding User Defined Error Messages

The EPUB 3 Content Documents spec reads:

The EPUB 3 CSS Profile includes @media and @import rules with media queries as defined in the Media Queries [MediaQueries] specification.

But it says nothing about stylesheets linked through a <link> element and Media Queries ! So, normatively per EPUB 3.0, <link> elements with Media Queries are currently forbidden...

If everything goes well, BlueGriffon EPUB Edition will be released 23-oct-2012 !

OS X, Windows, Linux. Full EPUB2 and EPUB3 metadata support. epub:type support. CSS Pro Editor. SVG Editor. And more ! Stay tuned !

Remember when I said the following?

any mechanism based on ID/IDREF introduces an extra keyword, the ID of the element ("mainauthor" here)... Asking a non-techie user to provide an ID is clearly suboptimal in terms of UX. That means the editing environment should pick one ID for the author, at the risk of using a human language not understood by the package's author or even a meaningless random ID.

Seems the situation is even worse than I expected... The EPUB3 Publications specification explicitly allows chains of subexpressions through ID/REF mechanisms and refines attributes. I guess it'll be clearer with this example:

<dc:identifier id="bookID">
  B64F547F-F2D1-4ACE-B8D2-D164785CDEB8
</dc:identifier>
<!-- meta element below refines the dc:identifier element above -->
<meta refines="#bookID"
      property="meta-auth"
      id="meta-authority1">My metadata Authority</meta>
<!-- link element below refines the meta element above... -->
<link refines="#meta-authority1"
      rel="xml-signature"
      href="../META-INF/signature.xml#Signature"/>

One ID/IDREF is already hard enough to deal with in UI but a chain?!? I can find one single word only to describe such a mechanism in a Standard oriented towards creation of visual products, i.e. oriented towards Wysiwyg-ness and nicely composed UIs: ridiculous. EPUB3 metadata are clearly a deep weakness in the EPUB3 format because of the incredible complexity they introduce in editing environments. Honestly, I am not sure to implement this, it would drastically uglify a UI that is already complex enough.

I have taken all packages available from the epub-samples EPUB3 repository and pushed them through my own EPUB3 validator. Some of these "clean" packages are not that clean, so here are the validation logs. Please note my validator also checks all stylesheets in the package according to the EPUB3 CSS Profile (AFAIK, no other validator is able to do that).

Validation of meta elements in EPUB3 triggers a few surprises... In fact, that's not the meta elements themselves, it's the properties they carry. Let me explain:

• EPUB3 meta elements carry a property attribute
• a property is a keyword optionally preceded by a property vocabulary prefix and a colon
• EPUB3 Media Overlays use the reserved media prefix mapped to namespace http://www.idpf.org/epub/vocab/overlays/#
• the Media Overlays says the media:duration property has a cardinality of "exactly one for the Publication and for each Media Overlay"

In fact, that's quite painful and costly to test in the following case:

1. suppose the namespace above is declared (yes, I know it's already reserved but it's not forbidden to do it !!!) on the package's prefix attribute with prefix foo
2. suppose we have one <meta property="media:duration"> in the OPF for the Publication
3. but also a <meta property="foo:duration"> for the same value...
4. that's invalid per spec, see the last item of the list above

You can't compare the prefixes, that's not enough. You really need to rely on URIs and that's where validation is expensive.

Conclusion: Media Overlays 3.0 have been released with EPUB3. The associated media prefix should be dropped and its related properties should be prefixless in EPUB3. If the issue is the media:narrator property too close to a value in another property vocabulary, it should be changed.

Hum, yet another bug in an EPUB3 spec... Excerpt (verbatim, nothing added or removed) from section 2.1.3.1.3 of EPUB Content Documents 3.0:

The prefix attribute

The prefix attribute definition is unchanged, but the attribute is defined to be in the namespace http://www.idpf.org/2007/ops when used in Content Documents.

Unchanged from what?!?

Dates and time... Dates and time are painful, dates and time are complex, "dates and time are the most complex objects we use on a daily basis" used to say Reuters' Misha Wolf in the HTML WG in the good ol'days.

Excerpt from section 2.2.7 of OPF 2.01:

The date element has one optional OPF event attribute. The set of values for event are not defined by this specification; possible values may include: creation, publication, and modification.

Excerpt from section 3.4.6 of EPUB Publications 3.0:

The date element must only be used to define the publication date of the EPUB Publication.
...
Only one date element is allowed.

Ahem... So EPUB3 allows only two dates for a package: the publication date and the last-modification date (through a <meta property="dcterms:modified"> element). It's impossible to preserve the date of creation or various modification dates. Is it only me or this change from EPUB2 to EPUB3 is weird?

Please note that to insert a publication date into an OPF document ready for distribution, you must modify it and rezip the whole package. So if an OPF document does have a <dc:date> publication element, its content value must be equal to the last-modified date!!! In other terms, this is totally useless. Wow.

Clearly, Wysiwyg-editability was, as too often in W3C specs, the last concern on EPUB3 spec editors' mind... EPUB3 is not a W3C spec but an IDPF spec, but the result is the same, unfortunately. I have a concrete example here: in EPUB2, here's how an author is specified in the OPF metadata of the package:

<dc:creator opf:file-as="Murakami, Haruki" opf:role="aut">Haruki Murakami</dc:creator>

That is quite easy to map into a simple and efficient UI.

But in EPUB3, that's pretty different since the properties refining the <dc:creator> element are expressed in standalone <meta> elements using an ID/IREF reference mechanism:

<dc:creator id="mainauthor">Haruki Murakami</dc:creator>
<meta refines="#mainauthor" property="role" scheme="marc:relators" id="role">aut</meta>
<meta refines="#mainauthor" property="alternate-script" xml:lang="ja">村上 春樹</meta>
<meta refines="#mainauthor" property="file-as">Murakami, Haruki</meta>
<meta refines="#mainauthor" property="display-seq">1</meta>

Please note the alternate-script property that was not expressable at all in EPUB2. Please also note the display-seq property that allows to specify an rendering order for the element (in the list of creators). These are cool features but...

1. any mechanism based on ID/IDREF introduces an extra keyword, the ID of the element ("mainauthor" here)... Asking a non-techie user to provide an ID is clearly suboptimal in terms of UX. That means the editing environment should pick one ID for the author, at the risk of using a human language not understood by the package's author or even a meaningless random ID.
2. there can be multiple alternate-script properties and that is really, really tricky to offer in a simple and clean UI.
3. CSS cannot style the above since there is no ID/IDREF mechanism in CSS; I understand that it was difficult to change the content model of DC elements but still, this is really a pity.
4. the display-seq property  is absolutely suboptimal: since <dc:creator> elements can be refined by a role property, the display sequence should apply for all elements of the same localName having the same role and not only all elements of the same localName; having the display sequence's value in PCDATA instead of inside an attribute on the <dc:creator> itself seems to me a design error. Even worse, the spec says in section 4.3.2 that "When the display-seq property is attached to some, but not all, of the members in a set, only the elements identified as having a sequence should be included in any rendering". Weird!
5. dealing with such metadata is expensive: for each <dc:creator> found, you have to look for all <meta> elements refining it. Please note the spec does not say what happens to a <meta> element when the IRI in the refines attributes has no target (I think such element should be made invalid).
6. unless I missed it, I think nothing is said about conflicting properties; for instance a creator having two roles specified, "auth" and ""pmn".

All in all, EPUB3 creator/contributor metadata are painful to deal with and edit, using ID/IDREF mechanisms in a specification (also) made for authoring environments while we have the full power of XML to avoid them seems to me a strategic error.

Don't get me wrong, I do understand why the <meta> and its refines attribute were introduced. But I think the cost to pay for that is too high.

1. we probably don't need a potentially multivalued alternate-script property. I think a monovalued original-script attribute on <dc:creator> or <dc:contributor> was probably enough.
2. file-as and role were perfect as attributes in EPUB2. Having the possibility to declare a scheme for role seems to me useless, most package authors will use marc roles anyway for compatibility reasons with EPUB2. file-as meta property is, compared to the file-as attribute of EPUB2, useless bloat in terms of footprint, speed of access, editability and maintainability.
3. the display-seq property seems to me far from meeting package authors' expectations. Its specification makes it painful to deal with since creators/contributors can become hidden if they have no display sequence while others have one. This property seems to me useless and the ordering of similar elements inside the <metadata> element is most certainly enough. I even bet that this feature will be drastically underused.
4. all in all, that says the <meta> element of EPUB3 is a suboptimal solution for problems touching only an extreme minority of package authors. The complexity it induces is, in my humble opinion, counter-productive and EPUB2 metadata were better designed and specified.