Balance chapter HTML and fix intro-material italics leaking into text

[LAfricain]

Follow-up to #921: while chapter intro material (book/section headings) is meant to render in italics via ".introMaterial { font- style: italic; }", two related bugs let that styling — and malformed markup in general — leak into the rest of the chapter on modules with rich introductions (e.g. FreCrampon).

• GTKChapDisp::introMaterial() (src/main/display.cc): SWORD's OSIS filter can emit an opening <div type="subSection" ...> for a section heading with no matching closing anywhere in the fetched text. Now counts

  /
  occurrences and appends any missing closing tags before closing our own wrapper, so the block is always self-balanced regardless of what the filter provides.

• wk_html_sanitize() (src/webkit/wk-html.c): parses the final assembled HTML with libxml2 (HTML_PARSE_RECOVER) and re-serializes it before handing it to WebKit, rather than letting WebKit's own parser silently repair mismatched tags unpredictably. General safety net for similar cases we may not have found yet.

• CSS: added ".introMaterial h2.chapterHeader { font-style: normal; }" so section/chapter titles stay upright — only the narrative intro text should be italic, not its headings.

[karlkleinpaste]

I had thought this change was benign so I merged it easily.

But now, having used it some more, I'm finding 2 serious problems.

• The code generates verse alignment anchors ("chapter * 1000 + verse"), and on navigation, sets top alignment to the current verse's anchor. With this latest update, anchors are not reliably top-aligned. (Sometimes they are. More often they are not.) With whole book rendering enabled, navigate to a sample spot in a large book; here, I used Deu 24:17. This top alignment does not occur at that point -- in my example, it "aligns" to page bottom. This is not specific to any module: NASB and ESV also go to page bottom.

• The placement of pre-verse headings is broken with respect to the text and to the end-of-chapter horizontal separator. See right column.

With change:

ESV, showing mis-aligned horizontal separator and excess whitespace following the next pre-verse heading:

With change reverted in a local branch, showing normal whitespace after pre-verse heading, and proper placement of horizontal separator:

Either I will revert this or an update is needed to restore correct anchor alignment and header + separator placement.

[LAfricain]

Let me work on it. Tomorrow or Monday.

[karlkleinpaste]

I really hate to add to this, but there is another issue of formatting. caused by this commit and continued by the fix for it that was merged a little while ago.

This is using Prefs → Fonts → Sizes and Faces → Justify margins.

I noticed, belatedly, that there is some line justification that has gotten absurd. These images are from Bible and commentary pane in latest build, showing 4-column at end of Genesis.

I used git bisect to find where this excessively-justified artifact began, and it happened with yesterday's first attempt at HTML balance.

git bisect bad
a3589177285025a68a5d491a47bf4baa02722b49 is the first bad commit

This is how things look after git reset --hard fe0cf25ab64f92a4b9644740abfc9bf93c84bb8f.

It's interesting to me that, in the commentary images, notes 1,2,3 are not overjustified but notes 4,5,6 are.

Something was done as part of this that causes a sort of overstatement of justification.

[karlkleinpaste]

Oh...dear...
In the first commentary image, the font size changes between notes 1,2,3 and notes 4,5,6.

[LAfricain]

I can't reproduce your issue??

[LAfricain]

See the theory of Claude (I can't test because I can't reproduice the issue):

Working theory: the change most likely responsible is the unconditional I wrapped around every verse's text to stop italics bleeding onto verse content (a fix that shipped in the follow-up commit, separate from the original div-balancing that caused your first report). Previously, verse text with no highlight/coloring had no wrapping inline element at all; adding one for every verse may be interfering with WebKit's last-line detection for text-align: justify. I'd like to replace it with a fix that doesn't add any new wrapping element around verse text at all -- balancing each fetched intro fragment individually, right where it's fetched, instead of stacking corrections at the end of the block.

Since I can't reproduce this locally, could you tell me:

• which commentary module you have open in those screenshots?
• whether it happens with whole-book rendering off, on a single chapter?
• whether it happens without a commentary pane open, Bible text alone?

In the meantime I'll prepare a candidate branch with the more targeted fix (no verse-wrapping span, localized div-balancing) and would appreciate if you could test it on your end before I ask for a merge, rather than repeating the merge-then-regress cycle from the last two rounds.

[LAfricain]

Can you try the last PR?

[karlkleinpaste]

Your screenshot shows the problem.

Near end of column 2.

Middle of column 3.

A little farther down column 3.

Modules are NET and NETnote. You're using NET; NETnote is also available from Xiphos repo.

Single chapter Gen 38.

[LAfricain]

Oh, right—I'd only noticed the ends of the paragraphs that didn't have the problem! So my fix isn't working.

[LAfricain]

Karl, it seems the problem is still NET. I don't see the problem with other module. And you?

[karlkleinpaste]

I'm glad of that, because I was about to say "it's NET only after all" and was going to schedule some time this evening to trace the generated HTML to find what's wrong, much as I had to do a few weeks ago for <p/>. If you've got a solution, I'll wait. I have to leave the house for the day, so I'll look here later.

[LAfricain]

Ok, something is wrong with the picture (chapter numbers are pictures).

[karlkleinpaste]

Let's take this from a different point of reference.

The original intent of these changes was to handle a (rare?) case where text returned from Sword was not self-consistent, insofar as it did not close <div> with </div> on its own. What and where is an example of that? Exactly where and how can I reproduce that error? What module, seen in which display, and with what options being set?

I find I really don't like the solution of wrapping every single verse in a font spec. That creates a great deal of extra work for WebKit and will slow down display considerably.

I am wanting to find a different solution, where the text returned from Sword is immediately repaired on the spot, so that no further analysis and repair is ever necessary. One and done.

And in any event, if Sword is returning improper text, a bug report should be filed against whatever part of Sword is returning unbalanced content.

[LAfricain]

I worked on this with Claude yesterday. He fixed the justification issue. Although I see there’s still some random justification on the last line. I wonder if it wasn’t already there before. Why did we take on this task? Well, because of a problem that’s actually quite serious. Chapter 1 of the module books appears in italics when the chapter titles are, for example, freCrampon, ktuvb, or freVulgglaire.

I think the simplest thing to do for now is probably to revert the PR. If you have a chance to look into what’s going on with FreCrampon, I’d appreciate it.

[karlkleinpaste]

Chapter 1 of the module books appears in italics when the chapter titles are, for example, freCrampon

I'm looking at FreCrampon and I must be misunderstanding what you're seeing.

Looks ordinary to me. No excess italics. Module settings are defaults. What should I be seeing as wrong?

Educate me a little further. What am I missing? I will chase it down, once I understand what's wrong.

[karlkleinpaste]

If FreCrampon is not an example of the problem, in what module can one be found?

[LAfricain]

I worked yesterday all day on the frecrampon, it was completely buggy. I really think that for now it's better to revert. But otherwise a module that has this italics problem is the ktuvb, chap. 1 of the books recovers the italics of the intros.

[karlkleinpaste]

This is KtuVB, from end of introductory material to beginning of Gen 1.

Intro italics ends with the start of text.

I'm just plain confused.

[LAfricain]

Yes indeed it's not very serious for the KTUVB, it's just the first title which is in italics, it's not very serious especially since it's the name of the book. So I repeat myself rather than waste time let's reverse the PR, to start again on a clean basis but with a correct NET and frecrampon module.

Unless you really see something relevant of course.

[karlkleinpaste]

Oh, "luyantiku" is italicized. I didn't even notice that until now. I was looking for italics in the text itself.

Well, anyway, as you say, I'll revert and get us back to a previously-known state.

[karlkleinpaste]

As regards KtuVB, where book names (only) are italicized...

I believe the problem is with the module markup. It's not happening in the XHTML generated by Xiphos. It's in how the pre-chapter content is encoded, and how Sword renders that into XHTML. It's not wrong, it's just unexpected.

Exodus 0:0, pre-book:

<div canonical="true" osisID="Exod" sID="gen799" type="book"/> <milestone n="Kub" type="x-usfm-h"/> <milestone n="MUKANDA YA KUBASIKA" type="x-usfm-toc1"/> <milestone n="KUBASIKA" type="x-usfm-toc2"/> <milestone n="Kub" type="x-usfm-toc3"/> <title level="1" type="main">KUBASIKA</title>

Sword renders this into XHTML:

<span class="x-usfm-h" data-n="Kub" data-type="x-usfm-h"></span> <span class="x-usfm-toc1" data-n="MUKANDA YA KUBASIKA" data-type="x-usfm-toc1"></span> <span class="x-usfm-toc2" data-n="KUBASIKA" data-type="x-usfm-toc2"></span> <span class="x-usfm-toc3" data-n="Kub" data-type="x-usfm-toc3"></span> <h1 class="bookHeader main">KUBASIKA</h1>

Most of this is simply <milestone> content converted into <span></span> blocks. But the <title> content becomes <h1 class="bookHeader main">KUBASIKA</h1>. I believe that this <h1> content is, by itself, what italicizes the book name.

I've traced carefully through the execution of GTKChapDisp::display and particularly GTKChapDisp::RenderOneChapter, and I see no evidence that Xiphos is inducing this, nor that Sword is returning unbalanced content from the OSIS source. It's just that a title becomes h1 which appears to have the italic attribute.

I've experimented with a simple change that adds <span style="font-style: normal;"> within <div dir=ltr>, enveloping the entire chapter content, which would appear to be the best minimalist forcing of non-italic content without doing so on a per-verse basis. But I don't want to commit such a thing because now I think it's unnecessary.

I'm open to contrary analysis and opinion.