Re: ox-md.el: Export TOC and Footnotes as Markdown rather than HTML

[Jake Romer <jkrmr.io@gmail.com>]

[-- Attachment #1.1: Type: text/plain, Size: 8665 bytes --]

Hi Nicolas,

Thanks for the quick feedback! Notes on your notes:


> However, AFAIU, rendering for footnote section is still HTML, albeit a
> lightweight one.


Ah yes, that's true—the enclosing section can be rendered in Markdown, but
the footnotes themselves (and their references) are in HTML. I think that
make sense because footnotes, afaik, aren't part of "vanilla" Markdown,
although some variants do support them.

I think this function should not be specific to footnote section header, i.e.,
> it could factor out the following code in `org-md-headline'


Good point -- I implemented this in the attached revision.

Ideally, this should handle the level and a scope so as to handle toc:3 or
> #+TOC: headlines local.


I'm not as familiar with Org's TOC facilities as I should be, so I'll
update to handle these cases after some research. I want to avoid letting
these patches get too big, too.

Here's the patch for generating the footnotes section as Markdown, let me
know what you think.

Cheers,

Jake




On Mon, Aug 8, 2016 at 9:35 AM, Nicolas Goaziou <mail@nicolasgoaziou.fr>
wrote:

> Hello,
>
> Jake Romer <jkrmr.io@gmail.com> writes:
>
> > I notice that in Org 8.3, `org-md-export-as-markdown` and
> > `org-md-export-to-markdown` render a document's Table of Contents and
> > Footnotes sections as HTML rather than Markdown.
>
> Correct.
>
> > I have a couple of patches that change this behavior so that both are
> > rendered as Markdown. I'd love to hear any thoughts or suggestions for
> > improvement if you think this would be useful to include in ox-md.el.
>
> That's very interesting. Thank you. Some comments follow.
>
> However, AFAIU, rendering for footnote section is still HTML, albeit
> a lightweight one.
>
> > From b64d21e6b5bb35b6446abf37233463e40df040c3 Mon Sep 17 00:00:00 2001
> > From: Jake Romer <jkrmr@github.com>
> > Date: Sun, 7 Aug 2016 16:04:39 -0400
> > Subject: [PATCH 1/2] Export Footnotes section as Markdown
>
> The commit message has to contain the name of new and modified
> functions. See commit log for examples.
> >
> > ---
> >  ox-md.el | 74 ++++++++++++++++++++++++++++++
> ++++++++++++++++++++++++++++++++--
> >  1 file changed, 72 insertions(+), 2 deletions(-)
> >
> > diff --git a/ox-md.el b/ox-md.el
> > index 0aaade6..865e123 100644
> > --- a/ox-md.el
> > +++ b/ox-md.el
> > @@ -50,6 +50,20 @@ This variable can be set to either `atx' or `setext'."
> >         (const :tag "Use \"atx\" style" atx)
> >         (const :tag "Use \"Setext\" style" setext)))
> >
> > +;;;; Footnotes
> > +
> > +(defcustom org-md-footnote-format "<sup>%s</sup>"
> > +  "The format for the footnote reference.
> > +%s will be replaced by the footnote reference itself."
> > +  :group 'org-export-md
> > +  :type 'string)
> > +
>
> > +(defcustom org-md-footnote-section-title "Footnotes"
> > +  "The title for the Footnotes section.
> > +Example: `Footnotes', `References', `Sources', etc."
> > +  :group 'org-export-md
> > +  :type 'string)
>
> I suggest to ignore the variable above and use an equivalent of
>
>    (org-html--translate "Table of Contents" info)
>
> instead.
>
> > +(defun org-md-footnote-section-header (info)
> > +  "Renders a template for the footnotes section header in the preferred
> style.
> > +INFO is used as a communication channel."
> > +  (let ((style (plist-get info :md-headline-style))
> > +        (section-title (plist-get info :md-footnote-section-title)))
> > +    (cond
> > +     ((equal style 'atx) (format "\n%s %s\n%%s\n" "##" section-title))
> > +     ((equal style 'setext) (format "\n%s\n%s\n%%s\n"
> > +                                    section-title
> > +                                    (make-string (length section-title)
> ?-))))))
>
> (if (eq style 'atx)
>     ...
>  ...)
>
> I think this function should not be specific to footnote section header,
> i.e., it could factor out the following code in `org-md-headline'
>
>        ;; Use "Setext" style.
>        ((eq style 'setext)
>         (concat heading tags anchor "\n"
>                 (make-string (length heading) (if (= level 1) ?= ?-))
>                 "\n\n"
>                 contents))
>        ;; Use "atx" style.
>        (t (concat (make-string level ?#) " " heading tags anchor "\n\n"
>                   contents))
>
> > +;;;; Footnotes Section
> > +
> > +(defun org-md-footnote-section (info)
> > +  "Format the footnote section as Markdown.
> > +INFO is a plist used as a communication channel."
> > +  (let* ((fn-alist (org-export-collect-footnote-definitions info))
> > +         (fn-alist
> > +          (loop for (n type raw) in fn-alist collect
> > +                (cons n (org-trim (org-export-data raw info))))))
> > +    (when fn-alist
> > +      (format
> > +       (org-md-footnote-section-header info)
> > +       (format
> > +        "\n%s\n"
> > +        (mapconcat
> > +         (lambda (fn)
> > +           (let ((n (car fn)) (def (cdr fn)))
> > +             (format
> > +              "%s %s\n"
> > +              (format
> > +               (plist-get info :md-footnote-format)
> > +               (org-html--anchor
> > +                (format "fn.%d" n)
> > +                n
> > +                (format " href=\"#fnr.%d\"" n)
> > +                info))
> > +              def)))
> > +         fn-alist
> > +         "\n"))))))
> > +
> > +
> >  ;;;; Template
> >
> >  (defun org-md-inner-template (contents info)
> > @@ -474,7 +536,15 @@ CONTENTS is the transcoded contents string.  INFO
> is a plist
> >  holding export options."
> >    ;; Make sure CONTENTS is separated from table of contents and
> >    ;; footnotes with at least a blank line.
> > -  (org-trim (org-html-inner-template (concat "\n" contents "\n") info)))
> > +  (let* ((depth (plist-get info :with-toc))
> > +         (headlines (and depth (org-export-collect-headlines info
> depth)))
> > +         (toc-string (org-html-toc depth info))
> > +         (toc-tail (if headlines "\n\n" ""))
> > +         (footnotes (org-md-footnote-section info)))
> > +    (org-trim (concat toc-string
> > +                      toc-tail
> > +                      contents
> > +                      footnotes))))
> >
> >  (defun org-md-template (contents info)
> >    "Return complete document string after Markdown conversion.
> > --
> > 2.9.2
> >
> >
> > From 31091e4bd4b48d1394482a1542e6d90abf04b32d Mon Sep 17 00:00:00 2001
> > From: Jake Romer <jkrmr@github.com>
> > Date: Sun, 7 Aug 2016 16:15:50 -0400
> > Subject: [PATCH 2/2] Export Table of Contents as Markdown
>
> This commit message is also incomplete.
> >
> > ---
> >  ox-md.el | 15 ++++++++++++++-
> >  1 file changed, 14 insertions(+), 1 deletion(-)
> >
> > diff --git a/ox-md.el b/ox-md.el
> > index 865e123..0e2a499 100644
> > --- a/ox-md.el
> > +++ b/ox-md.el
> > @@ -528,6 +528,19 @@ INFO is a plist used as a communication channel."
> >           "\n"))))))
> >
> >
> > +;;;; Table of contents
> > +
> > +(defun org-md-format-toc (headline)
>
> Ideally, this should handethe level and a scope so as to handle toc:3
> or #+TOC: headlines local.
>
> > +  "Return an appropriate table of contents entry for HEADLINE.
> > +INFO is a plist used as a communication channel."
> > +  (let* ((title (org-export-data (org-export-get-alt-title headline
> info) info))
> > +         (level (1- (org-element-property :level headline)))
> > +         (indent (concat (make-string (* level 2) ? )))
>
> "? " -> "?\s"
>
> Besides, the indentation is slightly wrong. IIRC, 4 spaces are expected
> between two levels. See, e.g., `org-md-item'.
>
> > +         (anchor (or (org-element-property :CUSTOM_ID headline)
> > +                     (org-export-get-reference headline info))))
> > +    (concat indent "- [" title "]" "(#" anchor ")")))
> > +
> > +
> >  ;;;; Template
> >
> >  (defun org-md-inner-template (contents info)
> > @@ -538,7 +551,7 @@ holding export options."
> >    ;; footnotes with at least a blank line.
> >    (let* ((depth (plist-get info :with-toc))
> >           (headlines (and depth (org-export-collect-headlines info
> depth)))
> > -         (toc-string (org-html-toc depth info))
> > +         (toc-string (or (mapconcat 'org-md-format-toc headlines "\n")
> ""))
>
> #'org-md-format-toc
>
> >           (toc-tail (if headlines "\n\n" ""))
>
> Maybe a better abstraction would be to let `org-md-format-toc' handle
> toc-string and toc-tail.
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #1.2: Type: text/html, Size: 11880 bytes --]

[-- Attachment #2: render-fn-as-md.patch --]
[-- Type: application/octet-stream, Size: 8416 bytes --]

From 63d985425d3995985bd340fe195d8e25bd7f2cc6 Mon Sep 17 00:00:00 2001
From: Jake Romer <jkrmr@github.com>
Date: Sat, 13 Aug 2016 11:45:34 -0400
Subject: [PATCH 1/4] Extract org-md-headline-title to generate title

In order to generate a headline title in the preferred Markdown
style ('atx or 'setext), I extracted this bit of logic from
`org-md-headline'.

Since `org-md-headline' doesn't process the headline for the footnotes
section, extracting `org-md-headline-title' will allow us to generate
the Footnote section header in the preferred Markdown style without
duplicating the logic for doing so.

As an enhancement, `org-md-headline-title' places the anchor tag above
the section header. This makes it so that in a browser, clicking on the
TOC entry for that section scrolls to the section header without
obscuring it.

Example generated section title:

    <a id="orgheadline1"></a>

    Summary of Changes
    ==================
---
 ox-md.el | 37 +++++++++++++++++++++++--------------
 1 file changed, 23 insertions(+), 14 deletions(-)

diff --git a/ox-md.el b/ox-md.el
index 0aaade6..4bd445c 100644
--- a/ox-md.el
+++ b/ox-md.el
@@ -216,20 +216,29 @@ a communication channel."
 			  (car (last (org-export-get-headline-number
 				      headline info))))
 			 "."))))
-	  (concat bullet (make-string (- 4 (length bullet)) ?\s) heading tags
-		  "\n\n"
-		  (and contents
-		       (replace-regexp-in-string "^" "    " contents)))))
-       ;; Use "Setext" style.
-       ((eq style 'setext)
-	(concat heading tags anchor "\n"
-		(make-string (length heading) (if (= level 1) ?= ?-))
-		"\n\n"
-		contents))
-       ;; Use "atx" style.
-       (t (concat (make-string level ?#) " " heading tags anchor "\n\n"
-		  contents))))))
-
+	  (concat bullet (make-string (- 4 (length bullet)) ?\s) heading tags "\n\n"
+		  (and contents (replace-regexp-in-string "^" "    " contents)))))
+       (t (concat (org-md-headline-title style level title anchor tags) contents))))))
+
+
+;; Headline Title
+
+(defun org-md-headline-title (style level title &optional anchor tags)
+  "Generate a headline title in the preferred Markdown headline style.
+STYLE is the preferred style ('atx or 'setext)
+LEVEL is the header level.
+TITLE is the headline title.
+ANCHOR is the HTML anchor tag for the section as a string.
+TAGS are the tags set on the section."
+  (let ((anchor-lines (if anchor (concat anchor "\n\n") nil)))
+    ;; Use "Setext" style
+    (if (and (eq style 'setext) (< level 3))
+        (let* ((underline-char (if (= level 1) ?= ?-))
+               (underline (concat (make-string (length title) underline-char) "\n")))
+          (concat "\n" anchor-lines title tags "\n" underline "\n"))
+        ;; Use "Atx" style
+        (let ((level-mark (make-string level ?#)))
+          (concat "\n" anchor-lines level-mark " " title tags "\n\n")))))
 
 ;;;; Horizontal Rule
 
-- 
2.9.2


From 10351a0eda50e921978329c0ecf94540188215b1 Mon Sep 17 00:00:00 2001
From: Jake Romer <jkrmr@github.com>
Date: Sat, 13 Aug 2016 11:50:28 -0400
Subject: [PATCH 2/4] Add customizable footnotes-related variables

Taking a cue from `ox-html.el', this patch adds
`org-md-footnotes-section' and `org-md-footnote-format' to allow user
customization of the formatting for the footnotes section and individual
footnotes, respectively.
---
 ox-md.el | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/ox-md.el b/ox-md.el
index 4bd445c..9a7182a 100644
--- a/ox-md.el
+++ b/ox-md.el
@@ -51,6 +51,21 @@ This variable can be set to either `atx' or `setext'."
 	  (const :tag "Use \"Setext\" style" setext)))
 
 
+;;;; Footnotes
+
+(defcustom org-md-footnotes-section "%s%s"
+  "Format for the footnotes section.
+The first %s placeholder will be replaced with the localized Footnotes section
+heading, the second with the contents of the Footnotes section."
+ :group 'org-export-md
+  :type 'string)
+
+(defcustom org-md-footnote-format "<sup>%s</sup>"
+  "The format for the footnote reference.
+The %s will be replaced by the footnote reference itself."
+  :group 'org-export-md
+  :type 'string)
+
 \f
 ;;; Define Back-End
 
-- 
2.9.2


From 4f6f932930f18228aa57c02ae8855ba96f5fce95 Mon Sep 17 00:00:00 2001
From: Jake Romer <jkrmr@github.com>
Date: Sat, 13 Aug 2016 11:52:33 -0400
Subject: [PATCH 3/4] Add footnote-generating functions

Introduces the following functions:

* `org-md-footnote-formatted'
* `org-md-footnote-section'

`org-md-footnote-formatted' formats a single footnote entry using the
user-customizable template string `org-md-footnote-format'.

Example:

    <sup><a id="fn.1" href="#fnr.1">1</a></sup> This is a footnote.

`org-md-footnote-section' formats the entire footnote section using the
user-customizable template string `org-md-footnotes-section'.

Example:

    Footnotes
    =========

    <sup><a id="fn.1" href="#fnr.1">1</a></sup> This is a footnote.

    <sup><a id="fn.2" href="#fnr.2">2</a></sup> This is another footnote.
---
 ox-md.el | 28 ++++++++++++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/ox-md.el b/ox-md.el
index 9a7182a..19b8912 100644
--- a/ox-md.el
+++ b/ox-md.el
@@ -490,6 +490,34 @@ a communication channel."
   contents)
 
 
+;;;; Footnote Section
+
+(defun org-md-footnote-formatted (footnote info)
+  "Formats a single footnote entry FOOTNOTE.
+INFO is a plist with contextual information."
+  (let* ((fn-num (car footnote))
+         (fn-text (cdr footnote))
+         (fn-format (plist-get info :md-footnote-format))
+         (fn-anchor (format "fn.%d" fn-num))
+         (fn-href (format " href=\"#fnr.%d\"" fn-num))
+         (fn-link-to-ref (org-html--anchor fn-anchor fn-num fn-href info)))
+    (concat (format fn-format fn-link-to-ref) " " fn-text "\n")))
+
+(defun org-md-footnote-section (info)
+  "Format the footnote section.
+INFO is a plist used as a communication channel."
+  (let* ((fn-alist (org-export-collect-footnote-definitions info))
+         (fn-alist (loop for (n type raw) in fn-alist collect
+                         (cons n (org-trim (org-export-data raw info)))))
+         (headline-style (plist-get info :md-headline-style))
+         (section-title (org-html--translate "Footnotes" info)))
+    (when fn-alist
+      (format (plist-get info :md-footnotes-section)
+              (org-md-headline-title headline-style 1 section-title)
+              (mapconcat #'(lambda (fn) (org-md-footnote-formatted fn info))
+                         fn-alist
+                         "\n")))))
+
 ;;;; Template
 
 (defun org-md-inner-template (contents info)
-- 
2.9.2


From a2e353da09d0db9bbde14641bd94e12e156a143c Mon Sep 17 00:00:00 2001
From: Jake Romer <jkrmr@github.com>
Date: Sat, 13 Aug 2016 13:11:26 -0400
Subject: [PATCH 4/4] Update org-md-inner-template to Markdown footnotes

Updates `org-md-inner-template' to use `org-md-footnotes-section' to
generate a Markdown version of the footnotes section.

Also appends `:md-footnote-format' and `md-footnotes-section' to the
exporter :options-alist.
---
 ox-md.el | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/ox-md.el b/ox-md.el
index 19b8912..7d78ff7 100644
--- a/ox-md.el
+++ b/ox-md.el
@@ -105,7 +105,10 @@ The %s will be replaced by the footnote reference itself."
 		     (src-block . org-md-example-block)
 		     (template . org-md-template)
 		     (verbatim . org-md-verbatim))
-  :options-alist '((:md-headline-style nil nil org-md-headline-style)))
+  :options-alist
+  '((:md-headline-style nil nil org-md-headline-style)
+    (:md-footnote-format nil nil org-md-footnote-format)
+    (:md-footnotes-section nil nil org-md-footnotes-section)))
 
 \f
 ;;; Filters
@@ -526,7 +529,15 @@ CONTENTS is the transcoded contents string.  INFO is a plist
 holding export options."
   ;; Make sure CONTENTS is separated from table of contents and
   ;; footnotes with at least a blank line.
-  (org-trim (org-html-inner-template (concat "\n" contents "\n") info)))
+  (org-trim (concat
+             ;; Table of contents.
+             (let ((depth (plist-get info :with-toc)))
+               (when depth (org-html-toc depth info)))
+             ;; Document contents.
+             contents
+             "\n"
+             ;; Footnotes section.
+             (org-md-footnote-section info))))
 
 (defun org-md-template (contents info)
   "Return complete document string after Markdown conversion.
-- 
2.9.2