Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 8990 bytes --]

Hi all,

I have fci-mode installed and enabled for programming modes.

When org exports to html, htmlize figures out the syntax highlighting of
the code by calling  =(funcall lang-mode)=.

That activates =fci-mode=.

=fci-mode= shows the fill column using a unicode character. The issue is
that org export to html exports that character as well.

Those characters show up in html as below!
http://i.imgur.com/8WplTqw.png

So the solution is to fix the =orx-html-fontify-code= function.

I have modified the function as below; my edits are just those 2 lines
added to the function. The 2 lines simply check if fci-mode is installed.
If installed it disables fci mode. Even if the package is not installed, no
error will be raised.

It will be great if this edit can be incorporated into the org source
code.

Thank you.



(defun org-html-fontify-code (code lang)
  "Color CODE with htmlize library.
CODE is a string representing the source code to colorize.  LANG
is the language used for CODE, as a string, or nil."
  (when code
    (cond
     ;; Case 1: No lang.  Possibly an example block.
     ((not lang)
      ;; Simple transcoding.
      (org-html-encode-plain-text code))
     ;; Case 2: No htmlize or an inferior version of htmlize
     ((not (and (require 'htmlize nil t) (fboundp
'htmlize-region-for-paste)))
      ;; Emit a warning.
      (message "Cannot fontify src block (htmlize.el >= 1.34 required)")
      ;; Simple transcoding.
      (org-html-encode-plain-text code))
     (t
      ;; Map language
      (setq lang (or (assoc-default lang org-src-lang-modes) lang))
      (let* ((lang-mode (and lang (intern (format "%s-mode" lang)))))
(cond
 ;; Case 1: Language is not associated with any Emacs mode
 ((not (functionp lang-mode))
  ;; Simple transcoding.
  (org-html-encode-plain-text code))
 ;; Case 2: Default.  Fontify code.
 (t
  ;; htmlize
  (setq code (with-temp-buffer
       ;; Switch to language-specific mode.
       (funcall lang-mode)

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
                       (when (require 'fill-column-indicator nil 'noerror)
                         (fci-mode -1))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

       (insert code)
       ;; Fontify buffer.
       (font-lock-fontify-buffer)
       ;; Remove formatting on newline characters.
       (save-excursion
 (let ((beg (point-min))
       (end (point-max)))
   (goto-char beg)
   (while (progn (end-of-line) (< (point) end))
     (put-text-property (point) (1+ (point)) 'face nil)
     (forward-char 1))))
       (org-src-mode)
       (set-buffer-modified-p nil)
       ;; Htmlize region.
       (org-html-htmlize-region-for-paste
(point-min) (point-max))))
  ;; Strip any enclosing <pre></pre> tags.
  (let* ((beg (and (string-match "\\`<pre[^>]*>\n*" code) (match-end 0)))
 (end (and beg (string-match "</pre>\\'" code))))
    (if (and beg end) (substring code beg end) code)))))))))



Remember to cover the basics, that is, what you expected to happen and
what in fact did happen.  You don't know how to make a good report?  See

     http://orgmode.org/manual/Feedback.html#Feedback

Your bug report will be posted to the Org-mode mailing list.
------------------------------------------------------------------------



Emacs  : GNU Emacs 24.3.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.10.4)
 of 2014-01-21
Package: Org-mode version 8.2.7c (8.2.7c-64-g01f736-elpa @
/home/kmodi/.emacs.d/elpa/org-20140915/)

current state:
==============
(setq
 org-src-lang-modes '(("dot" . graphviz-dot) ("systemverilog" . verilog)
("ocaml" . tuareg) ("elisp" . emacs-lisp)
                      ("ditaa" . artist) ("asymptote" . asy) ("dot" .
fundamental) ("sqlite" . sql)
                      ("calc" . fundamental) ("C" . c) ("cpp" . c++) ("C++"
. c++) ("screen" . shell-script))
 org-hide-leading-stars t
 org-latex-default-figure-position "H"
 org-reveal-mathjax t
 org-tab-first-hook '(org-hide-block-toggle-maybe
org-src-native-tab-command-maybe
                      org-babel-hide-result-toggle-maybe
org-babel-header-arg-expand)
 org-speed-command-hook '(org-speed-command-default-hook
org-babel-speed-command-hook)
 org-occur-hook '(org-first-headline-recenter)
 org-metaup-hook '(org-babel-load-in-session-maybe)
 org-html-format-drawer-function '(lambda (name contents) contents)
 org-log-done 'timestamp
 org-latex-minted-options '(("linenos") ("numbersep" "5pt") ("frame"
"none") ("framesep" "2mm"))
 org-latex-format-inlinetask-function 'ignore
 org-confirm-shell-link-function 'yes-or-no-p
 org-ascii-format-inlinetask-function 'org-ascii-format-inlinetask-default
 org-startup-folded 'showeverything
 org-latex-pdf-process '("xelatex -shell-escape %f" "xelatex -shell-escape
%f" "xelatex -shell-escape %f")
 org-pretty-entities t
 org-use-sub-superscripts '{}
 org-export-with-sub-superscripts '{}
 org-return-follows-link t
 org-latex-format-headline-function
'org-latex-format-headline-default-function
 org-todo-keyword-faces '(("TODO" . org-warning) ("SOMEDAY" . "#FFEF9F")
                          ("CANCELED" :foreground "#94BFF3" :weight bold
:strike-through t))
 org-capture-templates '(("t" "Project 1 Meeting Notes" entry
                          (file+datetree (concat project1-org-dir "/
dv_meeting_notes.org"))
                          "\n* %?\n  Entered on %U")
                         ("j" "Journal" entry (file+datetree (concat
org-directory "/journal.org"))
                          "\n* %?\n  Entered on %U")
                         ("n" "Note" entry (file (concat org-directory "/
notes.org"))
                          "\n* %?\n  Context:\n    %i\n  Entered on %U")
                         ("u" "UVM/System Verilog Notes" entry (file
(concat org-directory "/uvm.org"))
                          "\n* %?\n  Context:\n    %i\n  Entered on %U")
                         )
 org-after-todo-state-change-hook '(org-clock-out-if-current)
 org-latex-format-drawer-function '(lambda (name contents) contents)
 org-from-is-user-regexp "\\<Kaushal\\.Modi\\>"
 org-src-mode-hook '(org-src-babel-configure-edit-buffer
org-src-mode-configure-edit-buffer)
 org-agenda-before-write-hook '(org-agenda-add-entry-text)
 org-pretty-entities-include-sub-superscripts nil
 org-babel-pre-tangle-hook '(save-buffer)
 org-mode-hook '(turn-on-flyspell wrap-region-mode
turn-on-stripe-table-mode fci-mode er/add-org-mode-expansions
                 #[nil "\300\301\302\303\304$\207"
                   [org-add-hook change-major-mode-hook org-show-block-all
append local] 5]
                 #[nil "\300\301\302\303\304$\207"
                   [org-add-hook change-major-mode-hook
org-babel-show-result-all append local] 5]
                 org-babel-result-hide-spec org-babel-hide-all-hashes
turn-on-visual-line-mode)
 org-agenda-skip-comment-trees nil
 org-use-speed-commands t
 org-ascii-format-drawer-function '(lambda (name contents width) contents)
 org-ctrl-c-ctrl-c-hook '(org-babel-hash-at-point
org-babel-execute-safely-maybe)
 org-directory "/home/kmodi/org"
 org-enforce-todo-dependencies t
 org-cycle-hook '(org-cycle-hide-archived-subtrees org-cycle-hide-drawers
org-cycle-hide-inline-tasks
                  org-cycle-show-empty-lines
org-optimize-window-after-visibility-change)
 org-plantuml-jar-path "/home/kmodi/.emacs.d/plantuml/plantuml.7999.jar"
 org-confirm-elisp-link-not-regexp
"\\(.*switch\\-to\\-buffer.*\\|org-show\\)"
 org-export-before-processing-hook '(endless/update-includes)
 org-latex-create-formula-image-program 'imagemagick
 org-blank-before-new-entry '((heading) (plain-list-item))
 org-todo-keywords '((sequence "TODO" "SOMEDAY" "CANCELED" "DONE"))
 org-latex-packages-alist '(("" "parskip") ("" "minted") ("" "tikz")
("section" "placeins") ("mathscr" "eucal")
                            ("" "latexsym") ("" "float") ("" "caption"))
 org-babel-tangle-lang-exts '(("latex" . "tex") ("emacs-lisp" . "el"))
 org-confirm-elisp-link-function 'yes-or-no-p
 org-metadown-hook '(org-babel-pop-to-session-maybe)
 org-latex-listings 'minted
 org-ditaa-jar-path "/home/kmodi/.emacs.d/ditaa/ditaa0_9.jar"
 org-blocker-hook '(org-block-todo-from-children-or-siblings-or-parent)
 org-html-format-headline-function 'ignore
 org-export-with-smart-quotes t
 org-completion-use-ido t
 org-babel-load-languages '((ditaa . t) (plantuml . t) (latex . t) (dot .
t))
 org-html-format-inlinetask-function 'ignore
 org-agenda-mode-hook '((lambda nil (define-key org-agenda-mode-map "x"
(quote sacha/org-agenda-done))
                         (define-key org-agenda-mode-map "X" (quote
sacha/org-agenda-mark-done-and-add-followup))
                         (define-key org-agenda-mode-map "N" (quote
sacha/org-agenda-new)))
                        )
 org-agenda-files "/home/kmodi/org/agenda.files"
 org-clock-out-hook '(org-clock-remove-empty-clock-drawer)
 org-confirm-babel-evaluate 'my-org-confirm-babel-evaluate
 org-src-fontify-natively t
 )


--
Kaushal Modi

[-- Attachment #2: Type: text/html, Size: 27763 bytes --]

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Grant Rettke]

Works perfectly on this:

╭────
│ (print emacs-version)
│ (print org-version)
╰────

╭────
│ "24.3.1"
│
│ "8.2.7c"
╰────

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

Do you mean that the fix works perfectly?

Thanks.

--
Kaushal Modi
On Sep 25, 2014 7:55 PM, "Grant Rettke" <gcr@wisdomandwonder.com> wrote:

> Works perfectly on this:
>
> ╭────
> │ (print emacs-version)
> │ (print org-version)
> ╰────
>
> ╭────
> │ "24.3.1"
> │
> │ "8.2.7c"
> ╰────
>

[-- Attachment #2: Type: text/html, Size: 682 bytes --]

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Grant Rettke]

Yes. I faced the issue here:

https://github.com/alpaker/Fill-Column-Indicator/issues/45

On Thu, Sep 25, 2014 at 6:58 PM, Kaushal <kaushal.modi@gmail.com> wrote:
> Do you mean that the fix works perfectly?
>
> Thanks.
>
> --
> Kaushal Modi
>
> On Sep 25, 2014 7:55 PM, "Grant Rettke" <gcr@wisdomandwonder.com> wrote:
>>
>> Works perfectly on this:
>>
>> ╭────
>> │ (print emacs-version)
>> │ (print org-version)
>> ╰────
>>
>> ╭────
>> │ "24.3.1"
>> │
>> │ "8.2.7c"
>> ╰────



-- 
Grant Rettke
gcr@wisdomandwonder.com | http://www.wisdomandwonder.com/
“Wisdom begins in wonder.” --Socrates
((λ (x) (x x)) (λ (x) (x x)))
“Life has become immeasurably better since I have been forced to stop
taking it seriously.” --Thompson

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Thorsten Jolitz]

Grant Rettke <gcr@wisdomandwonder.com> writes:

> Yes. I faced the issue here:
>
> https://github.com/alpaker/Fill-Column-Indicator/issues/45

I don't use fci-mode, but have a long-standing problem with underscores
_ that once in a while appear almost randomly in html output that is
returned when calling emacsclient on the command-line (to export an org
file). 

This is really annoying, but also really hard to hunt down. There is
nothing in the exported Org buffer that seems to cause this underscore,
at least I see nothing special in the place where it appears, not even
with whitespace-mode on.

Could it be related to Org's export? Or rather to emacsclient first
converting whitespace to _ underscores and then underscores to
whitespace again ... and maybe missing an underscore sometimes (if I
understand the C sources correctly)?

I think I asked about this problem at least 2 times already (on Emacs
help too), but it seems nobody knows what this might be about. Maybe
related to the topic of this thread?

-- 
cheers,
Thorsten

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Nicolas Goaziou]

Hello,

Kaushal <kaushal.modi@gmail.com> writes:

> I have fci-mode installed and enabled for programming modes.
>
> When org exports to html, htmlize figures out the syntax highlighting of
> the code by calling  =(funcall lang-mode)=.
>
> That activates =fci-mode=.
>
> =fci-mode= shows the fill column using a unicode character. The issue is
> that org export to html exports that character as well.
>
> Those characters show up in html as below!
> http://i.imgur.com/8WplTqw.png
>
> So the solution is to fix the =orx-html-fontify-code= function.

I don't think so. `fill-column-indicator' is not even in core Emacs. We
shouldn't provide a workaround for every package out there.

Can't you simply disable this minor mode before exporting buffer, in
a hook such as `org-export-before-parsing-hook'?


Regards,

-- 
Nicolas Goaziou

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

The problem is that `(funcall prog-mode)' in `org-html-fontify-code'
enables fci-mode. So disabling it in `org-export-before-parsing-hook' won't
work.

But your reply gave me another idea..

I can remove enabling of fci-mode from my prog mode hook in
`org-export-before-parsing-hook'.

But then where can I rehook it? I can't find a hook like
`org-export-after-processing-hook'. Can you suggest a hook that can work
for this use case?

--
Kaushal Modi
On Sep 26, 2014 6:01 AM, "Nicolas Goaziou" <mail@nicolasgoaziou.fr> wrote:

> Hello,
>
> Kaushal <kaushal.modi@gmail.com> writes:
>
> > I have fci-mode installed and enabled for programming modes.
> >
> > When org exports to html, htmlize figures out the syntax highlighting of
> > the code by calling  =(funcall lang-mode)=.
> >
> > That activates =fci-mode=.
> >
> > =fci-mode= shows the fill column using a unicode character. The issue is
> > that org export to html exports that character as well.
> >
> > Those characters show up in html as below!
> > http://i.imgur.com/8WplTqw.png
> >
> > So the solution is to fix the =orx-html-fontify-code= function.
>
> I don't think so. `fill-column-indicator' is not even in core Emacs. We
> shouldn't provide a workaround for every package out there.
>
> Can't you simply disable this minor mode before exporting buffer, in
> a hook such as `org-export-before-parsing-hook'?
>
>
> Regards,
>
> --
> Nicolas Goaziou
>

[-- Attachment #2: Type: text/html, Size: 2036 bytes --]

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Aaron Ecay]

Hi Nicolas and Kaushal,

2014ko irailak 26an, Nicolas Goaziou-ek idatzi zuen:
> 
> Hello,
> 
> Kaushal <kaushal.modi@gmail.com> writes:
> 
>> I have fci-mode installed and enabled for programming modes.
>> 
>> When org exports to html, htmlize figures out the syntax highlighting of
>> the code by calling  =(funcall lang-mode)=.
>> 
>> That activates =fci-mode=.
>> 
>> =fci-mode= shows the fill column using a unicode character. The issue is
>> that org export to html exports that character as well.
>> 
>> Those characters show up in html as below!
>> http://i.imgur.com/8WplTqw.png
>> 
>> So the solution is to fix the =orx-html-fontify-code= function.
> 
> I don't think so. `fill-column-indicator' is not even in core Emacs. We
> shouldn't provide a workaround for every package out there.

For better or for worse, Bastien already installed this fix on master as
commit 44aaaad8, in response to Kaushal’s earlier bug report on this
issue.  See <http://mid.gmane.org/87oaw92y6s.fsf@bzg.ath.cx>.

The fix has not yet been merged to the release branch, so it won’t yet
show up in the ELPA version of org which Kaushal seems to be using.

-- 
Aaron Ecay

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Kaushal]

[-- Attachment #1: Type: text/plain, Size: 2740 bytes --]

Hi all,

I was finally able to solve this problem without tweaking anything inside
ox-html.el

I noticed that htmlize has provided few hooks and I used those.

Here is the code that I now have in my emacs init to disable fci (and
flyspell too) while htmlize-region is doing its job:

    (defvar modi/htmlize-initial-fci-state nil
      "Variable to store the state of `fci-mode' when `htmlize-buffer' is
called.")
    (defvar modi/htmlize-initial-flyspell-state nil
      "Variable to store the state of `flyspell-mode' when `htmlize-buffer'
is called.")

    (defun modi/htmlize-before-hook-fn ()
      (when (fboundp 'fci-mode)
        (setq modi/htmlize-initial-fci-state fci-mode)
        (when fci-mode
          (fci-mode -1)))
      (when (fboundp 'flyspell-mode)
        (setq modi/htmlize-initial-flyspell-state flyspell-mode)
        (when flyspell-mode
          (flyspell-mode -1))))
    (add-hook 'htmlize-before-hook #'modi/htmlize-before-hook-fn)

    (defun modi/htmlize-after-hook-fn ()
      (when (fboundp 'fci-mode)
        (when modi/htmlize-initial-fci-state
          (fci-mode 1)))
      (when (fboundp 'flyspell-mode)
        (when modi/htmlize-initial-flyspell-state
          (flyspell-mode 1))))
    (add-hook 'htmlize-after-hook #'modi/htmlize-after-hook-fn)

I would suggest that that fci-mode specific code be removed from the master.


On Sat, Sep 27, 2014 at 3:25 PM Aaron Ecay <aaronecay@gmail.com> wrote:

> Hi Nicolas and Kaushal,
>
> 2014ko irailak 26an, Nicolas Goaziou-ek idatzi zuen:
> >
> > Hello,
> >
> > Kaushal <kaushal.modi@gmail.com> writes:
> >
> >> I have fci-mode installed and enabled for programming modes.
> >>
> >> When org exports to html, htmlize figures out the syntax highlighting of
> >> the code by calling  =(funcall lang-mode)=.
> >>
> >> That activates =fci-mode=.
> >>
> >> =fci-mode= shows the fill column using a unicode character. The issue is
> >> that org export to html exports that character as well.
> >>
> >> Those characters show up in html as below!
> >> http://i.imgur.com/8WplTqw.png
> >>
> >> So the solution is to fix the =orx-html-fontify-code= function.
> >
> > I don't think so. `fill-column-indicator' is not even in core Emacs. We
> > shouldn't provide a workaround for every package out there.
>
> For better or for worse, Bastien already installed this fix on master as
> commit 44aaaad8, in response to Kaushal’s earlier bug report on this
> issue.  See <http://mid.gmane.org/87oaw92y6s.fsf@bzg.ath.cx>.
>
> The fix has not yet been merged to the release branch, so it won’t yet
> show up in the ELPA version of org which Kaushal seems to be using.
>
> --
> Aaron Ecay
>

[-- Attachment #2: Type: text/html, Size: 3849 bytes --]

Re: Bug: Export to html inserts strange unicode characters at line breaks because of fci-mode [8.2.7c (8.2.7c-64-g01f736-elpa @ /home/kmodi/.emacs.d/elpa/org-20140915/)]

[Nicolas Goaziou]

Hello,

Kaushal <kaushal.modi@gmail.com> writes:

> I would suggest that that fci-mode specific code be removed from the
> master.

Done in 9540ff8d7a7a480b18dd1af94e7eb062f020b9ff. Thank you.


Regards,

-- 
Nicolas Goaziou