org-mode export to (latex) PDF

org-mode export to (latex) PDF

[Jean-Christophe Helary]

I was busy last year going back to school and I wrote a research paper for my first year of master almost entirely in org-mode.

My workflow was trivial:

- write in org-mode
- enter the bibliography with Zotero
- export to ODT and open in NeoOffice
- modify in NeoOffice
- deliver

At first, I tried to export the document to PDF but all the Japanese quotes I had were removed from the document, which led me to prefer ODT export because it handled them properly.

I guess it is an issue with the Latex backend and could have been solved with the proper Latex settings, but it seems weird that the default settings do not allow for out-of-the-box multilingual support.

Is there an easy way to have that as the default behavior?

-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Hi Jean-Christophe,

Jean-Christophe Helary writes:

> I guess it is an issue with the Latex backend and could have been solved with the proper Latex settings, but it seems weird that the default settings do not allow for out-of-the-box multilingual support.

I agree with you that Org should have multilingual support. A few months
ago I started this thread here, with some proposals:
https://orgmode.org/list/87o8d95pvo.fsf@posteo.net/

Best regards,

Juan Manuel 

Re: org-mode export to (latex) PDF

[Jean-Christophe Helary]

> On Jul 10, 2021, at 22:52, Juan Manuel Macías <maciaschain@posteo.net> wrote:
> 
> Hi Jean-Christophe,
> 
> Jean-Christophe Helary writes:
> 
>> I guess it is an issue with the Latex backend and could have been solved with the proper Latex settings, but it seems weird that the default settings do not allow for out-of-the-box multilingual support.
> 
> I agree with you that Org should have multilingual support. A few months
> ago I started this thread here, with some proposals:
> https://orgmode.org/list/87o8d95pvo.fsf@posteo.net/

Juan,

That's a very interesting proposal. Thank you for the reference.

I had given up on Latex because mixing languages sounded like a huge pain in the butt but I see that without some org-level infrastructure it is not possible to achieve much when exporting to Latex/PDF (unless I missed something).

So I guess I'll just keep my current workflow for now, with ODT as my "pivot" format.

Thank you again, Juan.


-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Hi Jean-Christophe,

Jean-Christophe Helary writes:

> I had given up on Latex because mixing languages sounded like a huge
> pain in the butt but I see that without some org-level infrastructure
> it is not possible to achieve much when exporting to Latex/PDF (unless
> I missed something).

Well, LaTeX has excellent (typographic and orthotypographic)
multilingual support, using the babel or polyglossia packages. I
especially recommend babel:
http://mirrors.ctan.org/macros/latex/required/babel/base/babel.pdf

And LaTeX also has very good support for oriental languages or languages
with complex writing, especially in LuaTeX. In LuaTeX and XeTeX you can
also use opentype fonts and opentype features.

The problem is how to translate that from Org --in an org-centric way--
to LaTeX. Currently, you can apply LaTeX commands for multilingual
management directly in your Org document. For example:

#+LaTeX_Header: \usepackage[several langs]{babel}

@@latex:\begin{otherlanguage*}{german}@@

... some text in german ...

@@latex:\end{otherlanguage*}@@

Recently, I submitted a patch here that allows adding LaTeX attributes
to `quote' blocks. Then, you could do something like this:

#+LaTeX_Header:\usepackage[german,english]{babel}
#+LaTeX_Header:\usepackage{quoting}
#+LaTeX_Header:\usepackage[babel=true,autostyle=true,german=quotes]{csquotes}
#+LaTeX_Header:\SetBlockEnvironment{quoting}

#+ATTR_LaTeX: :environment foreigndisplayquote :options {german}
#+begin_quote
Eine Erklärung, wie sie einer Schrift in einer Vorrede nach der
Gewohnheit vorausgeschickt wird ---über den Zweck, den der Verfasser
sich in ihr vorgesetzt, sowie über die Veranlassungen und das
Verhältnis, worin er sie zu andern frühern oder gleichzeitigen
Behandlungen desselben Gegenstandes zu stehen glaubt--- scheint bei
einer philosophischen Schrift nicht nur überflüssig, sondern um der
Natur der Sache willen sogar unpassend und zweckwidrig zu sein (Hegel).
#+end_quote

Best regards,

Juan Manuel 

Re: org-mode export to (latex) PDF

[Tim Cross]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Hi Jean-Christophe,
>
> Jean-Christophe Helary writes:
>
>> I had given up on Latex because mixing languages sounded like a huge
>> pain in the butt but I see that without some org-level infrastructure
>> it is not possible to achieve much when exporting to Latex/PDF (unless
>> I missed something).
>
> Well, LaTeX has excellent (typographic and orthotypographic)
> multilingual support, using the babel or polyglossia packages. I
> especially recommend babel:
> http://mirrors.ctan.org/macros/latex/required/babel/base/babel.pdf
>
> And LaTeX also has very good support for oriental languages or languages
> with complex writing, especially in LuaTeX. In LuaTeX and XeTeX you can
> also use opentype fonts and opentype features.
>
> The problem is how to translate that from Org --in an org-centric way--
> to LaTeX. Currently, you can apply LaTeX commands for multilingual
> management directly in your Org document. For example:
>
> #+LaTeX_Header: \usepackage[several langs]{babel}
>
> @@latex:\begin{otherlanguage*}{german}@@
>
> ... some text in german ...
>
> @@latex:\end{otherlanguage*}@@
>
> Recently, I submitted a patch here that allows adding LaTeX attributes
> to `quote' blocks. Then, you could do something like this:
>
> #+LaTeX_Header:\usepackage[german,english]{babel}
> #+LaTeX_Header:\usepackage{quoting}
> #+LaTeX_Header:\usepackage[babel=true,autostyle=true,german=quotes]{csquotes}
> #+LaTeX_Header:\SetBlockEnvironment{quoting}

Just FYI for those who don't know, you can use the org-latex-classes
variable to define your own pseudo document classes, possibly using the
DEFAULT_PACKAGES, PACKAGES, EXTRA_PACKAGES macros and other latex
settings. So for example, you can add the babel or other packages you
want and either make that the 'default' class or specify which class you
want with the #+LATEX_CLASS header. I use this quite a bit because then
I don't have to remember which LATEX_HEADER lines to include in the
document, the specific option settings etc. I don't need support for
multilingual documents, but I do have a number of 'special' documents
(such as one with colours, logos and specific fonts for an employer to
match their 'style guide'. I also have ones for generating project
documents, letters, meeting minutes etc. They all use various different
Latex extensions (particularly ones which don't mix well and cannot be
included with other packages).

The LATEX_HEADER: options are useful for 'one off' documents, but become
a real pain to include all the time. however, I see this quite a lot and
just wanted to highlight that when you need to customise the latex
process, you do have these other options which can be very useful and I
suspect would be good for things like setting up support for
multilingual environments. I also use luatex rather than the default
plain 'latex' (mainly because of better/more flexible font support). 

I could be wrong as I've not looked at this in a long time, but one of
the problems with multilingual support in Latex was that it was somewhat
'fragile'. There were a number of packages which did not work well when
combined with certain fonts required for multilingual support and (from
memory) issues with hyphenation and packages which extended some
environments. While it was generally possible to tweak things to get
them to work, it was somewhat challenging to get them to work 'across
the board'. I don't know if this is still the case as it has been some
years incde I've needed to dig into Latex (mainly because now I do
almost everything just in org mode and don't need to!). This, combined
with a smaller user base for multilingual documents, may partly explain
the difficulty in gaining traction in this area. I do recall that
getting a stable general latex environment working for org exports was
somewhat challenging originally.

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Tim Cross writes:

> Just FYI for those who don't know, you can use the org-latex-classes
> variable to define your own pseudo document classes, possibly using the
> DEFAULT_PACKAGES, PACKAGES, EXTRA_PACKAGES macros and other latex
> settings. So for example, you can add the babel or other packages you
> want and either make that the 'default' class or specify which class you
> want with the #+LATEX_CLASS header. I use this quite a bit because then
> I don't have to remember which LATEX_HEADER lines to include in the
> document, the specific option settings etc. I don't need support for
> multilingual documents, but I do have a number of 'special' documents
> (such as one with colours, logos and specific fonts for an employer to
> match their 'style guide'. I also have ones for generating project
> documents, letters, meeting minutes etc. They all use various different
> Latex extensions (particularly ones which don't mix well and cannot be
> included with other packages).

I agree. `Org-latex-classes' is a very good option for create LaTeX
templates, and I have a few classes defined as well. The problem is when
you need really long and complex preambles (it is not a problem that
most users may have, though). In a recent project (a book) my preamble
had about 2000 lines (including macros and environments defined by me,
some functions in Lua for LuaTeX, etc.). With long or complex preambles
it's a bit awkward to do it in Elisp and org-latex-classes. In that
case, I usually write the preamble to an Org document and generate a
*.tex file using org-babel-tangle. Then I include that file at the very
beginning of my document with an \input macro. On the LaTeX side, there
is also the option to create your own sty file:
https://tex.stackexchange.com/questions/77/how-to-make-a-standard-preamble-into-a-package

As an alternative to #+LaTeX_Header you can also include the
preamble in the Org document itself using a LaTeX block:

#+NAME: preamble
#+begin_src latex :exports none
... a lot of latex code
#+end_src

and then, in another block with the keyword `:noweb':

#+begin_src latex :noweb yes :results raw
,#+LaTeX_Header: <<preamble>>
#+end_src

(This useful trick came from Charles Berry in this thread:
https://orgmode.org/list/225A3D45-0F47-4FFE-8BBA-F023CB8C9A6C@health.ucsd.edu/#r)

Best regards,

Juan Manuel

Re: org-mode export to (latex) PDF

[Tim Cross]

Juan Manuel Macías <maciaschain@posteo.net> writes:

> Tim Cross writes:
>
>> Just FYI for those who don't know, you can use the org-latex-classes
>> variable to define your own pseudo document classes, possibly using the
>> DEFAULT_PACKAGES, PACKAGES, EXTRA_PACKAGES macros and other latex
>> settings. So for example, you can add the babel or other packages you
>> want and either make that the 'default' class or specify which class you
>> want with the #+LATEX_CLASS header. I use this quite a bit because then
>> I don't have to remember which LATEX_HEADER lines to include in the
>> document, the specific option settings etc. I don't need support for
>> multilingual documents, but I do have a number of 'special' documents
>> (such as one with colours, logos and specific fonts for an employer to
>> match their 'style guide'. I also have ones for generating project
>> documents, letters, meeting minutes etc. They all use various different
>> Latex extensions (particularly ones which don't mix well and cannot be
>> included with other packages).
>
> I agree. `Org-latex-classes' is a very good option for create LaTeX
> templates, and I have a few classes defined as well. The problem is when
> you need really long and complex preambles (it is not a problem that
> most users may have, though). In a recent project (a book) my preamble
> had about 2000 lines (including macros and environments defined by me,
> some functions in Lua for LuaTeX, etc.). With long or complex preambles
> it's a bit awkward to do it in Elisp and org-latex-classes. In that
> case, I usually write the preamble to an Org document and generate a
> *.tex file using org-babel-tangle. Then I include that file at the very
> beginning of my document with an \input macro. On the LaTeX side, there
> is also the option to create your own sty file:
> https://tex.stackexchange.com/questions/77/how-to-make-a-standard-preamble-into-a-package
>
> As an alternative to #+LaTeX_Header you can also include the
> preamble in the Org document itself using a LaTeX block:
>
> #+NAME: preamble
> #+begin_src latex :exports none
>
> ... a lot of latex code
> #+end_src
>
> and then, in another block with the keyword `:noweb':
>
> #+begin_src latex :noweb yes :results raw
> ,#+LaTeX_Header: <<preamble>>
> #+end_src
>
> (This useful trick came from Charles Berry in this thread:
> https://orgmode.org/list/225A3D45-0F47-4FFE-8BBA-F023CB8C9A6C@health.ucsd.edu/#r)
>

Yes, I do pretty much the same. I have a number of personal *.sty files
which contain a lot of additional setup information which would be
difficult to include directly in org-latex-classes.

I have also used the idea of latex blocks and tangling, especially when
working out the details for a specific latex configuration. Once I have
it working, unless this is strictly a 'one off', I will typically move
that information into a *.sty file or similar. I also use a couple of
templates via either company or yasnipet which I use for some org
documents which have a 'standard' outline and header setup. 

I find the combination of these techniques makes it easy to create new
documents and maintain existing ones. I have a terrible memory for the
low level configuration stuff, so the more I can rely on pre-defined
configurations, the better. The nice thing about how I have things setup
now is that it seems pretty robust. I rarely encounter any problems when
updating org and am able to get 'up and running' on a new system fairly
quickly. The biggest challenge I've had lately has been my move from my
own standard configuration to using spacemacs. It has taken a bit of
work to get spacemacs to work the way I want, but I have grown to love
the modal editing of evil mode (probably because VI was my first
editor). I'm now very happy with my configuration and workflow. 

-- 
Tim Cross

Re: org-mode export to (latex) PDF

[Eric S Fraga]

On Monday, 12 Jul 2021 at 13:09, Tim Cross wrote:
> I also use a couple of templates via either company or yasnipet which
> I use for some org documents which have a 'standard' outline and
> header setup.

The autoinsert package (built-in) is quite useful here as well.  I have
a skeleton org file which has all the settings I typically need for both
articles and beamer presentations.  When I visit a new org file, Emacs
will ask if I want to insert the skeleton file contents.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-579-gfdb98a
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: org-mode export to (latex) PDF

[Jean-Christophe Helary]

> On Jul 10, 2021, at 23:38, Juan Manuel Macías <maciaschain@posteo.net> wrote:
> 
> Hi Jean-Christophe,
> 
> Jean-Christophe Helary writes:
> 
>> I had given up on Latex because mixing languages sounded like a huge
>> pain in the butt but I see that without some org-level infrastructure
>> it is not possible to achieve much when exporting to Latex/PDF (unless
>> I missed something).
> 
> Well, LaTeX has excellent (typographic and orthotypographic)
> multilingual support, using the babel or polyglossia packages. I
> especially recommend babel:
> http://mirrors.ctan.org/macros/latex/required/babel/base/babel.pdf

Thank you very much Juan.

I understand that Latex has excellent support for multilingual files. I use it sometimes for very complex multilingual texts (ancient Japanese and French for ex).

But with UTF-8 being ubiquitous on computers nowadays, I really can't be bothered to have to type all those sequences to have org-mode properly export to PDF something that it exports perfectly well to ODT without requiring anything extra.

Which is the reason why I was wondering about a "generic" default setting where the conversion engine behind org could just use a default multilingual package and a default "wide enough" generic font. I really mean a "good enough" export where all the characters are visible, nothing fancy.

Jean-Christophe 

> 
> And LaTeX also has very good support for oriental languages or languages
> with complex writing, especially in LuaTeX. In LuaTeX and XeTeX you can
> also use opentype fonts and opentype features.
> 
> The problem is how to translate that from Org --in an org-centric way--
> to LaTeX. Currently, you can apply LaTeX commands for multilingual
> management directly in your Org document. For example:
> 
> #+LaTeX_Header: \usepackage[several langs]{babel}
> 
> @@latex:\begin{otherlanguage*}{german}@@
> 
> ... some text in german ...
> 
> @@latex:\end{otherlanguage*}@@
> 
> Recently, I submitted a patch here that allows adding LaTeX attributes
> to `quote' blocks. Then, you could do something like this:
> 
> #+LaTeX_Header:\usepackage[german,english]{babel}
> #+LaTeX_Header:\usepackage{quoting}
> #+LaTeX_Header:\usepackage[babel=true,autostyle=true,german=quotes]{csquotes}
> #+LaTeX_Header:\SetBlockEnvironment{quoting}
> 
> #+ATTR_LaTeX: :environment foreigndisplayquote :options {german}
> #+begin_quote
> Eine Erklärung, wie sie einer Schrift in einer Vorrede nach der
> Gewohnheit vorausgeschickt wird ---über den Zweck, den der Verfasser
> sich in ihr vorgesetzt, sowie über die Veranlassungen und das
> Verhältnis, worin er sie zu andern frühern oder gleichzeitigen
> Behandlungen desselben Gegenstandes zu stehen glaubt--- scheint bei
> einer philosophischen Schrift nicht nur überflüssig, sondern um der
> Natur der Sache willen sogar unpassend und zweckwidrig zu sein (Hegel).
> #+end_quote
> 
> Best regards,
> 
> Juan Manuel 
> 

-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 10/07/2021 20:52, Juan Manuel Macías wrote:
> 
> I agree with you that Org should have multilingual support. A few months
> ago I started this thread here, with some proposals:
> https://orgmode.org/list/87o8d95pvo.fsf@posteo.net/

I tried to draw more attention to support of locale-aware formatting of 
numbers on emacs-devel mail list. Certainly a question concerning 
mixed-language buffers were raised. Result of attempts to discuss text 
properties to mark regions having alternative languages was just "Which 
property will help here? we don't have such properties" from Eli Zaretskii.

>  > Jean-Christophe Helary writes:
> 
>> I guess it is an issue with the Latex backend and could have been
>> solved with the proper Latex settings, but it seems weird that the
>> default settings do not allow for out-of-the-box multilingual
>> support.

(add-to-list 'org-latex-inputenc-alist '("utf8" . "utf8x"))

might help. I am not an active LaTeX user last years, so I am unaware 
whether \usepackage[utf8x]{inputenc} has any drawbacks.

Re: org-mode export to (latex) PDF

[Stefan Nobis]

Maxim Nikulin <manikulin@gmail.com> writes:

> (add-to-list 'org-latex-inputenc-alist '("utf8" . "utf8x"))

Do not do this. Both, utf8x and ucs, are obsolete and deprecated for
quite some time.

For proper unicode support, switch from pdflatex to lualatex or
xelatex. With these newer backends (and proper adjustments for the
LaTeX preamble generated by Org) Unicode should work out of the box
(if the font supports the requested Unicode characters).

My current packages setup to support all three engines looks like
this (should work for normal documents and beamer):

--8<---------------cut here---------------start------------->8---

(setq org-latex-default-packages-alist
        '(("AUTO" "inputenc" t ("pdflatex"))
          ("T1" "fontenc" t ("pdflatex"))
          ("" "fontspec" t ("lualatex" "xelatex"))
          ("AUTO" "babel" t ("pdflatex" "lualatex"))
          ("AUTO" "polyglossia" t ("xelatex"))
          ("" "graphicx" t)
          ("" "grffile" t)
          ("" "longtable" nil)
          ("" "wrapfig" nil)
          ("" "rotating" nil)
          ("normalem" "ulem" t)
          ("" "mathtools" t)
          ("" "amssymb" t)
          ("" "textcomp" t ("pdflatex"))
          ("warnings-off={mathtools-colon,mathtools-overbracket}" "unicode-math" t ("lualatex" "xelatex"))
          ("" "caption" nil)
          ("" "booktabs" t)
          ("" "hyperref" nil)
          "\\tolerance=1000"))

--8<---------------cut here---------------end--------------->8---

To switch e.g. to lualatex (and e.g. use latexmk for compiling), I
use:

--8<---------------cut here---------------start------------->8---

(setq org-latex-compiler "lualatex")
(setq org-latex-bib-compiler "biber")
(setq org-latex-pdf-process '("latexmk -f -pdf -%latex -outdir=%o %f"))

--8<---------------cut here---------------end--------------->8---

--
Until the next mail...,
Stefan.

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 10/07/2021 23:44, Stefan Nobis wrote:
> Maxim Nikulin writes:
> 
>> (add-to-list 'org-latex-inputenc-alist '("utf8" . "utf8x"))
>> so I am unaware whether \usepackage[utf8x]{inputenc} has any drawbacks.
> 
> Do not do this. Both, utf8x and ucs, are obsolete and deprecated for
> quite some time.

Maybe, I have seen such warnings. However I have tested neither utf8 nor 
utf8x on real examples. That is why I am unaware what can be broken in 
particular. For small examples with various symbols outside of ASCII, 
utf8x may give better support.

Last time I did something serious with LaTeX, bibtex was 8bit internally 
(unicode version had not reached stable Linux distributions, pybtex was 
buggy), so that time I used 8bit charset.

> For proper unicode support, switch from pdflatex to lualatex or
> xelatex. With these newer backends (and proper adjustments for the
> LaTeX preamble generated by Org) Unicode should work out of the box
> (if the font supports the requested Unicode characters).

Maybe my expectation from proper unicode support was too optimistic. I 
see the reason why preamble is necessary with 8bit encodings. I 
appreciate possibility to easily specify particular ttf or otf font. I 
do not like that it is necessary to specify *all* fonts, otherwise some 
characters are missed or compilation fails with error. I do not know if 
new engines allows to get list of available fonts and to choose a set of 
fonts with better coverage than lmodern.

I have tried LuaLaTeX with the following file. From some examples I have 
found, I expect, further tuning is required. Apparent problem is missed 
unicode characters in math mode. I admit that polyglossia is unavoidable 
for any real text but I do not like that \set…font commands are mandatory.

\documentclass{article}
\usepackage{fontspec}
\usepackage{polyglossia}
\setdefaultlanguage{russian}
\setotherlanguage{english}
\setmainfont{CMU Serif} % breaks ∞
\setsansfont{CMU Sans Serif}
\setmonofont{CMU Typewriter Text}

\begin{document}
Test¹ of superscript and ½ fraction.

\textbf{Теорема.} \emph{Пусть} $α → ∞$ и $\beta \to \infty$.

\verb=Катет= и \textsf{гипотенуза}.

Åå. Text Greek α.
\end{document}


> (setq org-latex-default-packages-alist
>          '(("AUTO" "inputenc" t ("pdflatex"))
>            ("T1" "fontenc" t ("pdflatex"))

I just have realized that fontenc behavior should be similar to inputenc 
and babel, e.g. something like \usepackage[T1,T2A]{fontenc} should be 
used for Russian.

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Hi Maxim,

Maxim Nikulin writes:

> I do not know if new engines allows to get list of available fonts and
> to choose a set of fonts with better coverage than lmodern.

LuaTeX and XeTeX use harfbuzz as OpenType rendering engine. On
LuaLaTeX and XeLaTeX you must use the fontspec package
(https://www.ctan.org/pkg/fontspec) to load otf or ttf fonts and add
opentype features. It is very powerful and its interface is very simple
to use. XeTeX has access to system fonts. LuaTeX has access to both
system fonts and any font you want to declare, simply by adding the
path.

For example:

\setmainfont{Palatino Linotype}[Ligatures=NoCommon,Numbers=Lowercase]

With LuaTeX you can also define new opentype features on the fly using
scripts in Lua, via the function fonts.handlers.otf.addfeature

For example, here I define a character substitution:

\directlua{
fonts.handlers.otf.addfeature{
name = "mysub",
type = "substitution",
data = {
periodcentered = "anoteleia",
},
}
}

And here I add that feature to Linux Libertine font:

\setmainfont{Linux Libertine O}[RawFeature=+mysub]

For multilingual management I recommend using Babel instead of
Polyglossia. You can, for example, assign with Babel families from fonts
and language definitions to non-Latin scripts (Cyrillic, Greek,
Devanagari, Arabic, etc.). For example

\babelprovide[onchar=ids fonts,hyphenrules=russian]{russian}
  \babelprovide[onchar=ids fonts,hyphenrules=ancientgreek]{greek}

\babelfont[russian]{rm}[%
  Numbers=Lowercase]{Linux Libertine O}
  \babelfont[greek]{rm}[%
  Numbers=Lowercase]{Old Standard}

Best regards,

Juan Manuel 

Re: org-mode export to (latex) PDF

[Stefan Nobis]

Maxim Nikulin <manikulin@gmail.com> writes:

[utf8x]
> Maybe, I have seen such warnings. However I have tested neither utf8
> nor utf8x on real examples. That is why I am unaware what can be
> broken in particular. For small examples with various symbols
> outside of ASCII, utf8x may give better support.

The main point: utf8x and the associated package ucs are not
maintained for quite some time (utf8x seems to be last changed in
2004) and as far as I understand have always been more of a workaround
than a solution. But I'm not an expert in this regard.

Nowadays the LaTeX kernel and input encodings like (plain) utf8 are
much more powerful and extensible and do play much better with other
packages.

Especially in the last few years the unicode support has become much
better (for all engines).

> I do not like that it is necessary to specify *all* fonts,

You need to specify all fonts that you want to use and that deviate
from the default (Latin Modern in the case of lualatex). How else
should the system now that you want something else?

And in the case of cyrillic: Sadly, the default fontset Latin Modern
has no good support for the cyrillic alphabet. But the name is at
least a small hint. :)

In LaTeX there are 4 groups of fonts: the main font (usually a serif
one), a sans serif font group, a monospace font group and the math
font set. If you use all kinds of groups and want differ from the
defaults, you need to say so explicitly. On the other hand: If you do
never use e.g. monospace glyphs you do not need to specify the
monospace font.

So here is a minimal version of your document that should work:

#+begin_src latex
\documentclass{article}
\usepackage{fontspec}
\usepackage{unicode-math}
\setmainfont{CMU Serif}
\setsansfont{CMU Sans Serif}
\setmonofont{CMU Typewriter Text}

\begin{document}
Test¹ of superscript and ½ fraction.

\textbf{Теорема.} \emph{Пусть} $\quad α → ∞$ и $\beta \to \infty$.

\verb=Катет= и \textsf{гипотенуза}.

Åå. Text Greek α.
\end{document}
#+end_src

The package "unicode-math" should always be used with lualatex and
xelatex, in order to support unicode math input. In your minimal
example neither polyglossia nor babel are required, but explicit font
selection is necessary to switch all font groups to a fontset with
cyrillic glyphs.

>> (setq org-latex-default-packages-alist
>>          '(("AUTO" "inputenc" t ("pdflatex"))
>>            ("T1" "fontenc" t ("pdflatex"))

> I just have realized that fontenc behavior should be similar to
> inputenc and babel, e.g. something like \usepackage[T1,T2A]{fontenc}
> should be used for Russian.

Yes, indeed. It would be nice to support this all from Org. So if one
chooses russian as language, that (in case of pdflatex engine) an
option "AUTO" for "fontenc" is supported that get expanded to
"[T1,T2A]" and that the necessary font selection is also generated (if
not overriden with an explicit set choosen by the user). But a full
fledged multi-language solution, that supports more than just latin
and russion may be quite a challenge.

-- 
Until the next mail...,
Stefan.

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 14/07/2021 13:44, Stefan Nobis wrote:
> 
> The main point: utf8x and the associated package ucs are not
> maintained for quite some time (utf8x seems to be last changed in
> 2004) and as far as I understand have always been more of a workaround
> than a solution. But I'm not an expert in this regard.
> 
> Nowadays the LaTeX kernel and input encodings like (plain) utf8 are
> much more powerful and extensible and do play much better with other
> packages.

I was unlucky to notice the utf8x option when ≠ character was discussed 
on this mail list in the context of pseudocode listings. I am afraid, 
year 2004 is impressive in other sense than you expected. "Better" 
solution still unable to handle ≠ (unlike deprecated one). Actually 
necessity to explicitly specify several fonts reminds me end of previous 
century when Cyrillic fonts was not in default package set and required 
manual adjustment of config files (not to mention missed support in 
babel and inputenc, so several partially incompatible variants were used 
at that time).

> You need to specify all fonts that you want to use and that deviate
> from the default (Latin Modern in the case of lualatex). How else
> should the system now that you want something else?

There are cm-super fonts for at least of 15 years. OK, they are Type1, 
not otf or ttf and conversion from metafont was not lossless. (Actually 
the only real problem I noticed is that some rare printers ignore 
hairlines.) Why is the Latin Modern family used if CMU is available? 
Nowadays most of applications have no problem with wide range of Unicode 
characters. However it is a kind of trade-off to preserve traditional 
Computer Modern font (a kind of feature of TeX) or to pick a system font 
with more characters.

> The package "unicode-math" should always be used with lualatex and
> xelatex, in order to support unicode math input.

Thank you for the hint. Do you think Org should use it by default? Are 
there any caveats?

> In your minimal example neither polyglossia nor babel are required

They are almost unavoidable in any real document unless it is preview of 
e.g. particular equation.

On 14/07/2021 00:53, Juan Manuel Macías wrote:> And here I add that 
feature to Linux Libertine font:
> 
> \setmainfont{Linux Libertine O}[RawFeature=+mysub]

It is wonderful that custom font can be chosen so easily. I was never 
brave enough and I did not have strong enough reason to follow lengthy 
guides how to use ttf font in latex+dvips+ps2pdf workflow. However 
custom fonts are for special documents. It perfectly suits for e.g. a 
book when camera ready variant is required. For routine notes it is 
better to keep from defaults as minimal as possible to minimize problems 
that may arise a decade later. I would prefer to avoid Linux Libertine 
if I am going to send source file to a colleague having another OS. I 
prefer to do fine tuning at the last stages of preparation of a 
document. It is sad that default fonts are often unusable for me.

> For multilingual management I recommend using Babel instead of
> Polyglossia.

I have no experience with polyglossia yet. I added it just because most 
of examples for LuaLaTeX or XeLaTeX use it.

TeX takes responsibility for a lot of things and it allows to get rather 
pleasant results with minimal efforts due to reasonable defaults. Unlike 
Apache FO processor for general formatting. Equations were looking 
disgusting in MS Word till ~2010.

This topic started from question concerning multilingual support out of 
the box. I can not help with Japanese quotes. However some problems can 
be noticed with e.g. (sorry for some raw LaTeX):

---- >8 ----
Test¹ of superscript and ½ fraction.

*«Теорема».* /Пусть/ $α → ∞$ и $\beta \to \infty$.

=Катет= и \textsf{гипотенуза}.

Åå. Text Greek α. µm.

Text utf8x ≠ utf8 and math $8 ≠ x$.
---- 8< -----

Current default in Org is pdflatex. It requires at least adjusting of 
fontspec by adding T2A option.

If LuaTeX and XeLaTeX handles Unicode better, is it possible to make any 
of them the default option and to leave pdflatex as a fallback? Is it 
possible to detect lualatex and xelatex in runtime? I have noticed that 
/usr/bin/lualatex belongs to texlive-latex-base package. Originally I 
did not have texlive-luatex package installed, so likely lualatex was 
rather broken despite presence of the binary in the system.

Should some packages for lualatex and xelatex be added to default list 
to minimize user problems and at the same time keeping configuration 
safe? (unicode-math, etc.)

Is it possible to provide reasonable defaults for fonts? Since lmodern 
is hardcoded in luatex and xetex, it may be done either by some usually 
available latex package or by org code and custom variables.

If some defaults can not be determined (e.g. \setmainfont) likely they 
should be explicitly mentioned in the org manual.

Re: org-mode export to (latex) PDF

[Stefan Nobis]

Maxim Nikulin <manikulin@gmail.com> writes:

> There are cm-super fonts for at least of 15 years.

There are many tradeoffs in many aspects. No single font pleases
everyone. So you want to say: Your requirements are more
important/common/stylish/whatever that the requirements of other
people?

I do need only latin characters and math (so Latin Modern would
suffice), but still I use different fonts from time to time (like
Libertine, Palatino and others) - and I also mix different fonts (not
all font families provide serif and sans serif and monospace glyphs).

And due to the history of TeX and the structure of font files, there
is no single command to set up all required information to switch all
font families with one command. Usually up to 3-4 command are required
(sometimes more for more advanced requirements) are needed to change
most relevant font information.

Frankly, I'm completely clueless why this should be a problem.

Yes, it may be unfortunate that not all fonts available support all
Unicode glyphs ever invented. But on the other hand: Most of the free
fonts are created by people in their free time and it takes VAST
amounts of time and talent to create nice looking fonts.

I appreciate the many fonts that creative people created to be used for
free. So if all I have to do to use this massive gift is drop a couple
of commands in some or all my documents, I do not complain - I'm
grateful.

I understand that it sometimes sucks to be forced to use tools that
are created with a massive US centric world view, that not only
focuses on latin characters, but even only respcect ASCII (e.g. even
today quite some systems have problems with german umlauts). But try
to get over it: At least in the case of Emacs, Org, and LaTeX it is
possible and in most cases quite easy to overcome the restrictions
that the default settings may impose.

[unicode-math]
> Thank you for the hint. Do you think Org should use it by default?
> Are there any caveats?

Yes, unicode-math should be seen as must have for lualatex and xelatex
(if math is used). As far as I know there are no downsides and it
should be part of the default packages (but only for lualatex and
xelatex, not for pdflatex).

> If LuaTeX and XeLaTeX handles Unicode better, is it possible to make
> any of them the default option and to leave pdflatex as a fallback?

That is possible today and you can easily change the LaTeX engine via
global options in your Emacs init.el or via local settings inside Org
documents.

> Is it possible to detect lualatex and xelatex in runtime?

At runtime of the LaTeX engine, so execute LaTeX commands depending on
the engine that processes the document containing these commands?

Yes, that is possible. The LaTeX package iftex provides macros to
execute commands based on the running engine (see
https://www.ctan.org/pkg/iftex?lang=en).

> Should some packages for lualatex and xelatex be added to default
> list to minimize user problems and at the same time keeping
> configuration safe? (unicode-math, etc.)

Maybe. I'm currently myself struggling a little bit with a flexible
configuration, that can be used with many different kind of documents
(short notes, larger reports, beamer presentations) and provides all
the extras I like to use. There is no clear best package list for
every use case (in some cases unnecessary loaded packages only waste
time, in other cases, especially with some individual set of package
options, there might be errors in some scenario or another).

> Is it possible to provide reasonable defaults for fonts?

I do not think so. You want Cyrillic. But what about Japanese,
Chinese, Devanagari, Tamil, Arabic etc? I doubt that there exists a
single font that supports all these scripts satisfactorily. Despite
the existence of the Unicode encoding(s), the glyphs and font designs
are still quite complex and demanding even for a single script.

But maybe we could assemble a list of good (enough) fonts for
different languages/scripts and provide a default setup in Org for
LaTeX export, that sets a proper font for the chosen document
language?

-- 
Until the next mail...,
Stefan.

Re: org-mode export to (latex) PDF

[Tim Cross]

Stefan Nobis <stefan-ml@snobis.de> writes:

> Maybe. I'm currently myself struggling a little bit with a flexible
> configuration, that can be used with many different kind of documents
> (short notes, larger reports, beamer presentations) and provides all
> the extras I like to use. There is no clear best package list for
> every use case (in some cases unnecessary loaded packages only waste
> time, in other cases, especially with some individual set of package
> options, there might be errors in some scenario or another).
>

I think this is the big stumbling bloc for having multi-lingual
documents 'out of the box'. It is hard enough to get a reliable default
configuration which will work for a majority of use cases with support
for just one language.

The good news is that threads like this have some really useful
information in them. My suggestion would be to create a page in worg
dedicated to configuring multi-lingual support. If we cannot make org
mode do what people need 'out of the box', lets at least make it easier
for them to find some good examples, suggestions and configuration
guidelines.

I have the disability of only knowing one language, so probably not
something I can help with much. However, I have always found configuring
desired fonts a drag when it comes to Latex based documents - not that
it is too hard, but more that each time I need to do it, I've forgotten
the steps and and wading through the information is a pain (especially
once you mix in outdated advice, different tex engines etc). I might see
if I can write up something concise which focuses on UTF-8 and selecting
a specific font. 

>> Is it possible to provide reasonable defaults for fonts?
>
> I do not think so. You want Cyrillic. But what about Japanese,
> Chinese, Devanagari, Tamil, Arabic etc? I doubt that there exists a
> single font that supports all these scripts satisfactorily. Despite
> the existence of the Unicode encoding(s), the glyphs and font designs
> are still quite complex and demanding even for a single script.
>
> But maybe we could assemble a list of good (enough) fonts for
> different languages/scripts and provide a default setup in Org for
> LaTeX export, that sets a proper font for the chosen document
> language?

I think such a list would be a really good addition to worg.

-- 
Tim Cross

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Tim Cross writes:

> Stefan Nobis <stefan-ml@snobis.de> writes:

>> But maybe we could assemble a list of good (enough) fonts for
>> different languages/scripts and provide a default setup in Org for
>> LaTeX export, that sets a proper font for the chosen document
>> language?

> I think such a list would be a really good addition to worg.

I think it's a great idea. Some resources on fonts and languages that
may be useful:

- The LaTeX Font Catalogue (https://tug.org/FontCatalogue/) is pretty
  complete and includes a lot of high quality fonts, with coverage for
  many languages. The fonts are from diverse origins, for example the
  excellent fonts for Greek and Latin alphabets from the Greek Font
  Society, the TeX Gyre project fonts, etc.

- To get quick information from an otf font (otf features, Unicode
  ranges, scripts, glyphs, etc.) otfinfo is very useful
  (https://man.archlinux.org/man/otfinfo.1.en). But the more powerful
  tool in this regard is fontforge (https://fontforge.org/en-US/), which
  is not only a complete professional font editor (and free as in
  freedom) but can also be used by everyone to audit all aspects from a
  font: glyphs, metadata, otf features, languages, scripts, ranges, etc.

- There are specialized fonts in a wide coverage of ranges and scripts,
  but many are low-quality or proprietary. Google's Noto Fonts ensure at
  least a reasonably complete coverage: I use them only for experiments
  (or for ensure certain "rare" scripts in Emacs buffers, as Linear B or
  Gothic), but they can also be used within a document.

- Finally, in case anyone is interested, I also wrote for my personal
  use a Helm source to list all system font families and insert the
  family name into a LaTeX document with the syntax of fontspec or Babel
  ("\babelfont", which is a frontend for fontspec).

Best regards,

Juan Manuel 

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 15/07/2021 02:05, Stefan Nobis wrote:
> Maxim Nikulin writes:
> 
>> There are cm-super fonts for at least of 15 years.
> 
> There are many tradeoffs in many aspects. No single font pleases
> everyone. So you want to say: Your requirements are more
> important/common/stylish/whatever that the requirements of other
> people?

I hope, it is possible to make Org export to PDF working out of the box 
for more people. Surprisingly Unicode TeX engines besides alleviating of 
some limitations of earlier implementation bring new burden. I see no 
problem to fix support of Cyrillic in PdfTeX. I have not realized if it 
can be done for LuaTeX or XeTeX despite original expectation that 
support of Unicode means that there should be no problem for any 
character of any script. I started to discuss Russian because it is a 
language for which most of problems are apparent for me. Perhaps a 
similar approach can be used for other scripts.

Before UTF-8 became wide spread on Linux, there were guides how to 
enable support of Cyrillic in e.g. Netscape Navigator. Now such problems 
are forgotten. Is it achievable for TeX?

In CSS it is possible to specify a list of fonts and a glyph is taken 
from the first font where it is present. Despite particular fonts have 
limited coverage, I see wide range of Unicode characters on web pages, 
that is why I am almost sure that system font libraries combine fonts.

Do Unicode TeX engines support such combination of fonts? Is it 
efficient enough to be used by default? Is it possible to write 
reasonably complete config for such purpose? It is preferable to have 
such config as a LaTeX style file, but it may be implemented in Org code 
as well.

There are two quite distinct cases: documents with carefully chosen 
fonts (e.g. a book) and everyday notes when particular font is not so 
important. For the former case a glyph taken from wrong font is a 
serious error. For the latter one, likely even low quality font is 
significantly better than a missed character. I think, both cases should 
be supported however.

> [unicode-math]
>> Thank you for the hint. Do you think Org should use it by default?
>> Are there any caveats?
> 
> Yes, unicode-math should be seen as must have for lualatex and xelatex
> (if math is used). As far as I know there are no downsides and it
> should be part of the default packages (but only for lualatex and
> xelatex, not for pdflatex).

Maybe it is possible to scan the document for presence of character from 
specific category, range, etc., to avoid overhead when the package is 
not necessary.

>> Is it possible to detect lualatex and xelatex in runtime?
> 
> Yes, that is possible. The LaTeX package iftex provides macros to
> execute commands based on the running engine (see
> https://www.ctan.org/pkg/iftex?lang=en).

I mean detection if LuaLaTeX or XeLaTeX is usable from Org code. The 
intention is to minimize customization required before successful 
export. Ideally Org should guess reasonable configuration form content 
of the document and available external tools (and maybe freeze it by 
adding properties to the document to make next compilations faster). 
Certainly user should have a way to force fixed values by disabling 
autoconfiguration as a whole or only for particular settings.

>> Is it possible to provide reasonable defaults for fonts?
> 
> I do not think so. You want Cyrillic. But what about Japanese,
> Chinese, Devanagari, Tamil, Arabic etc? I doubt that there exists a
> single font that supports all these scripts satisfactorily.

I hope, single font is not necessary since other applications can show 
all scripts simultaneously. Of course, my example was not complete, feel 
free to extend it, e.g.

Randomly chosen examples: "日本", "多喝水", "📞".

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Maxim Nikulin writes:

> In CSS it is possible to specify a list of fonts and a glyph is taken
> from the first font where it is present. Despite particular fonts have 
> limited coverage, I see wide range of Unicode characters on web pages,
> that is why I am almost sure that system font libraries combine fonts.

In LuaTeX you can associate a font family to a range or a group of
characters. In a book I typesetted some time ago I used the Cardo font
to represent the characters for Private Use Area.

\newfontfamily\cardo{Cardo} % a fontspec command

\def\puatext#1{{\cardo #1}}

  \begin{luacode*}
  function my_pua (text)
  texto = unicode.utf8.gsub ( text, "([\u{e000}-\u{f8ff}])", "\\puatext{%1}" )
  return text
  end
  \end{luacode*}

  \newcommand\activatepuatext{\directlua{luatexbase.add_to_callback
      ( "process_input_buffer" , my_pua , "my_pua" )}}

\AtBeginDocument{\activatepuatext}

(I add a simple substitution to the callback `process_imput_buffer'
[see: http://wiki.luatex.org/index.php/Callbacks], but these kinds of
overrides can also be do from Org using a custom filter).

Regards,

Juan Manuel 

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 16/07/2021 02:40, Juan Manuel Macías wrote:
> Maxim Nikulin writes:
> 
>> In CSS it is possible to specify a list of fonts and a glyph is taken
>> from the first font where it is present. Despite particular fonts have
>> limited coverage, I see wide range of Unicode characters on web pages,
>> that is why I am almost sure that system font libraries combine fonts.
> 
> In LuaTeX you can associate a font family to a range or a group of
> characters.
> 
>    texto = unicode.utf8.gsub ( text, "([\u{e000}-\u{f8ff}])", "\\puatext{%1}" )

I expect it is terribly inefficient for long spans of text in particular 
language. Command should not be per-character, preferably per-paragraph 
or at least per-word

"([\u{e000}-\u{f8ff}]+)"

However maybe you just do not use sequences of symbols from private use 
area.

I think that low level implementation in browser or in some underlying 
library is much faster

     <dl>
       <dt>LM Roman 12</dt>
       <dd style="font-family: 'LM Roman 12'">abc абв…с</dd>
       <dt>LM Roman 12, CMU Serif</dt>
       <dd style="font-family: 'LM Roman 12', 'CMU Serif'">abc абв…с</dd>
     </dl>

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Maxim Nikulin writes:

> I think that low level implementation in browser or in some underlying
> library is much faster
>
>     <dl>
>       <dt>LM Roman 12</dt>
>       <dd style="font-family: 'LM Roman 12'">abc абв…с</dd>
>       <dt>LM Roman 12, CMU Serif</dt>
>       <dd style="font-family: 'LM Roman 12', 'CMU Serif'">abc абв…с</dd>
>     </dl>

They are two different scenarios: web publishing and book typesetting
(Donald Knuth in the TexBook refers to TeX as: "...a new typesetting
system intended for the creation of beautiful books [...] you will be
telling a computer exactly how the manuscript is to be transformed into
pages whose typographic quality is comparable to that of the world's
finest printers"). LuaTeX, like the rest of TeX engines, is a highly
refined typesetting system, the digital evolution of the mechanical
printing press and the art of typographers. Its goal is printing press
instead of web browsers. All decisions regarding the chosen typefaces
should be taken before. When I prepare a book design, all those
decisions I take them before, and it takes time to test and calibrate
typefaces: choose the font family, or font family groups for certain
languages. Mixing fonts is not trivial for professional typography. This
is not the scenario you describe, nor is it necessary to ensure
readability in multiple browsers with "fallback fonts" for missed
glyphs. In TeX ecosystem it makes no sense (it can be done, anyway[1])
to add fallback fonts to ensure all characters are rendered by their
corresponding glyphs. I insist: they are two orthogonal scenarios and
two diametrically opposed design concepts.

If the font I want to use lacks certain glyphs, I can take various
decisions, depending on the glyphs I need. If I lack only certain
diacritics, often with some Lua code it is enough for me (some Lua is
also usually useful to adjust the position of combining diacritical
marks without editing the font with fontforge and add a 'mark' or
'marktomark' tag). But, generally, if a font doesn't have the glyphs I
need, I just don't use it. The same is true in the case of small caps.
If a font does not have small caps, you should never use those horrible
and illegible synthesized small caps from DTP programs ...

In LuaTeX and XeTeX you can define at high level, for example, your own
hybrid font families. If I want to use the GFS Porson as italics from
another font, a Didot typeface for example, I can do this:

\newfontfamily\mygreek{GFS Didot Classic}
[Script=Greek,ItalicFont=GFS Porson,ItalicFeatures={Scale=.90}]
\emfontdeclare{\itshape,\upshape}
\mygreek
γίγνονται παῖδες δύο, \emph{πρεσβύτερος} μὲν Ἀρταξέρξης

[1] If you want to have fallback fonts, you can also do it in
LuaTeX by adding some Lua code:
https://tex.stackexchange.com/questions/514940/define-fallback-font-for-missing-glyphs-in-lualatex

(anyway, I insist that combining glyphs is something you must
be done with care)

Best regards,

Juan Manuel 

Re: org-mode export to (latex) PDF

[Maxim Nikulin]

On 17/07/2021 01:34, Juan Manuel Macías wrote:
> Maxim Nikulin writes:
> 
>> I think that low level implementation in browser or in some underlying
>> library is much faster
>>
>>      <dl>
>>        <dt>LM Roman 12</dt>
>>        <dd style="font-family: 'LM Roman 12'">abc абв…с</dd>
>>        <dt>LM Roman 12, CMU Serif</dt>
>>        <dd style="font-family: 'LM Roman 12', 'CMU Serif'">abc абв…с</dd>
>>      </dl>
> 
> They are two different scenarios: web publishing and book typesetting

Juan Manuel, it seems you are too inclined to book typesetting. It is 
important case and it should be supported by org. I have repeated two 
times in this thread that there is another case, namely routine quick 
notes. Such documents have another balance concerning time required to 
get acceptable result. TeX takes responsibility for a lot of defaults 
such as what spaces should be added around "=" and what ones around 
right angle bracket. Users do not need to make decisions concerning such 
design details to get accurately typeset equations.

At the age of custom charsets (often 8bit) and encodings the problem of 
multilingual texts was obvious. Unicode and UTF-8 alleviate many issues. 
It happened that Cyrillic is an edge case for Unicode TeX engines. Since 
~2000 it works out of the box for LaTeX and PdfLaTeX. Last years I did 
not need to adjust config files and regenerate formats. Hyphenation, 
default fonts (OK, "apt install cm-super" to avoid rasterized fonts is a 
bit behind defaults though no manual config is required) just work. Each 
document needs a few universal lines to setup Russian. Some users may 
have specific style files but generally source documents are quite portable.

Default fonts for LuaTeX and XeTeX do not include Cyrillic any more. 
Every user have to do a decision which fonts should be used even if one 
does not care concerning particular fonts. It increases probability to 
get a file with font configuration that is specific to Mac or Windows.

I do not actively use characters from other Unicode planes, however 
sometimes I do it for fun. Getting them completely missing is less 
preferred than displaying them with low quality font. Specifying all 
fonts requires lengthy config, probably different for LuaTeX and XeTeX. 
At first it is necessary to realize which fonts are available for 
particular glyph. Finally it makes *source* document less portable.

"font-family: 'LM Roman 12', 'CMU Serif'" above is a dumb example of 
what I consider relatively high-level config for routine documents that 
do not need to be handcrafted. Unavailable glyph or even font is not an 
error, it just causes lookup of candidates in the following items. For 
TeX maybe it is reasonable to consider fallback to complete set of fonts 
at ones (roman + mono + math) if such combination is available.

> If I want to use the GFS Porson as italics from
> another font, a Didot typeface for example, I can do this:

If it is not a book or at the stage of early draft another scenario is 
possible. Text should just appear in the compiled document, particular 
font does not matter, its choice is postponed since text content has 
higher priority. Minimal setup in invaluable.

At least with minimal examples I faced another issue: characters 
silently disappears, no warning is generated. Adding babel changes it, 
but I still believe that especially for documents with carefully chosen 
fonts. It is a serious hidden error to get "invalid char glyph" instead 
of all missed characters.

> [1] If you want to have fallback fonts, you can also do it in
> LuaTeX by adding some Lua code:
> https://tex.stackexchange.com/questions/514940/define-fallback-font-for-missing-glyphs-in-lualatex

Would you recommend such code as default for Org? Let's assume that some 
information concerning system fonts are available. I suspect, the 
accepted answer is not fool-proof. In addition, XeLaTeX requires 
something different.

luaotfload provides fallback feature close to what I expect, however it 
is necessary to explicitly specify script that I would prefer to avoid. 
Moreover it significantly increases compilation time. Sometimes LuaLaTeX 
starts to eat CPU with no progress, emoji does not work for some reason.
I am unsure concerning particular "Noto Sans CJK" since several ones are 
available.

\documentclass{article}
\usepackage{fontspec}
\usepackage{unicode-math}
\directlua{luaotfload.add_fallback
   ("seriffallback",
     {
       "Noto Serif CJK SC:mode=harf;script=sc;",
       "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
     })
}
% TwemojiMozilla is not shown by viewers, visible in pdftotext
       %"Noto Color Emoji:mode=harf;"
% or
       %"file:/usr/share/fonts/truetype/noto/NotoColorEmoji.ttf:mode=harf;"
% </usr/share/fonts/truetype/noto/NotoColorEmoji.ttf
% ! error:  (file /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf) 
(ttf): loca
% table not found
% !  ==> Fatal error occurred, no output PDF file produced!
\setmainfont{CMU Serif}[RawFeature={fallback=seriffallback}]

\directlua{luaotfload.add_fallback
   ("sansfallback",
     {
       "Noto Sans CJK SC:mode=harf;script=sc;",
       "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
     })
}
\setsansfont{CMU Sans Serif}[RawFeature={fallback=sansfallback}]

\directlua{luaotfload.add_fallback
   ("monofallback",
     {
       "Noto Sans Mono CJK SC:mode=harf;script=sc;",
       "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
     })
}
\setmonofont{CMU Typewriter Text}[RawFeature={fallback=monofallback}]
\begin{document}
Test¹ of superscript and ½ fraction.

\textbf{«Теорема».} \emph{Пусть} $α → ∞$ и $\beta \to \infty$.

\verb=Катет= и \textsf{гипотенуза}.

Åå. Text Greek α.

Text utf8x ≠ utf8 and math $utf8x ≠ utf8$.

Randomly chosen examples: "日本", "多喝水", "📞".

\verb=Randomly chosen examples: "日本", "多喝水", "📞".=
\end{document}

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Hi Maxim,

I think the problem is not the fact that I may be inclined towards book
typesetting but that TeX itself and its workflow is inclined towards
book typesetting. This fact is something that must be taken into account
and that many LaTeX users sometimes forget. The concept of a 'fallback
font' is still something exotic for the TeX working method, despite that
LuaTeX can access TeX primitives using scripting in lua and can achieve
things like that, at the cost of resources. For example, the fontspec
package provides the conditional \IfFontExists{font}{True code}{False
code} which consumes a lot of resources.

Anyway, I would dare to recommend these two possibilities:

a) If you want to define a list of fallback fonts for the LaTeX
process, IMHO should be done org-centric with Elisp (something for
pdfTeX, XeTeX and LuaTeX. I'm afraid this would be a lot of work and too
much cholesterol for Org Mode, though).

b) However, my preference is something that has already been comented in
this thread: add to the documentation (or to Worg web site) a (not too
long) list of recommended fonts for different languages: of course,
fonts that are freely licensed and accessible to everyone. In any
collaborative work in LaTeX I find it's much more simple to share an
easily accessible and free (as in freedom) font than to add a list of
fallback fonts to the documents via code (a true bloat for TeX). The
LaTeX Font Catalogue includes lots of very good quality fonts. For
example, TeX Live includes an excellent font with support for Greek,
Cyrillic and Linguistics: Old Standard, originally designed by prof.
Alexey Kryukov and currently maintained by Robert Alessi:
https://www.ctan.org/pkg/oldstandard (I don't work on Cyrillic and I can
not comment on that aspect: I use this font more for Greek; but it is
often said that Old Standard is one of the best and most documented
options to represent Cyrillic).

Best regards,

Juan Manuel 

Maxim Nikulin writes:

> On 17/07/2021 01:34, Juan Manuel Macías wrote:
>> Maxim Nikulin writes:
>> 
>>> I think that low level implementation in browser or in some underlying
>>> library is much faster
>>>
>>>      <dl>
>>>        <dt>LM Roman 12</dt>
>>>        <dd style="font-family: 'LM Roman 12'">abc абв…с</dd>
>>>        <dt>LM Roman 12, CMU Serif</dt>
>>>        <dd style="font-family: 'LM Roman 12', 'CMU Serif'">abc абв…с</dd>
>>>      </dl>
>> They are two different scenarios: web publishing and book
>> typesetting
>
> Juan Manuel, it seems you are too inclined to book typesetting. It is
> important case and it should be supported by org. I have repeated two 
> times in this thread that there is another case, namely routine quick
> notes. Such documents have another balance concerning time required to 
> get acceptable result. TeX takes responsibility for a lot of defaults
> such as what spaces should be added around "=" and what ones around 
> right angle bracket. Users do not need to make decisions concerning
> such design details to get accurately typeset equations.
>
> At the age of custom charsets (often 8bit) and encodings the problem
> of multilingual texts was obvious. Unicode and UTF-8 alleviate many
> issues. It happened that Cyrillic is an edge case for Unicode TeX
> engines. Since ~2000 it works out of the box for LaTeX and PdfLaTeX.
> Last years I did not need to adjust config files and regenerate
> formats. Hyphenation, default fonts (OK, "apt install cm-super" to
> avoid rasterized fonts is a bit behind defaults though no manual
> config is required) just work. Each document needs a few universal
> lines to setup Russian. Some users may have specific style files but
> generally source documents are quite portable.
>
> Default fonts for LuaTeX and XeTeX do not include Cyrillic any more.
> Every user have to do a decision which fonts should be used even if
> one does not care concerning particular fonts. It increases
> probability to get a file with font configuration that is specific to
> Mac or Windows.
>
> I do not actively use characters from other Unicode planes, however
> sometimes I do it for fun. Getting them completely missing is less 
> preferred than displaying them with low quality font. Specifying all
> fonts requires lengthy config, probably different for LuaTeX and
> XeTeX. At first it is necessary to realize which fonts are available
> for particular glyph. Finally it makes *source* document less
> portable.
>
> "font-family: 'LM Roman 12', 'CMU Serif'" above is a dumb example of
> what I consider relatively high-level config for routine documents
> that do not need to be handcrafted. Unavailable glyph or even font is
> not an error, it just causes lookup of candidates in the following
> items. For TeX maybe it is reasonable to consider fallback to complete
> set of fonts at ones (roman + mono + math) if such combination is
> available.
>
>> If I want to use the GFS Porson as italics from
>> another font, a Didot typeface for example, I can do this:
>
> If it is not a book or at the stage of early draft another scenario is
> possible. Text should just appear in the compiled document, particular 
> font does not matter, its choice is postponed since text content has
> higher priority. Minimal setup in invaluable.
>
> At least with minimal examples I faced another issue: characters
> silently disappears, no warning is generated. Adding babel changes it, 
> but I still believe that especially for documents with carefully
> chosen fonts. It is a serious hidden error to get "invalid char glyph"
> instead of all missed characters.
>
>> [1] If you want to have fallback fonts, you can also do it in
>> LuaTeX by adding some Lua code:
>> https://tex.stackexchange.com/questions/514940/define-fallback-font-for-missing-glyphs-in-lualatex
>
> Would you recommend such code as default for Org? Let's assume that
> some information concerning system fonts are available. I suspect, the 
> accepted answer is not fool-proof. In addition, XeLaTeX requires
> something different.
>
> luaotfload provides fallback feature close to what I expect, however
> it is necessary to explicitly specify script that I would prefer to
> avoid. Moreover it significantly increases compilation time. Sometimes
> LuaLaTeX starts to eat CPU with no progress, emoji does not work for
> some reason.
> I am unsure concerning particular "Noto Sans CJK" since several ones
> are available.
>
> \documentclass{article}
> \usepackage{fontspec}
> \usepackage{unicode-math}
> \directlua{luaotfload.add_fallback
>   ("seriffallback",
>     {
>       "Noto Serif CJK SC:mode=harf;script=sc;",
>       "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
>     })
> }
> % TwemojiMozilla is not shown by viewers, visible in pdftotext
>       %"Noto Color Emoji:mode=harf;"
> % or
>       %"file:/usr/share/fonts/truetype/noto/NotoColorEmoji.ttf:mode=harf;"
> % </usr/share/fonts/truetype/noto/NotoColorEmoji.ttf
> % ! error:  (file /usr/share/fonts/truetype/noto/NotoColorEmoji.ttf)
>     (ttf): loca
> % table not found
> % !  ==> Fatal error occurred, no output PDF file produced!
> \setmainfont{CMU Serif}[RawFeature={fallback=seriffallback}]
>
> \directlua{luaotfload.add_fallback
>   ("sansfallback",
>     {
>       "Noto Sans CJK SC:mode=harf;script=sc;",
>       "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
>     })
> }
> \setsansfont{CMU Sans Serif}[RawFeature={fallback=sansfallback}]
>
> \directlua{luaotfload.add_fallback
>   ("monofallback",
>     {
>       "Noto Sans Mono CJK SC:mode=harf;script=sc;",
>       "file:/usr/lib/firefox/fonts/TwemojiMozilla.ttf:mode=harf;"
>     })
> }
> \setmonofont{CMU Typewriter Text}[RawFeature={fallback=monofallback}]
> \begin{document}
> Test¹ of superscript and ½ fraction.
>
> \textbf{«Теорема».} \emph{Пусть} $α → ∞$ и $\beta \to \infty$.
>
> \verb=Катет= и \textsf{гипотенуза}.
>
> Åå. Text Greek α.
>
> Text utf8x ≠ utf8 and math $utf8x ≠ utf8$.
>
> Randomly chosen examples: "日本", "多喝水", "📞".
>
> \verb=Randomly chosen examples: "日本", "多喝水", "📞".=
> \end{document}
>
>

Re: org-mode export to (latex) PDF

[Stefan Nobis]

Maxim Nikulin <manikulin@gmail.com> writes:

> Do Unicode TeX engines support such combination of fonts?

Yes, they do.

My rather long response was due to my impression that you are quite
surprised that not everything is supported with the default
configuration as you expected.

I wanted to highlight that it is even today a rather hard problem to
mix different scripts (even if typographic quality does not matter)
and that there are quite different expectations of what a sensible
default should look like.

> There are two quite distinct cases: documents with carefully chosen
> fonts (e.g. a book) and everyday notes when particular font is not
> so important.

Yes, indeed. And I hope you see that these two requirements are not
easily satisfied with the same default configuration (and I would say
this is a understatement). The LaTeX community has chosen to prefer a
minimum typographic quality for their defaults and the majority still
concentrates more on latin scripts.

And as I said: A good multi-lingual support for Org is a really huge
undertaking. Unicode alone solves only a rather minor part of these
challenges.

> I mean detection if LuaLaTeX or XeLaTeX is usable from Org code.

Org should be rather flexible about the configured engines. There are
reasons why today all three engine are quite alive and used by
different users. I think it would be possible to improve the support
for all three engines, make the selection easier for the Org user and
go some first steps in better supporting different scripts and
languages. But it is not easy and not a matter of a handful lines of
code.

> Randomly chosen examples: "日本", "多喝水", "📞".

The last glyph ("TELEPHONE RECEIVER") is not visible for me. Remember,
that Unicode gets expanded quite often and it is not easy for font
developers to keep up. I still think that the expectation, that Org
and/or LaTeX will support the whole Unicode range out of the box is a
little bit too far fetched.

-- 
Until the next mail...,
Stefan.

Re: org-mode export to (latex) PDF

[Jean-Christophe Helary]

> On Jul 16, 2021, at 18:20, Stefan Nobis <stefan-ml@snobis.de> wrote:
> 
> The last glyph ("TELEPHONE RECEIVER") is not visible for me. Remember,
> that Unicode gets expanded quite often and it is not easy for font
> developers to keep up. I still think that the expectation, that Org
> and/or LaTeX will support the whole Unicode range out of the box is a
> little bit too far fetched.

If I may insist, my original question was about a "good enough" *or* easily understandable setting for org export to PDF.

I *never* suggested that I, or someone, expected org to support the *whole Unicode range* at any moment.

Since org uses Latex to achieve export to PDF, which is quite a common demand nowadays, something that normal org users can understand should be posted somewhere (*should*, with the meaning that I'd love to do that if I understood even half the discussion here, but it is sadly not the case.)

-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: org-mode export to (latex) PDF

[Stefan Nobis]

Jean-Christophe Helary <lists@traduction-libre.org> writes:

> Since org uses Latex to achieve export to PDF, which is quite a
> common demand nowadays, something that normal org users can
> understand should be posted somewhere.

I second that! I just wanted to try to lower the expectation that
(most) scripts will work out of the box without any kind of manual
configuration.

-- 
Until the next mail...,
Stefan.

Re: org-mode export to (latex) PDF

[Jean-Christophe Helary]

> On Jul 15, 2021, at 4:05, Stefan Nobis <stefan-ml@snobis.de> wrote:
> 
>> Is it possible to provide reasonable defaults for fonts?
> 
> I do not think so. You want Cyrillic. But what about Japanese,
> Chinese, Devanagari, Tamil, Arabic etc? I doubt that there exists a
> single font that supports all these scripts satisfactorily.

The issue I'm pointing at is that we don't need "satisfactorily", we need "good enough".

There are a number of fonts that have 30k+ glyphs. That's good enough for a lot of org-mode users.

https://en.wikipedia.org/wiki/Unicode_font

What we need is 1 or 2 well-documented settings where we specify:

1) the back end
2) the font

It can be org proprieties, or extra elisp code, it doesn't matter. But we need to have good enough PDF export out of the box. We're in the 21st century. People expect (I know, free software, etc. but I'm not talking about that) proper font support for PDF, and well-documented workaround solutions.

I don't mind going through the org→ODF→PDF route, but it's a waste of time. Still, it works *well enough* so it's actually the only viable option.

-- 
Jean-Christophe Helary @brandelune
https://mac4translators.blogspot.com
https://sr.ht/~brandelune/omegat-as-a-book/

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Maxim Nikulin writes:

> It perfectly suits for e.g. a book when camera ready variant is
> required. For routine notes it is better to keep from defaults as
> minimal as possible to minimize problems that may arise a decade
> later. I would prefer to avoid Linux Libertine if I am going to send
> source file to a colleague having another OS.

Linux Libertine (otf version) is included in TeX live. Indeed TeX live
includes a extensive catalog of otf and ttf fonts to be used in LuaTeX
or XeTeX:

https://tug.org/FontCatalogue/opentypefonts.html

As for the original topic of this thread, I am not aware of Japanese
nor of its typographic norms, but it may be interesting to take a look
at the luatexja package: https://ctan.org/pkg/luatexja?lang=en

As for LuaTeX, I think this engine has been a great revolution within
the TeX ecosystem, to the point that it is setting the standard of
everything that is coming new and what is to come. I guess that LuaTeX
will be the natural replacement for pdfTeX (in fact, LuaTeX evolved from
pdfTeX and also took elements from another lesser known, experimental
TeX engine, Omega, which was the first attempt, quite avant-garde, to
create a TeX engine totally Unicode based:
https://en.wikipedia.org/wiki/Omega_(TeX).

Best regards,

Juan Manuel

Re: org-mode export to (latex) PDF

[Jonathan McHugh]

Hi Jean-Christophe,

I have heard positive things with regards to the document management system, Context, for handling non Latin characters.

It may be worth considering as an alternative to Latex.

I recall there was momentum re exporting to Context from Orgmode, hopefully a solid implementation is available.

Kind regards,


Jonathan McHugh
====================
indieterminacy@libre.brussels

July 10, 2021 3:43 PM, "Jean-Christophe Helary" <lists@traduction-libre.org> wrote:

> I was busy last year going back to school and I wrote a research paper for my first year of master
> almost entirely in org-mode.
> 
> My workflow was trivial:
> 
> - write in org-mode
> - enter the bibliography with Zotero
> - export to ODT and open in NeoOffice
> - modify in NeoOffice
> - deliver
> 
> At first, I tried to export the document to PDF but all the Japanese quotes I had were removed from
> the document, which led me to prefer ODT export because it handled them properly.
> 
> I guess it is an issue with the Latex backend and could have been solved with the proper Latex
> settings, but it seems weird that the default settings do not allow for out-of-the-box multilingual
> support.
> 
> Is there an easy way to have that as the default behavior?
> 
> -- 
> Jean-Christophe Helary @brandelune
> https://mac4translators.blogspot.com
> https://sr.ht/~brandelune/omegat-as-a-book

Re: org-mode export to (latex) PDF

[Juan Manuel Macías]

Hi Jonathan,

Jonathan McHugh writes:

> I recall there was momentum re exporting to Context from Orgmode, hopefully a solid implementation is available.

It seems that a member of this mailing list, Jason Ross, is working on a
ConTeXt backend for org: https://github.com/Jason-S-Ross/ox-context/

ConTeXt is a TeX format that uses the LuaTeX engine, but the same
results can be achieved using LaTeX format with the LuaLaTeX version.

Of course, LuaTeX, the TeX engine that runs behind ConTeXt and LuaLaTeX,
was born with a clear multilingual "vocation", and it is the natural
evolution of pdfTeX.

Another option for multilingual documents is to use xeTeX engine and
xeLaTeX format.

ConTeXt is very interesting (a more modular approach, and it is not extended by
macro packages like LaTeX, so that the user already has everything inside
almost out of the box). The problem is that it's not as documented as LaTeX.

Best regards,

Juan Manuel