Dumbquotes in exported source listings

Dumbquotes in exported source listings

[Avdi Grimm]

So I'm writing an eBook with lots of source code listings, using
org-mode-LaTeX->PDF with the "minted" package providing source code
highlighting. For the most part I'm really, really happy with the
toolchain; thank you to those that pointed me in the direction of
"minted".

One issue: my beta reviewers have noted that when copy-and-pasting
source code listings that contain single-quoted strings, they are
getting "smart quotes"--i.e. the first quote is a backquote, the
second quote is a single quote. This breaks the pasted code.

I'm sure I could track the problem down myself eventually, but I
thought someone here might know which part of the toolchain to look at
first for this quote-munging behavior.

Thanks!

--
Avdi Grimm
http://avdi.org

Re: Dumbquotes in exported source listings

[Ben Finney]

Avdi Grimm <groups@inbox.avdi.org> writes:

> One issue: my beta reviewers have noted that when copy-and-pasting
> source code listings that contain single-quoted strings, they are
> getting "smart quotes"--i.e. the first quote is a backquote, the
> second quote is a single quote. This breaks the pasted code.

Those aren't even “smart quotes” (the term usually applied to the quote
characters from Microsoft's standards-violating character set). Those
are what might be called “TeX quotes” (though the convention pre-dates
even TeX), since TeX uses ‘`’ for an opening single quote and ‘``’ for
an opening double quote.

Nowadays with Unicode available ubiquitously we can simply use the
correct typographical quotation marks directly in the plain text file,
but TeX and some other legacy systems don't work very well with Unicode.

-- 
 \      “Often, the surest way to convey misinformation is to tell the |
  `\               strict truth.” —Mark Twain, _Following the Equator_ |
_o__)                                                                  |
Ben Finney

Re: Dumbquotes in exported source listings

[Avdi Grimm]

On Sat, Apr 30, 2011 at 11:13 PM, Ben Finney > Those aren't even
“smart quotes” (the term usually applied to the quote
> characters from Microsoft's standards-violating character set). Those
> are what might be called “TeX quotes” (though the convention pre-dates
> even TeX), since TeX uses ‘`’ for an opening single quote and ‘``’ for
> an opening double quote.

I'll reiterate: this is in literal code-listing sections, not English
text sections. The listings in the plain org file ARE the correct
single-quote character. Something in the pipeline is "cleverly"
converting them to "TeX-quotes", and I don't know what.

-- 
Avdi Grimm
http://avdi.org

Re: Dumbquotes in exported source listings

[Nick Dokos]

Ben Finney <ben+emacs@benfinney.id.au> wrote:

> Avdi Grimm <groups@inbox.avdi.org> writes:
> 
> > ...
> 
> ...
>
> Nowadays with Unicode available ubiquitously we can simply use the
> correct typographical quotation marks directly in the plain text file,
> but TeX and some other legacy systems don't work very well with Unicode.
> 

AFAIK, TeX works fine with UTF8, given the defaults that org-mode export uses.
What problems do you see?

Nick

Re: Dumbquotes in exported source listings

[Sébastien Vauban]

Hi Nick,

Nick Dokos wrote:
> Ben Finney <ben+emacs-TqlCGjI+HWGnbCmf7pGUHw@public.gmane.org> wrote:
>> Avdi Grimm <groups-btwTPOiEE706sZKW81CQKw@public.gmane.org> writes:
>>
>> Nowadays with Unicode available ubiquitously we can simply use the
>> correct typographical quotation marks directly in the plain text file,
>> but TeX and some other legacy systems don't work very well with Unicode.

I really hate those "smart quotes", better named "dumb quotes", because:

- I don't find them attractive at all, but that's personal, and could depend
  on font families, etc.

- they're not correctly converted when playing with the coding system: from
  UTF-8 to Latin 1, for example. That's really nasty.

I was/am planning to have a function for converting them on the fly when
pasting... or when saving the Org buffer... or at any better moment. Though, I
didn't get the right code to do so yet...

> AFAIK, TeX works fine with UTF8, given the defaults that org-mode export uses.
> What problems do you see?

UTF-8 is correctly handled in LaTeX documents, but not in every environment:
Listings has problems, even if a Listings exist for UTF-8, and so for many
other environments.

So I would say that UTF-8 usage is still not yet fully supported in TeX.

Best regards,
  Seb

-- 
Sébastien Vauban

Re: Dumbquotes in exported source listings

[Nick Dokos]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> wrote:


> > AFAIK, TeX works fine with UTF8, given the defaults that org-mode export uses.
> > What problems do you see?
> 
> UTF-8 is correctly handled in LaTeX documents, but not in every environment:
> Listings has problems, even if a Listings exist for UTF-8, and so for many
> other environments.
> 
> So I would say that UTF-8 usage is still not yet fully supported in TeX.
> 

Yes, I misspoke: I should have said LaTeX, not TeX (but there are
versions of TeX that work correctly with UTF-8 - XeTeX and Omega are two
such I think, but I have never used them, so take with the appropriate
grain of salt.) And I'm sure I was being very parochial in my thinking:
inputenc/fontenc != "works fine".

And I'm also sure there are LaTeX packages that mishandle UTF-8, although I
didn't realize that listings is one of them. What are the problems?

But there are probably many pitfalls here that I've never even thought
of.  Can somebody refer me to a concise discussion of the problems
(before *this* discussion becomes even more OT for this list)? I don't
want to wade deep: just to understand what the main problems are.

Thanks,
Nick

Re: Dumbquotes in exported source listings

[Ben Finney]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com>
writes:

> Nick Dokos wrote:
> > Ben Finney <ben+emacs@benfinney.id.au> wrote:
> >> Nowadays with Unicode available ubiquitously we can simply use the
> >> correct typographical quotation marks directly in the plain text
> >> file, but TeX and some other legacy systems don't work very well
> >> with Unicode.
>
> I really hate those "smart quotes", better named "dumb quotes",

Again, that's just confusing the terminology, since both “smart quotes”
and “dumb quotes” already have connotations. If you want a neutral term
for the typographically-correct quotation marks, try “directional
quotes”.

Here is a discussion of the history of the “`” and “'” usage and why
it's a problem <URL:http://www.cl.cam.ac.uk/~mgk25/ucs/quotes.html>.

-- 
 \     “Please do not feed the animals. If you have any suitable food, |
  `\                     give it to the guard on duty.” —zoo, Budapest |
_o__)                                                                  |
Ben Finney

Re: Dumbquotes in exported source listings

[Ben Finney]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Ben Finney <ben+emacs@benfinney.id.au> wrote:
>
> > TeX and some other legacy systems don't work very well with Unicode.
>
> AFAIK, TeX works fine with UTF8, given the defaults that org-mode
> export uses.

To be clear, I don't have any issue with Org and Unicode (I've never
exported an Org file to TeX). I was mainly addressing the terminology
for the style of quotes described by the OP.

-- 
 \                   “Too many Indians spoil the golden egg.” —Sir Joh |
  `\                                                   Bjelke-Petersen |
_o__)                                                                  |
Ben Finney

Missing Introduction and About sections in LaTeX export

[Avdi Grimm]

I have an Org document I'm exporting to PDF through LaTeX using the
"report" class.

The two first top-level sections in the document, titled "About" and
"Introduction", are completely missing from the generated .tex and
PDF. The exported files start on the third section.

Coincidentally(?) those are the "front matter" sections which I don't
want included in the contents. However, I have done nothing to mark
them as such. They have no classes or properties. As far as I can tell
there is nothing making them different from other sections.

Any ideas why they might be elided by the LaTeX exporter in Org 7.5?

Follow-up: assuming I can get them to export at all, how can I mark
*just* those two headings to be exported using the "starred" form so
that they are not included in the chapter numbering?

Thanks,

-- 
Avdi Grimm
http://avdi.org

Re: Missing Introduction and About sections in LaTeX export

[Avdi Grimm]

Update: my initial problem with missing sections has gone away as
mysteriously as it came. Still looking for a way to use the "starred"
form for LaTeX export of certain headlines.

-- 
Avdi Grimm
http://avdi.org

Re: Missing Introduction and About sections in LaTeX export

[Eric S Fraga]

Avdi Grimm <groups@inbox.avdi.org> writes:

> Update: my initial problem with missing sections has gone away as
> mysteriously as it came.

Just to add that I do have similar problems every now and again: some
sections fail to export, usually the first or first two.  Subsequently,
after editing the document, these sections re-appear in the exported
document.  The problem is only with latex export when this happens:
other export targets will export everything I expect.  This has been
happening to me for >1 year but very sporadically and infrequently and
I've not seen it recently... (fingers crossed).

I've never managed to create a minimal example that illustrates this
bug, however, so it has been difficult to debug (or pose the problem
clearly enough for others to debug).

You have not imagined it! ;-)

-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.0.50.1
: using Org-mode version 7.5 (release_7.5.209.g1a687)

Re: Missing Introduction and About sections in LaTeX export

[Nick Dokos]

Avdi Grimm <groups@inbox.avdi.org> wrote:

> I have an Org document I'm exporting to PDF through LaTeX using the
> "report" class.
> 
> The two first top-level sections in the document, titled "About" and
> "Introduction", are completely missing from the generated .tex and
> PDF. The exported files start on the third section.
> 

Can't reproduce this:

--8<---------------cut here---------------start------------->8---
#+latex_class: report

* About

foo

* Introduction

bar

* First real section

baz
--8<---------------cut here---------------end--------------->8---

when exported to latex it gives:

,----
| ...
| \part{About}
| \label{sec-1}
| 
| 
| foo
| \part{Introduction}
| \label{sec-2}
| 
| 
| bar
| \part{First real section}
| \label{sec-3}
| 
| 
| baz
`----

and all three appear in the TOC.

Org-mode version 7.5 (release_7.5.181.g66caf)
GNU Emacs 24.0.50.1 (x86_64-unknown-linux-gnu, GTK+ Version 2.22.0) of 2011-04-13

> Coincidentally(?) those are the "front matter" sections which I don't
> want included in the contents. However, I have done nothing to mark
> them as such. They have no classes or properties. As far as I can tell
> there is nothing making them different from other sections.
> 
> Any ideas why they might be elided by the LaTeX exporter in Org 7.5?
> 
> Follow-up: assuming I can get them to export at all, how can I mark
> *just* those two headings to be exported using the "starred" form so
> that they are not included in the chapter numbering?
> 

Just star them by hand in the tex file after exporting for the last
time: it'll take two seconds. You may be able to do it from Org by
writing a custom function (C-h v org-export-latex-classes <RET> for
details) but I suspect that the effort is just not worth it.

Nick

Re: Missing Introduction and About sections in LaTeX export

[Avdi Grimm]

On Wed, Apr 27, 2011 at 5:41 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:
> Just star them by hand in the tex file after exporting for the last
> time: it'll take two seconds. You may be able to do it from Org by
> writing a custom function (C-h v org-export-latex-classes <RET> for
> details) but I suspect that the effort is just not worth it.


Ugh. This is a (somewhat) living document; manually tweaking the .tex
after export isn't really an option.

I was hoping there was a tag or property that I could set on a section
indicating it is frontmatter/backmatter.

-- 
Avdi Grimm
http://avdi.org

Re: Missing Introduction and About sections in LaTeX export

[Nick Dokos]

Avdi Grimm <avdi@avdi.org> wrote:

> On Wed, Apr 27, 2011 at 5:41 PM, Nick Dokos <nicholas.dokos@hp.com> wrote:
> > Just star them by hand in the tex file after exporting for the last
> > time: it'll take two seconds. You may be able to do it from Org by
> > writing a custom function (C-h v org-export-latex-classes <RET> for
> > details) but I suspect that the effort is just not worth it.
> 
> 
> Ugh. This is a (somewhat) living document; manually tweaking the .tex
> after export isn't really an option.
> 
> I was hoping there was a tag or property that I could set on a section
> indicating it is frontmatter/backmatter.
> 

I don't know of an easy way within Org - somebody else might have better
ideas.

If I were in your position, I'd probably write a simple Makefile to
produce the PDF and incorporate a simple post-processing awk script
to do the transformation. Or write an elisp function to run as
part of org-export-latex-final-hook perhaps.

Nick

Re: Missing Introduction and About sections in LaTeX export

[Sébastien Vauban]

Hi Avdi,

Nick Dokos wrote:
> Avdi Grimm <avdi-Y4aQArZDtos@public.gmane.org> wrote:
>> On Wed, Apr 27, 2011 at 5:41 PM, Nick Dokos <nicholas.dokos-VXdhtT5mjnY@public.gmane.org> wrote:
>> > Just star them by hand in the tex file after exporting for the last
>> > time: it'll take two seconds. You may be able to do it from Org by
>> > writing a custom function (C-h v org-export-latex-classes <RET> for
>> > details) but I suspect that the effort is just not worth it.
>> 
>> Ugh. This is a (somewhat) living document; manually tweaking the .tex
>> after export isn't really an option.
>> 
>> I was hoping there was a tag or property that I could set on a section
>> indicating it is frontmatter/backmatter.
>
> I don't know of an easy way within Org - somebody else might have better
> ideas.
>
> If I were in your position, I'd probably write a simple Makefile to produce
> the PDF and incorporate a simple post-processing awk script to do the
> transformation. Or write an elisp function to run as part of
> org-export-latex-final-hook perhaps.

What about just inserting

  #+LaTeX: \backmatter{}

and the like where applicable in the Org file?

Best regards,
  Seb

-- 
Sébastien Vauban

Re: Missing Introduction and About sections in LaTeX export

[Nick Dokos]

[forgot to cc: the list]

Sébastien Vauban <wxhgmqzgwmuf@spammotel.com> wrote:

> Hi Avdi,
> 
> Nick Dokos wrote:
> > Avdi Grimm <avdi@avdi.org> wrote:
> >> On Wed, Apr 27, 2011 at 5:41 PM, Nick Dokos <nicholas.dokos@hp.com> wrot=
> e:
> >> > Just star them by hand in the tex file after exporting for the last
> >> > time: it'll take two seconds. You may be able to do it from Org by
> >> > writing a custom function (C-h v org-export-latex-classes <RET> for
> >> > details) but I suspect that the effort is just not worth it.
> >>=20
> >> Ugh. This is a (somewhat) living document; manually tweaking the .tex
> >> after export isn't really an option.
> >>=20
> >> I was hoping there was a tag or property that I could set on a section
> >> indicating it is frontmatter/backmatter.
> >
> > I don't know of an easy way within Org - somebody else might have better
> > ideas.
> >
> > If I were in your position, I'd probably write a simple Makefile to produ=
> ce
> > the PDF and incorporate a simple post-processing awk script to do the
> > transformation. Or write an elisp function to run as part of
> > org-export-latex-final-hook perhaps.
> 
> What about just inserting
> 
>   #+LaTeX: \backmatter{}
> 
> and the like where applicable in the Org file?
> 

From a quick read of the description of \frontmatter, it is not clear to
me that it is going to work the way the OP wants, but it is indeed
worth trying: I forgot about that completely.

[trying it...]
I just ran a quick test: \frontmatter seems to be undefined in the
report class - it's only defined in the book class.

Thanks,
Nick

Re: Dumbquotes in exported source listings

[Nick Dokos]

Avdi Grimm <groups@inbox.avdi.org> wrote:

> So I'm writing an eBook with lots of source code listings, using
> org-mode-LaTeX->PDF with the "minted" package providing source code
> highlighting. For the most part I'm really, really happy with the
> toolchain; thank you to those that pointed me in the direction of
> "minted".
> 
> One issue: my beta reviewers have noted that when copy-and-pasting
> source code listings that contain single-quoted strings, they are
> getting "smart quotes"--i.e. the first quote is a backquote, the
> second quote is a single quote. This breaks the pasted code.
> 

In cases like this, an example would help: what you start with in the org
file, what it looks like in the tex file (and any other intermediate files[fn:1])
and what the resulting PDF file looks like. FWIW, I tried exporting to PDF
the following:

--8<---------------cut here---------------start------------->8---

* foo

#+begin_src elisp :results value :exports both

(setq foo '(bar baz))

#+end_src

#+results:
| bar | baz |

--8<---------------cut here---------------end--------------->8---

and then, while viewing the PDF file with xpdf, tried pasting the
lisp-expression from the PDF back into the org file (and even evaluated
it). This worked without a hitch.

So please give us more information.

Nick

PS. IIRC, a long time there was a problem like this: trying to copy and paste
a lisp expression from the org manual resulted in an error because of a "wrong"
quote. I couldn't remember the details so I searched the mailing list. The thread
starts at

       http://thread.gmane.org/gmane.emacs.orgmode/6610

but a) the warning does not appear in the Org manual any longer and b) I
now cannot reproduce *that* problem at all (not with xpdf, not with
evince and not even with acroread). But it *was* a problem with an older
version of acroread (and, apparently evince as well), so maybe that's
where you should concentrate your investigations: what PDF viewer are you
using and what is the version?

Footnotes:

[fn:1] Don't know anything about minted so this part of the toolchain is
opaque to me: I don't know if it produces other intermediate files.