Notes about citations in Org (part 3)

Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

This post is an update about "wip-cite-new" branch, which I rebased
a few minutes ago. It introduces changes that make previous reports
obsolete. In particular, the "demo" I wrote in the previous thread is no
longer applicable directly, even though the ideas developed there still
hold. Please comment on this thread from now on.

The first noteworthy changes are about the syntax. 

1. I re-introduced objects in prefixes and suffixes, which can again
   contain usual emphasis (bold, italic, strike-through, underscore,
   code, verbatim), subscript, superscript, entities and LaTeX
   fragments.

2. I removed support for "suppress author" minus symbol.

The complete syntax is now

  [cite/style:common prefix ;prefix @key suffix; ... ; common suffix]

Everything is optional, except the brackets, "cite" and the colon. Also
the citation must contain at least a key. So its minimal form is:

  [cite:@key]

Spaces are meaningful in prefixes and suffixes, except for those right
after the colon and those before the closing bracket.

As a reminder, the syntax also supports the following keywords.

  #+bibliography: file
  #+cite_export: [processor] [bibliography-style] [citation-style]
  #+print_bibliography: [style]

For complete information about styles, please refer to the last post,
"(Not so) Short note about citations in Org".

From a technical standpoint, as a consequence of item 1,

  (org-element-property :prefix citation)

or

  (org-element-property :prefix citation-reference)

may return so-called "secondary strings", i.e., lists of strings and
objects. In order to deal with this additional complexity, I implemented
some specific tooling in "oc.el": `org-cite-concat' and
`org-cite-mapconcat' are similar to `concat' and `mapconcat', but can
also operate on secondary strings and objects. For example, a processor
developer could write:

  (org-cite-concat "(" (org-element-property :prefix citation) ")")

and later export the result with `org-export-data' as if the central
argument were a string.

This new revision also introduces additional tooling, which I won't
comment but is worth looking at if you consider giving a try at writing
a citation processor: `org-cite-key-number', `org-cite-wrap-citation',
`org-cite-make-paragraph', and `org-cite-emphasize'.

The other incompatibility is about function signatures for
`:export-bibliography' and `:export-finalizer' keywords. This change
should not be an issue at this point in the development. Citation
processors are registered using `org-cite-register-processor'. Its
docstring follows:

--8<---------------cut here---------------start------------->8---
Mark citation processor NAME as available.

NAME is a symbol.  BODY is a property list, where the following optional keys
can be set:

  `:activate'

    Function activating a citation.  It is called with a single argument: a
    citation object extracted from the current buffer.  It may add text
    properties to the buffer.  If it is not provided, `org-cite-fontify-default'
    is used.

  `:export-bibliography'

    Function rendering a bibliography.  It is called with five arguments:
    the list of citation keys used in the document, as strings, a list
    of bibliography files, the style, as a string or nil, the export back-end,
    as a symbol, and the communication channel, as a property list.

    It is called at each \"print_bibliography\" keyword in the parse tree.
    It may return a string or nil.  When it returns nil, the keyword is ignored.
    Otherwise, the string it returns replaces the keyword in the export output.

  `:export-citation'    (mandatory for \"export\" capability)

    Function rendering citations.  It is called with four arguments: a citation
    object, the style, as a string or nil, the export back-end, as a symbol,
    and the communication channel, as a property list.

    It is called on each citation object in the parse tree.  It may return
    a string or nil.  When it returns nil, the citation is ignored.  Otherwise,
    the string it returns replaces the citation object in the export output.

  `:export-finalizer'

    Function called at the end of export process.  It must accept six arguments:
    the output, as a string, a list of citation keys used in the document,
    a list of bibliography files, a list of bibliography styles requested by
    \"print_bibliography\" keywords, as strings or nil, the export back-end,
    as a symbol, and the communication channel, as a property list

    It must return a string, which will become the final output from the export
    process, barring subsequent modifications from export filters.

  `:follow'

    Function called to follow a citation.  It accepts two arguments, the
    citation or citation reference object at point, and any prefix argument
    received during interactive call of `org-open-at-point'.
--8<---------------cut here---------------end--------------->8---

Ah! I almost forgot… This new revision introduces a poor man citation
processor for BibTeX bibliography files, named "basic", in the file
"oc-basic.el". Here is its commentary section:

--8<---------------cut here---------------start------------->8---
The `basic' citation processor provides "activate", "follow" and
"export" capabilities.

More specifically, "activate" capability re-uses default
fontification, but highlights with `error' face any unknown
citation key according to the bibliography defined in the document.

On a citation key, "follow" capability moves point to the
corresponding entry in the current bibliography.  Elsewhere on the
citation, it asks the user to follow any of the keys cited there,
with completion.

"export" capability supports the following styles for citations:
  - author
  - nocite
  - note
  - numeric
  - text
  - year
  - author-year (default)
It also supports the following styles for bibliography:
  - plain
  - numeric
  - author-year (default)

It assumes bibliography files are in BibTex format.

Disclaimer: this citation processor is meant to be a proof of
concept, and possibly a fall-back mechanism when nothing else is
available. It is too limited for any serious use case.

With that out of the way, if you still want to use this processor,
you may set `org-cite-activate-processor',
`org-cite-follow-processor', or `org-cite-export-processor' to
`basic'.
--8<---------------cut here---------------end--------------->8---

As pointed out above, don't expect too much from it. Nevertheless,
feedback is welcome. Improvements are welcome too, but at this point,
I would suggest to spend energy porting "citeproc-org" or "org-ref" to
this API instead.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Thu, Apr 29, 2021 at 10:06 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

>  ... at this point, I would suggest to spend energy porting "citeproc-org" or "org-ref" to
>  this API instead.

+1

Thanks for the update, and the work, Nicolas!

I agree any remaining details are best worked out in the process of
adapting existing solutions to this.

I assume it should be a relatively small step for citeproc-org to
support this, but obviously that's for Andras to confirm.

I will do my small part by updating my function to generate that
modified syntax from a list of keys.

Bruce

Re: Notes about citations in Org (part 3)

[Bastien]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> This post is an update about "wip-cite-new" branch, which I rebased
> a few minutes ago. 

I'm attaching a call for help to this thread so that more people can
test this before we merge it into master.

Thanks again for this important work!

-- 
 Bastien

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

Nicolas - just a little question on one detail:

On Thu, Apr 29, 2021 at 10:06 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> On a citation key, "follow" capability moves point to the
> corresponding entry in the current bibliography.  Elsewhere on the
> citation, it asks the user to follow any of the keys cited there,
> with completion.

What does this last clause mean?

I don't see completion mentioned in the oc.el file.

Bruce

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Nicolas - just a little question on one detail:
>
> On Thu, Apr 29, 2021 at 10:06 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> On a citation key, "follow" capability moves point to the
>> corresponding entry in the current bibliography.  Elsewhere on the
>> citation, it asks the user to follow any of the keys cited there,
>> with completion.
>
> What does this last clause mean?
>
> I don't see completion mentioned in the oc.el file.

Note the above is about "oc-basic.el", not "oc.el".

Anywhere on the following citation:

   [cite:@a]

`org-open-at-point' will try to find key "a" in the current
bibliography.

On the following citation:

  [cite:@a;@b]

if point is on "cite:", `org-open-at-point' will ask if it should find
key "a" or key "b".

If point is on @a, `org-open-at-point' will look for "a" without any
question.


Regards,

-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Sun, May 2, 2021 at 10:51 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> `org-open-at-point' will try to find key "a" in the current
> bibliography.

Got it.

I was thinking maybe it would complete the key. That can be for other
developers to add.

Would it be possible for you, or someone else on the list, to post a
minimal file we could load with `emacs -Q` to test the branch?

I guess this is a broader question: how do org developers test their branches?

Bruce

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> I was thinking maybe it would complete the key. That can be for other
> developers to add.

You missed the "basic" part in "oc-basic.el". ;)

You can write a function for creating citations with completion outside
of the oc API.

> Would it be possible for you, or someone else on the list, to post a
> minimal file we could load with `emacs -Q` to test the branch?

Make sure to checkout "wip-cite-new". Once there, you run "make".

Then, the following mini-init.el file should be enough.

--8<---------------cut here---------------start------------->8---
(let ((org-root "/path/to/org/git/directory/"))
  (when (file-directory-p org-root)
    (dolist (dir '("lisp" "contrib/lisp" "testing" "testing/lisp"))
      (add-to-list 'load-path (expand-file-name dir org-root)))))
--8<---------------cut here---------------end--------------->8---

> I guess this is a broader question: how do org developers test their
> branches?

Since I use development Org, I simply checkout the branch, run "make &&
org-reload", and test it.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Sun, May 2, 2021 at 3:32 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Make sure to checkout "wip-cite-new". Once there, you run "make".
>
> Then, the following mini-init.el file should be enough.

Perfect; thank you!

So if I want to add to that mini-init file the config so that the
basic processor is correctly setup, what am I doing wrong here?

(setq org-cite-activate-processor 'basic
      org-cite-follow-processor 'basic
      org-cite-export-processor 'basic)

Once I have that working, what sort of feedback are you most
interested in from testers?

Bruce

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

Hello all,

I've had a short play with this.  Looks nice!

A few points:

1. if the BiBTeX entry has, for instance, \& to escape the & for use
   with LaTeX, org translates the \ to $\backslash$ and then the &
   causes a problem compiling the resulting LaTeX.

2. the suppressed author case does not seem to work.

3. if I cite a paper with many authors, *all* of them get listed.  It
   would be useful to have some control over this, asking, for instance,
   for only the first or the first N to be listed in the citation (but
   all of them in the bibliography).
  
thank you,
eric

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-393-gd44bd5

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

Hi Eric,

On Mon, May 3, 2021 at 9:47 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> 2. the suppressed author case does not seem to work.

He removed the suppress author variant on the individual cited items,
so I think the same effect he means to achieve with the "year" style.

Bruce

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> So if I want to add to that mini-init file the config so that the
> basic processor is correctly setup, what am I doing wrong here?
>
> (setq org-cite-activate-processor 'basic
>       org-cite-follow-processor 'basic
>       org-cite-export-processor 'basic)

You need to (require 'oc-basic), otherwise, the `basic' processor is not
registered.

Moreover, `org-cite-export-processor' expects a more complex value than
the other variables. At the bare minimum, it should be '(basic).

Anyway, I suggest to let it nil, and select it at the document level
instead, with

  #+cite_export: basic bibstyle citestyle

syntax.

> Once I have that working, what sort of feedback are you most
> interested in from testers?

There is not much to test anyway. It boils down to trying the various
styles, with bibliography files containing entries of different types.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> I've had a short play with this.  Looks nice!

Thank you!

> A few points:
>
> 1. if the BiBTeX entry has, for instance, \& to escape the & for use
>    with LaTeX, org translates the \ to $\backslash$ and then the &
>    causes a problem compiling the resulting LaTeX.

Oops. I'll fix it in a few hours. I need to treat LaTeX-derived
back-ends specially in the context of BibTeX files. 

I'll let you know when it's done.

> 2. the suppressed author case does not seem to work.

As noted by Bruce D'Arcus, there is no more suppressed author shortcut.
You can use "year" style instead.

> 3. if I cite a paper with many authors, *all* of them get listed.  It
>    would be useful to have some control over this, asking, for instance,
>    for only the first or the first N to be listed in the citation (but
>    all of them in the bibliography).

I don't know how to do that cleanly. However, I think this is already
too smart a feature for "basic" back-end. So, I'd like to punt on this
one.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

FYI, I implemented support for the org-cite syntax and the core styles
that I think will likely end up in citeproc-org, on the `org-cite`
branch, represented by this PR.

https://github.com/bdarcus/bibtex-actions/pull/113

So if you run the 'bibtex-actions-insert-citations' command and select
one or more references, it will prompt you for the prefix and style,
and insert the correct cite syntax in the document.

And, of course, the 'new-org-cite' branch of org will immediately
recognize it as a citation!

This is the command I'm using to test the two together, with the
latter two loaded from the bibtex-actions repo:

emacs -Q -l cite-init.el -l ../bibtex-actions/test/install.el -l
../bibtex-actions/test/bibtex-actions.el

I still haven't managed to get the processor configured though.

Bruce

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Mon, May 3, 2021 at 12:42 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Anyway, I suggest to let it nil, and select it at the document level
> instead, with
>
>   #+cite_export: basic bibstyle citestyle

What is the significance of the last two items?

In a CSL implementation like citeproc-el, both are defined in a single style.

Are they defined separately in other systems?

And if the processor is citeproc-org, where does one put the "foo.csl" style?

Bruce

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Mon, May 3, 2021 at 12:42 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> Anyway, I suggest to let it nil, and select it at the document level
>> instead, with
>>
>>   #+cite_export: basic bibstyle citestyle
>
> What is the significance of the last two items?

These are the default styles used for bibliography and citations, in the
sense of Org "styles", i.e.,

  [cite/citestyle:...]
  #+print_bibliography: bibstyle

> In a CSL implementation like citeproc-el, both are defined in a single
> style.
>
> Are they defined separately in other systems?

IIUC, you can combine different bibliography and citation styles in
LaTeX. Maybe some LaTeX users could confirm this.

> And if the processor is citeproc-org, where does one put the "foo.csl"
> style?

That may be orthogonal. If you use a given "foo.csl", does it still make
sense to provide styles to print_bibliography keywords and citations?

If that's the case, the citeproc style file could be introduced with
a oc-citeproc specific keyword.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Denis Maier]

Hi,

Am 04.05.2021 um 08:47 schrieb Nicolas Goaziou:
> Hello,
> 
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> 
>> On Mon, May 3, 2021 at 12:42 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>>
>>> Anyway, I suggest to let it nil, and select it at the document level
>>> instead, with
>>>
>>>    #+cite_export: basic bibstyle citestyle
>>
>> What is the significance of the last two items?
> 
> These are the default styles used for bibliography and citations, in the
> sense of Org "styles", i.e.,
> 
>    [cite/citestyle:...]
>    #+print_bibliography: bibstyle
> 
>> In a CSL implementation like citeproc-el, both are defined in a single
>> style.
>>
>> Are they defined separately in other systems?
> 
> IIUC, you can combine different bibliography and citation styles in
> LaTeX. Maybe some LaTeX users could confirm this.
Yes, that's true for biblatex.

Denis

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Monday,  3 May 2021 at 09:58, Bruce D'Arcus wrote:
>> 2. the suppressed author case does not seem to work.
>
> He removed the suppress author variant on the individual cited items,
> so I think the same effect he means to achieve with the "year" style.

Makes sense.  Thank you.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-480-g479a3d

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Monday,  3 May 2021 at 18:48, Nicolas Goaziou wrote:
> I don't know how to do that cleanly. However, I think this is already
> too smart a feature for "basic" back-end. So, I'd like to punt on this
> one.

Point taken.  Reasonable approach.  Thank you.
-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-480-g479a3d

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Oops. I'll fix it in a few hours. I need to treat LaTeX-derived
> back-ends specially in the context of BibTeX files. 
>
> I'll let you know when it's done.

Done.

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 11:33, Nicolas Goaziou wrote:
> Done.

Thank you.  Seems to work just fine.

Question for the longer term: for LaTeX export, I will be wanting to
have the [cite:] constructs export to BiBTeX code.  Will this be
possible in due course?  For other targets (ODT, HTML), what has been
done in this branch is fantastic but, for LaTeX, publishers will expect
BiBTeX.

Thanks again,
eric

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Joost Kremers]

On Tue, May 04 2021, Eric S Fraga wrote:
> Question for the longer term: for LaTeX export, I will be wanting to
> have the [cite:] constructs export to BiBTeX code.  Will this be
> possible in due course?  For other targets (ODT, HTML), what has been
> done in this branch is fantastic but, for LaTeX, publishers will expect
> BiBTeX.

Pedantic nit-pick: they *should* be expecting and using biblatex. (But perhaps
that is what you meant already. :-) )

-- 
Joost Kremers
Life has its moments

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 464 bytes --]

On Tue, May 4, 2021, 5:51 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> On Tuesday,  4 May 2021 at 11:33, Nicolas Goaziou wrote:
> > Done.
>
> Thank you.  Seems to work just fine.
>
> Question for the longer term: for LaTeX export, I will be wanting to
> have the [cite:] constructs export to BiBTeX code.  Will this be
> possible in due course?


It should be trivial to map this to latex; cite/text -> \citet and such.

I agree it makes sense to include.

Bruce

[-- Attachment #2: Type: text/html, Size: 930 bytes --]

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Question for the longer term: for LaTeX export, I will be wanting to
> have the [cite:] constructs export to BiBTeX code.  Will this be
> possible in due course?  For other targets (ODT, HTML), what has been
> done in this branch is fantastic but, for LaTeX, publishers will expect
> BiBTeX.

The branch provides an API to develop citation processors. The goal is
not to ultimately use "oc-basic.el", but to activate more specialized
processors, like, for example "oc-citeproc.el", but also
"oc-biblatex.el" or "oc-natbib.el", depending on your requirements.
Those are not implemented yet, however.

You can help, for example, by providing a list of style that you would
like to be provided by hypothetical (at this point) "oc-natbib.el" (or
another one), with the desired LaTeX output in different cases:

  [cite:@key] => ???
  [cite:prefix @key suffix] => ???
  [cite:global; prefix @key1; @key2; suffix] => ???

Ditto for "print_bibliography" keywords. I think you get the idea.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Denis Maier]

Am 04.05.2021 um 12:10 schrieb Joost Kremers:
> 
> On Tue, May 04 2021, Eric S Fraga wrote:
>> Question for the longer term: for LaTeX export, I will be wanting to
>> have the [cite:] constructs export to BiBTeX code.  Will this be
>> possible in due course?  For other targets (ODT, HTML), what has been
>> done in this branch is fantastic but, for LaTeX, publishers will expect
>> BiBTeX.
> 
> Pedantic nit-pick: they *should* be expecting and using biblatex. (But perhaps
> that is what you meant already. :-) )
> 
But they most likely won't, or has that changed in recent years?

Denis

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 6:29 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> The branch provides an API to develop citation processors. The goal is
> not to ultimately use "oc-basic.el", but to activate more specialized
> processors, like, for example "oc-citeproc.el", but also
> "oc-biblatex.el" or "oc-natbib.el", depending on your requirements.
> Those are not implemented yet, however.
>
> You can help, for example, by providing a list of style that you would
> like to be provided by hypothetical (at this point) "oc-natbib.el" (or
> another one), with the desired LaTeX output in different cases:

It seems like those interested in this could collaborate on a table of
style/command mappings for the above export processors.

Anyone interested in working on this?

If yes, how/where?

I did add an issue to the citeproc-org repo, but it's aimed at that
processor, and we really need a wiki or something to do this.

https://github.com/andras-simonyi/citeproc-org/issues/16

Bruce

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 2:47 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> "Bruce D'Arcus" <bdarcus@gmail.com> writes:

> > And if the processor is citeproc-org, where does one put the "foo.csl"
> > style?
>
> That may be orthogonal. If you use a given "foo.csl", does it still make
> sense to provide styles to print_bibliography keywords and citations?
>
> If that's the case, the citeproc style file could be introduced with
> a oc-citeproc specific keyword.

Yes, that probably makes sense.

András can figure that it out in any case.

Bruce

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 6:48 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> It seems like those interested in this could collaborate on a table of
> style/command mappings for the above export processors.
>
> Anyone interested in working on this?
>
> If yes, how/where?

I added a wiki page on the bibtex-actions site here:

https://github.com/bdarcus/bibtex-actions/wiki/Org-cite

Perhaps the bibtex/biblatex folks can help with those details?

Bruce

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 12:10, Joost Kremers wrote:
> Pedantic nit-pick: they *should* be expecting and using biblatex. (But
> perhaps that is what you meant already. :-) )

Well, luckily, in my field most of the journals allow submissions in
LaTeX and provide style files.  The bibliography is usually the contents
as generated by BiBTeX inserted directly into the LaTeX file.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 06:11, Bruce D'Arcus wrote:
> It should be trivial to map this to latex; cite/text -> \citet and such.

Indeed.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 07:58, Bruce D'Arcus wrote:
> Perhaps the bibtex/biblatex folks can help with those details?

For natbib, there is a good summary at

http://merkel.texture.rocks/Latex/natbib.php

I only ever typically use "Author (year)" (\citet{key}) and "(Author,
year)" (\citep{key}) in most of my writing, i.e. my use case is very
simple compared with many here.  For some journals, the citing style
will be numeric, typically a superscript but sometimes "[1]" with
bibliography sorted by order of appearance.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> On Tuesday,  4 May 2021 at 07:58, Bruce D'Arcus wrote:
>> Perhaps the bibtex/biblatex folks can help with those details?
>
> For natbib, there is a good summary at
>
> http://merkel.texture.rocks/Latex/natbib.php

Thanks.

Naive question: it doesn't say how to print the references, how is it
done?

Also, what styles (including shortcuts if necessary) names could we
chose? Asterisk is not allowed in the citation style (but that could
change, if necessary).

E.g.,

  "parens" or "p" -> citep
  "text" or "t" -> citet (default?)
  "pf" or "parens-full" -> citep*

What about alt, alp, alt*, alt*, and all Cite variants?

It is possible to include sub-styles, e.g.,

 [cite/alt/caps:...] could be equivalent to [cite/caps/alt:...] and
 would return \Citealt{}

Naming is hard (and boring ;). Suggestions welcome.


Regards,

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 9:15 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Also, what styles (including shortcuts if necessary) names could we
> chose? Asterisk is not allowed in the citation style (but that could
> change, if necessary).

I'll let the bibtex folks speak to that perspective, but ...

> E.g.,
>
>   "parens" or "p" -> citep

... in the start of my mapping table, I suggest this as default. This
may be a CSL bias though.

>   "text" or "t" -> citet (default?)
>   "pf" or "parens-full" -> citep*
>
> What about alt, alp, alt*, alt*, and all Cite variants?
>
> It is possible to include sub-styles, e.g.,

I agree it would be better to allow variants, because it would make it
easier to have the top level consistent among export processors (and
therefore better for users in the end), without ending up with long
lists of style names that only apply to, say, biblatex.

Bruce

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

Sorry, just to be crytal clear ...

On Tue, May 4, 2021 at 9:21 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> > It is possible to include sub-styles, e.g.,
>
> I agree it would be better to allow variants ...

I should have written "sub-styles"; as in, I agree with Nicolas.

Also agree on shortcuts, which I've started to add to my table as well.

Bruce

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

I'm attaching the tentative style/command and shortcut mapping I came
up with, with help from Eric and Joost.

HTH.

I do have \autocite as default for biblatex. Joost can weigh in if he
thinks this is a problem.

Bruce

[-- Attachment #2: org-cite-styles.org --]
[-- Type: application/octet-stream, Size: 1487 bytes --]

This provides a mapping from possible =org-cite= style names to
different export formats.

For =citeproc-org=, the asterisk represents a suggested supported style.

| org-cite substyle  | org-cite shortcut | natbib      | biblatex    | citeproc-org |
|--------------------+-------------------+-------------+-------------+--------------|
| default (no style) |                   | \citep      | \autocite   | default      |
| text               | t                 | \citet      | \textcite   | *            |
| author             | a                 | \citeauthor | \citeauthor | *            |
| title              | ti                |             | \citetitle  | *            |
| year               | y                 | \citeyear   | \citeyear   | *            |
| locators           | l                 |             |             | *            |
| nocite             | n                 |             | \nocite     | *            |

Note that CSL implementations like =citeproc-el= are based around
biblatex autocite-like functionality as default. The CSL style controls
how the default cite command is formatted. While this is less-flexible
than biblatex, for example, it's simpler for the user, as you use the
same base command regardless of whether you are using an author-date,
numeric, or note-based style.

** Sub-styles
   :PROPERTIES:
   :CUSTOM_ID: sub-styles
   :END:
Nicolas suggested this, which would be a good way to handle some of the
natbib and biblatex variants.

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 11:20, Bruce D'Arcus wrote:
> I'm attaching the tentative style/command and shortcut mapping I came
> up with, with help from Eric and Joost.

Thank you for this.

I wonder whether it might be good to have a default LaTeX column as
well, i.e. not natbib, a column that represents what LaTeX supports out
of the box which is only the \cite{key} command.

Then there is the list of the default possible BiBTeX bibliography
styles (again, not natbib) itemised here:

   http://www.cs.stir.ac.uk/~kjt/software/latex/showbst.html


-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 11:46 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> I wonder whether it might be good to have a default LaTeX column as
> well, i.e. not natbib, a column that represents what LaTeX supports out
> of the box which is only the \cite{key} command.

Makes sense to me.

I guess the question at this stage is how to iterate this.

I already added a change or two based on feedback from Joost, and it
does make sense to add a new column for latex.

It seems easier to me to do it on the wiki, rather than post a new
version to the list every time, in part because it allows others to
edit it.

Any thoughts Nicolas?

Bruce

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Tue, May 4, 2021 at 11:46 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
>> I wonder whether it might be good to have a default LaTeX column as
>> well, i.e. not natbib, a column that represents what LaTeX supports out
>> of the box which is only the \cite{key} command.
>
> Makes sense to me.
>
> I guess the question at this stage is how to iterate this.
>
> I already added a change or two based on feedback from Joost, and it
> does make sense to add a new column for latex.
>
> It seems easier to me to do it on the wiki, rather than post a new
> version to the list every time, in part because it allows others to
> edit it.
>
> Any thoughts Nicolas?

Is the default \cite{key} command (without any other package) used? I'm
not sure we should provide it since we are working towards more complete
solutions.

I'd like to give a shot at "oc-natbib.el" in a few hours. It may give an
incentive to start out "oc-biblatex.el" (which is a bit harder).

AFAIU, the idea is to use styles from you wiki. However, sub-styles are
not mentioned. They could be "full" for the starred variants, "caps" for
the capitalized ones, and "alt" for those without parenthesis, so one
could write:

  [cite/text/alt/full/caps:...]  (in any order)

and obtain

  \Citealt*{...}

I hope we can find a shortcut, but it doesn't matter for now.

About sub-styles. The style is parsed as a string, so in the example
above, (org-element-property :style citation-object) returns
"text/alt/full/caps". 

Supporting sub-styles could be as simple as providing a function that
returns a non-nil value if a given sub-style belongs to the style. e.g.,

  (org-cite-has-substyle-p style "full") => t

  (org-cite-has-substyle-p style "author") => nil

I.e, sub-styles are not formally supported in the parser, but tooling
can take them into account.

Of course, we can also support them fully, i.e.
(org-element-property :style citation-object) returns a list of strings.
It depends how simple the common single style case should be.

To limit verbosity, we may want to stay (it's not mandatory) at one
sub-level instead, and use single characters for processor specific
variants, e.g.,

  [cite//aC*:...]  <-- default style with alt+full+caps

If style are in common use, processors might as well declare them in
`org-cite-register-processor' as a help for, e.g., functions inserting
citations. Hmm.

WDYT?

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 12:59 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
...
> > I guess the question at this stage is how to iterate this.
> >
> > I already added a change or two based on feedback from Joost, and it
> > does make sense to add a new column for latex.
> >
> > It seems easier to me to do it on the wiki, rather than post a new
> > version to the list every time, in part because it allows others to
> > edit it.
> >
> > Any thoughts Nicolas?
>
> Is the default \cite{key} command (without any other package) used? I'm
> not sure we should provide it since we are working towards more complete
> solutions.

Not ATM.

The table only has suggested mappings for natbib, biblatex, and
citeproc-el/org, though Eric was wondering about adding also one for
standard latex.

> I'd like to give a shot at "oc-natbib.el" in a few hours. It may give an
> incentive to start out "oc-biblatex.el" (which is a bit harder).
>
> AFAIU, the idea is to use styles from you wiki. However, sub-styles are
> not mentioned.

I just have the note they are a good idea, but wasn't sure on details.

> They could be "full" for the starred variants, "caps" for
> the capitalized ones, and "alt" for those without parenthesis, so one
> could write:
>
>   [cite/text/alt/full/caps:...]  (in any order)
>
> and obtain
>
>   \Citealt*{...}

I'll add them to the wiki page.

It did occur to me that ideally users could customize what the default
command is. My choices for natbib and biblatex defaults are I think
defensible, but some may disagree.

Bruce

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Tue, May 4, 2021 at 12:59 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> AFAIU, the idea is to use styles from you wiki. However, sub-styles are
> not mentioned. They could be "full" for the starred variants, "caps" for
> the capitalized ones, and "alt" for those without parenthesis, so one
> could write:

"Alt" and "full" I understand.

By "caps", you're referring to the biblatex variants?

Does this basically mean "force initial capitalization"?

Bruce

Re: Notes about citations in Org (part 3)

[András Simonyi]

Dear All,

this is just to indicate that I've started to look into the
wip-cite-new branch with an eye on updating citeproc-org to use the
new API -- everything seems to be working perfectly, thanks again for
this, Nicholas! One issue I also noticed is the separation of citation
style and bibliography style, which does not exist in CSL.
Accordingly, it would be nice to have ways of specifying a single
style for both of them (both in the Org syntax and in the Liisp API),
if this is not possible already.

best regards,
András

On Tue, 4 May 2021 at 20:25, Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
> On Tue, May 4, 2021 at 12:59 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> > AFAIU, the idea is to use styles from you wiki. However, sub-styles are
> > not mentioned. They could be "full" for the starred variants, "caps" for
> > the capitalized ones, and "alt" for those without parenthesis, so one
> > could write:
>
> "Alt" and "full" I understand.
>
> By "caps", you're referring to the biblatex variants?
>
> Does this basically mean "force initial capitalization"?
>
> Bruce
>

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Tue, May 4, 2021 at 12:59 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

>> Is the default \cite{key} command (without any other package) used? I'm
>> not sure we should provide it since we are working towards more complete
>> solutions.
>
> Not ATM.
>
> The table only has suggested mappings for natbib, biblatex, and
> citeproc-el/org, though Eric was wondering about adding also one for
> standard latex.

I meant: is it useful to even mention it? Is it used at all when other
more complete solutions exist?

>> AFAIU, the idea is to use styles from you wiki. However, sub-styles are
>> not mentioned.
>
> I just have the note they are a good idea, but wasn't sure on details.

OK. Let's go for the verbose path.

>> They could be "full" for the starred variants, "caps" for
>> the capitalized ones, and "alt" for those without parenthesis, so one
>> could write:
>>
>>   [cite/text/alt/full/caps:...]  (in any order)
>>
>> and obtain
>>
>>   \Citealt*{...}
>
> I'll add them to the wiki page.
>
> It did occur to me that ideally users could customize what the default
> command is. My choices for natbib and biblatex defaults are I think
> defensible, but some may disagree.

Customizing default command is the point of the third parameter in
"cite_export" keyword.

When using:

   #+cite_export: natbib harvard text

[cite:...] is equivalent to [cite/text: ...]

So the processor must choose a default value, but the user may choose
another one.

Regards,

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 18:59, Nicolas Goaziou wrote:
> Is the default \cite{key} command (without any other package) used? 

Yes, it is.  But I wouldn't be able to tell you anything about relative
frequency etc.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 22:22, Nicolas Goaziou wrote:
> I meant: is it useful to even mention it? Is it used at all when other
> more complete solutions exist?

Actually, to avoid misunderstanding, natbib is not more "complete" than
the default LaTeX support.  It simply supports a particular style and,
because of that style, it has more options that are necessary due to
what it exposes/uses for the actual citations.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Tuesday,  4 May 2021 at 21:04, András Simonyi wrote:
> One issue I also noticed is the separation of citation
> style and bibliography style, which does not exist in CSL.
> Accordingly, it would be nice to have ways of specifying a single
> style for both of them (both in the Org syntax and in the Liisp API),
> if this is not possible already.

And to add that the default LaTeX citation support also does not
separate citation style from bibliography style, both being defined by
the single \bibliographystyle{} command.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 362 bytes --]

On Wed, May 5, 2021, 1:48 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:

> On Tuesday,  4 May 2021 at 18:59, Nicolas Goaziou wrote:
> > Is the default \cite{key} command (without any other package) used?
>
> Yes, it is.  But I wouldn't be able to tell you anything about relative
> frequency etc.
>

Does it have any more commands than \cite and \nocite?

Bruce

>

[-- Attachment #2: Type: text/html, Size: 919 bytes --]

Re: Notes about citations in Org (part 3)

[Denis Maier]

Am 05.05.2021 um 11:37 schrieb Bruce D'Arcus:
> 
> 
> On Wed, May 5, 2021, 1:48 AM Eric S Fraga <e.fraga@ucl.ac.uk 
> <mailto:e.fraga@ucl.ac.uk>> wrote:
> 
>     On Tuesday,  4 May 2021 at 18:59, Nicolas Goaziou wrote:
>      > Is the default \cite{key} command (without any other package) used?
> 
>     Yes, it is.  But I wouldn't be able to tell you anything about relative
>     frequency etc.
> 
> 
> Does it have any more commands than \cite and \nocite?
> 
> Bruce
> 

I don't think so.

Denis

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Wednesday,  5 May 2021 at 05:37, Bruce D'Arcus wrote:
> Does it have any more commands than \cite and \nocite?

Vanilla LaTeX has only \cite and \nocite (page 74ff of the original book
by Leslie Lamport).

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-503-g501b2a

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Tue, May 4, 2021 at 12:59 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> AFAIU, the idea is to use styles from you wiki. However, sub-styles are
>> not mentioned. They could be "full" for the starred variants, "caps" for
>> the capitalized ones, and "alt" for those without parenthesis, so one
>> could write:
>
> "Alt" and "full" I understand.
>
> By "caps", you're referring to the biblatex variants?

I'm referring to \Citep vs. \citep natbib variants.
>
> Does this basically mean "force initial capitalization"?

Indeed.

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Nicolas Goaziou]

Hello,

András Simonyi <andras.simonyi@gmail.com> writes:

> this is just to indicate that I've started to look into the
> wip-cite-new branch with an eye on updating citeproc-org to use the
> new API -- everything seems to be working perfectly, thanks again for
> this, Nicholas! One issue I also noticed is the separation of citation
> style and bibliography style, which does not exist in CSL.
> Accordingly, it would be nice to have ways of specifying a single
> style for both of them (both in the Org syntax and in the Liisp API),
> if this is not possible already.

I don't think this is necessary. 

My (is it mine actually? I can't tell with all the discussions going on)
choice of "style" name for citations and bibliography may be
unfortunate. Actually citations/bibliography styles do not have to match
a known style in CSL. More precisely, the bibliography style is probably
close to what is expected by CSL, but I see citations styles more as
internal variants.

For example, one processor could implement an "draft" citation style
where the citation is in the document, but it isn't even processed
during export. So I think the single style you're talking about is
really the bibliography style.

Does that make sense?

Regards,
-- 
Nicolas Goaziou

Re: Notes about citations in Org (part 3)

[Maxim Nikulin]

On 04/05/2021 23:59, Nicolas Goaziou wrote:
> 
> Is the default \cite{key} command (without any other package) used? I'm
> not sure we should provide it since we are working towards more complete
> solutions.

In some fields simple "[3,7]" citations are traditional. A couple of 
randomly picked examples:
https://www.osapublishing.org/oe/fulltext.cfm?uri=oe-29-10-15413&id=450715
https://arxiv.org/abs/2105.01093

Personally, I do not mind to have a hint (e.g. a tooltip) if format 
allows such features, e.g.
[<span title="13. D. Upper. The unsuccessful self‐treatment of a case of 
“writer's block”. JABA 1974 v.7 p.497">13</span>]

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Wed, May 5, 2021 at 12:59 PM Maxim Nikulin <manikulin@gmail.com> wrote:
>
> On 04/05/2021 23:59, Nicolas Goaziou wrote:
> >
> > Is the default \cite{key} command (without any other package) used? I'm
> > not sure we should provide it since we are working towards more complete
> > solutions.
>
> In some fields simple "[3,7]" citations are traditional.

How do you achieve such output, with which formats, packages, commands?

For example, do you get that with the default \cite command in latex,
assuming the right bst file?

Something else?

Bruce

Re: Notes about citations in Org (part 3)

[Eric S Fraga]

On Wednesday,  5 May 2021 at 13:16, Bruce D'Arcus wrote:
> On Wed, May 5, 2021 at 12:59 PM Maxim Nikulin <manikulin@gmail.com> wrote:
>> In some fields simple "[3,7]" citations are traditional.
>
> How do you achieve such output, with which formats, packages, commands?
>
> For example, do you get that with the default \cite command in latex,
> assuming the right bst file?

yes, see, for instance, first example at:

https://www.overleaf.com/learn/latex/Bibtex_bibliography_styles

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.5-395-g82fbdd

Re: Notes about citations in Org (part 3)

[Maxim Nikulin]

On 06/05/2021 00:16, Bruce D'Arcus wrote:
> On Wed, May 5, 2021 at 12:59 PM Maxim Nikulin wrote:
>> On 04/05/2021 23:59, Nicolas Goaziou wrote:
>>>
>>> Is the default \cite{key} command (without any other package) used? I'm
>>> not sure we should provide it since we are working towards more complete
>>> solutions.
>>
>> In some fields simple "[3,7]" citations are traditional.
> 
> How do you achieve such output, with which formats, packages, commands?

It is default behavior. Minimal example:

--- 8< ---
\documentclass{article}
\begin{document}
New threads related to citations in Org~\cite{ml_natbib,ml_cite_org3}.

\begin{thebibliography}{9}
\bibitem{ml_cite_org3} Nicolas Goaziou. Notes about citations in Org 
(part 3).
\bibitem{ml_cite_status} Bruce D'Arcus. wip-cite status question and 
feedback.
\bibitem{ml_natbib} Nicolas Goaziou. [wip-cite-new] New natbib processor.
\end{thebibliography}
\end{document}
--- 8< ---
New threads related to citations in Org [3, 1].

References
[1] Nicolas Goaziou. Notes about citations in Org (part 3).
[2] Bruce D’Arcus. wip-cite status question and feedback.
[3] Nicolas Goaziou. [wip-cite-new] New natbib processor.
--- 8< ---

Additional packages are required to sort citations or to replace 
sequence with range.

> For example, do you get that with the default \cite command in latex,
> assuming the right bst file?

Do you think something is wrong with such citations? Papers may be 
published without using of BibTeX at all. "Right" bst file depends on 
particular journal. Even between styles, that generate simple 
\bibitem{key} causing numerical citations, difference may be rather 
significant in respect to what parts of entry are formatted, in which 
order how they are emphasized, what is omitted completely and what is 
shortened.

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Thu, May 6, 2021 at 7:37 AM Maxim Nikulin <manikulin@gmail.com> wrote:

> > For example, do you get that with the default \cite command in latex,
> > assuming the right bst file?
>
> Do you think something is wrong with such citations?

No.

I asked those questions because the focus now is on org-cite styles,
and sub-styles, and how those map to which output formats.

Your example should just use the default org-cite citation, without
any style or sub-style.

The rest would be handled by latex/bibtex.

Right?

Bruce

Re: Notes about citations in Org (part 3)

[Maxim Nikulin]

On 06/05/2021 18:53, Bruce D'Arcus wrote:
> 
> Your example should just use the default org-cite citation, without
> any style or sub-style.
> 
> The rest would be handled by latex/bibtex.
> 
> Right?

Yes, simple \cite{k1,k2} is mostly enough for numeric citations. E.g 
cases like separated list of references to author's works are handled by 
a customized \begin{thebibliography} and dedicated counter having some 
prefix.

There is might be some uncertainty concerning handling of prefixes and 
suffixes. However my impression is that even e.g. \cite[p.~7]{key} is 
quite rare. For completeness, keys without prefixes, suffixes could be 
combined into single \cite{key1,key2}, otherwise individual \cite{key1} 
may be required with additional text in between.

Re: Notes about citations in Org (part 3)

[Bruce D'Arcus]

On Thu, May 6, 2021 at 12:05 PM Maxim Nikulin <manikulin@gmail.com> wrote:

> There is might be some uncertainty concerning handling of prefixes and
> suffixes. However my impression is that even e.g. \cite[p.~7]{key} is
> quite rare. For completeness, keys without prefixes, suffixes could be
> combined into single \cite{key1,key2}, otherwise individual \cite{key1}
> may be required with additional text in between.

You should be covered.

[cite:prefix 1 @key1 suffix 1;prefix 2 @key2 suffix2]

Bruce