(Not so) Short note about citations in Org

(Not so) Short note about citations in Org

[Nicolas Goaziou]

Hello,

I just rebased "wip-cite-new" branch, which now includes citation
syntax, and an interface between external citation processors and the
rest of Org. I'll throw in a demo at the end of this message. TL;DR:
search for "Demo time".

As a reminder, the full citation syntax is

  [cite/style:common prefix ;prefix -@key suffix; ... ; common suffix]

Everything is optional, except the brackets, "cite" and the colon. Also
the citation must contain at least a key. So its minimal form is:

  [cite:@key]

or its "suppress author" variant:

  [cite:-@key]

A noteworthy difference with the last merge is that I removed the
opportunity to add some Org syntax (emphasis, sub/superscript…) in
prefixes or suffixes. It makes the code a bit simpler. Of course, I'm
open to discussion about this change.

The syntax also introduces three keywords (square brackets are for
optional arguments):

   #+bibliography: filename
   #+print_bibliography: [style]
   #+cite_export: backend-name [bibliography-style] [citation-style]

Now, the real novelty is the new "oc.el" library, which is an API to
operate simply on citations. I paste here its commentary section.


    This library provides tooling to handle citations in Org, e.g,
    follow, fontify, and export them, respectively called "follow",
    "activate" and "export" capabilities. Libraries responsible for
    providing some, or all, of these capabilities are called "citation
    processors".
    
    Such processors are defined using `org-cite-register-processor'.
    Using this function, it is possible, in addition to giving it
    a name, to attach functions associated to capabilities.  As such,
    a processor handling citation export must set the
    `:export-citation' property to an appropriate function.  Likewise,
    "activate" capability require an appropriate `:activate' property,
    and, unsurprisingly, "follow" capability implies `:follow'
    property.
    
    As a user, the first thing to do is setting a bibliography, either
    globally with `org-cite-global-bibliography', or locally using one
    ore more "bibliography" keywords.  Then one can select any
    registered processor for each capability by providing a processor
    name to the variables `org-cite-activate-processor' and
    `org-cite-follow-processor'.
    
    The "export" capability is slightly more involved as one need to
    select the processor providing it, but may also provide a default
    style for citations and bibliography.  These three parameters, or
    triplet, can be set in `org-cite-export-processor' variable, or in
    a document, through the "cite_export" keyword.
    
    Eventually, this library provides some tools, mainly targeted at
    processor implementors.  Most are export-specific and are located
    in the "Tools only available during export" section.  The few
    others can be used directly from an Org buffer, or operate on
    processors.  See "Generic tools" section.


There are two points of view to consider here. As a user, as stated
above, you first need to provide a bibliography, for all documents using
the `org-cite-global-bibliography' variable, or for a single document
(or a set of documents, using "setupfile" keyword) with

    #+bibliography: one-file.bib
    #+bibliography: another-file.ext
    #+bibliography: "some file with spaces"

You can use both the variable and the keywords, in which case files are
_accumulated_ in the list.

Then when you 

  (require 'org-cite-something)

the "oc-something.el" library, in addition to possibly many other tools,
registers a "citation processor", for example `something'. That
processor may operate on any of the three entry points "activate",
"follow", or "export". If you are not sure about which one it supports,
you may inspect the processor with, e.g.,

  (org-cite-processor-has-capability-p 'something 'follow)

If this is non-nil, you can set `org-cite-follow-processor' to
`something'. Once done, calling `org-open-at-point' on a citation starts
whatever action the processor defined. If the variable is nil, nothing
happens.

If you need to use a different processor for a given document, consider
using file local variables.

Likewise, you can fontify citations according to a given processor using
`org-cite-activate-processor'. This time, however, Org provides some
fontification even when the variable is nil. The default set-up merely
applies new `org-cite' and `org-cite-key' faces on citations.

The "export" capability introduces the concept of "style", which is an
_indication_ to the citation processor, which may or may consider
applying. More precisely, a style can be set for citations and
bibliography, at three levels from the most general to the most
specific:

  (setq org-cite-export-processor '(something "bibstyle" "citestyle"))

  #+cite_export: something bibstyle citestyle

  #+print_bibliography: bibstyle
  [cite/citestyle:...]

An export processor is required to support at least one default style
for citations and bibliography called the "nil" style. It may support
any number of other styles, and should treat any unknown style
indication as the "nil" style. So

  [cite/totallyunknownstyle:...]

may be treated as

  [cite/nil: ...]

which, in turn, is strictly equivalent to

  [cite: ...]

Now onto the developer point of view. A citation back-end can provide
many tools, but in order to interact with Org through the three entry
points listed earlier, it also needs to define a so-called processor,
using `org-cite-register-processor' function. For reference, here is its
docstring.


      Mark citation processor NAME as available.

    NAME is a symbol.  BODY is a property list, where the following optional keys
    can be set:

    `:activate'

      Function activating a citation.  It is called with a single argument: a
      citation object extracted from the current buffer.  It may add text
      properties to the buffer.  If it is not provided, `org-cite-fontify-default'
      is used.

    `:export-bibliography'

      Function rendering a bibliography.  It is called with five arguments: a list
      of citations, a list of bibliography files, the style, as a string or nil,
      the export back-end, as a symbol, and the communication channel, as a
      property list.

      It is called at each \"print_bibliography\" keyword in the parse tree.
      It may return a string or nil.  When it returns nil, the keyword is ignored.
      Otherwise, the string it returns replaces the keyword in the export output.

    `:export-citation'    (mandatory for \"export\" capability)

      Function rendering citations.  It is called with four arguments: a citation
      object, the style, as a string or nil, the export back-end, as a symbol,
      and the communication channel, as a property list.

      It is called on each citation object in the parse tree.  It may return
      a string or nil.  When it returns nil, the citation is ignored.  Otherwise,
      the string it returns replaces the citation object in the export output.

    `:export-finalizer'

      Function called at the end of export process.  It must accept five
      arguments: the output, as a string, a list of citations, a list of
      bibliography files, a list of bibliography styles requested by various
      \"print_bibliography\" keywords in the document, as strings or nil, and the
      export back-end, as a symbol.

      It must return a string, which will become the final output from the export
      process, barring subsequent modifications from export filters.

    `:follow'

      Function called to follow a citation.  It accepts two arguments, the
      citation or citation reference object at point, and any prefix argument
      received during interactive call of `org-open-at-point'.


The "follow" and "activate" capabilities are relatively simple to
implement because both require a single function to operate. On the
other hand, "export" capability may require up to three functions. Of
those, only the `export-citation' function cannot be omitted.

The "oc.el" library provides several tools to help developers writing
those functions. For example, `org-cite-list-bibliography-files' may be
useful when providing "follow" or "activate" capability. 

Likewise, `org-cite-get-references' lists all citation-reference objects
contained in a citation. For each object, you can extract boundaries as
buffer positions, the key, the prefix and the suffix. This can be very
useful for activating a citation reference, e.g., when apply some
specific key-map on a part of the buffer.

Most important helper functions available for export are
`org-cite-list-citations', which provides the _ordered_ list of
citations in the exported document, and `org-cite-inside-footnote-p',
which returns the closest (inline) footnote reference or footnote
definition containing a citation, if any. This function can be paired,
e.g., with `org-export-get-footnote-number'.

=== Demo time ===

Let's say I want to implement "oc-demo.el", which will provide
"activate" and "export" capabilities.

For activation, I want:
- the same default fontification (`org-cite' and `org-cite-key' faces),
- to display the cite key when mouse is over a citation reference,
- hide the unsightly "cite:" part of the citation.

I therefore write the following function:

  (defun org-cite-demo-activate (citation)
    ;; Apply default fontification, the lazy way.
    (org-cite-fontify-default citation)
    ;; Hide "cite:" prefix.
    (let* ((prefix-start (1+ (org-element-property :begin citation)))
           (prefix-end (+ 5 prefix-start)))
      (add-text-properties prefix-start prefix-end '(invisible t))
      (org-rear-nonsticky-at prefix-start)
      (org-rear-nonsticky-at prefix-end))
    ;; Apply `help-echo' on each key reference.
    (org-with-point-at (org-element-property :contents-begin citation)
      (let ((end (org-element-property :contents-end citation)))
        (while (re-search-forward org-element-citation-key-re end t)
          (add-text-properties
           (match-beginning 0) (match-end 0)
           ;; Drop the @ symbol.
           (list 'help-echo
                 (substring (match-string-no-properties 0) 1)))))))

I want to export citations as (Doe, 1999) or (1999) when suppress author
parameter is set. I also want to support the "plain" style, which
removes the parenthesis.

The bibliography should be inserted at the very end of the document,
listing all bibliography files used, but only if there is some citation
and some "print_bibliography" keyword in the document.

I won't use :export-bibliography, because it is called at each
"print_bibliography" keyword, so possibly in the middle of the document.
I'll use a finalizer instead.

  (defun org-cite-demo-export-citation (citation style back-end info)
    (concat
     ;; Global prefix, if any.  We use `org-export-data' as prefixes or
     ;; suffixes may contain special characters that need to be escaped
     ;; by the export process.
     (org-export-data (org-element-property :prefix citation) info)
     (and (not (equal style "plain")) "(")
     (mapconcat (lambda (ref)
                  (concat
                   (org-export-data (org-element-property :prefix ref) info)
                   (if (org-element-property :suppress-author ref)
                       "1999"
                     "Doe, 1999")
                   (org-export-data (org-element-property :suffix ref) info)))
                ;; Grab all references within the citation.
                (org-cite-get-references citation)
                ", ")
     (and (not (equal style "plain")) ")")
     ;; Global suffix, if any.
     (org-export-data (org-element-property :suffix citation) info)))

  (defun org-cite-demo-finalizer (output citations bibliography styles backend)
    (concat output
            ;; Print bibliography only if there are some citations and
            ;; a "print_bibliography" keyword in the document.
            (and citations
                 styles                ;proof of "print_bibliography"
                 (concat "\nBibliography: "
                         (mapconcat #'identity bibliography ", ")
                         "."))))

All is left to do is to register the "demo" processor.

  (org-cite-register-processor 'demo
    :activate #'org-cite-demo-activate
    :export-citation #'org-cite-demo-export-citation
    :export-finalizer #'org-cite-demo-finalizer)

That's about it.

You can, if you wish so, try out this demo on the following document:

--8<---------------cut here---------------start------------->8---
#+cite_export: demo

#+bibliography: bib.bib
#+bibliography: bib2.bib

Simple reference: [cite:@key]

Multiple references: [cite:@key1;@key2]

Suppress author: [cite:-@key]

Plain style: [cite/plain:@key]

Full syntax: [cite:common prefix ;prefix @key suffix ;@key2; common suffix]

#+print_bibliography:

# Local Variables:
# org-cite-activate-processor: demo
# End:
--8<---------------cut here---------------end--------------->8---

WDYT?

Regards,
-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Wed, Apr 21, 2021 at 7:34 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> I just rebased "wip-cite-new" branch, which now includes citation
> syntax, and an interface between external citation processors and the
> rest of Org. I'll throw in a demo at the end of this message. TL;DR:
> search for "Demo time".

Great!

I just pulled down the branch.

This is my first time trying to run off a development branch.

I ran make from the root of the repo, and then:

emacs -Q -l lisp/org.elc

... but then get this error:

load-with-code-conversion: Cannot open load file: No such file or directory, oc

Is there something else I should do?

Bruce

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Great!
>
> I just pulled down the branch.
>
> This is my first time trying to run off a development branch.
>
> I ran make from the root of the repo, and then:
>
> emacs -Q -l lisp/org.elc
>
> ... but then get this error:
>
> load-with-code-conversion: Cannot open load file: No such file or
> directory, oc

Not so great, then!

> Is there something else I should do?

Do you have "oc.el" file in "lisp/" directory? You may want to augment
your load-path at an early step.

You can also go low-tech and call `eval-buffer' on "org-element.el",
"oc.el", "org-element.el", "ox.el" and "org.el".

This should be enough for testing.

Regards,
-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Wed, Apr 21, 2021 at 8:35 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Great!
> >
> > I just pulled down the branch.
> >
> > This is my first time trying to run off a development branch.
> >
> > I ran make from the root of the repo, and then:
> >
> > emacs -Q -l lisp/org.elc
> >
> > ... but then get this error:
> >
> > load-with-code-conversion: Cannot open load file: No such file or
> > directory, oc
>
> Not so great, then!
>
> > Is there something else I should do?
>
> Do you have "oc.el" file in "lisp/" directory? You may want to augment
> your load-path at an early step.

Yeah, but it doesn't load.

I added =(add-to-list 'load-path "lisp/")= to a load file, and that
also didn't work for me.

> You can also go low-tech and call `eval-buffer' on "org-element.el",
> "oc.el", "org-element.el", "ox.el" and "org.el".

This, however, did.

I only tested export to HTML, but so far looks encouraging!

Bruce

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Wed, Apr 21, 2021, 7:34 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

>     `:export-bibliography'
>
>       Function rendering a bibliography.  It is called with five arguments: a list
>       of citations, a list of bibliography files, the style, as a string or nil,
>       the export back-end, as a symbol, and the communication channel, as a
>       property list.
>
>       It is called at each \"print_bibliography\" keyword in the parse tree.
>       It may return a string or nil.  When it returns nil, the keyword is ignored.
>       Otherwise, the string it returns replaces the keyword in the export output.
>
>     `:export-citation'    (mandatory for \"export\" capability)
>
>       Function rendering citations.  It is called with four arguments: a citation
>       object, the style, as a string or nil, the export back-end, as a symbol,
>       and the communication channel, as a property list.
>
>       It is called on each citation object in the parse tree.  It may return
>       a string or nil.  When it returns nil, the citation is ignored.  Otherwise,
>       the string it returns replaces the citation object in the export output.
>
>     `:export-finalizer'
>
>       Function called at the end of export process.  It must accept five
>       arguments: the output, as a string, a list of citations, a list of
>       bibliography files, a list of bibliography styles requested by various
>       \"print_bibliography\" keywords in the document, as strings or nil, and the
>       export back-end, as a symbol.

The haskell citeproc library has this:

"a CiteprocOptions structure (which currently just allows you to set
whether citations are hyperlinked to the bibliography)"

Would something like that make sense as an optional argument somewhere
here? Or would details like that get
set somewhere else?

BTW, on the demo, the "help echo" on my machine (a very recent Emacs
28) only flickers briefly, so I can't actually see it.

But I like how this is all looking; curious what others say.

Bruce

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> The haskell citeproc library has this:
>
> "a CiteprocOptions structure (which currently just allows you to set
> whether citations are hyperlinked to the bibliography)"
>
> Would something like that make sense as an optional argument somewhere
> here? Or would details like that get
> set somewhere else?

I am not sure to understand what it is meant to do. Anyway, I think such
options belong to the processor, an hypothetical "oc-citeproc.el" in
this case. "oc.el" does not pretend controlling what processors do with
their "capabilities"; it just makes it easy to plug citation processors
into Org. If some options are common to all processors, then it may be
reconsidered, tho.

Now, I hope to see some work towards "oc-citeproc.el" (or any other
processor). We need it to push the limits of "oc.el", e.g., to decide if
it should provide more tools, or more breadth in its API.

As stated already, I will be happy to provide guidance, and even code
for well defined tasks.

> BTW, on the demo, the "help echo" on my machine (a very recent Emacs
> 28) only flickers briefly, so I can't actually see it.

It works here when I enable `tooltip-mode'.

Regards,
-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Thu, Apr 22, 2021 at 9:14 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Now, I hope to see some work towards "oc-citeproc.el" (or any other
> processor). We need it to push the limits of "oc.el", e.g., to decide if
> it should provide more tools, or more breadth in its API.
>
> As stated already, I will be happy to provide guidance, and even code
for well defined tasks.

Sounds good!

Seems like the test would really be adapting citeproc-org/el to this,
so let's see what Andras thinks!

Bruce

Re: (Not so) Short note about citations in Org

[M. ‘quintus’ Gülker]

Hi,

Am 22. April 2021 um 01:34 Uhr +0200 schrieb Nicolas Goaziou:
> A noteworthy difference with the last merge is that I removed the
> opportunity to add some Org syntax (emphasis, sub/superscript…) in
> prefixes or suffixes. It makes the code a bit simpler. Of course, I'm
> open to discussion about this change.

I sometimes use emphasis around citations, but since my citation
styles all use footnotes, I think there should be no problem with
this. For example, I would write something like this (I hope I got the
syntax right):

    This is a test sentence[fn:1].

    * Footnotes

    [fn:1] See e.g. [cite:@doe1993 pp. 32-35], who explicitely argues
    /against/ the suggestion made by [cite:@someone1990 p. 50].

Since the emphasis markers on the word "against" are outside the two
citations, I would expect no problem here.

> WDYT?

Thank you so much for the hard work on this feature.

  -quintus

-- 
Dipl.-Jur. M. Gülker | https://mg.guelker.eu |    For security:
Passau, Germany      | kontakt@guelker.eu    | () Avoid HTML e-mail
European Union       | PGP: see homepage     | /\ http://asciiribbon.org

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

FYI, I started to implement a function to format the org-cite syntax here:

https://github.com/bdarcus/bibtex-actions/pull/113

When finished, and org-cite merged, I plan to submit it for inclusion
in bibtex-completion.

Bruce

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Hello,

M. ‘quintus’ Gülker <post+orgmodeml@guelker.eu> writes:

> I sometimes use emphasis around citations, but since my citation
> styles all use footnotes, I think there should be no problem with
> this. For example, I would write something like this (I hope I got the
> syntax right):
>
>     This is a test sentence[fn:1].
>
>     * Footnotes
>
>     [fn:1] See e.g. [cite:@doe1993 pp. 32-35], who explicitely argues
>     /against/ the suggestion made by [cite:@someone1990 p. 50].
>
> Since the emphasis markers on the word "against" are outside the two
> citations, I would expect no problem here.

Indeed, there is no problem above. The discussion is about emphasis
within prefix or suffix.

Regards,
-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

Nicolas,

Quick syntax question:

On Wed, Apr 21, 2021 at 7:34 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> As a reminder, the full citation syntax is
>
>   [cite/style:common prefix ;prefix -@key suffix; ... ; common suffix]

Is the space you have here before the semi-colon significant as part
of the delimiter, or is it part of the prefix?

prefix ;prefix

So if I enter common prefix text as "prefix" and common suffix as
"suffix", is this correct output?

 ELISP> (bibtex-actions-format-citation-org (list "doe1" "doe2"))
"[cite:prefix ;@doe1;@doe2; suffix]"

... or should it be:

"[cite:prefix;@doe1;@doe2;suffix]"

Bruce

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Nicolas,
>
> Quick syntax question:
>
> On Wed, Apr 21, 2021 at 7:34 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> As a reminder, the full citation syntax is
>>
>>   [cite/style:common prefix ;prefix -@key suffix; ... ; common suffix]
>
> Is the space you have here before the semi-colon significant as part
> of the delimiter, or is it part of the prefix?

The delimiter is the semi-colon. Space is part of the prefix.
>
> prefix ;prefix
>
> So if I enter common prefix text as "prefix" and common suffix as
> "suffix", is this correct output?
>
>  ELISP> (bibtex-actions-format-citation-org (list "doe1" "doe2"))
> "[cite:prefix ;@doe1;@doe2; suffix]"
>
> ... or should it be:
>
> "[cite:prefix;@doe1;@doe2;suffix]"

The latter if you don't want to see space in prefix or suffix, i.e., if
this is "prefix" and "suffix".

Regards,
-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Sat, Apr 24, 2021 at 3:53 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > Nicolas,
> >
> > Quick syntax question:
> >
> > On Wed, Apr 21, 2021 at 7:34 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> >
> >> As a reminder, the full citation syntax is
> >>
> >>   [cite/style:common prefix ;prefix -@key suffix; ... ; common suffix]
> >
> > Is the space you have here before the semi-colon significant as part
> > of the delimiter, or is it part of the prefix?
>
> The delimiter is the semi-colon. Space is part of the prefix.

Got it; thanks!

Now implemented, which some docstring examples borrowed from your
explanation, and adding a few to clarify the different prefix/suffix
levels, that could be valuable for org documentation?

https://github.com/bdarcus/bibtex-actions/pull/113/commits/d7869c409751f065197a689579bd5e13ff1ed2ff

Bruce

Re: (Not so) Short note about citations in Org

[Denis Maier]

Hi everyone,

great to see that there's been so much progress since this issue has 
last been discussed! Looks really impressive so far.

Two comments/questions:

- I think it should be possible to use some markup in prefixes and suffixes.
- Will this proposal allow multiple bibliographies? I think that's been 
discussed last year or so.

Best,
Denis

Am 22.04.2021 um 01:34 schrieb Nicolas Goaziou:
> Hello,
> 
> I just rebased "wip-cite-new" branch, which now includes citation
> syntax, and an interface between external citation processors and the
> rest of Org. I'll throw in a demo at the end of this message. TL;DR:
> search for "Demo time".
> 
> As a reminder, the full citation syntax is
> 
>    [cite/style:common prefix ;prefix -@key suffix; ... ; common suffix]
> 
> Everything is optional, except the brackets, "cite" and the colon. Also
> the citation must contain at least a key. So its minimal form is:
> 
>    [cite:@key]
> 
> or its "suppress author" variant:
> 
>    [cite:-@key]
> 
> A noteworthy difference with the last merge is that I removed the
> opportunity to add some Org syntax (emphasis, sub/superscript…) in
> prefixes or suffixes. It makes the code a bit simpler. Of course, I'm
> open to discussion about this change.
> 
> The syntax also introduces three keywords (square brackets are for
> optional arguments):
> 
>     #+bibliography: filename
>     #+print_bibliography: [style]
>     #+cite_export: backend-name [bibliography-style] [citation-style]
> 
> Now, the real novelty is the new "oc.el" library, which is an API to
> operate simply on citations. I paste here its commentary section.
> 
> 
>      This library provides tooling to handle citations in Org, e.g,
>      follow, fontify, and export them, respectively called "follow",
>      "activate" and "export" capabilities. Libraries responsible for
>      providing some, or all, of these capabilities are called "citation
>      processors".
>      
>      Such processors are defined using `org-cite-register-processor'.
>      Using this function, it is possible, in addition to giving it
>      a name, to attach functions associated to capabilities.  As such,
>      a processor handling citation export must set the
>      `:export-citation' property to an appropriate function.  Likewise,
>      "activate" capability require an appropriate `:activate' property,
>      and, unsurprisingly, "follow" capability implies `:follow'
>      property.
>      
>      As a user, the first thing to do is setting a bibliography, either
>      globally with `org-cite-global-bibliography', or locally using one
>      ore more "bibliography" keywords.  Then one can select any
>      registered processor for each capability by providing a processor
>      name to the variables `org-cite-activate-processor' and
>      `org-cite-follow-processor'.
>      
>      The "export" capability is slightly more involved as one need to
>      select the processor providing it, but may also provide a default
>      style for citations and bibliography.  These three parameters, or
>      triplet, can be set in `org-cite-export-processor' variable, or in
>      a document, through the "cite_export" keyword.
>      
>      Eventually, this library provides some tools, mainly targeted at
>      processor implementors.  Most are export-specific and are located
>      in the "Tools only available during export" section.  The few
>      others can be used directly from an Org buffer, or operate on
>      processors.  See "Generic tools" section.
> 
> 
> There are two points of view to consider here. As a user, as stated
> above, you first need to provide a bibliography, for all documents using
> the `org-cite-global-bibliography' variable, or for a single document
> (or a set of documents, using "setupfile" keyword) with
> 
>      #+bibliography: one-file.bib
>      #+bibliography: another-file.ext
>      #+bibliography: "some file with spaces"
> 
> You can use both the variable and the keywords, in which case files are
> _accumulated_ in the list.
> 
> Then when you
> 
>    (require 'org-cite-something)
> 
> the "oc-something.el" library, in addition to possibly many other tools,
> registers a "citation processor", for example `something'. That
> processor may operate on any of the three entry points "activate",
> "follow", or "export". If you are not sure about which one it supports,
> you may inspect the processor with, e.g.,
> 
>    (org-cite-processor-has-capability-p 'something 'follow)
> 
> If this is non-nil, you can set `org-cite-follow-processor' to
> `something'. Once done, calling `org-open-at-point' on a citation starts
> whatever action the processor defined. If the variable is nil, nothing
> happens.
> 
> If you need to use a different processor for a given document, consider
> using file local variables.
> 
> Likewise, you can fontify citations according to a given processor using
> `org-cite-activate-processor'. This time, however, Org provides some
> fontification even when the variable is nil. The default set-up merely
> applies new `org-cite' and `org-cite-key' faces on citations.
> 
> The "export" capability introduces the concept of "style", which is an
> _indication_ to the citation processor, which may or may consider
> applying. More precisely, a style can be set for citations and
> bibliography, at three levels from the most general to the most
> specific:
> 
>    (setq org-cite-export-processor '(something "bibstyle" "citestyle"))
> 
>    #+cite_export: something bibstyle citestyle
> 
>    #+print_bibliography: bibstyle
>    [cite/citestyle:...]
> 
> An export processor is required to support at least one default style
> for citations and bibliography called the "nil" style. It may support
> any number of other styles, and should treat any unknown style
> indication as the "nil" style. So
> 
>    [cite/totallyunknownstyle:...]
> 
> may be treated as
> 
>    [cite/nil: ...]
> 
> which, in turn, is strictly equivalent to
> 
>    [cite: ...]
> 
> Now onto the developer point of view. A citation back-end can provide
> many tools, but in order to interact with Org through the three entry
> points listed earlier, it also needs to define a so-called processor,
> using `org-cite-register-processor' function. For reference, here is its
> docstring.
> 
> 
>        Mark citation processor NAME as available.
> 
>      NAME is a symbol.  BODY is a property list, where the following optional keys
>      can be set:
> 
>      `:activate'
> 
>        Function activating a citation.  It is called with a single argument: a
>        citation object extracted from the current buffer.  It may add text
>        properties to the buffer.  If it is not provided, `org-cite-fontify-default'
>        is used.
> 
>      `:export-bibliography'
> 
>        Function rendering a bibliography.  It is called with five arguments: a list
>        of citations, a list of bibliography files, the style, as a string or nil,
>        the export back-end, as a symbol, and the communication channel, as a
>        property list.
> 
>        It is called at each \"print_bibliography\" keyword in the parse tree.
>        It may return a string or nil.  When it returns nil, the keyword is ignored.
>        Otherwise, the string it returns replaces the keyword in the export output.
> 
>      `:export-citation'    (mandatory for \"export\" capability)
> 
>        Function rendering citations.  It is called with four arguments: a citation
>        object, the style, as a string or nil, the export back-end, as a symbol,
>        and the communication channel, as a property list.
> 
>        It is called on each citation object in the parse tree.  It may return
>        a string or nil.  When it returns nil, the citation is ignored.  Otherwise,
>        the string it returns replaces the citation object in the export output.
> 
>      `:export-finalizer'
> 
>        Function called at the end of export process.  It must accept five
>        arguments: the output, as a string, a list of citations, a list of
>        bibliography files, a list of bibliography styles requested by various
>        \"print_bibliography\" keywords in the document, as strings or nil, and the
>        export back-end, as a symbol.
> 
>        It must return a string, which will become the final output from the export
>        process, barring subsequent modifications from export filters.
> 
>      `:follow'
> 
>        Function called to follow a citation.  It accepts two arguments, the
>        citation or citation reference object at point, and any prefix argument
>        received during interactive call of `org-open-at-point'.
> 
> 
> The "follow" and "activate" capabilities are relatively simple to
> implement because both require a single function to operate. On the
> other hand, "export" capability may require up to three functions. Of
> those, only the `export-citation' function cannot be omitted.
> 
> The "oc.el" library provides several tools to help developers writing
> those functions. For example, `org-cite-list-bibliography-files' may be
> useful when providing "follow" or "activate" capability.
> 
> Likewise, `org-cite-get-references' lists all citation-reference objects
> contained in a citation. For each object, you can extract boundaries as
> buffer positions, the key, the prefix and the suffix. This can be very
> useful for activating a citation reference, e.g., when apply some
> specific key-map on a part of the buffer.
> 
> Most important helper functions available for export are
> `org-cite-list-citations', which provides the _ordered_ list of
> citations in the exported document, and `org-cite-inside-footnote-p',
> which returns the closest (inline) footnote reference or footnote
> definition containing a citation, if any. This function can be paired,
> e.g., with `org-export-get-footnote-number'.
> 
> === Demo time ===
> 
> Let's say I want to implement "oc-demo.el", which will provide
> "activate" and "export" capabilities.
> 
> For activation, I want:
> - the same default fontification (`org-cite' and `org-cite-key' faces),
> - to display the cite key when mouse is over a citation reference,
> - hide the unsightly "cite:" part of the citation.
> 
> I therefore write the following function:
> 
>    (defun org-cite-demo-activate (citation)
>      ;; Apply default fontification, the lazy way.
>      (org-cite-fontify-default citation)
>      ;; Hide "cite:" prefix.
>      (let* ((prefix-start (1+ (org-element-property :begin citation)))
>             (prefix-end (+ 5 prefix-start)))
>        (add-text-properties prefix-start prefix-end '(invisible t))
>        (org-rear-nonsticky-at prefix-start)
>        (org-rear-nonsticky-at prefix-end))
>      ;; Apply `help-echo' on each key reference.
>      (org-with-point-at (org-element-property :contents-begin citation)
>        (let ((end (org-element-property :contents-end citation)))
>          (while (re-search-forward org-element-citation-key-re end t)
>            (add-text-properties
>             (match-beginning 0) (match-end 0)
>             ;; Drop the @ symbol.
>             (list 'help-echo
>                   (substring (match-string-no-properties 0) 1)))))))
> 
> I want to export citations as (Doe, 1999) or (1999) when suppress author
> parameter is set. I also want to support the "plain" style, which
> removes the parenthesis.
> 
> The bibliography should be inserted at the very end of the document,
> listing all bibliography files used, but only if there is some citation
> and some "print_bibliography" keyword in the document.
> 
> I won't use :export-bibliography, because it is called at each
> "print_bibliography" keyword, so possibly in the middle of the document.
> I'll use a finalizer instead.
> 
>    (defun org-cite-demo-export-citation (citation style back-end info)
>      (concat
>       ;; Global prefix, if any.  We use `org-export-data' as prefixes or
>       ;; suffixes may contain special characters that need to be escaped
>       ;; by the export process.
>       (org-export-data (org-element-property :prefix citation) info)
>       (and (not (equal style "plain")) "(")
>       (mapconcat (lambda (ref)
>                    (concat
>                     (org-export-data (org-element-property :prefix ref) info)
>                     (if (org-element-property :suppress-author ref)
>                         "1999"
>                       "Doe, 1999")
>                     (org-export-data (org-element-property :suffix ref) info)))
>                  ;; Grab all references within the citation.
>                  (org-cite-get-references citation)
>                  ", ")
>       (and (not (equal style "plain")) ")")
>       ;; Global suffix, if any.
>       (org-export-data (org-element-property :suffix citation) info)))
> 
>    (defun org-cite-demo-finalizer (output citations bibliography styles backend)
>      (concat output
>              ;; Print bibliography only if there are some citations and
>              ;; a "print_bibliography" keyword in the document.
>              (and citations
>                   styles                ;proof of "print_bibliography"
>                   (concat "\nBibliography: "
>                           (mapconcat #'identity bibliography ", ")
>                           "."))))
> 
> All is left to do is to register the "demo" processor.
> 
>    (org-cite-register-processor 'demo
>      :activate #'org-cite-demo-activate
>      :export-citation #'org-cite-demo-export-citation
>      :export-finalizer #'org-cite-demo-finalizer)
> 
> That's about it.
> 
> You can, if you wish so, try out this demo on the following document:
> 
> --8<---------------cut here---------------start------------->8---
> #+cite_export: demo
> 
> #+bibliography: bib.bib
> #+bibliography: bib2.bib
> 
> Simple reference: [cite:@key]
> 
> Multiple references: [cite:@key1;@key2]
> 
> Suppress author: [cite:-@key]
> 
> Plain style: [cite/plain:@key]
> 
> Full syntax: [cite:common prefix ;prefix @key suffix ;@key2; common suffix]
> 
> #+print_bibliography:
> 
> # Local Variables:
> # org-cite-activate-processor: demo
> # End:
> --8<---------------cut here---------------end--------------->8---
> 
> WDYT?
> 
> Regards,
> 

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Sun, Apr 25, 2021 at 5:20 PM Denis Maier <denismaier@mailbox.org> wrote:
>
> Hi everyone,
>
> great to see that there's been so much progress since this issue has
> last been discussed! Looks really impressive so far.
>
> Two comments/questions:
>
> - I think it should be possible to use some markup in prefixes and suffixes.

On this, on the other (big) wip-cite thread, Nicolas said the following:

"A drawback with allowing emphasis there is that prefix and suffix become
parsed data and not plain string anymore. As a consequence, searching
through them, e.g., when looking for locator names, requires an
additional level of indirection, since you need to first transform
parsed data back into plain text."

Nicolas: I get that you need more code to do this, but are there other
practical consequences of allowing more than plain-text?

Bruce

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On this, on the other (big) wip-cite thread, Nicolas said the following:
>
> "A drawback with allowing emphasis there is that prefix and suffix become
> parsed data and not plain string anymore. As a consequence, searching
> through them, e.g., when looking for locator names, requires an
> additional level of indirection, since you need to first transform
> parsed data back into plain text."
>
> Nicolas: I get that you need more code to do this, but are there other
> practical consequences of allowing more than plain-text?

The code I need to write does not count, really. It is 10 locs at most,
without the tests. 

My concern is the additional burden for processor developers. They will
need to shift from one representation to the other, this is not
convenient. Also some tasks then become an order of magnitude harder.

For example, let's consider the suffix: " p.32 and following" If it is
plain text, I can recognize p.32 as a locator, and replace it with "page
32" using something like `replace-regexp-in-string'.

Now, if it is a somewhat opaque parsed string, I need to first expand
it, for example with `org-element-interpret-data', modify the output as
above, and then parse it again to get a new parsed string.

I can no longer (concat "(" prefix author "," year suffix ")") either,
because prefix and suffix are not strings anymore. I need to

  (format "(%s%s, %s%s)"
          (org-export-data prefix info)
          (org-export-data author info)
          year    ;I assume year cannot contain problematic characters
          (org-export-data suffix info))

So, nothing impossible, but still slightly inconvenient.

Regards,
-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Denis Maier]

Am 26.04.2021 um 22:02 schrieb Nicolas Goaziou:
> Hello,
> 
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> 
>> On this, on the other (big) wip-cite thread, Nicolas said the following:
>>
>> "A drawback with allowing emphasis there is that prefix and suffix become
>> parsed data and not plain string anymore. As a consequence, searching
>> through them, e.g., when looking for locator names, requires an
>> additional level of indirection, since you need to first transform
>> parsed data back into plain text."
>>
>> Nicolas: I get that you need more code to do this, but are there other
>> practical consequences of allowing more than plain-text?
> 
> The code I need to write does not count, really. It is 10 locs at most,
> without the tests.
> 
> My concern is the additional burden for processor developers. They will
> need to shift from one representation to the other, this is not
> convenient. Also some tasks then become an order of magnitude harder.

While I cannot speak for the processor developers, I still want to throw 
in that at least the major CSL processors do handle formatting in 
affixes in one way or the other. The Haskell citeproc that is used by 
pandoc treats the suffixes as markdown, citeproc-js uses some sort of 
HTML-like tags for things like italics, bold, sub-/superscript, and a 
few more things.

> 
> For example, let's consider the suffix: " p.32 and following" If it is
> plain text, I can recognize p.32 as a locator, and replace it with "page
> 32" using something like `replace-regexp-in-string'.
> 
> Now, if it is a somewhat opaque parsed string, I need to first expand
> it, for example with `org-element-interpret-data', modify the output as
> above, and then parse it again to get a new parsed string.
> 
> I can no longer (concat "(" prefix author "," year suffix ")") either,
> because prefix and suffix are not strings anymore. I need to
> 
>    (format "(%s%s, %s%s)"
>            (org-export-data prefix info)
>            (org-export-data author info)
>            year    ;I assume year cannot contain problematic characters
>            (org-export-data suffix info))
> 
> So, nothing impossible, but still slightly inconvenient.
> 

If it isn't impossible I'd be strongly in favor of adding this. I'm 
pretty sure authors will miss it otherwises.

Denis

Re: (Not so) Short note about citations in Org

[Denis Maier]

Oh, and what do you think regarding the multiple bibliographies question?

Denis

Am 26.04.2021 um 22:02 schrieb Nicolas Goaziou:
> Hello,
> 
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> 
>> On this, on the other (big) wip-cite thread, Nicolas said the following:
>>
>> "A drawback with allowing emphasis there is that prefix and suffix become
>> parsed data and not plain string anymore. As a consequence, searching
>> through them, e.g., when looking for locator names, requires an
>> additional level of indirection, since you need to first transform
>> parsed data back into plain text."
>>
>> Nicolas: I get that you need more code to do this, but are there other
>> practical consequences of allowing more than plain-text?
> 
> The code I need to write does not count, really. It is 10 locs at most,
> without the tests.
> 
> My concern is the additional burden for processor developers. They will
> need to shift from one representation to the other, this is not
> convenient. Also some tasks then become an order of magnitude harder.
> 
> For example, let's consider the suffix: " p.32 and following" If it is
> plain text, I can recognize p.32 as a locator, and replace it with "page
> 32" using something like `replace-regexp-in-string'.
> 
> Now, if it is a somewhat opaque parsed string, I need to first expand
> it, for example with `org-element-interpret-data', modify the output as
> above, and then parse it again to get a new parsed string.
> 
> I can no longer (concat "(" prefix author "," year suffix ")") either,
> because prefix and suffix are not strings anymore. I need to
> 
>    (format "(%s%s, %s%s)"
>            (org-export-data prefix info)
>            (org-export-data author info)
>            year    ;I assume year cannot contain problematic characters
>            (org-export-data suffix info))
> 
> So, nothing impossible, but still slightly inconvenient.
> 
> Regards,
> 

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Denis Maier <denismaier@mailbox.org> writes:

> Oh, and what do you think regarding the multiple bibliographies
> question?

As far as Org is concerned, you can have multiple "bibliography"
keywords, e.g.,

  #+bibliography: file1.bib
  #+bibliography: file2.bib

Internally, the bibliography is stored as a list of absolute file names.

Is that what you have in mind or are you talking about something else?

Regards,

Re: (Not so) Short note about citations in Org

[Denis Maier]

Am 26.04.2021 um 22:33 schrieb Nicolas Goaziou:
> Denis Maier <denismaier@mailbox.org> writes:
>
>> Oh, and what do you think regarding the multiple bibliographies
>> question?
> As far as Org is concerned, you can have multiple "bibliography"
> keywords, e.g.,
>
>    #+bibliography: file1.bib
>    #+bibliography: file2.bib
>
> Internally, the bibliography is stored as a list of absolute file names.
>
> Is that what you have in mind or are you talking about something else?
>
> Regards,
No, I was not talking about having multiple input files, but about 
having multiple bibliographies in the output doc.
Perhaps each filtered in some way:

#+print_bibliography: [style] [filter1]
#+print_bibliography: [style] [filter2]

Obviously, filter1 and filter2 must be defined somewhere. The use case 
would be something along these lines:
- One bibliography with all the works by author X, a second bibliography 
with everything else.
- One bibliography with books, the other with webpages

Denis

Re: (Not so) Short note about citations in Org

[Nicolas Goaziou]

Denis Maier <denismaier@mailbox.org> writes:

> No, I was not talking about having multiple input files, but about
> having multiple bibliographies in the output doc.
> Perhaps each filtered in some way:
>
> #+print_bibliography: [style] [filter1]
> #+print_bibliography: [style] [filter2]
>
> Obviously, filter1 and filter2 must be defined somewhere. The use case
> would be something along these lines:
> - One bibliography with all the works by author X, a second
>   bibliography with everything else.
> - One bibliography with books, the other with webpages

It is already possible to write multiple "print_bibliography" keywords
(even though some citation systems, like biblatex, do not support it,
IIUC). However their value is limited to style at the moment.

It may be possible to send anything past the style string (i.e.,
starting after the first space) to the processor, as a sixth (!)
argument, and let it deal with it.

-- 
Nicolas Goaziou

Re: (Not so) Short note about citations in Org

[Denis Maier]

[-- Attachment #1: Type: text/html, Size: 2709 bytes --]

Re: (Not so) Short note about citations in Org

[Joost Kremers]

On Mon, Apr 26 2021, Denis Maier wrote:
> No, I was not talking about having multiple input files, but about having
> multiple bibliographies in the output doc.
> Perhaps each filtered in some way:
>
> #+print_bibliography: [style] [filter1]
> #+print_bibliography: [style] [filter2]
>
> Obviously, filter1 and filter2 must be defined somewhere. The use case would be
> something along these lines:
> - One bibliography with all the works by author X, a second bibliography with
>  everything else.
> - One bibliography with books, the other with webpages

Another use case would be a book in which each chapter has its own bibliography.

A quick synopsis of the biblatex way:

https://texblog.org/2012/10/22/multiple-bibliographies-with-biblatex/


-- 
Joost Kremers
Life has its moments

Re: (Not so) Short note about citations in Org

[Bruce D'Arcus]

On Mon, Apr 26, 2021 at 4:02 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> "Bruce D'Arcus" <bdarcus@gmail.com> writes:

> > Nicolas: I get that you need more code to do this, but are there other
> > practical consequences of allowing more than plain-text?
>
> The code I need to write does not count, really. It is 10 locs at most,
> without the tests.
>
> My concern is the additional burden for processor developers. They will
> need to shift from one representation to the other, this is not
> convenient. Also some tasks then become an order of magnitude harder.

I see.

So, for example, András currently has the function
'citeproc-org--parse-locator-affix', which takes as input a suffix
string.

That would need to be adapted to allow this, and it could be a hassle.

I agree with Denis that some users will want this, but I am unsure how
common the request might be.

Here's a thought:

How about having suffix plain text, since this is where locators are
parsed, and they are particularly important, but allow more on
prefixes?

Bruce