Org citations, CSL and citeproc-el

Org citations, CSL and citeproc-el

[Simonyi András]

Dear List Members,

a few days ago I've released the first public version of citeproc-el
(https://github.com/andras-simonyi/citeproc-el), a CSL 1.01 citation
processor library for Emacs. Since the main motivation of my work was to
contribute to org-mode's citation rendering I also implemented a "proof
of concept" add-on to org-ref that uses citeproc-el to render org-ref
citation links for non-LaTeX (and optionally even LaTeX) export backends
(see https://github.com/andras-simonyi/citeproc-orgref).

Both packages are in a relatively early stage of their development so
I'd be grateful to receive any feedback on them. In particular, in
citeproc-orgref I had to abuse the cite link descriptions to an even
greater degree than they were by org-ref to accommodate the full CSL
representation of citations (citations consist of cites and each cite
can have a prefix, postfix and a locator). The resulting link syntax is
rather cumbersome so I'd like to ask your opinion about introducing an
alternative org-mode citation syntax that handles all of these elements.
One option would be to use something very similar to pandoc's citation
syntax (which I tried to follow as much as possible in the cite link
descriptions of citeproc-orgref).

A more general question I'd like to raise how (or whether) you see
citeproc-el's (and CSL's) potential place in the org-mode ecosystem.
There are a lot of directions which the further development could take
(BibLaTeX support, citeproc-YAML bibliographies, CSL editing and CSL
extensions etc.) and I'd be grateful to receive your input on which ones
I should focus on.

thanks in advance & best wishes,

András Simonyi

Re: Org citations, CSL and citeproc-el

[Richard Lawrence]

Dear Simonyi,
 
Simonyi András <andras.simonyi@gmail.com> writes:

> a few days ago I've released the first public version of 
> citeproc-el (https://github.com/andras-simonyi/citeproc-el), a 
> CSL 1.01 citation processor library for Emacs.
 
Wow!  I don't know if you are aware, but we had discussed the 
possibility of building something like this on the list a few 
years back.  At the time it seemed like a lot more work than 
anyone was willing to do, and so the effort stalled.  Thanks so 
much for working on this -- it looks like you've done a ton of 
work!

> The resulting link syntax is rather cumbersome so I'd like to 
> ask your opinion about introducing an alternative org-mode 
> citation syntax that handles all of these elements.  One option 
> would be to use something very similar to pandoc's citation 
> syntax (which I tried to follow as much as possible in the cite 
> link descriptions of citeproc-orgref). 

We had a (very) long discussion about implementing a new citation 
syntax for Org in the spring of 2015.  Most of it took place in 
February and March; see:

https://lists.gnu.org/archive/html/emacs-orgmode/2015-02/threads.html
https://lists.gnu.org/archive/html/emacs-orgmode/2015-03/threads.html 

And here is how I summarized the state of that discussion in June:

https://lists.gnu.org/archive/html/emacs-orgmode/2015-06/msg00426.html

The upshot was that we came to agreement on quite a few points 
about what citation syntax should look like, and those points have 
been implemented in Org syntax in the wip-cite branch of the Org 
repo.  (This branch is now long out of date and at the very least 
in need of a rebase onto current master, I suspect.)  It would be 
really great to get things moving again.  Using your code to 
provide citation processing during Org export for the syntax 
that's already been implemented would be the place to start.

There are other aspects of the syntax we agreed on that are not 
implemented yet; as I recall, the idea was to get a minimal 
agreeable subset working, and then add to it based on real-world 
experience and feedback.
 
> A more general question I'd like to raise how (or whether) you 
> see citeproc-el's (and CSL's) potential place in the org-mode 
> ecosystem.  There are a lot of directions which the further 
> development could take (BibLaTeX support, citeproc-YAML 
> bibliographies, CSL editing and CSL extensions etc.) and I'd be 
> grateful to receive your input on which ones I should focus on. 

Here's my two cents on this. From what I recall about the 
discussion (it's hard to believe it's already been almost three 
years...), I would focus on two things:

1) BibLaTeX support.  Most Org users who want citation support 
want it primarily for LaTeX export, and BibLaTeX provides a good 
model of what we should try to achieve in Org citation support.

2) org-bibtex support.  Org-bibtex is a library that represents 
citation data via Org's property syntax.  Some people use this and 
it would be great to have it integrated with the new citation 
syntax and export.  That would provide an end-to-end citation 
solution that is completely native to Org and Emacs. 

Again, thanks for jump-starting this effort!  I look forward to 
contributing to it this spring.

-- 
Best,
Richard

Re: Org citations, CSL and citeproc-el

[Tory S. Anderson]

Just a second on this; it's a fantastic and useful idea. 

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Dear Simonyi,
>
> Simonyi András <andras.simonyi@gmail.com> writes:
>
>> a few days ago I've released the first public version of 
>> citeproc-el (https://github.com/andras-simonyi/citeproc-el), a 
>> CSL 1.01 citation processor library for Emacs.
>
> Wow!  I don't know if you are aware, but we had discussed the 
> possibility of building something like this on the list a few 
> years back.  At the time it seemed like a lot more work than 
> anyone was willing to do, and so the effort stalled.  Thanks so 
> much for working on this -- it looks like you've done a ton of 
> work!
>
>> The resulting link syntax is rather cumbersome so I'd like to 
>> ask your opinion about introducing an alternative org-mode 
>> citation syntax that handles all of these elements.  One option 
>> would be to use something very similar to pandoc's citation 
>> syntax (which I tried to follow as much as possible in the cite 
>> link descriptions of citeproc-orgref). 
>
> We had a (very) long discussion about implementing a new 
> citation syntax for Org in the spring of 2015.  Most of it took 
> place in February and March; see:
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2015-02/threads.html
> https://lists.gnu.org/archive/html/emacs-orgmode/2015-03/threads.html 
>
> And here is how I summarized the state of that discussion in 
> June:
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2015-06/msg00426.html
>
> The upshot was that we came to agreement on quite a few points 
> about what citation syntax should look like, and those points 
> have been implemented in Org syntax in the wip-cite branch of 
> the Org repo.  (This branch is now long out of date and at the 
> very least in need of a rebase onto current master, I suspect.) 
> It would be really great to get things moving again.  Using your 
> code to provide citation processing during Org export for the 
> syntax that's already been implemented would be the place to 
> start.
>
> There are other aspects of the syntax we agreed on that are not 
> implemented yet; as I recall, the idea was to get a minimal 
> agreeable subset working, and then add to it based on real-world 
> experience and feedback.
>
>> A more general question I'd like to raise how (or whether) you 
>> see citeproc-el's (and CSL's) potential place in the org-mode 
>> ecosystem.  There are a lot of directions which the further 
>> development could take (BibLaTeX support, citeproc-YAML 
>> bibliographies, CSL editing and CSL extensions etc.) and I'd be 
>> grateful to receive your input on which ones I should focus 
>> on. 
>
> Here's my two cents on this. From what I recall about the 
> discussion (it's hard to believe it's already been almost three 
> years...), I would focus on two things:
>
> 1) BibLaTeX support.  Most Org users who want citation support 
> want it primarily for LaTeX export, and BibLaTeX provides a good 
> model of what we should try to achieve in Org citation support.
>
> 2) org-bibtex support.  Org-bibtex is a library that represents 
> citation data via Org's property syntax.  Some people use this 
> and it would be great to have it integrated with the new 
> citation syntax and export.  That would provide an end-to-end 
> citation solution that is completely native to Org and Emacs. 
>
> Again, thanks for jump-starting this effort!  I look forward to 
> contributing to it this spring.

Re: Org citations, CSL and citeproc-el

[Christian Moe]

This is exciting news, and I look forward to trying it out.

Regarding syntax:

1. A couple of years back, there was an active discussion on this list to
flesh out a native citation syntax for Org. I fell off the discussion,
but I think it resulted in a non-link-based, feature-rich syntax spec,
but not much code to make use of it. Perhaps someone could comment?

2. Not many people liked my proposal: a link-based syntax similar to yours
but also including author and date in the link description. The idea was
to have descriptions that are both (1) human-readable even though the
citekeys are hidden away in the link path and (2) machine-readable to
determine the form of the citation (parenthetical or not, date-only or
not, etc.). I have used one to work with Zotero, and have a simple
parser for it, if you're interested in pursuing something similar.

Yours,
Christian

Simonyi András writes:

> Dear List Members,
>
> a few days ago I've released the first public version of citeproc-el
> (https://github.com/andras-simonyi/citeproc-el), a CSL 1.01 citation
> processor library for Emacs. Since the main motivation of my work was to
> contribute to org-mode's citation rendering I also implemented a "proof
> of concept" add-on to org-ref that uses citeproc-el to render org-ref
> citation links for non-LaTeX (and optionally even LaTeX) export backends
> (see https://github.com/andras-simonyi/citeproc-orgref).
>
> Both packages are in a relatively early stage of their development so
> I'd be grateful to receive any feedback on them. In particular, in
> citeproc-orgref I had to abuse the cite link descriptions to an even
> greater degree than they were by org-ref to accommodate the full CSL
> representation of citations (citations consist of cites and each cite
> can have a prefix, postfix and a locator). The resulting link syntax is
> rather cumbersome so I'd like to ask your opinion about introducing an
> alternative org-mode citation syntax that handles all of these elements.
> One option would be to use something very similar to pandoc's citation
> syntax (which I tried to follow as much as possible in the cite link
> descriptions of citeproc-orgref).
>
> A more general question I'd like to raise how (or whether) you see
> citeproc-el's (and CSL's) potential place in the org-mode ecosystem.
> There are a lot of directions which the further development could take
> (BibLaTeX support, citeproc-YAML bibliographies, CSL editing and CSL
> extensions etc.) and I'd be grateful to receive your input on which ones
> I should focus on.
>
> thanks in advance & best wishes,
>
> András Simonyi

Re: Org citations, CSL and citeproc-el

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 899 bytes --]

On Friday,  5 Jan 2018 at 08:52, Richard Lawrence wrote:

[...]

As Richard says, wow!  This would seem to fit nicely with what we
(i.e. the mailing list as I don't think I actually said anything...)
did discuss about what was needed going forward.

> Here's my two cents on this. From what I recall about the discussion
> (it's hard to believe it's already been almost three years...), I
> would focus on two things:

> 2) org-bibtex support.  Org-bibtex is a library that represents

+1

All of my references are in org-bibtex format and supporting this
would be really welcome.  At present, I export these to bibtex files
for use with org->PDF via LaTeX but it would be great to have:

1. an easy to use citing mechanism in org that knows org-bibtex and
2. export to ODT and/or html with citations.

thanks,
eric

-- 
Eric S Fraga via Emacs 27.0.50, Org release_9.1.6

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: Org citations, CSL and citeproc-el

[Nicolas Goaziou]

Hell,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Simonyi András <andras.simonyi@gmail.com> writes:
>
>> a few days ago I've released the first public version of citeproc-el
>> (https://github.com/andras-simonyi/citeproc-el), a CSL 1.01 citation
>> processor library for Emacs.

Excellent! Thank you for this work.

>> The resulting link syntax is rather cumbersome so I'd like to ask
>> your opinion about introducing an alternative org-mode citation
>> syntax that handles all of these elements.  One option would be to
>> use something very similar to pandoc's citation syntax (which
>> I tried to follow as much as possible in the cite link descriptions
>> of citeproc-orgref). 

[...]

> The upshot was that we came to agreement on quite a few points about
> what citation syntax should look like, and those points have been
> implemented in Org syntax in the wip-cite branch of the Org repo.
> (This branch is now long out of date and at the very least in need of
> a rebase onto current master, I suspect.)

The "wip-cite" branch is up-to-date with master (give or take last
commits). It implements a good basis for a citation syntax in Org (tests
included).

Also, there is "wip-cite-awe" branch from Aaron Ecay (Cc'ed) which
implements some support for exporting citations. "wip-cite-awe" is
up-to-date and base on top of "wip-cite".

> There are other aspects of the syntax we agreed on that are not
> implemented yet; as I recall, the idea was to get a minimal agreeable
> subset working, and then add to it based on real-world experience and
> feedback.

AFAIR, everything we agreed on is implemented in "wip-cite" branch.

That's a good thing to know that citations may move forward. Let us know
if you need help. Meanwhile, as Richard suggests, please have a look at
"wip-cite" branch.

Regards,

-- 
Nicolas Goaziou

Re: Org citations, CSL and citeproc-el

[András Simonyi]

Dear All,

thanks a lot for the helpful responses!

Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> I don't know if you are aware, but we had discussed the possibility
  > of building something like this on the list a few years back. [...]
  > And here is how I summarized the state of that discussion in June:
  > https://lists.gnu.org/archive/html/emacs-orgmode/2015-06/msg00426.html
  > The upshot was that we came to agreement on quite a few points about what
  > citation syntax should look like, and those points have been implemented in Org
  > syntax in the wip-cite branch of the Org repo.

I found some of the discussion, but not your summary and the
cite-related branches -- these are very useful.

  > 1) BibLaTeX support.
  > 2) org-bibtex support.

I'll have to look into BibLaTeX more but I suspect that both of these
should be doable relatively easily on the basis of the already existing
(albeit by no means complete) BibTeX support in citeproc-el. I'll create
corresponding enhancement issues in the github repo. A related thought:
was there any initiative to create something like org-bibtex but for CSL
bibliography descriptions? It could be an interesting alternative to
citeproc-YAML (see
http://blog.martinfenner.org/2013/07/30/citeproc-yaml-for-bibliographies/).

I've checked the citation/cite syntax in the wip-cite branch and it
seems to be a good starting point for trying to use citeproc-el to
render the citations. As a first step I could simply abstract away the
syntax and add a switch in citeproc-orgref to choose between the org-ref
and the the org-mode citation syntax (as it is in the wip-cite branch),
without touching other parts of the code. Currently citation rendering
is done (following org-ref's citeproc) by means of the
`org-export-before-parsing-hook' -- all citations are replaced with
their rendered versions before parsing. I'm not familiar enough with
org-mode's internals to assess how hackish this solution is, maybe there
is a much better way. One thing to consider is that I need all citations
together with their positional information (order, footnote number etc.)
for rendering in order to deal with disambiguation, ibid. forms etc.

Another question is the syntax for specifying the bibliography to be
used and, maybe, the location where the bibliography should be placed.
Org-ref (ab)uses links for these purposes as well. Was there a syntax
proposal for these things too?

Christian Moe <mail@christianmoe.com> wrote:
  > Not many people liked my proposal: a link-based syntax similar to yours
  > but also including author and date in the link description. The idea was
  > to have descriptions that are both (1) human-readable even though the
  > citekeys are hidden away in the link path and (2) machine-readable to
  > determine the form of the citation (parenthetical or not, date-only or
  > not, etc.).

This is an interesting proposal -- I like the idea of human readable
citations in the unexported org-mode documents but I'm not sure what
would be the best way to achieve them. citeproc-el can render isolated
references on the fly so they could be perhaps used as tool-tips or
overlays? Another difficulty I see is the generation of this added
information -- would it be automatic? What happens if the author or the
year is missing, or is ambiguous etc. If I could get hold of your parser
then perhaps this could also be added as an alternative citation syntax
in citeproc-orgref. Another option could be to normalize the keys to a
disambiguated author-year format, but this would modify the bibliography
file which is not necessarily a good idea...

thanks again and best regards,

András

Re: Org citations, CSL and citeproc-el

[Richard Lawrence]

Hi András and all: 

Sorry for the delay.  
 
András Simonyi <andras.simonyi@gmail.com> writes:
 
> Another question is the syntax for specifying the bibliography 
> to be used and, maybe, the location where the bibliography 
> should be placed.  Org-ref (ab)uses links for these purposes as 
> well. Was there a syntax proposal for these things too? 

As far as I know, there was not.  This was not settled because in 
order to settle it, you need to know a bit more about the backend. 
(Where is the citation data stored? do you have to get it from an 
external program? if so, which? etc. etc.)

I think there was some agreement that we can do this with option 
keywords.  There is an existing syntax for specifying 
bibliographies in contrib/ox-bibtex.el, which probably provides a 
good model:

#+BIBLIOGRAPHY: bibfilename stylename optional-options  When I was 
playing around with hooking up the citation syntax to an external 
citation processor, I was using something similar: I had keywords 
like

#+CSL_FILE: the-csl-style-file.csl +BIBDB: bibtex testdoc.bib 

in the header, and used

#+BIBLIOGRAPHY: here

to specify where the bibliography should be placed in the 
document.  That is obviously not the nicest solution, but the 
basic idea might be a good starting point. 


-- 
Best,
Richard