Citation syntax: a revised proposal

Citation syntax: a revised proposal

[Richard Lawrence]

[-- Attachment #1: Type: text/plain, Size: 1764 bytes --]

Hi everyone,

Since discussion seems to have petered out on the previous thread (see:
http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
go back over the discussion and write up a concrete proposal for
citation syntax.

This proposal represents my attempt to formulate a syntax that is easy
to read, easy to parse, and covers all the use-cases that people
mentioned as being important.  It is surely not perfect, but I learned a
lot from the previous thread, and I hope something like this will serve
the community's needs.

The proposal is below, both inline (for easy quoting) and attached (for
easy reading).  To keep it relatively short, I have mostly not explained
my reasoning for the choices I made, but I am happy to do so here if
anyone has questions.

I welcome feedback, comments, criticisms, and objections on any point.
However, since we've already had a long discussion about this, I
respectfully request that we try to keep this thread focused.  To that
end, I suggest:

  1) If you have criticisms or objections, please try to indicate
     whether you think they are `substantive' (e.g., you see a problem
     that would prevent you from using this syntax, or prevent Org from
     implementing it) or not (e.g., you would prefer a slightly
     different but equivalent way of expressing something).

  2) If you wish to express an opinion about the proposal without
     offering further comments, let us know by just replying with +1
     (meaning you'd like to see this syntax, or something reasonably
     similar to it, be adopted), 0, or -1 (meaning you'd prefer not to
     see this syntax or anything similar to it adopted).

I guess this is my Valentine to the Org community. :) Thanks for reading!

Best,
Richard


[-- Attachment #2: citation-syntax.org --]
[-- Type: text/plain, Size: 13614 bytes --]

#+TITLE: Citation syntax, a revised proposal
#+DATE: <2015-02-14 Sat>
#+AUTHOR: Richard Lawrence
#+EMAIL: richard.lawrence@berkeley.edu
#+LANGUAGE: en
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport

* Citation syntax
** Requirements
A citation is a textual reference to one or more individual works,
together with other information about those works, grouped together in
a single place.  

Within a citation, each reference to an individual work needs to be
capable of containing:
  1) a database key that references the cited work
  2) prefix / pre-note
  3) suffix / post-note
     
Whole citations also need:
  4) [@4] a way of specifying whether the citation is in-text or
     parenthetical
  5) a way of representing a common prefix and suffix, if the citation
     is a multi-cite
  6) a way of specifying whether the citation should produce a
     complete bibliography entry in-place
  7) an extensible way of specifying formatting properties to export
     filters and/or specific export backends 
     
** Citation definitions
*** Citation keys; bibliography references vs. complete entries 
A citation key consists of a unique label preceded by a flag, which is
optionally preceded by a hyphen.

The flag is either `@' or `&'.  `@' indicates that the citation should
produce a normal reference to the bibliography entry for the cited
work (in whatever style the document uses), located elsewhere.

The `&' flag indicates that the citation should produce a complete
bibliography entry for the cited work in the place where the citation
appears.

The optional hyphen (`-') indicates that the author's name should be
suppressed from the rendered citation.  (Note that this is only useful
in author-X citation styles; it should have no effect in numeric
styles.)

*** Basic citations: Parenthetical vs. in-text 
There are two basic types of citation: /parenthetical/ and /in-text/.
Each of these may contain references to one or more individual works.

The difference between parenthetical and in-text citations is
expressed using parentheses around the /first/ citation key.  A
parenthetical citation has such parentheses around the first citation
key; an in-text citation lacks them.  (Parentheses around non-initial
keys are permitted for visual consistency and to keep the grammar
simple, but have no meaning.)

A citation thus consists in general of a bracketed list, beginning
with `cite:', of one or more individual references, each of which:
  - may contain a prefix,
  - must contain a citation key, which may or may not be surrounded by `(...)'
  - and may contain a suffix
Individual references are separated by semi-colons.

There are also two special cases to make simple-but-common uses very
easy to type and read:
  1) a parenthetical citation for a single work with no prefix and
     suffix may be written by just surrounding the key with brackets,
     like: [@Doe99].
  2) an in-text citation for a single work with no prefix and suffix
     may be written as a /bare/ key, without brackets, like: @Doe99.
(Thus, in both of the `simple' cases, one less level of bracketing is
required.)

Prefix and suffix text are regular Org text, which are allowed to
contain various kinds of Org markup (see the grammar below for a
complete list).

*** Multi-cite citations 
Multi-cite citations are distinguished from basic parenthetical and
in-text citations by the presence of an optional common prefix or
common suffix (which may not contain keys).  If present, the common
prefix must occur before the first individual reference, and the
common suffix must occur after the last individual reference.  The
common prefix and suffix are separated from the individual references
by semi-colons.

*** Examples of main citation syntax
Basic parenthetical citation:
#+BEGIN_QUOTE
The nineteenth century was very interesting. [cite: (@Doe99)]
#+END_QUOTE

Basic parenthetical citation using special-case syntax:
#+BEGIN_QUOTE
The nineteenth century was very interesting. [@Doe99]
#+END_QUOTE

Parenthetical citation with multiple works and prefix and suffix:
#+BEGIN_QUOTE
The nineteenth century was in fact lovely [cite: see (@Doe99) p. 44;
@Smith2000 has a review].
#+END_QUOTE

Basic in-text citation with a suffix:
#+BEGIN_QUOTE
As [cite: @Doe99 p. 44] says, the nineteenth century was very interesting.
#+END_QUOTE

In-text citation using special-case syntax:
#+BEGIN_QUOTE
@Doe2000 explains that the twentieth century was even more interesting. 
#+END_QUOTE

In-text citation with author suppressed:
#+BEGIN_QUOTE
As Doe explained in his -@Doe2003, the twentieth century was somewhat
less interesting than previously thought.
#+END_QUOTE

Parenthetical citation with full-entry key:
#+BEGIN_QUOTE
A complete bibliography entry follows in parentheses. [cite: (&Doe99)]
A complete bibliography entry follows in parentheses. [&Doe99]
#+END_QUOTE

In-text citation with full-entry key:
#+BEGIN_QUOTE
A complete bibliography entry follows: [cite: &Doe99].
A complete bibliography entry follows: &Doe99.
#+END_QUOTE

Full-entry in-text citation, in a footnote:
#+BEGIN_QUOTE
Doe exhibits unusual scholarship.[fn:: &Doe99.]
#+END_QUOTE

In-text citation, with a complete bibliography entry minus the author
in a footnote, plus a suffix:
#+BEGIN_QUOTE
@Doe99 exhibits unusual scholarship.[fn:1]

[fn:1] [cite: -&Doe99 Cf. especially section 4.]
#+END_QUOTE

In-text multi-cite:
#+BEGIN_QUOTE
Speculation abounds about what the twenty-first century will
bring. [cite: For an overview of this topic, see; @Smith1998;
@Jones1999; @Miller2001; and references therein.]
#+END_QUOTE

Parenthetical multi-cite:
#+BEGIN_QUOTE
Speculation abounds about what the twenty-first century will
bring. [cite: For an overview of this topic, see; (@Smith1998);
@Jones1999; @Miller2001; and references therein.]
#+END_QUOTE

*** Syntax for extensions 
Additional information can be supplied in a citation that may affect
how export filters or particular backends format it.

This additional information may be supplied following the brackets of
a citation between the following delimiters: `%%( ... )'.

(Note: I am proposing that this expression go /after/ the main
citation brackets both because it visually separates this extra
information from the main citation, and in order to avoid imposing any
further syntactic restriction on suffixes.)

At least for now, any information supplied this way is /strictly the
user's responsibility/ to interpret (e.g., using an export filter).
This means that citations that have information like this are not
portable and might not be exported correctly:
  - in other users' setups
  - by particular backends
  - by future versions of Org

I will not deal with the details of how this additional information
should be syntactically represented, since this has not really been
discussed.  But I suggest that, to deal with the complexities of
additional information in full generality, something like a complete
Lisp list is required.  Thus, I suggest that this additional
information simply be represented as a Lisp list.  (Besides
generality, this has the benefit of making the syntax easy to parse:
the parser can just call Elisp's read function with a marker after the
`%%'.)

I provide these examples merely to illustrate the possibilities here:
#+BEGIN_QUOTE
@vonNeumann1930 %%(:type genitive :capitalize t) model can only handle
a limited range of observed cases.

@McCarthy1950 %%('s) clever use of Lisp syntax was also used to
express the Saxon genitive.

For more, see Ref. @Doe99 %%(:type refnum :follow-to "some.pdf").

Even more complicated examples occur after Doe's famous article from
[cite: @Doe99] %%(:type date-only).

And in [cite: @Doe2000] %%(:attr_latex (:format-string
"\citeyear{%KEY}") :attr_html (:only-fields (month year))), Doe
finally realized that arbitrary complexity was a powerful but
double-edged sword.

@_aParticularlyUGLYkey:is-this-one %%(:overlay "Nice Display")
#+END_QUOTE 

** Grammar
This section formally documents the syntax of citations discussed
above.  

To represent the syntax of citations, we need a category of /citation/
objects, which require the following properties (the names here are not
important and could be changed):
  - is-parenthetical (boolean; nil means is in-text)
  - common-prefix (text)
  - common-suffix (text)
  - references (list)
  - extra-info (list)

Each reference in the list of references should be a plist with the
following properties:
  - prefix (text)
  - suffix (text)
  - key (string)
  - is-parenthesized (boolean; t means key was parenthesized; only
    significant for the first reference in a citation)
  - suppress-author (boolean; t means author name should not be output)
  - is-full (boolean; t means a full bibliography entry should be
    output in-place) 

The category of citations has the following grammar:
  - A CITATION is a PARENTHETICAL-CITATION or an IN-TEXT citation.
  - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
    CITATION-LIST whose first individual INDIVIDUAL-REFERENCE is a
    PARENTHESIZED-KEY
  - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
    CITATION-LIST whose first INDIVIDUAL-REFERENCE is a BARE-KEY.
  - A SIMPLE-PARENTHETICAL is a KEY immediately surrounded by square
    brackets, optionally followed by an EXTRA-INFO clause.
  - A SIMPLE-IN-TEXT is a BARE-KEY, optionally followed by an
    EXTRA-INFO clause
  - A CITATION-LIST has the format
       [cite: PREFIX; INDIVIDUAL-REFERENCE; ... INDIVIDUAL-REFERENCE; SUFFIX] EXTRA-INFO
    where the initial PREFIX, final SUFFIX, and EXTRA-INFO clause are
    optional.  At least one INDIVIDUAL-REFERENCE must be present. 
  - An INDIVIDUAL-REFERENCE has the format: 
       PREFIX KEY-MAYBE-PARENS SUFFIX
    The KEY-MAYBE-PARENS is obligatory, and the prefix and suffix
    are optional.
  - A KEY-MAYBE-PARENS is either a BARE-KEY or PARENTHESIZED-KEY
  - A BARE-KEY is a KEY with immediately-preceding whitespace
  - A PARENTHESIZED-KEY is a KEY immediately surrounded by `(' and `)'.
  - A KEY optionally begins with `-', and obligatorily contains `@' or
    `&' followed by a string of characters which begins with a letter
    or `_', and may contain alphanumeric characters and the following
    internal punctuation characters:
       :.#$%&-+?<>~/
  - A PREFIX or SUFFIX is arbitrary text (except `;', `]', and
    KEY-MAYBE-PARENs) which may contain only the following Org
    objects:
    - bold
    - code
    - entity
    - italic
    - latex-fragment
    - line-break
    - strike-through
    - subscript
    - superscript
    - underline
    - superscript
    (Note that this list could be extended somewhat if necessary.)
  - An EXTRA-INFO clause consists of data not specified by this
    grammar, in between `%%(' and `)'

** Outstanding issues
It seems to me that there are potential problems with the above
proposal in a number of areas, but I cannot tell how serious they are,
or what changes (if any) should be made to solve them.  I don't
pretend that this is an exhaustive list:
  1) *Nesting.*  I have favored LaTeX compatibility for in-text
     citations with multiple references; but this means there is no
     way to `nest' citations.  Thus, there is no way to express (in
     the main syntax) what Pandoc expresses as:
        @Doe99 [p. 34; see also @DoeRoe2000]
     which renders like:
        Doe (1999, p. 34; see also Doe and Roe 2000)
     Instead, since a citation is in-text or parenthetical as a whole,
     the equivalent in the above syntax
        [cite: @Doe99 p. 34; see also @DoeRoe2000]
     should render like:
	Doe (1999, p. 34), see also Doe and Roe (2000).
     I am not certain if Pandoc-like output is important in this case.
     The few people who commented on this said that it was not. 
  2) *Limitations on prefixes and suffixes.*  There may be legitimate
     uses of `@', `;', `]', etc. inside prefix or suffix text that the
     above syntax does not allow.  Examples might include:
     - use of semi-colons as part of the prefix/suffix text
     - footnotes, links, or timestamps inside a prefix/suffix
     I am not certain how important these cases are.  If they are
     important, some of them might be able to be worked around with
     entities.
  3) *Edge cases.* The above syntax may make it possible to express
     things that don't make sense, or would be too difficult to
     export.  The only one I can think of is that it is possible to
     mix `@'-style and `&'-style keys in the same citation.  I am not
     sure if this should be forbidden; it may sometimes make sense.
     It may also be possible to express things that external tools,
     such as citeproc-js, don't know how to process.  I do not have a
     good sense of what, if anything, falls into that category, and
     what should be done about it.
  4) *Citation commands.*  Rather than introduce an explicit
     representation for different citation commands/types, I have used
     different parts of the syntax to express the common distinctions
     that people mentioned.  I suggest that, for now, anything beyond
     these basic distinctions be left to the user-extension syntax.
     However, if it becomes clear in the future that there is a need
     to add a representation for a command to the main syntax, there
     is a natural place to do so: immediately after the `cite:' tag
     (as Nicolas suggested).

Also, I have not said anything in this proposal to address how other
document metadata should be represented, which has not been discussed
much on the list.  I think this should be discussed separately.
      

[-- Attachment #3: citation-syntax.org --]
[-- Type: text/plain, Size: 13614 bytes --]

#+TITLE: Citation syntax, a revised proposal
#+DATE: <2015-02-14 Sat>
#+AUTHOR: Richard Lawrence
#+EMAIL: richard.lawrence@berkeley.edu
#+LANGUAGE: en
#+SELECT_TAGS: export
#+EXCLUDE_TAGS: noexport

* Citation syntax
** Requirements
A citation is a textual reference to one or more individual works,
together with other information about those works, grouped together in
a single place.  

Within a citation, each reference to an individual work needs to be
capable of containing:
  1) a database key that references the cited work
  2) prefix / pre-note
  3) suffix / post-note
     
Whole citations also need:
  4) [@4] a way of specifying whether the citation is in-text or
     parenthetical
  5) a way of representing a common prefix and suffix, if the citation
     is a multi-cite
  6) a way of specifying whether the citation should produce a
     complete bibliography entry in-place
  7) an extensible way of specifying formatting properties to export
     filters and/or specific export backends 
     
** Citation definitions
*** Citation keys; bibliography references vs. complete entries 
A citation key consists of a unique label preceded by a flag, which is
optionally preceded by a hyphen.

The flag is either `@' or `&'.  `@' indicates that the citation should
produce a normal reference to the bibliography entry for the cited
work (in whatever style the document uses), located elsewhere.

The `&' flag indicates that the citation should produce a complete
bibliography entry for the cited work in the place where the citation
appears.

The optional hyphen (`-') indicates that the author's name should be
suppressed from the rendered citation.  (Note that this is only useful
in author-X citation styles; it should have no effect in numeric
styles.)

*** Basic citations: Parenthetical vs. in-text 
There are two basic types of citation: /parenthetical/ and /in-text/.
Each of these may contain references to one or more individual works.

The difference between parenthetical and in-text citations is
expressed using parentheses around the /first/ citation key.  A
parenthetical citation has such parentheses around the first citation
key; an in-text citation lacks them.  (Parentheses around non-initial
keys are permitted for visual consistency and to keep the grammar
simple, but have no meaning.)

A citation thus consists in general of a bracketed list, beginning
with `cite:', of one or more individual references, each of which:
  - may contain a prefix,
  - must contain a citation key, which may or may not be surrounded by `(...)'
  - and may contain a suffix
Individual references are separated by semi-colons.

There are also two special cases to make simple-but-common uses very
easy to type and read:
  1) a parenthetical citation for a single work with no prefix and
     suffix may be written by just surrounding the key with brackets,
     like: [@Doe99].
  2) an in-text citation for a single work with no prefix and suffix
     may be written as a /bare/ key, without brackets, like: @Doe99.
(Thus, in both of the `simple' cases, one less level of bracketing is
required.)

Prefix and suffix text are regular Org text, which are allowed to
contain various kinds of Org markup (see the grammar below for a
complete list).

*** Multi-cite citations 
Multi-cite citations are distinguished from basic parenthetical and
in-text citations by the presence of an optional common prefix or
common suffix (which may not contain keys).  If present, the common
prefix must occur before the first individual reference, and the
common suffix must occur after the last individual reference.  The
common prefix and suffix are separated from the individual references
by semi-colons.

*** Examples of main citation syntax
Basic parenthetical citation:
#+BEGIN_QUOTE
The nineteenth century was very interesting. [cite: (@Doe99)]
#+END_QUOTE

Basic parenthetical citation using special-case syntax:
#+BEGIN_QUOTE
The nineteenth century was very interesting. [@Doe99]
#+END_QUOTE

Parenthetical citation with multiple works and prefix and suffix:
#+BEGIN_QUOTE
The nineteenth century was in fact lovely [cite: see (@Doe99) p. 44;
@Smith2000 has a review].
#+END_QUOTE

Basic in-text citation with a suffix:
#+BEGIN_QUOTE
As [cite: @Doe99 p. 44] says, the nineteenth century was very interesting.
#+END_QUOTE

In-text citation using special-case syntax:
#+BEGIN_QUOTE
@Doe2000 explains that the twentieth century was even more interesting. 
#+END_QUOTE

In-text citation with author suppressed:
#+BEGIN_QUOTE
As Doe explained in his -@Doe2003, the twentieth century was somewhat
less interesting than previously thought.
#+END_QUOTE

Parenthetical citation with full-entry key:
#+BEGIN_QUOTE
A complete bibliography entry follows in parentheses. [cite: (&Doe99)]
A complete bibliography entry follows in parentheses. [&Doe99]
#+END_QUOTE

In-text citation with full-entry key:
#+BEGIN_QUOTE
A complete bibliography entry follows: [cite: &Doe99].
A complete bibliography entry follows: &Doe99.
#+END_QUOTE

Full-entry in-text citation, in a footnote:
#+BEGIN_QUOTE
Doe exhibits unusual scholarship.[fn:: &Doe99.]
#+END_QUOTE

In-text citation, with a complete bibliography entry minus the author
in a footnote, plus a suffix:
#+BEGIN_QUOTE
@Doe99 exhibits unusual scholarship.[fn:1]

[fn:1] [cite: -&Doe99 Cf. especially section 4.]
#+END_QUOTE

In-text multi-cite:
#+BEGIN_QUOTE
Speculation abounds about what the twenty-first century will
bring. [cite: For an overview of this topic, see; @Smith1998;
@Jones1999; @Miller2001; and references therein.]
#+END_QUOTE

Parenthetical multi-cite:
#+BEGIN_QUOTE
Speculation abounds about what the twenty-first century will
bring. [cite: For an overview of this topic, see; (@Smith1998);
@Jones1999; @Miller2001; and references therein.]
#+END_QUOTE

*** Syntax for extensions 
Additional information can be supplied in a citation that may affect
how export filters or particular backends format it.

This additional information may be supplied following the brackets of
a citation between the following delimiters: `%%( ... )'.

(Note: I am proposing that this expression go /after/ the main
citation brackets both because it visually separates this extra
information from the main citation, and in order to avoid imposing any
further syntactic restriction on suffixes.)

At least for now, any information supplied this way is /strictly the
user's responsibility/ to interpret (e.g., using an export filter).
This means that citations that have information like this are not
portable and might not be exported correctly:
  - in other users' setups
  - by particular backends
  - by future versions of Org

I will not deal with the details of how this additional information
should be syntactically represented, since this has not really been
discussed.  But I suggest that, to deal with the complexities of
additional information in full generality, something like a complete
Lisp list is required.  Thus, I suggest that this additional
information simply be represented as a Lisp list.  (Besides
generality, this has the benefit of making the syntax easy to parse:
the parser can just call Elisp's read function with a marker after the
`%%'.)

I provide these examples merely to illustrate the possibilities here:
#+BEGIN_QUOTE
@vonNeumann1930 %%(:type genitive :capitalize t) model can only handle
a limited range of observed cases.

@McCarthy1950 %%('s) clever use of Lisp syntax was also used to
express the Saxon genitive.

For more, see Ref. @Doe99 %%(:type refnum :follow-to "some.pdf").

Even more complicated examples occur after Doe's famous article from
[cite: @Doe99] %%(:type date-only).

And in [cite: @Doe2000] %%(:attr_latex (:format-string
"\citeyear{%KEY}") :attr_html (:only-fields (month year))), Doe
finally realized that arbitrary complexity was a powerful but
double-edged sword.

@_aParticularlyUGLYkey:is-this-one %%(:overlay "Nice Display")
#+END_QUOTE 

** Grammar
This section formally documents the syntax of citations discussed
above.  

To represent the syntax of citations, we need a category of /citation/
objects, which require the following properties (the names here are not
important and could be changed):
  - is-parenthetical (boolean; nil means is in-text)
  - common-prefix (text)
  - common-suffix (text)
  - references (list)
  - extra-info (list)

Each reference in the list of references should be a plist with the
following properties:
  - prefix (text)
  - suffix (text)
  - key (string)
  - is-parenthesized (boolean; t means key was parenthesized; only
    significant for the first reference in a citation)
  - suppress-author (boolean; t means author name should not be output)
  - is-full (boolean; t means a full bibliography entry should be
    output in-place) 

The category of citations has the following grammar:
  - A CITATION is a PARENTHETICAL-CITATION or an IN-TEXT citation.
  - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
    CITATION-LIST whose first individual INDIVIDUAL-REFERENCE is a
    PARENTHESIZED-KEY
  - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
    CITATION-LIST whose first INDIVIDUAL-REFERENCE is a BARE-KEY.
  - A SIMPLE-PARENTHETICAL is a KEY immediately surrounded by square
    brackets, optionally followed by an EXTRA-INFO clause.
  - A SIMPLE-IN-TEXT is a BARE-KEY, optionally followed by an
    EXTRA-INFO clause
  - A CITATION-LIST has the format
       [cite: PREFIX; INDIVIDUAL-REFERENCE; ... INDIVIDUAL-REFERENCE; SUFFIX] EXTRA-INFO
    where the initial PREFIX, final SUFFIX, and EXTRA-INFO clause are
    optional.  At least one INDIVIDUAL-REFERENCE must be present. 
  - An INDIVIDUAL-REFERENCE has the format: 
       PREFIX KEY-MAYBE-PARENS SUFFIX
    The KEY-MAYBE-PARENS is obligatory, and the prefix and suffix
    are optional.
  - A KEY-MAYBE-PARENS is either a BARE-KEY or PARENTHESIZED-KEY
  - A BARE-KEY is a KEY with immediately-preceding whitespace
  - A PARENTHESIZED-KEY is a KEY immediately surrounded by `(' and `)'.
  - A KEY optionally begins with `-', and obligatorily contains `@' or
    `&' followed by a string of characters which begins with a letter
    or `_', and may contain alphanumeric characters and the following
    internal punctuation characters:
       :.#$%&-+?<>~/
  - A PREFIX or SUFFIX is arbitrary text (except `;', `]', and
    KEY-MAYBE-PARENs) which may contain only the following Org
    objects:
    - bold
    - code
    - entity
    - italic
    - latex-fragment
    - line-break
    - strike-through
    - subscript
    - superscript
    - underline
    - superscript
    (Note that this list could be extended somewhat if necessary.)
  - An EXTRA-INFO clause consists of data not specified by this
    grammar, in between `%%(' and `)'

** Outstanding issues
It seems to me that there are potential problems with the above
proposal in a number of areas, but I cannot tell how serious they are,
or what changes (if any) should be made to solve them.  I don't
pretend that this is an exhaustive list:
  1) *Nesting.*  I have favored LaTeX compatibility for in-text
     citations with multiple references; but this means there is no
     way to `nest' citations.  Thus, there is no way to express (in
     the main syntax) what Pandoc expresses as:
        @Doe99 [p. 34; see also @DoeRoe2000]
     which renders like:
        Doe (1999, p. 34; see also Doe and Roe 2000)
     Instead, since a citation is in-text or parenthetical as a whole,
     the equivalent in the above syntax
        [cite: @Doe99 p. 34; see also @DoeRoe2000]
     should render like:
	Doe (1999, p. 34), see also Doe and Roe (2000).
     I am not certain if Pandoc-like output is important in this case.
     The few people who commented on this said that it was not. 
  2) *Limitations on prefixes and suffixes.*  There may be legitimate
     uses of `@', `;', `]', etc. inside prefix or suffix text that the
     above syntax does not allow.  Examples might include:
     - use of semi-colons as part of the prefix/suffix text
     - footnotes, links, or timestamps inside a prefix/suffix
     I am not certain how important these cases are.  If they are
     important, some of them might be able to be worked around with
     entities.
  3) *Edge cases.* The above syntax may make it possible to express
     things that don't make sense, or would be too difficult to
     export.  The only one I can think of is that it is possible to
     mix `@'-style and `&'-style keys in the same citation.  I am not
     sure if this should be forbidden; it may sometimes make sense.
     It may also be possible to express things that external tools,
     such as citeproc-js, don't know how to process.  I do not have a
     good sense of what, if anything, falls into that category, and
     what should be done about it.
  4) *Citation commands.*  Rather than introduce an explicit
     representation for different citation commands/types, I have used
     different parts of the syntax to express the common distinctions
     that people mentioned.  I suggest that, for now, anything beyond
     these basic distinctions be left to the user-extension syntax.
     However, if it becomes clear in the future that there is a need
     to add a representation for a command to the main syntax, there
     is a natural place to do so: immediately after the `cite:' tag
     (as Nicolas suggested).

Also, I have not said anything in this proposal to address how other
document metadata should be represented, which has not been discussed
much on the list.  I think this should be discussed separately.
      

Re: Citation syntax: a revised proposal

[Richard Lawrence]

...and of course, immediately sending, I noticed a small problem in the
grammar:

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

>   - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
>     CITATION-LIST whose first INDIVIDUAL-REFERENCE is a
>     PARENTHESIZED-KEY
>   - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
>     CITATION-LIST whose first INDIVIDUAL-REFERENCE is a BARE-KEY.

In both of these clauses, `INDIVIDUAL-REFERENCE is' should be
`INDIVIDUAL-REFERENCE contains', so that they read:

  - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
    CITATION-LIST whose first INDIVIDUAL-REFERENCE contains a
    PARENTHESIZED-KEY
  - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
    CITATION-LIST whose first INDIVIDUAL-REFERENCE contains a BARE-KEY.
    
Also, I wanted to mention that people may want to start by reading the
examples in the proposal, which are under the heading ``Examples of main
citation syntax''.  (There are further, unofficial examples under the
heading ``Syntax for extensions''.)

Best,
Richard

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I welcome feedback, comments, criticisms, and objections on any point.
> However, since we've already had a long discussion about this, I
> respectfully request that we try to keep this thread focused.  To that
> end, I suggest:
>
>   1) If you have criticisms or objections, please try to indicate
>      whether you think they are `substantive' (e.g., you see a problem
>      that would prevent you from using this syntax, or prevent Org from
>      implementing it) or not (e.g., you would prefer a slightly
>      different but equivalent way of expressing something).
>
>   2) If you wish to express an opinion about the proposal without
>      offering further comments, let us know by just replying with +1
>      (meaning you'd like to see this syntax, or something reasonably
>      similar to it, be adopted), 0, or -1 (meaning you'd prefer not to
>      see this syntax or anything similar to it adopted).

0

A syntax that relegates citation commands to an extension that might not
export properly in future versions of Org mode isn't useful in my work.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Tom,

tsd@tsdye.com (Thomas S. Dye) writes:

> 0
>
> A syntax that relegates citation commands to an extension that might not
> export properly in future versions of Org mode isn't useful in my work.

Sorry to disappoint!  

I tried really hard to represent in the [cite: ...] syntax all the
particular cases of citation commands that people said were important,
including the ones you suggested.  But maybe I missed something, or
maybe you have other suggestions.  If so, I am happy to try again.

It is true that the [cite: ...] syntax does not support specifying an
arbitrary citation command.  The point of this is to keep the number of
features that Org would initially commit to making work in all backends
at a reasonable level, while still encompassing all the important cases.

If you need arbitrary extensions, that is what the %%(...) part of the
syntax is for.  Let me try to clarify what I was thinking about this,
since I didn't say much about it in the proposal.  I don't know if this
will address your concern, but I hope it will at least partly do so.

Basically, I'm thinking of the %%(...) part of the syntax as a kind of
playground; it represents `everything we haven't decided about yet', and
is meant to encompass both user-extensions and backend-specific
extensions.  

If we treat %%(...) as an arbitrary s-expression, then it should already
be flexible enough to allow people who need even the most esoteric
citation commands to implement them via export filters.  But like all
user extensions that execute arbitrary code on arbitrary data, Org is
not in a position to *guarantee* that it will work forevermore.  (For
example, your filter might call an Org function that could one day be
removed or renamed, at which point it will break.)  Hence this warning:

> *At least for now*, any information supplied this way is /strictly the
> user's responsibility/ to interpret (e.g., using an export filter).
> This means that citations that have information like this are not
> portable and might not be exported correctly:
>  - in other users' setups
>  - by particular backends
>  - by future versions of Org

Note that exactly the same warning applies to links with custom types
and similar user extensions, even if the manual is less explicit about
it.  In practice, I expect that the information in %%(...) clauses will
be at least as stable and portable as information in custom links.

(By the way, just to be super clear: `correctly' in the above warning
means `handling the extra information correctly'.  I didn't mean to
imply that even the default export behavior could go awry.)

Eventually, we may find that lots of people have similar needs that they
are dealing with in %%(...) syntax.  At that point, we may want to adopt
stricter conventions for some parts of this syntax.  For example, I
assume we might eventually want a convention that the LaTeX backend will
correctly handle properties that look like:

%%(... :attr_latex (:command "\somecitecommand[%PRE][%POST]{%KEY}") ...) 

Once such a convention is discussed and adopted, Org could then
officially support it, and the above warning would no longer apply,
except the part about other backends.  

Likewise, if we find cases that are important enough that they need to
work on all backends, representations for those cases could be added to
the [cite: ...] part of the syntax.

So really, the idea I had in splitting the syntax into [cite: ...] and
%%(...) was to give us a good starting point that can be built on in the
future -- not to relegate important features to the dustbin.  Sorry that
I did not make that clearer in the proposal!

Best,
Richard

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Hi Richard,

Thanks for your thoughtful response.  I had indeed misunderstood the bit
about not being able to export in future versions of Org mode.  Thanks
for the clarification.

However, I'm still 0 because our goals are different.

I want a syntax that recognizes arbitrary citation commands because I
write in Org mode for publication.  You want a syntax that recognizes a
few commands that it might be possible to support in Org mode backends,
some of which are tied loosely, if at all, to publication.  Yours might
be a noble goal that many Org mode users will find useful (I hope it
is!), but I don't think it is (or will be) a syntax useful in my work,
for the following reasons:

1) It is easier for me to have the citation command in one place.  The
decision to represent selected aspects of the citation command in the
syntax and other parts in extensions means that I'd have to learn the
syntax and then remember which aspects were chosen for representation
and which I'd need to develop through extensions of my own.  This is a
lot more work than I do now to get exactly what I want through links.
I'm keen to simplify the authoring process, not make it more complex.

2) Treating footnote citations differently from author-date citations is
a non-starter for me.  When Science turns me away and the editor
suggests that my rant is well suited for another journal, one that
happens to use author-date citations, I'll just search all my citation
links and replace footcite with parencite before exporting the rant to
the suggested journal.  IIUC, with the official Org mode syntax, I'd be
faced with the tedious process of cutting and pasting footnote text back
into the document body.

Also, I'm 0 because there is no need to satisfy me with the official
citation syntax.  I'm able to achieve my goals fully in Org mode without
it.

All the best,
Tom

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi Tom,
>
> tsd@tsdye.com (Thomas S. Dye) writes:
>
>> 0
>>
>> A syntax that relegates citation commands to an extension that might not
>> export properly in future versions of Org mode isn't useful in my work.
>
> Sorry to disappoint!  
>
> I tried really hard to represent in the [cite: ...] syntax all the
> particular cases of citation commands that people said were important,
> including the ones you suggested.  But maybe I missed something, or
> maybe you have other suggestions.  If so, I am happy to try again.
>
> It is true that the [cite: ...] syntax does not support specifying an
> arbitrary citation command.  The point of this is to keep the number of
> features that Org would initially commit to making work in all backends
> at a reasonable level, while still encompassing all the important cases.
>
> If you need arbitrary extensions, that is what the %%(...) part of the
> syntax is for.  Let me try to clarify what I was thinking about this,
> since I didn't say much about it in the proposal.  I don't know if this
> will address your concern, but I hope it will at least partly do so.
>
> Basically, I'm thinking of the %%(...) part of the syntax as a kind of
> playground; it represents `everything we haven't decided about yet', and
> is meant to encompass both user-extensions and backend-specific
> extensions.  
>
> If we treat %%(...) as an arbitrary s-expression, then it should already
> be flexible enough to allow people who need even the most esoteric
> citation commands to implement them via export filters.  But like all
> user extensions that execute arbitrary code on arbitrary data, Org is
> not in a position to *guarantee* that it will work forevermore.  (For
> example, your filter might call an Org function that could one day be
> removed or renamed, at which point it will break.)  Hence this warning:
>
>> *At least for now*, any information supplied this way is /strictly the
>> user's responsibility/ to interpret (e.g., using an export filter).
>> This means that citations that have information like this are not
>> portable and might not be exported correctly:
>>  - in other users' setups
>>  - by particular backends
>>  - by future versions of Org
>
> Note that exactly the same warning applies to links with custom types
> and similar user extensions, even if the manual is less explicit about
> it.  In practice, I expect that the information in %%(...) clauses will
> be at least as stable and portable as information in custom links.
>
> (By the way, just to be super clear: `correctly' in the above warning
> means `handling the extra information correctly'.  I didn't mean to
> imply that even the default export behavior could go awry.)
>
> Eventually, we may find that lots of people have similar needs that they
> are dealing with in %%(...) syntax.  At that point, we may want to adopt
> stricter conventions for some parts of this syntax.  For example, I
> assume we might eventually want a convention that the LaTeX backend will
> correctly handle properties that look like:
>
> %%(... :attr_latex (:command "\somecitecommand[%PRE][%POST]{%KEY}") ...) 
>
> Once such a convention is discussed and adopted, Org could then
> officially support it, and the above warning would no longer apply,
> except the part about other backends.  
>
> Likewise, if we find cases that are important enough that they need to
> work on all backends, representations for those cases could be added to
> the [cite: ...] part of the syntax.
>
> So really, the idea I had in splitting the syntax into [cite: ...] and
> %%(...) was to give us a good starting point that can be built on in the
> future -- not to relegate important features to the dustbin.  Sorry that
> I did not make that clearer in the proposal!
>
> Best,
> Richard
>
>
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1820 bytes --]

In
On Feb 15, 2015 2:43 PM, "Thomas S. Dye" <tsd@tsdye.com> wrote:
>
>
> 1) It is easier for me to have the citation command in one place.  The
> decision to represent selected aspects of the citation command in the
> syntax and other parts in extensions means that I'd have to learn the
> syntax and then remember which aspects were chosen for representation
> and which I'd need to develop through extensions of my own.  This is a
> lot more work than I do now to get exactly what I want through links.
> I'm keen to simplify the authoring process, not make it more complex.
>
> 2) Treating footnote citations differently from author-date citations is
> a non-starter for me.  When Science turns me away and the editor
> suggests that my rant is well suited for another journal, one that
> happens to use author-date citations, I'll just search all my citation
> links and replace footcite with parencite before exporting the rant to
> the suggested journal.  IIUC, with the official Org mode syntax, I'd be
> faced with the tedious process of cutting and pasting footnote text back
> into the document body.

I am generally much more positive than Thomas,  being,  for the most part,
ecstatic at the thought of a built-in citation syntax which will make
citations in org workable for bumbling nonprogrammers like myself.

However,  I agree that the distinction between parenthetical and footnotes
citations is unhelpful for me.  Whenever I switch between Chicago and APA,
for instance,  zotero converts my footnotes to parenthetical expressions.
To me this seems an essential feature.

Thanks Richard and everyone else for the continued hard work!

Matt

Matt

> Also, I'm 0 because there is no need to satisfy me with the official
> citation syntax.  I'm able to achieve my goals fully in Org mode without
> it.
>
>

[-- Attachment #2: Type: text/html, Size: 2229 bytes --]

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Hello,

Matt Price <moptop99@gmail.com> writes:

> However,  I agree that the distinction between parenthetical and footnotes
> citations is unhelpful for me.  Whenever I switch between Chicago and APA,
> for instance,  zotero converts my footnotes to parenthetical expressions.
> To me this seems an essential feature.

There is no distinction between parenthetical and footnotes citations in
the syntax. Richard showed that citations can be included in footnote
definitions, but that's about it.

AFAICT, nothing prevents you from writing

  A citation[cite ... @key ...]{latex :type footnotecite} within
  a paragraph.

and let "ox-latex" turn it into a footnote citation for you.


Regards,

-- 
Nicolas Goaziou

Re: Citation syntax: a revised proposal

[Rasmus]

Matt Price <moptop99@gmail.com> writes:

> I am generally much more positive than Thomas,  being,  for the most part,
> ecstatic at the thought of a built-in citation syntax which will make
> citations in org workable for bumbling nonprogrammers like myself.
>
> However,  I agree that the distinction between parenthetical and footnotes
> citations is unhelpful for me.  Whenever I switch between Chicago and APA,
> for instance,  zotero converts my footnotes to parenthetical expressions.
> To me this seems an essential feature.

Maybe Richard stepped back on this, but I think at some point the
consensus was that you'd have two main cite types, namely @· and [@·]
which can be mapped to whatever, e.g. @· → textcite by default and [@·] →
\parencite.

Then the switch would be.

#+INLINE-CITE: textcite
#+SQUARE-CITE: parencite

—Rasmus

-- 
Governments should be afraid of their people

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Tom,

tsd@tsdye.com (Thomas S. Dye) writes:

> I want a syntax that recognizes arbitrary citation commands because I
> write in Org mode for publication.  You want a syntax that recognizes a
> few commands that it might be possible to support in Org mode backends,
> some of which are tied loosely, if at all, to publication.  Yours might
> be a noble goal that many Org mode users will find useful (I hope it
> is!), but I don't think it is (or will be) a syntax useful in my work,
> for the following reasons:
>
> 1) It is easier for me to have the citation command in one place.  The
> decision to represent selected aspects of the citation command in the
> syntax and other parts in extensions means that I'd have to learn the
> syntax and then remember which aspects were chosen for representation
> and which I'd need to develop through extensions of my own.  This is a
> lot more work than I do now to get exactly what I want through links.
> I'm keen to simplify the authoring process, not make it more complex.
>
> 2) Treating footnote citations differently from author-date citations is
> a non-starter for me.  When Science turns me away and the editor
> suggests that my rant is well suited for another journal, one that
> happens to use author-date citations, I'll just search all my citation
> links and replace footcite with parencite before exporting the rant to
> the suggested journal.  IIUC, with the official Org mode syntax, I'd be
> faced with the tedious process of cutting and pasting footnote text back
> into the document body.

I do think it is important to support these kinds of uses, and I think
it would be a shame if the official Org syntax did not make them
relatively straightforward.  You are surely not the only person using
Org to prepare documents for publication, and I'm sure this kind of
per-journal `refactoring' is common and important to make easy.

(You're right that our goals differ to some extent.  I am still in grad
school.  Preparing documents for academic publication is a privilege I
hope to have one day; but I am not presently one of the people using Org
for this on a regular basis, though I hope I can in the future.  One
reason I am concerned to have a citation syntax that can be exported by
other backends is this: I am anxious that, unless I can also export my
dissertation to HTML, the final document may never be read by anyone
except backup programs on the library servers.  Another, more serious
reason is that I work in a field where some journals do not accept LaTeX
submissions, or disprefer them; so having some citation support in ODT
export is important.)

I *think* it should be possible to do the kinds of things you've
described here using the syntax I proposed, but I may not understand
everything you'd like to do.  If not, let's figure out what the other
things are, and how to accommodate them.

Basically, I think you could ignore the distinctions that the [cite:
...] syntax is capable of expressing, and just write all your citations
like:

  [cite: See @Doe99 for more on this point.] %%(:type footnoted)

or, in the syntax Nicolas proposed, something like:

  [cite: See @Doe99 for more on this point.]{:type footnoted}

You would then use an export filter to transform citations with this
:type into the appropriate command, something along the lines of:

(defun footnoted-citation (citation backend info)
  (let ((type (get-citation-type citation)
        (pre (get-citation-prefix citation))
        (post (get-citation-suffix citation))
        (key (get-citation-key citation)))
     (when (and (org-export-derived-backend-p backend 'latex)
                (eq type 'footnoted))
       (format "\footcite[%s][%s]{%s}" pre post key))))

(It would be more complicated than this in the general case, since a
citation can contain more than one reference as well as common prefix
and suffix text, but hopefully that illustrates the idea.)

Then, when Science sends you elsewhere, you can just query-replace
":type footnoted" with ":type author-date", or whatever the appropriate
type for the new journal is, which will have a different export filter
(or a different clause in the same filter).

That is more work than letting Org export citations for you, because it
means manually processing every :type you use.  But maybe it is about
the same amount of work as what you are doing now with custom links.

This way, although you wouldn't be relying much on the default export
behavior of citations, you could still get the other advantages of
having them represented in Org syntax.  Those are things like having
prefix/suffix text stand in a more readable relation to the key, having
Org parse the different parts out for you, and having individual keys be
clickable so you can look up the reference in your reference database or
find an associated PDF.

Would that be sufficient?  And are there other kinds of situation where
you don't think the proposed syntax would work well?

Best,
Richard

Re: Citation syntax: a revised proposal

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Basically, I think you could ignore the distinctions that the [cite:
> ...] syntax is capable of expressing, and just write all your citations
> like:
>
>   [cite: See @Doe99 for more on this point.] %%(:type footnoted)
>
> or, in the syntax Nicolas proposed, something like:
>
>   [cite: See @Doe99 for more on this point.]{:type footnoted}
>
> You would then use an export filter to transform citations with this
> :type into the appropriate command, something along the lines of:

This seems backward and wrong /to me/.  Why not global options?  I
actually think we "agreed" on this earlier.

#+cite: text-cite
#+(cite): parenthesis-cite 

Or

#+cite: numeric
#+(cite): footnote-cite

This is why I asked Tom earlier if he normally uses more than two citation
TYPES.  Presumably citeauthor and citeyear works irrespective of the main
citation type.  So I still think that having two TYPES handy will meet
most requirements.

—Rasmus
 
-- 
m-mm-mmm-mmmm bacon!

Re: Citation syntax: a revised proposal

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1441 bytes --]

On Feb 17, 2015 1:12 PM, "Rasmus" <rasmus@gmx.us> wrote:
>
> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
> > Basically, I think you could ignore the distinctions that the [cite:
> > ...] syntax is capable of expressing, and just write all your citations
> > like:
> >
> >   [cite: See @Doe99 for more on this point.] %%(:type footnoted)
> >
> > or, in the syntax Nicolas proposed, something like:
> >
> >   [cite: See @Doe99 for more on this point.]{:type footnoted}
> >
> > You would then use an export filter to transform citations with this
> > :type into the appropriate command, something along the lines of:
>
> This seems backward and wrong /to me/.  Why not global options?  I
> actually think we "agreed" on this earlier.
>
> #+cite: text-cite
> #+(cite): parenthesis-cite
>
> Or
>
> #+cite: numeric
> #+(cite): footnote-cite
>
> This is why I asked Tom earlier if he normally uses more than two citation
> TYPES.  Presumably citeauthor and citeyear works irrespective of the main
> citation type.  So I still think that having two TYPES handy will meet
> most requirements.

This is clearly preferable.  Using zotero in odt ,  rtf ,  or docx,  you
would normally just set a citation style and let zotero take care of the
reformatting.  Rasmus's solution is almost as convenient as that,  and
presumably backend-independent.
>
> —Rasmus
>
> --
> m-mm-mmm-mmmm bacon!
>
>

[-- Attachment #2: Type: text/html, Size: 1921 bytes --]

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Rasmus,

Rasmus <rasmus@gmx.us> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Basically, I think you could ignore the distinctions that the [cite:
>> ...] syntax is capable of expressing, and just write all your citations
>> like:
>>
>>   [cite: See @Doe99 for more on this point.] %%(:type footnoted)
>>
>> You would then use an export filter to transform citations with this
>> :type into the appropriate command, something along the lines of:
>
> This seems backward and wrong /to me/.  Why not global options?  I
> actually think we "agreed" on this earlier.
>
> #+cite: text-cite
> #+(cite): parenthesis-cite 
>
> Or
>
> #+cite: numeric
> #+(cite): footnote-cite
>
> This is why I asked Tom earlier if he normally uses more than two citation
> TYPES.  Presumably citeauthor and citeyear works irrespective of the main
> citation type.  So I still think that having two TYPES handy will meet
> most requirements.

That may be so.  But surely it is at least conceivable that more than
three types will be used in a document (say, mostly textcite and
parencite, with a few footfullcites sprinkled in); and Tom's concern was
that he doesn't want to have to worry about which parts of the syntax of
a citation he has to change to effect a change from one type to another.

So, if this approach doesn't seem right to you, don't do it that way.
The point is just that Tom's style of working is possible to achieve in
this syntax.  Hopefully, yours is too, though it may be quite different.

Best,
Richard

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Hi Richard,

Thanks for your thoughtful responses and your work on the citation
syntax.  My "author" concerns have been addressed in this thread and I
look forward to development now.  I'm +1 and optimistic about the switch
from home-brew links to citations in my Org mode work.

Thanks for your patience as I digested your proposals.  Let me know if
you think I can help in some way.

All the best,
Tom

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi Tom,
>
> tsd@tsdye.com (Thomas S. Dye) writes:
>
>> I want a syntax that recognizes arbitrary citation commands because I
>> write in Org mode for publication.  You want a syntax that recognizes a
>> few commands that it might be possible to support in Org mode backends,
>> some of which are tied loosely, if at all, to publication.  Yours might
>> be a noble goal that many Org mode users will find useful (I hope it
>> is!), but I don't think it is (or will be) a syntax useful in my work,
>> for the following reasons:
>>
>> 1) It is easier for me to have the citation command in one place.  The
>> decision to represent selected aspects of the citation command in the
>> syntax and other parts in extensions means that I'd have to learn the
>> syntax and then remember which aspects were chosen for representation
>> and which I'd need to develop through extensions of my own.  This is a
>> lot more work than I do now to get exactly what I want through links.
>> I'm keen to simplify the authoring process, not make it more complex.
>>
>> 2) Treating footnote citations differently from author-date citations is
>> a non-starter for me.  When Science turns me away and the editor
>> suggests that my rant is well suited for another journal, one that
>> happens to use author-date citations, I'll just search all my citation
>> links and replace footcite with parencite before exporting the rant to
>> the suggested journal.  IIUC, with the official Org mode syntax, I'd be
>> faced with the tedious process of cutting and pasting footnote text back
>> into the document body.
>
> I do think it is important to support these kinds of uses, and I think
> it would be a shame if the official Org syntax did not make them
> relatively straightforward.  You are surely not the only person using
> Org to prepare documents for publication, and I'm sure this kind of
> per-journal `refactoring' is common and important to make easy.
>
> (You're right that our goals differ to some extent.  I am still in grad
> school.  Preparing documents for academic publication is a privilege I
> hope to have one day; but I am not presently one of the people using Org
> for this on a regular basis, though I hope I can in the future.  One
> reason I am concerned to have a citation syntax that can be exported by
> other backends is this: I am anxious that, unless I can also export my
> dissertation to HTML, the final document may never be read by anyone
> except backup programs on the library servers.  Another, more serious
> reason is that I work in a field where some journals do not accept LaTeX
> submissions, or disprefer them; so having some citation support in ODT
> export is important.)
>
> I *think* it should be possible to do the kinds of things you've
> described here using the syntax I proposed, but I may not understand
> everything you'd like to do.  If not, let's figure out what the other
> things are, and how to accommodate them.
>
> Basically, I think you could ignore the distinctions that the [cite:
> ...] syntax is capable of expressing, and just write all your citations
> like:
>
>   [cite: See @Doe99 for more on this point.] %%(:type footnoted)
>
> or, in the syntax Nicolas proposed, something like:
>
>   [cite: See @Doe99 for more on this point.]{:type footnoted}
>
> You would then use an export filter to transform citations with this
> :type into the appropriate command, something along the lines of:
>
> (defun footnoted-citation (citation backend info)
>   (let ((type (get-citation-type citation)
>         (pre (get-citation-prefix citation))
>         (post (get-citation-suffix citation))
>         (key (get-citation-key citation)))
>      (when (and (org-export-derived-backend-p backend 'latex)
>                 (eq type 'footnoted))
>        (format "\footcite[%s][%s]{%s}" pre post key))))
>
> (It would be more complicated than this in the general case, since a
> citation can contain more than one reference as well as common prefix
> and suffix text, but hopefully that illustrates the idea.)
>
> Then, when Science sends you elsewhere, you can just query-replace
> ":type footnoted" with ":type author-date", or whatever the appropriate
> type for the new journal is, which will have a different export filter
> (or a different clause in the same filter).
>
> That is more work than letting Org export citations for you, because it
> means manually processing every :type you use.  But maybe it is about
> the same amount of work as what you are doing now with custom links.
>
> This way, although you wouldn't be relying much on the default export
> behavior of citations, you could still get the other advantages of
> having them represented in Org syntax.  Those are things like having
> prefix/suffix text stand in a more readable relation to the key, having
> Org parse the different parts out for you, and having individual keys be
> clickable so you can look up the reference in your reference database or
> find an associated PDF.
>
> Would that be sufficient?  And are there other kinds of situation where
> you don't think the proposed syntax would work well?
>
> Best,
> Richard
>
>
>

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Tom,

tsd@tsdye.com (Thomas S. Dye) writes:

> Thanks for your thoughtful responses and your work on the citation
> syntax.  My "author" concerns have been addressed in this thread and I
> look forward to development now.  I'm +1 and optimistic about the switch
> from home-brew links to citations in my Org mode work.

Great!  I'm really glad to hear that.

Actually, your post has convinced me that it may be worth allowing some
explicit name for a type in the [cite: ...] part of the syntax, although
I am still leery about what this would mean for non-LaTeX backends.  I
did not appreciate before that switching from one type to another is
something you probably want to be able to do really easily, like with
query-replace, even if you are making use of the other parts of the
syntax to express distinctions like in-text vs. parenthetical citations.

So, two questions for the group:

1) Is it worth allowing a name for a user-defined type in the [cite: ...]
part, or is it OK to confine user-defined types to the second part
(like: [cite: ...] %%(:type foo) or [cite: ...]{:type foo})?

2) If a user-defined type can go in the [cite: ...] part, where should
it go?  Nicolas has suggested:

  [cite:subtype ...]

or 

  [cite:subtype: ...]

I would personally (aesthetically, don't ask me why) prefer:

  [cite/subtype: ...]

or

  [cite|subtype: ...]

But maybe there are other options I haven't thought of.  

Best,
Richard

Re: Citation syntax: a revised proposal

[Stefan Nobis]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> 1) Is it worth allowing a name for a user-defined type in the [cite:
> ...] part, or is it OK to confine user-defined types to the second
> part (like: [cite: ...] %%(:type foo) or [cite: ...]{:type foo})?

IMHO it is better to have such an important part of the citation in a
prominent position, therefore I'm in favour of Nicolas suggestion of

    [cite:subtype: ...]{backend options}

with the four variations for "cite" (i.e. "[cite:...]", "[Cite:...]",
"[(cite):...]", and "[(Cite):...]").

The drawback is that now subtype is hard or even impossible to vary
for different backends. Therefore I would suggest that either org has
to define the allowed values of subtype or else we should define that
subtype has to be handled by the user (e.g. for use in private filter
functions) and is out of the scope of org (maybe this would be a good
place of extensions like org-ref to plug in their machinery).

> 2) If a user-defined type can go in the [cite: ...] part, where should
> it go?

>   [cite:subtype ...]
>   [cite:subtype: ...]
>   [cite/subtype: ...]
>   [cite|subtype: ...]

I favor [cite:subtype: ...] a very tiny bit over the other variants.

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[Eric S Fraga]

On Tuesday, 17 Feb 2015 at 20:03, Richard Lawrence wrote:

[...]

> I would personally (aesthetically, don't ask me why) prefer:
>
>   [cite/subtype: ...]

as do I as it mirrows mime-types, e.g. text/plain, text/x-org-mode, ...
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.50.1, Org release_8.3beta-843-ga5f1a3.dirty

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Actually, your post has convinced me that it may be worth allowing some
> explicit name for a type in the [cite: ...] part of the syntax, although
> I am still leery about what this would mean for non-LaTeX backends.

Each back-end can decide to use it or simply ignore it. Also [cite:...]
should be equivalent to [cite:default: ...], for some value of "default"
decided by the target back-end.

> I did not appreciate before that switching from one type to another is
> something you probably want to be able to do really easily, like with
> query-replace, even if you are making use of the other parts of the
> syntax to express distinctions like in-text vs. parenthetical
> citations.
>
> So, two questions for the group:
>
> 1) Is it worth allowing a name for a user-defined type in the [cite: ...]
> part, or is it OK to confine user-defined types to the second part
> (like: [cite: ...] %%(:type foo) or [cite: ...]{:type foo})?

Expecting subtype in the header doesn't add a limitation to pre or post
text.

Moreover [cite: ...]{...} syntax really makes sense if it is the
equivalent to #+attr_... keywords, so we can generalize it to links. As
a consequence, {...} should include a reference to back-end. E.g.,

  [cite:...]{latex :color pink}

> 2) If a user-defined type can go in the [cite: ...] part, where should
> it go?  Nicolas has suggested:
>
>   [cite:subtype ...]
>
> or 
>
>   [cite:subtype: ...]
>
> I would personally (aesthetically, don't ask me why) prefer:
>
>   [cite/subtype: ...]
>
> or
>
>   [cite|subtype: ...]
>
> But maybe there are other options I haven't thought of.

I'm fine with any of these, although the latter looks less nice to me.


Regards,

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas and all,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Actually, your post has convinced me that it may be worth allowing some
>> explicit name for a type in the [cite: ...] part of the syntax, although
>> I am still leery about what this would mean for non-LaTeX backends.
>
> Each back-end can decide to use it or simply ignore it. Also [cite:...]
> should be equivalent to [cite:default: ...], for some value of "default"
> decided by the target back-end.

I know that this is technically easy to handle from the backend's
perspective.  But I have a concern related to Stefan's: 

Stefan Nobis <stefan-ml@snobis.de> writes:

> The drawback is that now subtype is hard or even impossible to vary
> for different backends. Therefore I would suggest that either org has
> to define the allowed values of subtype or else we should define that
> subtype has to be handled by the user (e.g. for use in private filter
> functions) and is out of the scope of org (maybe this would be a good
> place of extensions like org-ref to plug in their machinery).

Basically, I am worried that neither of these options is very good.  

If the subtype is interpreted by backends, I think it will become hard
to use as an author.  If each backend treats different subtypes as
significant, but some backends overlap in which subtypes they accept,
then as an author you have to keep track of which subtypes work in which
backends, and which don't, and whether the default behavior is OK for
your citations in backends you care about.  If you are targeting more
than one backend, that could mean a lot of trips to the manual.  And if
user-supplied subtypes are also permissible, you have to keep track of
the distinction between `official' subtypes and your own.

On the other hand, if the subtype is just a user-supplied label, which
can be interpreted by a filter but which backends will otherwise ignore,
things are nice and simple.  You just don't use a subtype unless you are
also supplying a way to interpret it on every backend that you care
about.  But that kind of situation is exactly what the extra-info part
of the syntax is for; in that case,

  [cite/my-subtype: ...] 

would just be an exceptional way of writing

  [cite: ...]{:type my-subtype}

or whatever.  I'm not totally convinced yet that the convenience of the
former is worth blurring the line between `stuff that definitely works
on all backends' and `stuff that might not work on all backends'.

> Moreover [cite: ...]{...} syntax really makes sense if it is the
> equivalent to #+attr_... keywords, so we can generalize it to links. As
> a consequence, {...} should include a reference to back-end. E.g.,
>
>   [cite:...]{latex :color pink}

We have already seen a couple of examples in this thread of properties
that one might want to specify in a backend-agnostic way:
  - special-case capitalization
  - user-defined type/command/label/etc.

Other things one might want to do include:
  - indicating when a citation should be placed in a footnote 
  - extracting arbitrary bibliographic field(s)
  - providing an overlay string for displaying complicated/ugly citations 

And I think there is no reason, at the moment, to expect that these are
the only possible uses for arbitrary backend-agnostic properties that a
user can interpret, either via export filters or via in-buffer
customizations.

So I suggest that we also allow backend-agnostic properties here in
addition to backend-specific ones.  My preference would be that we just
let these backend-agnostic properties occur at the front, and separate
the backend-specific ones with labeled pipes, like you suggested
earlier:

  [cite: ...]{:my-key1 my-val1 ... |latex: :lkey1 lval1 ... |html: ...} 

I'm OK with disallowing the backend-agnostic properties for other types
of objects if that is necessary, though I don't see how it would hurt to
allow them, as long as it's not Org's responsibility to interpret them.
 
Best,
Richard

Re: Citation syntax: a revised proposal

[Samuel Wales]

i have a silly question: whatever syntax we choose, will it be able to
be used, in the future, for new org features nobody has thought of
yet, that are unrelated to citations?

my preference is to forestall future syntax creep, enhance
consistency, and amortize the effort in supporting a syntax [including
e.g. supporting quoting, escaping, hiding, labeling, nesting,
debugging, parsing, searching, etc.].

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  And
ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: Citation syntax: a revised proposal

[Samuel Wales]

[disclaimer: i do not currently use citations, so i have no stake in
citation syntax per se, just a tendency to ask highly generic silly
questions.]

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Samuel,

2015ko otsailak 18an, Samuel Wales-ek idatzi zuen:
>
> i have a silly question: whatever syntax we choose, will it be able to
> be used, in the future, for new org features nobody has thought of
> yet, that are unrelated to citations?

Do you mean how the syntax and implementation of links was used to
support a new org feature completely unrelated to web URLs, namely
citations?  ;)

[It’s entirely unclear from your message, at least to me, whether you
think these future features would be a good thing or a bad thing.]

--
Aaron Ecay

Re: Citation syntax: a revised proposal

[Samuel Wales]

hi aaron,

On 2/18/15, Aaron Ecay <aaronecay@gmail.com> wrote:
> Do you mean how the syntax and implementation of links was used to
> support a new org feature completely unrelated to web URLs, namely
> citations?  ;)

heh heh.  :]  that applies to the outer syntax only.  i mean, for
example, plists vs. something less general.

> [It’s entirely unclear from your message, at least to me, whether you
> think these future features would be a good thing or a bad thing.]

they haven't happened yet, so i don't have an opinion on them.  :]


samuel

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  And
ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: Citation syntax: a revised proposal

[John Kitchin]

I was thinking the same thing as Aaron ;) What could one do with a
"link-like" object with arbitrary attributes/properties... Hmm... maybe
this idea:
http://kitchingroup.cheme.cmu.edu/blog/2015/02/05/Extending-the-org-mode-link-syntax-with-attributes/
without the kludgy parsing? It will be so confusing to future people why
they have to have a cite: in them ;)

Aaron Ecay writes:

> Hi Samuel,
>
> 2015ko otsailak 18an, Samuel Wales-ek idatzi zuen:
>>
>> i have a silly question: whatever syntax we choose, will it be able to
>> be used, in the future, for new org features nobody has thought of
>> yet, that are unrelated to citations?
>
> Do you mean how the syntax and implementation of links was used to
> support a new org feature completely unrelated to web URLs, namely
> citations?  ;)
>
> [It’s entirely unclear from your message, at least to me, whether you
> think these future features would be a good thing or a bad thing.]

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I know that this is technically easy to handle from the backend's
> perspective.  But I have a concern related to Stefan's: 
>
> Stefan Nobis <stefan-ml@snobis.de> writes:
>
>> The drawback is that now subtype is hard or even impossible to vary
>> for different backends. Therefore I would suggest that either org has
>> to define the allowed values of subtype or else we should define that
>> subtype has to be handled by the user (e.g. for use in private filter
>> functions) and is out of the scope of org (maybe this would be a good
>> place of extensions like org-ref to plug in their machinery).
>
> Basically, I am worried that neither of these options is very good.  
>
> If the subtype is interpreted by backends, I think it will become hard
> to use as an author.  If each backend treats different subtypes as
> significant, but some backends overlap in which subtypes they accept,
> then as an author you have to keep track of which subtypes work in which
> backends, and which don't, and whether the default behavior is OK for
> your citations in backends you care about.  If you are targeting more
> than one backend, that could mean a lot of trips to the manual.  And if
> user-supplied subtypes are also permissible, you have to keep track of
> the distinction between `official' subtypes and your own.

I wasn't clear. Subtype should be interpreted by back-ends means it has
no impact on syntax. However a user should be able to dictate what the
back-end should do with it, much like `org-add-link-type'.

A new library, e.g. "org-cite.el" would provide all the tools necessary
to build custom handlers for subtypes. Thus, power users can do whatever
they want with cite objects.

Nevertheless, Org needs to provide at least one built-in handler so
casual users are not required to do anything when writing

  [cite: see @Doe98 p.87] or even @Doe99.

This default handler may be also customized by power users, but we're
not there yet.

> On the other hand, if the subtype is just a user-supplied label, which
> can be interpreted by a filter but which backends will otherwise ignore,
> things are nice and simple.  You just don't use a subtype unless you are
> also supplying a way to interpret it on every backend that you care
> about.  But that kind of situation is exactly what the extra-info part
> of the syntax is for; in that case,
>
>   [cite/my-subtype: ...] 
>
> would just be an exceptional way of writing
>
>   [cite: ...]{:type my-subtype}
>
> or whatever.  I'm not totally convinced yet that the convenience of the
> former is worth blurring the line between `stuff that definitely works
> on all backends' and `stuff that might not work on all backends'.
>
>> Moreover [cite: ...]{...} syntax really makes sense if it is the
>> equivalent to #+attr_... keywords, so we can generalize it to links. As
>> a consequence, {...} should include a reference to back-end. E.g.,
>>
>>   [cite:...]{latex :color pink}
>
> We have already seen a couple of examples in this thread of properties
> that one might want to specify in a backend-agnostic way:
>   - special-case capitalization
>   - user-defined type/command/label/etc.
>
> Other things one might want to do include:
>   - indicating when a citation should be placed in a footnote 
>   - extracting arbitrary bibliographic field(s)

I disagree. These properties should be associated to the subtype.
Having two places to set them is asking for trouble.

IMO, there is little incentive to define a set of options for a single
use. Creating a dedicated subtype /once/ makes more sense.

Also the syntax stays clean.

>   - providing an overlay string for displaying complicated/ugly
>   citations

I don't think Org should provide this. Overlays are slow. Also, there is
no good default for displaying citations. However, an external library
could do that.

> And I think there is no reason, at the moment, to expect that these are
> the only possible uses for arbitrary backend-agnostic properties that a
> user can interpret, either via export filters or via in-buffer
> customizations.

Export filters are orthogonal to the problem at hand. They are applied
after handlers. We are discussing about handlers here (e.g., there are
custom link types and export filters for links).

> So I suggest that we also allow backend-agnostic properties here in
> addition to backend-specific ones.  My preference would be that we just
> let these backend-agnostic properties occur at the front, and separate
> the backend-specific ones with labeled pipes, like you suggested
> earlier:
>
>   [cite: ...]{:my-key1 my-val1 ... |latex: :lkey1 lval1 ... |html: ...} 
>
> I'm OK with disallowing the backend-agnostic properties for other types
> of objects if that is necessary, though I don't see how it would hurt to
> allow them, as long as it's not Org's responsibility to interpret
> them.

I think we should postpone the idea of attributes for object, as it gets
in the way of the discussion. IMO,

  [cite/subtype: ...]

is all we need, syntax-wise.


Regards,

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Nicolas,

2015ko otsailak 18an, Nicolas Goaziou-ek idatzi zuen:
>
> I think we should postpone the idea of attributes for object, as it gets
> in the way of the discussion. IMO,
>
>   [cite/subtype: ...]
>
> is all we need, syntax-wise.

The question of attributes for objects arose out of the desire to attach
attributes to citations.  So, I think it needs to be decided whether to
support a special syntax for citation attributes, or defer that to the
more general question of attributes for objects.  In the latter case, a
significant number of “power user” (read: early adopter) features won’t
be available until object attributes are also implemented.

IOW, citation syntax and object attributes are separate discussions,
but are also (possibly) both part of the path from the status quo to
citation support in org core.

--
Aaron Ecay

Re: Citation syntax: a revised proposal

[Rasmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Nicolas,
>
> 2015ko otsailak 18an, Nicolas Goaziou-ek idatzi zuen:
>>
>> I think we should postpone the idea of attributes for object, as it gets
>> in the way of the discussion. IMO,
>>
>>   [cite/subtype: ...]
>>
>> is all we need, syntax-wise.
>
> The question of attributes for objects arose out of the desire to attach
> attributes to citations.  So, I think it needs to be decided whether to
> support a special syntax for citation attributes, or defer that to the
> more general question of attributes for objects.  In the latter case, a
> significant number of “power user” (read: early adopter) features won’t
> be available until object attributes are also implemented.

Attributes are useful in their own right, but for the problem at hand they
are a distraction (IMO, but I'm also not a power-user).

> IOW, citation syntax and object attributes are separate discussions,
> but are also (possibly) both part of the path from the status quo to
> citation support in org core.

Why?  With one λ the sky is the limit!  It can call whatever crazy
functions that do whatever crazy things you can imagine.

IOW: if you are old enough to color your citations pink in latex you are
also old enough to write a macro that adds [cite:COLOR] or
[cite:subtype-color] for all colors in the rainbow...

—Rasmus

-- 
Me gusta la noche, me gustas tú

Re: Citation syntax: a revised proposal

[Rasmus]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>> We have already seen a couple of examples in this thread of properties
>> that one might want to specify in a backend-agnostic way:
>>   - special-case capitalization
>>   - user-defined type/command/label/etc.
>>
>> Other things one might want to do include:
>>   - indicating when a citation should be placed in a footnote 
>>   - extracting arbitrary bibliographic field(s)
>
> I disagree. These properties should be associated to the subtype.
> Having two places to set them is asking for trouble.
>
> IMO, there is little incentive to define a set of options for a single
> use. Creating a dedicated subtype /once/ makes more sense.
>
> Also the syntax stays clean.

+1.  A subtype is ∞ customizable.  With a highlevel API, e.g. based on
org-bibtex plist is enough for any relevant task.

>>   - providing an overlay string for displaying complicated/ugly
>>   citations
>
> I don't think Org should provide this. Overlays are slow. Also, there is
> no good default for displaying citations. However, an external library
> could do that.

I think overlays would be nice, e.g. for multiple authors, but [cite: pre
@key post] it's almost effortless to read, so I don't care much.

Someone mentioned that e.g. Zotero has poor keys by default.  Of course
one way to get around this is inserting a zotero entry as a org-bibtex
entry and give it nice key.  Problem solved.

>> And I think there is no reason, at the moment, to expect that these are
>> the only possible uses for arbitrary backend-agnostic properties that a
>> user can interpret, either via export filters or via in-buffer
>> customizations.
>
> Export filters are orthogonal to the problem at hand. They are applied
> after handlers. We are discussing about handlers here (e.g., there are
> custom link types and export filters for links).

+1.  Handling subtypes in a filter is a bad idea.  It's already hard doing
stuff without extracting the element from the text-properties.

Customization should be done over a list of plist of entries imo, e.g.
((:common-pre "pre" :common-post "post" :type "cite" :subtype "subtype")
 (:type "article" :title "t" :author "a" :pre "pre" :post "post")
 (:type "article" :title "t" :author "a" :pre "pre" :post "post"))

Utility functions like citeauthor should be available so that you can
generate e.g. genetive cite as
  (cite-mapconcat
  (lambda (cite) (concat (citeauthor cite) "'s (" (citeyeat cite) ")" ))
  citations)

> I think we should postpone the idea of attributes for object, as it gets
> in the way of the discussion. IMO,
>
>   [cite/subtype: ...]
>
> is all we need, syntax-wise.

+1

Though [cite:subtype:whatever] has a higher RK metric. . . 

—Rasmus

-- 
Me gusta la noche, me gustas tú

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:


> I wasn't clear. Subtype should be interpreted by back-ends means it has
> no impact on syntax. However a user should be able to dictate what the
> back-end should do with it, much like `org-add-link-type'.
>
> A new library, e.g. "org-cite.el" would provide all the tools necessary
> to build custom handlers for subtypes. Thus, power users can do whatever
> they want with cite objects.

Ah, OK, I think I see...so this is basically the second option, with
users interpreting subtypes via a separate protocol, instead of via
filters.  Right?

What about this concern, then?

>> But that kind of situation is exactly what the extra-info part
>> of the syntax is for; in that case,
>>
>>   [cite/my-subtype: ...] 
>>
>> would just be an exceptional way of writing
>>
>>   [cite: ...]{:type my-subtype}
>>
>> or whatever.  I'm not totally convinced yet that the convenience of the
>> former is worth blurring the line between `stuff that definitely works
>> on all backends' and `stuff that might not work on all backends'.


>> We have already seen a couple of examples in this thread of properties
>> that one might want to specify in a backend-agnostic way:
>>   - special-case capitalization
>>   - user-defined type/command/label/etc.
>>
>> Other things one might want to do include:
>>   - indicating when a citation should be placed in a footnote 
>>   - extracting arbitrary bibliographic field(s)
>
> I disagree. These properties should be associated to the subtype.
> Having two places to set them is asking for trouble.
>
> IMO, there is little incentive to define a set of options for a single
> use. Creating a dedicated subtype /once/ makes more sense.

Hmm, OK.  I agree in the abstract, but as far as I understand what a
`subtype' is, I am not sure how well that idea applies in this case.  If
a subtype encompasses a set of options, and the only way to specify
those options is by specifying a subtype, then changing one option means
changing to a whole different subtype, even when you want all the other
options to stay the same.  That will lead to a lot of duplication in
subtypes.  That in turn could lead to a lot of author frustration: now I
have to remember exactly which subtype encompasses the set of options I
need for this particular citation.

Look at the LaTeX citation commands.  There, you have a different
`type'/command for every possible combination of options, and there are
many commands because there are many options to deal with special cases.
In my mind, that is the wrong abstraction to emulate.  You end up making
trips to the manual to figure out exactly which one you need.  

To me, it makes much more sense to have a basic citation command, and
then a way to `answer some questions' to toggle various things related
to special-case behavior, like:
  - should it be footnoted?
  - should it have special-case capitalization?
  - should it cite the volume?
  - should it deal with brackets in a context-sensitive manner?
  - should it use the genitive form of the author's name?
  - ...

I would guess that the answers to these questions *usually* have nothing
to do with one another, so it makes sense to be able to answer them
independently, and change your answer to one independently of your
answers to the others.  That's why I would want to be able to write

[cite: ...]{:footnoted t :capitalized t :author-title t ...}
 
rather than

[cite/footnoted-capitalized-author-title: ...]

In the former case, if I later figure out I don't need the `:capitalized
t' part, that's easy to remove without changing the other options.  In
the latter case, I need to remember what name I gave to the subtype for
footnoted-but-not-capitalized-author-titles, or whatever.

I don't know; maybe this is not a very serious concern in practice.
Maybe, in practice, you generally only need one `special case' option at
a time, so the number of subtypes will be small.  Still, I do not feel
comfortable declaring *in advance* that that is so; and the bewildering
array of LaTeX citation commands is at least some evidence that it
isn't.  Thus, I am in favor of allowing the greater flexibility provided
by the former syntax.

That's not to say that subtypes never make sense: you are quite right
that sometimes options *should* be grouped together into a subtype,
namely, when it makes sense to set or remove them *conjointly*.  But
that means the two syntaxes above actually have different purposes, even
though for some range of cases either one would do the job. 

So although I would not be in favor of *only* allowing

[cite/subtype: ...]    

I am basically OK with allowing this if we also allow the {:key val ...}
syntax.  Still, the /subtype form is not needed if we also allow

[cite: ...]{:type subtype}

Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Ah, OK, I think I see...so this is basically the second option, with
> users interpreting subtypes via a separate protocol, instead of via
> filters.  Right?

Correct.

> What about this concern, then?
>
>>> But that kind of situation is exactly what the extra-info part
>>> of the syntax is for; in that case,
>>>
>>>   [cite/my-subtype: ...] 
>>>
>>> would just be an exceptional way of writing
>>>
>>>   [cite: ...]{:type my-subtype}
>>>
>>> or whatever.  I'm not totally convinced yet that the convenience of the
>>> former is worth blurring the line between `stuff that definitely works
>>> on all backends' and `stuff that might not work on all backends'.

I think everything is in the category "stuff that might not work on all
backends". Citations do not even make sense in all back-ends.

> Hmm, OK.  I agree in the abstract, but as far as I understand what a
> `subtype' is, I am not sure how well that idea applies in this case.  If
> a subtype encompasses a set of options, and the only way to specify
> those options is by specifying a subtype, then changing one option means
> changing to a whole different subtype, even when you want all the other
> options to stay the same.  That will lead to a lot of duplication in
> subtypes.  That in turn could lead to a lot of author frustration: now I
> have to remember exactly which subtype encompasses the set of options I
> need for this particular citation.

AFAICT, the most advanced use of citations is Thomas', and he is
basically only using "subtype". So I'm pretty confident that 99.9% of
users will be fine with only these subtypes.

> Look at the LaTeX citation commands.  There, you have a different
> `type'/command for every possible combination of options, and there are
> many commands because there are many options to deal with special cases.
> In my mind, that is the wrong abstraction to emulate.  You end up making
> trips to the manual to figure out exactly which one you need.  
>
> To me, it makes much more sense to have a basic citation command, and
> then a way to `answer some questions' to toggle various things related
> to special-case behavior, like:
>   - should it be footnoted?
>   - should it have special-case capitalization?
>   - should it cite the volume?
>   - should it deal with brackets in a context-sensitive manner?
>   - should it use the genitive form of the author's name?
>   - ...
>
> I would guess that the answers to these questions *usually* have nothing
> to do with one another, so it makes sense to be able to answer them
> independently, and change your answer to one independently of your
> answers to the others.  That's why I would want to be able to write
>
> [cite: ...]{:footnoted t :capitalized t :author-title t ...}
>  
> rather than
>
> [cite/footnoted-capitalized-author-title: ...]
>
> In the former case, if I later figure out I don't need the `:capitalized
> t' part, that's easy to remove without changing the other options.  In
> the latter case, I need to remember what name I gave to the subtype for
> footnoted-but-not-capitalized-author-titles, or whatever.

A good naming scheme can certainly help here. I'd choose

  [cite:FAT:...]

over

  [cite: ...]{:footnoted t :capitalized t :author-title t}

Moreover, we certainly can provide a menu offering the defined subtypes
(with a short summary) for completion.

> I don't know; maybe this is not a very serious concern in practice.
> Maybe, in practice, you generally only need one `special case' option at
> a time, so the number of subtypes will be small.  Still, I do not feel
> comfortable declaring *in advance* that that is so; and the bewildering
> array of LaTeX citation commands is at least some evidence that it
> isn't.  Thus, I am in favor of allowing the greater flexibility provided
> by the former syntax.

In fact, the former syntax provides less flexibility since you declare
/in advance/ what features, or properties, will be supported (or you let
users declare their own keywords, which is even worse).

It doesn't even help users because the handler for "foo" subtype has to
take into account all possible combinations of properties, and so has
"bar" subtype's handler. IOW, handlers become complicated to write.

> That's not to say that subtypes never make sense: you are quite right
> that sometimes options *should* be grouped together into a subtype,
> namely, when it makes sense to set or remove them *conjointly*.  But
> that means the two syntaxes above actually have different purposes, even
> though for some range of cases either one would do the job.

They have the same purpose. Your concern is that one may need a large
number of subtypes. That can be handled by appropriate tools.

> So although I would not be in favor of *only* allowing
>
> [cite/subtype: ...]    
>
> I am basically OK with allowing this if we also allow the {:key val ...}
> syntax.  Still, the /subtype form is not needed if we also allow
>
> [cite: ...]{:type subtype}

Again, I don't think we need {:key val} at the moment. Also, it would be
nice to eschew having once again at least two different ways to write
the same thing (footnotes, links...).


Regards,

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> AFAICT, the most advanced use of citations is Thomas', and he is
> basically only using "subtype". So I'm pretty confident that 99.9% of
> users will be fine with only these subtypes.
> ...
> Again, I don't think we need {:key val} at the moment. Also, it would be
> nice to eschew having once again at least two different ways to write
> the same thing (footnotes, links...).

OK.  I don't anticipate needing {:key val} myself anytime soon; I was
just trying to future-proof the syntax, and I don't want to lobby for it
if you feel strongly that this is problematic.

If there are others (John? Aaron? Samuel?) who think they really need
the {:key val} syntax *over and above* a subtype designation, please
speak up!

Best,
Richard

Re: Citation syntax: a revised proposal

[Samuel Wales]

On 2/20/15, Richard Lawrence <richard.lawrence@berkeley.edu> wrote:
> If there are others (John? Aaron? Samuel?) who think they really need
> the {:key val} syntax *over and above* a subtype designation, please
> speak up!

i have no comments on citations per se.  i just think the syntax we
design should, if possible, be so general that it can be used for
future features, including 100% unrelated features, and also for
future subfeatures of any feature, including citations.  to me, that
means plist or similar.

basically, i am concerned about syntax creep in the big picture and
its downstream consequences.  for example, it's more efficient to
support, and for the user to remember, a single general syntax than a
whole bunch of special syntaxes.

if everybody is already thinking along the same lines, great.


samuel

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  And
ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: Citation syntax: a revised proposal

[Rasmus]

Samuel Wales <samologist@gmail.com> writes:

> if everybody is already thinking along the same lines, great.

I think everybody is thinking along the lines, but some people want to not
have another link-morass :) In particular, I think we are trying hard to
avoid this situation:

    i just think the syntax we design should, if possible, be so general
    that it can be used for future features, *including 100% unrelated
    features*, and also for future subfeatures of any feature, including
    citations.

These days, my impression is that Org developers like to have [fn:·]
always be of a footnote type and *bold* always be of bold type.

> to me, that means plist or similar.

A lambda (that is a cite-subtype) is ∞ more customizable than a plist.
But at least by having a cite-like prefix Org forces you to write
something unappealing like,

        [cite:color-bar-pink-and-change-bar-to-baz: foo @bar]

Which might discourage you to something stupid.

A generalization of, say macros and link which look like [FUN: :key value]
or [FUN: arg]{:key value} may be appropriate, but it's something
different from the discussion at hand.

—Rasmus

-- 
Hvor meget poesi tror De kommer ud af et glas isvand?

Re: Citation syntax: a revised proposal

[Samuel Wales]

hi rasmus,

On 2/20/15, Rasmus <rasmus@gmx.us> wrote:
> I think everybody is thinking along the lines, but some people want to not
> have another link-morass :) In particular, I think we are trying hard to
> avoid this situation:
>
>     i just think the syntax we design should, if possible, be so general
>     that it can be used for future features, *including 100% unrelated
>     features*, and also for future subfeatures of any feature, including
>     citations.

this means that we are not thinking along the same lines.

what i am describing is what i described years ago in several posts.
it was mentioned recently [and on john's blog], then discussion went
back to citation-specific syntax.

> These days, my impression is that Org developers like to have [fn:·]
> always be of a footnote type and *bold* always be of bold type.

i am not proposing hijacking existing syntax; i am proposing the
opposite.  i am proposing a single, new, unambiguous syntax.  e.g.

  $[feature args... :key value ...]

for more than just "the feature we need today".  i don't care about
the details of the outer syntax.  and i misspoke when i said plist.  i
meant specifiable via a lambda list.

for today's feature, that can mean e.g.

  $[cite blah :blah2 blah3]

if you want to keep the mnemonic, that's fine too:

  $[(Cite) blah ...]

but suppose we want, oh, say, backend-independent color in 5 years:

  $[color-start "red"]red$[color-end "red"]

[i am just making this up as i go along to give you the general idea.]

notice how we did not need to invent new syntax!

>> to me, that means plist or similar.
>
> A lambda (that is a cite-subtype) is ∞ more customizable than a plist.

i don't think i'd favor anything that must eval.  security issues,
among other things.

> A generalization of, say macros and link which look like [FUN: :key value]
> or [FUN: arg]{:key value} may be appropriate, but it's something
> different from the discussion at hand.

i'm not sure i am explaining my point well here.


samuel

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.  And
ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: Citation syntax: a revised proposal

[Rasmus]

Hi Samuel,

Samuel Wales <samologist@gmail.com> writes:

> On 2/20/15, Rasmus <rasmus@gmx.us> wrote:
>> I think everybody is thinking along the lines, but some people want to not
>> have another link-morass :) In particular, I think we are trying hard to
>> avoid this situation:
>>
>>     i just think the syntax we design should, if possible, be so general
>>     that it can be used for future features, *including 100% unrelated
>>     features*, and also for future subfeatures of any feature, including
>>     citations.
>
> this means that we are not thinking along the same lines.
>
> what i am describing is what i described years ago in several posts.
> it was mentioned recently [and on john's blog], then discussion went
> back to citation-specific syntax.

As I said an arbitrary [fun: arg :key val] is great.  It might solve what
I (perhaps unfairly) dubbed the "link-morass", since it has no
description.

> i am not proposing hijacking existing syntax; i am proposing the
> opposite.  i am proposing a single, new, unambiguous syntax.  e.g.
>
>   $[feature args... :key value ...]
> ...
>   $[color-start "red"]red$[color-end "red"]

^^^ This is already supported via a macros (for export at least):

{{{color-start red, red}}}
#+MACRO: color-start @@html:<span ⋯ style=f($1)>$2</span@@@@latex:⋯@@

> [i am just making this up as i go along to give you the general idea.]
>
> notice how we did not need to invent new syntax!

I sympathize with the idea.  Surely (some years ago I *wanted* to write
the generalized "link", but lacked time and skillz).

But citations is a different beast and fixed syntax is what is needed.

>>> to me, that means plist or similar.
>>
>> A lambda (that is a cite-subtype) is ∞ more customizable than a plist.
>
> i don't think i'd favor anything that must eval.  security issues,
> among other things.

I too worry about the NSA backdoors in self-insert-command. . .

If you don't allow a generalized link to follow a user-specified λs then
you don't have a flexible syntax that you expressed desire for above.
You'd still have to wait for somebody "upstream" to develop
[color-start:⋯].

>> A generalization of, say macros and link which look like [FUN: :key value]
>> or [FUN: arg]{:key value} may be appropriate, but it's something
>> different from the discussion at hand.
>
> i'm not sure i am explaining my point well here.

You are.  I just don't agree citation support should be generalized to a
more abstract level at this point.  What Org desperately needs in terms of
reproducible, scientific writing is a rigorous, standard syntax.

—Rasmus

-- 
Dung makes an excellent fertilizer

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Rasmus <rasmus@gmx.us> writes:

>  What Org desperately needs in terms of
> reproducible, scientific writing is a rigorous, standard syntax.

IMHO, the citation syntax discussion is more about making citations work
correctly in the various backends and less about reproducible documents.
Babel was a key development for reproducible documents.  With babel,
"non-standard" functions could be stuffed in a noexport section, made
buffer local with Emacs local variables, and distributed so others can
reproduce the document.

This model has a certain beauty.  A community of scholars can share
reproducible documents and concentrate on the content without having to
agree upon and use a standard syntax for document preparation.

Or, a sub-community of scholars, like the Kitchin research group, can
adopt a package built on top of extensible syntax, then distribute their
work in reproducible form to scholars in the larger community.

At any rate, I'm just reacting to "desperately needs" and "rigorous,
standard".

I think Samuel's ideas on extensible syntax are compatible with
reproducible scientific writing.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Samuel Wales]

hi rasmus,

dunno if this is of any value [possibly negative value via confusion].

cannot respond to all of it, so just a little.

On 2/21/15, Rasmus <rasmus@gmx.us> wrote:
> But citations is a different beast and fixed syntax is what is needed.

the syntax for citations can be defined to be fixed.

> If you don't allow a generalized link to follow a
> user-specified λs then you don't have a flexible syntax
> that you expressed desire for above.  You'd still have to
> wait for somebody "upstream" to develop [color-start:⋯].

not sure why you are talking about links.

you can write color-start as a user.  you can even define a
feature that requires a lambda: $[rasmus-color-start (lambda
(x) (rasmus-stuff))].  or $[rasmus-color ...].

but if we are going to define citations with this syntax,
then we can do so strictly.  no need for lambda.

> You are.  I just don't agree citation support should be
> generalized to a more abstract level at this point.  What

citations can be specific and concrete.

the first atom says it's a citation.  it's up to the
developers of citation syntax to decide how specific and
concrete it should be.

maybe you are saying that you don't think it's a good idea
to allow other first atoms.  not sure why.

> reproducible, scientific writing is a rigorous, standard syntax.

also, those adjectives are up to the developers.

just make it error upon nonstandardness.

the only difference is that you won't have to create the
infrastructure the NEXT time you as a developer or user want
to add syntax.

===


samuel

-- 
The Kafka Pandemic: http://thekafkapandemic.blogspot.com

The disease DOES progress.  MANY people have died from it.
And ANYBODY can get it.

Denmark: free Karina Hansen NOW.

Re: Citation syntax: a revised proposal

[Stefan Nobis]

Samuel Wales <samologist@gmail.com> writes:

> the only difference is that you won't have to create the
> infrastructure the NEXT time you as a developer or user want to add
> syntax.

But the hard part of e.g. the citation syntax is not the parser. The
hart part are the semantics. The discussion in this thread is not
about aesthectics of the syntax (at least that's not the main part),
but about what the right abstraction is.

The other hard part is implementing the machinery to make citations
really great, i.e. jumping to the right entry in any of the different
possible citation backends etc.

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[Rasmus]

Samuel Wales <samologist@gmail.com> writes:

>> If you don't allow a generalized link to follow a
>> user-specified λs then you don't have a flexible syntax
>> that you expressed desire for above.  You'd still have to
>> wait for somebody "upstream" to develop [color-start:⋯].
>
> not sure why you are talking about links.

'Cause [type/subtype: whaterver]{:key val} or [type/subtype:
whaterver :key val] is like links, only meant to operate as functions,
treating data.  Much like links are used, making the description part
rather annoying at times...

> you can write color-start as a user.  you can even define a
> feature that requires a lambda: $[rasmus-color-start (lambda
> (x) (rasmus-stuff))].  or $[rasmus-color ...].
>
> but if we are going to define citations with this syntax,
> then we can do so strictly.  no need for lambda.

Consider:

        [cite/color-me-pink:this will be pink]

The subtype is color-me-pink.  Color-me-pink is a user-written function (a
λ).  E.g.

     (add-to-cite-types "color-me-pink" (lambda (cite) (my/pink cite)))

Note that this example can already be implemented via macros.

> maybe you are saying that you don't think it's a good idea
> to allow other first atoms.  not sure why.

No.  I would be thrilled about that.  It would be much better than links
for most purposes.

But it's a different issue.

> also, those adjectives are up to the developers.

Agreed.

—Rasmus

-- 
Hooray!

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Samuel,

Samuel Wales <samologist@gmail.com> writes:

> basically, i am concerned about syntax creep in the big picture and
> its downstream consequences.  for example, it's more efficient to
> support, and for the user to remember, a single general syntax than a
> whole bunch of special syntaxes.

In general, I share your point of view here, which is one reason why I
proposed the s-expression part of the syntax for citations.  I still
like that idea, and I think Nicolas is right that it would be good to
extend it to other sorts of objects in Org.  You're right that
consistency in that syntax would be a good thing.  Moreover, Org already
has syntax that looks a lot like plists in #+ATTR_BACKEND lines and in
Babel source block headers, so it seems natural to adopt something like
it for other sorts of objects.  I think that is the right way to go,
both from the perspective of consistency and from the perspective of
expressiveness.

I guess the rejoinder to all this, though, is that this point of view
can be taken too far.  I am glad, for example, that Org uses different
syntax for delimiting source blocks and delimiting emphasized text.
Insisting on syntactic consistency there would give you something like
HTML...and I am glad that writing Org is not like writing HTML.
(Likewise, I will be glad when we have a syntax specially-designed for
writing citations, even if it looks nothing like the syntax other
things.)

So I think the right question is: what is the correct range of
application for a uniform syntax for user-specifiable extensions, for
citations or otherwise?  I don't know the answer to that, but
`properties of existing objects' seems like a good start.  This needs
more thought, in another thread, but thank you for reminding us to keep
it in mind.

Best,
Richard

Re: Citation syntax: a revised proposal

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> You're right that consistency in that syntax would be a good thing.
> Moreover, Org already has syntax that looks a lot like plists in
> #+ATTR_BACKEND lines and in Babel source block headers, so it seems
> natural to adopt something like it for other sorts of objects.

Org also already has [fn:·:·] that already is supported inline and looks
an awful lot like [cite:·:·]. . .

—Rasmus

-- 
Hooray!

Re: Citation syntax: a revised proposal

[Samuel Wales]

hi rasmus!

thanks for your comments.

we are definitely not communicating about the same things.
i do not even understand what you think i am proposing.  :)
if you get a chance to re-read my old posts on the subject,
i think it might help?


samuel

Re: Citation syntax: a revised proposal

[Samuel Wales]

hi richard,

indeed i am not proposing changing any existing syntax.


samuel

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Richard,

2015ko otsailak 20an, Richard Lawrence-ek idatzi zuen:
> OK.  I don't anticipate needing {:key val} myself anytime soon; I was
> just trying to future-proof the syntax, and I don't want to lobby for it
> if you feel strongly that this is problematic.
> 
> If there are others (John? Aaron? Samuel?) who think they really need
> the {:key val} syntax *over and above* a subtype designation, please
> speak up!

Speaking for myself, I think the discussion so far has revealed a
number of “advanced” uses of citations, such as possessive citations,
citations as footnotes, the insertion of only author/year/etc., ...
At least for academic publishing, citations are pretty demanding and
there isn’t much room for “close enough;” a paper’s citations either
conform to a particular style guide or they don’t.

I think these various applications of citations, and others not yet
mentioned or thought of, are best represented as binary switches.  Many
of these distinctions will factor well into independent implementations.
For example, a citation that is :footnote t can (probably) be generated
by taking the citation, whatever it is, and wrapping it in
\footnote{...}.  (For the latex case; other backends will have different
specifics but the idea is the same.)  If this is implemented in terms of
subtypes, it will lead to an explosion of 2^n subtypes being necessary.

Of course, not all 2^n combinations will be realized (I don’t think it
makes sense for a citation to be both possessive and a footnote, for
example).  Ultimately, it’s an empirical question how well different
types of citation factor, and how many of the combinations make sense or
are ever realized.

Nicolas has given reasons why the inline attr syntax is needed
independently.  I think no-subtype citations + inline attr is a superset
of with-subtype citations.  I’d rather see the superset be implemented.
Subtypes would constrain the expressivity of citations and lead to
more fragile implementations.  Since we’re designing the syntax from
scratch, I would like to avoid that.

However, the most important thing is to implement something.  The
semipermanent beta status of master allows a period of experimentation
with a citation syntax before something is made official in a release.

Aaron

PS A note on implementation: I envision a sort of pattern matching on
key-value combinations.  Something like:

(((:possessive t :footnote t) (error "wtf"))
 ;; the generated citation command will be inserted at the %s
 ((:footnote t) (wrap "\footnote{%s}"))
 ;; slightly artificial example to illustrate pattern matching with binding
 ((:color _c) (wrap (format "\color{%s}{%%s}" _c)))
 ((:possessive t) (cite "\citeposs{%s}" ...))
 ;; cite provides a list of four format strings for the
 ;; (non-)capitalized (non-)parenthesized
 ;; variants encoded in the citation type
 (default (cite "\cite{%s}" "\parencite{%s}" "\Cite{%s}" "\Parencite{%s}")))

Where the list of attributes is pattern-matched, and the first matching
cite command is composed with all matching wrap commands.  I’ve just
shown one-place format strings for the cite key, but a full
implementation would have to handle pre- and post-note.  It would
probably also need to handle multicites as a fifth type (or set of 4
types).  Though it’s worth considering whether the latex \multicite
family of commands provides anything above and beyond a series of
sequential \cite’s.  It might be possible to handle multicites by just
using elisp to concatenate individual citation commands, and not letting
them vary by backend.

The specifics of whether cite and wrap are sufficient primitives needs
to be decided on.  Probably we need to allow functions not just format
strings, for the benefit of non-latex backends where the citation needs
to be formatted by emacs.  Then cite and wrap would just be predefined
shortcuts, with the ability to drop into full elisp for more complicated
cases.

A small version of the 2^n problem is already visible: the 4 types of
citation necessitate providing 4 strings/functions for the default case,
and also for the possessive case (though I think this is unavoidable
under any implementation).

This is a very rough sketch, but I hope it helps stimulate thinking.
There’s already a pattern matching library in emacs (pcase.el), though
it would need to be extended for plist pattern matching.

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> Speaking for myself, I think the discussion so far has revealed a
> number of “advanced” uses of citations, such as possessive citations,
> citations as footnotes, the insertion of only author/year/etc., ...
> At least for academic publishing, citations are pretty demanding and
> there isn’t much room for “close enough;” a paper’s citations either
> conform to a particular style guide or they don’t.
>
> I think these various applications of citations, and others not yet
> mentioned or thought of, are best represented as binary switches.  Many
> of these distinctions will factor well into independent implementations.
> For example, a citation that is :footnote t can (probably) be generated
> by taking the citation, whatever it is, and wrapping it in
> \footnote{...}.  (For the latex case; other backends will have different
> specifics but the idea is the same.)  If this is implemented in terms of
> subtypes, it will lead to an explosion of 2^n subtypes being necessary.

Yes, I share this worry.

> Of course, not all 2^n combinations will be realized (I don’t think it
> makes sense for a citation to be both possessive and a footnote, for
> example).  Ultimately, it’s an empirical question how well different
> types of citation factor, and how many of the combinations make sense or
> are ever realized.

Indeed.  It is hard to tell in advance how much of a practical concern
this is, but I can see subtypes becoming unpleasant to use unless one
thinks ahead carefully about all the kinds of subtypes one needs, and
how to relate them.

It would be good to hear from people who know already that they will
need something like subtypes or arbitrary key-value properties.  How
would you folks prefer to trade off between these options?
  1) subtypes which are specified by a label and easy to write handlers
     for, but potentially introduce a lot of redundancy in handling
     different formatting properties
  2) arbitrary key-value pairs (one of which could be `:type t'), which are
     harder to write handlers for, but introduce less redundancy and
     make it easier to handle formatting properties individually

> Nicolas has given reasons why the inline attr syntax is needed
> independently.  I think no-subtype citations + inline attr is a superset
> of with-subtype citations.  I’d rather see the superset be implemented.
> Subtypes would constrain the expressivity of citations and lead to
> more fragile implementations.  Since we’re designing the syntax from
> scratch, I would like to avoid that.

> However, the most important thing is to implement something.  The
> semipermanent beta status of master allows a period of experimentation
> with a citation syntax before something is made official in a release.

Agreed.  I'd like to see an implementation of a parser for the
[cite:...] part of the syntax as a first step.  If we can get that far,
I'd guess that extending the parser to include either a subtype label or
{:key val ...} syntax will not be too difficult to do.

I am OK with Elisp, but I should probably not lead the charge here,
since I am not very familiar with how org-element works internally.  If
I hack this up myself, it will probably take longer and result in a
less-acceptable patch than if someone more experienced does it.  But if
no one volunteers, I will start to work on it, with what skills I've
got.

Erik Hetzner started work on an Elisp parser for Pandoc syntax, which is
here:

https://gitlab.com/egh/org-pdcite/blob/master/org-pdcite.el

Erik seems to have taken a break from the discussion since we moved away
from the Pandoc syntax, but this might be a good starting point, either
for him or for someone else.

> PS A note on implementation: I envision a sort of pattern matching on
> key-value combinations.  Something like:
>
> (((:possessive t :footnote t) (error "wtf"))
>  ;; the generated citation command will be inserted at the %s
>  ((:footnote t) (wrap "\footnote{%s}"))
>  ;; slightly artificial example to illustrate pattern matching with binding
>  ((:color _c) (wrap (format "\color{%s}{%%s}" _c)))
>  ((:possessive t) (cite "\citeposs{%s}" ...))
>  ;; cite provides a list of four format strings for the
>  ;; (non-)capitalized (non-)parenthesized
>  ;; variants encoded in the citation type
>  (default (cite "\cite{%s}" "\parencite{%s}" "\Cite{%s}" "\Parencite{%s}")))

I like this idea.  Especially if there is fall-through in the `wrap'
clauses, it would make it pretty easy to write your own handlers for
arbitrary key-value pairs atop the default handlers, though one would
still have to be careful because the order of the clauses would be
significant.

> Where the list of attributes is pattern-matched, and the first matching
> cite command is composed with all matching wrap commands.  I’ve just
> shown one-place format strings for the cite key, but a full
> implementation would have to handle pre- and post-note.  It would
> probably also need to handle multicites as a fifth type (or set of 4
> types).  Though it’s worth considering whether the latex \multicite
> family of commands provides anything above and beyond a series of
> sequential \cite’s.  It might be possible to handle multicites by just
> using elisp to concatenate individual citation commands, and not letting
> them vary by backend.
>
> The specifics of whether cite and wrap are sufficient primitives needs
> to be decided on.  Probably we need to allow functions not just format
> strings, for the benefit of non-latex backends where the citation needs
> to be formatted by emacs.  Then cite and wrap would just be predefined
> shortcuts, with the ability to drop into full elisp for more complicated
> cases.
>
> A small version of the 2^n problem is already visible: the 4 types of
> citation necessitate providing 4 strings/functions for the default case,
> and also for the possessive case (though I think this is unavoidable
> under any implementation).
>
> This is a very rough sketch, but I hope it helps stimulate thinking.
> There’s already a pattern matching library in emacs (pcase.el), though
> it would need to be extended for plist pattern matching.

Yes, that is all food for further thought.  Thanks for illustrating the
idea! 

Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Agreed.  I'd like to see an implementation of a parser for the
> [cite:...] part of the syntax as a first step.  If we can get that far,
> I'd guess that extending the parser to include either a subtype label or
> {:key val ...} syntax will not be too difficult to do.

I'll do it by the end of the week.

Regards,

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Agreed.  I'd like to see an implementation of a parser for the
>> [cite:...] part of the syntax as a first step.  If we can get that far,
>> I'd guess that extending the parser to include either a subtype label or
>> {:key val ...} syntax will not be too difficult to do.
>
> I'll do it by the end of the week.

That would be wonderful!  Will you publish a patch or, better, a branch
somewhere, even if it's not ready for master?
 
I don't know if this is helpful, but I'm posting an updated grammar
below for the [cite: ...] part of the syntax that uses parentheses
around `cite' to indicate the distinction between in-text and
parenthetical citations.  It doesn't include any clauses for a subtype
label or for extra key-value pairs.

Best,
Richard

We need a category of /citation/ objects, which require the following
properties (the names here are not important and could be changed):
  - is-parenthetical (boolean; nil means is in-text)
  - common-prefix (text)
  - common-suffix (text)
  - references (list)

Each reference in the list of references should be a plist with the
following properties:
  - prefix (text)
  - suffix (text)
  - key (string)
  - suppress-author (boolean; t means author name should not be output)
  - is-full (boolean; t means a full bibliography entry should be
    output in-place) 

The category of citations has the following grammar:
  - A CITATION is a PARENTHETICAL-CITATION or an IN-TEXT citation.
  - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
    CITATION-LIST whose TAG is a PARENTHESIZED-TAG.
  - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
    CITATION-LIST whose TAG is a BARE-TAG.
  - A SIMPLE-PARENTHETICAL is a KEY immediately surrounded by square
    brackets.
  - A SIMPLE-IN-TEXT is a BARE-KEY.
  - A CITATION-LIST has the format
       [TAG: PREFIX; INDIVIDUAL-REFERENCE; ... INDIVIDUAL-REFERENCE; SUFFIX]
    where the initial PREFIX and final SUFFIX are optional.  At least
    one INDIVIDUAL-REFERENCE must be present.  The colon and
    semicolons here are literal and indicate the end of the TAG and
    the end of a PREFIX or INDIVIDUAL-REFERENCE respectively. 
  - A TAG is either a PARENTHESIZED-TAG or a BARE-TAG.
  - A PARENTHESIZED-TAG is a BARE-TAG, immediately surrounded by parentheses.
  - A BARE-TAG is: `cite'.
  - An INDIVIDUAL-REFERENCE has the format: 
       PREFIX KEY SUFFIX
    The KEY is obligatory, and the prefix and suffix are optional.
  - A BARE-KEY is a KEY with immediately-preceding whitespace
  - A KEY optionally begins with `-', and obligatorily contains `@' or
    `&' followed by a string of characters which begins with a letter
    or `_', and may contain alphanumeric characters and the following
    internal punctuation characters:
       :.#$%&-+?<>~/
  - A PREFIX or SUFFIX is arbitrary text (except `;', `]', and KEYs)
    which may contain only the following Org objects:
    - bold
    - code
    - entity
    - italic
    - latex-fragment
    - line-break
    - strike-through
    - subscript
    - superscript
    - underline
    - superscript

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> That would be wonderful!  Will you publish a patch or, better, a branch
> somewhere, even if it's not ready for master?

I created a new branch: "wip-cite". It introduces support for @key
[@key] [cite:pre @key post] and [(cite):pre @key post] constructs.

As a reminder, I prefer subkeys over plists because they have a smaller
footprint in the document. Also, as already explained, having many
subkeys is not a problem with proper tooling (e.g., some completion with
descriptions). Note that this is closer to "org-ref" requirements,
probably making easier to port some features into core.

However, opinion from advanced citation users on this ML has more weight
that mine. Instead of trying to figure out hypothetical crazy uses for
citations (e.g., using 50 different citation commands), I'd rather hear
from people with real citation requirements who are willing to use this
machinery.

At this point, we probably need to implement a BIBLIOGRAPHY keyword
(files) and BIBLIOGRAPHY_BACKEND (bibtex, zotero, jabref...) and provide
basic tools to handle citations in an Org document.


Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Hi,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> That would be wonderful!  Will you publish a patch or, better, a branch
>> somewhere, even if it's not ready for master?
>
> I created a new branch: "wip-cite". It introduces support for @key
> [@key] [cite:pre @key post] and [(cite):pre @key post] constructs.

Cool.  I'll check it out.

> However, opinion from advanced citation users on this ML has more weight
> that mine. Instead of trying to figure out hypothetical crazy uses for
> citations (e.g., using 50 different citation commands), I'd rather hear
> from people with real citation requirements who are willing to use this
> machinery.

I use a similar setup and beside the above I use citeauthor a lot.
Cite/(cite) covers something like 80% of my needs, but (later) I think
something like org-cite-add-new-subtype should be there.  With a nice API.
Or we can add a couple of more subtypes.

> At this point, we probably need to implement a BIBLIOGRAPHY keyword
> (files) and BIBLIOGRAPHY_BACKEND (bibtex, zotero, jabref...) and provide
> basic tools to handle citations in an Org document.

Probably a CITATION_STYLE as well, e.g. "numeric", "author-year", etc.

I'll try to look at biblatex support for ox-latex, which should be the
easiest target, but ATM I'm a bit busy.  For bibtex-outside-of-latex,
reftex-cite.el is decent, but not great¹.  Still, it may be easier to fix
it up that to write our own bibtex parser.  

Or did John already solve this problem?  Perhaps, org-bibtex offer good
support for parsing data?

Note, something like author-parsing is non-trivial, since bibtex both
support firstname lastname and lastname, firstname.  Further, for
author-year style, you'd sometimes want to support no. of authors (see %a
in reftex-format-citation).

I have noticed tex4ht manages to do "proper" citations in odt.  Perhaps we
can study the resulting xml and how it adds a entries.  Formatting is
tricky...  Perhaps only zotero is useful here.

Latexml has some support for bibtex in html, but I haven't studied it
properly.  In any case for author-year html is "easy" up to the point of
creating the bibliography (as with odt).

Cheers,
Rasmus

Footnotes: 
¹   E.g. if curly brackets are not removed from year it will
    include it, it only understands "lastname, firstname" for author (not
    "firstname lastname") etc.


-- 
The Kids call him Billy the Saint

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Rasmus <rasmus@gmx.us> writes:

> Probably a CITATION_STYLE as well, e.g. "numeric", "author-year", etc.

I suggest we keep Patrick Daly's distinction between "citation style"
and "citation mode".  Hence, #+CITATION_MODE instead
of #+CITATION_STYLE. 
 
IIUC, there are three citation modes:
 1. Harvard, author-date, author-year, etc.  In this mode, some
    citation detail is given at the point of the Org mode citation.
 2. Vancouver, footnote, etc.  In this mode, the point of the Org mode
    citation is indicated by a number that references a footnote or
    endnote where some citation detail is given.  In some variants of
    this mode, the full citation information is provided in the
    footnote or endnote and no separate bibliography is used, while in
    others only partial citation detail is provided in the footnote or
    endnote and a bibliography is provided.
 3. Numerical.  In this mode, no citation detail is inserted, only a
    number that references an entry in the bibliography.

Within each of these three modes, there are numerous citation styles,
which refer to the placement of items and the kinds of punctuation in
the citation.

In the wild, a particular citation style might appear with one
bibliographic style or another.  Hence, the separation of citation
styles from bibliographic styles in systems such as biblatex.

The following tables describe my uncertainty about how the Org mode
citations map to these three modes.  The tables show example (modulo
style) replacement text at the point of the Org mode citation.  Note
that the Vancouver mode will also require the addition of citation
information in a footnote or endnote, separate from the position of the
Org mode citation.

#+name: first-map
| citation | Harvard       | Vancouver | numerical |
|----------+---------------+-----------+-----------|
| @key     | author year   |         1 |         1 |
| [@key]   | (author year) |       (1) |       (1) |
| cite:    | author year   |         1 |         1 |
| (cite):  | (author year) |       (1) |       (1) |

#+name: second-map
| citation | Harvard       | Vancouver  | numerical  |
|----------+---------------+------------+------------|
| @key     | author (year) | author (1) | author (1) |
| [@key]   | (author year) | (1)        | (1)        |
| cite:    | author (year) | author (1) | author (1) |
| (cite):  | (author year) | (1)        | (1)        |

Or, is there a third-map that better captures what we're discussing?

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Eric S Fraga]

On Sunday,  1 Mar 2015 at 14:24, Thomas S. Dye wrote:
> Rasmus <rasmus@gmx.us> writes:
>
>> Probably a CITATION_STYLE as well, e.g. "numeric", "author-year", etc.
>
> I suggest we keep Patrick Daly's distinction between "citation style"
> and "citation mode".  Hence, #+CITATION_MODE instead
> of #+CITATION_STYLE. 

Agreed.

[...]

> The following tables describe my uncertainty about how the Org mode
> citations map to these three modes.  The tables show example (modulo
> style) replacement text at the point of the Org mode citation.  Note
> that the Vancouver mode will also require the addition of citation
> information in a footnote or endnote, separate from the position of the
> Org mode citation.

I'm very much in line with the second of your two maps:

> #+name: second-map
> | citation | Harvard       | Vancouver  | numerical  |
>
> |----------+---------------+------------+------------|
> | @key     | author (year) | author (1) | author (1) |
> | [@key]   | (author year) | (1)        | (1)        |
> | cite:    | author (year) | author (1) | author (1) |
> | (cite):  | (author year) | (1)        | (1)        |


thanks,
eric

-- 
: Eric S Fraga (0xFFFCF67D), Emacs 24.4.1, Org release_8.3beta-820-gd92ef9

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Rasmus <rasmus@gmx.us> writes:

> I have noticed tex4ht manages to do "proper" citations in odt.  Perhaps we
> can study the resulting xml and how it adds a entries.  Formatting is
> tricky...  Perhaps only zotero is useful here.

IIUC, tex4ht uses the dvi (device independent format of Knuth) file
produced by LaTeX.  Thus, it is able to take advantage of the work of
sophisticated citations managers built on bibtex.  I think of it,
perhaps naively, as a program that can be configured to generate dvi
drivers for odt, html, xml, etc.

hth,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Rasmus]

tsd@tsdye.com (Thomas S. Dye) writes:

> Rasmus <rasmus@gmx.us> writes:
>
>> I have noticed tex4ht manages to do "proper" citations in odt.  Perhaps we
>> can study the resulting xml and how it adds a entries.  Formatting is
>> tricky...  Perhaps only zotero is useful here.
>
> IIUC, tex4ht uses the dvi (device independent format of Knuth) file
> produced by LaTeX.  Thus, it is able to take advantage of the work of
> sophisticated citations managers built on bibtex.  I think of it,
> perhaps naively, as a program that can be configured to generate dvi
> drivers for odt, html, xml, etc.

Thanks.

The point I was trying to raise is that it adds "proper" citations in LO.
That means a citation is a gray "field" rather than white "plaintext".
Much like the macro \citet{·} differs from plaintext "A (Y)" in LaTeX.

—Rasmus

-- 
May the Force be with you

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Rasmus,

2015ko martxoak 1an, Rasmus-ek idatzi zuen:
> 
>> At this point, we probably need to implement a BIBLIOGRAPHY keyword
>> (files) and BIBLIOGRAPHY_BACKEND (bibtex, zotero, jabref...) and provide
>> basic tools to handle citations in an Org document.
> 
> Probably a CITATION_STYLE as well, e.g. "numeric", "author-year", etc.
> 
> I'll try to look at biblatex support for ox-latex, which should be the
> easiest target, but ATM I'm a bit busy.  

If you have time, I’d appreciate your opinion on whether the approach
I’ve started of doing latex and non-latex together in ox-cite is a good
approach, or whether instead you’d rather handle latex within ox-latex.

Should we also support “plain” bibtex and natbib?

> For bibtex-outside-of-latex, reftex-cite.el is decent, but not great¹.
> Still, it may be easier to fix it up that to write our own bibtex
> parser.

It would also be possible to just use an external program like
citeproc-java.  WDYT?

Thanks,

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Rasmus]

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Rasmus,
>
> 2015ko martxoak 1an, Rasmus-ek idatzi zuen:
>> 
>>> At this point, we probably need to implement a BIBLIOGRAPHY keyword
>>> (files) and BIBLIOGRAPHY_BACKEND (bibtex, zotero, jabref...) and provide
>>> basic tools to handle citations in an Org document.
>> 
>> Probably a CITATION_STYLE as well, e.g. "numeric", "author-year", etc.
>> 
>> I'll try to look at biblatex support for ox-latex, which should be the
>> easiest target, but ATM I'm a bit busy.  
>
> If you have time, I’d appreciate your opinion on whether the approach
> I’ve started of doing latex and non-latex together in ox-cite is a good
> approach, or whether instead you’d rather handle latex within ox-latex.

Good that I didn't start hacking on ox-latex in the plain, but went for
org-element instead :)

I will check them out.  I think ox-cite will be a beast.  Still, since
citation is a single object it should probably be in backends.
E.g. export of inlinetasks are handled in backends.

Still, general functionality and backend support and/or API should
probably be in a separate library.

WDYT?

> Should we also support “plain” bibtex and natbib?

I think John said that journals often require natbib.  At this point I'm
using biblatex only.

For ox-latex, it might make sense to have a :citation-backend which is
maps supported citations types to packages.  Until somebody complains, we
could support biblatex only.

>> For bibtex-outside-of-latex, reftex-cite.el is decent, but not great¹.
>> Still, it may be easier to fix it up that to write our own bibtex
>> parser.
>
> It would also be possible to just use an external program like
> citeproc-java.  WDYT?

This is the preferred method by far!  The closer we can get to the latex
citation where we just insert "naïve" commands the better IMO.

—Rasmus

-- 
In theory, practice and theory are the same. In practice they are not

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Aaron Ecay <aaronecay@gmail.com> writes:

> It would also be possible to just use an external program like
> citeproc-java.  WDYT?

I agree with Rasmus that using an external tool is the preferred way to
go here.  I don't think introducing a dependency is really a problem, so
long as we choose the right dependency -- LaTeX is a dependency for the
PDF exporter, after all.

Is there any reason to go with citeproc-java over a different CSL
implementation, like citeproc-js or pandoc-citeproc?  I am a little
nervous about shelling out to something that sounds it like it requires
loading the JVM...

I think Zotero also has a built-in CSL processor (actually, I think it
uses citeproc-js), and Erik Hetzner's zotxt plugin looks like it lets
you communicate with it in client-server fashion:

https://bitbucket.org/egh/zotxt/src

So maybe Zotero + zotxt is a good candidate for a `blessed' citation
manager and CSL processor?  

Best,
Richard

Re: Citation syntax: a revised proposal

[Avram Lyon]

[-- Attachment #1: Type: text/plain, Size: 851 bytes --]

On Mon, Mar 2, 2015 at 7:16 PM Richard Lawrence <
richard.lawrence@berkeley.edu> wrote:
>
> Is there any reason to go with citeproc-java over a different CSL
> implementation, like citeproc-js or pandoc-citeproc?  I am a little
> nervous about shelling out to something that sounds it like it requires
> loading the JVM...
>

citeproc-java just calls citeproc-js from Rhino or Nashorn, so there's
little reason to go with citeproc-java for any application not already
running on the JVM.

Zotero is indeed using citeproc-js directly from XULrunner/Firefox, and
that is the best-supported usage of the library.

If you're looking for something with citation management and CSL proessing,
perhaps zotxt is best. If you just want CSL processing, it would be best to
run citeproc-js by itself (there is a citeproc-node, but it's not quite
plug-and-play).

[-- Attachment #2: Type: text/html, Size: 1170 bytes --]

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Avram,

Avram Lyon <ajlyon@gmail.com> writes:

> On Mon, Mar 2, 2015 at 7:16 PM Richard Lawrence <
> richard.lawrence@berkeley.edu> wrote:
>>
>> Is there any reason to go with citeproc-java over a different CSL
>> implementation, like citeproc-js or pandoc-citeproc?  I am a little
>> nervous about shelling out to something that sounds it like it requires
>> loading the JVM...
>
> citeproc-java just calls citeproc-js from Rhino or Nashorn, so there's
> little reason to go with citeproc-java for any application not already
> running on the JVM.

Hmm, good to know.

> Zotero is indeed using citeproc-js directly from XULrunner/Firefox, and
> that is the best-supported usage of the library.
>
> If you're looking for something with citation management and CSL proessing,
> perhaps zotxt is best. If you just want CSL processing, it would be best to
> run citeproc-js by itself (there is a citeproc-node, but it's not quite
> plug-and-play).

That sounds right.  And I agree with Aaron that we probably don't want a
hard dependency on Zotero on the output side, so maybe citeproc-js is
the way to go.  On the other hand, as Aaron points out, citeproc-java
has a BibTeX parser, and citeproc-js doesn't look like it would be easy
to run from the command line...some sort of JS engine is required in
addition to citeproc-js itself.

I wonder if citeproc-js would run under Guile??  Maybe that would be the
easiest way to turn citeproc-js into a lightweight command line utility
that Org (and hence Emacs) could feel good about depending on.
 
Best,
Richard

Re: Citation syntax: a revised proposal

[Avram Lyon]

[-- Attachment #1: Type: text/plain, Size: 1060 bytes --]

On Tue, Mar 3, 2015 at 9:28 AM Richard Lawrence <
richard.lawrence@berkeley.edu> wrote:
>
> That sounds right.  And I agree with Aaron that we probably don't want a
> hard dependency on Zotero on the output side, so maybe citeproc-js is
> the way to go.  On the other hand, as Aaron points out, citeproc-java
> has a BibTeX parser, and citeproc-js doesn't look like it would be easy
> to run from the command line...some sort of JS engine is required in
> addition to citeproc-js itself.
>
> I wonder if citeproc-js would run under Guile??  Maybe that would be the
> easiest way to turn citeproc-js into a lightweight command line utility
> that Org (and hence Emacs) could feel good about depending on.
>

I know that citeproc-js has tried to be engine-agnostic, so perhaps it can
work with Guile. If not, you may also want to look at citeproc-hs and
citeproc-rb, both of which are quite complete (they, I believe, pass the
entire test suite) and which may be easier to bring in as dependencies (JS
engines are still a rarer dependency than Ruby or Haskell).

[-- Attachment #2: Type: text/html, Size: 1376 bytes --]

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Avram Lyon <ajlyon@gmail.com> writes:

> I know that citeproc-js has tried to be engine-agnostic, so perhaps it can
> work with Guile. 

It looks like I was too quick.  Although the homepage makes it seem like
Guile supports JS, the manual says:

"ECMAScript was not the first non-Schemey language implemented by Guile,
but it was the first implemented for Guile's bytecode compiler. The goal
was to support ECMAScript version 3.1, a relatively small language, but
the implementor was completely irresponsible and got distracted by other
things before finishing the standard library, and even some bits of the
syntax. So, ECMAScript does deserve a mention in the manual, but it
doesn't deserve an endorsement until its implementation is completed,
perhaps by some more responsible hacker."

https://www.gnu.org/software/guile/manual/html_node/ECMAScript.html#ECMAScript

...and I can't even get my local Guile to interpret "var x = 2;", though
I doubt that is entirely Guile's fault.  So it looks to me like Guile's
JS support is not quite mature enough to run citeproc-js, though I'd be
happy to be wrong about this.

> If not, you may also want to look at citeproc-hs and citeproc-rb, both
> of which are quite complete (they, I believe, pass the entire test
> suite) and which may be easier to bring in as dependencies (JS engines
> are still a rarer dependency than Ruby or Haskell).

This is the citeproc-ruby you have in mind, right?

https://github.com/inukshuk/citeproc

(I found another under the name `citeproc-rb', but it doesn't look
nearly as complete.)

Both the Ruby citeproc and citeproc-hs seem to have support for reading
BibTeX databases.

Anyway, maybe a good intermediate step would be translating citation
objects and Org-bibtex data to JSON.  It looks like citeproc-hs and
citeproc-js both accept JSON as input formats (though I do not know if
these formats are compatible...).  The Ruby citeproc implementation
speaks some amount of JSON (though mostly as an output format, I think),
and I guess it could probably be taught to read JSON pretty easily.
This needs some more investigation, but if Org can produce a JSON
representations of citation data, it seems like it could be made to work
with several citeproc implementations that look reasonably complete.

Best,
Richard

Re: Citation syntax: a revised proposal

[Rasmus]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Is there any reason to go with citeproc-java over a different CSL
> implementation, like citeproc-js or pandoc-citeproc?  I am a little
> nervous about shelling out to something that sounds it like it requires
> loading the JVM...

For the longest of time, mathtoweb.jar was the blessed MathML producer.
Java is great 'cause it works equally mediocre on all platforms!

This might make citeproc-java very attractive (from its Github page):

     "... citeproc-java contains a BibTeX converter that is able to map
     BibTeX database entries to CSL citations."

But I really know the details of CSL.

—Rasmus


-- 
To err is human. To screw up 10⁶ times per second, you need a computer

Re: Citation syntax: a revised proposal

[Rasmus]

Rasmus <rasmus@gmx.us> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Is there any reason to go with citeproc-java over a different CSL
>> implementation, like citeproc-js or pandoc-citeproc?  I am a little
>> nervous about shelling out to something that sounds it like it requires
>> loading the JVM...
>
> For the longest of time, mathtoweb.jar was the blessed MathML producer.
> Java is great 'cause it works equally mediocre on all platforms!
>
> This might make citeproc-java very attractive (from its Github page):
>
>      "... citeproc-java contains a BibTeX converter that is able to map
>      BibTeX database entries to CSL citations."

Actually, Richard, check out the cli tool of citeproc-java:

    http://michel-kraemer.github.io/citeproc-java/using/command-line-tool/

It looks pretty cool.  It reads bib, json and mendeley(?) out of the box
and support text, html output.  From html it's not far to odt.

Unfortunately, Zotero stores its database in sqlite according to their
wiki...  Perhaps, a zotero back-end could be added...

Are there other equivalently "nice" CSL cli-tools?

How would CSL work to get e.g. parentheses citations?  Is it a question of
passing an alternative style, e.g. "format with @foo with
chicago-author-date" versus "format with @foo with
chicago-author-date-parenthesis"?

—Rasmus

-- 
The second rule of Fight Club is: You do not talk about Fight Club

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Richard,

2015ko martxoak 3an, Richard Lawrence-ek idatzi zuen:
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> It would also be possible to just use an external program like
>> citeproc-java.  WDYT?
> 
> I agree with Rasmus that using an external tool is the preferred way to
> go here.  I don't think introducing a dependency is really a problem, so
> long as we choose the right dependency -- LaTeX is a dependency for the
> PDF exporter, after all.

OK, I’m satisfied that there is good rationale for using an external
tool.  This will make some things a lot easier.

> 
> Is there any reason to go with citeproc-java over a different CSL
> implementation, like citeproc-js or pandoc-citeproc?  I am a little
> nervous about shelling out to something that sounds it like it requires
> loading the JVM...

citeproc-java has a bibtex parser; citeproc-js doesn’t.  (The simplest
of many reasons.)

I looked at pandoc-citeproc, but it seems tightly integrated with
pandoc’s internal JSON representation, and not as flexible for general
purpose usage as citeproc-(js,java).

> 
> I think Zotero also has a built-in CSL processor (actually, I think it
> uses citeproc-js), and Erik Hetzner's zotxt plugin looks like it lets
> you communicate with it in client-server fashion:
> 
> https://bitbucket.org/egh/zotxt/src
> 
> So maybe Zotero + zotxt is a good candidate for a `blessed' citation
> manager and CSL processor?

Requiring a running zotero seems worse to me than requiring the JVM.  I
guess it should be an option, but probably not the default.

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> That would be wonderful!  Will you publish a patch or, better, a branch
>> somewhere, even if it's not ready for master?
>
> I created a new branch: "wip-cite". It introduces support for @key
> [@key] [cite:pre @key post] and [(cite):pre @key post] constructs.

Great!  I'll take a look.

What's the next step here?  Adding support for multiple references?
multi-cites? `&'-keys?  

> As a reminder, I prefer subkeys over plists because they have a smaller
> footprint in the document. Also, as already explained, having many
> subkeys is not a problem with proper tooling (e.g., some completion with
> descriptions). Note that this is closer to "org-ref" requirements,
> probably making easier to port some features into core.
>
> However, opinion from advanced citation users on this ML has more weight
> that mine. Instead of trying to figure out hypothetical crazy uses for
> citations (e.g., using 50 different citation commands), I'd rather hear
> from people with real citation requirements who are willing to use this
> machinery.

Yes, me too.

> At this point, we probably need to implement a BIBLIOGRAPHY keyword
> (files) and BIBLIOGRAPHY_BACKEND (bibtex, zotero, jabref...) and provide
> basic tools to handle citations in an Org document.

Could we guess the backend from the file extension on the BIBLIOGRAPHY,
to keep things simple here?  I don't use a citation manager, so I don't
know if this is possible for anything other than Bib(La)TeX.  

Also, as mentioned earlier, it would be really nice to support
org-bibtex as one of the reference database formats.  (It's what I use,
so naturally it's what I think we should bless. :) This would allow
storing your reference database in-document.

Some things to think about:

1) Is there ever a need to mix reference database formats in the same
document (e.g., zotero and org-bibtex)?  (I would think not, but my
needs are simple.)

2) Is there ever a need to mix multiple reference databases in the
*same* format (e.g., two different .bib files)?  (I would think so,
given the existence in BibLaTeX of \addbibresource.)

3) If the answer to either 1 or 2 is yes, how should we decide
precedence between multiple reference databases?  (Two databases might
contain the same key.)

Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> What's the next step here?  Adding support for multiple references?
> multi-cites? `&'-keys?

To support multi cites, we must first decide how the parsed will present
information, i.e., what are the properties in the following case

  [cite:pre; pre1 @k1 post1; pre2 @k2 post2; post]

There is also the following degenerate case

  [cite:pre; pre ;post]

What should be done about it?

I didn't implement &-keys as there is no consensus on them.

Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> What's the next step here?  Adding support for multiple references?
>> multi-cites? `&'-keys?
>
> To support multi cites, we must first decide how the parsed will present
> information, i.e., what are the properties in the following case
>
>   [cite:pre; pre1 @k1 post1; pre2 @k2 post2; post]

I was actually looking at this today and wondering why this was not
supported.

I think a citation object should always member of a citations object.  So
the above would be

   (citations (:begin n :end N :prefix pre :suffix post
               :citations
               '((citation (:key k1 :begin n1 :end N1 :prefix pre1))
                 (citation (:key k2 :begin n2 :end N2 :prefix pre2 :suffix post2)))))

This makes it naturally to operate over one many citations.  I don't know
if this should be some sort of pseudo-object or what.  Also, one issue I
ran into when trying to get [@k1; @k2] working was that @k2 is recognized
as an inline citation (which means that I probably did something wrong)...

Of course, a quasi-tricky part (I think) is that [cite: pre @key post]
should be (with no "global" :prefix and :suffix):

   (citations (:begin n :end N
               :citations
               '((citation (:key key :begin n1 :end N1 :prefix pre :suffix post)))))

Which imply that citations are parsed from "the middle" and outwards.


Nicolas: I wrote a patch for subtypes (with "/" as a separator as most
people seemed to like that).  Should I post it or will you take care of it
eventually?  I don't know if you have got a game-plan in mind?
[I also have other outstanding patches, so I just want to put my limited
Org-time where it makes sense].

Cheers,
Rasmus

-- 
There are known knowns; there are things we know that we know

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> I was actually looking at this today and wondering why this was not
> supported.

Not enough specifications.

> I think a citation object should always member of a citations object.  So
> the above would be
>
>    (citations (:begin n :end N :prefix pre :suffix post
>                :citations
>                '((citation (:key k1 :begin n1 :end N1 :prefix pre1))
>                  (citation (:key k2 :begin n2 :end N2 :prefix pre2 :suffix post2)))))

OK. 

However mixing `citations' and `citation' is confusing. I'd rather keep
the outer one as `citation'. What could go inside? Maybe `cite'?

Moreover,

  [cite:@key]

will be parsed as

  [citation (:begin n :end N
             :whatever '((whatever (:key key :begin n1 :end N1 :prefix pre1))))]

Is that correct?

> This makes it naturally to operate over one many citations.  I don't know
> if this should be some sort of pseudo-object or what.  Also, one issue I
> ran into when trying to get [@k1; @k2] working was that @k2 is recognized
> as an inline citation (which means that I probably did something
> wrong)...

[@k1; @k2] ?

This is unspecified. [@k1] is a shortcut for [(cite):@k1], nothing more.
Anything more complicated should go in a [cite:...] object.

> Of course, a quasi-tricky part (I think) is that [cite: pre @key post]
> should be (with no "global" :prefix and :suffix):
>
>    (citations (:begin n :end N
>                :citations
>                '((citation (:key key :begin n1 :end N1 :prefix pre :suffix post)))))
>
> Which imply that citations are parsed from "the middle" and outwards.

I don't see any ambiguity here, since semi colons are forbidden in PRE
and POST.

> Nicolas: I wrote a patch for subtypes (with "/" as a separator as most
> people seemed to like that).  Should I post it or will you take care of it
> eventually?  I don't know if you have got a game-plan in mind?

I didn't add subtypes because we didn't reach a consensus on it.
I suggest to wait until everybody realizes subtypes are the superior
choice.


Regards,

-- 
Nicolas Goaziou

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> However mixing `citations' and `citation' is confusing. I'd rather keep
> the outer one as `citation'. What could go inside? Maybe `cite'?

Perhaps the difference is too subtle.  Note that you would never deal with
a `citation' other than through a mapping.

> Moreover,
>
>   [cite:@key]
>
> will be parsed as
>
>   [citation (:begin n :end N
>              :whatever '((whatever (:key key :begin n1 :end N1 :prefix pre1))))]
>
> Is that correct?

Yeah, exactly.  So the format would be a map over the list of whatevers in
ox.

>> This makes it naturally to operate over one many citations.  I don't know
>> if this should be some sort of pseudo-object or what.  Also, one issue I
>> ran into when trying to get [@k1; @k2] working was that @k2 is recognized
>> as an inline citation (which means that I probably did something
>> wrong)...
>
> [@k1; @k2] ?
>
> This is unspecified. [@k1] is a shortcut for [(cite):@k1], nothing more.
> Anything more complicated should go in a [cite:...] object.

Right, I was trying *to add* support for [@k1; ⋯;@kN].  One issue doing
this was that the regexp for inline citations also captured @k2 in [@k1;
@k2], but it's cause I did a mistake.

>> Of course, a quasi-tricky part (I think) is that [cite: pre @key post]
>> should be (with no "global" :prefix and :suffix):
>>
>>    (citations (:begin n :end N
>>                :citations
>>                '((citation (:key key :begin n1 :end N1 :prefix pre :suffix post)))))
>>
>> Which imply that citations are parsed from "the middle" and outwards.
>
> I don't see any ambiguity here, since semi colons are forbidden in PRE
> and POST.

I was talking about paring [cite: pre @k post] as 


    (citation (:begin n :end N
               :cites
                '((cite (:key key :begin n1 :end N1 :prefix pre :suffix post)))))

And not:

    (citation (:begin n :end N :prefix pre :suffix post
               :cites '((cite (:key key :begin n1 :end N1)))))

Perhaps I'm worrying about things that need not be worried about.

—Rasmus

-- 
This message is brought to you by the department of redundant departments

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> Perhaps the difference is too subtle.  Note that you would never deal with
> a `citation' other than through a mapping.

Hmm. I still find `citations/citation' pair confusing. What about
`citation/part'?

> Right, I was trying *to add* support for [@k1; ⋯;@kN].

Please don't. Let's keep shortcuts simple.

> I was talking about paring [cite: pre @k post] as 
>
>
>     (citation (:begin n :end N
>                :cites
>                 '((cite (:key key :begin n1 :end N1 :prefix pre :suffix post)))))
>
> And not:
>
>     (citation (:begin n :end N :prefix pre :suffix post
>                :cites '((cite (:key key :begin n1 :end N1)))))
>
> Perhaps I'm worrying about things that need not be worried about.

The latter would be obtained with

  [cite: pre; @k; post]


Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> Perhaps the difference is too subtle.  Note that you would never deal with
>> a `citation' other than through a mapping.
>
> Hmm. I still find `citations/citation' pair confusing. What about
> `citation/part'?

So (citation ⋯ :parts ((part ⋯) (part ⋯))).  Fine with me.  Other
possibilities are citation/entry and :entries.  Or less nice:
citation/cite and :cites.

>> Right, I was trying *to add* support for [@k1; ⋯;@kN].
>
> Please don't. Let's keep shortcuts simple.

It would enable "complicated argument [@k1;@k2;@k3]", which is pretty
nice.  There's still no pre and post notes etc, only keys.

—Rasmus

-- 
Together we'll stand, divided we'll fall

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> So (citation ⋯ :parts ((part ⋯) (part ⋯))).  Fine with me.  Other
> possibilities are citation/entry and :entries.  Or less nice:
> citation/cite and :cites.

I don't mind using :entries and `entry' either. I'll update "wip-cite"
in a few days.

> It would enable "complicated argument [@k1;@k2;@k3]", which is pretty
> nice.  There's still no pre and post notes etc, only keys.

Shortcuts are simple citations that ought to be available in most
back-ends. I don't see your example as a particularly straightforward
one. I suggest to use

  [(cite):@k1;@k2;@k3]

Note that your example doesn't even provide an in-text equivalent while
[@k1] has @k1.


Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> It would enable "complicated argument [@k1;@k2;@k3]", which is pretty
>> nice.  There's still no pre and post notes etc, only keys.
>
> Shortcuts are simple citations that ought to be available in most
> back-ends. I don't see your example as a particularly straightforward
> one. I suggest to use
>
>   [(cite):@k1;@k2;@k3]

That also fine, though for the simple case slightly less readable.

> Note that your example doesn't even provide an in-text equivalent while
> [@k1] has @k1.

But that's OK since I can easily formulate this in "human-language":

   "@k1, @k2, and @k3 argues ⋯" → "A1 (Y1), A2 (Y2), and A3 (Y3) argues"

Whereas I can't easily do

   "Argument (@k1;@k2;@k3)" →  "Argument (A1 Y1; A2 Y2; A3 Y3)"

Perhaps I'm thinking too much in terms of text/parenthesis citations here.

—Rasmus

-- 
. . . The proofs are technical in nature and provides no real
understanding

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> To support multi cites, we must first decide how the parsed will present
> information, i.e., what are the properties in the following case
>
>   [cite:pre; pre1 @k1 post1; pre2 @k2 post2; post]

I was thinking that this should yield a citation object with a structure like:

('citation ...
  :common-prefix pre
  :common-suffix post
  :references ((:prefix pre1
                :key "k1"
                :suffix post1 ...)
               (:prefix pre2
                :key "k2"
                :suffix post2 ...))
  ...)

Would that work?

> There is also the following degenerate case
>
>   [cite:pre; pre ;post]
>
> What should be done about it?

Hmm, I don't quite understand what you mean.  This is not allowed by the
grammar (as far as I can tell), so that should not parse as a citation
object, IMO.

> I didn't implement &-keys as there is no consensus on them.

Oh, I did not realize there were outstanding issues with this.  I
remember Rasmus not liking `&'.  I'm fine with changing it, though I
cannot think of a better symbol.  Does someone think we should not have
a way of indicating that a reference should produce a full bibliography
entry?  Or that we should indicate it in some other way?  

Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

>> To support multi cites, we must first decide how the parsed will present
>> information, i.e., what are the properties in the following case
>>
>>   [cite:pre; pre1 @k1 post1; pre2 @k2 post2; post]
>
> I was thinking that this should yield a citation object with a structure like:
>
> ('citation ...
>   :common-prefix pre
>   :common-suffix post
>   :references ((:prefix pre1
>                 :key "k1"
>                 :suffix post1 ...)
>                (:prefix pre2
>                 :key "k2"
>                 :suffix post2 ...))
>   ...)
>
> Would that work?

Yes. I find it better than "entries/entry" as discussed with Rasmus.
I'll implement it in a few days.

> Oh, I did not realize there were outstanding issues with this.  I
> remember Rasmus not liking `&'.  I'm fine with changing it, though I
> cannot think of a better symbol.  Does someone think we should not have
> a way of indicating that a reference should produce a full bibliography
> entry?  Or that we should indicate it in some other way?

AFAIC, I don't think a dedicated symbol is useful. It can be implemented
through subtypes/properties. Besides LaTeX, could other back-end provide
that feature anyway?

I have no opinion about the :suppress-author symbol.


Regards,

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> Oh, I did not realize there were outstanding issues with this.  I
>> remember Rasmus not liking `&'.  I'm fine with changing it, though I
>> cannot think of a better symbol.  Does someone think we should not have
>> a way of indicating that a reference should produce a full bibliography
>> entry?  Or that we should indicate it in some other way?
>
> AFAIC, I don't think a dedicated symbol is useful. It can be implemented
> through subtypes/properties. 

It would be useful to have a per-key symbol rather than a subtype if one
wants to mix regular and `full' references in the same citation.  But I
am not sure there are any realistic use cases for this, so I am fine
with expressing `full' citations via a subtype.
 
> Besides LaTeX, could other back-end provide that feature anyway?

Yes, I don't think that would be a problem, if we are using a CSL
processor.  You can ask a CSL processor for either a citation or a full
bibliography entry.

> I have no opinion about the :suppress-author symbol.

The case I can think of where it would be most useful to have this
expressed via the key is when you have multiple works by the same author
in a citation, like:

[cite: This was originally proposed by @Doe1999.; See his -@Doe2014 for a recent review.]

Apart from that, I would think a subtype/property would suffice.

Best,
Richard

Re: Citation syntax: a revised proposal

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 1689 bytes --]

On Mar 3, 2015 3:43 AM, "Nicolas Goaziou" <mail@nicolasgoaziou.fr> wrote:
>
> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
> >> To support multi cites, we must first decide how the parsed will
present
> >> information, i.e., what are the properties in the following case
> >>
> >>   [cite:pre; pre1 @k1 post1; pre2 @k2 post2; post]
> >
> > I was thinking that this should yield a citation object with a
structure like:
> >
> > ('citation ...
> >   :common-prefix pre
> >   :common-suffix post
> >   :references ((:prefix pre1
> >                 :key "k1"
> >                 :suffix post1 ...)
> >                (:prefix pre2
> >                 :key "k2"
> >                 :suffix post2 ...))
> >   ...)
> >
> > Would that work?
>
> Yes. I find it better than "entries/entry" as discussed with Rasmus.
> I'll implement it in a few days.
>
> > Oh, I did not realize there were outstanding issues with this.  I
> > remember Rasmus not liking `&'.  I'm fine with changing it, though I
> > cannot think of a better symbol.  Does someone think we should not have
> > a way of indicating that a reference should produce a full bibliography
> > entry?  Or that we should indicate it in some other way?
>
> AFAIC, I don't think a dedicated symbol is useful. It can be implemented
> through subtypes/properties. Besides LaTeX, could other back-end provide
> that feature anyway?
>

I have done this with zotxt for HTML and odt export, so it should be
possible. I personally really like this, as I have a personal use case
(course syllabi) where I need this all the time. But I may be unusual in
this.

Matt
> I have no opinion about the :suppress-author symbol.
>
>
> Regards,
>

[-- Attachment #2: Type: text/html, Size: 2395 bytes --]

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I was thinking that this should yield a citation object with a structure like:
>
> ('citation ...
>   :common-prefix pre
>   :common-suffix post
>   :references ((:prefix pre1
>                 :key "k1"
>                 :suffix post1 ...)
>                (:prefix pre2
>                 :key "k2"
>                 :suffix post2 ...))
>   ...)
>
> Would that work?

Done in b0474b09a97ca20290b6f664b2e49a8c7a0d0c17.

Note that, as a consequence, the new object is incompatible with the
previous one, since every citation is a multi-cite citation. See commit
message for details.

Regards,

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Richard (again),

2015ko martxoak 2an, Richard Lawrence-ek idatzi zuen:
> 
> Could we guess the backend from the file extension on the BIBLIOGRAPHY,
> to keep things simple here?  I don't use a citation manager, so I don't
> know if this is possible for anything other than Bib(La)TeX.
> 
> Also, as mentioned earlier, it would be really nice to support
> org-bibtex as one of the reference database formats.  (It's what I use,
> so naturally it's what I think we should bless. :) This would allow
> storing your reference database in-document.

I too use org-bibtex, and I agree that in-document references would be
nice.  My skeletal implementation supports only org-bibtex at the
moment.

> 
> Some things to think about:
> 
> 1) Is there ever a need to mix reference database formats in the same
> document (e.g., zotero and org-bibtex)?  (I would think not, but my
> needs are simple.)

I think this is most likely in collaborative situations: you have your
carefully curated org-bibtex database, and your co-author sends you a
bunch of references in some other database format.  I think it’s pretty
easy to support (convert everything to bibtex and concat it all
together).

> 
> 2) Is there ever a need to mix multiple reference databases in the
> *same* format (e.g., two different .bib files)?  (I would think so,
> given the existence in BibLaTeX of \addbibresource.)

Certainly yes (this can be seen as a degenerate case of (1) above).

> 
> 3) If the answer to either 1 or 2 is yes, how should we decide
> precedence between multiple reference databases?  (Two databases might
> contain the same key.)

Initially I think an admonition of “don’t do that.”  Eventually, we
could raise a warning (or error) on detecting multiply-defined keys.  I
think allowing, and trying to make sense of, multiple definitions is
more trouble than it’s worth.

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 5192 bytes --]

Hi Nicolas,

Thanks for implementing the parser support.  I decided to go ahead and
see what I could make of it.  The result has been pushed to the org mode
repo to the branch wip-cite-awe.  (I didn’t want to push to your branch
without asking, but if you prefer I’ll do that and delete my own.)

First I’ll describe two minor issues with the parser support I
uncovered.  Then I’ll go on to describe what the code I’ve written does.

The first issue is that the parser includes trailing punctuation in
“bare” @key citations.  So the following does not work as expected (the
:key includes the period): “This was demonstrated most recently by
@Smith2015.”  I’m not sure what the right approach is – one option
would be to say that keys can contain punctuation, but must end (and
begin) with an alphanumeric character.

The second issue is that the :key property of the citation element
includes the @.  This is not right IMO: it’s a detail of the syntax.
And it means that consumers of the syntax, who might want to look up
the key in a database, will always have to remember to strip the @.
I’ve pushed a provisional fix for this in my branch.

=====

The code I have pushed introduces three document-level keywords: BIBDB,
CITATION_MODE, and CITATION_STYLE.  Each of these has a corresponding
extension point in the code: org-export-cite-add-lookup-type,
org-export-cite-add-citation-mode, and
org-export-cite-add-citation-style.

Lookup types are concerned with mapping from a key to bibliographic
information.  Currently an org-bibtex lookup is implemented.  Others
that could be added straightforwardly are bibtex and the internet DOI
resolver as described here:
https://tex.stackexchange.com/questions/6848/automatically-dereference-doi-to-bib

Citation modes are responsible for formatting the in-text citation
(usually, a reference to a full citation stored elsewhere).  Currently a
crude author-year style is implemented.  Footnote and numbered citations
could both be added with relatively little effort (I think), though
they’d be a bit more complicated because they require keeping track of a
counter.

Citation styles are responsible for formatting the citation in the
bibliography (or footnote, or wherever the “full” citation lands).
Currently a hyper-simple one is implemented that just outputs the
author, year, and title.

I’ve attached a zip file to this email which contains a very simple org
document, a very simple bibliography, and the results of exporting it to
HTML (body only), plain text, and Latex.  (These are the only three
supported backends so far).

The code is very rough and ready, has lots of TODO comments in it, is
missing tests, documentation in the manual, etc.  Nonetheless, I want to
get feedback on it early, given that many people have already contributed
so much useful information to this discussion.

The two main questions that arise for me at this point are:

-> How much is it worth trying to keep latex and the other backends
together.

The current implementation uses some common functions (in ox-cite.el)
for all backends, including latex.  However, latex basically does its
own thing.  So it would be possible to include in ox-cite only code for
non-latex backends, and then implement latex citations solely in
ox-latex.  Separation would make the initial implementation very
simple.  On the other hand, I worry that it would perpetuate the present
situation where latex and non-latex citations are two separate
universes.  On the third hand, serious latex users will probably just
say you should use latex of you want nice citations, since latex is so
much better.  ;)

-> How much of the non-latex citation code is it worth implementing in
elisp.

One quick strategy would be to depend on citeproc-java
<https://github.com/michel-kraemer/citeproc-java>.  This can generate
bibliographies in HTML format from a bibtex file, which can be parsed,
turned into org syntax, and used wherever they are needed.  However, it
introduces a hard dependency on this program.  Is it worth trying to put
together a bare-bones elisp implementation, so people can have
dependency-free bibliographies?  My inclination is to say “no,” to avoid
duplicating the considerable effort put into the CSL universe (over
7,000 citation styles available!).  However, it also seems a bit wrong
to have a core feature depend on a third party executable.

(How much it would slow down export to continually shell out to an
external program is also an open question.)

=====

I didn’t try to do anything about support for editing/inserting citations
within org mode.  It would be good to know whether John Kitchin is willing
to contribute bits of his org-ref code for that.  (I see on Worg that he
has an FSF assignment on file, but I don’t want to take without asking –
or do porting/integration work that he’s happy to do himself!)

Thanks,
Aaron

PS the code uses relatively new functions from the subr-x and let-alist
libraries, so it probably works best on a recent-ish (past few months)
trunk version of emacs.


[-- Attachment #2: org-citations.zip --]
[-- Type: application/zip, Size: 2514 bytes --]

[-- Attachment #3: Type: text/plain, Size: 15 bytes --]

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> I decided to go ahead and see what I could make of it. The result has
> been pushed to the org mode repo to the branch wip-cite-awe. (I didn’t
> want to push to your branch without asking, but if you prefer I’ll do
> that and delete my own.)

This is not *my* branch. However, I suggest to push only consensual
features with documentation and tests there, and experiment features in
other branches.

> The first issue is that the parser includes trailing punctuation in
> “bare” @key citations.  So the following does not work as expected (the
> :key includes the period): “This was demonstrated most recently by
> @Smith2015.”  I’m not sure what the right approach is – one option
> would be to say that keys can contain punctuation, but must end (and
> begin) with an alphanumeric character.

I'll update the parser once there is a new syntax for keys. At the
moment, it is correct wrt syntax.

> The second issue is that the :key property of the citation element
> includes the @.  This is not right IMO: it’s a detail of the syntax.
> And it means that consumers of the syntax, who might want to look up
> the key in a database, will always have to remember to strip the @.
> I’ve pushed a provisional fix for this in my branch.

Please apply it to wip-cite. A dedicated test would be nice, too.

> The code is very rough and ready, has lots of TODO comments in it, is
> missing tests, documentation in the manual, etc.  Nonetheless, I want to
> get feedback on it early, given that many people have already contributed
> so much useful information to this discussion.

I didn't look closely at the code, but I suggest to use "org-cite.el"
instead of "ox-cite.el". Even though this is only related to export at
the moment, this library will also contain commands to manipulate
citation objects. It also shorten prefix for these functions.

Thanks for your work,

Regards,

-- 
Nicolas Goaziou

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> The first issue is that the parser includes trailing punctuation in
>> “bare” @key citations.  So the following does not work as expected (the
>> :key includes the period): “This was demonstrated most recently by
>> @Smith2015.”  I’m not sure what the right approach is – one option
>> would be to say that keys can contain punctuation, but must end (and
>> begin) with an alphanumeric character.
>
> I'll update the parser once there is a new syntax for keys. At the
> moment, it is correct wrt syntax.

Sorry, I may not have emphasized this enough, but in the grammar, I wrote:

  - A KEY optionally begins with `-', and obligatorily contains `@' or
    `&' followed by a string of characters which begins with a letter
    or `_', and may contain alphanumeric characters and the following
    *internal* punctuation characters:
       :.#$%&-+?<>~/
       
The `internal' here was meant to express exactly what Aaron said: keys
can contain punctuation, but may not end (or begin) with punctuation.

Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Sorry, I may not have emphasized this enough, but in the grammar, I wrote:
>
>   - A KEY optionally begins with `-', and obligatorily contains `@' or
>     `&' followed by a string of characters which begins with a letter
>     or `_', and may contain alphanumeric characters and the following
>     *internal* punctuation characters:
>        :.#$%&-+?<>~/
>        
> The `internal' here was meant to express exactly what Aaron said: keys
> can contain punctuation, but may not end (or begin) with punctuation.

OK. I misunderstood the "internal" part. 

What about "@_" and "@a" ? Are they valid keys?


Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Sorry, I may not have emphasized this enough, but in the grammar, I wrote:
>>
>>   - A KEY optionally begins with `-', and obligatorily contains `@' or
>>     `&' followed by a string of characters which begins with a letter
>>     or `_', and may contain alphanumeric characters and the following
>>     *internal* punctuation characters:
>>        :.#$%&-+?<>~/

AFAIK Bibtex keys don't understand '#%~', so I'd remove those.  I would
leave out '$' as well, as it's also the math symbol (think of display
support).

The regexp used by bibtex.el is bibtex-entry-head and keys are matched by:

          \\([][[:alnum:].:;?!`'/*@+|()<>&_^$-]+\\)

> What about "@_" and "@a" ? Are they valid keys?

What is wrong with @a?  That seems like a perfectly legit key and one that
you would even use in real life, for a one-citation document, say.

@_ Would be supported by bibtex, but I don't see a reason for supporting
it here (what is "@_1"?  Why would citations take precedence over
subscripts?)


-- 
Together we'll stand, divided we'll fall

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Rasmus <rasmus@gmx.us> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>>
>>> Sorry, I may not have emphasized this enough, but in the grammar, I wrote:
>>>
>>>   - A KEY optionally begins with `-', and obligatorily contains `@' or
>>>     `&' followed by a string of characters which begins with a letter
>>>     or `_', and may contain alphanumeric characters and the following
>>>     *internal* punctuation characters:
>>>        :.#$%&-+?<>~/
>
> AFAIK Bibtex keys don't understand '#%~', so I'd remove those.  I would
> leave out '$' as well, as it's also the math symbol (think of display
> support).

I don't think we should remove these just because BibTeX doesn't support
them.  They might be used as keys by other citation databases, and
removing them might break them.

The syntax of KEYs (except for `&') is wholly borrowed from Pandoc,
which I figured had a reason for permitting those characters.  Note that
Pandoc reads reference databases in a lot of different formats,
including MODS, EndNote, etc., which might be more liberal with symbols
in keys:

http://pandoc.org/README.html#citations

So unless we decide we don't want to support people who use these
formats, I suggest we be more liberal here, too.  This won't cause a
problem for BibTeX users. 

Best,
Richard

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Nicolas,

2015ko martxoak 2an, Nicolas Goaziou-ek idatzi zuen:
> 
> Hello,
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> I decided to go ahead and see what I could make of it. The result has
>> been pushed to the org mode repo to the branch wip-cite-awe. (I didn’t
>> want to push to your branch without asking, but if you prefer I’ll do
>> that and delete my own.)
> 
> This is not *my* branch. However, I suggest to push only consensual
> features with documentation and tests there, and experiment features in
> other branches.

OK, I wasn’t sure what the etiquette was.

> 
>> The first issue is that the parser includes trailing punctuation in
>> “bare” @key citations.  So the following does not work as expected (the
>> :key includes the period): “This was demonstrated most recently by
>> @Smith2015.”  I’m not sure what the right approach is – one option
>> would be to say that keys can contain punctuation, but must end (and
>> begin) with an alphanumeric character.
> 
> I'll update the parser once there is a new syntax for keys. At the
> moment, it is correct wrt syntax.
> 
>> The second issue is that the :key property of the citation element
>> includes the @.  This is not right IMO: it’s a detail of the syntax.
>> And it means that consumers of the syntax, who might want to look up
>> the key in a database, will always have to remember to strip the @.
>> I’ve pushed a provisional fix for this in my branch.
> 
> Please apply it to wip-cite. A dedicated test would be nice, too.

OK.

> 
>> The code is very rough and ready, has lots of TODO comments in it, is
>> missing tests, documentation in the manual, etc.  Nonetheless, I want to
>> get feedback on it early, given that many people have already contributed
>> so much useful information to this discussion.
> 
> I didn't look closely at the code, but I suggest to use "org-cite.el"
> instead of "ox-cite.el". Even though this is only related to export at
> the moment, this library will also contain commands to manipulate
> citation objects. It also shorten prefix for these functions.

OK.  It will certainly save some typing.

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Rasmus]

[-- Attachment #1: Type: text/plain, Size: 4480 bytes --]

Looks cool Aaron.  Thanks!

Aaron Ecay <aaronecay@gmail.com> writes:

> The first issue is that the parser includes trailing punctuation in
> “bare” @key citations.  So the following does not work as expected (the
> :key includes the period): “This was demonstrated most recently by
> @Smith2015.”  I’m not sure what the right approach is – one option
> would be to say that keys can contain punctuation, but must end (and
> begin) with an alphanumeric character.

I also tried to solve this in the attached patch.  Feel free to use it in
the unlikely case that it adds anything to your fix (I didn't check).
Note, cf. my other recent post, this should probably be part of a
citations object rather than a citation object (assuming you take my idea
at good value).

> The second issue is that the :key property of the citation element
> includes the @.  This is not right IMO: it’s a detail of the syntax.
> And it means that consumers of the syntax, who might want to look up
> the key in a database, will always have to remember to strip the @.
> I’ve pushed a provisional fix for this in my branch.

I agree that the @ should not be part of the key.

> Citation modes are responsible for formatting the in-text citation
> (usually, a reference to a full citation stored elsewhere).  [...]
> Citation styles are responsible for formatting the citation in the
> bibliography [...]

Cool!


> -> How much is it worth trying to keep latex and the other backends
> together.

A lot!  In the sense that you get approximately the same output across
backends when using support citation commands.  When you are using citepos
or similar you are on your own, but a highlevel API would be nice.

> The current implementation uses some common functions (in ox-cite.el)
> for all backends, including latex.  However, latex basically does its
> own thing.  So it would be possible to include in ox-cite only code for
> non-latex backends, and then implement latex citations solely in
> ox-latex.  Separation would make the initial implementation very
> simple.  On the other hand, I worry that it would perpetuate the present
> situation where latex and non-latex citations are two separate
> universes.  On the third hand, serious latex users will probably just
> say you should use latex of you want nice citations, since latex is so
> much better.  ;)

I use LaTeX and in a recent funding request .doc (no x) was *mandatory*!
I'm told that some journals only accept word...

> -> How much of the non-latex citation code is it worth implementing in
> elisp.

As little as possible, though a highlevel API is nice, e.g. being able to
make a citepos using (concat (citeauthor cite) "'s" (citeyear cite)).

> Is it worth trying to put together a bare-bones elisp implementation,
> so people can have dependency-free bibliographies?  My inclination is to
> say “no,”

+1 (so "2 × no").

> to avoid duplicating the considerable effort put into the CSL
> universe (over 7,000 citation styles available!).  However, it also
> seems a bit wrong to have a core feature depend on a third party
> executable.

Afaik, we need latexml or mathtoweb.jar to turn math into mathml for odt.

> (How much it would slow down export to continually shell out to an
> external program is also an open question.)

Does it matter with async?

> I didn’t try to do anything about support for editing/inserting citations
> within org mode.  It would be good to know whether John Kitchin is willing
> to contribute bits of his org-ref code for that.  (I see on Worg that he
> has an FSF assignment on file, but I don’t want to take without asking –
> or do porting/integration work that he’s happy to do himself!)

I haven't tried org-ref, but isn't it just reftex?  If so something like
this

(add-to-list 'reftex-cite-format-builtin
               '(org "Org-mode citation"
                     ((?t . "[cite:%l]")
                      (?p . "[parencite:%l]"))))

Should work.  We'd automatically add subtypes and maybe make an
intelligent choice based on whether prefix and postfix is used.  Another
idea would be to use the ox-export dispatcher code, somehow...

> PS the code uses relatively new functions from the subr-x and let-alist
> libraries, so it probably works best on a recent-ish (past few months)
> trunk version of emacs.

In another post, Nicolas said we can target (at least?) 24.4 for v8.4.
I'm assuming citation won't make 8.3...

—Rasmus

-- 
Send from my Emacs

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-org-element.el-Add-subkeys-to-citation-objects.patch --]
[-- Type: text/x-diff, Size: 2773 bytes --]

From 60e7b587ccda20e63fe0d90ce27315831be27d76 Mon Sep 17 00:00:00 2001
From: Rasmus <rasmus@gmx.us>
Date: Mon, 2 Mar 2015 18:18:02 +0100
Subject: [PATCH] org-element.el: Add subkeys to citation objects

* org-element.el (org-element--set-regexps),
  (org-element-citation-parser),
  (org-element-citation-interpreter): Add citation subtypes.
---
 lisp/org-element.el | 19 ++++++++++++++-----
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/lisp/org-element.el b/lisp/org-element.el
index b341cef..7745558 100644
--- a/lisp/org-element.el
+++ b/lisp/org-element.el
@@ -197,8 +197,8 @@ specially in `org-element--object-lex'.")
 		      (format "\\[\\(?:%s\\)"
 			      (mapconcat
 			       #'identity
-			       (list "cite:"
-				     "(cite):"
+			       (list "cite\\(?:/\\([A-Za-z_-]+\\)\\)?:"
+				     "(cite\\(?:/\\([A-Za-z_-]+\\)\\)?):"
 				     "@[_A-Za-z][A-Za-z0-9:.#$%&-+?<>~/]*\\]"
 				     "fn:"
 				     "\\(?:[0-9]\\|\\(?:%\\|/[0-9]*\\)\\]\\)"
@@ -408,7 +408,7 @@ This alist also applies to secondary string.  For example, an
 still has an entry since one of its properties (`:title') does.")
 
 (defconst org-element-secondary-value-alist
-  '((citation :prefix :suffix)
+  '((citation :prefix :suffix :subtype)
     (headline :title)
     (inlinetask :title)
     (item :tag))
@@ -2721,7 +2721,8 @@ Assume point is at the beginning of the citation."
 	  (t
 	   (let ((begin (point))
 		 (before-end (with-syntax-table org-element--pair-square-table
-			       (ignore-errors (scan-lists (point) 1 0)))))
+			       (ignore-errors (scan-lists (point) 1 0))))
+		 (subtype (org-match-string-no-properties 1)))
 	     (save-excursion
 	       (search-forward ":")
 	       ;; Ignore blanks between cite type and prefix or key.
@@ -2741,6 +2742,9 @@ Assume point is at the beginning of the citation."
 				:post-blank (progn (goto-char before-end)
 						   (skip-chars-forward " \t"))
 				:end (point)))))
+		   (when subtype
+		     (org-element-put-property
+		      cite :subkey subkey))
 		   (when (< post-tag (match-beginning 0))
 		     (org-element-put-property
 		      cite :prefix
@@ -2765,7 +2769,12 @@ Assume point is at the beginning of the citation."
   "Interpret CITATION object as Org syntax.
 CONTENTS is nil."
   (concat "["
-	  (if (org-element-property :parentheticalp citation) "(cite):" "cite:")
+	  (format
+	   (if (org-element-property :parentheticalp citation)
+	       "(%s%s):" "%s%s:")
+	   "cite"
+	   (let ((subtype (org-element-property :subtype citation)))
+	     (if subtype (concat "/" subtype) "")))
 	  (org-element-interpret-data (org-element-property :prefix citation))
 	  (org-element-property :key citation)
 	  (org-element-interpret-data (org-element-property :suffix citation))
-- 
2.3.1

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Rasmus,

2015ko martxoak 2an, Rasmus-ek idatzi zuen:
> 
> Looks cool Aaron.  Thanks!
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> The first issue is that the parser includes trailing punctuation in
>> “bare” @key citations.  So the following does not work as expected (the
>> :key includes the period): “This was demonstrated most recently by
>> @Smith2015.”  I’m not sure what the right approach is – one option
>> would be to say that keys can contain punctuation, but must end (and
>> begin) with an alphanumeric character.
> 
> I also tried to solve this in the attached patch.  Feel free to use it in
> the unlikely case that it adds anything to your fix (I didn't check).
> Note, cf. my other recent post, this should probably be part of a
> citations object rather than a citation object (assuming you take my idea
> at good value).

It looks like the patch you sent also adds support for subtypes.  Rather
than try to distill the essential bit from it, I’ll wait for you and/or
Nicolas to get to it (there’s no particular hurry...)

I’m not sure I’m happy about the citations/citation proposal (under any
assignment of different names to the pieces).  A citations containing
only one citation is degenerate: it can never have a :prefix or :suffix
(these will rather be attached to the lone daughter citation).  That’s a
smell in my book.

It also makes the latex export more complicated: a one-daughter
citations should export as \textcite etc., whereas a multi-daughter one
will be \multicite.  From my perspective it would be more
straightforward to only have the citations wrapper be generated if there
are actually multiple citations.  But I don’t know how that would affect
the parser, so you should do what seems best to you.

Another tangentially related issue is what does (org-element-context)
return when point is in a multi-citation.  It would be nice if it
returned the citation daughter, rather than the wrapping citations
element.  This would make implementing goto-citation-at-point very
easy.

>> -> How much of the non-latex citation code is it worth implementing in
>> elisp.
> 
> As little as possible, though a highlevel API is nice, e.g. being able to
> make a citepos using (concat (citeauthor cite) "'s" (citeyear cite)).

I think I have an idea for this (though based on string templates rather
than list forms).

> I haven't tried org-ref, but isn't it just reftex?  

I haven’t either, but my impression is that, while perhaps based on
reftex, there’s a lot of spit and polish that goes into making it work
nicely.  It would be nice to be able to “steal” that because...

> If so something like this
> 
> (add-to-list 'reftex-cite-format-builtin
>                '(org "Org-mode citation"
>                      ((?t . "[cite:%l]")
>                       (?p . "[parencite:%l]"))))
> 
> Should work.  We'd automatically add subtypes and maybe make an
> intelligent choice based on whether prefix and postfix is used.  

Every time I’ve looked at the reftex internals I get grumpy, because
they are heavily tuned towards latex usecases, and involve some weird
cache structure which manipulates the plist of an interned symbol.  So
customizing it for org always seems precarious.  In the long term I’d be
happy if we built something out of more easily composable pieces.

> Another idea would be to use the ox-export dispatcher code, somehow...

If this was a composable piece I’d be happy (I haven’t looked at it).
There was also an attempt to factor the menu code out of magit (called
makey IIRC).  I’ve used it for some small things, but it wasn’t terribly
pleasant.

> 
>> PS the code uses relatively new functions from the subr-x and let-alist
>> libraries, so it probably works best on a recent-ish (past few months)
>> trunk version of emacs.
> 
> In another post, Nicolas said we can target (at least?) 24.4 for v8.4.
> I'm assuming citation won't make 8.3...

What about XEmacs?  That’s another stumbling block.

Thanks,

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Rasmus]

Hi,

> I’m not sure I’m happy about the citations/citation proposal (under any
> assignment of different names to the pieces).  A citations containing
> only one citation is degenerate: it can never have a :prefix or :suffix
> (these will rather be attached to the lone daughter citation).

This is one of my concerns.  Getting [cite: pre @k post] right.  I
mentioned it in another post, but Nicolas seemed not to worry about it.

> That’s a smell in my book.

I my book it's very pleasant since it means that I can use the same
function no matter the amount of "keys" within the citation.  It's always
a list that you map on.  You don't even need a cond, though in some cases
the length of the list may be nice to have.

> It also makes the latex export more complicated: a one-daughter
> citations should export as \textcite etc., whereas a multi-daughter one
> will be \multicite.  From my perspective it would be more
> straightforward to only have the citations wrapper be generated if there
> are actually multiple citations.  But I don’t know how that would affect
> the parser, so you should do what seems best to you.

With biblatex, which is the only latex backend that has sophisticated
notes functionality, this scheme makes it *super easy*!  Here's the latex
part of the hack I use for citation "in production":

;; type is e.g. textcite, and citations look like
;; [[cite: common pre; pre0 @key0 post0; pre1 @key1 post1; common post ]]
(if (org-export-derived-backend-p backend 'latex)
    (format "\\%s%s%s%s%s"
                    type (if (> (length citations) 1) "s" "")
                    common-pre common-post
                    (mapconcat
                     (lambda (cite)
                       (let ((pre  (org-string-nw-p (plist-get cite :pre)))
                             (post (org-string-nw-p (plist-get cite :post))))
                           (concat (cond
                                    ((and pre post)
                                     (format "[%s][%s]" pre post))
                                    (post (format "[%s]" post))
                                    (pre (format "[%s][]" pre))
                                    (t ""))
                                   (format "{%s}" (plist-get cite :key)))))
                             citations ""))
                   ⋯

It works well irrespective of the number of citations because a citation
is a list of keys.

> Another tangentially related issue is what does (org-element-context)
> return when point is in a multi-citation.  It would be nice if it
> returned the citation daughter, rather than the wrapping citations
> element.  This would make implementing goto-citation-at-point very
> easy.

This is a technical detail that I don't know enough about to have an
opinion on.  If it would not be as you say it should be easy enough to
extract :begin and :end from daughters and figure out which one point it
on.

>> As little as possible, though a highlevel API is nice, e.g. being able to
>> make a citepos using (concat (citeauthor cite) "'s" (citeyear cite)).
>
> I think I have an idea for this (though based on string templates rather
> than list forms).

OK.  Although I don't want to make you angry, do you know
reftex-format-citation?  It's what I use for citations outside of latex.

>> I haven't tried org-ref, but isn't it just reftex?  
>
> I haven’t either, but my impression is that, while perhaps based on
> reftex, there’s a lot of spit and polish that goes into making it work
> nicely.  It would be nice to be able to “steal” that because...

I think *that* part of reftex is fairly easy to bend.  Still, Reftex has
some quirkiness.  Initially I think it's OK.

Here's John's setup:

  https://github.com/jkitchin/org-ref/blob/master/org-ref.org#org-mode--reftex-setup

>  In the long term I’d be happy if we built something out of more easily
> composable pieces.

I agree.

>> Another idea would be to use the ox-export dispatcher code, somehow...
>
> If this was a composable piece I’d be happy (I haven’t looked at it).

Isn't it?  Maybe I remember wrongly.

> There was also an attempt to factor the menu code out of magit (called
> makey IIRC).  I’ve used it for some small things, but it wasn’t terribly
> pleasant.

Magit isn't part of ELPA...  There's the new hydra thing in ELPA (haven't
tried it), which seems similar to the export dispatcher.

> What about XEmacs?  That’s another stumbling block.

I don't know.  I don't even have it installed to be honest...

–Rasmus

-- 
Summon the Mothership!

Re: Citation syntax: a revised proposal

[Ken Mankoff]

On 2015-03-03 at 10:41, Rasmus <rasmus@gmx.us> wrote:
> ;; type is e.g. textcite, and citations look like
> ;; [[cite: common pre; pre0 @key0 post0; pre1 @key1 post1; common post ]]

My Biblatex keys are currently Author:YYYYFour-words-from-title. I don't think I'll ever have an author with "[" or "]" characters in their name. But will this work if I have an author with last-name "Cite"?

  -k.
  

Re: Citation syntax: a revised proposal

[Rasmus]

Hi,

Ken Mankoff <mankoff@gmail.com> writes:

> On 2015-03-03 at 10:41, Rasmus <rasmus@gmx.us> wrote:
>> ;; type is e.g. textcite, and citations look like
>> ;; [[cite: common pre; pre0 @key0 post0; pre1 @key1 post1; common post ]]
>
> My Biblatex keys are currently Author:YYYYFour-words-from-title. I
> don't think I'll ever have an author with "[" or "]" characters in
> their name. But will this work if I have an author with last-name
> "Cite"?

I think I'm misunderstanding.

The above was an example of the poor hack I use to support citations to
show Aaron that multicite and normal cite is really easy when the
individual references/daughters are stored in a list.

AFAIK, in the implementation that will be part of org eventually, both "-"
and "cite" would be supported as long as you don't have "[cite:" in the
name.  Then I don't know what would happen.

—Rasmus

-- 
I hear there's rumors on the, uh, Internets. . .

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> Another tangentially related issue is what does (org-element-context)
> return when point is in a multi-citation.  It would be nice if it
> returned the citation daughter, rather than the wrapping citations
> element.  This would make implementing goto-citation-at-point very
> easy.

This is an important issue that I think requires more thought.

I can see two ways of going here:

1) Introduce individual references (really, probably just keys) as
first-class objects, which are contained by citation objects.

2) Keep individual references/keys as properties of citation objects,
but store :begin/:end properties individually for them.

I don't really have an informed opinion here, but the second option
seems simpler to me, since it doesn't involve a new class of object (and
therefore doesn't require distinguishing a key-within-a-citation from a
key-which-is-a-citation, as in the `shortcut' syntax).

With the second option, given a (multi-)citation object and a position,
it is still pretty easy to tell which individual reference the position
is in, so it is still pretty easy to implement goto-citation-at-point.

Best,
Richard

Re: Citation syntax: a revised proposal

[Aaron Ecay]

[-- Attachment #1: Type: text/plain, Size: 4009 bytes --]

Hello all,

I’ve pushed an update to my branch.  The major change is to use
citeproc-java for the generation of the bibliography and the parsing of
names.  The former is straightforward.  For the latter, I have created a
CSL file which outputs author-year citations in an easy-to-parse format.
These are then slurped by org, and used to fill in printf-style
templates.  Some people mentioned using citations as generated by
citeproc-java directly.  However, I don’t believe this is reliable since
(as also mentioned), it is difficult to control whether a certain style
uses parentheses around a citation or not, whether the citation is
capitalized*, the insertion of prefixes/suffixes within the parentheses,
etc.  So I think the only solution is to implement the formatting of the
in-text portion of citations ourselves, and use citeproc-java only to
extract authors and years.

* NB neither my branch nor the parser currently handles capitalized
  citations either.

Using citeproc gives us for free sophisticated disambiguation of authors
that share a last name (by adding first initial) or works by the same
author in the same year (by adding a letter suffix to the year).

My code still does not implement numeric or footnote citations.

It’s hard to keep straight all of the traffic in citation-related
threads.  I’ll try to respond to what I see as the major points raised,
but I apologize if I inadvertently skip something.

Some people have talked about supporting other CSL processors.  I don’t
see much utility in that, since CSL is a standard that all processors
should implement faithfully.  I judge the java implementation to be the
most complete and cross-platform.  With respect to Zotero, it is possible,
through the “lookup” facility I’ve implemented, to implement fetching of
bibliography data from Zotero.  I merely do not want to talk to Zotero (or
other tools) for the *formatting* of the data.

Rasmus <http://mid.gmane.org/87r3sxubgl.fsf@gmx.us> writes about the
insertion of punctuation by biblatex.  I’ve noticed it too, and it’s a
thorny problem.  Perhaps the best and easiest solution is to say that
org-latex documents must do \renewcommand{\postnotedelim}{} in their
preamble.  (Thanks also to Rasmus for discussion of points I raised in
previous mails about reftex and org syntax.  I have no specific reply
but the responses were all extremely helpful.  I haven’t had time to act
on any of them because I’ve been concentrating on citeproc support.)

I haven’t updated the branch to the new multicitation syntax yet, but
thanks as always to Nicolas for working on it.  I think that is next on
my list, along with getting biblatex support completely ironed out.

I have not had much time to study Vaidheeswaran’s jabref integration
code.  In any case I would be hesitant to do so until there is
confirmation of the code’s copyright status, since he(?) sometimes posts
to the mailing list using an email address belonging to Jambunathan,
who at some points in the past was not willing to provide copyright
assignment.  However, it would be good to know:
1. Is any important functionality lost by using citeproc-java as the CSL
   processor, rather than jabref?
2. Is it possible to support for implement importing citations from
   jabref through the “lookup types” facility in my code?
3. How are citations formatted for export to ODT?  An example of the
   ODT/XML code for something like “as demonstrated by Smith (2015)”,
   where “Smith (2015)” has whatever fancy formatting a citation is
   expected to have in ODT, would be helpful.

I’m sorry that I do not have enough time to both work on the code and
participate very much in the discussion.  But reading it has been quite
helpful to me in creating the code, so thank you all.

Finally, the attachments demonstrate the output from ascii export of a
sample document + bibliography (in org-bibtex format).


[-- Attachment #2: doc.txt --]
[-- Type: text/plain, Size: 736 bytes --]

Aaron Ecay





1 Section one
=============

  Here is a citation (Smith 2015) .

  Here is another with parens J. Doe (2016b).

  Here is one with a prefix and suffix (see Smith 2015 and other such
  works).

  Here is one with a prefix and suffix and parens see Smith (2015 and
  other such works).

  Here’s a citation to demonstrate the disambiguation of names: (B. Doe
  2016) .

  And the disambiguation of years: (J. Doe 2016a) .


2 Bibliography
==============

  Currently the bibliography keyword doesn’t create its own section.

  - Doe, Bill. 2016. “Work Three.”
  - Doe, Jane. 2016a. “Work Two Bis.”
  - ———. 2016b. “Work Two.”
  - Smith, John. 2015. “Work One.”

[-- Attachment #3: doc.org --]
[-- Type: application/vnd.lotus-organizer, Size: 595 bytes --]

[-- Attachment #4: bib.org --]
[-- Type: application/vnd.lotus-organizer, Size: 526 bytes --]

[-- Attachment #5: Type: text/plain, Size: 15 bytes --]


--
Aaron Ecay

Re: Citation syntax: a revised proposal

[Rasmus]

Hi,

Aaron Ecay <aaronecay@gmail.com> writes:

> I’ve pushed an update to my branch.

Thank you Aaron!  I appreciate the work.  While the below may sound
bitter, it's not!

> These are then slurped by org, and used to fill in printf-style
> templates.  Some people mentioned using citations as generated by
> citeproc-java directly.  However, I don’t believe this is reliable since
> (as also mentioned), it is difficult to control whether a certain style
> uses parentheses around a citation or not, whether the citation is
> capitalized*, the insertion of prefixes/suffixes within the parentheses,
> etc.

If so, is there *any* advantage is using a processors over say
bibtex.el/reftex-cite.el/org-bibtex-parser.el?  We'd get a couple of more
backends, right?  But this step can presumably be handled by other tools
as necessary.

If we have to do processing to the extend you describe (or I imagine that
you describe), I'm not sure there's a point in relying on a external tool,
which we cannot easily extend as opposed to fixing bugs we might encounter
in reftex-cite or "forking"/reimplementing functions we need from there.

> So I think the only solution is to implement the formatting of the
> in-text portion of citations ourselves, and use citeproc-java only to
> extract authors and years.

See above.  I then don't see the point.

> Using citeproc gives us for free sophisticated disambiguation of authors
> that share a last name (by adding first initial) or works by the same
> author in the same year (by adding a letter suffix to the year).

That's a fair point, but I don't understand it.  I haven't looked at your
code!  I guess you'd have to generate the whole bibliography to get this
data?  It sounds messy...  Note, reftex.el + friends is probably equally
messy.  And I don't know if it offers any support for bibliographies.

What I gather is that a big selling point of csl is the plentifulness of
styles.  But you say we anyway need curated CSL styles to make it work.
But then, is there any advantage of it left?

A similar idea, explored on the list, is using Jabref as a processor.  I
don't know about the copyright status, though. 

> Some people have talked about supporting other CSL processors.  I don’t
> see much utility in that, since CSL is a standard that all processors
> should implement faithfully.

+1.

> 1. Is any important functionality lost by using citeproc-java as the CSL
>    processor, rather than jabref?

I guess Jabref supports odt about of the box.  I don't know what the
sophistication of Jabref style language is viz-a-viz CSL.  I don't
understand if your work aims to support CSL or an "approved" subset of
CSL.

> 2. Is it possible to support for implement importing citations from
>    jabref through the “lookup types” facility in my code?

This is major.  I don't want to Jabref to manage citation and I would
rather not give it write access to my citation library as it has encoding
issues.

> 3. How are citations formatted for export to ODT?  An example of the
>    ODT/XML code for something like “as demonstrated by Smith (2015)”,
>    where “Smith (2015)” has whatever fancy formatting a citation is
>    expected to have in ODT, would be helpful.

It's a big mystery.  tex4ht/oolatex also manages.  Some sort of internal
database is generated.  I don't know how styling is managed.

—Rasmus

-- 
What will be next?

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Rasmus,

Thanks for your comments.  Some replies:

2015ko martxoak 10an, Rasmus-ek idatzi zuen:
>> These are then slurped by org, and used to fill in printf-style
>> templates.  Some people mentioned using citations as generated by
>> citeproc-java directly.  However, I don’t believe this is reliable since
>> (as also mentioned), it is difficult to control whether a certain style
>> uses parentheses around a citation or not, whether the citation is
>> capitalized*, the insertion of prefixes/suffixes within the parentheses,
>> etc.
>
> If so, is there *any* advantage is using a processors over say
> bibtex.el/reftex-cite.el/org-bibtex-parser.el?  We'd get a couple of more
> backends, right?  But this step can presumably be handled by other tools
> as necessary.
>
> If we have to do processing to the extend you describe (or I imagine that
> you describe), I'm not sure there's a point in relying on a external tool,
> which we cannot easily extend as opposed to fixing bugs we might encounter
> in reftex-cite or "forking"/reimplementing functions we need from
> there.

I have made citeproc-java give output like:

Smith////2014
Doe////1999
Smith et al.////2005

I parse that into lists of (author, year) pairs by splitting on the
////.  Then I expose a template to elisp: “%p%a (%y%s)” (for prefix,
author, year, and suffix).  This template could be changed however it is
desired by a user: using square brackets instead of round parens, for
example.  I think people will likely want to customize things like this,
and it seems difficult to get such flexibility from CSL.

So far, this has proved much easier than trying to implement all the
“proper” citation functionality in elisp.

CSL also gives the bibliography basically for free, including support
for zillions of highly specific formats.  (We do have to convert it from
HTML, but it is highly structured, in the sense that only a few tags and
styles can occur.  So the conversion can basically just handle those
cases one-by-one.)

>
>> So I think the only solution is to implement the formatting of the
>> in-text portion of citations ourselves, and use citeproc-java only to
>> extract authors and years.
>
> See above.  I then don't see the point.

It’s only the template for author-year in-text citations that I have
implemented in elisp.  CSL still gives us name parsing, disambiguation,
and bibliography formatting, which are each very complex.

> What I gather is that a big selling point of csl is the plentifulness of
> styles.  But you say we anyway need curated CSL styles to make it
> work.

I need just one such style, for our extraction of author + year for
in-text citations.  CSL styles are still used off-the-shelf for the
bibliography.

>> 2. Is it possible to support for implement importing citations from
>> jabref through the “lookup types” facility in my code?
>
> This is major.  I don't want to Jabref to manage citation and I would
> rather not give it write access to my citation library as it has encoding
> issues.

It would be one among many options: bibtex, org-bibtex, jabref, ...  You
would be free to pick the one(s) that hold your reference database, and
not bother with the others.

I hope that’s helpful,

--
Aaron Ecay

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Aloha Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> I have made citeproc-java give output like:
>
> Smith////2014
> Doe////1999
> Smith et al.////2005
>
> I parse that into lists of (author, year) pairs by splitting on the
> ////.  Then I expose a template to elisp: “%p%a (%y%s)” (for prefix,
> author, year, and suffix).  This template could be changed however it is
> desired by a user: using square brackets instead of round parens, for
> example.  I think people will likely want to customize things like this,
> and it seems difficult to get such flexibility from CSL.

The CSL web page says:

,-------------------------------------------------------------------------
| While CSL styles only define complete citations, e.g. “(Doe, 2002)”, the
| word processor plugins of Zotero, Mendeley, and Papers all allow you to 
| suppress the author(s) in individual citations, which would leave you   
| with just “(2002)”. You then have to write the author’s name by hand.   
`-------------------------------------------------------------------------

So, your implementation seems to be ahead in this respect.

I'd expect the author would want to have a list of templates for each
author-date CSL style, one for parenthetical citations, one for text
citations, another for genitive citations, etc.  It might even be nice
down the road to have a place where these could be stored and shared.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Aaron and all,

Aaron Ecay <aaronecay@gmail.com> writes:

> I’ve pushed an update to my branch.  The major change is to use
> citeproc-java for the generation of the bibliography and the parsing of
> names.

That is awesome!  Thank you for your work.

> The former is straightforward.  For the latter, I have created a CSL
> file which outputs author-year citations in an easy-to-parse format.
> These are then slurped by org, and used to fill in printf-style
> templates.  Some people mentioned using citations as generated by
> citeproc-java directly.  However, I don’t believe this is reliable
> since (as also mentioned), it is difficult to control whether a
> certain style uses parentheses around a citation or not, whether the
> citation is capitalized*, the insertion of prefixes/suffixes within
> the parentheses, etc.  So I think the only solution is to implement
> the formatting of the in-text portion of citations ourselves, and use
> citeproc-java only to extract authors and years.

I have actually been working on the same problem, using citeproc-hs as
the CSL processor instead of citeproc-java.  I took a (very) brief look
at your code; it seems like you are only communicating with
citeproc-java via command line arguments and stdout.  Is that right?

My approach to the problems you mention has been the following:
  1) Generate JSON from citation objects on the Org side.
  2) Pass that JSON to the processor via stdin.  
  3) Pass the output format, the CSL file, and the bibliography database
     to the processor via command line arguments.
  4) Return, to stdout, the formatted citations and the bibliography.
     These are formatted such that there is one citation or entry per line, 
     and a recognizable separator separates the citations from the
     bibliography. 

This allows passing formatting options for individual citations via the
JSON object for that citation, so it allows citeproc-hs to do more of
the work of formatting citations.

(See http://gsl-nagoya-u.net/http/pub/citeproc-doc.html#citation-data-object
for documentation of the citation data JSON format.)

I don't know whether this will ultimately be a good design, but the way
I am picturing it right now is that exporting citations will work sort
of like footnotes: the exporter will gather them all together as they
are encountered, then generate the JSON and run a single call to the CSL
processor at the end of the export process.  It can then replace the
citations in the document with the result from the CSL processor, and
insert the bibliography at the end of the document.

(The code is not very pretty yet, but it does generate citations and
bibliographies in both plain text and HTML, and it would be
straightforward to extend it to other output formats.  I can post it
somewhere if anyone is interested in taking a look.)

> Some people have talked about supporting other CSL processors.  I don’t
> see much utility in that, since CSL is a standard that all processors
> should implement faithfully.  

Indeed!  Though as you have observed, `should' and `already does' come
apart.  I doubt there are any implementations that are perfectly
complete.  So it may be worth thinking about how Org can talk to CSL
processors in a processor-independent way.  That way, different users
can use different CSL processors if one works better for their
particular document or environment.

I think the generate-and-pass-JSON approach is promising for that
reason.  That is what citeproc-js accepts as input (so maybe that is
what citeproc-java is doing internally?), and my code aims to allow
citeproc-hs to interpret the same JSON format as citeproc-js.  I don't
know Ruby, but I think it would be easy to make citeproc-ruby accept the
same JSON format.  Do you have a sense of how easy it would be to coax
citeproc-java into accepting JSON on stdin?  If it could be done, I
think that would give us a common format for talking to the
best-developed CSL implementations in a processor-independent way.

Best,
Richard

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Richard,

Thanks for your comments, and for your work on an implementation.

2015ko martxoak 10an, Richard Lawrence-ek idatzi zuen:
> I have actually been working on the same problem, using citeproc-hs as
> the CSL processor instead of citeproc-java.

This is an interesting approach.  What version of citeproc-hs are you
using?  The version under that name is no longer maintained, and I had
some trouble getting it to build.  The pandoc fork (under the name
pandoc-citeproc) seemed to me to lack command-line functionality.  (But
I only looked briefly and could have missed something.)

> I took a (very) brief look at your code; it seems like you are only
> communicating with citeproc-java via command line arguments and stdout.
> Is that right?

Yes.

>
> My approach to the problems you mention has been the following:
>   1) Generate JSON from citation objects on the Org side.
>   2) Pass that JSON to the processor via stdin.
>   3) Pass the output format, the CSL file, and the bibliography database
>      to the processor via command line arguments.
>   4) Return, to stdout, the formatted citations and the bibliography.
>      These are formatted such that there is one citation or entry per line,
>      and a recognizable separator separates the citations from the
>      bibliography.
>
> This allows passing formatting options for individual citations via the
> JSON object for that citation, so it allows citeproc-hs to do more of
> the work of formatting citations.
>
> (See http://gsl-nagoya-u.net/http/pub/citeproc-doc.html#citation-data-object
> for documentation of the citation data JSON format.)

Very interesting.  But it looks like the CSL standard does not
differentiate parenthesized/not and capitalized/not citations, whereas
biblatex (taken as the best representative of the latex family of
citation processors) does.  I think we have decided we need to support
these.  So we will always need to do some post-processing of the CSL
output.  Then the question arises (for example) whether it’s better to
let CSL/citeproc handle the prefix and suffix, or to do it ourselves.

I don’t think we can decide this without looking at a working “sketch”
of both implementations.  It would be very good to see your draft code.

>
> I don't know whether this will ultimately be a good design, but the way
> I am picturing it right now is that exporting citations will work sort
> of like footnotes: the exporter will gather them all together as they
> are encountered, then generate the JSON and run a single call to the CSL
> processor at the end of the export process.  It can then replace the
> citations in the document with the result from the CSL processor, and
> insert the bibliography at the end of the document.

My code does something similar.  It processes all citations at the
beginning of export and stashes the data in the info plist, so that
it’s available to transcoders during the “main” export process.  IDK if
footnotes are handled in the same way, or rather processed in a late
step after the transcoders.  But it’s six of one, half a dozen of the
other I think.

>
> (The code is not very pretty yet, but it does generate citations and
> bibliographies in both plain text and HTML, and it would be
> straightforward to extend it to other output formats.  I can post it
> somewhere if anyone is interested in taking a look.)

Please do!

>
>> Some people have talked about supporting other CSL processors.  I don’t
>> see much utility in that, since CSL is a standard that all processors
>> should implement faithfully.
>
> Indeed!  Though as you have observed, `should' and `already does' come
> apart.  I doubt there are any implementations that are perfectly
> complete.  So it may be worth thinking about how Org can talk to CSL
> processors in a processor-independent way.  That way, different users
> can use different CSL processors if one works better for their
> particular document or environment.

I take your point, but any differences in the implementation just make
it potentially harder to be processor-independent.  I think we should
tightly integrate with one processor, working around whatever warts it
may have.

>
> I think the generate-and-pass-JSON approach is promising for that
> reason.  That is what citeproc-js accepts as input (so maybe that is
> what citeproc-java is doing internally?), and my code aims to allow
> citeproc-hs to interpret the same JSON format as citeproc-js.

... hmm.  Do you mean you’ve written Haskell code?

> I don't know Ruby, but I think it would be easy to make citeproc-ruby
> accept the same JSON format.  Do you have a sense of how easy it would
> be to coax citeproc-java into accepting JSON on stdin?

My understanding is that citeproc-java in its current form can read JSON
database files (in addition to bibtex).  However, it does not accept JSON
to control output of citations – it merely allows passing a key, for
which a “default” citation will be generated (no prefix/suffix/author
suppression/...).

It would not be easy for me to extend it, because I’m not fluent in
Java.

Thanks,

--
Aaron Ecay

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Aaron and all,

I cleaned up my efforts a bit and posted them here:

https://github.com/wyleyr/org-citeproc

(This program is just a modified version of John MacFarlane's citeproc
program:

https://github.com/jgm/citeproc/

which reads JSON in a slightly different format, and produces JSON
instead of fully-rendered output.)

I wrote a few functions to produce JSON from (the current implementation
of) citations in Org; these live in json.el.  Note that I haven't been
able to test the new parser because I am on Emacs 23, so these have only
been tested with my hand-constructed Lisp objects, not with parser
output...they might need a little adjusting.  I also haven't done
anything to hook them up to the export process.

Aaron Ecay <aaronecay@gmail.com> writes:

> ... hmm.  Do you mean you’ve written Haskell code?

Yes, but not very much.  As long as I only have to write pure functions,
it seems to go alright. :) 

I went with Haskell purely for practical reasons:
  1) I at least know a little Haskell, and I've been wanting to learn more;
     I don't know Java, Ruby, or JS
  2) I already have the Haskell platform installed, but I don't have
     Java, Ruby, or a JS engine (other than a browser) installed

> 2015ko martxoak 10an, Richard Lawrence-ek idatzi zuen:
>> I have actually been working on the same problem, using citeproc-hs as
>> the CSL processor instead of citeproc-java.
>
> This is an interesting approach.  What version of citeproc-hs are you
> using?  The version under that name is no longer maintained, and I had
> some trouble getting it to build.  

I am in fact using the version under that name (I have not had trouble
installing/building it via cabal).  I went this way because it seemed to
have everything we'd need, and I wanted to avoid a dependency on all of
pandoc.  Maybe this was shortsighted of me, though...  

> The pandoc fork (under the name pandoc-citeproc) seemed to me to lack
> command-line functionality.  (But I only looked briefly and could have
> missed something.)

Well, pandoc-citeproc does read and produce JSON at the command line,
but it takes (and returns) a complete pandoc document.

I think what I've got would be pretty easy to port to pandoc-citeproc,
if maintenance of citeproc-hs is an issue.  If a full dependency on
pandoc is OK, we could also just use Pandoc's rendering functions for
the various backends we wish to support.

>> My approach to the problems you mention has been the following:
>>   1) Generate JSON from citation objects on the Org side.
>>   2) Pass that JSON to the processor via stdin.
>>   3) Pass the output format, the CSL file, and the bibliography database
>>      to the processor via command line arguments.
>>   4) Return, to stdout, the formatted citations and the bibliography.
>>      These are formatted such that there is one citation or entry per line,
>>      and a recognizable separator separates the citations from the
>>      bibliography.
>>
>> This allows passing formatting options for individual citations via the
>> JSON object for that citation, so it allows citeproc-hs to do more of
>> the work of formatting citations.
>>
>> (See http://gsl-nagoya-u.net/http/pub/citeproc-doc.html#citation-data-object
>> for documentation of the citation data JSON format.)
>
> Very interesting.  But it looks like the CSL standard does not
> differentiate parenthesized/not and capitalized/not citations, whereas
> biblatex (taken as the best representative of the latex family of
> citation processors) does.  

Yikes, really?  Surely the CSL standard has something to say about this.
It seems like it would be pretty useless otherwise, and there are all
those options for representing names...

Anyway, whether it's standard or not, citeproc-hs has an `authorInText'
option for citations, which I am using to capture the
parenthetical-vs-in-text distinction.  I haven't thought about
capitalization, though.

>> I don't know whether this will ultimately be a good design, but the way
>> I am picturing it right now is that exporting citations will work sort
>> of like footnotes: the exporter will gather them all together as they
>> are encountered, then generate the JSON and run a single call to the CSL
>> processor at the end of the export process.  It can then replace the
>> citations in the document with the result from the CSL processor, and
>> insert the bibliography at the end of the document.
>
> My code does something similar.  It processes all citations at the
> beginning of export and stashes the data in the info plist, so that
> it’s available to transcoders during the “main” export process.

Ah, that sounds like a much better idea than what I was thinking of.

Best,
Richard

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Aaron and all,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

>> What version of citeproc-hs are you using?  The version under that
>> name is no longer maintained, and I had some trouble getting it to
>> build.
>
> I am in fact using the version under that name (I have not had trouble
> installing/building it via cabal).  I went this way because it seemed to
> have everything we'd need, and I wanted to avoid a dependency on all of
> pandoc.  Maybe this was shortsighted of me, though...  

I've posted a new version that uses pandoc-citeproc rather than
citeproc-hs to do the CSL processing, and uses pandoc itself to render
the output.  This fixes some spacing issues I was having with
citeproc-hs, and made it trivial to add an OpenDocument writer.

I'll take some time this weekend to see if I can wire this together with
the Elisp Aaron wrote for the Org exporter side.

Code is still at: https://github.com/wyleyr/org-citeproc

Best,
Richard

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Aaron and all,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I'll take some time this weekend to see if I can wire this together with
> the Elisp Aaron wrote for the Org exporter side.

I've had some success with this.  I would not say that my efforts are
complete yet, but I thought I should send an update to let everyone
know.  I've published a branch here:

https://github.com/wyleyr/org-mode

which derives from Aaron's wip-cite-awe branch.  Basically, what I've
been able to do so far is process citations in a document via
org-citeproc, using a bibtex database file, then insert the processed
citations and bibliography in the document during export.  (In theory,
this should work with org-bibtex too, though I think I may have
introduced a bug...the library is not producing well-formed bibtex from
org-bibtex entries for me at the moment.)  It works pretty slick, at
least for the simple cases I've tested.
 
Aaron, now that I've had time to go through your code a bit more
carefully, here are a few comments:

In general, I like the design of the org-cite library!  It was pretty
easy for me to understand what's going on, and it was also pretty easy
for me to make it use org-citeproc, just by plugging in different helper
functions.  This speaks to the modularity of the code and a
well-designed API.  I like the general approach of pre-processing
citations via org-cite-export-prepare and stashing the information on
the info plist.  I also like the extensibility of the lookup types
mechanism.

I didn't quite understand all the code surrounding citation modes and
styles.  I didn't dive into this too much, because I just wanted to get
org-citeproc to process citations and produce a bibliography using a CSL
file.  Rather than figure out how to infer a CSL file from
CITATION_STYLE and CITATION_MODE, I just added a new keyword (CSL_FILE)
to allow specifying it directly.  I am not sure which approach is
better, but I certainly found it simpler to use CSL_FILE and let
org-citeproc deal with formatting.

(One relevant issue here, as you mentioned in an earlier post, is how
much we want to worry about keeping LaTeX and non-LaTeX backends in
sync.  If it's important that they stay very close, the high-level way
of specifying citation formatting via CITATION_STYLE and _MODE is
probably the way to go.  These can be mapped to a BibLaTeX style on the
LaTeX side, and a CSL file elsewhere.  It might still be good to have
low-level keywords like CSL_FILE and, e.g., LATEX_BIB_STYLE too.  The
LaTeX vs. non-LaTeX issue is also important for dealing with
multi-cites, which are not handled in this branch.  I am waiting to hear
back about the right way to get the list of references associated with a
citation before I tackle that.)

As for the lookup types mechanism, I think this works pretty well,
though I have one suggestion: lookup functions should just be
responsible for jumping to the entry referenced by a citation in a
BIBDB, not for returning data about it.  Returning data only really
makes sense if we need to produce a new bibliography file, as in
org-cite--make-bibtex, but there are better mechanisms for that case
(see org-bibtex-export-to-kill-ring).  Jumping to the referenced entry
enables this, and makes lookup functions useful interactively.

In general, I did find that the cl-lib idioms and other stuff from
external libraries made the code more difficult for me to understand
(though that's on me), and in at least one case, I had to do without one
of the libraries you are loading (namely, let-alist) to get the code to
run in a vanilla Emacs 24.  I would personally prefer it if this library
could stay away from bleeding-edge Emacs features, as I run Org from
master, but generally on an older Emacs (my main Emacs is the one in
Debian stable).

There are just a few points where I had to modify your code, or comment
it out, to get it to run in my setup.  As a result, I might have broken
some things due to lack of understanding.  In all cases, my intent was
just to make my new code run alongside the existing code via whatever
minimal change was necessary, not to remove what was working for you.

Best,
Richard

Re: Citation syntax: a revised proposal

[Andreas Leha]

Hi all,


Richard Lawrence <richard.lawrence@berkeley.edu> writes:
> Hi Aaron and all,
>
> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> I'll take some time this weekend to see if I can wire this together with
>> the Elisp Aaron wrote for the Org exporter side.
>
> I've had some success with this.  I would not say that my efforts are
> complete yet, but I thought I should send an update to let everyone
> know.  I've published a branch here:
>
> https://github.com/wyleyr/org-mode
>
> which derives from Aaron's wip-cite-awe branch.  Basically, what I've
> been able to do so far is process citations in a document via
> org-citeproc, using a bibtex database file, then insert the processed
> citations and bibliography in the document during export.  (In theory,
> this should work with org-bibtex too, though I think I may have
> introduced a bug...the library is not producing well-formed bibtex from
> org-bibtex entries for me at the moment.)  It works pretty slick, at
> least for the simple cases I've tested.
>  

[ ... ]

I have been following this thread from (quite) some distance as I am
very interested in more general citation support from orgmode.  Please
allow some basic questions:


1. For the LaTeX user
This change means that the LaTeX user can use org syntax for citations
rather than bare LaTeX (or links of some sort)?

And this org syntax could then (I guess is the future) be extended with
additional link-like functionality (maybe similar to org-ref)?

2. The non-LaTeX exports
These are all treated the same and will contain just text, that is
produced to mimic LaTeX's output to some extent?

I think I read some question about e.g. having zotero handling the
citations in the odt export.  Are there plans for such thing?

3. The database
As I understand the database currently is either bibtex or org?  But
that list could in principle be extended to zotero ... ?

And any other database X (zotero ...) would need translations X ->
bibtex to make the LaTeX export work with it as well?

4. Example
Could you post one of your examples?  I'd love to see the prototype in
action to have a proper picture of this.


Thanks to all working on this!
Andreas

Re: Citation syntax: a revised proposal

[Richard Lawrence]

[-- Attachment #1: Type: text/plain, Size: 6040 bytes --]

Hi Andreas,

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

> I have been following this thread from (quite) some distance as I am
> very interested in more general citation support from orgmode.  Please
> allow some basic questions:
>
> 1. For the LaTeX user
> This change means that the LaTeX user can use org syntax for citations
> rather than bare LaTeX (or links of some sort)?
>
> And this org syntax could then (I guess is the future) be extended with
> additional link-like functionality (maybe similar to org-ref)?

Yes, exactly.

> 2. The non-LaTeX exports
> These are all treated the same and will contain just text, that is
> produced to mimic LaTeX's output to some extent?

Well, that depends on what you mean by `just' text.  Citations can still
contain or be wrapped in markup that is specific to the output format,
like span tags or anchor tags in HTML.

> I think I read some question about e.g. having zotero handling the
> citations in the odt export.  Are there plans for such thing?

Maybe.  As far as I know, no one has done any work on Zotero integration
yet.  But Vaidheeswaran did some work to make JabRef (a different
reference database) handle citations in ODT export.

> 3. The database
> As I understand the database currently is either bibtex or org?  

Yes.

> But that list could in principle be extended to zotero ... ?

Yes, in principle.  Any database that can export to bibtex format is
supported to some degree, if we support that format.  Whether we want to
integrate with any particular reference database more tightly than that
hasn't really been discussed, as far as I'm aware.  I don't have a good
sense of what other software Org users rely on for this purpose.

> And any other database X (zotero ...) would need translations X ->
> bibtex to make the LaTeX export work with it as well?

Not necessarily.  Obviously, if you want to have bibtex/biblatex do the
processing of the citations and bibliography in the document, that is
required.  But there is another approach, which is the one that Pandoc
takes: directly rendering citations and bibliography into the output
.tex file.  If you're not relying on bibtex/biblatex to do the
rendering, you don't need to have the database in its format.

The org-citeproc tool I've been working on supports reading databases in
any of these formats (via pandoc-citeproc):

Format            File extension
------------      --------------
MODS              .mods
BibLaTeX          .bib
BibTeX            .bibtex
RIS               .ris
EndNote           .enl
EndNote XML       .xml
ISI               .wos
MEDLINE           .medline
Copac             .copac
JSON citeproc     .json 

org-citeproc (via pandoc-citeproc) should already be able to read
databases in these other formats.  It would be trivial to add a LaTeX
writer to org-citeproc that would allow rendering citations and
bibliographies directly, as Pandoc does (since org-citeproc is just a
small wrapper around pandoc and pandoc-citeproc).

> 4. Example
> Could you post one of your examples?  I'd love to see the prototype in
> action to have a proper picture of this.

A couple of other people (Vaidheeswaran, Aaron Ecay) have posted
examples in other messages.  Here's an example of a simple Org document
being processed by org-citeproc:

#+BEGIN_SRC org
    #+OPTIONS: toc:nil todo:t |:t
    #+TITLE: Org-Citeproc Test
    #+AUTHOR: Richard Lawrence
    #+LANGUAGE: en
    #+BIBDB: bibtex ~/Documents/philosophy/dissertation/build/dissertation.bib
    #+CSL_FILE: /tmp/chicago-author-date.csl

    Citations and Bibliography are supported using org-citeproc.

    Here's a couple of them.
    [(cite): @Brandom1994] [(cite): @Russell1919] [(cite): @Vaanaanen2011]

    And one textual cite to throw in the mix:
    [cite: See @Caponigro2003].

    #+BIBLIOGRAPHY: here

#+END_SRC
 
In plain text, the body renders like:

#+BEGIN_QUOTE
    Citations and Bibliography are supported using org-citeproc.

    Here's a couple of them.  (Brandom 1994) (Russell 2001) (Väänäänen 2011)

    And one textual cite to throw in the mix: Caponigro (See 2003).

    Brandom, Robert. 1994. _Making It Explicit_. Harvard University Press.

    Caponigro, Ivano. 2003. “Free Not to Ask: On the Semantics of Free Relatives and
    Wh-Words Cross-Linguistically.” PhD thesis, University of California, Los
    Angeles.

    Russell, Bertrand. 2001. “Descriptions.” In _The Philosophy of Language_, edited
    by A. P. Martinich, Fourth, 221–27. Oxford University Press.

    Väänäänen, Jouko. 2011. _Models and Games_. Vol. 132. Cambrdige Studies in
    Advanced Mathematics. Cambridge University Press.
#+END_QUOTE

And in HTML, the relevant parts of the body look like this (hopefully,
you can see this unescaped HTML in your mail/news reader):

#+BEGIN_QUOTE
    <p>
    Citations and Bibliography are supported using org-citeproc.
    </p>

    <p>
    Here's a couple of them.
    <span class="citation">(Brandom 1994)</span> <span class="citation">(Russell 2001)</span> <span class="citation">(Väänäänen 2011)</span>
    </p>

    <p>
    And one textual cite to throw in the mix:
    <span class="citation">Caponigro (See 2003)</span>.
    </p>

    <div class="bibliography"><div class="bibliography"><p>Brandom, Robert. 1994. <em>Making It Explicit</em>. Harvard University Press.</p><p>Caponigro, Ivano. 2003. “Free Not to Ask: On the Semantics of Free Relatives and Wh-Words Cross-Linguistically.” PhD thesis, University of California, Los Angeles.</p><p>Russell, Bertrand. 2001. “Descriptions.” In <em>The Philosophy of Language</em>, edited by A. P. Martinich, Fourth, 221–27. Oxford University Press.</p><p>Väänäänen, Jouko. 2011. <em>Models and Games</em>. Vol. 132. Cambrdige Studies in Advanced Mathematics. Cambridge University Press.</p></div>
    </div>
    </div>
#+END_QUOTE

I've attached the full files, too.

Note that citations containing multiple references are not supported at
the moment, but that is coming soon.

Best,
Richard


[-- Attachment #2: citetest.org --]
[-- Type: text/plain, Size: 467 bytes --]

#+OPTIONS: toc:nil todo:t |:t
#+TITLE: Org-Citeproc Test
#+AUTHOR: Richard Lawrence
#+LANGUAGE: en
#+BIBDB: bibtex ~/Documents/philosophy/dissertation/build/dissertation.bib
#+CSL_FILE: /tmp/chicago-author-date.csl

Citations and Bibliography are supported using org-citeproc.

Here's a couple of them.
[(cite): @Brandom1994] [(cite): @Russell1919] [(cite): @Vaanaanen2011]

And one textual cite to throw in the mix:
[cite: See @Caponigro2003].

#+BIBLIOGRAPHY: here

[-- Attachment #3: citetest.txt --]
[-- Type: text/plain, Size: 826 bytes --]

			  ___________________

			   ORG-CITEPROC TEST

			    Richard Lawrence
			  ___________________


Citations and Bibliography are supported using org-citeproc.

Here's a couple of them.  (Brandom 1994) (Russell 2001) (Väänäänen 2011)

And one textual cite to throw in the mix: Caponigro (See 2003).

Brandom, Robert. 1994. _Making It Explicit_. Harvard University Press.

Caponigro, Ivano. 2003. “Free Not to Ask: On the Semantics of Free Relatives and
Wh-Words Cross-Linguistically.” PhD thesis, University of California, Los
Angeles.

Russell, Bertrand. 2001. “Descriptions.” In _The Philosophy of Language_, edited
by A. P. Martinich, Fourth, 221–27. Oxford University Press.

Väänäänen, Jouko. 2011. _Models and Games_. Vol. 132. Cambrdige Studies in
Advanced Mathematics. Cambridge University Press.

[-- Attachment #4: citetest.html --]
[-- Type: text/html, Size: 3512 bytes --]

Re: Citation syntax: a revised proposal

[Andreas Leha]

Hi Richard,

Thank you very much for your detailed answer.

Richard Lawrence <richard.lawrence@berkeley.edu> writes:
> Hi Andreas,
>
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
>> I have been following this thread from (quite) some distance as I am
>> very interested in more general citation support from orgmode.  Please
>> allow some basic questions:
>>
>> 1. For the LaTeX user
>> This change means that the LaTeX user can use org syntax for citations
>> rather than bare LaTeX (or links of some sort)?
>>
>> And this org syntax could then (I guess is the future) be extended with
>> additional link-like functionality (maybe similar to org-ref)?
>
> Yes, exactly.
>
>> 2. The non-LaTeX exports
>> These are all treated the same and will contain just text, that is
>> produced to mimic LaTeX's output to some extent?
>
> Well, that depends on what you mean by `just' text.  Citations can still
> contain or be wrapped in markup that is specific to the output format,
> like span tags or anchor tags in HTML.
>

I meant especially link-like functionality in the exported document.
Just as references in a LaTeX generated PDF can link to the bibliography
(and back).

>> I think I read some question about e.g. having zotero handling the
>> citations in the odt export.  Are there plans for such thing?
>
> Maybe.  As far as I know, no one has done any work on Zotero integration
> yet.  But Vaidheeswaran did some work to make JabRef (a different
> reference database) handle citations in ODT export.
>
>> 3. The database
>> As I understand the database currently is either bibtex or org?
>
> Yes.
>
>> But that list could in principle be extended to zotero ... ?
>
> Yes, in principle.  Any database that can export to bibtex format is
> supported to some degree, if we support that format.  Whether we want to
> integrate with any particular reference database more tightly than that
> hasn't really been discussed, as far as I'm aware.  I don't have a good
> sense of what other software Org users rely on for this purpose.
>
>> And any other database X (zotero ...) would need translations X ->
>> bibtex to make the LaTeX export work with it as well?
>
> Not necessarily.  Obviously, if you want to have bibtex/biblatex do the
> processing of the citations and bibliography in the document, that is
> required.  But there is another approach, which is the one that Pandoc
> takes: directly rendering citations and bibliography into the output
> .tex file.  If you're not relying on bibtex/biblatex to do the
> rendering, you don't need to have the database in its format.
>
> The org-citeproc tool I've been working on supports reading databases in
> any of these formats (via pandoc-citeproc):
>
> Format            File extension
> ------------      --------------
> MODS              .mods
> BibLaTeX          .bib
> BibTeX            .bibtex
> RIS               .ris
> EndNote           .enl
> EndNote XML       .xml
> ISI               .wos
> MEDLINE           .medline
> Copac             .copac
> JSON citeproc     .json
>
> org-citeproc (via pandoc-citeproc) should already be able to read
> databases in these other formats.  It would be trivial to add a LaTeX
> writer to org-citeproc that would allow rendering citations and
> bibliographies directly, as Pandoc does (since org-citeproc is just a
> small wrapper around pandoc and pandoc-citeproc).
>

I see.  Very interesting.

>> 4. Example
>> Could you post one of your examples?  I'd love to see the prototype in
>> action to have a proper picture of this.
>
> A couple of other people (Vaidheeswaran, Aaron Ecay) have posted
> examples in other messages.  Here's an example of a simple Org document
> being processed by org-citeproc:
>

[ deleted the examples ]

Thank you very much.  These look really nice!

>
> Note that citations containing multiple references are not supported at
> the moment, but that is coming soon.
>
> Best,
> Richard

This citation support will make my lock-in into orgmode perfect, I
guess.....

Thanks again,
Andreas

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Andreas,

Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:

>>> 2. The non-LaTeX exports
>>> These are all treated the same and will contain just text, that is
>>> produced to mimic LaTeX's output to some extent?
>>
>> Well, that depends on what you mean by `just' text.  Citations can still
>> contain or be wrapped in markup that is specific to the output format,
>> like span tags or anchor tags in HTML.
>>
>
> I meant especially link-like functionality in the exported document.
> Just as references in a LaTeX generated PDF can link to the bibliography
> (and back).

I don't see why this couldn't be done, at least in the direction from
link to bibliography entry.  (I don't think the other direction will
work, as the relationship is generally one-many.)  I'll think about the
best way to add this; the problem is adding unique identifiers to
bibliography entries when generating them, which hopefully shouldn't be
too difficult.

Best,
Richard

Re: Citation syntax: a revised proposal

[Matt Price]

[-- Attachment #1: Type: text/plain, Size: 7585 bytes --]

On Tue, Mar 17, 2015 at 12:26 PM, Richard Lawrence <
richard.lawrence@berkeley.edu> wrote:

> Hi Andreas,
>
> Andreas Leha <andreas.leha@med.uni-goettingen.de> writes:
>
> > I have been following this thread from (quite) some distance as I am
> > very interested in more general citation support from orgmode.  Please
> > allow some basic questions:
> >
> > 1. For the LaTeX user
> > This change means that the LaTeX user can use org syntax for citations
> > rather than bare LaTeX (or links of some sort)?
> >
> > And this org syntax could then (I guess is the future) be extended with
> > additional link-like functionality (maybe similar to org-ref)?
>
> Yes, exactly.
>
> > 2. The non-LaTeX exports
> > These are all treated the same and will contain just text, that is
> > produced to mimic LaTeX's output to some extent?
>
> Well, that depends on what you mean by `just' text.  Citations can still
> contain or be wrapped in markup that is specific to the output format,
> like span tags or anchor tags in HTML.
>
> > I think I read some question about e.g. having zotero handling the
> > citations in the odt export.  Are there plans for such thing?
>
> Maybe.  As far as I know, no one has done any work on Zotero integration
> yet.  But Vaidheeswaran did some work to make JabRef (a different
> reference database) handle citations in ODT export.
>
> > 3. The database
> > As I understand the database currently is either bibtex or org?
>
> Yes.
>
> > But that list could in principle be extended to zotero ... ?
>
> Yes, in principle.  Any database that can export to bibtex format is
> supported to some degree, if we support that format.  Whether we want to
> integrate with any particular reference database more tightly than that
> hasn't really been discussed, as far as I'm aware.  I don't have a good
> sense of what other software Org users rely on for this purpose.
>

Just a note about Zotero:  I think for most of us, the reason to export
into ODT and/or DOC is to circulate a paper either for review or
collaboration.  Either case will likely involve some revision to citations,
which would ideally be handled through Zotero.  for this to work properly,
we would want to export a full Zotero citation.  To do that, we would
likely need to speak directly to Zotero iteself.  Here is a snippet of xml
from a Zotero-generated citation in an odt document:
--------------
      <text:p text:style-name="P1">Here is a cite.
      <text:note text:id="ftn0" text:note-class="footnote">
        <text:note-citation>1
        </text:note-citation>
        <text:note-body>
          <text:p text:style-name="Footnote">
            <text:reference-mark-start text:name="ZOTERO_ITEM CSL_CITATION
{&quot;citationID&quot;:&quot;ECYx4ePh&quot;,&quot;properties&quot;:{&quot;formattedCitation&quot;:&quot;{\\rtf
Lea Schick, \\uc0\\u8220{}Bodies, Embodiment and Ubiquitous
Computing\\uc0\\u8221{} 21, no. 1 (March 1, 2010): 63\\uc0\\u8211{}69,
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml.}&quot;,&quot;plainCitation&quot;:&quot;Lea
Schick, “Bodies, Embodiment and Ubiquitous Computing” 21, no. 1 (March 1,
2010): 63–69,
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml.&quot;
},&quot;citationItems&quot;:[{&quot;id&quot;:5262,&quot;uris&quot;:[&quot;
http://zotero.org/users/20/items/AJGK5FDE&quot;],&quot;uri&quot;:[&quot;
http://zotero.org/users/20/items/AJGK5FDE&quot;],&quot;itemData&quot;:{&quot;id&quot;:5262,&quot;type&quot;:&quot;article-journal&quot;,&quot;title&quot;:&quot;Bodies,
embodiment and ubiquitous
computing&quot;,&quot;page&quot;:&quot;63-69&quot;,&quot;volume&quot;:&quot;21&quot;,&quot;issue&quot;:&quot;1&quot;,&quot;URL&quot;:&quot;
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml&quot
;,&quot;ISSN&quot;:&quot;14626268&quot;,&quot;author&quot;:[{&quot;family&quot;:&quot;Schick&quot;,&quot;given&quot;:&quot;Lea&quot;}],&quot;issued&quot;:{&quot;date-parts&quot;:[[&quot;2010&quot;,3,1]]}}}],&quot;schema&quot;:&quot;
https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot;}
RNDXQmSceXdkV"/>
            <text:span text:style-name="T1">Lea Schick, “Bodies, Embodiment
and Ubiquitous Computing” 21, no. 1 (March 1, 2010): 63–69,
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml.
            </text:span>
            <text:reference-mark-end text:name="ZOTERO_ITEM CSL_CITATION
{&quot;citationID&quot;:&quot;ECYx4ePh&quot;,&quot;properties&quot;:{&quot;formattedCitation&quot;:&quot;{\\rtf
Lea Schick, \\uc0\\u8220{}Bodies, Embodiment and Ubiquitous
Computing\\uc0\\u8221{} 21, no. 1 (March 1, 2010): 63\\uc0\\u8211{}69,
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml.}&quot;,&quot;plainCitation&quot;:&quot;Lea
Schick, “Bodies, Embodiment and Ubiquitous Computing” 21, no. 1 (March 1,
2010): 63–69,
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml.&quot;
},&quot;citationItems&quot;:[{&quot;id&quot;:5262,&quot;uris&quot;:[&quot;
http://zotero.org/users/20/items/AJGK5FDE&quot;],&quot;uri&quot;:[&quot;
http://zotero.org/users/20/items/AJGK5FDE&quot;],&quot;itemData&quot;:{&quot;id&quot;:5262,&quot;type&quot;:&quot;article-journal&quot;,&quot;title&quot;:&quot;Bodies,
embodiment and ubiquitous
computing&quot;,&quot;page&quot;:&quot;63-69&quot;,&quot;volume&quot;:&quot;21&quot;,&quot;issue&quot;:&quot;1&quot;,&quot;URL&quot;:&quot;
http://resolver.scholarsportal.info/resolve/14626268/v21i0001/63_beauc.xml&quot
;,&quot;ISSN&quot;:&quot;14626268&quot;,&quot;author&quot;:[{&quot;family&quot;:&quot;Schick&quot;,&quot;given&quot;:&quot;Lea&quot;}],&quot;issued&quot;:{&quot;date-parts&quot;:[[&quot;2010&quot;,3,1]]}}}],&quot;schema&quot;:&quot;
https://github.com/citation-style-language/schema/raw/master/csl-citation.json&quot;}
RNDXQmSceXdkV"/>
          </text:p>
        </text:note-body>
      </text:note>
      </text:p>
-------------------

Among other pieces of data, it contains a reference to a zotero URL which i
don't think org-citeproc will know about.  Can you describe what would be
necessary to get text like this into an ODT, using ythe code you've been
working on?  THis conversation has gone a bit beyond my competence/comfort
level, but I would be interested i nhelping a bit as soon as our semester
is over here (just a couple of weeks left).

Thanks,
Matt



> > And any other database X (zotero ...) would need translations X ->
> > bibtex to make the LaTeX export work with it as well?
>
> Not necessarily.  Obviously, if you want to have bibtex/biblatex do the
> processing of the citations and bibliography in the document, that is
> required.  But there is another approach, which is the one that Pandoc
> takes: directly rendering citations and bibliography into the output
> .tex file.  If you're not relying on bibtex/biblatex to do the
> rendering, you don't need to have the database in its format.
>
> The org-citeproc tool I've been working on supports reading databases in
> any of these formats (via pandoc-citeproc):
>
> Format            File extension
> ------------      --------------
> MODS              .mods
> BibLaTeX          .bib
> BibTeX            .bibtex
> RIS               .ris
> EndNote           .enl
> EndNote XML       .xml
> ISI               .wos
> MEDLINE           .medline
> Copac             .copac
> JSON citeproc     .json
>
>

[-- Attachment #2: Type: text/html, Size: 10682 bytes --]

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Matt,

Matt Price <moptop99@gmail.com> writes:

> Just a note about Zotero:  I think for most of us, the reason to export
> into ODT and/or DOC is to circulate a paper either for review or
> collaboration.  Either case will likely involve some revision to citations,
> which would ideally be handled through Zotero.  for this to work properly,
> we would want to export a full Zotero citation.  To do that, we would
> likely need to speak directly to Zotero iteself.  
> ...
> Among other pieces of data, it contains a reference to a zotero URL which i
> don't think org-citeproc will know about.  Can you describe what would be
> necessary to get text like this into an ODT, using ythe code you've been
> working on?  

Hmm, interesting.  It looks like the ODT XML encodes the actual JSON
representing the citation, with the Zotero URL you refer to going in the
"uri" or "uris" property.  What is this used for, exactly?  I get an
Access Denied error when I try to visit the URL...is it something you
can share with collaborators?

I'm not sure exactly what's required to get this information through a
processing pipeline that looks like Zotero -> Org -> org-citeproc ->
pandoc-citeproc -> pandoc -> org-citeproc -> Org.  The question is
whether the pandoc(-citeproc) data structures have slots for URLs
associated with a work in the reference database, and whether the
rendering functions will pay attention to those URLs.  

I know that Pandoc is capable of reading in citation data that includes
URLs, but I don't know whether that URL information makes it to the
other side of the pandoc-citeproc CSL processing.  It may (in fact,
probably does) depend on the CSL stylesheet.  I would need to look into
this a bit, but it's possible that with an appropriate stylesheet, it's
pretty trivial to generate output like this.

Another option would be to post-process the org-citeproc output within
the Org exporter.  If Org has got the data from Zotero, wrapping the
actual citation in <text:reference-mark-start ...> and
<text:reference-mark-end ...> tags that contain this data shouldn't be
too hard to do from Elisp.

Finally, a third option is not to use org-citeproc at all in this case.
If you are already using Zotero, which contains its own CSL processor
(citeproc-js), then you may be able to just use Zotero to render the
citations and bibliography, by passing it JSON data.  The zotxt plugin
might help here: https://bitbucket.org/egh/zotxt/src.  In that case, the
processing pipeline would be much simpler:
Org -> (zotxt ->) Zotero (-> zotxt) -> Org. 

The nice thing about using the citeproc-js JSON format as the input to
org-citeproc is that it makes having this alternative pipeline pretty
simple.  I already have Elisp functions to generate this JSON from an
Org citation object (see the *-to-json functions in the org-citeproc
repo).  Setting up this alternative pipeline would involve figuring out
how to send that data to Zotero and how to parse the response, which
would basically mean writing a function parallel to
org-cite--process-citations (in the org-cite library in my
wip-cite-org-citeproc branch) that talks to Zotero instead of
org-citeproc.

Best,
Richard

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Aloha Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> I think these various applications of citations, and others not yet
> mentioned or thought of, are best represented as binary switches.  Many
> of these distinctions will factor well into independent implementations.
> For example, a citation that is :footnote t can (probably) be generated
> by taking the citation, whatever it is, and wrapping it in
> \footnote{...}.  (For the latex case; other backends will have different
> specifics but the idea is the same.)  If this is implemented in terms of
> subtypes, it will lead to an explosion of 2^n subtypes being necessary.

BibLaTeX has 6 standard "subtypes", which it calls "standard commands".

A citation style can provide any number of specialized commands in
addition to the 6 standard commands.

The various citation styles that ship with BibLaTeX together include
seven specialized commands, for a total of 13.

In this design, the potential explosion in subtypes has been pretty well
kept in check.  Does that make the design of BibLaTeX a good model for
Org mode?

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Thomas,

2015ko otsailak 25an, "Thomas S. Dye"-ek idatzi zuen:
> 
> BibLaTeX has 6 standard "subtypes", which it calls "standard commands".
> 
> A citation style can provide any number of specialized commands in
> addition to the 6 standard commands.
> 
> The various citation styles that ship with BibLaTeX together include
> seven specialized commands, for a total of 13.

I count roughly 50 commands in sections 3.7.1 – 3.7.6 of the biblatex
user’s manual (version 2.9a of 24/06/2014).  Some of these are quite
esoteric, of course, but they are all provided.

> 
> In this design, the potential explosion in subtypes has been pretty well
> kept in check.  Does that make the design of BibLaTeX a good model for
> Org mode?

I don’t know, but I suspect not.  Latex allows users to create powerful
macros, but has relatively few built-in niceties (some are provided by
auctex and friends, but that’s separate).  Org’s macro facilities,
though also powerful, are not well-integrated into its considerable
interactive features.

By way of illustration, Biblatex (AFAICT) doesn’t provide a possessive
citation command, which was mentioned by someone in this thread (or its
predecessor) as a desideratum.  I’d expect a savvy latex user to put in
their preamble:

\newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}

That doesn’t really work in org.  (It could be put together with an org
macro, but would lose the kind of click-to-view functionality that
org-ref already provides and which would be ported to the new syntax as
well.)

Org needs to be smarter about anticipating users’ needs, because it
doesn’t rely on them to program their own solution using the markup
language.  And, insofar as all 50+ biblatex commands are actually
needed, it would be good to see if it’s possible to cut them into more
digestible chunks.

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Hi Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Thomas,
>
> 2015ko otsailak 25an, "Thomas S. Dye"-ek idatzi zuen:
>> 
>> BibLaTeX has 6 standard "subtypes", which it calls "standard commands".
>> 
>> A citation style can provide any number of specialized commands in
>> addition to the 6 standard commands.
>> 
>> The various citation styles that ship with BibLaTeX together include
>> seven specialized commands, for a total of 13.
>
> I count roughly 50 commands in sections 3.7.1 – 3.7.6 of the biblatex
> user’s manual (version 2.9a of 24/06/2014).  Some of these are quite
> esoteric, of course, but they are all provided.

>> 
>> In this design, the potential explosion in subtypes has been pretty well
>> kept in check.  Does that make the design of BibLaTeX a good model for
>> Org mode?
>
> I don’t know, but I suspect not.  Latex allows users to create powerful
> macros, but has relatively few built-in niceties (some are provided by
> auctex and friends, but that’s separate).  Org’s macro facilities,
> though also powerful, are not well-integrated into its considerable
> interactive features.
>
> By way of illustration, Biblatex (AFAICT) doesn’t provide a possessive
> citation command, which was mentioned by someone in this thread (or its
> predecessor) as a desideratum.  I’d expect a savvy latex user to put in
> their preamble:
>
> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}
>
> That doesn’t really work in org.  (It could be put together with an org
> macro, but would lose the kind of click-to-view functionality that
> org-ref already provides and which would be ported to the new syntax as
> well.)
>
> Org needs to be smarter about anticipating users’ needs, because it
> doesn’t rely on them to program their own solution using the markup
> language.  And, insofar as all 50+ biblatex commands are actually
> needed, it would be good to see if it’s possible to cut them into more
> digestible chunks.

OK, you folks know this much better than I do.  As an author, I'd much
rather put together a citation style once and remember (or look up) a
funky command name, than try to remember how to construct the style each
time I make a citation.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Stefan Nobis]

Aaron Ecay <aaronecay@gmail.com> writes:

> I count roughly 50 commands in sections 3.7.1 – 3.7.6 of the
> biblatex user’s manual (version 2.9a of 24/06/2014). Some of these
> are quite esoteric, of course, but they are all provided.

There are many commands (and even more private commands are possible)
in order to help reproducing all the various citation styles out there.

The critical question for org and org users is: How many different
citation commands are needed in a single document (or needed by a
single author within all her documents)?

If no single author ever needs more than about a dozen different
commands (including variations like the genetive versions), the
[cite:subcommand ...] syntax should suffice.

> By way of illustration, Biblatex (AFAICT) doesn’t provide a
> possessive citation command, which was mentioned by someone in this
> thread (or its predecessor) as a desideratum. I’d expect a savvy
> latex user to put in their preamble:

> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}

This is what the subcommand is for. An author may define "poss" as a
subcommand and use [cite:poss ...]. Then all the nice gimmicks will
still work.

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Stefan,

Stefan Nobis <stefan-ml@snobis.de> writes:

> Aaron Ecay <aaronecay@gmail.com> writes:
>
>> I count roughly 50 commands in sections 3.7.1 – 3.7.6 of the
>> biblatex user’s manual (version 2.9a of 24/06/2014). Some of these
>> are quite esoteric, of course, but they are all provided.
>
> There are many commands (and even more private commands are possible)
> in order to help reproducing all the various citation styles out there.
>
> The critical question for org and org users is: How many different
> citation commands are needed in a single document (or needed by a
> single author within all her documents)?

These are very different questions.  I think we need to answer the
second, since the syntax for citations will be common to all documents.

> If no single author ever needs more than about a dozen different
> commands (including variations like the genetive versions), the
> [cite:subcommand ...] syntax should suffice.

Yes, I don't disagree.  But how do we know that no single author ever
needs more than a dozen different types/commands, across all her
documents, now and in the future?  I for one do not feel comfortable
making this judgment a priori.  As Aaron said, it's an empirical
question.

>> By way of illustration, Biblatex (AFAICT) doesn’t provide a
>> possessive citation command, which was mentioned by someone in this
>> thread (or its predecessor) as a desideratum. I’d expect a savvy
>> latex user to put in their preamble:
>
>> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}
>
> This is what the subcommand is for. An author may define "poss" as a
> subcommand and use [cite:poss ...]. Then all the nice gimmicks will
> still work.

If I understand correctly, Aaron and I have two worries about this
approach.  First, doing things this way potentially leads to a lot of
redundancy in the code you have to write when defining subtypes, which
is a maintenance hassle and an extra burden on users.  Second, it means
that an author who has to deal with a lot of `uncommon' commands has to
remember a lot of subtype names, which are likely to be very similar,
because they represent overlapping groups of options.

Look at, e.g., the commands required in BibLaTeX to support citing
multi-volume works.  There are 24(!) of them, because every one of them
is simply a multi-volume version of other combinations of options that
BibLaTeX supports (multi-cite or not? parenthetical or in-text?
footnoted or not? etc.).

For my part, if I were citing multi-volume works a lot, I would
appreciate being able to express all those options using orthogonal
distinctions.  Then I could just add

{... :volume 2 ...}

at the end of a citation that already expresses most of those options in
the [cite: ...] part of the syntax.  I would personally prefer *not* to
have to write

[cite/SUBTYPE: ...]

and remember which, among those 24 command types, I need to use (and
define) here.  

(Also, note that this syntax does not have a way to pass additional
arguments like the volume, so it *can't* say "cite volume 2", even if
with a subtype label it can say "cite this multi-volume work".  Citing a
volume in a multi-volume work seems like it might require some ugly
hacking on the subtypes approach, but I have no idea how often people
actually use such citations.)

On the other hand, Tom says that he prefers the latter way of working,
even if it requires a lot of subtypes.  This is because it makes the
subtype easy to find-and-replace when changing a document from one
citation style to another.  Also, it is simpler to write custom handlers
that dispatch just on the subtype, rather than on a plist of options.

So there are considerations on both sides.  In the end, I prefer the 

[cite: ...]{:key val ...}

syntax because:

  1) it supports both styles of working (subtypes can be represented
     like `:type fvolcites')
  2) it allows passing additional arguments (e.g., volume number)
  3) something like this is needed anyway elsewhere in Org

but I recognize there are some drawbacks to this over the simpler
approach of subtype labels.
  
Best,
Richard

Re: Citation syntax: a revised proposal

[Rasmus]

Aaron Ecay <aaronecay@gmail.com> writes:

>> In this design, the potential explosion in subtypes has been pretty well
>> kept in check.  Does that make the design of BibLaTeX a good model for
>> Org mode?
>
> I don’t know, but I suspect not.  Latex allows users to create powerful
> macros, but has relatively few built-in niceties (some are provided by
> auctex and friends, but that’s separate).  Org’s macro facilities,
> though also powerful, are not well-integrated into its considerable
> interactive features.
>
> By way of illustration, Biblatex (AFAICT) doesn’t provide a possessive
> citation command, which was mentioned by someone in this thread (or its
> predecessor) as a desideratum.
                    ^^^^^^^^^^^

According to my dictionary, that might be a bit strong.  It was used an
example of why you need userwritten types.

> I’d expect a savvy latex user to put in
> their preamble:
>
> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}
>
> That doesn’t really work in org.  (It could be put together with an org
> macro, but would lose the kind of click-to-view functionality that
> org-ref already provides and which would be ported to the new syntax as
> well.)

And this is why I say that you need to be able to define you own subtypes.
Adding the naïve version of citepos should be something like:

(cite-mapcar (λ (cite) (concat (citeauthor cite) "'s" (citeyear cite))) cites)

—Rasmus

-- 
Got mashed potatoes. Ain't got no T-Bone. No T-Bone

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Aloha Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> By way of illustration, Biblatex (AFAICT) doesn’t provide a possessive
> citation command, which was mentioned by someone in this thread (or its
> predecessor) as a desideratum.  I’d expect a savvy latex user to put in
> their preamble:
>
> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}
>
> That doesn’t really work in org.  (It could be put together with an org
> macro, but would lose the kind of click-to-view functionality that
> org-ref already provides and which would be ported to the new syntax as
> well.)

#+name: define-citeposs-link
#+begin_src emacs-lisp :results silent :exports none
    (org-add-link-type 
     "citeposs" 'ebib-open-org-link
     (lambda (path desc format)
       (cond
        ((eq format 'html)
         (format "(<cite>%s</cite>)" path))
        ((eq format 'latex)
         (format "\\citeauthor{%s}'s (\\citeyear{%s})" path path)))))
#+end_src

I haven't tested this, but I think it would work in Org mode.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Tom,

2015ko martxoak 2an, "Thomas S. Dye"-ek idatzi zuen:
> 
> Aloha Aaron,
> 
> Aaron Ecay <aaronecay@gmail.com> writes:
> 
>> By way of illustration, Biblatex (AFAICT) doesn’t provide a possessive
>> citation command, which was mentioned by someone in this thread (or its
>> predecessor) as a desideratum.  I’d expect a savvy latex user to put in
>> their preamble:
>> 
>> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}
>> 
>> That doesn’t really work in org.  (It could be put together with an org
>> macro, but would lose the kind of click-to-view functionality that
>> org-ref already provides and which would be ported to the new syntax as
>> well.)
> 
> #+name: define-citeposs-link
> #+begin_src emacs-lisp :results silent :exports none
>     (org-add-link-type 
>      "citeposs" 'ebib-open-org-link
>      (lambda (path desc format)
>        (cond
>         ((eq format 'html)
>          (format "(<cite>%s</cite>)" path))
>         ((eq format 'latex)
>          (format "\\citeauthor{%s}'s (\\citeyear{%s})" path path)))))
> #+end_src
> 
> I haven't tested this, but I think it would work in Org mode.

The main thrust of this thread, and the previous one, has been to
define a citation syntax in org.  I don’t think anyone contests that
link-based solutions basically do the trick for Latex (only).  And
yet, (almost?) everyone has agreed that something more is needed, or
at least inevitable.  So I’m puzzled why you brought this up.

Are you trying to argue for subtype-based citations?  This is what I
infer from your messages (not just this one, and please do correct me if
I’m wrong).  If so, it would make it easier for me to understand you if
you said so outright.  My own opinion is that plists are better than
subtypes, but I’ve also said that I don’t think the correct decision can
be made a priori.  So don’t let me stop you (in general, not just Tom)
from going ahead with an implementation of subtypes, if that’s your
preferred solution.  I would like to help out with coding or testing,
though I haven’t yet been able to figure out where my efforts would be
best applied.  So if there’s something you (again, in general) think
would be helpful, don’t hesitate to ask.

Thanks,

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Aloha Aaron,

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Tom,
>
> 2015ko martxoak 2an, "Thomas S. Dye"-ek idatzi zuen:
>> 
>> Aloha Aaron,
>> 
>> Aaron Ecay <aaronecay@gmail.com> writes:
>> 
>>> By way of illustration, Biblatex (AFAICT) doesn’t provide a possessive
>>> citation command, which was mentioned by someone in this thread (or its
>>> predecessor) as a desideratum.  I’d expect a savvy latex user to put in
>>> their preamble:
>>> 
>>> \newcommand{\citeposs}[1]{\citeauthor{#1}’s (\citeyear{#1})}
>>> 
>>> That doesn’t really work in org.  (It could be put together with an org
>>> macro, but would lose the kind of click-to-view functionality that
>>> org-ref already provides and which would be ported to the new syntax as
>>> well.)
>> 
>> #+name: define-citeposs-link
>> #+begin_src emacs-lisp :results silent :exports none
>>     (org-add-link-type 
>>      "citeposs" 'ebib-open-org-link
>>      (lambda (path desc format)
>>        (cond
>>         ((eq format 'html)
>>          (format "(<cite>%s</cite>)" path))
>>         ((eq format 'latex)
>>          (format "\\citeauthor{%s}'s (\\citeyear{%s})" path path)))))
>> #+end_src
>> 
>> I haven't tested this, but I think it would work in Org mode.
>
> The main thrust of this thread, and the previous one, has been to
> define a citation syntax in org.  I don’t think anyone contests that
> link-based solutions basically do the trick for Latex (only).  And
> yet, (almost?) everyone has agreed that something more is needed, or
> at least inevitable.  So I’m puzzled why you brought this up.
>
> Are you trying to argue for subtype-based citations?  This is what I
> infer from your messages (not just this one, and please do correct me if
> I’m wrong).  If so, it would make it easier for me to understand you if
> you said so outright.  My own opinion is that plists are better than
> subtypes, but I’ve also said that I don’t think the correct decision can
> be made a priori.  So don’t let me stop you (in general, not just Tom)
> from going ahead with an implementation of subtypes, if that’s your
> preferred solution.  I would like to help out with coding or testing,
> though I haven’t yet been able to figure out where my efforts would be
> best applied.  So if there’s something you (again, in general) think
> would be helpful, don’t hesitate to ask.

I'm not able to understand the full implications of subtypes
vs. plists, so don't have a preferred solution along those lines.

I brought this up in reaction to "This doesn't really work in org."

I'm hoping for an Org mode citation syntax where there is an analogous
org-add-cite-type function, so I only have to remember the syntax one
time and can forget about it when I'm writing.

Sorry if this is noise and thanks for your patience.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Tom,

2015ko martxoak 2an, "Thomas S. Dye"-ek idatzi zuen:
> 
> I'm not able to understand the full implications of subtypes
> vs. plists, so don't have a preferred solution along those lines.
> 
> I brought this up in reaction to "This doesn't really work in org."
> 
> I'm hoping for an Org mode citation syntax where there is an analogous
> org-add-cite-type function, so I only have to remember the syntax one
> time and can forget about it when I'm writing.
> 
> Sorry if this is noise and thanks for your patience.

Not noise at all.  I think I understand where you are coming from
better.

Your comment inspired me to implement
org-export-cite-add-citation-mode-latex in the experimental citation
support I just pushed.  So you can do:

(org-export-cite-add-citation-mode-latex "tsd" "\\mycitecommand[%s][%s]{%s}" "\\myparencitecommand[%s][%s]{%s}")

Add to your document:

#+CITATION_MODE: tsd

And citations should just work with your chosen commands.  I’m sure when
advanced citation support comes along (whether from subtypes or plists),
a similarly simple wrapper can be implemented.

I also found your discussion of citation modes vs. styles at
<http://mid.gmane.org/m2k2z0mekp.fsf@tsdye.com> illuminating, so thanks
for that.

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Melanie Bacou]

Just want to point out RMarkdown/Pandoc implementation of bibliographies 
and citations here 
http://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html

One useful option is a `csl: biomed-central.csl` option in the preamble.

--Mel.


On 2/19/2015 12:06 PM, Richard Lawrence wrote:
> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:
>
>
>> I wasn't clear. Subtype should be interpreted by back-ends means it has
>> no impact on syntax. However a user should be able to dictate what the
>> back-end should do with it, much like `org-add-link-type'.
>>
>> A new library, e.g. "org-cite.el" would provide all the tools necessary
>> to build custom handlers for subtypes. Thus, power users can do whatever
>> they want with cite objects.
>
> Ah, OK, I think I see...so this is basically the second option, with
> users interpreting subtypes via a separate protocol, instead of via
> filters.  Right?
>
> What about this concern, then?
>
>>> But that kind of situation is exactly what the extra-info part
>>> of the syntax is for; in that case,
>>>
>>>    [cite/my-subtype: ...]
>>>
>>> would just be an exceptional way of writing
>>>
>>>    [cite: ...]{:type my-subtype}
>>>
>>> or whatever.  I'm not totally convinced yet that the convenience of the
>>> former is worth blurring the line between `stuff that definitely works
>>> on all backends' and `stuff that might not work on all backends'.
>
>
>>> We have already seen a couple of examples in this thread of properties
>>> that one might want to specify in a backend-agnostic way:
>>>    - special-case capitalization
>>>    - user-defined type/command/label/etc.
>>>
>>> Other things one might want to do include:
>>>    - indicating when a citation should be placed in a footnote
>>>    - extracting arbitrary bibliographic field(s)
>>
>> I disagree. These properties should be associated to the subtype.
>> Having two places to set them is asking for trouble.
>>
>> IMO, there is little incentive to define a set of options for a single
>> use. Creating a dedicated subtype /once/ makes more sense.
>
> Hmm, OK.  I agree in the abstract, but as far as I understand what a
> `subtype' is, I am not sure how well that idea applies in this case.  If
> a subtype encompasses a set of options, and the only way to specify
> those options is by specifying a subtype, then changing one option means
> changing to a whole different subtype, even when you want all the other
> options to stay the same.  That will lead to a lot of duplication in
> subtypes.  That in turn could lead to a lot of author frustration: now I
> have to remember exactly which subtype encompasses the set of options I
> need for this particular citation.
>
> Look at the LaTeX citation commands.  There, you have a different
> `type'/command for every possible combination of options, and there are
> many commands because there are many options to deal with special cases.
> In my mind, that is the wrong abstraction to emulate.  You end up making
> trips to the manual to figure out exactly which one you need.
>
> To me, it makes much more sense to have a basic citation command, and
> then a way to `answer some questions' to toggle various things related
> to special-case behavior, like:
>    - should it be footnoted?
>    - should it have special-case capitalization?
>    - should it cite the volume?
>    - should it deal with brackets in a context-sensitive manner?
>    - should it use the genitive form of the author's name?
>    - ...
>
> I would guess that the answers to these questions *usually* have nothing
> to do with one another, so it makes sense to be able to answer them
> independently, and change your answer to one independently of your
> answers to the others.  That's why I would want to be able to write
>
> [cite: ...]{:footnoted t :capitalized t :author-title t ...}
>
> rather than
>
> [cite/footnoted-capitalized-author-title: ...]
>
> In the former case, if I later figure out I don't need the `:capitalized
> t' part, that's easy to remove without changing the other options.  In
> the latter case, I need to remember what name I gave to the subtype for
> footnoted-but-not-capitalized-author-titles, or whatever.
>
> I don't know; maybe this is not a very serious concern in practice.
> Maybe, in practice, you generally only need one `special case' option at
> a time, so the number of subtypes will be small.  Still, I do not feel
> comfortable declaring *in advance* that that is so; and the bewildering
> array of LaTeX citation commands is at least some evidence that it
> isn't.  Thus, I am in favor of allowing the greater flexibility provided
> by the former syntax.
>
> That's not to say that subtypes never make sense: you are quite right
> that sometimes options *should* be grouped together into a subtype,
> namely, when it makes sense to set or remove them *conjointly*.  But
> that means the two syntaxes above actually have different purposes, even
> though for some range of cases either one would do the job.
>
> So although I would not be in favor of *only* allowing
>
> [cite/subtype: ...]
>
> I am basically OK with allowing this if we also allow the {:key val ...}
> syntax.  Still, the /subtype form is not needed if we also allow
>
> [cite: ...]{:type subtype}
>
> Best,
> Richard
>
>
>

-- 
Melanie BACOU
International Food Policy Research Institute
Snr. Program Manager, HarvestChoice
E-mail m.bacou@cgiar.org
Visit www.harvestchoice.org

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Melanie,

Melanie Bacou <mel@mbacou.com> writes:

> Just want to point out RMarkdown/Pandoc implementation of
> bibliographies and citations here
> http://rmarkdown.rstudio.com/authoring_bibliographies_and_citations.html

Thanks for joining the discussion!  Actually, the Pandoc/RMarkdown
syntax was sort of the starting point of the proposal we are currently
discussing, though it has evolved a lot since then because of Org users'
diverse needs.
 
> One useful option is a `csl: biomed-central.csl` option in the preamble.

Yes indeed.  We have yet to discuss syntax for this type of
citation-related metadata.  I think we should start another thread for
this.

Best,
Richard

Re: Citation syntax: a revised proposal

[Vaidheeswaran C]

On Tuesday 17 February 2015 10:48 PM, Richard Lawrence wrote:

> Another, more serious reason is that I work in a field where some
> journals do not accept LaTeX submissions, or disprefer them; so
> having some citation support in ODT export is important.)

You are lobbying for two things:

1. An improvement in existing citation syntax.
2. Citation support for ODT backend.

If you need help with ODT/JabRef integration, I am willing to lend a
hand.  (Only thing is) I would expect that someone hand-hold me wrt
what one wants in the final exporter on a case-by-case basis.  I would
rather build bottom-up, rather than top-down.

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Vaidheeswaran C <vaidheeswaran.chinnaraju@gmail.com> writes:

> If you need help with ODT/JabRef integration, I am willing to lend a
> hand.  (Only thing is) I would expect that someone hand-hold me wrt
> what one wants in the final exporter on a case-by-case basis.  I would
> rather build bottom-up, rather than top-down.

That's great, thanks!

I don't know if JabRef will be a/the tool we eventually want to
`bless'...from what you say, it sounds more limited than Zotero, though
I don't know because I haven't personally used either...but it's nice to
have volunteers!

Best,
Richard

Re: Citation syntax: a revised proposal

[Vaidheeswaran C]

On Wednesday 25 February 2015 09:59 AM, Richard Lawrence wrote:
> Vaidheeswaran C<vaidheeswaran.chinnaraju@gmail.com>  writes:
>
>> If you need help with ODT/JabRef integration, I am willing to lend a
>> hand.  (Only thing is) I would expect that someone hand-hold me wrt
>> what one wants in the final exporter on a case-by-case basis.  I would
>> rather build bottom-up, rather than top-down.
>
> That's great, thanks!
>
> I don't know if JabRef will be a/the tool we eventually want to
> `bless'...from what you say, it sounds more limited than Zotero, though
> I don't know because I haven't personally used either...but it's nice to
> have volunteers!

I agree that we should go for the best tool chain that is available. I
wish you success with your new initiative.

Do write to me if you ever feel that JabRef-based workflow will be of
some interest to you with your document production needs.

Re: Citation syntax: a revised proposal

[Tory S. Anderson]

+1

Thanks for the work substantiating the idea. 

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Hi everyone,
>
> Since discussion seems to have petered out on the previous thread (see:
> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
> go back over the discussion and write up a concrete proposal for
> citation syntax.
>
> This proposal represents my attempt to formulate a syntax that is easy
> to read, easy to parse, and covers all the use-cases that people
> mentioned as being important.  It is surely not perfect, but I learned a
> lot from the previous thread, and I hope something like this will serve
> the community's needs.
>
> The proposal is below, both inline (for easy quoting) and attached (for
> easy reading).  To keep it relatively short, I have mostly not explained
> my reasoning for the choices I made, but I am happy to do so here if
> anyone has questions.
>
> I welcome feedback, comments, criticisms, and objections on any point.
> However, since we've already had a long discussion about this, I
> respectfully request that we try to keep this thread focused.  To that
> end, I suggest:
>
>   1) If you have criticisms or objections, please try to indicate
>      whether you think they are `substantive' (e.g., you see a problem
>      that would prevent you from using this syntax, or prevent Org from
>      implementing it) or not (e.g., you would prefer a slightly
>      different but equivalent way of expressing something).
>
>   2) If you wish to express an opinion about the proposal without
>      offering further comments, let us know by just replying with +1
>      (meaning you'd like to see this syntax, or something reasonably
>      similar to it, be adopted), 0, or -1 (meaning you'd prefer not to
>      see this syntax or anything similar to it adopted).
>
> I guess this is my Valentine to the Org community. :) Thanks for reading!
>
> Best,
> Richard
>
> #+TITLE: Citation syntax, a revised proposal
> #+DATE: <2015-02-14 Sat>
>
> #+AUTHOR: Richard Lawrence
> #+EMAIL: richard.lawrence@berkeley.edu
>
> #+LANGUAGE: en
> #+SELECT_TAGS: export
>
> #+EXCLUDE_TAGS: noexport
>
> * Citation syntax
> ** Requirements
> A citation is a textual reference to one or more individual works,
> together with other information about those works, grouped together in
> a single place.  
>
> Within a citation, each reference to an individual work needs to be
> capable of containing:
>   1) a database key that references the cited work
>   2) prefix / pre-note
>   3) suffix / post-note
>      
> Whole citations also need:
>   4) [@4] a way of specifying whether the citation is in-text or
>      parenthetical
>   5) a way of representing a common prefix and suffix, if the citation
>      is a multi-cite
>   6) a way of specifying whether the citation should produce a
>      complete bibliography entry in-place
>   7) an extensible way of specifying formatting properties to export
>      filters and/or specific export backends 
>      
> ** Citation definitions
> *** Citation keys; bibliography references vs. complete entries 
> A citation key consists of a unique label preceded by a flag, which is
> optionally preceded by a hyphen.
>
> The flag is either `@' or `&'.  `@' indicates that the citation should
> produce a normal reference to the bibliography entry for the cited
> work (in whatever style the document uses), located elsewhere.
>
> The `&' flag indicates that the citation should produce a complete
> bibliography entry for the cited work in the place where the citation
> appears.
>
> The optional hyphen (`-') indicates that the author's name should be
> suppressed from the rendered citation.  (Note that this is only useful
> in author-X citation styles; it should have no effect in numeric
> styles.)
>
> *** Basic citations: Parenthetical vs. in-text 
> There are two basic types of citation: /parenthetical/ and /in-text/.
> Each of these may contain references to one or more individual works.
>
> The difference between parenthetical and in-text citations is
> expressed using parentheses around the /first/ citation key.  A
> parenthetical citation has such parentheses around the first citation
> key; an in-text citation lacks them.  (Parentheses around non-initial
> keys are permitted for visual consistency and to keep the grammar
> simple, but have no meaning.)
>
> A citation thus consists in general of a bracketed list, beginning
> with `cite:', of one or more individual references, each of which:
>   - may contain a prefix,
>   - must contain a citation key, which may or may not be surrounded by `(...)'
>   - and may contain a suffix
> Individual references are separated by semi-colons.
>
> There are also two special cases to make simple-but-common uses very
> easy to type and read:
>   1) a parenthetical citation for a single work with no prefix and
>      suffix may be written by just surrounding the key with brackets,
>      like: [@Doe99].
>   2) an in-text citation for a single work with no prefix and suffix
>      may be written as a /bare/ key, without brackets, like: @Doe99.
> (Thus, in both of the `simple' cases, one less level of bracketing is
> required.)
>
> Prefix and suffix text are regular Org text, which are allowed to
> contain various kinds of Org markup (see the grammar below for a
> complete list).
>
> *** Multi-cite citations 
> Multi-cite citations are distinguished from basic parenthetical and
> in-text citations by the presence of an optional common prefix or
> common suffix (which may not contain keys).  If present, the common
> prefix must occur before the first individual reference, and the
> common suffix must occur after the last individual reference.  The
> common prefix and suffix are separated from the individual references
> by semi-colons.
>
> *** Examples of main citation syntax
> Basic parenthetical citation:
>
> #+BEGIN_QUOTE
> The nineteenth century was very interesting. [cite: (@Doe99)]
> #+END_QUOTE
>
>
> Basic parenthetical citation using special-case syntax:
>
> #+BEGIN_QUOTE
> The nineteenth century was very interesting. [@Doe99]
> #+END_QUOTE
>
>
> Parenthetical citation with multiple works and prefix and suffix:
>
> #+BEGIN_QUOTE
> The nineteenth century was in fact lovely [cite: see (@Doe99) p. 44;
> @Smith2000 has a review].
> #+END_QUOTE
>
>
> Basic in-text citation with a suffix:
>
> #+BEGIN_QUOTE
> As [cite: @Doe99 p. 44] says, the nineteenth century was very interesting.
> #+END_QUOTE
>
>
> In-text citation using special-case syntax:
>
> #+BEGIN_QUOTE
> @Doe2000 explains that the twentieth century was even more interesting. 
> #+END_QUOTE
>
>
> In-text citation with author suppressed:
>
> #+BEGIN_QUOTE
> As Doe explained in his -@Doe2003, the twentieth century was somewhat
> less interesting than previously thought.
> #+END_QUOTE
>
>
> Parenthetical citation with full-entry key:
>
> #+BEGIN_QUOTE
> A complete bibliography entry follows in parentheses. [cite: (&Doe99)]
> A complete bibliography entry follows in parentheses. [&Doe99]
> #+END_QUOTE
>
>
> In-text citation with full-entry key:
>
> #+BEGIN_QUOTE
> A complete bibliography entry follows: [cite: &Doe99].
> A complete bibliography entry follows: &Doe99.
> #+END_QUOTE
>
>
> Full-entry in-text citation, in a footnote:
>
> #+BEGIN_QUOTE
> Doe exhibits unusual scholarship.[fn:: &Doe99.]
> #+END_QUOTE
>
>
> In-text citation, with a complete bibliography entry minus the author
> in a footnote, plus a suffix:
>
> #+BEGIN_QUOTE
> @Doe99 exhibits unusual scholarship.[fn:1]
>
> [fn:1] [cite: -&Doe99 Cf. especially section 4.]
> #+END_QUOTE
>
>
> In-text multi-cite:
>
> #+BEGIN_QUOTE
> Speculation abounds about what the twenty-first century will
> bring. [cite: For an overview of this topic, see; @Smith1998;
> @Jones1999; @Miller2001; and references therein.]
> #+END_QUOTE
>
>
> Parenthetical multi-cite:
>
> #+BEGIN_QUOTE
> Speculation abounds about what the twenty-first century will
> bring. [cite: For an overview of this topic, see; (@Smith1998);
> @Jones1999; @Miller2001; and references therein.]
> #+END_QUOTE
>
>
> *** Syntax for extensions 
> Additional information can be supplied in a citation that may affect
> how export filters or particular backends format it.
>
> This additional information may be supplied following the brackets of
> a citation between the following delimiters: `%%( ... )'.
>
> (Note: I am proposing that this expression go /after/ the main
> citation brackets both because it visually separates this extra
> information from the main citation, and in order to avoid imposing any
> further syntactic restriction on suffixes.)
>
> At least for now, any information supplied this way is /strictly the
> user's responsibility/ to interpret (e.g., using an export filter).
> This means that citations that have information like this are not
> portable and might not be exported correctly:
>   - in other users' setups
>   - by particular backends
>   - by future versions of Org
>
> I will not deal with the details of how this additional information
> should be syntactically represented, since this has not really been
> discussed.  But I suggest that, to deal with the complexities of
> additional information in full generality, something like a complete
> Lisp list is required.  Thus, I suggest that this additional
> information simply be represented as a Lisp list.  (Besides
> generality, this has the benefit of making the syntax easy to parse:
> the parser can just call Elisp's read function with a marker after the
> `%%'.)
>
> I provide these examples merely to illustrate the possibilities here:
>
> #+BEGIN_QUOTE
> @vonNeumann1930 %%(:type genitive :capitalize t) model can only handle
> a limited range of observed cases.
>
> @McCarthy1950 %%('s) clever use of Lisp syntax was also used to
> express the Saxon genitive.
>
> For more, see Ref. @Doe99 %%(:type refnum :follow-to "some.pdf").
>
> Even more complicated examples occur after Doe's famous article from
> [cite: @Doe99] %%(:type date-only).
>
> And in [cite: @Doe2000] %%(:attr_latex (:format-string
> "\citeyear{%KEY}") :attr_html (:only-fields (month year))), Doe
> finally realized that arbitrary complexity was a powerful but
> double-edged sword.
>
> @_aParticularlyUGLYkey:is-this-one %%(:overlay "Nice Display")
> #+END_QUOTE 
>
> ** Grammar
> This section formally documents the syntax of citations discussed
> above.  
>
> To represent the syntax of citations, we need a category of /citation/
> objects, which require the following properties (the names here are not
> important and could be changed):
>   - is-parenthetical (boolean; nil means is in-text)
>   - common-prefix (text)
>   - common-suffix (text)
>   - references (list)
>   - extra-info (list)
>
> Each reference in the list of references should be a plist with the
> following properties:
>   - prefix (text)
>   - suffix (text)
>   - key (string)
>   - is-parenthesized (boolean; t means key was parenthesized; only
>     significant for the first reference in a citation)
>   - suppress-author (boolean; t means author name should not be output)
>   - is-full (boolean; t means a full bibliography entry should be
>     output in-place) 
>
> The category of citations has the following grammar:
>   - A CITATION is a PARENTHETICAL-CITATION or an IN-TEXT citation.
>   - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
>     CITATION-LIST whose first individual INDIVIDUAL-REFERENCE is a
>     PARENTHESIZED-KEY
>   - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
>     CITATION-LIST whose first INDIVIDUAL-REFERENCE is a BARE-KEY.
>   - A SIMPLE-PARENTHETICAL is a KEY immediately surrounded by square
>     brackets, optionally followed by an EXTRA-INFO clause.
>   - A SIMPLE-IN-TEXT is a BARE-KEY, optionally followed by an
>     EXTRA-INFO clause
>   - A CITATION-LIST has the format
>        [cite: PREFIX; INDIVIDUAL-REFERENCE; ... INDIVIDUAL-REFERENCE; SUFFIX] EXTRA-INFO
>     where the initial PREFIX, final SUFFIX, and EXTRA-INFO clause are
>     optional.  At least one INDIVIDUAL-REFERENCE must be present. 
>   - An INDIVIDUAL-REFERENCE has the format: 
>        PREFIX KEY-MAYBE-PARENS SUFFIX
>     The KEY-MAYBE-PARENS is obligatory, and the prefix and suffix
>     are optional.
>   - A KEY-MAYBE-PARENS is either a BARE-KEY or PARENTHESIZED-KEY
>   - A BARE-KEY is a KEY with immediately-preceding whitespace
>   - A PARENTHESIZED-KEY is a KEY immediately surrounded by `(' and `)'.
>   - A KEY optionally begins with `-', and obligatorily contains `@' or
>     `&' followed by a string of characters which begins with a letter
>     or `_', and may contain alphanumeric characters and the following
>     internal punctuation characters:
>        :.#$%&-+?<>~/
>   - A PREFIX or SUFFIX is arbitrary text (except `;', `]', and
>     KEY-MAYBE-PARENs) which may contain only the following Org
>     objects:
>     - bold
>     - code
>     - entity
>     - italic
>     - latex-fragment
>     - line-break
>     - strike-through
>     - subscript
>     - superscript
>     - underline
>     - superscript
>     (Note that this list could be extended somewhat if necessary.)
>   - An EXTRA-INFO clause consists of data not specified by this
>     grammar, in between `%%(' and `)'
>
> ** Outstanding issues
> It seems to me that there are potential problems with the above
> proposal in a number of areas, but I cannot tell how serious they are,
> or what changes (if any) should be made to solve them.  I don't
> pretend that this is an exhaustive list:
>   1) *Nesting.*  I have favored LaTeX compatibility for in-text
>      citations with multiple references; but this means there is no
>      way to `nest' citations.  Thus, there is no way to express (in
>      the main syntax) what Pandoc expresses as:
>         @Doe99 [p. 34; see also @DoeRoe2000]
>      which renders like:
>         Doe (1999, p. 34; see also Doe and Roe 2000)
>      Instead, since a citation is in-text or parenthetical as a whole,
>      the equivalent in the above syntax
>         [cite: @Doe99 p. 34; see also @DoeRoe2000]
>      should render like:
> 	Doe (1999, p. 34), see also Doe and Roe (2000).
>      I am not certain if Pandoc-like output is important in this case.
>      The few people who commented on this said that it was not. 
>   2) *Limitations on prefixes and suffixes.*  There may be legitimate
>      uses of `@', `;', `]', etc. inside prefix or suffix text that the
>      above syntax does not allow.  Examples might include:
>      - use of semi-colons as part of the prefix/suffix text
>      - footnotes, links, or timestamps inside a prefix/suffix
>      I am not certain how important these cases are.  If they are
>      important, some of them might be able to be worked around with
>      entities.
>   3) *Edge cases.* The above syntax may make it possible to express
>      things that don't make sense, or would be too difficult to
>      export.  The only one I can think of is that it is possible to
>      mix `@'-style and `&'-style keys in the same citation.  I am not
>      sure if this should be forbidden; it may sometimes make sense.
>      It may also be possible to express things that external tools,
>      such as citeproc-js, don't know how to process.  I do not have a
>      good sense of what, if anything, falls into that category, and
>      what should be done about it.
>   4) *Citation commands.*  Rather than introduce an explicit
>      representation for different citation commands/types, I have used
>      different parts of the syntax to express the common distinctions
>      that people mentioned.  I suggest that, for now, anything beyond
>      these basic distinctions be left to the user-extension syntax.
>      However, if it becomes clear in the future that there is a need
>      to add a representation for a command to the main syntax, there
>      is a natural place to do so: immediately after the `cite:' tag
>      (as Nicolas suggested).
>
> Also, I have not said anything in this proposal to address how other
> document metadata should be represented, which has not been discussed
> much on the list.  I think this should be discussed separately.
>       

Re: Citation syntax: a revised proposal

[Rasmus]

0

Parts I like:  

   1) a parenthetical citation for a single work with no prefix and
      suffix may be written by just surrounding the key with brackets,
      like: [@Doe99].
   2) an in-text citation for a single work with no prefix and suffix
      may be written as a /bare/ key, without brackets, like: @Doe99.

I recently cracked up something similar for a paper we are working on, and
I think it's nice.  I have yet to get the verdict from my coauthor,
though.

Parts that I don't care for:

     [cite: whatever (@Doe99) whatever]

Not intuitive to me, but I could get used to it.

Parts I hate:

    The flag is either `@' or `&'.  `@' [...] The optional hyphen (`-') 

Too many weird symbols that I won't be able to remember, much less explain
to somebody else.

    `%%( ... )'.

Just too odd.  Extensibilty should not be delegated to some weird
construct outside of the element in question.

—Rasmus

--       

A page of history is worth a volume of logic

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Rasmus,

Rasmus <rasmus@gmx.us> writes:

> 0
>
> Parts I like:  
>
>    1) a parenthetical citation for a single work with no prefix and
>       suffix may be written by just surrounding the key with brackets,
>       like: [@Doe99].
>    2) an in-text citation for a single work with no prefix and suffix
>       may be written as a /bare/ key, without brackets, like: @Doe99.
>
> I recently cracked up something similar for a paper we are working on, and
> I think it's nice.  I have yet to get the verdict from my coauthor,
> though.

Cool. :)

> Parts that I don't care for:
>
>      [cite: whatever (@Doe99) whatever]
>
> Not intuitive to me, but I could get used to it.

It's not super intuitive to me either, and it just occurred to me
yesterday, so maybe there's a better way.  The reason I went this way
was so that we could represent the difference between parenthetical and
in-text citations without moving the cite key and without using
different tags (citet: vs. citep:).  That makes it easy to write a
function that will quickly switch a citation between the two styles,
without using the tag to express the difference, which Nicolas was
worried would slow down the parser.

> Parts I hate:
>
>     The flag is either `@' or `&'.  `@' [...] The optional hyphen (`-') 
>
> Too many weird symbols that I won't be able to remember, much less explain
> to somebody else.

I don't love these either, but I am not sure what a better alternative
would be.  The `@' is vestigial inspiration from Pandoc, and is used
infrequently enough elsewhere in Org syntax that Nicolas at one point
said it would be OK performance-wise to have `@key' appear alone in the
text.  `&' seemed like a natural counterpart for the same reasons, but I
agree it isn't terribly intuitive.  (Though maybe there's one supporting
intuition: `&' is used to introduce keys in URL parameters...)

I disagree with you about the hyphen, but I wouldn't use it enough to
lobby for it (it is just vestigial Pandoc).  If others think we should
take it out, that's fine with me.

>     `%%( ... )'.
>
> Just too odd.  Extensibilty should not be delegated to some weird
> construct outside of the element in question.

Again, I don't exactly love this either, but I chose this syntax because
%%(...) is already used elsewhere in Org to represent embedded
s-expressions (notably in timestamps -- see section 8.1 of the manual).
%(...) is also used for s-expressions in capture templates, though I'm
not sure why the first case uses two `%'s and the second only one.

The only reason to use these delimiters is consistency; I'm not opposed
to something prettier if there's a better alternative.

Keeping this part of the syntax outside the [cite: ...] brackets allows
it to be used with bare keys for simple in-text citations, and prevents
a further syntactic restriction on prefix/suffix text inside the
brackets.  I'm not opposed to moving it inside if that seems really
important, but these two considerations weigh against it in my opinion.

Best,
Richard

Re: Citation syntax: a revised proposal

[Stefan Nobis]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Rasmus <rasmus@gmx.us> writes:

>> Parts I hate:

>>     The flag is either `@' or `&'.  `@' [...] The optional hyphen (`-') 

>> Too many weird symbols that I won't be able to remember, much less explain
>> to somebody else.

> I don't love these either, but I am not sure what a better
> alternative would be.

I would say, just keep "@" to mark the key. The others are not really
needed. Both, "&" and "-" are better handled by a nice internal
syntax, something like

  [cite:command ...]

or

  [cite: ... @key :part year ...]

These internal extensions via keywords are IMHO much nicer that the
"%%(...)" variant (as a programmer I also like "%%(...)", but not as
an author).

I think this kind of syntax (only plain "@key" or maybe "[@key]" as
shortcut and everything else within "[cite:...]") is also easier to
handle with overlays, user input helpers etc.

Some input helper can make remembering all the options and keywords
inside [cite:...] a non-issue and overlays will render it nice in the
text. Therefore the syntax should be rather simple and regular with as
few exceptions and shorthands as sensible.

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Aloha Stefan,

Stefan Nobis <stefan-ml@snobis.de> writes:

> Richard Lawrence <richard.lawrence@berkeley.edu> writes:
>
>> Rasmus <rasmus@gmx.us> writes:
>
>>> Parts I hate:
>
>>>     The flag is either `@' or `&'.  `@' [...] The optional hyphen (`-') 
>
>>> Too many weird symbols that I won't be able to remember, much less explain
>>> to somebody else.
>
>> I don't love these either, but I am not sure what a better
>> alternative would be.
>
> I would say, just keep "@" to mark the key. The others are not really
> needed. Both, "&" and "-" are better handled by a nice internal
> syntax, something like
>
>   [cite:command ...]
>
> or
>
>   [cite: ... @key :part year ...]
>
> These internal extensions via keywords are IMHO much nicer that the
> "%%(...)" variant (as a programmer I also like "%%(...)", but not as
> an author).
>
> I think this kind of syntax (only plain "@key" or maybe "[@key]" as
> shortcut and everything else within "[cite:...]") is also easier to
> handle with overlays, user input helpers etc.
>
> Some input helper can make remembering all the options and keywords
> inside [cite:...] a non-issue and overlays will render it nice in the
> text. Therefore the syntax should be rather simple and regular with as
> few exceptions and shorthands as sensible.

This makes good sense to me.  Something like [cite:command pre-note key
post-note] for an individual citation is capable of carrying all the
necessary information and is easy for an author to read in case overlays
aren't active.

IIUC, one motivation for privileging "basic" citation commands in the
syntax was brevity during input.  An input helper would make this
motivation mostly moot and would result in a simpler syntax.

Another motivation for privileging "basic" citation commands is that the
syntax would then advertise the commands officially supported by Org
mode.  Using a simpler syntax, as Stefan suggests, would it be possible
for the various ox-* files to indicate for the input helper which
commands are supported?  Then, the author could specify the output
targets somewhere in the header and the input helper could serve up the
intersection of supported commands, yielding a document portable across
the indicated backends and with the richest possible set of citation
commands.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> Parts I hate:
>
>     The flag is either `@' or `&'.  `@' [...] The optional hyphen (`-') 
>
> Too many weird symbols that I won't be able to remember, much less explain
> to somebody else.

Honestly, Org is already full of cryptic symbols, e.g., {{{...}}}
@@...@@, <<<....>>>, <<...>>, and so on. This is not worse than the rest
of Org.


Regards,

Re: Citation syntax: a revised proposal

[Sebastien Vauban]

Hello Nicolas,

Nicolas Goaziou wrote:
> Honestly, Org is already full of cryptic symbols, e.g., {{{...}}}
> @@...@@, <<<....>>>, <<...>>, and so on. This is not worse than the
> rest of Org.

What is <<<...>>>?

PS- Not easy to search for that in the Org manual in Emacs: neither <<
    nor <<< are in the index, though I know about the first one (Noweb).

PPS- Before, I could use C-s to go through pages which would contain
     <<. Now, I don't know if that's because of a change on my side,
     that does not work anymore...

Best regards,
  Seb

-- 
Sebastien Vauban

Re: Citation syntax: a revised proposal

[Vaidheeswaran C]

On Monday 09 March 2015 04:10 PM, Sebastien Vauban wrote:
> What is <<<...>>>?

(info "(org) Radio targets")

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Hello,

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> Since discussion seems to have petered out on the previous thread (see:
> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
> go back over the discussion and write up a concrete proposal for
> citation syntax.
>
> This proposal represents my attempt to formulate a syntax that is easy
> to read, easy to parse, and covers all the use-cases that people
> mentioned as being important.  It is surely not perfect, but I learned a
> lot from the previous thread, and I hope something like this will serve
> the community's needs.

Thanks for your proposal. Some comments follow.

> The difference between parenthetical and in-text citations is
> expressed using parentheses around the /first/ citation key.  A
> parenthetical citation has such parentheses around the first citation
> key; an in-text citation lacks them.  (Parentheses around non-initial
> keys are permitted for visual consistency and to keep the grammar
> simple, but have no meaning.)

I think it would be nicer to differentiate between in-text and
parenthetical citations at the type level, e.g.:


  [cite: this @key citation is in-text]
  [(cite): this @key citation is parenthetical]

or, as already suggested

  [citet: ...]
  [citep: ...]

I prefer the former.

>   1) a parenthetical citation for a single work with no prefix and
>      suffix may be written by just surrounding the key with brackets,
>      like: [@Doe99].
>   2) an in-text citation for a single work with no prefix and suffix
>      may be written as a /bare/ key, without brackets, like: @Doe99.
> (Thus, in both of the `simple' cases, one less level of bracketing is
> required.)

Sounds good.

> *** Syntax for extensions 
> Additional information can be supplied in a citation that may affect
> how export filters or particular backends format it.
>
> This additional information may be supplied following the brackets of
> a citation between the following delimiters: `%%( ... )'.

As pointed out, this is very odd. But I cannot see any clean solution.
However, it would be nice to integrate it somehow with the syntax. Maybe
something like

  [cite: ... @key ...; ... @key2 ... |latex: :prop val |html: :prop val]

> ** Outstanding issues
> It seems to me that there are potential problems with the above
> proposal in a number of areas, but I cannot tell how serious they are,
> or what changes (if any) should be made to solve them.  I don't
> pretend that this is an exhaustive list:
>   1) *Nesting.*  I have favored LaTeX compatibility for in-text
>      citations with multiple references; but this means there is no
>      way to `nest' citations.  Thus, there is no way to express (in
>      the main syntax) what Pandoc expresses as:
>         @Doe99 [p. 34; see also @DoeRoe2000]
>      which renders like:
>         Doe (1999, p. 34; see also Doe and Roe 2000)
>      Instead, since a citation is in-text or parenthetical as a whole,
>      the equivalent in the above syntax
>         [cite: @Doe99 p. 34; see also @DoeRoe2000]
>      should render like:
> 	Doe (1999, p. 34), see also Doe and Roe (2000).
>      I am not certain if Pandoc-like output is important in this case.
>      The few people who commented on this said that it was not.

AFAIU, when using in-text citation, only the first key is extracted out
of the parenthesis, so

  [cite: @Doe99 p. 34; see also @DoeRoe2000]

should really render like

  Doe (1999, p. 34; see also Doe and Roe 2000).

IOW, why do you think that "a citation is in-text or parenthetical as
a whole"?

>   2) *Limitations on prefixes and suffixes.*  There may be legitimate
>      uses of `@', `;', `]', etc. inside prefix or suffix text that the
>      above syntax does not allow.  Examples might include:
>      - use of semi-colons as part of the prefix/suffix text
>      - footnotes, links, or timestamps inside a prefix/suffix
>      I am not certain how important these cases are.  If they are
>      important, some of them might be able to be worked around with
>      entities.

Indeed. Entities are the way to go. If it isn't sufficient we'll need to
provide an escaping mechanism. This may be used elsewhere in Org.

>   4) *Citation commands.*  Rather than introduce an explicit
>      representation for different citation commands/types, I have used
>      different parts of the syntax to express the common distinctions
>      that people mentioned.  I suggest that, for now, anything beyond
>      these basic distinctions be left to the user-extension syntax.
>      However, if it becomes clear in the future that there is a need
>      to add a representation for a command to the main syntax, there
>      is a natural place to do so: immediately after the `cite:' tag
>      (as Nicolas suggested).

With extensions, it is not necessary to support another location for
commands. E.g.,

  [cite: @Doe |latex: :command citedwim]

or whatever extension syntax is chosen.

> Also, I have not said anything in this proposal to address how other
> document metadata should be represented, which has not been discussed
> much on the list.  I think this should be discussed separately.

Indeed. One step at a time. And this one is rather big.


Regards,

-- 
Nicolas Goaziou

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> The difference between parenthetical and in-text citations is
>> expressed using parentheses around the /first/ citation key.  A
>> parenthetical citation has such parentheses around the first citation
>> key; an in-text citation lacks them.  (Parentheses around non-initial
>> keys are permitted for visual consistency and to keep the grammar
>> simple, but have no meaning.)
>
> I think it would be nicer to differentiate between in-text and
> parenthetical citations at the type level, e.g.:
>
>
>   [cite: this @key citation is in-text]
>   [(cite): this @key citation is parenthetical]
>
> or, as already suggested
>
>   [citet: ...]
>   [citep: ...]
>
> I prefer the former.

I prefer the latter.  It's explicit, shorter and doesn't hitting shift for
'()' (on my kb).  No voodoo.  I don't mind either, though.

>
>> *** Syntax for extensions 
>> Additional information can be supplied in a citation that may affect
>> how export filters or particular backends format it.
>>
>> This additional information may be supplied following the brackets of
>> a citation between the following delimiters: `%%( ... )'.
>
> As pointed out, this is very odd. But I cannot see any clean solution.
> However, it would be nice to integrate it somehow with the syntax. Maybe
> something like
>
>   [cite: ... @key ...; ... @key2 ... |latex: :prop val |html: :prop val]

I prefer to have more expressive keys, e.g. the 'cite' part.  But perhaps
it's a good way express extra properties.  The thing is, for latex the
extra property is a citation type.


> AFAIU, when using in-text citation, only the first key is extracted out
> of the parenthesis, so
>
>   [cite: @Doe99 p. 34; see also @DoeRoe2000]
>
> should really render like
>
>   Doe (1999, p. 34; see also Doe and Roe 2000).
>
> IOW, why do you think that "a citation is in-text or parenthetical as
> a whole"?

No!  I believe (but correct me if I'm wrong) that neither John, Eric, Tom
nor myself have seen a citation like this in the wild.  If you have I
might be wrong.  It's no easily supported in latex.  The latex equivalent
of the above is:

\citeauthor{doe} (\citeyear[p.\ 34]{doe}; see also \textcite*{roe})

Or something like that.

AFAIK, 
  [cite: @Doe99 p. 34; see also @DoeRoe2000]
→  Doe (1999, p. 34) and see also Doe et al (2000)
or maybe
   Doe (1999, p. 34) and  Doe et al (see also 2000).

I don't remember.

—Rasmus

-- 
Together we'll stand, divided we'll fall

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Rasmus <rasmus@gmx.us> writes:

> Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> I think it would be nicer to differentiate between in-text and
>> parenthetical citations at the type level, e.g.:
>>
>>
>>   [cite: this @key citation is in-text]
>>   [(cite): this @key citation is parenthetical]
>>
>> or, as already suggested
>>
>>   [citet: ...]
>>   [citep: ...]
>>
>> I prefer the former.
>
> I prefer the latter.

OK.

> It's explicit,

No, sir. (cite) is explicit. It means "a citation enclosed within
parenthesis". citep is only explicit if you come from LaTeX world.

> shorter

cite and (cite) have the same mean length!

> and doesn't hitting shift for '()' (on my kb).

OK. "Rasmus' keyboard" (aka. a Rk) is a decent unit of measurement for
syntax quality, I guess. ;)

> No voodoo. I don't mind either, though.

What colour are voodoo sheds these days?

>> As pointed out, this is very odd. But I cannot see any clean solution.
>> However, it would be nice to integrate it somehow with the syntax. Maybe
>> something like
>>
>>   [cite: ... @key ...; ... @key2 ... |latex: :prop val |html: :prop val]
>
> I prefer to have more expressive keys, e.g. the 'cite' part.

Please don't touch (too much) the "cite" part. This is for Org, not for
export back-ends.

> But perhaps it's a good way express extra properties. The thing is,
> for latex the extra property is a citation type.

Then

  [cite: ... |latex: :type citedwim]

How many "Rk" does this score?

>> AFAIU, when using in-text citation, only the first key is extracted
>> out of the parenthesis, so
>>
>>   [cite: @Doe99 p. 34; see also @DoeRoe2000]
>>
>> should really render like
>>
>>   Doe (1999, p. 34; see also Doe and Roe 2000).
>>
>> IOW, why do you think that "a citation is in-text or parenthetical as
>> a whole"?
>
> No!  I believe (but correct me if I'm wrong) that neither John, Eric, Tom
> nor myself have seen a citation like this in the wild.  If you have I
> might be wrong.  It's no easily supported in latex.  The latex equivalent
> of the above is:
>
> \citeauthor{doe} (\citeyear[p.\ 34]{doe}; see also \textcite*{roe})
>
> Or something like that.
>
> AFAIK, 
>   [cite: @Doe99 p. 34; see also @DoeRoe2000]
> →  Doe (1999, p. 34) and see also Doe et al (2000)
> or maybe
>    Doe (1999, p. 34) and  Doe et al (see also 2000).
>
> I don't remember.

OK. Then let's forget about my remark.


Regards,

Re: Citation syntax: a revised proposal

[John Kitchin]

Rasmus writes:

>> or, as already suggested
>>
>>   [citet: ...]
>>   [citep: ...]
>>
>> I prefer the former.
>
> I prefer the latter.  It's explicit, shorter and doesn't hitting shift for
> '()' (on my kb).  No voodoo.  I don't mind either, though.

Nobody should be typing these by hand anyway ;) you should have a nice
key binding (preferrably with a prefix arg) that helps you select from a
database, and inserts the formatted key in the right syntax for the vast
majority of the time.



--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas,

Thanks for your comments.  A couple of replies follow.

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>> The difference between parenthetical and in-text citations is
>> expressed using parentheses around the /first/ citation key.  A
>> parenthetical citation has such parentheses around the first citation
>> key; an in-text citation lacks them.  (Parentheses around non-initial
>> keys are permitted for visual consistency and to keep the grammar
>> simple, but have no meaning.)
>
> I think it would be nicer to differentiate between in-text and
> parenthetical citations at the type level, e.g.:
>
>
>   [cite: this @key citation is in-text]
>   [(cite): this @key citation is parenthetical]
>
> or, as already suggested
>
>   [citet: ...]
>   [citep: ...]
>
> I prefer the former.

Either of these is fine with me if it is OK with you.  I was trying to
avoid any variation in the position after `[' to reduce strain on the
parser, but if these are simple enough to support without too many false
positives, even better.

>> This additional information may be supplied following the brackets of
>> a citation between the following delimiters: `%%( ... )'.
>
> As pointed out, this is very odd. But I cannot see any clean solution.

As I said in reply to Rasmus, my only reason for this choice was to be
consistent with the s-expression syntax in timestamps and capture
templates.  I am fine with (and even would prefer) something that looks
prettier.

> However, it would be nice to integrate it somehow with the syntax. Maybe
> something like
>
>   [cite: ... @key ...; ... @key2 ... |latex: :prop val |html: :prop val]
>

But I think there are three reasons this does not quite improve on what
I proposed:

1) It looks like it only supports properties directed at specific
backends.  How should users specify custom properties that they want to
be handled in multiple backends (by their own filter)?  

2) It requires us to decide *now* on conventions for specifying
properties to specific backends (and also to build a parser for them,
instead of just calling `read'), instead of just using arbitrary
s-expressions and seeing what people come up with in the future.  (See
my reply to Tom for more about how I was thinking this part of the
syntax would evolve.)

3) Putting the properties inside the brackets introduces an (admittedly
very minor) additional restriction on suffix text, and can't be used
with the simple syntax for in-text citations.  (See my reply to Rasmus
on this point.)

That said, I have no objections to doing something like this if we can
come to an agreement now about what it should look like.

>> ** Outstanding issues
>> It seems to me that there are potential problems with the above
>> proposal in a number of areas, but I cannot tell how serious they are,
>> or what changes (if any) should be made to solve them.  I don't
>> pretend that this is an exhaustive list:
>>   1) *Nesting.*  I have favored LaTeX compatibility for in-text
>>      citations with multiple references; but this means there is no
>>      way to `nest' citations.  Thus, there is no way to express (in
>>      the main syntax) what Pandoc expresses as:
>>         @Doe99 [p. 34; see also @DoeRoe2000]
>>      which renders like:
>>         Doe (1999, p. 34; see also Doe and Roe 2000)
>>      Instead, since a citation is in-text or parenthetical as a whole,
>>      the equivalent in the above syntax
>>         [cite: @Doe99 p. 34; see also @DoeRoe2000]
>>      should render like:
>> 	Doe (1999, p. 34), see also Doe and Roe (2000).
>>      I am not certain if Pandoc-like output is important in this case.
>>      The few people who commented on this said that it was not.
>
> AFAIU, when using in-text citation, only the first key is extracted out
> of the parenthesis, so
>
>   [cite: @Doe99 p. 34; see also @DoeRoe2000]
>
> should really render like
>
>   Doe (1999, p. 34; see also Doe and Roe 2000).
>
> IOW, why do you think that "a citation is in-text or parenthetical as
> a whole"?

This is a LaTeX compatibility issue, and it is an area where Pandoc
departs from LaTeX behavior.  If you're a LaTeX user, the natural way to
think of the Org syntax you quoted is (I think) as equivalent to one of:

  \textcites[p. 34]{Doe99}[see also][]{DoeRoe2000}
  \textcite[p. 34]{Doe99}, \textcite[see also][]{DoeRoe2000} 

which IIUC both render like

  Doe (1999, p. 34), see also Doe and Roe (2000).

modulo some subtle details about the separator.  No one so far has said
that they need or want the Pandoc behavior.  I personally am fine
without it.

Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

>> However, it would be nice to integrate it somehow with the syntax. Maybe
>> something like
>>
>>   [cite: ... @key ...; ... @key2 ... |latex: :prop val |html: :prop val]
>>
>
> But I think there are three reasons this does not quite improve on what
> I proposed:
>
> 1) It looks like it only supports properties directed at specific
> backends.  How should users specify custom properties that they want to
> be handled in multiple backends (by their own filter)?

They cannot (unless they want to use something like "|custom: ...").

Note they cannot either for regular elements using attributes. The
reason is that multiple back-end properties are very rare. For
example, :width hasn't the same unit in "ox-latex" and "ox-html".

> 2) It requires us to decide *now* on conventions for specifying
> properties to specific backends

Indeed. This is also part of the syntax we're trying to define, isn't
it?

> (and also to build a parser for them, instead of just calling `read'),

We won't be calling "read". OTOH, there's already
`org-export-read-attribute', which does a reasonable job.

> instead of just using arbitrary s-expressions and seeing what people
> come up with in the future. (See my reply to Tom for more about how
> I was thinking this part of the syntax would evolve.)

The point is that this syntax (which isn't new in this thread, excepted
the "|" character) is extensible at will. It can evolve.

> 3) Putting the properties inside the brackets introduces an (admittedly
> very minor) additional restriction on suffix text, and can't be used
> with the simple syntax for in-text citations.  (See my reply to Rasmus
> on this point.)

I don' think this is a real issue. Each back-end can decide what command
should be used for simple syntax. It is even possible to provide
a defcustom for it.


Regards,

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Nicolas,

2015ko otsailak 15an, Nicolas Goaziou-ek idatzi zuen:
>> 1) It looks like it only supports properties directed at specific
>> backends.  How should users specify custom properties that they want to
>> be handled in multiple backends (by their own filter)?
> 
> They cannot (unless they want to use something like "|custom: ...").
> 
> Note they cannot either for regular elements using attributes. The
> reason is that multiple back-end properties are very rare. For
> example, :width hasn't the same unit in "ox-latex" and "ox-html".

It seems like these might occur for citations.  The :capitalize property
discussed in the predecessor to this thread is one example.  So perhaps
there could be a |all: list which would be merged with the
backend-specific one(s).

Thanks,

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Hello,

Aaron Ecay <aaronecay@gmail.com> writes:

> Hi Nicolas,
>
> 2015ko otsailak 15an, Nicolas Goaziou-ek idatzi zuen:
>>> 1) It looks like it only supports properties directed at specific
>>> backends.  How should users specify custom properties that they want to
>>> be handled in multiple backends (by their own filter)?
>> 
>> They cannot (unless they want to use something like "|custom: ...").
>> 
>> Note they cannot either for regular elements using attributes. The
>> reason is that multiple back-end properties are very rare. For
>> example, :width hasn't the same unit in "ox-latex" and "ox-html".
>
> It seems like these might occur for citations.  The :capitalize property
> discussed in the predecessor to this thread is one example.  So perhaps
> there could be a |all: list which would be merged with the
> backend-specific one(s).

Perhaps. But [cite: ... |latex: :cap t |html: :cap t] isn't impossible
to write either.

Anyway, time for another proposal. In fact, it seems that it would be
better to externalize these properties, e.g.,

  [cite: ...]{latex :prop val}{html :prop val}

or

  [cite: ...]{latex :prop val | html :prop val}

No space allowed between the citation and the attributes. The big
advantage with this is that it can be extended to other objects while
still being backward-compatible.

It would solve one long standing limitation:

 Text [[file:img1.png]]{html :width 50px} and
 [[file:img2.png]]{html :width 60 px}


Regards,

Re: Citation syntax: a revised proposal

[Aaron Ecay]

Hi Nicolas,

2015ko otsailak 15an, Nicolas Goaziou-ek idatzi zuen:
> Perhaps. But [cite: ... |latex: :cap t |html: :cap t] isn't impossible
> to write either.

It violates DRY.  It (thus) makes it annoying to export a document to a
new backend: you have to search through all the citations and copy any
:cap keys to the new backend.

Crazy idea: what if there was a “root” backend from which all other
backends were derived?  This would maintain consistency with existing
attribute syntax (and possibly new, as proposed below), but also allow
certain attributes to be passed to all backends.

> 
> Anyway, time for another proposal. In fact, it seems that it would be
> better to externalize these properties, e.g.,
> 
>   [cite: ...]{latex :prop val}{html :prop val}
> 
> or
> 
>   [cite: ...]{latex :prop val | html :prop val}
> 
> No space allowed between the citation and the attributes. The big
> advantage with this is that it can be extended to other objects while
> still being backward-compatible.
> 
> It would solve one long standing limitation:
> 
>  Text [[file:img1.png]]{html :width 50px} and
>  [[file:img2.png]]{html :width 60 px}

Nice, I like it.

Thanks,

-- 
Aaron Ecay

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Aaron Ecay <aaronecay@gmail.com> writes:

> It violates DRY.  It (thus) makes it annoying to export a document to a
> new backend: you have to search through all the citations and copy any
> :cap keys to the new backend.
>
> Crazy idea: what if there was a “root” backend from which all other
> backends were derived?  This would maintain consistency with existing
> attribute syntax (and possibly new, as proposed below), but also allow
> certain attributes to be passed to all backends.

The model doesn't hold because attributes are not inherited. Anyway I'm
not against adding a special "all" attribute, but if it can be avoided,
I'm all for it.

Time for another crazy idea. Last one on my side for today

  [cite ...] [(cite) ...] [Cite ...] [(Cite) ...]

It should solve the :capitalize issue.


Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>   [cite ...] [(cite) ...] [Cite ...] [(Cite) ...]
>
> It should solve the :capitalize issue.

This is what biblatex does.

—Rasmus


-- 
. . . The proofs are technical in nature and provides no real
understanding

Re: Citation syntax: a revised proposal

[Stefan Nobis]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Time for another crazy idea. Last one on my side for today

>   [cite ...] [(cite) ...] [Cite ...] [(Cite) ...]

> It should solve the :capitalize issue.

+1

I really like it - even when looking at the org file with something
weird like vim, it's very clear and intuitive what's meant.

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Nicolas,

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Time for another crazy idea. Last one on my side for today
>
>   [cite ...] [(cite) ...] [Cite ...] [(Cite) ...]
>
> It should solve the :capitalize issue.

I am OK with this if it is important, though I am a little hesitant.

In the last thread, you expressed concern that we not have too much
variation after the opening `[' for performance reasons, which is why I
kept all the (non-simple) citations to `[cite: ...]'.

Unless you have changed your mind, I assume this means we should try not
to have very many options for this position.  Expressing capitalization
here would mean there are now four options, two of which are devoted to
expressing capitalization.  Is capitalization important enough to
introduce the complexity for it at *this* crucial syntactic position?
If we're trying to keep the number of variants after `[' low, we should
think carefully about what is important enough to go there.  (I think
parenthetical vs. in-text does meet that bar, but I am not sure
special-case capitalization does.)

Aesthetically, this feels a little *too* much like BibLaTeX to me.  I
would actually prefer [cite: @vanOrden60] %%(:capitalize t) or 
[cite: @vanOrden60]{:capitalize t}.  But like I said, I'm fine with this
if it's important.
 
Best,
Richard

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

Richard Lawrence <richard.lawrence@berkeley.edu> writes:

> I am OK with this if it is important, though I am a little hesitant.

I don't know if it is important. Just thinking out loud.

> In the last thread, you expressed concern that we not have too much
> variation after the opening `[' for performance reasons, which is why I
> kept all the (non-simple) citations to `[cite: ...]'.

Sorry if I wasn't clear. Variations after the opening "[" are OK as long
as they are contained in a _fixed_ set. Of course, the smaller the set
the better.

However, a customizable "cite" keyword (à la `org-add-link-type') is
a no-go. If this is really needed, I already suggested

  [cite:subtype: ...]

where "subtype" can be associated to any number of attributes, at user's
discretion.

> Unless you have changed your mind, I assume this means we should try not
> to have very many options for this position.  Expressing capitalization
> here would mean there are now four options, two of which are devoted to
> expressing capitalization.  Is capitalization important enough to
> introduce the complexity for it at *this* crucial syntactic position?

Again, I don't know if capitalization is important enough, but the added
complexity in this case is negligible. Anyhow, I am not wedded to the
idea.

> If we're trying to keep the number of variants after `[' low, we should
> think carefully about what is important enough to go there.  (I think
> parenthetical vs. in-text does meet that bar, but I am not sure
> special-case capitalization does.)

OK.

> Aesthetically, this feels a little *too* much like BibLaTeX to me.

I didn't know BibLaTeX used it at the time I suggested the idea.
I didn't know BibLaTeX was deemed as aesthetically wrong either (why is
it so?).


Regards,

Re: Citation syntax: a revised proposal

[Rasmus]

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

>   [cite:subtype: ...]
>
> where "subtype" can be associated to any number of attributes, at user's
> discretion.

I like CITE:subtype for customization, where CITE is a member some set,
e.g. {cite citep/(cite) fncite citeauthor} or whatever. I like this cause
it's ∞ customizable due to the subtype.

I also like [·]{:key val}, though less so for citations.  It could also be
used for "true" *inline* tasks, which would sometimes be quite nice.

> Again, I don't know if capitalization is important enough, but the added
> complexity in this case is negligible. Anyhow, I am not wedded to the
> idea.

Previously, I thought not.  But since M-c is so nice I don't see why not.

[Then again, perhaps Cite could be "captured" automatically if it's after a
 sentence-end (wait I see you use French spacing...! *sigh*).]

>> Aesthetically, this feels a little *too* much like BibLaTeX to me.
>
> I didn't know BibLaTeX used it at the time I suggested the idea.
> I didn't know BibLaTeX was deemed as aesthetically wrong either (why is
> it so?).

Biblatex is the gold standard.  Maybe not in input-aesthetics..., but in
terms of amenability, usability and output it surely is.  (No, I have
nothing to back this up).

—Rasmus

-- 
This message is brought to you by the department of redundant departments

Re: Citation syntax: a revised proposal

[Thomas S. Dye]

Rasmus <rasmus@gmx.us> writes:

> Biblatex is the gold standard.  Maybe not in input-aesthetics..., but in
> terms of amenability, usability and output it surely is.  (No, I have
> nothing to back this up).

Compare the bibtex style, chicago.bst, with biblatex-chicago and note
how much more closely the biblatex version approximates the Chicago
Manual of Style.

Also, the various multicites implemented in biblatex have made it a
viable option for the humanities.  Bibtex was created for use in the
(hard) sciences and it lacked facilities that authors and editors in the
humanities take for granted.

Also, biber is required for some biblatex features that bibtex doesn't
support.  I haven't followed this development and am not sure what they
are, though.

All the best,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Citation syntax: a revised proposal

[Rasmus]

tsd@tsdye.com (Thomas S. Dye) writes:

>> Biblatex is the gold standard.  Maybe not in input-aesthetics..., but in
>> terms of amenability, usability and output it surely is.  (No, I have
>> nothing to back this up).
>
> Compare the bibtex style, chicago.bst, with biblatex-chicago and note
> how much more closely the biblatex version approximates the Chicago
> Manual of Style.

Before biblatex-chicago, I used to generate my own bst files with
custom-bib.  It was awful 'cause I would often answer questions wrong and
would have to start over...

> Also, biber is required for some biblatex features that bibtex doesn't
> support.  I haven't followed this development and am not sure what they
> are, though.

My understanding is that a major limitation of bibtex was that it didn't
handle sorting of anything more complex than [a-zA-Z].  Bibtex8 extended
this, but you still needed special support, see e.g. dk-bib on CTAN.

Lars Madsen, of "Avoid eqnarray!"-fame (and much else in the TeX-verse),
has an excellent intro in Danish.  If my memory serves me correctly,
earlier β-versions had instructions on how to do hacks to get better
support when writing in Danish.

—Rasmus

-- 
Lasciate ogni speranza, voi che leggete questo.

Re: Citation syntax: a revised proposal

[John Kitchin]

I guess this looks workable. The syntax is generally more verbose than I
am accustomed to, and less explicit in my (latex centric) opinion. But,
the majority of our citations would be the simplest form, which is maybe
even shorter.  It looks like the citation insertion could be practically
automated, e.g. a command with a convenient key-binding looks up
references in some database, selects them, and then inserts them as
properly formatted citations at point. I think the usual suspects
reftex, helm-bibtex, and probably ebib could be taught to output most of
this syntax for whatever type, and they could give human readable hints
about the intended format, e.g. intext, parenthetical, noauthor,
etc... Or you could have dedicated commands with key completion to do
that. So many options, this should not be an issue.

Presumably each &/@key will be clickable like a link, and the function
it runs would get the key (and maybe additional info about the cite)? If
not, that would be a show-stopper to me. Not because of any syntax
reason, but a functional one. Right now every link-based citation is a
one click gateway to every scientific search engine I know, the pdf, the
bibtex entry, functions that copy citations, summarize citations, email
citations, etc... for the key that was clicked on. That is too useful to
give up for any syntax. But if each key was clickable, to a
user-definable, or default function (which might even be nothing), then
no problem. Probably the user should have to define this function, since
a key is no good without getting information about where the database is
from somewhere (e.g. a variable or in the file). I gather we are split
between bibtex, org-bibtex, and zotero as backends, and maybe there are
others (RIS, Mendeley, ...). Maybe one day there is good support for all
of them. I have tried them all, and for 15+ years, I keep coming back to
bibtex ;)

I am a little concerned about what the latex export will eventually look
like. Out of the box, I suppose a handful of types will be pretty well
supported, something like: (cite, citet, citep, citeauthor, citeyear,
citenum) which I suspect would cover many people's needs. There is no
question in my mind that some people will want to extend this, as there
are just too few of the latex citation commands supported out of the
box, especially for biblatex users (who used that because of limitations
in bibtex ;). My sense is the syntax may then be too verbose, and
difficult to write exporters for and they would go back to links. That
is probably a small number of people, and maybe I am wrong about it. I
am anyway supportive enough to see it tried out.

My final comment is that I suggest two additional things to go with this
syntax:

[bibliographystyle: some-kind-of-information-probably-unsrt/alpha/chicago]
This would tell some backend how to style the bibliography entries. This
does not need to be clickable (I don't know what a click would do
anyway, at most select the style? edit the style?).

[bibliography: @some-kind-of-source-probably-a-file; @maybe-more-than-one]
This is where the keys are stored. And, it would also indicate where the
bibliography should actually be placed. This should also be clickable,
with a default action to just open the file that was clicked on.

I prefer those to file attributes, e.g.
#+BIBLIOGRAPHY: @some-kind-of-source-probably-a-file;
@maybe-more-than-one

I don't think that can be used to specify where a bibliography should be
placed, and it doesn't make sense to me to use two things to specify the
same information.


Each of these would need customizable export.

I am pretty sure those are backend independent (even though I took the
names straight from LaTeX ;). With Endnote/Word for example you have to
choose a bibliography and style.

So, overall, I am on the positive side of zero.


Richard Lawrence writes:

> Hi everyone,
>
> Since discussion seems to have petered out on the previous thread (see:
> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
> go back over the discussion and write up a concrete proposal for
> citation syntax.
>
> This proposal represents my attempt to formulate a syntax that is easy
> to read, easy to parse, and covers all the use-cases that people
> mentioned as being important.  It is surely not perfect, but I learned a
> lot from the previous thread, and I hope something like this will serve
> the community's needs.
>
> The proposal is below, both inline (for easy quoting) and attached (for
> easy reading).  To keep it relatively short, I have mostly not explained
> my reasoning for the choices I made, but I am happy to do so here if
> anyone has questions.
>
> I welcome feedback, comments, criticisms, and objections on any point.
> However, since we've already had a long discussion about this, I
> respectfully request that we try to keep this thread focused.  To that
> end, I suggest:
>
>   1) If you have criticisms or objections, please try to indicate
>      whether you think they are `substantive' (e.g., you see a problem
>      that would prevent you from using this syntax, or prevent Org from
>      implementing it) or not (e.g., you would prefer a slightly
>      different but equivalent way of expressing something).
>
>   2) If you wish to express an opinion about the proposal without
>      offering further comments, let us know by just replying with +1
>      (meaning you'd like to see this syntax, or something reasonably
>      similar to it, be adopted), 0, or -1 (meaning you'd prefer not to
>      see this syntax or anything similar to it adopted).
>
> I guess this is my Valentine to the Org community. :) Thanks for reading!
>
> Best,
> Richard
>
> #+TITLE: Citation syntax, a revised proposal
> #+DATE: <2015-02-14 Sat>
> #+AUTHOR: Richard Lawrence
> #+EMAIL: richard.lawrence@berkeley.edu
> #+LANGUAGE: en
> #+SELECT_TAGS: export
> #+EXCLUDE_TAGS: noexport
>
> * Citation syntax
> ** Requirements
> A citation is a textual reference to one or more individual works,
> together with other information about those works, grouped together in
> a single place.
>
> Within a citation, each reference to an individual work needs to be
> capable of containing:
>   1) a database key that references the cited work
>   2) prefix / pre-note
>   3) suffix / post-note
>
> Whole citations also need:
>   4) [@4] a way of specifying whether the citation is in-text or
>      parenthetical
>   5) a way of representing a common prefix and suffix, if the citation
>      is a multi-cite
>   6) a way of specifying whether the citation should produce a
>      complete bibliography entry in-place
>   7) an extensible way of specifying formatting properties to export
>      filters and/or specific export backends
>
> ** Citation definitions
> *** Citation keys; bibliography references vs. complete entries
> A citation key consists of a unique label preceded by a flag, which is
> optionally preceded by a hyphen.
>
> The flag is either `@' or `&'.  `@' indicates that the citation should
> produce a normal reference to the bibliography entry for the cited
> work (in whatever style the document uses), located elsewhere.
>
> The `&' flag indicates that the citation should produce a complete
> bibliography entry for the cited work in the place where the citation
> appears.
>
> The optional hyphen (`-') indicates that the author's name should be
> suppressed from the rendered citation.  (Note that this is only useful
> in author-X citation styles; it should have no effect in numeric
> styles.)
>
> *** Basic citations: Parenthetical vs. in-text
> There are two basic types of citation: /parenthetical/ and /in-text/.
> Each of these may contain references to one or more individual works.
>
> The difference between parenthetical and in-text citations is
> expressed using parentheses around the /first/ citation key.  A
> parenthetical citation has such parentheses around the first citation
> key; an in-text citation lacks them.  (Parentheses around non-initial
> keys are permitted for visual consistency and to keep the grammar
> simple, but have no meaning.)
>
> A citation thus consists in general of a bracketed list, beginning
> with `cite:', of one or more individual references, each of which:
>   - may contain a prefix,
>   - must contain a citation key, which may or may not be surrounded by `(...)'
>   - and may contain a suffix
> Individual references are separated by semi-colons.
>
> There are also two special cases to make simple-but-common uses very
> easy to type and read:
>   1) a parenthetical citation for a single work with no prefix and
>      suffix may be written by just surrounding the key with brackets,
>      like: [@Doe99].
>   2) an in-text citation for a single work with no prefix and suffix
>      may be written as a /bare/ key, without brackets, like: @Doe99.
> (Thus, in both of the `simple' cases, one less level of bracketing is
> required.)
>
> Prefix and suffix text are regular Org text, which are allowed to
> contain various kinds of Org markup (see the grammar below for a
> complete list).
>
> *** Multi-cite citations
> Multi-cite citations are distinguished from basic parenthetical and
> in-text citations by the presence of an optional common prefix or
> common suffix (which may not contain keys).  If present, the common
> prefix must occur before the first individual reference, and the
> common suffix must occur after the last individual reference.  The
> common prefix and suffix are separated from the individual references
> by semi-colons.
>
> *** Examples of main citation syntax
> Basic parenthetical citation:
> #+BEGIN_QUOTE
> The nineteenth century was very interesting. [cite: (@Doe99)]
> #+END_QUOTE
>
> Basic parenthetical citation using special-case syntax:
> #+BEGIN_QUOTE
> The nineteenth century was very interesting. [@Doe99]
> #+END_QUOTE
>
> Parenthetical citation with multiple works and prefix and suffix:
> #+BEGIN_QUOTE
> The nineteenth century was in fact lovely [cite: see (@Doe99) p. 44;
> @Smith2000 has a review].
> #+END_QUOTE
>
> Basic in-text citation with a suffix:
> #+BEGIN_QUOTE
> As [cite: @Doe99 p. 44] says, the nineteenth century was very interesting.
> #+END_QUOTE
>
> In-text citation using special-case syntax:
> #+BEGIN_QUOTE
> @Doe2000 explains that the twentieth century was even more interesting.
> #+END_QUOTE
>
> In-text citation with author suppressed:
> #+BEGIN_QUOTE
> As Doe explained in his -@Doe2003, the twentieth century was somewhat
> less interesting than previously thought.
> #+END_QUOTE
>
> Parenthetical citation with full-entry key:
> #+BEGIN_QUOTE
> A complete bibliography entry follows in parentheses. [cite: (&Doe99)]
> A complete bibliography entry follows in parentheses. [&Doe99]
> #+END_QUOTE
>
> In-text citation with full-entry key:
> #+BEGIN_QUOTE
> A complete bibliography entry follows: [cite: &Doe99].
> A complete bibliography entry follows: &Doe99.
> #+END_QUOTE
>
> Full-entry in-text citation, in a footnote:
> #+BEGIN_QUOTE
> Doe exhibits unusual scholarship.[fn:: &Doe99.]
> #+END_QUOTE
>
> In-text citation, with a complete bibliography entry minus the author
> in a footnote, plus a suffix:
> #+BEGIN_QUOTE
> @Doe99 exhibits unusual scholarship.[fn:1]
>
> [fn:1] [cite: -&Doe99 Cf. especially section 4.]
> #+END_QUOTE
>
> In-text multi-cite:
> #+BEGIN_QUOTE
> Speculation abounds about what the twenty-first century will
> bring. [cite: For an overview of this topic, see; @Smith1998;
> @Jones1999; @Miller2001; and references therein.]
> #+END_QUOTE
>
> Parenthetical multi-cite:
> #+BEGIN_QUOTE
> Speculation abounds about what the twenty-first century will
> bring. [cite: For an overview of this topic, see; (@Smith1998);
> @Jones1999; @Miller2001; and references therein.]
> #+END_QUOTE
>
> *** Syntax for extensions
> Additional information can be supplied in a citation that may affect
> how export filters or particular backends format it.
>
> This additional information may be supplied following the brackets of
> a citation between the following delimiters: `%%( ... )'.
>
> (Note: I am proposing that this expression go /after/ the main
> citation brackets both because it visually separates this extra
> information from the main citation, and in order to avoid imposing any
> further syntactic restriction on suffixes.)
>
> At least for now, any information supplied this way is /strictly the
> user's responsibility/ to interpret (e.g., using an export filter).
> This means that citations that have information like this are not
> portable and might not be exported correctly:
>   - in other users' setups
>   - by particular backends
>   - by future versions of Org
>
> I will not deal with the details of how this additional information
> should be syntactically represented, since this has not really been
> discussed.  But I suggest that, to deal with the complexities of
> additional information in full generality, something like a complete
> Lisp list is required.  Thus, I suggest that this additional
> information simply be represented as a Lisp list.  (Besides
> generality, this has the benefit of making the syntax easy to parse:
> the parser can just call Elisp's read function with a marker after the
> `%%'.)
>
> I provide these examples merely to illustrate the possibilities here:
> #+BEGIN_QUOTE
> @vonNeumann1930 %%(:type genitive :capitalize t) model can only handle
> a limited range of observed cases.
>
> @McCarthy1950 %%('s) clever use of Lisp syntax was also used to
> express the Saxon genitive.
>
> For more, see Ref. @Doe99 %%(:type refnum :follow-to "some.pdf").
>
> Even more complicated examples occur after Doe's famous article from
> [cite: @Doe99] %%(:type date-only).
>
> And in [cite: @Doe2000] %%(:attr_latex (:format-string
> "\citeyear{%KEY}") :attr_html (:only-fields (month year))), Doe
> finally realized that arbitrary complexity was a powerful but
> double-edged sword.
>
> @_aParticularlyUGLYkey:is-this-one %%(:overlay "Nice Display")
> #+END_QUOTE
>
> ** Grammar
> This section formally documents the syntax of citations discussed
> above.
>
> To represent the syntax of citations, we need a category of /citation/
> objects, which require the following properties (the names here are not
> important and could be changed):
>   - is-parenthetical (boolean; nil means is in-text)
>   - common-prefix (text)
>   - common-suffix (text)
>   - references (list)
>   - extra-info (list)
>
> Each reference in the list of references should be a plist with the
> following properties:
>   - prefix (text)
>   - suffix (text)
>   - key (string)
>   - is-parenthesized (boolean; t means key was parenthesized; only
>     significant for the first reference in a citation)
>   - suppress-author (boolean; t means author name should not be output)
>   - is-full (boolean; t means a full bibliography entry should be
>     output in-place)
>
> The category of citations has the following grammar:
>   - A CITATION is a PARENTHETICAL-CITATION or an IN-TEXT citation.
>   - A PARENTHETICAL-CITATION is either a SIMPLE-PARENTHETICAL or a
>     CITATION-LIST whose first individual INDIVIDUAL-REFERENCE is a
>     PARENTHESIZED-KEY
>   - An IN-TEXT-CITATION is either a SIMPLE-IN-TEXT, or a
>     CITATION-LIST whose first INDIVIDUAL-REFERENCE is a BARE-KEY.
>   - A SIMPLE-PARENTHETICAL is a KEY immediately surrounded by square
>     brackets, optionally followed by an EXTRA-INFO clause.
>   - A SIMPLE-IN-TEXT is a BARE-KEY, optionally followed by an
>     EXTRA-INFO clause
>   - A CITATION-LIST has the format
>        [cite: PREFIX; INDIVIDUAL-REFERENCE; ... INDIVIDUAL-REFERENCE; SUFFIX] EXTRA-INFO
>     where the initial PREFIX, final SUFFIX, and EXTRA-INFO clause are
>     optional.  At least one INDIVIDUAL-REFERENCE must be present.
>   - An INDIVIDUAL-REFERENCE has the format:
>        PREFIX KEY-MAYBE-PARENS SUFFIX
>     The KEY-MAYBE-PARENS is obligatory, and the prefix and suffix
>     are optional.
>   - A KEY-MAYBE-PARENS is either a BARE-KEY or PARENTHESIZED-KEY
>   - A BARE-KEY is a KEY with immediately-preceding whitespace
>   - A PARENTHESIZED-KEY is a KEY immediately surrounded by `(' and `)'.
>   - A KEY optionally begins with `-', and obligatorily contains `@' or
>     `&' followed by a string of characters which begins with a letter
>     or `_', and may contain alphanumeric characters and the following
>     internal punctuation characters:
>        :.#$%&-+?<>~/
>   - A PREFIX or SUFFIX is arbitrary text (except `;', `]', and
>     KEY-MAYBE-PARENs) which may contain only the following Org
>     objects:
>     - bold
>     - code
>     - entity
>     - italic
>     - latex-fragment
>     - line-break
>     - strike-through
>     - subscript
>     - superscript
>     - underline
>     - superscript
>     (Note that this list could be extended somewhat if necessary.)
>   - An EXTRA-INFO clause consists of data not specified by this
>     grammar, in between `%%(' and `)'
>
> ** Outstanding issues
> It seems to me that there are potential problems with the above
> proposal in a number of areas, but I cannot tell how serious they are,
> or what changes (if any) should be made to solve them.  I don't
> pretend that this is an exhaustive list:
>   1) *Nesting.*  I have favored LaTeX compatibility for in-text
>      citations with multiple references; but this means there is no
>      way to `nest' citations.  Thus, there is no way to express (in
>      the main syntax) what Pandoc expresses as:
>         @Doe99 [p. 34; see also @DoeRoe2000]
>      which renders like:
>         Doe (1999, p. 34; see also Doe and Roe 2000)
>      Instead, since a citation is in-text or parenthetical as a whole,
>      the equivalent in the above syntax
>         [cite: @Doe99 p. 34; see also @DoeRoe2000]
>      should render like:
>       Doe (1999, p. 34), see also Doe and Roe (2000).
>      I am not certain if Pandoc-like output is important in this case.
>      The few people who commented on this said that it was not.
>   2) *Limitations on prefixes and suffixes.*  There may be legitimate
>      uses of `@', `;', `]', etc. inside prefix or suffix text that the
>      above syntax does not allow.  Examples might include:
>      - use of semi-colons as part of the prefix/suffix text
>      - footnotes, links, or timestamps inside a prefix/suffix
>      I am not certain how important these cases are.  If they are
>      important, some of them might be able to be worked around with
>      entities.
>   3) *Edge cases.* The above syntax may make it possible to express
>      things that don't make sense, or would be too difficult to
>      export.  The only one I can think of is that it is possible to
>      mix `@'-style and `&'-style keys in the same citation.  I am not
>      sure if this should be forbidden; it may sometimes make sense.
>      It may also be possible to express things that external tools,
>      such as citeproc-js, don't know how to process.  I do not have a
>      good sense of what, if anything, falls into that category, and
>      what should be done about it.
>   4) *Citation commands.*  Rather than introduce an explicit
>      representation for different citation commands/types, I have used
>      different parts of the syntax to express the common distinctions
>      that people mentioned.  I suggest that, for now, anything beyond
>      these basic distinctions be left to the user-extension syntax.
>      However, if it becomes clear in the future that there is a need
>      to add a representation for a command to the main syntax, there
>      is a natural place to do so: immediately after the `cite:' tag
>      (as Nicolas suggested).
>
> Also, I have not said anything in this proposal to address how other
> document metadata should be represented, which has not been discussed
> much on the list.  I think this should be discussed separately.
>

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi John,

I don't have time for a long reply but I wanted to express a couple
points of agreement:

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> I think the usual suspects reftex, helm-bibtex, and probably ebib
> could be taught to output most of this syntax for whatever type, and
> they could give human readable hints about the intended format,
> e.g. intext, parenthetical, noauthor, etc... Or you could have
> dedicated commands with key completion to do that. So many options,
> this should not be an issue.

Yes, I would hope the syntax is fairly straightforward to generate.  

> Presumably each &/@key will be clickable like a link, and the function
> it runs would get the key (and maybe additional info about the cite)? If
> not, that would be a show-stopper to me.

Yes, that is certainly what I had in mind.  org-element may even be able
to provide support for this, so you don't have to parse the keys out
yourself in Elisp (though I think maybe this would require making keys,
in addition to complete citations, a category of object -- is that
right, Nicolas?).

> There is no question in my mind that some people will want to extend
> this, as there are just too few of the latex citation commands
> supported out of the box, especially for biblatex users (who used that
> because of limitations in bibtex ;). 

Do you think there are important commands that I missed?  I did try to
make sure that all the major distinctions in biblatex were covered,
though I ignored some more esoteric things like smartcite and volcite.
 
> My sense is the syntax may then be too verbose, and difficult to write
> exporters for and they would go back to links. That is probably a
> small number of people, and maybe I am wrong about it. I am anyway
> supportive enough to see it tried out.

I am a bit worried about this too; but one reason I suggested an
arbitrary s-expression for the %%(...) part was that it is flexible
enough to let people get very creative in making simple expressions for
particular needs.  For example, I thought this was a cute hack for
genitive citations:

> @McCarthy1958 %%('s) clever use of Lisp syntax...

If you use those enough, that's a lot nicer to type and read than

> @McCarthy1958 %%(:type genitive) clever use of Lisp syntax...

So I suggest we let a thousand flowers bloom, and see what people come
up with, rather than trying to cut down on the verbosity up front.

> My final comment is that I suggest two additional things to go with this
> syntax:
>
> [bibliographystyle: some-kind-of-information-probably-unsrt/alpha/chicago]
> This would tell some backend how to style the bibliography entries. This
> does not need to be clickable (I don't know what a click would do
> anyway, at most select the style? edit the style?).
>
> [bibliography: @some-kind-of-source-probably-a-file; @maybe-more-than-one]
> This is where the keys are stored. And, it would also indicate where the
> bibliography should actually be placed. This should also be clickable,
> with a default action to just open the file that was clicked on.
>
> I prefer those to file attributes, e.g.
> #+BIBLIOGRAPHY: @some-kind-of-source-probably-a-file;
> @maybe-more-than-one
>
> I don't think that can be used to specify where a bibliography should be
> placed, and it doesn't make sense to me to use two things to specify the
> same information.

Hmm, OK.  Let's discuss this in another thread.

> So, overall, I am on the positive side of zero.

Haha, leave it to a physical scientist to turn a discrete interval into
a continuous one... ;)

Best,
Richard

Re: Citation syntax: a revised proposal

[John Kitchin]

>
>> There is no question in my mind that some people will want to extend
>> this, as there are just too few of the latex citation commands
>> supported out of the box, especially for biblatex users (who used that
>> because of limitations in bibtex ;).
>
> Do you think there are important commands that I missed?  I did try to
> make sure that all the major distinctions in biblatex were covered,
> though I ignored some more esoteric things like smartcite and volcite.

I think the most common ones are there. It is just that over a decade in
academia has taught me that there are always people that do things
another way, for some reason, including to be difficult ;)


> So I suggest we let a thousand flowers bloom, and see what people come
> up with, rather than trying to cut down on the verbosity up front.

Also fine with me.


> Hmm, OK.  Let's discuss this in another thread.
>
>> So, overall, I am on the positive side of zero.
>
> Haha, leave it to a physical scientist to turn a discrete interval into
> a continuous one... ;)

Indeed! I could have gone with

+i

for I imagine it might work ;)

>
> Best,
> Richard

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Eric S Fraga]

On Saturday, 14 Feb 2015 at 18:29, Richard Lawrence wrote:
> Hi everyone,
>
> Since discussion seems to have petered out on the previous thread (see:
> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
> go back over the discussion and write up a concrete proposal for
> citation syntax.

+1 emphatically.

I would like to see something along these lines implemented so we can
"suck it and see"...

For my use case, you've covered everything and, as John Kitchin says,
most cite insertions could be automated through appropriate code and key
bindings but, very importantly, common uses are quite straightforward to
type directly.

With respect to the bibliography database, for completeness, I would
like to see linking with org-bibtex data instead of bibtex etc.

Thanks,
eric
-- 
: Eric S Fraga (0xFFFCF67D), Emacs 25.0.50.1, Org release_8.3beta-834-g836d9d

Re: Citation syntax: a revised proposal

[William Denton]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1087 bytes --]

+1 from me too, observing from the sidelines but also wanting to be able to 
handle citations.

Bill

On 16 February 2015, Eric S Fraga wrote:

> On Saturday, 14 Feb 2015 at 18:29, Richard Lawrence wrote:
>> Hi everyone,
>>
>> Since discussion seems to have petered out on the previous thread (see:
>> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
>> go back over the discussion and write up a concrete proposal for
>> citation syntax.
>
> +1 emphatically.
>
> I would like to see something along these lines implemented so we can
> "suck it and see"...
>
> For my use case, you've covered everything and, as John Kitchin says,
> most cite insertions could be automated through appropriate code and key
> bindings but, very importantly, common uses are quite straightforward to
> type directly.
>
> With respect to the bibliography database, for completeness, I would
> like to see linking with org-bibtex data instead of bibtex etc.
>
> Thanks,
> eric
>

-- 
William Denton ↔  Toronto, Canada ↔  https://www.miskatonic.org/

Re: Citation syntax: a revised proposal

[John Kitchin]

Eric S Fraga writes:

> On Saturday, 14 Feb 2015 at 18:29, Richard Lawrence wrote:
>> Hi everyone,
>>
>> Since discussion seems to have petered out on the previous thread (see:
>> http://thread.gmane.org/gmane.emacs.orgmode/94524), I took some time to
>> go back over the discussion and write up a concrete proposal for
>> citation syntax.
>
> +1 emphatically.

I still remain somewhat on the positive side of 0. While the focus of
these conversations has been on syntax (a necessary step to move
forward), there has been little focus on function. Citations in org are
/far/ more than just references in the text for me. They are functional
links, gateways to a lot of information connected to the citation. My
org-files are much more useful than the PDF manuscripts that get
exported. It is hard to explain what that means exactly, so if you have
the time check these videos out. The last one shows the most integrated
capabilities we use.

org-ref show: https://www.youtube.com/watch?v=JyvpSVl4_dg
org-ref in action: https://www.youtube.com/watch?v=Zya8SfmCtFA
org-ref + helm: https://www.youtube.com/watch?v=8cEb6F9AEu0

These features beat the pants off every citation manager software I have
ever seen, and over the past decade I have tried a lot of them. There is
in my opinion no syntax worth giving this up for. I am only moderately
sure the proposed syntax can be used the way we use links. I am only
mildly concerned some things will be too difficult to do that are
currently possible, e.g. sorting a multcite, or rearranging them with
arrow keys. Who knows, some things might be simpler.

> I would like to see something along these lines implemented so we can
> "suck it and see"...

I also would like to see if it is workable. There is nothing to lose in
trying. org-ref will always be an alternative option, and if the new
syntax is extendable enough it might even replace the citation links.

> For my use case, you've covered everything and, as John Kitchin says,
> most cite insertions could be automated through appropriate code and key
> bindings but, very importantly, common uses are quite straightforward to
> type directly.
>
> With respect to the bibliography database, for completeness, I would
> like to see linking with org-bibtex data instead of bibtex etc.

I think we can come up with a reasonable, user-customizable api that
would support multiple data sources. A user would customize the click
and export function. There might even be simple out of the box default
functions for org-bibtex and bibtex, and zotero through zotxt. For the
first two, the click function could just open the entry in the
bibliography file.

>
> Thanks,
> eric

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Nicolas Goaziou]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> I still remain somewhat on the positive side of 0. While the focus of
> these conversations has been on syntax (a necessary step to move
> forward), there has been little focus on function.

One step at a time. It's already difficult to agree on a syntax.

> Citations in org are /far/ more than just references in the text for
> me. They are functional links, gateways to a lot of information
> connected to the citation. My org-files are much more useful than the
> PDF manuscripts that get exported. It is hard to explain what that
> means exactly, so if you have the time check these videos out. The
> last one shows the most integrated capabilities we use.
>
> org-ref show: https://www.youtube.com/watch?v=JyvpSVl4_dg
> org-ref in action: https://www.youtube.com/watch?v=Zya8SfmCtFA
> org-ref + helm: https://www.youtube.com/watch?v=8cEb6F9AEu0

Very interesting. I wish we can provide some of these features out of
the box in a future release.

> These features beat the pants off every citation manager software I have
> ever seen, and over the past decade I have tried a lot of them. There is
> in my opinion no syntax worth giving this up for. I am only moderately
> sure the proposed syntax can be used the way we use links. I am only
> mildly concerned some things will be too difficult to do that are
> currently possible, e.g. sorting a multcite, or rearranging them with
> arrow keys. Who knows, some things might be simpler.

I'm really puzzled here. The very point of this new syntax is to provide
at least as much information as links, while being more readable in most
cases.

For example, I suggested

  [cite:subtype: whatever]

which is, in my book, equivalent to 

  [[subtype:whatever]]

I think other suggestions are as capable.

Again, the problem is not what you could do with the new syntax, but
what information it brings. AFAIU, you only need a "type" property,
which is pretty easy to provide. 

So, why do you think it would not be equivalent, feature-wise to links?


Regards,

Re: Citation syntax: a revised proposal

[John Kitchin]

Nicolas Goaziou writes:

> John Kitchin <jkitchin@andrew.cmu.edu> writes:
>
>> I still remain somewhat on the positive side of 0. While the focus of
>> these conversations has been on syntax (a necessary step to move
>> forward), there has been little focus on function.
>
> One step at a time. It's already difficult to agree on a syntax.
True enough.

>
>> Citations in org are /far/ more than just references in the text for
>> me. They are functional links, gateways to a lot of information
>> connected to the citation. My org-files are much more useful than the
>> PDF manuscripts that get exported. It is hard to explain what that
>> means exactly, so if you have the time check these videos out. The
>> last one shows the most integrated capabilities we use.
>>
>> org-ref show: https://www.youtube.com/watch?v=JyvpSVl4_dg
>> org-ref in action: https://www.youtube.com/watch?v=Zya8SfmCtFA
>> org-ref + helm: https://www.youtube.com/watch?v=8cEb6F9AEu0
>
> Very interesting. I wish we can provide some of these features out of
> the box in a future release.

Thanks! I would be happy to help where I can.

>
> I'm really puzzled here. The very point of this new syntax is to provide
> at least as much information as links, while being more readable in most
> cases.
>
> For example, I suggested
>
>   [cite:subtype: whatever]
>
> which is, in my book, equivalent to
>
>   [[subtype:whatever]]
>
> I think other suggestions are as capable.
>
> Again, the problem is not what you could do with the new syntax, but
> what information it brings. AFAIU, you only need a "type" property,
> which is pretty easy to provide.
>
> So, why do you think it would not be equivalent, feature-wise to links?

They are probably minor, but for example I am not sure how easy it would
be to sort a multicite with all of the syntax options. I guess it can be
done, I just do not see it clearly. It may not be necessary to do this
either.

When on a citation, I know how to append a new citation to it, or to
replace the citation at point, and I know what kind of trickery was used
to enable that. I don't know how simple it would be to do that with full
support of the new syntax. But again, maybe it is not necessary. It may
suffice to handle the simple cases.

An example of something easier might be if each @key is natively
clickable. I could do away with the code that has to figure out which
key you clicked on in a multicite.

These are only based on my experience building org-ref, the evolution
of ideas that came with it, and methods used to achieve it. I would not
let this get in the way of this syntax, but it is the baggage I have
thinking about it ;) These are the kinds of things I do almost every day
in our scientific writing. They are all kind of small, but collectively
they make it easy to do.

so, it is not just about the syntax, but what the syntax makes possible
to do. Real writing is messy, and not just about inserting citations. We
have to edit these when students do a bad job, or when we were sloppy or
incomplete. As the chief editor (a very different job than author!) I
want a syntax that enables easy editing too.

I am not too worried about it. Two years ago I would not have thought
org-ref was even possible. A year ago I did not think it would have the
features it has today. This community is very clever at solving
problems.


>
>
> Regards,

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Rasmus]

John Kitchin <jkitchin@andrew.cmu.edu> writes:

> They are probably minor, but for example I am not sure how easy it would
> be to sort a multicite with all of the syntax options. I guess it can be
> done, I just do not see it clearly. It may not be necessary to do this
> either.

Out of curiosity:

Why would this be hard?  Would it amount to more than replacing identity
in the below snippet with something that gets the true author?  You can
split out global pre and post note beforehand and reinsert it.  I'm
probably missing something.

(sort '((:key "foo" :pre "pre")
        (:key "bar" :post "pp. 22")
        (:key "baz"))
      (lambda (a b) (string-collate-lessp
                (funcall 'identity (plist-get a :key))
                (funcall 'identity (plist-get b :key)))))

—Rasmus

-- 
A clever person solves a problem. A wise person avoids it

Re: Citation syntax: a revised proposal

[John Kitchin]

That is probably something like it. I meant sorted by year (others may
prefer author), which also means looking up in the database, decorating,
and then sorting, but that doesn't fundamentally change your idea. Then,
from the sorted list, you have to regenerate the org-syntax and replace
the original text with the new string. hard may be an overstatment. I
just have the experience from org-ref that there were often corner cases
that I did not anticipate, with an even simpler syntax. I would hope the
new syntax and data structure make this easier, but I will reserve
judgment until it exists and we can try it.

I am confident some ambitious soul will set out to make a citation using
every possible variation, and then want to do something like sort it,
and expect it to be done correctly, and will file bug reports if it is
not. If not hard, it will definitely be more difficult! Still, as long
as it is possible, it is only hard one time, and then it is
done. Possible is the key. That said, there is a famous statment that it
is twice as hard to debug code as to write it, so if you your cleverest
at writing the code you are by definition not clever enough to debug it!

Like I said, this community is very clever at solving these kinds of
issues, so probably it is not a reason to not try something.

Rasmus writes:

> John Kitchin <jkitchin@andrew.cmu.edu> writes:
>
>> They are probably minor, but for example I am not sure how easy it would
>> be to sort a multicite with all of the syntax options. I guess it can be
>> done, I just do not see it clearly. It may not be necessary to do this
>> either.
>
> Out of curiosity:
>
> Why would this be hard?  Would it amount to more than replacing identity
> in the below snippet with something that gets the true author?  You can
> split out global pre and post note beforehand and reinsert it.  I'm
> probably missing something.
>
> (sort '((:key "foo" :pre "pre")
>         (:key "bar" :post "pp. 22")
>         (:key "baz"))
>       (lambda (a b) (string-collate-lessp
>                 (funcall 'identity (plist-get a :key))
>                 (funcall 'identity (plist-get b :key)))))
>
> —Rasmus

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Vaidheeswaran]

On Monday 16 February 2015 09:49 PM, Nicolas Goaziou wrote:

> [cite:subtype: whatever]

Nicolas, if you could circulate a one-off patch that handles the above 
syntax I will bump it against the ODT backend and JabRef engine.

I am waiting for the FSF representative to counter-sign my assignment 
papers (which is in transit).  I would very much like to see Citation 
support in the ODT exporter and work with you see the feature all the 
way through to the main-line.

Re: Citation syntax: a revised proposal

[Jorge A. Alfaro-Murillo]

0.

John Kitchin writes:

> Citations in org are /far/ more than just references in the text 
> for me. They are functional links, gateways to a lot of 
> information connected to the citation. My org-files are much 
> more useful than the PDF manuscripts that get exported. 

I completely agree.
 
> I also would like to see if it is workable. 

Same here.

> There is nothing to lose in trying. org-ref will always be an 
> alternative option, and if the new syntax is extendable enough 
> it might even replace the citation links.

The big advantage of org-ref is that it builds on BibTeX. From 
what I read in this and the previous thread, the new proposal 
tries more or less to reimplement BibTeX in org. (If it is not so, 
I apologize, because I confess that I haven't read every single 
email in these two veery long threads. If you plan to use BibTeX 
as a base, disregard the rest of the email).

BibTeX is IMHO one of the best pieces of software ever created. It 
had a stable version from 1988 to 2010, in 2010 it was last 
updated, because we started to cite with long URLs (not the norm 
in 1988). It is something that just works as it is supposed to and 
has all you need for creating bibliographies.

The biggest advantage of having something org/elisp native as in 
the proposal would be the implementation of functions to create 
bibliographies with a specific style, what Oren Patashnik called 
"Bibliography-style hacking", which is very cumbersome in BibTeX 
(maybe is just that I cannot read WEB/Pascal and have a strong 
preference for Lisp dialects).

Just let me drop the mandatory reading:

http://www.tex.ac.uk/tex-archive/bibliography/bibtex/base/btxdoc.pdf
http://www.tex.ac.uk/tex-archive/bibliography/bibtex/base/btxhak.pdf

and references therein.

Best,

-- 
Jorge.

Re: Citation syntax: a revised proposal

[Stefan Nobis]

jorge.alfaro-murillo@yale.edu (Jorge A. Alfaro-Murillo) writes:

> From what I read in this and the previous thread, the new proposal
> tries more or less to reimplement BibTeX in org.

No, that's wrong, not the database should be replaced. The goal is to
make citations a first class citizen in the org world (so no fallback
to LaTeX commands or links with special handlings are needed).

> The biggest advantage of having something org/elisp native as in the
> proposal would be the implementation of functions to create
> bibliographies with a specific style, what Oren Patashnik called
> "Bibliography-style hacking", which is very cumbersome in BibTeX
> (maybe is just that I cannot read WEB/Pascal and have a strong
> preference for Lisp dialects).

Hmmm... nowadays one uses biblatex[fn:1] (with its companion biber)
which makes hacking bibliography styles quite easy (in LaTeX; compared
to customizing bst files). I do not think that the current discussion
will lead to writing bib-styles in Lisp instead of LaTeX (at least not
in the foreseeable future).

[fn:1] http://ctan.org/pkg/biblatex

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[John Kitchin]

Stefan Nobis writes:

> jorge.alfaro-murillo@yale.edu (Jorge A. Alfaro-Murillo) writes:
>
>
> Hmmm... nowadays one uses biblatex[fn:1] (with its companion biber)
> which makes hacking bibliography styles quite easy (in LaTeX; compared
> to customizing bst files). I do not think that the current discussion
> will lead to writing bib-styles in Lisp instead of LaTeX (at least not
> in the foreseeable future).

This is not universally true. None of the journals we submit to accept
biblatex, only bibtex. I don't know of any major science publishers that
use biblatex.

>
> [fn:1] http://ctan.org/pkg/biblatex

--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

Re: Citation syntax: a revised proposal

[Jorge A. Alfaro-Murillo]

John Kitchin writes:

> Stefan Nobis writes: 
> 
>> Hmmm... nowadays one uses biblatex[fn:1] (with its companion 
>> biber) which makes hacking bibliography styles quite easy (in 
>> LaTeX; compared to customizing bst files). I do not think that 
>> the current discussion will lead to writing bib-styles in Lisp 
>> instead of LaTeX (at least not in the foreseeable future). 
> 
> This is not universally true. None of the journals we submit to 
> accept biblatex, only bibtex. I don't know of any major science 
> publishers that use biblatex.

Same here. Journals either do not support LaTeX or support LaTeX 
and BibTeX.

-- 
Jorge.

Re: Citation syntax: a revised proposal

[Jorge A. Alfaro-Murillo]

Stefan Nobis writes:

> jorge.alfaro-murillo@yale.edu (Jorge A. Alfaro-Murillo) writes: 
> 
>> From what I read in this and the previous thread, the new 
>> proposal tries more or less to reimplement BibTeX in org. 
> 
> No, that's wrong, not the database should be replaced. The goal 
> is to make citations a first class citizen in the org world (so 
> no fallback to LaTeX commands or links with special handlings 
> are needed).

I see, so in the examples provided Doe99 is only the key, org 
would not have to know that the author name is Doe and its year is 
1999, or any other information about the citation. I thought it 
was more or less the equivalent of implementing natbib 
(http://merkel.zoneo.net/Latex/natbib.php) in org, a way to decide 
if I wanted textual, parenthesis or numerical citations, and thus 
you would have to go to the process of determining what 
information each citation needs, for example: an article always 
has an author, a book always has a publisher, but a book can have 
an author or an editor if each chapter has a different author, 
that kind of thing, among many other complicated things. Sorry for 
the noise.

But now it is not clear to me what the actual org reference points 
to.  If it is the actual reference, I mean the article's PDF or 
URL, what would you do when you need to cite a physical book? Also 
in this case you will not be able to produce a proper bibliography 
when exporting since the key cannot contain all the information 
needed. Now if the reference points to be entry for that reference 
in a database, isn't there a lot of compatibility problems if one 
uses one type of database vs another? Again here you will not be 
able to produce the bibliography when exporting

-- 
Jorge.

Re: Citation syntax: a revised proposal

[Stefan Nobis]

jorge.alfaro-murillo@yale.edu (Jorge A. Alfaro-Murillo) writes:

> I see, so in the examples provided Doe99 is only the key, org would
> not have to know that the author name is Doe and its year is 1999,
> or any other information about the citation.

Yes and no. In the first place org should only get a special syntax
for citations. That means there will be special data structures for
citations and backends get a uniform interface for these parts of the
source text. In the simple case that's all, i.e. the backends get more
information to generate the correct commands (in the case of LaTeX) or
to call some tool that will generate the text/object to be inserted in
the resulting document.

On the other hand org should be able to show additional information
for citations, like linking to its data (in some bib file, in zotero
or wherever). But that's a second step.

> But now it is not clear to me what the actual org reference points
> to. If it is the actual reference, I mean the article's PDF or URL,
> what would you do when you need to cite a physical book?

The org element, say "[cite: see @doe99]", will point to some
data source, to be defined in the same org document (e.g. with
"#+BIBIOGRAPHY:..."). This data source for citations may be a bib
file, a zotero database, maybe even Endnote or something else.

As said above, org will not handle every aspect of citation. It should
only know a little more about these things in order to enable some
extra features (e.g. special UI for citations or exporting citations to
different backends instead of the need to fallback to LaTeX commands).

-- 
Until the next mail...,
Stefan.

Re: Citation syntax: a revised proposal

[Richard Lawrence]

Hi Eric,

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> +1 emphatically.

Thanks!

> With respect to the bibliography database, for completeness, I would
> like to see linking with org-bibtex data instead of bibtex etc.

Me too, as I keep all my reference data in org-bibtex.  I suggest we
discuss the bibliography and other document metadata in another thread.
 
Best,
Richard