Re: Citation processing via Zotero + zotxt

[Richard Lawrence <richard.lawrence@berkeley.edu>]

Hi Rasmus and all,

Rasmus <rasmus@gmx.us> writes:

> Also, last I checked Zotero also existed as a standalone manager.  If this
> also works re your points below, using Zotero is a non-issue.  There
> exists a plugin to keep a bibtex file up to date.  I don’t know if it’s
> two way, though.

I haven't looked into whether things work the same way when Zotero is
installed as a standalone program, but I don't imagine they'd be
significantly different.  (What do you mean by "using Zotero is a
non-issue"?  Do you think installing as a standalone program would be
better, or did I misunderstand?)


>> Previously, I thought that it would be a similar amount of work to
>> communicate with Zotero from Emacs as any of the other CSL
>> implementations out there.  However, after looking at zotxt a bit more
>> closely, I discovered that it has an (undocumented) API endpoint [3]
>
> This sounds amazing, but also dangerous.  Do you know whether stabilizing
> the API has been discussed upstream?

As Matt mentioned, "upstream" in this case is zotxt, not Zotero.  I have
heard from Erik that he's open to changing and stabilizing this API.

>> that pretty much does exactly what we need: it accepts a list of
>> citation objects, and returns a list of formatted citations and a
>> formatted bibliography, which can be inserted into the exported
>> document.
>
> Could you give an example of the sort of input you give?  I.e. is it based
> on keys?  From my bibtex-centric world view I imagine something like:
>
>    I send key or pointer @K to a DB entry as well as a CSL-file pointer C,
>    and maybe a desired output format F.  I get a string back that is the
>    formatting of the data behind @K formatted according to the rules in C,
>    adapted to F.

Yes, that's correct, except that ideally you send the data for all
citations at once (because context is important).  So for example, if
Erik accepts a patch for some small changes I wrote, one can query

/zotxt/bibliography?outputFormat=html&style=chicago-fullnote-bibliography

sending POST data like

[
    { "citationItems": [{"key": "0_ZOTKEY1"}],
      "properties": {"noteIndex": 0}},
    { "citationItems": [{"key": "0_ZOTKEY2"}]
      "properties": {"noteIndex": 0}},
    ...
]

and get back data that basically looks like

{
  "bibliography": [
    {
      ...
      "bibstart": "<div class=\"csl-bib-body\">\n",
      "bibend": "</div>"
    },
    [
      "  <div class=\"csl-entry\">Formatted entry for 0_ZOTKEY1</div>\n",
      "  <div class=\"csl-entry\">Formatted entry for 0_ZOTKEY2</div>\n",
      ...
    ]
  ],
  "citationClusters": [
    "Formatted citation for 0_ZOTKEY1",
    "Formatted citation for 0_ZOTKEY2",
    ...
  ]
}

> Is that correct?  If so, does it support html, text and odt?

At the moment it supports html and text.  I suppose it could be made to
support ODT, though I'm not sure how difficult it is.  However, I think
a better solution would actually be to have it return *Org* markup, and
then replace citation objects in the document with that.  The main
problem this solves concerns note-based styles: we can insert Org
footnotes into the document at the beginning of the export process, and
then let Org figure out how to number the total set of footnotes in its
usual way.  I think it's a lot harder to deal with citation footnotes if
they come already-formatted in the output format.

>> Erik has also written a package for communicating with zotxt from Emacs,
>> zotxt-emacs [4], which is available on MELPA.  This package already
>> contains a lot of useful functions for querying the Zotero database and
>> inserting reference data into documents, including links in Org
>> documents.  I think it would be pretty straightforward to extend this
>> package to provide a nice UI for writers who are inserting citations
>> into Org documents, including search-based lookups of keys, etc.
>> Perhaps org-ref could also be taught to communicate with zotxt (with or
>> without zotxt-emacs) without too much work.
>
> I guess we’d need to convince Erik to move it to GELPA.  Fortunately (from
> that point of view), neither zotxt nor zotxt-emacs seems to have too many
> contributors.

Is the concern here that Org should not officially depend on anything
that isn't in GELPA?  Or just that it would make installing easier for
users?

>> I know that many people (perhaps especially the `power users' who have
>> been active in the citations discussion so far) prefer to maintain their
>> reference database without the aid of a GUI reference manager like
>> Zotero.
>
> I think this solves part of that issue.  Again it could be a dangerous
> solution.  Alternatively, perhaps it’s possible to feed-in a bibtex file
> to Zotero via an API.  This would also limit the damage.

I am hoping it will prove simple to feed a .bib file to Zotero via an
existing API, or to add such an API to zotxt.  But at the moment that's
just a hope.  (If anyone knows more about how to access a BibTeX
translator in a Zotero plugin, please get in touch!)

> IMO we can leverage zotero as a tool, but we cannot enforce it as a
> bibliography manager.

Agreed.  

Best,
Richard