Re: Citation processing via Zotero + zotxt

[Richard Lawrence <richard.lawrence@berkeley.edu>]

Hi Matt and all,

Matt Lundin <mdl@imapmail.org> writes:

> But for bibtex users, wouldn't there presumably have to be another
> zotero plugin that would allow for live, automated importing of bibtex
> into zotero? (If anyone knows whether such a plugin exists, please do
> let me know.)

Well, my hope is that this could be added to zotxt without much effort,
so we could still just depend on Zotero and zotxt.  The translation
capability already exists in Zotero; it's just a matter of exposing it
as an API, and I imagine that Erik would happily accept a patch to zotxt
that does so.

> A thought experiment... Do we need a fork of pandoc-citeproc?

(The name "org-citeproc" might be a bit misleading: it's not a fork of
pandoc-citeproc, but a small wrapper script that takes care of
marshalling data into and out of Pandoc, in a format that's easy to
generate and read from Emacs lisp.  But because it's written in Haskell,
building it still requires a full Haskell build chain, and distributing
it is non-trivial.)

> Or could we rather write an emacs-lisp wrapper that would feed
> citation data and a bibliography to pandoc and receive a string
> containing citations formatted in org syntax (one of pandoc's
> outputs). This could be done via an export filter, with all the
> necessary manipulation being done on the emacs/org side of things.

Yes, you're basically describing the approach that I eventually realized
org-citeproc should take: use the full capabilities of Pandoc to render
citations and bibliography in Org format, then re-parse these on the Org
side.  I did start to work on this, though I didn't finish and I'm not
sure if I pushed it to the public repo.

If we want to use pandoc-citeproc directly, instead of wrapping it with
something like org-citeproc, what we'd need to do is be able to
translate an Org document (or at least the citations within it) both to
and from pandoc-compatible JSON, since pandoc-citeproc reads and writes
in that format.  This can certainly be done in Emacs Lisp, and maybe it
would be worth doing, because it would mean that we'd gain a nice
serialization format for Org documents.

I am not opposed to this idea -- indeed, I kind of like it, which is why
I started work on org-citeproc in the first place.  Still, it would be a
non-trivial amount of work to develop this solution even to the point
that it can do what Zotero and zotxt can do right now.

> Javascript interpreters/engines are widely available for all platforms
> if we create a wrapper script around citeproc-js. Node itself is also
> easily available for most platforms. But we wouldn't need to set it up
> as a node server à la citeproc-node.

My concern here is with the wrapper script.  Yes, it's pretty easy to
install a javascript interpreter; but getting from there to the point
where you have a fully-working toolchain for processing citations from
Org mode is the problem.  What I think we should avoid is a process that
looks like:

1) Install node (or whatever interpreter)
2) Install citeproc-js and the wrapper script
3) Make sure the wrapper script is available as an executable that can
be called from Emacs
4) ...

Steps 2 and 3 are a bit much to ask people to do manually, and we don't
have the option of doing them through the system package manager.  They
could of course be automated by packaging up citeproc-js and a wrapper
script into a node package.  But again, that requires some work that no
one so far has volunteered to do; and anyway, it doesn't get away from
the worry about having to install packages/plugins from heterogeneous
sources.  (Also, we'd have to develop the wrapper script, and our own
way of translating items stored in BibTeX format into a format
compatible with citeproc-js.  This means duplicating efforts that have
already been put into both pandoc-citeproc and Zotero.)

It's a question of where to focus the limited resources we've got.  My
impression is that going with the combination of Zotero and zotxt will
represent the least amount of effort to get citations working on
non-LaTeX backends, for both Org developers and users.  I understand
that it won't be ideal for everyone, and there is time enough later on
to develop an alternative toolchain if that would better suit the needs
of people who don't want to depend on a GUI, etc.  I fully support that.
But until more people have time to work on this, it seems to me that
Zotero and zotxt represent the most practical path forward.
 
Best,
Richard