Re: citations: org-cite vs org-ref 3.0

[John Kitchin <jkitchin@andrew.cmu.edu>]

I do not think it is productive for the community to say or consider it
is a sad situation. Many good things have emerged from these
discussions, even if it is not yet consensus on a solution. It is a
complex problem, with many years of effort by many people on each side.
That is an indication of how ambitious this project is and that there
may be more than one solution that is needed. It pains me quite a bit
there is a sentiment of fractionation, and that I may be contributing to
it.

My regular job workload the past few years has been crushing, and I have
not had the time to participate in this that I wish I had. I am not sure
I can add much here without sounding or feeling defensive about org-ref,
and my decision to continue supporting and developing it. I have thought
about this for most of the day, and in the (very long, apologies in
advance) response that follows I will do my best to provide a balanced
perspective (from my point of view) on the situation.

Some specific context that is important to me is that I wrote org-ref
long ago to solve a specific problem for me in the preparation of
scientific publications that are destined for LaTeX export. I intended
it to provide nearly equivalent bib(la)tex citation export, and as
reasonable an export as possible for everything else. I use org-ref
professionally, and it is a complete solution for me. I simply cannot
compromise on the capability org-ref provides me, or wait for an
alternative complete solution in org-mode. I have work I have to do now,
and org-ref lets me do it. This alone is reason enough for me to
continue using, developing and supporting org-ref. I understand org is
not intended to be a substitute for writing LaTeX, but it is a fact of
my job that I have to do that.

There are more than 8 years of legacy org-ref documents. I have written
40+ scientific papers with it, and countless technical documents with
more than 8000 cite links among them. org-ref has exceeded 190K
downloads from MELPA, so I feel obligated to maintain org-ref for
myself, and those users. org-ref may be heavyweight in bundling a lot of
capability together that could be separated into individual packages,
but it is also convenient for people who need it, and a GitHUB issue or
pull request away from new features. I remain committed to supporting
this, and I do it in a way I can manage, hence the monolithic package
design.

org-cite was also developed to solve some specific citation problems for
others that org-ref did not address well at the time it was started. I
believe those were issues like better pre/post note support, and
integration with CSL.

I think org-ref and org-cite have different priorities, they solve
different problems with different approaches, and they have different
pros and cons. I believe there are mutually incompatible compromises one
must make here because the specific choices you make determine what is
easy and what is possible. There is not a direct mapping of bib(la)tex
and CSL as a citation processor. They are different programs and they
don't share citation commands, or style information. It is inevitable
that something will be lost in the translation between these from a
single source like org, and whether that loss is acceptable depends on
what you need. For many things, close enough is ok for me, but for
manuscripts and proposals, they must be perfect, and in bib(la)tex form
for the journals I publish in. It is not that one thing is possible in
one and not the other; it is that you have to compromise one to do the
other no matter what you choose and that typically makes one thing or
the other easier to do. I am not even sure it is possible to do
everything one can do in bib(la)tex with CSL, for example, I don't know
the equivalent of \citenum in CSL. org-ref sides on making bib(la)tex
easy, and CSL is possible.

With org-cite, CSL can be readily used across export backends, more
bibliography database formats are supported, and it is also possible to
get LaTeX export, although there is no goal to fully support all of
bib(la)tex. A pure org approach (e.g. org-files as bibliography
database) can be used, there are a lot of CSL styles to work with, and
you can develop or adapt your own style if needed. With reasonable
defaults this should be a straightforward introduction to using
citations for new users that works out of the box. I support that.

With org-ref, bib(la)tex export is almost fully supported, and is easy,
and other exports via CSL are possible. Of course, you must have a
working LaTeX installation, only bibtex databases are supported, and
working knowledge of bib(la)tex is required to leverage this. Whether
you want it or not, you get a lot of extra utilities for getting bibtex
entries from a DOI, and support for cross-references, indexes,
glossaries and acronyms. This is a lot to ask of people who don't need
it, and convenient for those who do. I support this.

Which one of these approaches is right depends on what you want to be
easy. Neither is right or wrong, that is determined by what you need at
the time and what you prioritize in your solution. It is even possible
you need both approaches at different times. The two approaches are not
compatible, but it is org-mode after all, and you can certainly convert
back and forth between them for the most part.

Both projects have benefited from this discussion a lot. org has
org-cite now, and org-ref now handles pre/post notes like org-cite does,
it supports CSL much better, and is even a little more modular, lighter
weight, and more easily integrated with other completion backends than
ivy or helm. That should broadly be viewed as a win-win situation.

Here are some factors that have prevented me from deprecating org-ref. I
spent about a month and half trying to get a solution to this at
https://github.com/jkitchin/org-ref-cite, and I don't make these
observations lightly. That solution was complete and highly functional
for org-cite, but as I describe below not a replacement for org-ref at
this time. I still welcome a new maintainer for this code.

Cross-references are critical for me; without them, there is no path
forward for me with org-cite. I did work on a cross-reference approach
that leveraged org-cite syntax
(https://github.com/jkitchin/org-ref-cite/issues/16), but there was not
much appetite for the approach so I abandoned that. There are
cross-reference capabilities in org, that may be suitable for some
applications, but they do not come close to what org-ref offers, and
that the kind of technical documents I write require. For me, any
cross-reference capability would also have to support what is possible
in LaTeX, and preferrably look similar to the LaTeX commands. It has not
been possible to write an orthogonal package that could co-exist with
org-ref to address this; this would require a new syntax for
cross-references in my opinion. My opinion is that practically citations
and cross-references are just links to a place in your document (either
a figure/table/section/etc, or an entry in a bibliography). In org-ref,
both are represented by links; of course, they have different types and
functions, but they both have follow like actions. My opinion seems to
be in the minority on this.

I do not like the abstraction away from LaTeX cite commands in org-cite.
This is an example of a compromise between LaTeX and CSL. They do not
share common citation commands, so you can either choose one or the
other, or make a new abstraction that generalizes them. I strongly favor
the LaTeX commands because I write for LaTeX export, and there is
extensive documentation for how to cite in LaTeX and what to expect.
Clearly org-cite favors using the standardized abstractions in org-cite,
and then mapping them to the LaTeX commands I think. My perspective on
this is partially one of an educator; I have taught a lot of people how
to use org-mode for technical writing. This layer of abstraction adds
additional complexity to documentation, and in teaching people what to
do. As a LaTeX-centric user, that abstraction adds an additional
cognitive load while writing I find unwelcome. I concede that it also
reflects "what I do", that org is not LaTeX, and that others may have a
more CSL centric perspective. org-cite is for them. I can see that the
abstraction away from the LaTeX cite commands strengthens the org is not
LaTeX philosophy, and will serve part of the community well. If I wasn't
required to generate LaTeX documents, and teach others how to do it, I
would not feel so strongly.

It is my opinion that the modularity of org-cite is a challenge. I think
it is too difficult to configure, and difficult to support, even more so
than org-ref. I know my opinion differs from many on the list who want
modularity and configurability. I have supported org-ref since around
2014, and even the modularity there (helm or ivy) has been a challenge
to support. org-ref has always been configurable, but monolithic. Still,
I learned a lot from Bruce (thank you!) that pushed me to redesign parts
of org-ref to be more modular and more easily pluggable to other
completion backends, and less specific on ivy/helm where practical.
There are limitations to this I learned about, compromises one has to
choose in doing it, and consequences in maintenance, support and
documentation from them. We still don't fully agree on some of these
points, but org-ref is closer to that ideal than it was. Maybe one day I
will abandon ivy like I did helm many years ago and feel differently
about this, but that day is sufficiently far away that I don't see it
now. 

Finally, the org-cite code is magnificent, and written at a level well
above my coding skills. I am grateful to those who wrote it, and
especially to Nicolas, for the opportunity to learn from it. The code I
wrote in org-ref-cite was challenging. org-cite uses (IMO) advanced
emacs-lisp techniques, and more complex data structures than I am
accustomed to. I learned a lot studying the org-cite code, but I will be
honest that I find it difficult to make contributions to. That gave me
pause in continuing to develop it. It is fair to say that org-cite
showed me some ways to address limitations of org-ref that I did not see
before, org-ref is better for it, and the writing community that uses
pre/post notes and biblatex is much better served as a result.

Where does this leave me, org-ref and org-cite? I still have differences
of opinion on design choices between them, and those differences are
likely irreconcilable. These differences arise from my experiences in
writing, teaching, using, developing and supporting org-ref. For those
who need high fidelity LaTeX export like I do, I think org-ref is still
a superior solution. For everyone else, and especially if you do not
need sophisticated cross-references and don't want the dependencies of
org-ref, org-cite is likely the better solution. 

I am content to agree to disagree on these points and move forward with
both packages because they solve different problems, are suitable for
different communities, and they continue to benefit each other. I can
see not everyone sees this as a positive situation though, and that has
weighed heavily upon me lately. These times are heavy enough. Anyway,
this turned out much longer than I expected, so thanks everyone who has
contributed to making org better, I hope I got things mostly correct,
you found it a fair assessment, we might still be friends, and thanks
for reading to the end.

j

Nicolas Goaziou <mail@nicolasgoaziou.fr> writes:

> Hello,
>
> Vikas Rawal <vikasrawal@gmail.com> writes:
>
>> This obviously creates many problems including that two people using
>> different citation systems cannot share org files.
>
> Indeed.
>>
>> What is the general view of the community about this?
>
> I don't know about the general view of the community, but, as a data
> point, I find it very sad.
>
> That's not helpful, tho.
>
> Regards,


-- 
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his