Expanding how the new cite syntax is used to include cross-references - thoughts?

Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 2730 bytes --]

To preface this, let me note that I am looking for a way to replace org-ref
with a complete and orthogonal package for writing scientific papers with
citations and cross-references. (I can probably live without the glossary
and acronym support). The cross-references in org-ref were handled with
org-links, and I think it would be a mistake to try to do that again
because the links will be defined by the last package loaded which will
lead to issues. So, I would like what I call an orthogonal approach for
cross-references, orthogonal in the sense that it can coexist with org-ref,
but not require org-ref if you don't use it. The built in internal links
like [[label]] lack the flexibility I need (I think), e.g. to differentiate
a typical reference from an equation reference (these map to \ref{label}
and \eqref{label} in latex for example), and there are other references
that are relevant in latex also like \pageref{label}, \cref{label} etc.

While working on org-ref-cite, it occurred to me that we can use the new
org-cite syntax for cross-references, without any changes to the syntax, we
can just use a different style. This would enable a near complete and
orthogonal replacement of org-ref. These cross-references are like a
citation IMO, but to a separate set of targets, i.e. not to bibliography
entries but rather to locations within the document, and possibly even to
other documents.

e.g. [cite/ref:@label] would be a cross-reference to "label" defined in a
CUSTOM_ID on a heading, a NAME keyword, or in raw latex like \label{label}.

You could use substyles for equations, or page refs, etc. e.g.
[cite/ref/eq:@label] for an equation.

You could also use a prefix to indicate a figure, table, equation, e.g.
[cite/ref/eq:Equation @label].

This does add a bit of complexity to the processors, which have to do
different things for the different kinds of cites (for activation,
insertion, following and exporting), and that is based only on the style. I
would not expect these styles to be fully supported by all the processors.
For LaTeX though, it would make for a near total and orthogonal replacement
of org-ref.

You can see what this would look like at
https://github.com/jkitchin/org-ref-cite/issues/16. I use style-dependent
fontification to differentiate bibliographic citations from
cross-references. They also have different follow behaviors.

Anyway, I am curious what you all think. Is it a good idea? Worth waiting
for or thinking of a better solution?


John

-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[-- Attachment #2: Type: text/html, Size: 3261 bytes --]

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

Hi John,

On Tue, Aug 10, 2021 at 8:58 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> ... I would like what I call an orthogonal approach for cross-references, orthogonal in the sense that it can coexist with org-ref, but not require org-ref if you don't use it. The built in internal links like [[label]] lack the flexibility I need (I think), e.g. to differentiate a typical reference from an equation reference (these map to \ref{label} and \eqref{label} in latex for example), and there are other references that are relevant in latex also like \pageref{label}, \cref{label} etc.

Granting the current "lack of flexibility" of internal links, what
about extending them to allow an optional type; like:

[[eq:label]]
[[table:label]]

... etc?

It could work based on similar logic as citation styles, namely that
content preceding some delimiter (in the examples above, the hyphen)
would be an optional link type. As you say, it could even have a
similar style/variant structure as citations.

Perhaps a handful such link types could be reserved, for obvious org
link targets (like tables, equations, figures, etc), but it would be
flexible beyond that.

Would that not work?

Because if it could, that would seem to be a better solution, both for
cross-references, and for citations.

I do grant there may be a challenge with legacy documents adding such
an optional type, but perhaps there's a solution to that problem?

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Tom Gillespie]

In general I like John's suggestion. Org link syntax can be made to do
nearly anything because it is possible to bind link actions to
arbitrary elisp functions (I have used them to create buttons that run
source blocks for some of my non-technical colleagues). The grouping
of cross references under org-cite seems reasonable to me, and I would
love it if they could handle arbitrary references, e.g. to hypothesis
web annotation links or org-capture links.

Actually, having written this now, I think that both solutions have
their own use cases. Org cite is clearly about providing evidence for,
or a scholarly reference for something, and critically it can embed
some metadata about that reference in the document as a citation or
perhaps as an excerpt (and extension of what org-ref does now when the
cursor is over a reference?). Regular links do not provide any way to
embed metadata within the document, they are purely pointers.

That being said, it seems that there are a number of use cases where
org-ref links are simply internal document links that can point to an
element with a specific #+name: and no embedded information about the
target is needed. However, I think it would be a mistake to use up
equation/eq and table/tbl or figure/fig prefixes for references that
are internal to org, because it implicitly limits/collides with the
#+link: keyword.

Best,
Tom

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

Here's a recent subthread on this question:

https://lists.gnu.org/archive/html/emacs-orgmode/2021-07/msg00233.html

At the end of that discussion, my argument against using citations for
cross-references:

1. Cross-references are not citations, neither conceptually, nor in
software implementations. In LaTeX, MS Word, Libre office, InDesign,
etc, cross-references are handled differently than citations. There,
they are typed internal links. You can get a sense of how this works
in this tutorial for Word, which includes a list of cross-reference
types, and so hints at the range of things people need to internally
reference:

https://www.customguide.com/word/how-to-cross-reference-in-word

2. As John and Joost noted on that thread, because they're different,
they raise a range of implementation questions, most notably for me
what org-cite processors are supposed to do with these citations that
are not citations. As it is now, the user would just get errors and/or
unexpected output.

On Wed, Aug 11, 2021 at 1:28 AM Tom Gillespie <tgbugs@gmail.com> wrote:

...

> Actually, having written this now, I think that both solutions have
> their own use cases. Org cite is clearly about providing evidence for,
> or a scholarly reference for something, and critically it can embed
> some metadata about that reference in the document as a citation or
> perhaps as an excerpt (and extension of what org-ref does now when the
> cursor is over a reference?). Regular links do not provide any way to
> embed metadata within the document, they are purely pointers.

Right, which is what a cross-reference is.

It's just there needs to be some way to distinguish among types of
targets, I think.

> I think it would be a mistake to use up equation/eq and table/tbl or
> figure/fig prefixes for references that are internal to org, because it implicitly
> limits/collides with the #+link: keyword.

Is there a workaround for this somehow, or an alternative that gets
the same thing in the end?

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Wed, Aug 11, 2021 at 7:13 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> Is there a workaround for this somehow, or an alternative that gets
> the same thing in the end?

Like, if not typing the link, type the target?

From this example from org-mode manual, the "fig" in the name keyword
could provide that type?

#+CAPTION: This is the caption for the next figure link (or table)
#+NAME:   fig:SED-HR4049
[[./img/a.jpg]]

Or some other metadata on the target?

Bruce

https://orgmode.org/manual/Images.html
https://orgmode.org/manual/Internal-Links.html

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Hi John,
>
> On Tue, Aug 10, 2021 at 8:58 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>
>> ... I would like what I call an orthogonal approach for cross-references,
>> orthogonal in the sense that it can coexist with org-ref, but not require
>> org-ref if you don't use it. The built in internal links like [[label]] lack
>> the flexibility I need (I think), e.g. to differentiate a typical reference
>> from an equation reference (these map to \ref{label} and \eqref{label} in
>> latex for example), and there are other references that are relevant in latex
>> also like \pageref{label}, \cref{label} etc.
>
> Granting the current "lack of flexibility" of internal links, what
> about extending them to allow an optional type; like:
>
> [[eq:label]]
> [[table:label]]

Those look like regular org links, and are handled differently from [[label]]
which is called a fuzzy link if the links are defined. Interestingly, if
the org-links are not defined, they are still fuzzy links, but that is
going to be super confusing because the syntax is so similar. In any
case, this is substituting using one syntax for another purpose for
another syntax.

No extension is needed for regular links, it is already a feature of
org-links which is used heavily in org-ref already does via ref:label,
eqref:label, etc. But to use them, you would have to install org-ref,
and it would come with helm and ivy, which many people do not like. I
don't want to redefine these links because some people (like me) will
have to have both packages installed for legacy documents, and the link
behavior is what ever is loaded last.

It is not necessary to differentiate figures and tables that way, imo, and you
can use different kinds of references to the same label anyway. E.g.
\ref{tab-1} (in LaTeX) would refer to the table number, whereas
\pageref{tab-1} would refer to the page number it is on. Obviously, that
has less meaning for html where there is a single page.


>
> ... etc?
>
> It could work based on similar logic as citation styles, namely that
> content preceding some delimiter (in the examples above, the hyphen)
> would be an optional link type. As you say, it could even have a
> similar style/variant structure as citations.
>
> Perhaps a handful such link types could be reserved, for obvious org
> link targets (like tables, equations, figures, etc), but it would be
> flexible beyond that.
>
> Would that not work?

I don't think this would work. org-ref already defines most or all of
these links. Maybe we could use new names like ref/:label ref/eq:label,
etc though. They seem to be legal names. It still seems like there is a
lot of opportunity for mixing exports from org-ref and org-ref-cite this
way though, because the syntax is so similar. I want as clear a
separation of these packages as can be had.

>
> Because if it could, that would seem to be a better solution, both for
> cross-references, and for citations.
>
> I do grant there may be a challenge with legacy documents adding such
> an optional type, but perhaps there's a solution to that problem?
>
> Bruce


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Wed, Aug 11, 2021 at 7:13 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
>> Is there a workaround for this somehow, or an alternative that gets
>> the same thing in the end?
>
> Like, if not typing the link, type the target?
>
> From this example from org-mode manual, the "fig" in the name keyword
> could provide that type?

The "fig" part is only useful imo, for prefix text, eg. If I write "see
\cref{fig:SED-HR4049}" in text, it will get exported in LaTeX as
something like "see Figure 1", where the cleveref package inferred that
the Figure prefix should go in front of the number.

Otherwise, "see \ref{fig:SED-HR4049}" would just export to "see 1",
which is not what you want to see. You can manually add prefix text of
course, like "see Figure \ref{fig:SED-HR4049}".

However, if I am in a long document (like a book), I may want to refer
to a page number, e.g. "See pg. \pageref{fig:SED-HR4049}" to indicate to
the reader where to find some information. In this case the fig prefix
of the label is not important.

With sections, you may want \ref{sec-intro} to get the number of the
section, or \nameref{sec-intro} to get the name of the section. Both
cases use the same label.

>
> #+CAPTION: This is the caption for the next figure link (or table)
> #+NAME:   fig:SED-HR4049
>
> [[./img/a.jpg]]
>
> Or some other metadata on the target?

I don't think metadata on the target helps with the cases described
above, you can reference a label in different ways at different times to
get different meanings.

>
> Bruce
>
> https://orgmode.org/manual/Images.html
> https://orgmode.org/manual/Internal-Links.html


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> Here's a recent subthread on this question:
>
> https://lists.gnu.org/archive/html/emacs-orgmode/2021-07/msg00233.html
>
> At the end of that discussion, my argument against using citations for
> cross-references:
>
> 1. Cross-references are not citations, neither conceptually, nor in
> software implementations. In LaTeX, MS Word, Libre office, InDesign,
> etc, cross-references are handled differently than citations. There,
> they are typed internal links. You can get a sense of how this works
> in this tutorial for Word, which includes a list of cross-reference
> types, and so hints at the range of things people need to internally
> reference:
>
> https://www.customguide.com/word/how-to-cross-reference-in-word
>
> 2. As John and Joost noted on that thread, because they're different,
> they raise a range of implementation questions, most notably for me
> what org-cite processors are supposed to do with these citations that
> are not citations. As it is now, the user would just get errors and/or
> unexpected output.

This is also the case with org-ref links, if you haven't installed or
loaded org-ref.



>
> On Wed, Aug 11, 2021 at 1:28 AM Tom Gillespie <tgbugs@gmail.com> wrote:
>
> ...
>
>> Actually, having written this now, I think that both solutions have
>> their own use cases. Org cite is clearly about providing evidence for,
>> or a scholarly reference for something, and critically it can embed
>> some metadata about that reference in the document as a citation or
>> perhaps as an excerpt (and extension of what org-ref does now when the
>> cursor is over a reference?). Regular links do not provide any way to
>> embed metadata within the document, they are purely pointers.
>
> Right, which is what a cross-reference is.
>
> It's just there needs to be some way to distinguish among types of
> targets, I think.

At the risk of repeating myself, I don't think this is true. There needs
to be a way to distinguish among styles of cross-references, just like
there is for citations, and for the same reason: different contexts
require different styles. In citations, you might want to see the year,
author, number, or some combination of those, and there are different
styles for those that all use the same key.

In cross-references, you might want to see a figure/table number, and
these may be styled differently than an equation. You might also want
the page number a label is on, or the name of a section. In MS Word, it
is even possible to use the caption of a figure or table. These are just
different styles to use on a label.

As I have thought about this more, the line between citations and
cross-references has blurred. In org-ref, they were handled the same
way, with org-links (although each link had its own export function).
For both cites and refs, the links are just pointers, and in both cases
it is possible for them to point to things within the same document.
Even in LaTeX, when we submit a manuscript, the citation references are
embedded in a standalone tex file, so every link is to an internal
location.

The line blurs even further if you use something like org-bibtex, with
entries in the same document. Then, you can make a cross-reference to a
heading, or a citation reference to a reference entry.

>
>> I think it would be a mistake to use up equation/eq and table/tbl or
>> figure/fig prefixes for references that are internal to org, because it implicitly
>> limits/collides with the #+link: keyword.
>
> Is there a workaround for this somehow, or an alternative that gets
> the same thing in the end?




>
> Bruce


--
Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu
Pronouns: he/him/his

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Wed, Aug 11, 2021 at 9:53 AM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> > #+CAPTION: This is the caption for the next figure link (or table)
> > #+NAME:   fig:SED-HR4049
> >
> > [[./img/a.jpg]]
> >
> > Or some other metadata on the target?
>
> I don't think metadata on the target helps with the cases described
> above, you can reference a label in different ways at different times to
> get different meanings.

OK, this clarifies a key piece then.

I agree then: target metadata alone isn't enough.

So I see two options:

1. per my original response, allow optional typed internal links; e.g.:

[[fig/foo:file]]

To be clear, with this idea, I'm suggesting an alignment between
external links (which already have similar types) and internal (which
do not).

IUC, org-ref is using *external* link types (not internal links) to
make these distinctions?

2. extend citations, per your idea here, which to me means the
org-cite code would need to be revised and expanded to handle both
cross-references and citations, but do so distinctly.

E.g. an export processor like oc-csl would need to handle these
cross-references in that code, but pass the citations per se to the
citeproc-el backend, which should not need to worry about
cross-references.

Likewise, as a front-end developer, I don't want to deal with
cross-references at all, so I should be able to ignore them in my
insert and follow processors, in the same way you would need to be
able to include support for them.

And activate processors would need to be updated to distinguish them somehow.

Right?

It seems to me 1 is the easier path, both practically and
conceptually, unless I'm still missing something (which is certainly
possible).

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 3959 bytes --]

On Wed, Aug 11, 2021 at 10:32 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> On Wed, Aug 11, 2021 at 9:53 AM John Kitchin <jkitchin@andrew.cmu.edu>
> wrote:
>
> > > #+CAPTION: This is the caption for the next figure link (or table)
> > > #+NAME:   fig:SED-HR4049
> > >
> > > [[./img/a.jpg]]
> > >
> > > Or some other metadata on the target?
> >
> > I don't think metadata on the target helps with the cases described
> > above, you can reference a label in different ways at different times to
> > get different meanings.
>
> OK, this clarifies a key piece then.
>
> I agree then: target metadata alone isn't enough.
>
> So I see two options:
>
> 1. per my original response, allow optional typed internal links; e.g.:
>
> [[fig/foo:file]]
>

This is a bad idea to me. It is only a fuzzy link (what you call a typed
internal link) if no one has defined it as an external link. As soon as
that happens, then you will lose the export behavior defined for fuzzy
links, and get the behavior defined by the export function. The syntax for
these is the same (I guess a fuzzy link must be wrapped in [[ ]], but an
external link may also be wrapped that way). I don't see a  way to
differentiate these.

>
> To be clear, with this idea, I'm suggesting an alignment between
> external links (which already have similar types) and internal (which
> do not).
>
> IUC, org-ref is using *external* link types (not internal links) to
> make these distinctions?
>

Yes, org-ref uses external links, defined by org-set-link-parameters.


>
> 2. extend citations, per your idea here, which to me means the
> org-cite code would need to be revised and expanded to handle both
> cross-references and citations, but do so distinctly.
>

I don't think so. You only have to extend them where you want to have the
capability. org-cite can stay a citation only body of code, and if you want
cross-references too you just use my processors, or write one that does
what you want.


>
> E.g. an export processor like oc-csl would need to handle these
> cross-references in that code, but pass the citations per se to the
> citeproc-el backend, which should not need to worry about
> cross-references.
>

Again, not necessarily, this is handled in the export processor, and it can
differentiate which citations are sent to oc-csl and which are handled
differently based on the style.


>
> Likewise, as a front-end developer, I don't want to deal with
> cross-references at all, so I should be able to ignore them in my
> insert and follow processors, in the same way you would need to be
> able to include support for them.
>

You don't have to. Your processors should fail gracefully in any case,
because you just cannot control what people will do with the styles.


>
> And activate processors would need to be updated to distinguish them
> somehow.
>

Right now this just looks like

(when (org-ref-cite-ref-p reference) ...)

My activate processor just runs a bunch of functions and those functions
can do nothing if needed.


> Right?
>
> It seems to me 1 is the easier path, both practically and
> conceptually, unless I'm still missing something (which is certainly
> possible).
>

The biggest issue with 1 is I don't think it is possible to differentiate
internal typed links from external links because they use the same syntax.
It would be very difficult to support and troubleshoot a scenario where the
same syntax shows different behavior depending on whether an external link
is defined or not. Finally, it is so close to what org-ref already does, I
think it is too tricky to differentiate [[ref:label]] from [[ref/:label]],
etc.

Maybe these links would look different enough (as external links) that it
would be really clear you were not using org-ref.

ref/:@label
ref/eq:@label
ref/page:@label
ref/name:@label
ref/c:@label
ref/C:@label

 However, this doesn't support using prefix/suffix text, which is one of
the main reasons the cite syntax came to be.


> Bruce
>

[-- Attachment #2: Type: text/html, Size: 6077 bytes --]

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Wed, Aug 11, 2021 at 10:56 AM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> This is a bad idea to me. It is only a fuzzy link (what you call a typed internal link) if no one has defined it as an external link. As soon as that happens, then you will lose the export behavior defined for fuzzy links, and get the behavior defined by the export function. The syntax for these is the same (I guess a fuzzy link must be wrapped in [[ ]], but an external link may also be wrapped that way). I don't see a  way to differentiate these.

I guess this is a key question then; sounds like maybe internal links
are too loosely defined?

BTW, I was using the language from the org-mode manual to avoid confusion.

https://orgmode.org/manual/Internal-Links.html

>> 2. extend citations, per your idea here, which to me means the
>> org-cite code would need to be revised and expanded to handle both
>> cross-references and citations, but do so distinctly.
>
> I don't think so. You only have to extend them where you want to have the capability. org-cite can stay a citation only body of code, and if you want cross-references too you just use my processors, or write one that does what you want.

Except now, org-cite and all the extant processors assume a
citation-reference key corresponds to a bibliographic (mostly either
bibtex or csl json) entry, which identifies an external source.

Surely that has implications for compatibility if adding
cross-referencing, unless existing code is changed to account for
that?

With your suggestion, for example, I presume org-cite-insert would be
used to insert and edit cross-references, rather than org-insert-link
or org-insert-cross-reference? Aside from whether that makes sense
from a UX perspective, those references wouldn't be found unless a UI
explicitly accounted for them. And an activate processor that wasn't
updated would treat them as bad citations.

Maybe the changes would be small; IDK. Hopefully Nicolas can weigh in on this.

> You don't have to. Your processors should fail gracefully in any case, because you just cannot control what people will do with the styles.

The issue for me is less what you are proposing to do with styles
(which in my case I construct from org-cite-supported styles), than
with citation-reference. Per above, a citation-reference is just not a
cross-reference.

IDK; perhaps if this path makes sense in the end, it might be cleaner
to add a citation-cross-reference element, or maybe even better to
just have a parallel cross-reference object that borrows some of the
design of org-cite?

> The biggest issue with 1 is I don't think it is possible to differentiate internal typed links from external links because they use the same syntax. It would be very difficult to support and troubleshoot a scenario where the same syntax shows different behavior depending on whether an external link is defined or not. Finally, it is so close to what org-ref already does, I think it is too tricky to differentiate [[ref:label]] from [[ref/:label]], etc.
>
> Maybe these links would look different enough (as external links) that it would be really clear you were not using org-ref.
>
> ref/:@label
> ref/eq:@label
> ref/page:@label
> ref/name:@label
> ref/c:@label
> ref/C:@label
>
>  However, this doesn't support using prefix/suffix text, which is one of the main reasons the cite syntax came to be.

OK. That would be another requirement that tips the balance to
citation, or something very similar.

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

Hi Bruce, John,

Regarding the issues with overloading [cite:@] syntax, would it be too much to
introduce [ref:@] which would function identically, but deal with references?

All the best,
*Timothy*

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

That'd be my preferred solution, with (notwithstanding name overlap with
org-ref) an org-ref-insert entry point.

On Wed, Aug 11, 2021, 12:10 PM Timothy <tecosaur@gmail.com> wrote:

> Hi Bruce, John,
>
> Regarding the issues with overloading [cite:@] syntax, would it be too
> much to
> introduce [ref:@] which would function identically, but deal with
> references?
>
> All the best,
> *Timothy*
>

[-- Attachment #2: Type: text/html, Size: 680 bytes --]

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Wed, Aug 11, 2021 at 1:31 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

...

> > It's just there needs to be some way to distinguish among types of
> > targets, I think.
>
> At the risk of repeating myself, I don't think this is true.

Just to clarify ...yes, I agree with you; I wrote that before the
subsequent back-and-forth yesterday.

So no need to repeat: we are now in closer agreement than we were
before the discussion :-)

> As I have thought about this more, the line between citations and
> cross-references has blurred. In org-ref, they were handled the same
> way, with org-links (although each link had its own export function).
> For both cites and refs, the links are just pointers, and in both cases
> it is possible for them to point to things within the same document.
> Even in LaTeX, when we submit a manuscript, the citation references are
> embedded in a standalone tex file, so every link is to an internal
> location.

Let's step back though:

What is the ideal UX for org users?

Per my response to Timothy yesterday, and what I have said elsewhere
about how citations and cross-references are handled elsewhere in the
software universe, I think it's not to remove the distinction, but
rather to clarify it.

In effect, the choice is between:

1. org-cite-insert for both citations and cross-refernces
2. org-cite-insert for the former, org-ref-insert for the latter

Even in org-ref, you adopted 2.; there's a separate command and menu
entry for each, and a different UI.

Do you now think that was a bad decision in retrospect?

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 2328 bytes --]

> As I have thought about this more, the line between citations and
> > cross-references has blurred. In org-ref, they were handled the same
> > way, with org-links (although each link had its own export function).
> > For both cites and refs, the links are just pointers, and in both cases
> > it is possible for them to point to things within the same document.
> > Even in LaTeX, when we submit a manuscript, the citation references are
> > embedded in a standalone tex file, so every link is to an internal
> > location.
>
> Let's step back though:
>
> What is the ideal UX for org users?
>
> Per my response to Timothy yesterday, and what I have said elsewhere
> about how citations and cross-references are handled elsewhere in the
> software universe, I think it's not to remove the distinction, but
> rather to clarify it.
>
> In effect, the choice is between:
>
> 1. org-cite-insert for both citations and cross-refernces
> 2. org-cite-insert for the former, org-ref-insert for the latter
>
> Even in org-ref, you adopted 2.; there's a separate command and menu
> entry for each, and a different UI.
>
> Do you now think that was a bad decision in retrospect?
>

I would say the UI that I like, have used for many years, and is the
default of org-ref is:

1. You type C-c ] to insert a citation
2. You type C-u C-c ] to insert a cross-reference
3. you type C-u C-u C-c ] to insert a new label.

That is, there is a single entry point for inserting things, that is
differentiable with a prefix arg.

So, you remember one key binding to insert something, with the default
being the thing you insert the most of (for me citations), and you just
have to remember to prefix it for a reference or label. These are all done
with completion, and citations use a different set of candidates than a
cross-reference. Labels also use completion to help you insert unique ones.

There are lots of ways to implement that UI behind the scenes, and as you
noted the org-ref-insert-link function simply calls one of three
user-defined functions for a cite, ref or label. An alternative could be a
hydra or transient menu, with some keybinding that lets you choose to
insert a cite, ref (or a label perhaps).

so, I guess I am in agreement that it is option 2, and that there needs to
be a single entry point for each one.


>
> Bruce
>

[-- Attachment #2: Type: text/html, Size: 3130 bytes --]

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Thu, Aug 12, 2021 at 1:19 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:

> so, I guess I am in agreement that it is option 2, and that there needs to be a single entry point for each one.

+1

So I guess open questions:

1. a new syntax that mirrors cite (and accompanying new
cross-reference element?), per Timothy's suggestion, or reuse cite? I
still think a new syntax and element is cleaner, but depends in part
on ...
2. if the former, what new code would be required, and can some from
OC (and maybe org-ref??) be reused somehow? if the latter, what
changes to the OC code?

A related question on syntax is if one ever needs multiple
sub-references, as one does multi-cite citations? I was assuming no,
so a hypothetical new syntax could be simpler than citations, but IDK.

Of course these questions are only relevant if the org maintainers are
open to them also.

BTW, I thought this earlier post from Joost Kremers was interesting
and relevant:

https://lists.gnu.org/archive/html/emacs-orgmode/2021-07/msg00238.html

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Eric S Fraga]

Hello John & co.,

I need to chime in when it comes to the UI:

On Thursday, 12 Aug 2021 at 13:19, John Kitchin wrote:
> I would say the UI that I like, have used for many years, and is the
> default of org-ref is:
>
> 1. You type C-c ] to insert a citation
> 2. You type C-u C-c ] to insert a cross-reference
> 3. you type C-u C-u C-c ] to insert a new label.

UI is a very personal thing.  So long as there are 3 different functions
that implement these actions, I would be happy.  Despite using Emacs for
approximately 35 years, I use evil mode to avoid making my RSI worse
(RSI due originally to mouse use in the 80s, not Emacs ;-)).  In that
context, typing C-u is not ideal.  (C-c is worse, mind you.)  I have a
"," leader key to which I add org specific commands when in org mode.

Just my 2¢.

In terms of the bigger picture, citations and cross-references, to me,
are different things (one is external and the other usually internal,
respectively) and I see no benefit in conflating them.  I am also still
not convinced that org links are not sufficient but I don't mind if
there is a new syntax for cross-references, if this is deemed
beneficial.  However, these feelings (as opposed to the UI issue) are
not strong and will easily adapt to whatever is decided.

eric
-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.6-625-ge7454c
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

Can we go back to this question of whether internal links are adequate
for cross-references, and if not, what's missing?

On Fri, Aug 13, 2021 at 11:22 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
> Hello John & co.,
>
> I need to chime in when it comes to the UI:
>
> On Thursday, 12 Aug 2021 at 13:19, John Kitchin wrote:
> > I would say the UI that I like, have used for many years, and is the
> > default of org-ref is:
> >
> > 1. You type C-c ] to insert a citation
> > 2. You type C-u C-c ] to insert a cross-reference
> > 3. you type C-u C-u C-c ] to insert a new label.
>
> UI is a very personal thing.  So long as there are 3 different functions
> that implement these actions, I would be happy...

[snip]

> In terms of the bigger picture, citations and cross-references, to me,
> are different things (one is external and the other usually internal,
> respectively) and I see no benefit in conflating them.  I am also still
> not convinced that org links are not sufficient ...

In reviewing and playing with this functionality a bit, it's clear to
me that internal links are intended to be cross-references, even if
they fall short for certain classes of users.

Let's take a simple example of a figure, adapted from the manual:

#+begin_example
#+CAPTION: This is the caption.
#+NAME: fig:SED-HR4049
[[./img/a.jpg]]

Here's a cross-reference to Figure [[fig:SED-HR4049]].
#+end_example

This internal link seems to produce consistent and correct output
across latex, html, and odt.

The "fig" prefix is meaningful to ref in latex to enable it to type
that cross-reference. I'm unsure how typing in the odt context works,
but it does.

The only downside is one has to manually, per the example, insert the
prefix (here "Figure ") in text (though this is a small price to pay
in my view).

If one wants to use cleveref or autoref instead, as is now possible
with a recent commit, those packages handle that automatically. But of
course, this only works in latex, so you lose the consistency across
the backends.

Also, you then need to be able to specify different variants locally
(for example, cref vs Cref), which is not possible currently.

So I'm unclear: is that added functionality and complexity really
needed? Is there something else I'm missing?

A separate, possibly more important (?), matter is UI and supporting
functionality.

I think it would be nice to have better UI support for inserting these
references.

To John's point, could we add interactive functions to insert labels
and cross-references (say as customizable functions?), using the
existing internal link support, and iterate that support over time?

Bruce

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 4416 bytes --]

I think the existing link capability of org-mode is completely adequate for
cross-references. LaTeX exports are almost completely covered in org-ref. I
made a refproc (
https://github.com/jkitchin/org-ref/blob/org-ref-3/org-ref-refproc.el) that
provides similar export behavior for other backends in a pre-processing
hook. It supports the concepts in autoref and cleveref by replacing the
org-links with an appropriate org syntax that then exports natively. For
example, if cref:name references a table, it would get replaced by "Table
[[name]]" in the buffer and then exported to the desired backend.

All of these are references to two tables named b1 and b2

- ref :: ref:b1
- autoref :: autoref:b1
- nameref :: nameref:b1
- pageref :: pageref:b1
- cref :: cref:b1
- cref, two :: cref:b1,b2
- Cref :: Cref:b1
- Cref, two :: Cref:b1,b2

and they are converted to this plain org syntax (which basically mirrors
what would happen in a LaTeX export).

- ref :: [[b1]]
- autoref :: Tab. [[b1]]
- nameref :: [[b1][this is a table b1.]]
- pageref :: [[b1]]
- cref ::  Tab. [[b1]]
- cref, two :: Tabs. [[b1]] and [[b2]]
- Cref :: Tab. [[b1]]
- Cref, two :: Tabs. [[b1]] and [[b2]]

There are some options to control if the clever parts are abbreviated
and/or capitalized.

This should be part of org-ref on Melpa next week.

John

-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Sun, Oct 10, 2021 at 9:30 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> Can we go back to this question of whether internal links are adequate
> for cross-references, and if not, what's missing?
>
> On Fri, Aug 13, 2021 at 11:22 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
> >
> > Hello John & co.,
> >
> > I need to chime in when it comes to the UI:
> >
> > On Thursday, 12 Aug 2021 at 13:19, John Kitchin wrote:
> > > I would say the UI that I like, have used for many years, and is the
> > > default of org-ref is:
> > >
> > > 1. You type C-c ] to insert a citation
> > > 2. You type C-u C-c ] to insert a cross-reference
> > > 3. you type C-u C-u C-c ] to insert a new label.
> >
> > UI is a very personal thing.  So long as there are 3 different functions
> > that implement these actions, I would be happy...
>
> [snip]
>
> > In terms of the bigger picture, citations and cross-references, to me,
> > are different things (one is external and the other usually internal,
> > respectively) and I see no benefit in conflating them.  I am also still
> > not convinced that org links are not sufficient ...
>
> In reviewing and playing with this functionality a bit, it's clear to
> me that internal links are intended to be cross-references, even if
> they fall short for certain classes of users.
>
> Let's take a simple example of a figure, adapted from the manual:
>
> #+begin_example
> #+CAPTION: This is the caption.
> #+NAME: fig:SED-HR4049
> [[./img/a.jpg]]
>
> Here's a cross-reference to Figure [[fig:SED-HR4049]].
> #+end_example
>
> This internal link seems to produce consistent and correct output
> across latex, html, and odt.
>
> The "fig" prefix is meaningful to ref in latex to enable it to type
> that cross-reference. I'm unsure how typing in the odt context works,
> but it does.
>
> The only downside is one has to manually, per the example, insert the
> prefix (here "Figure ") in text (though this is a small price to pay
> in my view).
>
> If one wants to use cleveref or autoref instead, as is now possible
> with a recent commit, those packages handle that automatically. But of
> course, this only works in latex, so you lose the consistency across
> the backends.
>
> Also, you then need to be able to specify different variants locally
> (for example, cref vs Cref), which is not possible currently.
>
> So I'm unclear: is that added functionality and complexity really
> needed? Is there something else I'm missing?
>
> A separate, possibly more important (?), matter is UI and supporting
> functionality.
>
> I think it would be nice to have better UI support for inserting these
> references.
>
> To John's point, could we add interactive functions to insert labels
> and cross-references (say as customizable functions?), using the
> existing internal link support, and iterate that support over time?
>
> Bruce
>

[-- Attachment #2: Type: text/html, Size: 5682 bytes --]

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Tue, Oct 12, 2021 at 5:16 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>
> I think the existing link capability of org-mode is completely adequate for cross-references. LaTeX exports are almost completely covered in org-ref. I made a refproc (https://github.com/jkitchin/org-ref/blob/org-ref-3/org-ref-refproc.el) that provides similar export behavior for other backends in a pre-processing hook. It supports the concepts in autoref and cleveref by replacing the org-links with an appropriate org syntax that then exports natively. For example, if cref:name references a table, it would get replaced by "Table [[name]]" in the buffer and then exported to the desired backend.

Nice work John; I'm glad you were able to sort it out.

I haven't had a chance to play with it, but FYI I'm seeing a number of
compilation warnings (see below), which should be easy enough to fix.

Any reason this, or something like it, shouldn't be added to org?

Bruce

org-ref-refproc.el:15:1: Warning: defcustom for
    ‘org-ref-refproc-clever-prefixes’ fails to specify type
org-ref-refproc.el:15:1: Warning: defcustom for
    ‘org-ref-refproc-clever-prefixes’ fails to specify type

In org-ref-get-ref-links:
org-ref-refproc.el:47:54: Warning: reference to free variable
    ‘org-ref-ref-types’

In org-ref-refproc-referenceables:
org-ref-refproc.el:113:60: Warning: reference to free variable
    ‘org-ref-label-re’

In org-ref-refproc-cref-replacement:
org-ref-refproc.el:379:41: Warning: assignment to free variable ‘prefix-data’
org-ref-refproc.el:382:27: Warning: reference to free variable ‘prefix-data’

In end of data:
org-ref-refproc.el:244:7: Warning: the function ‘org-ref-ref-jump-to’ is not
    known to be defined.

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[John Kitchin]

[-- Attachment #1: Type: text/plain, Size: 3385 bytes --]

I am not sure it really fits the model of the exporters that are in org,
since it is basically just a pre-processing hook. I don't think there are
any exporters in org that rely on that, e.g. you can't just C-c C-e choose
a backend and export with a pre-processing hook. It doesn't make sense to
make a command that does that either I think, since you might want to use
other pre-processing hooks too, e.g. I have one called natmove (inspired by
the latex package) that moves superscript citations to the right hand side
of punctuation, and in ox-ipynb I use pre-processing functions for removing
solutions, and things like that. So some exports might use all three, and
others only two, some just one, and others none. One solution for that is
file-local variables I guess.

It also is (IMO) a part of org-ref since it uses the links that are defined
in org-ref. I see it as more an extension of org-mode than something that
needs to be in the core. For simple cross-references the built in internal
links are adequate I think.

Anyway, it should be tested for a while to see if it is the right way to do
this. Maybe there are better ideas for it.


John

-----------------------------------
Professor John Kitchin (he/him/his)
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu



On Tue, Oct 12, 2021 at 5:58 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> On Tue, Oct 12, 2021 at 5:16 PM John Kitchin <jkitchin@andrew.cmu.edu>
> wrote:
> >
> > I think the existing link capability of org-mode is completely adequate
> for cross-references. LaTeX exports are almost completely covered in
> org-ref. I made a refproc (
> https://github.com/jkitchin/org-ref/blob/org-ref-3/org-ref-refproc.el)
> that provides similar export behavior for other backends in a
> pre-processing hook. It supports the concepts in autoref and cleveref by
> replacing the org-links with an appropriate org syntax that then exports
> natively. For example, if cref:name references a table, it would get
> replaced by "Table [[name]]" in the buffer and then exported to the desired
> backend.
>
> Nice work John; I'm glad you were able to sort it out.
>
> I haven't had a chance to play with it, but FYI I'm seeing a number of
> compilation warnings (see below), which should be easy enough to fix.
>
> Any reason this, or something like it, shouldn't be added to org?
>
> Bruce
>
> org-ref-refproc.el:15:1: Warning: defcustom for
>     ‘org-ref-refproc-clever-prefixes’ fails to specify type
> org-ref-refproc.el:15:1: Warning: defcustom for
>     ‘org-ref-refproc-clever-prefixes’ fails to specify type
>
> In org-ref-get-ref-links:
> org-ref-refproc.el:47:54: Warning: reference to free variable
>     ‘org-ref-ref-types’
>
> In org-ref-refproc-referenceables:
> org-ref-refproc.el:113:60: Warning: reference to free variable
>     ‘org-ref-label-re’
>
> In org-ref-refproc-cref-replacement:
> org-ref-refproc.el:379:41: Warning: assignment to free variable
> ‘prefix-data’
> org-ref-refproc.el:382:27: Warning: reference to free variable
> ‘prefix-data’
>
> In end of data:
> org-ref-refproc.el:244:7: Warning: the function ‘org-ref-ref-jump-to’ is
> not
>     known to be defined.
>

[-- Attachment #2: Type: text/html, Size: 4303 bytes --]

Re: Expanding how the new cite syntax is used to include cross-references - thoughts?

[Bruce D'Arcus]

On Tue, Oct 12, 2021 at 7:28 PM John Kitchin <jkitchin@andrew.cmu.edu> wrote:
>
> I am not sure it really fits the model of the exporters that are in org, since it is basically just a pre-processing hook.

[snip]

> It also is (IMO) a part of org-ref since it uses the links that are defined in org-ref. I see it as more an extension of org-mode than something that needs to be in the core. For simple cross-references the built in internal links are adequate I think.
>
> Anyway, it should be tested for a while to see if it is the right way to do this. Maybe there are better ideas for it.

Probably does take more testing and thought, but I guess I was
thinking of something like an org-crossref-insert interactive
function, that could make use of functions like
org-ref-refproc-referenceables, to provide better out-of-box support
for at least inserting cross-references.

Perhaps like org-cite-insert, that could be configurable, so one could
plug in a different such functions.

I was toying with an idea like this earlier:

https://gist.github.com/bdarcus/1cebcff17fca762a363f89138214848d

Bruce