[wip-cite-new] Adjust punctuation around citations

[wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

Following discussion with Bruce D'Arcus and Denis Maier, I pushed, in
the "wip-cite-new" branch, the first version of a tool for adjusting the
location of the citation and surrounding punctuation according to fixed
rules. The name is `org-cite-adjust-punctuation' and its docstring is:

  Adjust punctuation around CITATION object.

  When CITATION follows a quotation, or when there is punctuation next to it,
  the function tries to normalize the location of punctuation and citation
  according to some RULE.

  RULE is a triplet of symbols (PUNCT POSITION RELATIVE):

    PUNCT is the desired location of the punctuation with regards to the
    quotation, if any.  It may be `inside', `outside', or`strict', the latter
    meaning the punctuation should not be moved.

    POSITION is the desired location of the citation with regards to the
    quotation, if any.  It may be `inside' or `outside'.

    RELATIVE is the relative position of the citation with regards to the closest
    punctuation.  It may be `after' or `before'.

  For example,

    (inside outside after) corresponds to American typography;
    (strict outside after) corresponds to German typography;
    (strict inside before) corresponds to French typography.

  INFO is the export state, as a property list.

  Optional argument PUNCT is a list of punctuation marks to be considered.
  When nil, it includes the following: \".\" \",\" \";\" \":\" \"!\" and \"?\".

  Parse tree is modified by side-effect.

  Note: if you are calling both `org-cite-adjust-punctuation' and
  `org-cite-wrap-citation' on the same object, call `org-cite-adjust-punctuation'
  first.

Citation processors focused on export may choose to use it, particularly
when using note style. 

As an example, the following code implements a processor named `test'
that uses note style, and adjust punctuation according to the language
specified for the document.

--8<---------------cut here---------------start------------->8---
(defun org-test--language-to-rule (info)
  (pcase (plist-get info :language)
    ("en-us" '(inside outside after))
    ((or "en" "de" "en-gb") '(strict outside after))
    ("fr" '(strict inside before))
    (_ nil)))

(defun org-test-export-citation (citation _style _backend info)
  (pcase (org-test--language-to-rule info)
    (`nil nil)
    (rule (org-cite-adjust-punctuation citation rule info)))
  (unless (org-cite-inside-footnote-p citation)
    (org-cite-wrap-citation citation info))
  "...")

(org-cite-register-processor 'test
  :export-citation #'org-test-export-citation)
--8<---------------cut here---------------end--------------->8---

Once evaluated, you can test it, for example, by exporting the following
document:

--8<---------------cut here---------------start------------->8---
#+language: de
#+cite_export: test

"This is a complete sentence." [cite:@key]

"This is an incomplete sentence" [cite:@key].

This is a complete sentence. [cite:@key]

This is an incomplete sentence [cite:@key].
--8<---------------cut here---------------end--------------->8---

and changing language value.

WDYT?

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Thanks Nicolas! That looks quite good already.

Your test cases give good results for German.
I've also added another language property for when you want to switch to 
an in-text citation style:

(defun org-test--language-to-rule (info)
   (pcase (plist-get info :language)
     ("en-us" '(inside outside after))
     ((or "en" "de" "en-gb") '(strict outside after))
     ("de-author-year" '(outside outside before))
     ("fr" '(strict inside before))
     (_ nil)))

Exporting your example with #+language: de-author-year gives me:

=========================
"This is a complete sentence"[1].

"This is an incomplete sentence"[2].

"This is an incomplete sentence"[3].

This is a complete sentence[4].

This is an incomplete sentence[5].
=========================

The only quirk here is that you'll obviously want spaces before the 
citaitons, but I guess this is because citation end up in footnotes. 
With an in-text style spaces won't be collapsed here, right?

Again, thanks for all your work on this one.

Denis

Am 13.05.2021 um 23:33 schrieb Nicolas Goaziou:
> Hello,
> 
> Following discussion with Bruce D'Arcus and Denis Maier, I pushed, in
> the "wip-cite-new" branch, the first version of a tool for adjusting the
> location of the citation and surrounding punctuation according to fixed
> rules. The name is `org-cite-adjust-punctuation' and its docstring is:
> 
>    Adjust punctuation around CITATION object.
> 
>    When CITATION follows a quotation, or when there is punctuation next to it,
>    the function tries to normalize the location of punctuation and citation
>    according to some RULE.
> 
>    RULE is a triplet of symbols (PUNCT POSITION RELATIVE):
> 
>      PUNCT is the desired location of the punctuation with regards to the
>      quotation, if any.  It may be `inside', `outside', or`strict', the latter
>      meaning the punctuation should not be moved.
> 
>      POSITION is the desired location of the citation with regards to the
>      quotation, if any.  It may be `inside' or `outside'.
> 
>      RELATIVE is the relative position of the citation with regards to the closest
>      punctuation.  It may be `after' or `before'.
> 
>    For example,
> 
>      (inside outside after) corresponds to American typography;
>      (strict outside after) corresponds to German typography;
>      (strict inside before) corresponds to French typography.
> 
>    INFO is the export state, as a property list.
> 
>    Optional argument PUNCT is a list of punctuation marks to be considered.
>    When nil, it includes the following: \".\" \",\" \";\" \":\" \"!\" and \"?\".
> 
>    Parse tree is modified by side-effect.
> 
>    Note: if you are calling both `org-cite-adjust-punctuation' and
>    `org-cite-wrap-citation' on the same object, call `org-cite-adjust-punctuation'
>    first.
> 
> Citation processors focused on export may choose to use it, particularly
> when using note style.
> 
> As an example, the following code implements a processor named `test'
> that uses note style, and adjust punctuation according to the language
> specified for the document.
> 
> --8<---------------cut here---------------start------------->8---
> (defun org-test--language-to-rule (info)
>    (pcase (plist-get info :language)
>      ("en-us" '(inside outside after))
>      ((or "en" "de" "en-gb") '(strict outside after))
>      ("fr" '(strict inside before))
>      (_ nil)))
> 
> (defun org-test-export-citation (citation _style _backend info)
>    (pcase (org-test--language-to-rule info)
>      (`nil nil)
>      (rule (org-cite-adjust-punctuation citation rule info)))
>    (unless (org-cite-inside-footnote-p citation)
>      (org-cite-wrap-citation citation info))
>    "...")
> 
> (org-cite-register-processor 'test
>    :export-citation #'org-test-export-citation)
> --8<---------------cut here---------------end--------------->8---
> 
> Once evaluated, you can test it, for example, by exporting the following
> document:
> 
> --8<---------------cut here---------------start------------->8---
> #+language: de
> #+cite_export: test
> 
> "This is a complete sentence." [cite:@key]
> 
> "This is an incomplete sentence" [cite:@key].
> 
> This is a complete sentence. [cite:@key]
> 
> This is an incomplete sentence [cite:@key].
> --8<---------------cut here---------------end--------------->8---
> 
> and changing language value.
> 
> WDYT?
> 
> Regards,
> 

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Thu, May 13, 2021 at 5:33 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

....

> WDYT?

Looks really good; thanks for this!

But I get this error when I run the export to test.

org-export-as: Wrong number of arguments: #<subr
org-macro-initialize-templates>, 1

Not sure what I'm doing wrong; I:

1. pulled the branch
2. ran make
3. evaled your "test" code
4. ran the export

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 14.05.2021 um 01:21 schrieb Bruce D'Arcus:
> On Thu, May 13, 2021 at 5:33 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
> 
> ....
> 
>> WDYT?
> 
> Looks really good; thanks for this!
> 
> But I get this error when I run the export to test.
> 
> org-export-as: Wrong number of arguments: #<subr
> org-macro-initialize-templates>, 1
> 
> Not sure what I'm doing wrong; I:
> 
> 1. pulled the branch
> 2. ran make
> 3. evaled your "test" code
> 4. ran the export

Exactly, what I was doing. It works without problems here.

Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 13.05.2021 um 23:33 schrieb Nicolas Goaziou:
> Hello,
> 
> [...]
> 
>    RULE is a triplet of symbols (PUNCT POSITION RELATIVE):
> 
>      PUNCT is the desired location of the punctuation with regards to the
>      quotation, if any.  It may be `inside', `outside', or`strict', the latter
>      meaning the punctuation should not be moved.
> 
>      POSITION is the desired location of the citation with regards to the
>      quotation, if any.  It may be `inside' or `outside'.
> 
>      RELATIVE is the relative position of the citation with regards to the closest
>      punctuation.  It may be `after' or `before'.
> 
>    For example,
> 
>      (inside outside after) corresponds to American typography;
>      (strict outside after) corresponds to German typography;
>      (strict inside before) corresponds to French typography.
These things are not only language dependent, but depend also on the 
chosen citation style. E.g. with in-text styles you'll have (outside 
outside before) in German, and American typography, in French likely as 
well.
So, perhaps another setting:
#+type-of-citation-style: in-text
#+type-of-citation-style: note

And ideally this RULE should be configurable directly as well:
#+punct-moving-rule: strict inside before

> 
>    INFO is the export state, as a property list.
> 
>    Optional argument PUNCT is a list of punctuation marks to be considered.
>    When nil, it includes the following: \".\" \",\" \";\" \":\" \"!\" and \"?\".

Here as well, it should be possible to configure these on a user level. 
Something like:
#+moved-punctuation: ,.?!

WDYT?

Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> But I get this error when I run the export to test.
>
> org-export-as: Wrong number of arguments: #<subr
> org-macro-initialize-templates>, 1

This looks like an unrelated error.
>
> Not sure what I'm doing wrong; I:
>
> 1. pulled the branch
> 2. ran make

You probably need to reload Org (M-x org-reload) at this point.

> 3. evaled your "test" code
> 4. ran the export

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Fri, May 14, 2021 at 4:44 AM Denis Maier
<denis.maier.lists@mailbox.org> wrote:

> These things are not only language dependent, but depend also on the
> chosen citation style. E.g. with in-text styles you'll have (outside
> outside before) in German, and American typography, in French likely as
> well.
> So, perhaps another setting:
> #+type-of-citation-style: in-text
> #+type-of-citation-style: note

Shouldn't in general this be up to the export processor?

I wouldn't expect to have to set this if I'm using citeproc-org or biblatex.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Fri, May 14, 2021 at 6:26 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
>
> > But I get this error when I run the export to test.
> >
> > org-export-as: Wrong number of arguments: #<subr
> > org-macro-initialize-templates>, 1
>
> This looks like an unrelated error.

Just confirming this was in fact the case, and I now have it working.

It seems to work as expected from my perspective.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Hi,

>      (strict inside before) corresponds to French typography.
>

Regarding French typography.
With this example:
---------------------------------------------------
#+language: fr
#+cite_export: test

"This is a complete sentence."[cite:@key]

"This is an incomplete sentence" [cite:@key].

"This is an incomplete sentence". [cite:@key]

This is a complete sentence. [cite:@key]

This is an incomplete sentence [cite:@key].
----------------------------------------------------

I'm getting:

----------------------------------------------------
"This is a complete sentence[1]."

"This is an incomplete sentence[2]" .

"This is an incomplete sentence"[3].

This is a complete sentence[4].

This is an incomplete sentence[5].
----------------------------------------------------

1, 3, 4, 5 look ok.
2 is a bit odd.
- there's the spurios space before the period.
- I think in that case the footnote mark should be just before the 
period, i.e. after the quotation mark. Basically like example 3.

I think the textbook rule regarding French typography is:
Place the footnote mark just before the final punctuation.
Punctuation is placed inside quotation marks if we have a full 
quotation, otherwise outside. (OTOH, that seems to be a textbook rule. 
As I've said, I haven't seen that in real books.)

I don't know if that is feasible with the current parameters.

Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

Denis Maier writes:

> Regarding French typography.
> With this example:
> ---------------------------------------------------
>
> #+language: fr
> #+cite_export: test
>
> "This is a complete sentence."[cite:@key]
>
> "This is an incomplete sentence" [cite:@key].

[...]

> I'm getting:
>
> ----------------------------------------------------
> "This is a complete sentence[1]."
>
> "This is an incomplete sentence[2]" .

[...]

> 2 is a bit odd.
> - there's the spurios space before the period.
> - I think in that case the footnote mark should be just before the 
> period, i.e. after the quotation mark. Basically like example 3.

[...]

> I don't know if that is feasible with the current parameters.

It is not. I added a `same' parameter for citation to achieve it. I also
fixed the space error.

You can try again with the following processor:

--8<---------------cut here---------------start------------->8---
(defun org-test--language-to-rule (info)
  (pcase (plist-get info :language)
    ("en-us" '(inside outside after))
    ((or "en" "de" "en-gb") '(static outside after))
    ("fr" '(static same before))
    (_ nil)))

(defun org-test-export-citation (citation style _backend info)
  (pcase style
    ("author-year"
     (org-cite-adjust-punctuation citation '(outside outside before) info nil t)
     "(John Doe, 1999)")
    (_
     (pcase (org-test--language-to-rule info)
       (`nil nil)
       (rule (org-cite-adjust-punctuation citation rule info)))
     (unless (org-cite-inside-footnote-p citation)
       (org-cite-wrap-citation citation info))
     "...")))

(org-cite-register-processor 'test
  :export-citation #'org-test-export-citation)
--8<---------------cut here---------------end--------------->8---

I'll comment further in another message.

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

Denis Maier <denis.maier.lists@mailbox.org> writes:

> Am 13.05.2021 um 23:33 schrieb Nicolas Goaziou:
>> Hello,
>> [...]
>>    RULE is a triplet of symbols (PUNCT POSITION RELATIVE):
>>      PUNCT is the desired location of the punctuation with regards
>> to the
>>      quotation, if any.  It may be `inside', `outside', or`strict', the latter
>>      meaning the punctuation should not be moved.
>>      POSITION is the desired location of the citation with regards
>> to the
>>      quotation, if any.  It may be `inside' or `outside'.
>>      RELATIVE is the relative position of the citation with regards
>> to the closest
>>      punctuation.  It may be `after' or `before'.
>>    For example,
>>      (inside outside after) corresponds to American typography;
>>      (strict outside after) corresponds to German typography;
>>      (strict inside before) corresponds to French typography.
> These things are not only language dependent, but depend also on the
> chosen citation style. E.g. with in-text styles you'll have (outside 
> outside before) in German, and American typography, in French likely
> as well.

Since the location of the punctuation is meaningful in French (and
German), I have doubts about (outside ...) being appropriate. More on
this below.

> So, perhaps another setting:
> #+type-of-citation-style: in-text
> #+type-of-citation-style: note
>
> And ideally this RULE should be configurable directly as well:
> #+punct-moving-rule: strict inside before
>
>>    INFO is the export state, as a property list.
>>    Optional argument PUNCT is a list of punctuation marks to be
>> considered.
>>    When nil, it includes the following: \".\" \",\" \";\" \":\" \"!\" and \"?\".
>
> Here as well, it should be possible to configure these on a user
> level. Something like:
> #+moved-punctuation: ,.?!
>
> WDYT?

At the moment, the `org-cite-adjust-punctuation' function is designed
with author-year to note conversion in mind, not the other way. I don't
have enough examples of the opposite transformation to even be sure the
current interface would be appropriate. I even believe this is not often
possible, as it would imply rewording, i.e., add information that is not
present in the original document.

This doesn't seem to be a limitation, though. This feature is useful for
users having to switch between author-year and note styles. The rule of
thumb is to always write author-year in source.

In any case, this explains why the docstring has a bias. I updated it to
insist on the fact that these are rules for author-year to note
conversion.

Also, this function is not meant to be accessible to the end user. It is
called from the processor, which knows the type (or style) of the
citation. It may also choose not to use this function. So, I agree with
Bruce D'Arcus: selecting an appropriate rule and punctuation ought to
happen at that level.

More importantly, I don't think fine-grain configuration is required.
For specific needs, this "smart" feature should be disabled, and
elements (punctuation, citation) positioned manually. But, again, if
configuration is needed, the processor should provide it, e.g., through
variables, not Org Cite. For example, a defcustom could offer to 1) not
use this feature 2) rely on "language" keyword 3) apply a user-defined
rule and punctuation set.

What would be nice, however, would be an association between language
and default rules and punctuation characters.

WDYT?

Meanwhile, I modified `org-cite-adjust-punctuation' function a bit. Here
is its new docstring.

--8<---------------cut here---------------start------------->8---
Adjust punctuation around CITATION object.

When CITATION follows a quotation, or when there is punctuation next to it,
the function tries to normalize the location of punctuation and citation
according to some RULE.

RULE is a triplet of symbols (PUNCTUATION CITE ORDER):

  PUNCTUATION is the desired location of the punctuation with regards to the
  quotation, if any.  It may be `inside', `outside', or`static'.  When set to
  `static', the punctuation is not moved.

  CITE is the desired location of the citation with regards to the quotation
  mark, if any.  It may be `inside', `outside', or `same'.  When set to `same',
  the citation is moved on the same side as the punctuation, but does not move
  if there is punctuation on both sides or on none.

  ORDER is the relative position of the citation with regards to the closest
  punctuation.  It may be `after' or `before'.

For example, when changing from author-date to note style,

  (inside outside after) corresponds to American typography;
  (static outside after) corresponds to German typography;
  (static same before) corresponds to French typography.

INFO is the export state, as a property list.

Optional argument PUNCT is a list of punctuation marks to be considered.
When nil, it includes the following: \".\" \",\" \";\" \":\" \"!\" and \"?\".

When optional argument ADD-SPACE is non-nil, add a space before citation.  This
is useful, for example, when switching from note to author-year style.

Parse tree is modified by side-effect.

Note: if you are calling both `org-cite-adjust-punctuation' and
`org-cite-wrap-citation' on the same object, call `org-cite-adjust-punctuation'
first.
--8<---------------cut here---------------end--------------->8---

Note that previous `strict' became `static', and I introduced a `same'
value for the second rule.  I also added a new ADD-SPACE optional
argument as an attempt to ease note to author-year style conversion.

As written already in another message, you can test the following
updated processor:

--8<---------------cut here---------------start------------->8---
(defun org-test--language-to-rule (info)
  (pcase (plist-get info :language)
    ("en-us" '(inside outside after))
    ((or "en" "de" "en-gb") '(static outside after))
    ("fr" '(static same before))
    (_ nil)))

(defun org-test-export-citation (citation style _backend info)
  (pcase style
    ("author-year"
     (org-cite-adjust-punctuation citation '(outside outside before) info nil t)
     "(John Doe, 1999)")
    (_
     (pcase (org-test--language-to-rule info)
       (`nil nil)
       (rule (org-cite-adjust-punctuation citation rule info)))
     (unless (org-cite-inside-footnote-p citation)
       (org-cite-wrap-citation citation info))
     "...")))

(org-cite-register-processor 'test
  :export-citation #'org-test-export-citation)
--8<---------------cut here---------------end--------------->8---


Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sat, May 15, 2021 at 7:56 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

...

> At the moment, the `org-cite-adjust-punctuation' function is designed
> with author-year to note conversion in mind, not the other way. I don't
> have enough examples of the opposite transformation to even be sure the
> current interface would be appropriate. I even believe this is not often
> possible, as it would imply rewording, i.e., add information that is not
> present in the original document.
>
> This doesn't seem to be a limitation, though. This feature is useful for
> users having to switch between author-year and note styles. The rule of
> thumb is to always write author-year in source.
>
> In any case, this explains why the docstring has a bias. I updated it to
> insist on the fact that these are rules for author-year to note
> conversion.

FWIW, I agree with this decision.

High level, the basic logic says to a user if you need to cite
something, insert a citation, and let your citation processor take
care of the rest.

Whether that final output is a number, a footnote mark, or an
author-date representation, it should "just work."

Going the other way doesn't really make sense from that perspective. A
user can still place a citation in a footnote manually.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sat, May 15, 2021 at 8:03 AM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
> On Sat, May 15, 2021 at 7:56 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

...

> > In any case, this explains why the docstring has a bias. I updated it to
> > insist on the fact that these are rules for author-year to note
> > conversion.
>
> FWIW, I agree with this decision.
>
> High level, the basic logic says to a user if you need to cite
> something, insert a citation, and let your citation processor take
> care of the rest.
>
> Whether that final output is a number, a footnote mark, or an
> author-date representation, it should "just work."

To make explicit what is implicit above:

What I'm saying is it's not actually an author-date bias, even if the
examples we've discussed have focused on that.

Rather, this functionality allows an export processor to handle
automatic footnoting of in-text, as opposed to in-note, citations,
regardless of what the initial output expectation for those citations
are.

I don't know how to elegantly express this for the docstring, but just
wanted to clarify.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 15.05.2021 um 13:56 schrieb Nicolas Goaziou:
> Hello,
> 
[...]
> 
> At the moment, the `org-cite-adjust-punctuation' function is designed
> with author-year to note conversion in mind, not the other way. I don't
> have enough examples of the opposite transformation to even be sure the
> current interface would be appropriate. I even believe this is not often
> possible, as it would imply rewording, i.e., add information that is not
> present in the original document.
> 
> This doesn't seem to be a limitation, though. This feature is useful for
> users having to switch between author-year and note styles. The rule of
> thumb is to always write author-year in source.

Well, I have to admit I was a bit confused when I first read this since 
the examples we're currently working with wouldn't be correct examples 
for author-date citations in German and English:

--8<---------------cut here---------------start------------->8---
#+language: de
#+cite_export: test

1. "This is a complete sentence." [cite:@key]

2. "This is an incomplete sentence" [cite:@key].

3. This is a complete sentence. [cite:@key]

4. This is an incomplete sentence [cite:@key].
--8<---------------cut here---------------end--------------->8---

In German and English author-date styles example 1. and 2. will both be 
rendered as:
"This is a complete sentence" (author year).
"This is an incomplete sentence" (author year).
So, in both cases the punctuation comes after the citation.

After looking up a few guidelines from French speaking universities in 
Canada, it looks like in French you'll actually render these citations as:
"This is a complete sentence." (author year)
"This is an incomplete sentence" (author year).
(Don't know if that is consistent across la francophonie.)
I.e., with a complete sentence you'll have the final punctuation inside 
the quotation marks with the citation following the citation. So, as you 
say the location of the punctuation is semantically meaningful in French 
even with author-date styles, but that isn't the case in German and 
English. In German and British English, the location of the punctuation 
is only semantically meaningful with note citation styles.

Now, interestingly, the way you'll place the punctuation marks in German 
and British English seems to conform to French author-date punctuation 
placement.

This means that in German and British English a conversion from
"This is a complete sentence." [cite:@key]
to
"This is a complete sentence" (citation).
is actually not adding, but removing information.

OTOH, if you write targeting German/English author-date styles, it's not 
possible to switch correctly to a note style in German and British 
English. (You'll have to indicate the location of the punctuation in the 
original material, which means it has to be moved when targeting an 
author-year style.)

So, I still think (outside outside before) should work in general for 
English and German. If I understand correctly you've already added it as 
  (pcase style ("author-year" ...
Correct?

There's only one further complication: if the quotation is a set off 
block quote, the citation comes after the punctuation mark:
This is a complete sentence. (author year)
Can surrounding context be considered in that transformation?

Denis

> 
> In any case, this explains why the docstring has a bias. I updated it to
> insist on the fact that these are rules for author-year to note
> conversion.
> 
> Also, this function is not meant to be accessible to the end user. It is
> called from the processor, which knows the type (or style) of the
> citation. It may also choose not to use this function. So, I agree with
> Bruce D'Arcus: selecting an appropriate rule and punctuation ought to
> happen at that level.
> 
> More importantly, I don't think fine-grain configuration is required.
> For specific needs, this "smart" feature should be disabled, and
> elements (punctuation, citation) positioned manually. But, again, if
> configuration is needed, the processor should provide it, e.g., through
> variables, not Org Cite. For example, a defcustom could offer to 1) not
> use this feature 2) rely on "language" keyword 3) apply a user-defined
> rule and punctuation set.
> 
> What would be nice, however, would be an association between language
> and default rules and punctuation characters.
> 
> WDYT?
> 
> Meanwhile, I modified `org-cite-adjust-punctuation' function a bit. Here
> is its new docstring.
> 
> --8<---------------cut here---------------start------------->8---
> Adjust punctuation around CITATION object.
> 
> When CITATION follows a quotation, or when there is punctuation next to it,
> the function tries to normalize the location of punctuation and citation
> according to some RULE.
> 
> RULE is a triplet of symbols (PUNCTUATION CITE ORDER):
> 
>    PUNCTUATION is the desired location of the punctuation with regards to the
>    quotation, if any.  It may be `inside', `outside', or`static'.  When set to
>    `static', the punctuation is not moved.
> 
>    CITE is the desired location of the citation with regards to the quotation
>    mark, if any.  It may be `inside', `outside', or `same'.  When set to `same',
>    the citation is moved on the same side as the punctuation, but does not move
>    if there is punctuation on both sides or on none.
> 
>    ORDER is the relative position of the citation with regards to the closest
>    punctuation.  It may be `after' or `before'.
> 
> For example, when changing from author-date to note style,
> 
>    (inside outside after) corresponds to American typography;
>    (static outside after) corresponds to German typography;
>    (static same before) corresponds to French typography.
> 
> INFO is the export state, as a property list.
> 
> Optional argument PUNCT is a list of punctuation marks to be considered.
> When nil, it includes the following: \".\" \",\" \";\" \":\" \"!\" and \"?\".
> 
> When optional argument ADD-SPACE is non-nil, add a space before citation.  This
> is useful, for example, when switching from note to author-year style.
> 
> Parse tree is modified by side-effect.
> 
> Note: if you are calling both `org-cite-adjust-punctuation' and
> `org-cite-wrap-citation' on the same object, call `org-cite-adjust-punctuation'
> first.
> --8<---------------cut here---------------end--------------->8---
> 
> Note that previous `strict' became `static', and I introduced a `same'
> value for the second rule.  I also added a new ADD-SPACE optional
> argument as an attempt to ease note to author-year style conversion.
> 
> As written already in another message, you can test the following
> updated processor:
> 
> --8<---------------cut here---------------start------------->8---
> (defun org-test--language-to-rule (info)
>    (pcase (plist-get info :language)
>      ("en-us" '(inside outside after))
>      ((or "en" "de" "en-gb") '(static outside after))
>      ("fr" '(static same before))
>      (_ nil)))
> 
> (defun org-test-export-citation (citation style _backend info)
>    (pcase style
>      ("author-year"
>       (org-cite-adjust-punctuation citation '(outside outside before) info nil t)
>       "(John Doe, 1999)")
>      (_
>       (pcase (org-test--language-to-rule info)
>         (`nil nil)
>         (rule (org-cite-adjust-punctuation citation rule info)))
>       (unless (org-cite-inside-footnote-p citation)
>         (org-cite-wrap-citation citation info))
>       "...")))
> 
> (org-cite-register-processor 'test
>    :export-citation #'org-test-export-citation)
> --8<---------------cut here---------------end--------------->8---
> 
> 
> Regards,
> 

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sun, May 16, 2021 at 5:29 PM Denis Maier
<denis.maier.lists@mailbox.org> wrote:

...

> There's only one further complication: if the quotation is a set off
> block quote, the citation comes after the punctuation mark:
> This is a complete sentence. (author year)

I've not seen that with the cases I'm familiar with (US and UK English).

The citation would just be inside the final period of the blockquote,
so depending on style class:

- ... ending of blockquote (Doe, 2019).
- ... ending of blockquote [doe19].
- ... ending of blockquote.[1]

So there would be need for any special handling.

I mean blockquotes don't normally include quotation marks around them.

Can you clarify the rule/situation that would use that approach?

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier via General discussions about Org-mode.]

Am 16.05.2021 um 23:38 schrieb Bruce D'Arcus:
> On Sun, May 16, 2021 at 5:29 PM Denis Maier
> <denis.maier.lists@mailbox.org> wrote:
> 
> ...
> 
>> There's only one further complication: if the quotation is a set off
>> block quote, the citation comes after the punctuation mark:
>> This is a complete sentence. (author year)
> 
> I've not seen that with the cases I'm familiar with (US and UK English).
> 
> The citation would just be inside the final period of the blockquote,
> so depending on style class:
> 
> - ... ending of blockquote (Doe, 2019).
> - ... ending of blockquote [doe19].
> - ... ending of blockquote.[1]
> 
> So there would be need for any special handling.
> 
> I mean blockquotes don't normally include quotation marks around them.
> 
> Can you clarify the rule/situation that would use that approach?
> 

Chicago Manual of Style 15.26:
"When the source of a block quotation is given in parentheses at the end 
of the quotation, the opening parenthesis appears after the final 
punctuation mark of the quoted material. No period either precedes or 
follows the closing parenthesis."

And the example:
If you happen to be fishing, and you get a strike, and whatever it is 
starts off with the preliminaries of a vigorous fight; and by and by, 
looking down over the side through the glassy water, you see a rosy 
golden gleam, the mere specter of a fish, shining below in the clear 
depths; and when you look again a sort of glory of golden light flashes 
and dazzles as it circles nearer beneath and around and under the boat; 
. . . and you land a slim and graceful and impossibly beautiful 
three-foot goldfish, whose fierce and vivid yellow is touched around the 
edges with a violent red—when all these things happen to you, fortunate 
but bewildered fisherman, then you may know you have been fishing in the 
Galapagos Islands and have taken a Golden Grouper. (Pinchot 1930, 123)

Seems to be the same in MLA and APA by the way:
https://writingcenter.uagc.edu/block-quotations

Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sun, May 16, 2021 at 6:03 PM Denis Maier
<denis.maier.lists@mailbox.org> wrote:
>
> Am 16.05.2021 um 23:38 schrieb Bruce D'Arcus:

....

> > Can you clarify the rule/situation that would use that approach?
> >
>
> Chicago Manual of Style 15.26:
> "When the source of a block quotation is given in parentheses at the end
> of the quotation, the opening parenthesis appears after the final
> punctuation mark of the quoted material. No period either precedes or
> follows the closing parenthesis."

Maybe me and a lot of scholars aren't very good at following these rules :-)

Or maybe the stuff I read and write doesn't follow Chicago, etc for
whatever reason.

> Seems to be the same in MLA and APA by the way:
> https://writingcenter.uagc.edu/block-quotations

Very helpful!

So what are you thinking; this punctuation-manipulation should not
apply to blockquotes at all?

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 17.05.2021 um 00:24 schrieb Bruce D'Arcus:
> On Sun, May 16, 2021 at 6:03 PM Denis Maier
> <denis.maier.lists@mailbox.org> wrote:
>>
>> Am 16.05.2021 um 23:38 schrieb Bruce D'Arcus:
> 
> ....
> 
>>> Can you clarify the rule/situation that would use that approach?
>>>
>>
>> Chicago Manual of Style 15.26:
>> "When the source of a block quotation is given in parentheses at the end
>> of the quotation, the opening parenthesis appears after the final
>> punctuation mark of the quoted material. No period either precedes or
>> follows the closing parenthesis."
> 
> Maybe me and a lot of scholars aren't very good at following these rules :-)
> 
> Or maybe the stuff I read and write doesn't follow Chicago, etc for
> whatever reason.
> 
>> Seems to be the same in MLA and APA by the way:
>> https://writingcenter.uagc.edu/block-quotations
> 
> Very helpful!
> 
> So what are you thinking; this punctuation-manipulation should not
> apply to blockquotes at all?

I think there are a couple of options and combinations thereof:

1. Do nothing special, just treat block quotations like regular paragraphs.

2. Don't apply this punctuation-manipulation to blockquotes at all, i.e. 
let authors handle this manually.

3. Adjust RULE in the manipulation function: Let it accept two patterns:
'((outside outside before) (outside outside after))
The first would apply to inline quotations, the second to blockquotes.

4. IIUC, it's possible to set certain properties at block elements. So, 
maybe you could actually add citation information there, e.g.:

#+BEGIN_QUOTE :cite @doe p.45
This is a blockquote in its own paragraph.
#+END_QUOTE

Unsure about the syntax, but you get the idea. This should allow for 
complex manipulations: the citation can be added before or after the 
punctuation mark; if a journal requires quotation marks even around 
blockquotes => just add them during export; if the period is not part of 
the original quotation you could just do this:

#+BEGIN_QUOTE :cite @doe p.45 :punct .
This is a blockquote in its own paragraph
#+END_QUOTE

Then the period will be added during export, but if a journal wants an 
ellipsis in that case you can also just add that during export.
	This is a blockquote in its own paragraph [...].[1]

Now, this is of course rather complex and might add a lot of overhead. I 
don't think users should be required to use this, but this could 
nevertheless be added as an additional and more powerful mechanism.

5. Don't apply punctuation-manipulation to individual blockquotes.

#+BEGIN_QUOTE :punctuation-manipulation nil
This is a blockquote in its own paragraph. [@doe p. 45]
#+END_QUOTE

(Maybe adding someting like this even for inline citations would be a 
good idea: "asdf." [cite: @doe p. 45 :punctuation-manipulation nil]
I can imagine many users will be happy with the automatism 95% of the 
time, but there will always be egde cases.)

Again, as I've said above, a combination of these options might be the 
way to go. E.g., the default could be to disable 
punctuation-manipulation for blockquotes. In addition to that there 
could also be a properties based mechanism that offers more options. 
(Thinking more about it, 3 and 4 would actually also play nicely together.)

Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

Denis Maier <denismaier@mailbox.org> writes:

> I think there are a couple of options and combinations thereof:
>
> 1. Do nothing special, just treat block quotations like regular paragraphs.
>
> 2. Don't apply this punctuation-manipulation to blockquotes at all, i.e. 
> let authors handle this manually.
>
> 3. Adjust RULE in the manipulation function: Let it accept two patterns:
> '((outside outside before) (outside outside after))
> The first would apply to inline quotations, the second to blockquotes.
>
> 4. IIUC, it's possible to set certain properties at block elements. So, 
> maybe you could actually add citation information there, e.g.:
>
> #+BEGIN_QUOTE :cite @doe p.45
> This is a blockquote in its own paragraph.
> #+END_QUOTE
>
>
> Unsure about the syntax, but you get the idea. This should allow for 
> complex manipulations: the citation can be added before or after the 
> punctuation mark; if a journal requires quotation marks even around 
> blockquotes => just add them during export; if the period is not part of 
> the original quotation you could just do this:
>
> #+BEGIN_QUOTE :cite @doe p.45 :punct .
> This is a blockquote in its own paragraph
> #+END_QUOTE
>
>
> Then the period will be added during export, but if a journal wants an 
> ellipsis in that case you can also just add that during export.
> 	This is a blockquote in its own paragraph [...].[1]
>
> Now, this is of course rather complex and might add a lot of overhead. I 
> don't think users should be required to use this, but this could 
> nevertheless be added as an additional and more powerful mechanism.
>
> 5. Don't apply punctuation-manipulation to individual blockquotes.
>
> #+BEGIN_QUOTE :punctuation-manipulation nil
> This is a blockquote in its own paragraph. [@doe p. 45]
> #+END_QUOTE
>
> (Maybe adding someting like this even for inline citations would be a 
> good idea: "asdf." [cite: @doe p. 45 :punctuation-manipulation nil]
> I can imagine many users will be happy with the automatism 95% of the 
> time, but there will always be egde cases.)
>
> Again, as I've said above, a combination of these options might be the 
> way to go. E.g., the default could be to disable 
> punctuation-manipulation for blockquotes. In addition to that there 
> could also be a properties based mechanism that offers more options. 
> (Thinking more about it, 3 and 4 would actually also play nicely
> together.)

Getting back to this thread… I'd like to see it moving forward.

You lost me here. Punctuation moving was implemented with notes cites in
mind (much like Pandoc, I think). More exactly, punctuation shuffling
may happen whenever a cite turns into a footnote. For any other case,
the author is expected to place the cite object manually. So, the
automation does not apply to blockquotes, in the sense that inline
quotes or blockquotes use the same location for footnotes.

Does that make sense?

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sat, Jun 5, 2021 at 5:35 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Getting back to this thread… I'd like to see it moving forward.

Yes. This is the last missing piece I see!

> You lost me here. Punctuation moving was implemented with notes cites in
> mind (much like Pandoc, I think).

Correct. And also biblatex.

> More exactly, punctuation shuffling
> may happen whenever a cite turns into a footnote. For any other case,
> the author is expected to place the cite object manually. So, the
> automation does not apply to blockquotes, in the sense that inline
> quotes or blockquotes use the same location for footnotes.

I guess to bottomline it, see the link that Denis provided, including
to the end with footnotes.

https://writingcenter.uagc.edu/block-quotations

I haven't tested the punctuation moving for this specifically, but it
would suggest a user would place this in a blockquote:

... end of sentence. [cite:@doe]

... and the result should remove the space for a footnote.

... end of sentence.[1]

I don't know how quotes work here, but perhaps there's no difference. Denis?

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Hi,

Am 05.06.2021 um 23:45 schrieb Bruce D'Arcus:
> On Sat, Jun 5, 2021 at 5:35 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
>> Getting back to this thread… I'd like to see it moving forward.
> Yes. This is the last missing piece I see!
Yes, good this is coming.
>
>> You lost me here. Punctuation moving was implemented with notes cites in
>> mind (much like Pandoc, I think).
> Correct. And also biblatex.
>
>> More exactly, punctuation shuffling
>> may happen whenever a cite turns into a footnote. For any other case,
>> the author is expected to place the cite object manually. So, the
>> automation does not apply to blockquotes, in the sense that inline
>> quotes or blockquotes use the same location for footnotes.
> I guess to bottomline it, see the link that Denis provided, including
> to the end with footnotes.
>
> https://writingcenter.uagc.edu/block-quotations
>
> I haven't tested the punctuation moving for this specifically, but it
> would suggest a user would place this in a blockquote:
>
> ... end of sentence. [cite:@doe]
>
> ... and the result should remove the space for a footnote.
>
> ... end of sentence.[1]
>
> I don't know how quotes work here, but perhaps there's no difference. Denis?
As as English and German are concerned, I think there is no difference 
between the two, yes. OTOH, French may be again different as the 
footnote marker may appear before the final punctuation.

... end of sentence[1].


Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

Denis Maier <denismaier@mailbox.org> writes:

> Yes, good this is coming.

As a step forward, I rebased wip-cite-new branch with more support for
note numbers handling.

I added three customizable variables:

- org-cite-adjust-note-numbers, which simply allows the user to toggle
  punctuation and note number moving (on by default).

- org-cite-note-rules, which defines what rules to apply according to
  locale, expressed as a language tag, as in RFC 4646.

- org-cite-punctuation-marks, which lists strings recognized as
  punctuation in the process.

`csl' and `basic' processors now both make use of this.

I'd appreciate some feedback, in particular about the docstrings of the
variables above. I focused on the "note numbers" topic instead of
"punctuation" since I found the latter too generic.

Also, there are some points that may need to be discussed:

- I'm not sure about the `org-cite-punctuation-marks' variable being
  global, i.e., not locale-specific.

- There is no support for this in LaTeX-derived back-ends, because
  I don't know when a citation is going to become a footnote. As
  a reminder, there is no "\footcite" command in `biblatex' processor.
  OTOH, users might prefer using a more advanced mechanism, e.g.,
  csquotes.

- It doesn't do anything special in quote blocks, because I'm still not
  sure there is something to do. AFAIU, special casing there only
  applies to author-date location, which out of the scope of this code.

WDYT?

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

I'm not able to test this closely ATM, but a few things ...

On Sat, Jun 12, 2021 at 5:39 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:
>
> Hello,
>
> Denis Maier <denismaier@mailbox.org> writes:
>
> > Yes, good this is coming.
>
> As a step forward, I rebased wip-cite-new branch with more support for
> note numbers handling.
>
> I added three customizable variables:
>
> - org-cite-adjust-note-numbers, which simply allows the user to toggle
>   punctuation and note number moving (on by default).
>
> - org-cite-note-rules, which defines what rules to apply according to
>   locale, expressed as a language tag, as in RFC 4646.
>
> - org-cite-punctuation-marks, which lists strings recognized as
>   punctuation in the process.
>
> `csl' and `basic' processors now both make use of this.
>
> I'd appreciate some feedback, in particular about the docstrings of the
> variables above. I focused on the "note numbers" topic instead of
> "punctuation" since I found the latter too generic.
>
> Also, there are some points that may need to be discussed:
>
> - I'm not sure about the `org-cite-punctuation-marks' variable being
>   global, i.e., not locale-specific.

One of the things I've wondered about, but cannot offer any expertise
on (really, I have no clue), is how this might work for languages like
Arabic or Mandarin.

> - There is no support for this in LaTeX-derived back-ends, because
>   I don't know when a citation is going to become a footnote. As
>   a reminder, there is no "\footcite" command in `biblatex' processor.
>   OTOH, users might prefer using a more advanced mechanism, e.g.,
>   csquotes.

So the upshot is if users want this functionality for LaTeX/PDF, they
should use oc-biblatex for that export target?

> - It doesn't do anything special in quote blocks, because I'm still not
>   sure there is something to do. AFAIU, special casing there only
>   applies to author-date location, which out of the scope of this code.

Here's the scenario I believe Denis was concerned about:

#+begin_quote
... block with citation at end. [cite:@doe]
#+end_quote

I think in standard author-year styles, we'd want:

... block with citation at end. (Doe, 2020)

... while in note-based and "US" (space removed):

... block with citation at end.[1]

Not sure if that has any practical import, but just to clarify.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

>> - I'm not sure about the `org-cite-punctuation-marks' variable being
>>   global, i.e., not locale-specific.
>
> One of the things I've wondered about, but cannot offer any expertise
> on (really, I have no clue), is how this might work for languages like
> Arabic or Mandarin.

Of course, different punctuation marks exist, so characters are at least
locale-dependant. But I don't know if the set is entirely determined by
the locale or if it also depends on the "style" of the document.

>> - There is no support for this in LaTeX-derived back-ends, because
>>   I don't know when a citation is going to become a footnote. As
>>   a reminder, there is no "\footcite" command in `biblatex' processor.
>>   OTOH, users might prefer using a more advanced mechanism, e.g.,
>>   csquotes.
>
> So the upshot is if users want this functionality for LaTeX/PDF, they
> should use oc-biblatex for that export target?

Actually, that's the opposite. I don't know if it is possible, or even
appropriate, to port that feature to `biblatex' and `natbib' processors.
So, ATM, if you want it, and target LaTeX/PDF, your only option is
`csl'.

>> - It doesn't do anything special in quote blocks, because I'm still not
>>   sure there is something to do. AFAIU, special casing there only
>>   applies to author-date location, which out of the scope of this code.
>
> Here's the scenario I believe Denis was concerned about:
>
> #+begin_quote
> ... block with citation at end. [cite:@doe]
> #+end_quote
>
> I think in standard author-year styles, we'd want:
>
> ... block with citation at end. (Doe, 2020)
>
> ... while in note-based and "US" (space removed):
>
> ... block with citation at end.[1]
>
> Not sure if that has any practical import, but just to clarify.

I understand this, but, AFAICT, this is the output we get already. So
I don't think there is anything to do.

Thank you for the feedback.

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sat, Jun 12, 2021 at 6:04 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> Of course, different punctuation marks exist, so characters are at least
> locale-dependant. But I don't know if the set is entirely determined by
> the locale or if it also depends on the "style" of the document.

It's a good question. I need to think about this one.

> > So the upshot is if users want this functionality for LaTeX/PDF, they
> > should use oc-biblatex for that export target?
>
> Actually, that's the opposite. I don't know if it is possible, or even
> appropriate, to port that feature to `biblatex' and `natbib' processors.
> So, ATM, if you want it, and target LaTeX/PDF, your only option is
> `csl'.

OIC; I misread your post.

To clarify (I realized after sending that message I should have done
so), biblatex, at least, has this sort of punctuation moving included
out-of-box. So there's indeed no need to include it in oc-biblatex.

I thought you were saying you were not including such moving in oc-csl
latex export, hence my question.

So I think we're good on that then.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

[-- Attachment #1: Type: text/html, Size: 4222 bytes --]

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 12.06.2021 um 11:39 schrieb Nicolas Goaziou:
> Hello,
> 
> Denis Maier <denismaier@mailbox.org> writes:
> 
>> Yes, good this is coming.
> 
> As a step forward, I rebased wip-cite-new branch with more support for
> note numbers handling.
> 
> I added three customizable variables:
> 
> - org-cite-adjust-note-numbers, which simply allows the user to toggle
>    punctuation and note number moving (on by default).
> 
> - org-cite-note-rules, which defines what rules to apply according to
>    locale, expressed as a language tag, as in RFC 4646.
> 
> - org-cite-punctuation-marks, which lists strings recognized as
>    punctuation in the process.
> 
> `csl' and `basic' processors now both make use of this.
> 
> I'd appreciate some feedback, in particular about the docstrings of the
> variables above. I focused on the "note numbers" topic instead of
> "punctuation" since I found the latter too generic.
> 
> Also, there are some points that may need to be discussed:
> 
> - I'm not sure about the `org-cite-punctuation-marks' variable being
>    global, i.e., not locale-specific.
> 
> - There is no support for this in LaTeX-derived back-ends, because
>    I don't know when a citation is going to become a footnote. As
>    a reminder, there is no "\footcite" command in `biblatex' processor.
>    OTOH, users might prefer using a more advanced mechanism, e.g.,
>    csquotes.
> 
> - It doesn't do anything special in quote blocks, because I'm still not
>    sure there is something to do. AFAIU, special casing there only
>    applies to author-date location, which out of the scope of this code.
> 
> WDYT?

Ok, I've managed to test this a bit, and I think this looks pretty good 
so far.

The only question I'd still have is if this could somehow also cover the 
reverse situation (going from a note style to author-date). I've noticed 
that simply adding a new language rule doesn't work anymore---as opposed 
to my initial tests with earlier iterations of that mechanism. Seems 
like this mechanism is now only triggered when using a note based style.

Best,
Denis

> 
> Regards,
> 

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 2419 bytes --]

Nicolas explained the reverse is out of scope, and gave a reasonable
explanation why (because much harder to reconstruct missing information
IIRC).

On Sun, Jun 13, 2021, 2:54 PM Denis Maier <denismaier@mailbox.org> wrote:

> Am 12.06.2021 um 11:39 schrieb Nicolas Goaziou:
> > Hello,
> >
> > Denis Maier <denismaier@mailbox.org> writes:
> >
> >> Yes, good this is coming.
> >
> > As a step forward, I rebased wip-cite-new branch with more support for
> > note numbers handling.
> >
> > I added three customizable variables:
> >
> > - org-cite-adjust-note-numbers, which simply allows the user to toggle
> >    punctuation and note number moving (on by default).
> >
> > - org-cite-note-rules, which defines what rules to apply according to
> >    locale, expressed as a language tag, as in RFC 4646.
> >
> > - org-cite-punctuation-marks, which lists strings recognized as
> >    punctuation in the process.
> >
> > `csl' and `basic' processors now both make use of this.
> >
> > I'd appreciate some feedback, in particular about the docstrings of the
> > variables above. I focused on the "note numbers" topic instead of
> > "punctuation" since I found the latter too generic.
> >
> > Also, there are some points that may need to be discussed:
> >
> > - I'm not sure about the `org-cite-punctuation-marks' variable being
> >    global, i.e., not locale-specific.
> >
> > - There is no support for this in LaTeX-derived back-ends, because
> >    I don't know when a citation is going to become a footnote. As
> >    a reminder, there is no "\footcite" command in `biblatex' processor.
> >    OTOH, users might prefer using a more advanced mechanism, e.g.,
> >    csquotes.
> >
> > - It doesn't do anything special in quote blocks, because I'm still not
> >    sure there is something to do. AFAIU, special casing there only
> >    applies to author-date location, which out of the scope of this code.
> >
> > WDYT?
>
> Ok, I've managed to test this a bit, and I think this looks pretty good
> so far.
>
> The only question I'd still have is if this could somehow also cover the
> reverse situation (going from a note style to author-date). I've noticed
> that simply adding a new language rule doesn't work anymore---as opposed
> to my initial tests with earlier iterations of that mechanism. Seems
> like this mechanism is now only triggered when using a note based style.
>
> Best,
> Denis
>
> >
> > Regards,
> >
>
>

[-- Attachment #2: Type: text/html, Size: 3210 bytes --]

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

[-- Attachment #1: Type: text/html, Size: 4058 bytes --]

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

[-- Attachment #1: Type: text/plain, Size: 2735 bytes --]

I'll let you two sort it out; I don't have a position.

On Sun, Jun 13, 2021, 3:23 PM Denis Maier <denismaier@mailbox.org> wrote:

>
> Bruce D'Arcus <bdarcus@gmail.com> hat am 14.06.2021 00:04 geschrieben:
>
>
> Nicolas explained the reverse is out of scope,
>
> IIRC, it was out of scope ATM.
>
> and gave a reasonable explanation why (because much harder to reconstruct
> missing information IIRC).
>
> That's where I disagree. I think the opposite is true.
>
> On Sun, Jun 13, 2021, 2:54 PM Denis Maier <denismaier@mailbox.org> wrote:
>
> Am 12.06.2021 um 11:39 schrieb Nicolas Goaziou:
> > Hello,
> >
> > Denis Maier <denismaier@mailbox.org> writes:
> >
> >> Yes, good this is coming.
> >
> > As a step forward, I rebased wip-cite-new branch with more support for
> > note numbers handling.
> >
> > I added three customizable variables:
> >
> > - org-cite-adjust-note-numbers, which simply allows the user to toggle
> >    punctuation and note number moving (on by default).
> >
> > - org-cite-note-rules, which defines what rules to apply according to
> >    locale, expressed as a language tag, as in RFC 4646.
> >
> > - org-cite-punctuation-marks, which lists strings recognized as
> >    punctuation in the process.
> >
> > `csl' and `basic' processors now both make use of this.
> >
> > I'd appreciate some feedback, in particular about the docstrings of the
> > variables above. I focused on the "note numbers" topic instead of
> > "punctuation" since I found the latter too generic.
> >
> > Also, there are some points that may need to be discussed:
> >
> > - I'm not sure about the `org-cite-punctuation-marks' variable being
> >    global, i.e., not locale-specific.
> >
> > - There is no support for this in LaTeX-derived back-ends, because
> >    I don't know when a citation is going to become a footnote. As
> >    a reminder, there is no "\footcite" command in `biblatex' processor.
> >    OTOH, users might prefer using a more advanced mechanism, e.g.,
> >    csquotes.
> >
> > - It doesn't do anything special in quote blocks, because I'm still not
> >    sure there is something to do. AFAIU, special casing there only
> >    applies to author-date location, which out of the scope of this code.
> >
> > WDYT?
>
> Ok, I've managed to test this a bit, and I think this looks pretty good
> so far.
>
> The only question I'd still have is if this could somehow also cover the
> reverse situation (going from a note style to author-date). I've noticed
> that simply adding a new language rule doesn't work anymore---as opposed
> to my initial tests with earlier iterations of that mechanism. Seems
> like this mechanism is now only triggered when using a note based style.
>
> Best,
> Denis
>
> >
> > Regards,
> >
>
>

[-- Attachment #2: Type: text/html, Size: 4596 bytes --]

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

[-- Attachment #1: Type: text/plain, Size: 6544 bytes --]

Below a few examples of what I mean.

WDYT? Am I missing something?

Denis
===========================================================
#+cite_export: csl
#+cite_export: csl 
"C:/Users/denis/Zotero/styles/chicago-note-bibliography.csl"
#+bibliography: test.bib

* Original source

"A quotation ending with a period."

"A quotation ending without punctuation"

* Author-date style input (= semantically non-strict input)

"A quotation ending with a period" [cite: @hoel-71-whole].

"A quotation ending without punctuation" [cite: @hoel-71-whole].

** author-date output with language: en-us
Expected: "A quotation ending with a period" (Hoel 1971).
Actual:   "A quotation ending with a period" (Hoel 1971).

Expected: "A quotation ending without punctuation" (Hoel 1971).
Actual:   "A quotation ending without punctuation" (Hoel 1971).

=> ok

** author-date output with language: de
Expected: "A quotation ending with a period" (Hoel 1971).
Actual:   "A quotation ending with a period" (Hoel 1971).

Expected: "A quotation ending without punctuation" (Hoel 1971).
Actual:   "A quotation ending without punctuation" (Hoel 1971).

=> ok

** note style output with language: en-us
Expected: "A quotation ending with a period."[1]
Actual:   "A quotation ending with a period."[1]

Expected: "A quotation ending without punctuation."[1]
Actual:   "A quotation ending without punctuation."[1]

=> ok

** note style output with language: en-gb or de
Expected: "A quotation ending with a period."[1]
Actual:   "A quotation ending with a period".[1]

Expected: "A quotation ending without punctuation".[1]
Actual:   "A quotation ending without punctuation".[1]

=> Here, we cannot distinguish between the two cases as we don't know 
whether punctuation appears in the original source.

* Note style input (=semantically strict input)

"A quotation ending with a period." [cite: @hoel-71-whole]

"A quotation ending without punctuation". [cite: @hoel-71-whole]

As the input preserves the location of punctuation in the original 
material, I'd say it should be much easier to deal with this. We don't 
have to add information which isn't in the input, but rather we'll just 
have to move any punctuation to after the citation object. Maybe I'm 
missing something, but to me this looks like a much simpler operation 
than going in the opposite direction.

Maybe we should stop talking about author date vs note style input, but 
rather about strict vs. non-strict input. And, I think that's the whole 
issue: going from strict to non-strict is easy while the other way is 
more complicated; at least, it would require some more efforts to 
support the last case (going from non-strict input to note style output 
with a language that requires strict output.
=========================================================================

Am 14.06.2021 um 00:47 schrieb Bruce D'Arcus:
> I'll let you two sort it out; I don't have a position.
>
> On Sun, Jun 13, 2021, 3:23 PM Denis Maier <denismaier@mailbox.org 
> <mailto:denismaier@mailbox.org>> wrote:
>
>
>>     Bruce D'Arcus <bdarcus@gmail.com <mailto:bdarcus@gmail.com>> hat
>>     am 14.06.2021 00:04 geschrieben:
>>
>>
>>     Nicolas explained the reverse is out of scope,
>     IIRC, it was out of scope ATM.
>>     and gave a reasonable explanation why (because much harder to
>>     reconstruct missing information IIRC).
>     That's where I disagree. I think the opposite is true.
>
>>     On Sun, Jun 13, 2021, 2:54 PM Denis Maier <denismaier@mailbox.org
>>     <mailto:denismaier@mailbox.org>> wrote:
>>
>>         Am 12.06.2021 um 11:39 schrieb Nicolas Goaziou:
>>         > Hello,
>>         >
>>         > Denis Maier <denismaier@mailbox.org
>>         <mailto:denismaier@mailbox.org>> writes:
>>         >
>>         >> Yes, good this is coming.
>>         >
>>         > As a step forward, I rebased wip-cite-new branch with more
>>         support for
>>         > note numbers handling.
>>         >
>>         > I added three customizable variables:
>>         >
>>         > - org-cite-adjust-note-numbers, which simply allows the
>>         user to toggle
>>         >    punctuation and note number moving (on by default).
>>         >
>>         > - org-cite-note-rules, which defines what rules to apply
>>         according to
>>         >    locale, expressed as a language tag, as in RFC 4646.
>>         >
>>         > - org-cite-punctuation-marks, which lists strings
>>         recognized as
>>         >    punctuation in the process.
>>         >
>>         > `csl' and `basic' processors now both make use of this.
>>         >
>>         > I'd appreciate some feedback, in particular about the
>>         docstrings of the
>>         > variables above. I focused on the "note numbers" topic
>>         instead of
>>         > "punctuation" since I found the latter too generic.
>>         >
>>         > Also, there are some points that may need to be discussed:
>>         >
>>         > - I'm not sure about the `org-cite-punctuation-marks'
>>         variable being
>>         >    global, i.e., not locale-specific.
>>         >
>>         > - There is no support for this in LaTeX-derived back-ends,
>>         because
>>         >    I don't know when a citation is going to become a
>>         footnote. As
>>         >    a reminder, there is no "\footcite" command in
>>         `biblatex' processor.
>>         >    OTOH, users might prefer using a more advanced
>>         mechanism, e.g.,
>>         >    csquotes.
>>         >
>>         > - It doesn't do anything special in quote blocks, because
>>         I'm still not
>>         >    sure there is something to do. AFAIU, special casing
>>         there only
>>         >    applies to author-date location, which out of the scope
>>         of this code.
>>         >
>>         > WDYT?
>>
>>         Ok, I've managed to test this a bit, and I think this looks
>>         pretty good
>>         so far.
>>
>>         The only question I'd still have is if this could somehow
>>         also cover the
>>         reverse situation (going from a note style to author-date).
>>         I've noticed
>>         that simply adding a new language rule doesn't work
>>         anymore---as opposed
>>         to my initial tests with earlier iterations of that
>>         mechanism. Seems
>>         like this mechanism is now only triggered when using a note
>>         based style.
>>
>>         Best,
>>         Denis
>>
>>         >
>>         > Regards,
>>         >
>>


[-- Attachment #2: Type: text/html, Size: 10180 bytes --]

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Just one addition: I think it will be perfectly fine if the current 
suggestion is added now. I guess adding additional features later will 
be always possible, right? I don't know what the plans are for the next 
org release, and I don't want this question here to stand in the way.

Denis

Am 14.06.2021 um 13:45 schrieb Denis Maier:
> Below a few examples of what I mean.
> 
> WDYT? Am I missing something?
> 
> Denis
> ===========================================================
> #+cite_export: csl
> #+cite_export: csl 
> "C:/Users/denis/Zotero/styles/chicago-note-bibliography.csl"
> #+bibliography: test.bib
> 
> * Original source
> 
> "A quotation ending with a period."
> 
> "A quotation ending without punctuation"
> 
> * Author-date style input (= semantically non-strict input)
> 
> "A quotation ending with a period" [cite: @hoel-71-whole].
> 
> "A quotation ending without punctuation" [cite: @hoel-71-whole].
> 
> ** author-date output with language: en-us
> Expected: "A quotation ending with a period" (Hoel 1971).
> Actual:   "A quotation ending with a period" (Hoel 1971).
> 
> Expected: "A quotation ending without punctuation" (Hoel 1971).
> Actual:   "A quotation ending without punctuation" (Hoel 1971).
> 
> => ok
> 
> ** author-date output with language: de
> Expected: "A quotation ending with a period" (Hoel 1971).
> Actual:   "A quotation ending with a period" (Hoel 1971).
> 
> Expected: "A quotation ending without punctuation" (Hoel 1971).
> Actual:   "A quotation ending without punctuation" (Hoel 1971).
> 
> => ok
> 
> ** note style output with language: en-us
> Expected: "A quotation ending with a period."[1]
> Actual:   "A quotation ending with a period."[1]
> 
> Expected: "A quotation ending without punctuation."[1]
> Actual:   "A quotation ending without punctuation."[1]
> 
> => ok
> 
> ** note style output with language: en-gb or de
> Expected: "A quotation ending with a period."[1]
> Actual:   "A quotation ending with a period".[1]
> 
> Expected: "A quotation ending without punctuation".[1]
> Actual:   "A quotation ending without punctuation".[1]
> 
> => Here, we cannot distinguish between the two cases as we don't know 
> whether punctuation appears in the original source.
> 
> * Note style input (=semantically strict input)
> 
> "A quotation ending with a period." [cite: @hoel-71-whole]
> 
> "A quotation ending without punctuation". [cite: @hoel-71-whole]
> 
> As the input preserves the location of punctuation in the original 
> material, I'd say it should be much easier to deal with this. We don't 
> have to add information which isn't in the input, but rather we'll just 
> have to move any punctuation to after the citation object. Maybe I'm 
> missing something, but to me this looks like a much simpler operation 
> than going in the opposite direction.
> 
> Maybe we should stop talking about author date vs note style input, but 
> rather about strict vs. non-strict input. And, I think that's the whole 
> issue: going from strict to non-strict is easy while the other way is 
> more complicated; at least, it would require some more efforts to 
> support the last case (going from non-strict input to note style output 
> with a language that requires strict output.
> =========================================================================
> 
> Am 14.06.2021 um 00:47 schrieb Bruce D'Arcus:
>> I'll let you two sort it out; I don't have a position.
>>
>> On Sun, Jun 13, 2021, 3:23 PM Denis Maier <denismaier@mailbox.org 
>> <mailto:denismaier@mailbox.org>> wrote:
>>
>>
>>>     Bruce D'Arcus <bdarcus@gmail.com <mailto:bdarcus@gmail.com>> hat
>>>     am 14.06.2021 00:04 geschrieben:
>>>
>>>
>>>     Nicolas explained the reverse is out of scope,
>>     IIRC, it was out of scope ATM.
>>>     and gave a reasonable explanation why (because much harder to
>>>     reconstruct missing information IIRC).
>>     That's where I disagree. I think the opposite is true.
>>
>>>     On Sun, Jun 13, 2021, 2:54 PM Denis Maier <denismaier@mailbox.org
>>>     <mailto:denismaier@mailbox.org>> wrote:
>>>
>>>         Am 12.06.2021 um 11:39 schrieb Nicolas Goaziou:
>>>         > Hello,
>>>         >
>>>         > Denis Maier <denismaier@mailbox.org
>>>         <mailto:denismaier@mailbox.org>> writes:
>>>         >
>>>         >> Yes, good this is coming.
>>>         >
>>>         > As a step forward, I rebased wip-cite-new branch with more
>>>         support for
>>>         > note numbers handling.
>>>         >
>>>         > I added three customizable variables:
>>>         >
>>>         > - org-cite-adjust-note-numbers, which simply allows the
>>>         user to toggle
>>>         >    punctuation and note number moving (on by default).
>>>         >
>>>         > - org-cite-note-rules, which defines what rules to apply
>>>         according to
>>>         >    locale, expressed as a language tag, as in RFC 4646.
>>>         >
>>>         > - org-cite-punctuation-marks, which lists strings
>>>         recognized as
>>>         >    punctuation in the process.
>>>         >
>>>         > `csl' and `basic' processors now both make use of this.
>>>         >
>>>         > I'd appreciate some feedback, in particular about the
>>>         docstrings of the
>>>         > variables above. I focused on the "note numbers" topic
>>>         instead of
>>>         > "punctuation" since I found the latter too generic.
>>>         >
>>>         > Also, there are some points that may need to be discussed:
>>>         >
>>>         > - I'm not sure about the `org-cite-punctuation-marks'
>>>         variable being
>>>         >    global, i.e., not locale-specific.
>>>         >
>>>         > - There is no support for this in LaTeX-derived back-ends,
>>>         because
>>>         >    I don't know when a citation is going to become a
>>>         footnote. As
>>>         >    a reminder, there is no "\footcite" command in
>>>         `biblatex' processor.
>>>         >    OTOH, users might prefer using a more advanced
>>>         mechanism, e.g.,
>>>         >    csquotes.
>>>         >
>>>         > - It doesn't do anything special in quote blocks, because
>>>         I'm still not
>>>         >    sure there is something to do. AFAIU, special casing
>>>         there only
>>>         >    applies to author-date location, which out of the scope
>>>         of this code.
>>>         >
>>>         > WDYT?
>>>
>>>         Ok, I've managed to test this a bit, and I think this looks
>>>         pretty good
>>>         so far.
>>>
>>>         The only question I'd still have is if this could somehow
>>>         also cover the
>>>         reverse situation (going from a note style to author-date).
>>>         I've noticed
>>>         that simply adding a new language rule doesn't work
>>>         anymore---as opposed
>>>         to my initial tests with earlier iterations of that
>>>         mechanism. Seems
>>>         like this mechanism is now only triggered when using a note
>>>         based style.
>>>
>>>         Best,
>>>         Denis
>>>
>>>         >
>>>         > Regards,
>>>         >
>>>
> 

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Mon, Jun 14, 2021 at 7:45 AM Denis Maier <denismaier@mailbox.org> wrote:
>
> Below a few examples of what I mean.
>
> WDYT? Am I missing something?
>
> Denis
> ===========================================================
> #+cite_export: csl
> #+cite_export: csl "C:/Users/denis/Zotero/styles/chicago-note-bibliography.csl"
> #+bibliography: test.bib
>
> * Original source
>
> "A quotation ending with a period."
>
> "A quotation ending without punctuation"
>
> * Author-date style input (= semantically non-strict input)
>
> "A quotation ending with a period" [cite: @hoel-71-whole].
>
> "A quotation ending without punctuation" [cite: @hoel-71-whole].
>
> ** author-date output with language: en-us
> Expected: "A quotation ending with a period" (Hoel 1971).
> Actual:   "A quotation ending with a period" (Hoel 1971).
>
> Expected: "A quotation ending without punctuation" (Hoel 1971).
> Actual:   "A quotation ending without punctuation" (Hoel 1971).
>
> => ok
>
> ** author-date output with language: de
> Expected: "A quotation ending with a period" (Hoel 1971).
> Actual:   "A quotation ending with a period" (Hoel 1971).
>
> Expected: "A quotation ending without punctuation" (Hoel 1971).
> Actual:   "A quotation ending without punctuation" (Hoel 1971).
>
> => ok
>
> ** note style output with language: en-us
> Expected: "A quotation ending with a period."[1]
> Actual:   "A quotation ending with a period."[1]
>
> Expected: "A quotation ending without punctuation."[1]
> Actual:   "A quotation ending without punctuation."[1]
>
> => ok
>
> ** note style output with language: en-gb or de
> Expected: "A quotation ending with a period."[1]
> Actual:   "A quotation ending with a period".[1]
>
> Expected: "A quotation ending without punctuation".[1]
> Actual:   "A quotation ending without punctuation".[1]
>
> => Here, we cannot distinguish between the two cases as we don't know whether punctuation appears in the original source.
>
> * Note style input (=semantically strict input)
>
> "A quotation ending with a period." [cite: @hoel-71-whole]
>
> "A quotation ending without punctuation". [cite: @hoel-71-whole]
>
> As the input preserves the location of punctuation in the original material, I'd say it should be much easier to deal with this. We don't have to add information which isn't in the input, but rather we'll just have to move any punctuation to after the citation object. Maybe I'm missing something, but to me this looks like a much simpler operation than going in the opposite direction.
>
> Maybe we should stop talking about author date vs note style input, but rather about strict vs. non-strict input.

It's definitely not author-date vs note. I see it as in-text citations
vs note citations. As in, the former applies to other styles beyond
author-date.

The example you are highlighlighting here was why I was earlier
suggesting for a rule that would allow something like this input:

"A quotation ending with a period." [cite: @hoel-71-whole].

... where the second would be dropped, hence getting the expected output.

But IDK what Nicolas thinks of considering that now, or deferring for
later consideration.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

Missing word on the key sentence ...

On Mon, Jun 14, 2021 at 7:37 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> The example you are highlighlighting here was why I was earlier
> suggesting for a rule that would allow something like this input:
>
> "A quotation ending with a period." [cite: @hoel-71-whole].
>
> ... where the second would be dropped, hence getting the expected output.

Should be "second period would ..."

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Mon, Jun 14, 2021 at 7:45 AM Denis Maier <denismaier@mailbox.org> wrote:

>> * Note style input (=semantically strict input)
>>
>> "A quotation ending with a period." [cite: @hoel-71-whole]
>>
>> "A quotation ending without punctuation". [cite: @hoel-71-whole]
>>
>> As the input preserves the location of punctuation in the original
>> material, I'd say it should be much easier to deal with this. We
>> don't have to add information which isn't in the input, but rather
>> we'll just have to move any punctuation to after the citation
>> object. Maybe I'm missing something, but to me this looks like
>> a much simpler operation than going in the opposite direction.

This cannot be. We don't know anything about the cite after the
quotation. A bare cite could be starting out a new sentence:

  "A quotation ending with a period." [cite: @hoel-71-whole] pretends…

OTOH, we know perfectly when a citation is meant to become a footnote
(at least in basic and csl processors). And we know — almost, as you
demonstrated — where to put that footnote.

Moreover, I think the syntax you propose has another drawback: it
doesn't correspond to any desired output (note or something else). As
this looks artificial, I fear it might hinder readability of the Org
document.

  ... period." [cite:@doe21] [cite/text:@doe21] pretends…

>> Maybe we should stop talking about author date vs note style input, but rather about strict vs. non-strict input.
>
> It's definitely not author-date vs note. I see it as in-text citations
> vs note citations. As in, the former applies to other styles beyond
> author-date.

I think the current patch is purely about note citations. I mentioned
"author-date" in a docstring just because I didn't know how to express
it otherwise. So, in a way, I agree it can be considered as in-text
citations vs note citations, indeed.

> The example you are highlighlighting here was why I was earlier
> suggesting for a rule that would allow something like this input:
>
> "A quotation ending with a period." [cite: @hoel-71-whole].
>
> ... where the second would be dropped, hence getting the expected output.

This is interesting, but we might get false positives, as in the
following (far-fetched) example

  … the so-called "foobar". [cite/text: See @hoel-71-whole p. 42].

which bites us because we need to process even non-note citations to
remove the spurious punctuation while ignoring the necessity of a given
punctuation character.

As another, imperfect, workaround, I submit the following idea for
consideration:

  "A quotation ending without punctuation" [cite: @hoel-71-whole].
  "A quotation ending with a period"[cite: @hoel-71-whole].

IOW, the presence or absence of a space before the citation determines,
according to a note rule, if the punctuation should go inside or outside
the quotation. When processing non-note citations, we just need to
ensure there is at least a space after the previous element, which is
less "dangerous" than removing punctuation.

I find it a bit too subtle, and so error-prone, but so is punctuation
anyway.

WDYT?

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Bruce D'Arcus]

On Sun, Jun 20, 2021 at 3:41 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

...

> > The example you are highlighlighting here was why I was earlier
> > suggesting for a rule that would allow something like this input:
> >
> > "A quotation ending with a period." [cite: @hoel-71-whole].
> >
> > ... where the second would be dropped, hence getting the expected output.
>
> This is interesting, but we might get false positives, as in the
> following (far-fetched) example
>
>   … the so-called "foobar". [cite/text: See @hoel-71-whole p. 42].
>
> which bites us because we need to process even non-note citations to
> remove the spurious punctuation while ignoring the necessity of a given
> punctuation character.

Yeah, I wasn't sure of the details.

> As another, imperfect, workaround, I submit the following idea for
> consideration:
>
>   "A quotation ending without punctuation" [cite: @hoel-71-whole].
>   "A quotation ending with a period"[cite: @hoel-71-whole].
>
> IOW, the presence or absence of a space before the citation determines,
> according to a note rule, if the punctuation should go inside or outside
> the quotation. When processing non-note citations, we just need to
> ensure there is at least a space after the previous element, which is
> less "dangerous" than removing punctuation.
>
> I find it a bit too subtle, and so error-prone, but so is punctuation
> anyway.
>
> WDYT?

Just to confirm, Nicolas, your proposal would basically say the second
example would override the default punctuation-moving behavior for the
locale?

And for an in-text/author-date style, what would the output be with
that example?

I, someone who may have an "en-US" bias you could say, would expect:

 "A quotation ending with a period" (Hoel, 1971).

As in, the in-quote period would be dropped regardless, and therefore
a space would also need to be added.

Is that right Denis? It wouldn't make sense to do any of these; right?

 "A quotation ending with a period." (Hoel, 1971).
 "A quotation ending with a period." (Hoel, 1971)
 "A quotation ending with a period"(Hoel, 1971).
etc.

Bruce

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Sun, Jun 20, 2021 at 3:41 AM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

>> As another, imperfect, workaround, I submit the following idea for
>> consideration:
>>
>>   "A quotation ending without punctuation" [cite: @hoel-71-whole].
>>   "A quotation ending with a period"[cite: @hoel-71-whole].
>>
>> IOW, the presence or absence of a space before the citation determines,
>> according to a note rule, if the punctuation should go inside or outside
>> the quotation. When processing non-note citations, we just need to
>> ensure there is at least a space after the previous element, which is
>> less "dangerous" than removing punctuation.
>>
>> I find it a bit too subtle, and so error-prone, but so is punctuation
>> anyway.
>>
>> WDYT?
>
> Just to confirm, Nicolas, your proposal would basically say the second
> example would override the default punctuation-moving behavior for the
> locale?

No, this behaviour is still customizable. For example,
`org-cite-note-rules' would contain the following entry:

  ("fr" adaptive same before)

where `adaptive' means the proposal above.

OTOH, "en-us" would still be

  ("en-us" inside outside after)

where the space does not matter since the punctuation always goes
inside.

> And for an in-text/author-date style, what would the output be with
> that example?
>
> I, someone who may have an "en-US" bias you could say, would expect:
>
>  "A quotation ending with a period" (Hoel, 1971).
>
> As in, the in-quote period would be dropped regardless, and therefore
> a space would also need to be added.

In any non-note situation, we just make sure the citation is preceded by
a space, so the example would be equivalent to

  "A quotation ending without punctuation" [cite: @hoel-71-whole].
  "A quotation ending with a period" [cite: @hoel-71-whole].

without moving punctuation.

Does that make sense?

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 20.06.2021 um 09:41 schrieb Nicolas Goaziou:
> Hello,
> 
> "Bruce D'Arcus" <bdarcus@gmail.com> writes:
> 
>> On Mon, Jun 14, 2021 at 7:45 AM Denis Maier <denismaier@mailbox.org> wrote:
> 
>>> * Note style input (=semantically strict input)
>>>
>>> "A quotation ending with a period." [cite: @hoel-71-whole]
>>>
>>> "A quotation ending without punctuation". [cite: @hoel-71-whole]
>>>
>>> As the input preserves the location of punctuation in the original
>>> material, I'd say it should be much easier to deal with this. We
>>> don't have to add information which isn't in the input, but rather
>>> we'll just have to move any punctuation to after the citation
>>> object. Maybe I'm missing something, but to me this looks like
>>> a much simpler operation than going in the opposite direction.
> 
> This cannot be. [...]

> 
> As another, imperfect, workaround, I submit the following idea for
> consideration:
> 
>    "A quotation ending without punctuation" [cite: @hoel-71-whole].
>    "A quotation ending with a period"[cite: @hoel-71-whole].
> 
> IOW, the presence or absence of a space before the citation determines,
> according to a note rule, if the punctuation should go inside or outside
> the quotation. When processing non-note citations, we just need to
> ensure there is at least a space after the previous element, which is
> less "dangerous" than removing punctuation.

Yes, I guess we should be safe here. I can't think of a case where you 
wouldn't want a space before a citation.
> 
> I find it a bit too subtle, and so error-prone, but so is punctuation
> anyway. >
> WDYT?


Not a bad approach.

Using a space for this is perhaps too subtle as you say. Also, the 
question is which one should be the default. I'd actually suggest to 
turn it around:

    "A quotation ending without punctuation"[cite: @hoel-71-whole].
    "A quotation ending with a period" [cite: @hoel-71-whole].

Reason for this: People who don't care for this distinction---either 
because they use en-us only, or because they never switch from in-text 
to notes styles---will probably prefer to have a space between quotation 
and citation (in input and output). On the other hand, this here feels 
also a bit odd:
    "A quotation ending without punctuation"[cite: @hoel-71-whole].

In your example the rule would simply be that punctuation cannot jump 
across spaces.

What about some sort of escaping for punctuation that should stay 
outside the quotation marks?
    "A quotation ending without punctuation" [cite: @hoel-71-whole]\.

But, of course, that's imperfect as well. Don't know which option is 
less odd.

Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Nicolas Goaziou]

Hello,

Denis Maier <denismaier@mailbox.org> writes:

> Using a space for this is perhaps too subtle as you say. Also, the
> question is which one should be the default. I'd actually suggest to 
> turn it around:
>
>    "A quotation ending without punctuation"[cite: @hoel-71-whole].
>    "A quotation ending with a period" [cite: @hoel-71-whole].
>
> Reason for this: People who don't care for this distinction---either
> because they use en-us only, or because they never switch from in-text 
> to notes styles---will probably prefer to have a space between
> quotation and citation (in input and output).

People who don't care for this distinction can write 

   "A quotation ending without punctuation" [cite: @hoel-71-whole].
   "A quotation ending with a period" [cite: @hoel-71-whole].

So I guess the default doesn't matter in this case?

> What about some sort of escaping for punctuation that should stay
> outside the quotation marks?
>    "A quotation ending without punctuation" [cite: @hoel-71-whole]\.
>
> But, of course, that's imperfect as well. Don't know which option is
> less odd.

Actual usage may tell. I implemented the spacing-tweak in
"wip-cite-new". You (and others) may want to try it out on real
documents and see if it feels somewhat natural.

Regards,
-- 
Nicolas Goaziou

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Am 21.06.2021 um 10:45 schrieb Nicolas Goaziou:
> Hello,
> 
> Denis Maier <denismaier@mailbox.org> writes:
> 
>> Using a space for this is perhaps too subtle as you say. Also, the
>> question is which one should be the default. I'd actually suggest to
>> turn it around:
>>
>>     "A quotation ending without punctuation"[cite: @hoel-71-whole].
>>     "A quotation ending with a period" [cite: @hoel-71-whole].
>>
>> Reason for this: People who don't care for this distinction---either
>> because they use en-us only, or because they never switch from in-text
>> to notes styles---will probably prefer to have a space between
>> quotation and citation (in input and output).
> 
> People who don't care for this distinction can write
> 
>     "A quotation ending without punctuation" [cite: @hoel-71-whole].
>     "A quotation ending with a period" [cite: @hoel-71-whole].
> 
> So I guess the default doesn't matter in this case?

That's probably correct.

> 
>> What about some sort of escaping for punctuation that should stay
>> outside the quotation marks?
>>     "A quotation ending without punctuation" [cite: @hoel-71-whole]\.
>>
>> But, of course, that's imperfect as well. Don't know which option is
>> less odd.
> 
> Actual usage may tell. I implemented the spacing-tweak in
> "wip-cite-new". You (and others) may want to try it out on real
> documents and see if it feels somewhat natural.

Ok, let's see if others have comments. I can for now at least confirm 
that the mechanism seems to work as intended.

Thanks for all your work on this!

Best,
Denis

Re: [wip-cite-new] Adjust punctuation around citations

[Denis Maier]

Hi,

Am 21.06.2021 um 10:45 schrieb Nicolas Goaziou:
> Hello,
> 
> Denis Maier <denismaier@mailbox.org> writes:
> 
>> Using a space for this is perhaps too subtle as you say. Also, the
>> question is which one should be the default. I'd actually suggest to
>> turn it around:
>>
>>     "A quotation ending without punctuation"[cite: @hoel-71-whole].
>>     "A quotation ending with a period" [cite: @hoel-71-whole].
>>
>> Reason for this: People who don't care for this distinction---either
>> because they use en-us only, or because they never switch from in-text
>> to notes styles---will probably prefer to have a space between
>> quotation and citation (in input and output).
> 
> People who don't care for this distinction can write
> 
>     "A quotation ending without punctuation" [cite: @hoel-71-whole].
>     "A quotation ending with a period" [cite: @hoel-71-whole].
> 
> So I guess the default doesn't matter in this case?

That's probably correct.

> 
>> What about some sort of escaping for punctuation that should stay
>> outside the quotation marks?
>>     "A quotation ending without punctuation" [cite: @hoel-71-whole]\.
>>
>> But, of course, that's imperfect as well. Don't know which option is
>> less odd.
> 
> Actual usage may tell. I implemented the spacing-tweak in
> "wip-cite-new". You (and others) may want to try it out on real
> documents and see if it feels somewhat natural.

Ok, let's see if others have comments. I can for now at least confirm 
that this works as intended.

Thanks again for all your work!

Best,
Denis

> 
> Regards,
>