re-scanning bibliography for org-cite

re-scanning bibliography for org-cite

[Eric S Fraga]

Dear all,

simple question: how can I get org-cite to rescan my bibliography (.bib)
file when it changes?  When I'm writing a paper, as I'm doing now, I add
entries to my bibliography as I write but org-cite is not aware of any
new entries.

I have gone through the code but cannot see where the bib entries are
actually read.  (my bad)

Thank you,
eric

-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-149-g781080
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: re-scanning bibliography for org-cite

[Uwe Brauer]

[-- Attachment #1: Type: text/plain, Size: 697 bytes --]

>>> "ESF" == Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Dear all,

> simple question: how can I get org-cite to rescan my bibliography
> (.bib) file when it changes? When I'm writing a paper, as I'm doing
> now, I add entries to my bibliography as I write but org-cite is not
> aware of any new entries.

> I have gone through the code but cannot see where the bib entries are
> actually read.  (my bad)

Do you use 
org-ref-insert-cite-link

Or 
org-ref-helm-insert-cite-link

I use the latter in that case, if I finish an entry and update the bib
file, then when I call the function again, it is updated, not sure about
org-ref-insert-cite-link



> Thank you,
> eric

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 5673 bytes --]

Re: re-scanning bibliography for org-cite

[Eric S Fraga]

On Thursday, 21 Oct 2021 at 16:51, Uwe Brauer wrote:
> Do you use 
> org-ref-insert-cite-link
> org-ref-helm-insert-cite-link

Neither.  I use org-cite-insert.

-- 
: Eric S Fraga via Emacs 28.0.60, Org release_9.5-149-g781080
: Latest paper written in org: https://arxiv.org/abs/2106.05096

Re: re-scanning bibliography for org-cite

[Bruce D'Arcus]

On Thu, Oct 21, 2021 at 10:12 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>
> Dear all,
>
> simple question: how can I get org-cite to rescan my bibliography (.bib)
> file when it changes?  When I'm writing a paper, as I'm doing now, I add
> entries to my bibliography as I write but org-cite is not aware of any
> new entries.

Do you mean for org-cite-insert?

Which insert processor?

If it's bibtex-actions, you have to configure the filenotify support
mentioned on the README.

I'm not sure how the oc-basic one handles this.

Bruce

Re: re-scanning bibliography for org-cite

[Nicolas Goaziou]

Hello,

"Bruce D'Arcus" <bdarcus@gmail.com> writes:

> On Thu, Oct 21, 2021 at 10:12 AM Eric S Fraga <e.fraga@ucl.ac.uk> wrote:
>>
>> Dear all,
>>
>> simple question: how can I get org-cite to rescan my bibliography (.bib)
>> file when it changes?  When I'm writing a paper, as I'm doing now, I add
>> entries to my bibliography as I write but org-cite is not aware of any
>> new entries.
>
> Do you mean for org-cite-insert?
>
> Which insert processor?
>
> If it's bibtex-actions, you have to configure the filenotify support
> mentioned on the README.
>
> I'm not sure how the oc-basic one handles this.

oc-basic relies on a cache. The cache key contains a hash of the
contents of the bib file. So whenever the bib file is modified, the
cache is invalidated, and oc-basic parses again the file.

IOW, rescanning happens automatically in oc-basic.

Regards,
-- 
Nicolas Goaziou

Re: re-scanning bibliography for org-cite

[Bruce D'Arcus]

On Wed, Nov 3, 2021 at 12:21 PM Nicolas Goaziou <mail@nicolasgoaziou.fr> wrote:

> oc-basic relies on a cache. The cache key contains a hash of the
> contents of the bib file. So whenever the bib file is modified, the
> cache is invalidated, and oc-basic parses again the file.

Of course; clever!

Bruce

Re: re-scanning bibliography for org-cite

[Eric S Fraga]

On Wednesday,  3 Nov 2021 at 17:21, Nicolas Goaziou wrote:
> oc-basic relies on a cache. The cache key contains a hash of the
> contents of the bib file. So whenever the bib file is modified, the
> cache is invalidated, and oc-basic parses again the file.
>
> IOW, rescanning happens automatically in oc-basic.

Hi Nicolas,

sorry to revisit this but I continue to run into the problem of the
bibliography not being rescanned automatically.  What is the easiest way
(other than leaving Emacs and restarting) to force it, assuming there is
one?  I.e. maybe I can invalidate or clear the cache?

I do not know why rescanning doesn't happen.  It usually does but then
sometimes it doesn't without any noticeable difference in the actions I
have taken.  Frustrating.

I hate restarting Emacs... ;-)

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.5.3-511-g8e69ad in Emacs 29.0.50

Re: re-scanning bibliography for org-cite

[Ihor Radchenko]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> sorry to revisit this but I continue to run into the problem of the
> bibliography not being rescanned automatically.  What is the easiest way
> (other than leaving Emacs and restarting) to force it, assuming there is
> one?  I.e. maybe I can invalidate or clear the cache?
>
> I do not know why rescanning doesn't happen.  It usually does but then
> sometimes it doesn't without any noticeable difference in the actions I
> have taken.  Frustrating.

In theory, it might be related to one of the changes I made in this
area. I am not 100% sure, but bibliography might not be rescanned if you
edit a bibliography file in Emacs buffer, but do not actually save it.
Is this what is happening?

Best,
Ihor

Re: re-scanning bibliography for org-cite

[Eric S Fraga]

On Thursday, 26 May 2022 at 13:00, Ihor Radchenko wrote:
> In theory, it might be related to one of the changes I made in this
> area. I am not 100% sure, but bibliography might not be rescanned if you
> edit a bibliography file in Emacs buffer, but do not actually save it.
> Is this what is happening?

No and, to be fair, I wouldn't expect the scanning to take place if I
hadn't saved the file.

I have my bibliography in an org file which I export to .bib using
org-bibtex.

Is there a function I can call to force a rescan?  That would satisfy me
as I could put it into a hook for org-bibtex (or advice the function if
no hook available).

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.5.3-472-gd2a459 in Emacs 29.0.50

Re: re-scanning bibliography for org-cite

[Ihor Radchenko]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Is there a function I can call to force a rescan?  That would satisfy me
> as I could put it into a hook for org-bibtex (or advice the function if
> no hook available).

The relevant function is org-cite-basic--parse-bibliography.

You can try 2 things:
1. Set org-cite-basic--file-id-cache to nil in :around advice before
   calling the function
2. Remove :cite-basic/bibliography from INFO plist in :around advice
   before calling the function

I you find any of these methods helpful, please report back. I have a
suspicion that :cite-basic/bibliography might be the cause.

Best,
Ihor

Re: re-scanning bibliography for org-cite

[Eric S Fraga]

Hi Ihor,

I will try these out, hopefully tomorrow.  The difficulty will be
knowing whether they work or not given the sporadic nature of the
problem...

Thank you,
eric

-- 
: Eric S Fraga, with org release_9.5.3-513-g292116 in Emacs 29.0.50

Re: re-scanning bibliography for org-cite

[Timothy]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

Hi Eric,

> The difficulty will be knowing whether they work or not given the sporadic
> nature of the problem…

If it’s any help, I consistently have this issue when producing the bib file by tangling.

All the best,
Timothy

Re: re-scanning bibliography for org-cite

[Ihor Radchenko]

Timothy <tecosaur@gmail.com> writes:

> Hi Eric,
>
>> The difficulty will be knowing whether they work or not given the sporadic
>> nature of the problem…
>
> If it’s any help, I consistently have this issue when producing the bib file by tangling.

Then, can you also try the proposed steps? If you can reproduce the
problem consistently, you can reveal what helps (or not) far quicker.

Best,
Ihor

Re: re-scanning bibliography for org-cite

[Fraga, Eric]

On Thursday, 26 May 2022 at 18:24, Ihor Radchenko wrote:
> The relevant function is org-cite-basic--parse-bibliography.

To finally follow up on this, the problem is not with org-cite but
appears to be due to my use of citar.  Citar seems to cache the keys
somewhere/somehow and this cache is not updated when the bibliography
file changes.  If I set org-cite-insert-processor to 'basic, everything
is fine.

Sorry for the noise.

-- 
: Eric S Fraga, with org release_9.5.4-521-g1105da in Emacs 29.0.50

Re: re-scanning bibliography for org-cite

[Bruce D'Arcus]

On Thu, Jun 9, 2022 at 10:19 AM Fraga, Eric <e.fraga@ucl.ac.uk> wrote:
>
> On Thursday, 26 May 2022 at 18:24, Ihor Radchenko wrote:
> > The relevant function is org-cite-basic--parse-bibliography.
>
> To finally follow up on this, the problem is not with org-cite but
> appears to be due to my use of citar.  Citar seems to cache the keys
> somewhere/somehow and this cache is not updated when the bibliography
> file changes.

Yes; not currently by default, but can be configured.

https://github.com/emacs-citar/citar#refreshing-the-library-display

But more importantly I'm currently working on a major change in the
caching and completion architecture of citar, where I've actually
borrowed ideas and code from oc-basic, so it should work the same WRT
to cache updating, without need for any configuration.

https://github.com/emacs-citar/citar/pull/628

It should take another few days of development and testing before I'm
ready to merge it, but it pretty much works now, and any help with
code review and/or testing would be much appreciated.

Bruce

Re: re-scanning bibliography for org-cite

[Fraga, Eric]

On Thursday,  9 Jun 2022 at 10:34, Bruce D'Arcus wrote:
> It should take another few days of development and testing before I'm
> ready to merge it, but it pretty much works now, and any help with
> code review and/or testing would be much appreciated.

Okay, let me know.  It's easy enough for me to replace 'basic with
'citar again!

-- 
: Eric S Fraga, with org release_9.5.4-521-g1105da in Emacs 29.0.50

Re: re-scanning bibliography for org-cite

[Bruce D'Arcus]

BTW, just to give some context of why we didn't implement it by
default earlier; there are two reasons:

1. it seems vast differences in the size of user bib files, and those
with very large ones would be annoyed if it re-generated the
completions on every change.
2. citar also covers notes and related files, which get complicated,
and subject to further user variety; and the UI needs to keep this in
sync with the bib data

I think we understand the problem better now, however.

On Thu, Jun 9, 2022 at 12:00 PM Fraga, Eric <e.fraga@ucl.ac.uk> wrote:
>
> On Thursday,  9 Jun 2022 at 10:34, Bruce D'Arcus wrote:
> > It should take another few days of development and testing before I'm
> > ready to merge it, but it pretty much works now, and any help with
> > code review and/or testing would be much appreciated.
>
> Okay, let me know.  It's easy enough for me to replace 'basic with
> 'citar again!
>
> --
> : Eric S Fraga, with org release_9.5.4-521-g1105da in Emacs 29.0.50

Re: re-scanning bibliography for org-cite

[Bruce D'Arcus]

On Thu, Jun 9, 2022 at 12:00 PM Fraga, Eric <e.fraga@ucl.ac.uk> wrote:
>
> On Thursday,  9 Jun 2022 at 10:34, Bruce D'Arcus wrote:
> > It should take another few days of development and testing before I'm
> > ready to merge it, but it pretty much works now, and any help with
> > code review and/or testing would be much appreciated.
>
> Okay, let me know.  It's easy enough for me to replace 'basic with
> 'citar again!

Took a bit longer than "days," because we ended up making a ton of
other changes, but it's now merged.

The result should be a snappy UI that appropriately adapts to local
buffer context, and updates the cache as needed.