I can confirm that the key turns red on insert when I altered the key outside of emacs (with that second version of `org-cite-basic--parse-bibliography`).

However, I'm now noticing that the hang improvement earlier (< a second down from ~10) doesn't always occur.

Specifically, if I
  1. emacs -Q
  2. eval your code in scratch
  3. C-x C-f to the org file
I get the hang. However, if I then

  1. kill the org buffer
  2. eval the code again
  3. re-find the org file
the hang is gone. Without eval​ing your code in between, killing the org buffer and finding it again in the same emacs session reproduces the hang everytime, which was probably what I did the first time I report the improvement, as in I didn't check it worked from startup.



From: Ihor Radchenko <yantar92@gmail.com>
Sent: Saturday, March 19, 2022 10:28 AM
To: Jamie Matthews <jdm204@cam.ac.uk>
Cc: emacs-orgmode@gnu.org <emacs-orgmode@gnu.org>
Subject: Re: [BUG] org-cite: 10 second hang opening a ~4k org file with 10MB bibtex library [9.5.2 (9.5.2-g91681f @ /home/jdm204/.config/emacs/straight/build/org/)]
 
Jamie Matthews <jdm204@cam.ac.uk> writes:

> It seems a very significant improvement - the lag on scrolling after the buffer is loaded is not noticeable, and the initial loading hang essentially disappears!

Great! Do you mean that there is no apparent slowness at all?

> I'm slightly surprised that elp-results​ doesn't seem to show as dramatic an improvement as it feels for me:
>
> ```
> org-cite-basic-activate             123         6.5652978389  0.0533764051
> org-cite-basic--all-keys            123         6.2679872230  0.0509592457
> org-cite-basic--parse-bibliography  270         6.2513217860  0.0231530436
> org-cite-basic--get-entry           147         0.1550045990  0.0010544530
> ```

It does. You need to look at the last column. The total time in the
second column is large probably because you interacted (scrolled?) with
the buffer a bit more than first time and the functions got called more
(see second column).

Also, can you try the following version of
org-cite-basic--parse-bibliography again and try to change cite key of
one of the cited entries of the bibliography outside emacs? The key
should be marked red if you try to insert the changed entry in Org.

(defvar org-cite-basic--file-id-cache nil
  "Hash table linking files to their hash.")
(defun org-cite-basic--parse-bibliography (&optional info)
  "List all entries available in the buffer.

Each association follows the pattern

  (FILE . ENTRIES)

where FILE is the absolute file name of the BibTeX file, and ENTRIES is a hash
table where keys are references and values are association lists between fields,
as symbols, and values as strings or nil.

Optional argument INFO is the export state, as a property list."
  (unless (hash-table-p org-cite-basic--file-id-cache)
    (setq org-cite-basic--file-id-cache (make-hash-table :test #'equal)))
  (if (plist-member info :cite-basic/bibliography)
      (plist-get info :cite-basic/bibliography)
    (let ((results nil))
      (dolist (file (org-cite-list-bibliography-files))
        (when (file-readable-p file)
          (with-temp-buffer
            (when (or (file-has-changed-p file)
                      (not (gethash file org-cite-basic--file-id-cache)))
              (insert-file-contents file)
              (puthash file (org-buffer-hash) org-cite-basic--file-id-cache))
            (let* ((file-id (cons file (gethash file org-cite-basic--file-id-cache)))
                   (entries
                    (or (cdr (assoc file-id org-cite-basic--bibliography-cache))
                        (let ((table
                               (pcase (file-name-extension file)
                                 ("json" (org-cite-basic--parse-json))
                                 ("bib" (org-cite-basic--parse-bibtex 'biblatex))
                                 ("bibtex" (org-cite-basic--parse-bibtex 'BibTeX))
                                 (ext
                                  (user-error "Unknown bibliography extension: %S"
                                              ext)))))
                          (push (cons file-id table) org-cite-basic--bibliography-cache)
                          table))))
              (push (cons file entries) results)))))
      (when info (plist-put info :cite-basic/bibliography results))
      results)))

Best,
Ihor