You can try something like this

#+BEGIN_SRC emacs-lisp
(setq bibtex-completion-format-citation-functions
      '((org-mode      . (lambda (keys) (org-ref-insert-cite-keys keys) ""))
        (latex-mode    . bibtex-completion-format-citation-cite)
        (markdown-mode . bibtex-completion-format-citation-pandoc-citeproc)
        (default       . bibtex-completion-format-citation-default)))
#+END_SRC

it seems like ivy-bibtex at least expects some kind of string to be returned, but org-ref-insert-cite-keys does the insertion for you.

On Sun, Feb 26, 2023 at 11:19 AM Sven Bretfeld <sven.bretfeld@ntnu.no> wrote:
Hi everybody

Is it at all possible to produce a correct org-ref citation format
from ivy-bibtex (or helm-bibtex, I tried both)? What I mean by
"correct" is the format: [[cite:&citekey]].

The problem seems to come down to the question what has to be inserted
in the 2nd line here:

(setq bibtex-completion-format-citation-functions
      '((org-mode      . ??????)
        (latex-mode    . bibtex-completion-format-citation-cite)
        (markdown-mode . bibtex-completion-format-citation-pandoc-citeproc)
        (default       . bibtex-completion-format-citation-default)))

(A) Not setting the variable at all, defaults to producing an ebib
link which is of no use to me at all.

(B) Putting org-ref-helm-bibtex-insert-citation here (as found on
reddit) does not work, since the function does not seem to exist
anymore.

(C) Putting org-ref-insert-cite-link here is obviously wrong. Choosing
that from the list of ivy-bibtex-actions, just calls the function
interactively, you have to search for the bib-entry again and the
resulting link looks like: [[cite:&citekey]]^&. The terminating '^&'
obviously makes subsequent LaTeX-exports fall.

(D) Putting "bibtex-completion-format-citation-org-cite" here,
produces standard org-links: [cite:@citekey]. These are not correctly
exported to a tex file when the org-ref package is loaded.

The only working solution I have found is to use org-ref's own cite
function org-ref-insert-cite-link. I know that ivy-bibtex can be set
as backend (require 'org-ref-ivy). But in this way ivy-bibtex's other
actions (like opening an associated PDF or note file) are lost.

It would be nice to be able to use a SINGLE key-binding to select a
bib-entry and then decide what to do with it (insert org reference,
insert tex reference, open pdf, etc.). This was possible earlier (I
did that for years with older versions of org-ref-helm) but is
apparently no longer possible now.

Many thanks for help and suggestions,

Sven