emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* [wip-cite-new] Exporting to pandoc md (and from there to zotero odt)
@ 2021-06-23 19:56 Anders Johansson
  2021-06-23 20:02 ` Bruce D'Arcus
  0 siblings, 1 reply; 6+ messages in thread
From: Anders Johansson @ 2021-06-23 19:56 UTC (permalink / raw)
  To: emacs-orgmode

[-- Attachment #1: Type: text/plain, Size: 1956 bytes --]

Hi all,
I appreciate the work done on the cite functionality!
I use zotero for managing my library and have previously used zotxt
and org-zotxt (along with some additions in my zotxt-extra library)
for being able to cite in org-mode and export correctly to various
formats.
https://gitlab.com/egh/zotxt
https://gitlab.com/egh/zotxt-emacs
https://gitlab.com/andersjohansson/emacs-zotxt-extra

Zotxt has used a custom link format for citations (similar to org-ref)
which links to zotero but optionally also contains a citekey (defined
via the zotero-better-bibtex extension to zotero). I actually wrote my
PhD thesis with this setup, but the connection to Zotero has always
felt a little wobbly and Zotero is also quite slow for my large
library.

Now seems to be a great time to convert to a workflow with the new
cite format. To enable a conversion from org mode to word processors
with Zotero ”active” citations I created two libraries to manage this
via pandoc and the pandoc filter created for the zotero-better-bibtex
extension.
(see: https://retorque.re/zotero-better-bibtex/exporting/pandoc/)

This allows me a workflow like:
1. keep an updated  .bib-file of my library in zotero (with the help
of zotero-better-bibtex)
2. cite in org-mode (for me with completion via helm-bibtex and
https://github.com/bdarcus/bibtex-actions/pull/113)
3. Export to an odt-file with active zotero citations (with oc-pandoc
and ox-md-pandoc-zotero)

I attach them here in case anyone else finds some part of this workflow usable.
https://gitlab.com/andersjohansson/org-cite-pandoc
https://gitlab.com/andersjohansson/ox-md-pandoc-zotero

I couldn’t find any previous definitions for exporting to the pandoc
cite format so I quickly hacked up org-cite-pandoc. I suppose it could
be be widely useful and included in org. Feel free to use it as
suitable (I have FSF copyright assignment for emacs).

Best,
Anders Johansson

[-- Attachment #2: oc-pandoc.el --]
[-- Type: text/x-emacs-lisp, Size: 3363 bytes --]

;;; oc-pandoc.el --- Export org to pandoc markdown with citations  -*- lexical-binding: t; -*-

;; Copyright (C) 2021  Anders Johansson

;; Author: Anders Johansson <mejlaandersj@gmail.com>
;; Created: 2021-06-23
;; Modified: 2021-06-23
;; Keywords: org, wp

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;;

;;; Code:

(require 'oc)
(require 'ox)
(require 'org-element)

(defun org-cite-pandoc--get-key (ref)
  "Return @key from REF object."
  (concat "@" (org-element-property :key ref)))

(defun org-cite-pandoc--format-citation (citation noauthor info)
  "Format CITATION object according to pandoc format.
NOAUTHOR non-nil for noauthor style.
INFO is the export state, as a property list."
  (org-export-data
   (org-cite-concat "["
                    (when noauthor "-")
                    ;; no support for global prefix or suffix
                    ;; (org-element-property :prefix citation)
                    (org-cite-mapconcat
                     (lambda (ref)
                       (org-cite-concat
                        (org-element-property :prefix ref)
                        (org-cite-pandoc--get-key ref)
                        (when-let ((suf (org-element-property :suffix ref)))
                          (org-cite-concat "," suf))))
                     (org-cite-get-references citation)
                     ";")
                    ;; (org-element-property :suffix citation)
                    "]")
   info))

(defun org-cite-pandoc-export-citation (citation style _ info)
  "Export CITATION object.
STYLE is the expected citation style, as a pair of strings or nil.  INFO is the
export communication channel, as a property list."
  (pcase style
    (`(,(or "text" "t") . ,_)
     ;; This would generate multiple in-text citations like:
     ;; AuthorA (2020), Author B (2021)
     (mapconcat #'org-cite-pandoc--get-key (org-cite-get-references citation) ", "))
    ;; "noauthor" style.
    (`(,(or "noauthor" "na") . ,_)
     (org-cite-pandoc--format-citation citation t info))
    ;; Default ("nil") style.
    (`(,_ . ,_)
     (org-cite-pandoc--format-citation citation nil info))
    ;; This should not happen.
    (_ (error "Invalid style: %S" style))))

(defun org-cite-pandoc-export-bibliography (_k _f _s _p backend _i)
  "Generate bibliography.
Just outputs a #refs section when BACKEND is markdown."
  (when (org-export-derived-backend-p backend 'md)
    "::: {#refs}\n:::"))

\f
;;; Register processor
(org-cite-register-processor 'pandoc
  :export-citation #'org-cite-pandoc-export-citation
  :export-bibliography #'org-cite-pandoc-export-bibliography)

(provide 'org-cite-pandoc)
(provide 'oc-pandoc)
;;; oc-pandoc.el ends here

;; Local Variables:
;; nameless-current-name: "org-cite-pandoc"
;; End:

[-- Attachment #3: ox-md-pandoc-zotero.el --]
[-- Type: text/x-emacs-lisp, Size: 2747 bytes --]

;;; ox-md-pandoc-zotero.el --- Export org citations to md and then to zotero formats via pandoc  -*- lexical-binding: t; -*-

;; Copyright (C) 2021  Anders Johansson

;; Author: Anders Johansson <mejlaandersj@gmail.com>
;; Created: 2021-06-23
;; Modified: 2021-06-23
;; Keywords: org, wp

;; This program is free software; you can redistribute it and/or modify
;; it under the terms of the GNU General Public License as published by
;; the Free Software Foundation, either version 3 of the License, or
;; (at your option) any later version.

;; This program is distributed in the hope that it will be useful,
;; but WITHOUT ANY WARRANTY; without even the implied warranty of
;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
;; GNU General Public License for more details.

;; You should have received a copy of the GNU General Public License
;; along with this program.  If not, see <https://www.gnu.org/licenses/>.

;;; Commentary:

;;

;;; Code:

(require 'ox-md)

(defgroup ox-md-pandoc-zotero nil
  "Customization for org export to zotero via pandoc md."
  :group 'org-export)

(defcustom ox-md-pandoc-path-to-bbt-filter ""
  "Path to the betterbibtex pandoc filter for converting md to zotero odt/word.

Latest file should be here:
https://raw.githubusercontent.com/retorquere/zotero-better-bibtex/master/site/content/exporting/zotero.lua"
  :type 'file)

(defcustom ox-md-pandoc-format "odt"
  "Final export format. odt or docx."
  :type '(choice
          (const "odt")
          (const "docx")))

(defcustom ox-md-pandoc-style "apa"
  "File to set in zotero output file. A valid csl style."
  :type 'string)

(org-export-define-derived-backend 'md-pandoc-zotero 'md
  :menu-entry
  '(?p "MD→pandoc zotero export"
       ((?z "As zotero odt" ox-md-pandoc-zotero-odt))))

(defun ox-md-pandoc-zotero-odt (async subtreep visible-only body-only)
  "Export to md and then to odt with pandoc and zotero-better-bibtex filter."
  (interactive)
  (let ((outfile (org-export-output-file-name ".md" subtreep)))
    (org-export-to-file 'md outfile async subtreep visible-only body-only nil
                        #'ox-md-pandoc-process-zotero-wpfile)))

(defun ox-md-pandoc-process-zotero-wpfile (file)
  "Process markdown FILE to zotero odt or docx."
  (org-open-file
   (org-compile-file
    file
    (list
     (format
      "pandoc -s -o %%b.%s --lua-filter=%s --metadata=zotero_author_in_text=true --metadata=zotero_csl_style=%s %%f"
      ox-md-pandoc-format
      ox-md-pandoc-path-to-bbt-filter
      ox-md-pandoc-style))
    ox-md-pandoc-format
    "See *ox-md-pandoc-zotero output* for details"
    (get-buffer-create "*ox-md-pandoc-zotero output*"))))


(provide 'ox-md-pandoc-zotero)
;;; ox-md-pandoc-zotero.el ends here

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [wip-cite-new] Exporting to pandoc md (and from there to zotero odt)
  2021-06-23 19:56 [wip-cite-new] Exporting to pandoc md (and from there to zotero odt) Anders Johansson
@ 2021-06-23 20:02 ` Bruce D'Arcus
  2021-06-29 18:29   ` Matt Price
  0 siblings, 1 reply; 6+ messages in thread
From: Bruce D'Arcus @ 2021-06-23 20:02 UTC (permalink / raw)
  To: Anders Johansson, org-mode-email

[-- Attachment #1: Type: text/plain, Size: 2247 bytes --]

Cool!

We should also get pandoc to support updated org-cite syntax.

https://github.com/jgm/pandoc/issues/7329


On Wed, Jun 23, 2021, 12:57 PM Anders Johansson <mejlaandersj@gmail.com>
wrote:

> Hi all,
> I appreciate the work done on the cite functionality!
> I use zotero for managing my library and have previously used zotxt
> and org-zotxt (along with some additions in my zotxt-extra library)
> for being able to cite in org-mode and export correctly to various
> formats.
> https://gitlab.com/egh/zotxt
> https://gitlab.com/egh/zotxt-emacs
> https://gitlab.com/andersjohansson/emacs-zotxt-extra
>
> Zotxt has used a custom link format for citations (similar to org-ref)
> which links to zotero but optionally also contains a citekey (defined
> via the zotero-better-bibtex extension to zotero). I actually wrote my
> PhD thesis with this setup, but the connection to Zotero has always
> felt a little wobbly and Zotero is also quite slow for my large
> library.
>
> Now seems to be a great time to convert to a workflow with the new
> cite format. To enable a conversion from org mode to word processors
> with Zotero ”active” citations I created two libraries to manage this
> via pandoc and the pandoc filter created for the zotero-better-bibtex
> extension.
> (see: https://retorque.re/zotero-better-bibtex/exporting/pandoc/)
>
> This allows me a workflow like:
> 1. keep an updated  .bib-file of my library in zotero (with the help
> of zotero-better-bibtex)
> 2. cite in org-mode (for me with completion via helm-bibtex and
> https://github.com/bdarcus/bibtex-actions/pull/113)
> 3. Export to an odt-file with active zotero citations (with oc-pandoc
> and ox-md-pandoc-zotero)
>
> I attach them here in case anyone else finds some part of this workflow
> usable.
> https://gitlab.com/andersjohansson/org-cite-pandoc
> https://gitlab.com/andersjohansson/ox-md-pandoc-zotero
>
> I couldn’t find any previous definitions for exporting to the pandoc
> cite format so I quickly hacked up org-cite-pandoc. I suppose it could
> be be widely useful and included in org. Feel free to use it as
> suitable (I have FSF copyright assignment for emacs).
>
> Best,
> Anders Johansson
>

[-- Attachment #2: Type: text/html, Size: 3504 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [wip-cite-new] Exporting to pandoc md (and from there to zotero odt)
  2021-06-23 20:02 ` Bruce D'Arcus
@ 2021-06-29 18:29   ` Matt Price
  2021-07-02 15:49     ` Matt Price
  0 siblings, 1 reply; 6+ messages in thread
From: Matt Price @ 2021-06-29 18:29 UTC (permalink / raw)
  To: Bruce D'Arcus; +Cc: org-mode-email, Anders Johansson

[-- Attachment #1: Type: text/plain, Size: 2658 bytes --]

Anders, just to be clear:  with your new exporters I can, as of today,
insert the new [cite:@citekey] links using a better-bibtex-generated
bibliography file, export to odt via pandoc, and then manage my citations
with zotero in the resultant odt?

it so... cool.

On Wed, Jun 23, 2021 at 4:04 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:

> Cool!
>
> We should also get pandoc to support updated org-cite syntax.
>
> https://github.com/jgm/pandoc/issues/7329
>
>
> On Wed, Jun 23, 2021, 12:57 PM Anders Johansson <mejlaandersj@gmail.com>
> wrote:
>
>> Hi all,
>> I appreciate the work done on the cite functionality!
>> I use zotero for managing my library and have previously used zotxt
>> and org-zotxt (along with some additions in my zotxt-extra library)
>> for being able to cite in org-mode and export correctly to various
>> formats.
>> https://gitlab.com/egh/zotxt
>> https://gitlab.com/egh/zotxt-emacs
>> https://gitlab.com/andersjohansson/emacs-zotxt-extra
>>
>> Zotxt has used a custom link format for citations (similar to org-ref)
>> which links to zotero but optionally also contains a citekey (defined
>> via the zotero-better-bibtex extension to zotero). I actually wrote my
>> PhD thesis with this setup, but the connection to Zotero has always
>> felt a little wobbly and Zotero is also quite slow for my large
>> library.
>>
>> Now seems to be a great time to convert to a workflow with the new
>> cite format. To enable a conversion from org mode to word processors
>> with Zotero ”active” citations I created two libraries to manage this
>> via pandoc and the pandoc filter created for the zotero-better-bibtex
>> extension.
>> (see: https://retorque.re/zotero-better-bibtex/exporting/pandoc/)
>>
>> This allows me a workflow like:
>> 1. keep an updated  .bib-file of my library in zotero (with the help
>> of zotero-better-bibtex)
>> 2. cite in org-mode (for me with completion via helm-bibtex and
>> https://github.com/bdarcus/bibtex-actions/pull/113)
>> 3. Export to an odt-file with active zotero citations (with oc-pandoc
>> and ox-md-pandoc-zotero)
>>
>> I attach them here in case anyone else finds some part of this workflow
>> usable.
>> https://gitlab.com/andersjohansson/org-cite-pandoc
>> https://gitlab.com/andersjohansson/ox-md-pandoc-zotero
>>
>> I couldn’t find any previous definitions for exporting to the pandoc
>> cite format so I quickly hacked up org-cite-pandoc. I suppose it could
>> be be widely useful and included in org. Feel free to use it as
>> suitable (I have FSF copyright assignment for emacs).
>>
>> Best,
>> Anders Johansson
>>
>

[-- Attachment #2: Type: text/html, Size: 4218 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [wip-cite-new] Exporting to pandoc md (and from there to zotero odt)
  2021-06-29 18:29   ` Matt Price
@ 2021-07-02 15:49     ` Matt Price
  2021-07-02 22:10       ` Anders Johansson
  0 siblings, 1 reply; 6+ messages in thread
From: Matt Price @ 2021-07-02 15:49 UTC (permalink / raw)
  To: Bruce D'Arcus; +Cc: org-mode-email, Anders Johansson

[-- Attachment #1: Type: text/plain, Size: 3095 bytes --]

Anders, I just want to say that I tested this out and it works great. It's
quite shocking, actually, to have such an easy path from org to odt with
live citations.  I'd love to see this stuff in MELPA or otherwise made more
generally accessible, as i think it fills a substantial need.

On Tue, Jun 29, 2021 at 2:29 PM Matt Price <moptop99@gmail.com> wrote:

> Anders, just to be clear:  with your new exporters I can, as of today,
> insert the new [cite:@citekey] links using a better-bibtex-generated
> bibliography file, export to odt via pandoc, and then manage my citations
> with zotero in the resultant odt?
>
> it so... cool.
>
> On Wed, Jun 23, 2021 at 4:04 PM Bruce D'Arcus <bdarcus@gmail.com> wrote:
>
>> Cool!
>>
>> We should also get pandoc to support updated org-cite syntax.
>>
>> https://github.com/jgm/pandoc/issues/7329
>>
>>
>> On Wed, Jun 23, 2021, 12:57 PM Anders Johansson <mejlaandersj@gmail.com>
>> wrote:
>>
>>> Hi all,
>>> I appreciate the work done on the cite functionality!
>>> I use zotero for managing my library and have previously used zotxt
>>> and org-zotxt (along with some additions in my zotxt-extra library)
>>> for being able to cite in org-mode and export correctly to various
>>> formats.
>>> https://gitlab.com/egh/zotxt
>>> https://gitlab.com/egh/zotxt-emacs
>>> https://gitlab.com/andersjohansson/emacs-zotxt-extra
>>>
>>> Zotxt has used a custom link format for citations (similar to org-ref)
>>> which links to zotero but optionally also contains a citekey (defined
>>> via the zotero-better-bibtex extension to zotero). I actually wrote my
>>> PhD thesis with this setup, but the connection to Zotero has always
>>> felt a little wobbly and Zotero is also quite slow for my large
>>> library.
>>>
>>> Now seems to be a great time to convert to a workflow with the new
>>> cite format. To enable a conversion from org mode to word processors
>>> with Zotero ”active” citations I created two libraries to manage this
>>> via pandoc and the pandoc filter created for the zotero-better-bibtex
>>> extension.
>>> (see: https://retorque.re/zotero-better-bibtex/exporting/pandoc/)
>>>
>>> This allows me a workflow like:
>>> 1. keep an updated  .bib-file of my library in zotero (with the help
>>> of zotero-better-bibtex)
>>> 2. cite in org-mode (for me with completion via helm-bibtex and
>>> https://github.com/bdarcus/bibtex-actions/pull/113)
>>> 3. Export to an odt-file with active zotero citations (with oc-pandoc
>>> and ox-md-pandoc-zotero)
>>>
>>> I attach them here in case anyone else finds some part of this workflow
>>> usable.
>>> https://gitlab.com/andersjohansson/org-cite-pandoc
>>> https://gitlab.com/andersjohansson/ox-md-pandoc-zotero
>>>
>>> I couldn’t find any previous definitions for exporting to the pandoc
>>> cite format so I quickly hacked up org-cite-pandoc. I suppose it could
>>> be be widely useful and included in org. Feel free to use it as
>>> suitable (I have FSF copyright assignment for emacs).
>>>
>>> Best,
>>> Anders Johansson
>>>
>>

[-- Attachment #2: Type: text/html, Size: 4885 bytes --]

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [wip-cite-new] Exporting to pandoc md (and from there to zotero odt)
  2021-07-02 15:49     ` Matt Price
@ 2021-07-02 22:10       ` Anders Johansson
  2022-03-10 19:50         ` Anders Johansson
  0 siblings, 1 reply; 6+ messages in thread
From: Anders Johansson @ 2021-07-02 22:10 UTC (permalink / raw)
  To: Matt Price; +Cc: org-mode-email, Bruce D'Arcus

> Anders, I just want to say that I tested this out and it works great. It's quite shocking, actually, to have such an easy path from org to odt with live citations.  I'd love to see this stuff in MELPA or otherwise made more generally accessible, as i think it fills a substantial need.

Hi Matt,
Great that it works for you! I have also been testing it out a bit
over the last few days (along with transferring from helm to the newer
completion packages, spurred by testing the nice completion given by
bibtex-actions. Thanks Bruce).

I do miss some of the possibilities of customizing the odt export
(easier when exporting via ox-odt). In particular having numbered
sections, which doesn't seem to be supported for odt exports from
pandoc (https://pandoc.org/MANUAL.html#option--number-sections). I
just added a custom variable for passing additional parameters to
pandoc in ox-md-pandoc-zotero (at
https://gitlab.com/andersjohansson/ox-md-pandoc-zotero), but
apparently, --number-sections didn't make a difference for odt.


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [wip-cite-new] Exporting to pandoc md (and from there to zotero odt)
  2021-07-02 22:10       ` Anders Johansson
@ 2022-03-10 19:50         ` Anders Johansson
  0 siblings, 0 replies; 6+ messages in thread
From: Anders Johansson @ 2022-03-10 19:50 UTC (permalink / raw)
  To: Matt Price; +Cc: org-mode-email, Bruce D'Arcus

Hi,
>We should also get pandoc to support updated org-cite syntax.
> https://github.com/jgm/pandoc/issues/7329

Now that pandoc has org-cite support, I found it simpler to reuse
ox-pandoc (https://github.com/emacsorphanage/ox-pandoc) and just do
this very simple configuration. ox-pandoc exports to an intermediary
org file, so we can just make sure to reinsert citations in org
format.

(use-package ox-pandoc
  :custom (org-pandoc-options-for-odt
           '((lua-filter . "/home/aj/lib/bbt-pandoc-zotero.lua")
             (metadata . "zotero_author_in_text=true"))))

(org-cite-register-processor 'pandoc
  :export-citation #'oc-pandoc-export-citation
  :export-bibliography #'oc-pandoc-export-bibliography)

(defun oc-pandoc-export-citation (citation _style _ _info)
  "Export CITATION object in org format."
  (org-element-citation-interpreter
   citation
   (mapconcat (lambda (r) (org-element-citation-reference-interpreter r nil))
              (org-cite-get-references citation))))

(defun oc-pandoc-export-bibliography (_k _f _s _p _b _i)
  "Generate a #+print_bibliography:"
  "#+print_bibliography:")


^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2022-03-10 19:51 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-23 19:56 [wip-cite-new] Exporting to pandoc md (and from there to zotero odt) Anders Johansson
2021-06-23 20:02 ` Bruce D'Arcus
2021-06-29 18:29   ` Matt Price
2021-07-02 15:49     ` Matt Price
2021-07-02 22:10       ` Anders Johansson
2022-03-10 19:50         ` Anders Johansson

Code repositories for project(s) associated with this public inbox

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).