Re: [PATCH] org-bibtex.el --- convert between Org headings and bibtex entries

["Eric Schulte" <schulte.eric@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 1258 bytes --]

Matt Lundin <mdl@imapmail.org> writes:

> Hi Eric,
>
> "Eric Schulte" <schulte.eric@gmail.com> writes:
>
>> In an attempt to organize my reading notes, I've written the following
>> tool which allows both for exporting Org-mode headlines with bibtex
>> meta-data to bibtex entries, and for reading existing bibtex entries
>> into Org-mode headings.
>>
>> One nice feature of these functions is the ability to check that all
>> required fields are present in a given headline based on the bibtex type
>> (e.g., :article, :inproceedings), and prompt for missing fields.
>>
>> See the top of the elisp file for more usage information.
>> https://github.com/eschulte/org-bibtex/blob/master/org-bibtex.el
>
> Thanks for announcing this! One note: I believe the name conflicts with
> a core org-module (org-bibtex.el), which is used to store and open
> bibtex links.
>

As one possible solution to this name conflict, I've folded my
org-bibtex.el into the existing org-bibtex.el in the Org-mode core.  The
attached patch performs this integration adding the functionality in my
version of org-bibtex.el into the Org-mode core.

Does this seem like an appropriate addition?

Thanks -- Eric

(p.s. created with "git format-patch" and can be applied using "git am")


[-- Attachment #2: 0001-org-bibtex-translating-between-Org-mode-headings-and.patch --]
[-- Type: text/x-diff, Size: 20239 bytes --]

From d0ec36c11152c342584d5a2145871ebb2b195127 Mon Sep 17 00:00:00 2001
From: Eric Schulte <schulte.eric@gmail.com>
Date: Fri, 22 Apr 2011 06:19:30 -0600
Subject: [PATCH] org-bibtex: translating between Org-mode headings and Bibtex entries

* lisp/org-bibtex.el: Updating Copyright dates, author information,
  commentary and history notes.
  (org-bibtex-types): List of bibtex types with descriptions and
  required and optional fields.
  (org-bibtex-fields): List of bibtex fields with descriptions.
  (*org-bibtex-entries*): Special variable to hold parsed bibtex
  entries.
  (org-bibtex-autogen-keys): Custom variable controlling whether
  bibtex keys are automatically generated
  (org-bibtex-prefix): Custom variable allowing use of optional prefix
  for bibtex properties in Org-mode headlines.
  (org-bibtex-get): Helper function for accessing bibtex elements of a
  property list.
  (org-bibtex-put): Helper function for inserting bibtex element into
  a property list.
  (org-bibtex-headline): Return a bibtex entry of the given headline
  as a string.
  (org-bibtex-ask): Prompt the user to fill in the value of a bibtex
  field.
  (org-bibtex-autokey): Generate a bibtex key for the current
  headline.
  (org-bibtex-fleshout): Fill in missing bibtex properties of the
  current headline.
  (org-bibtex): Export the current Org-mode buffer to a bibtex buffer.
  (org-bibtex-check): Check that all bibtex properties are present in
  the current headline.
  (org-bibtex-check-all): Check all headlines in the current buffer.
  (org-bibtex-create): Create a new bibtex headline at the current
  level.
  (org-bibtex-read): Read the current bibtex entry from a bibtex file.
  (org-bibtex-write): Write the most recently read bibtex entry into
  an Org-mode file.
---
 lisp/org-bibtex.el |  317 +++++++++++++++++++++++++++++++++++++++++++++++++--
 1 files changed, 304 insertions(+), 13 deletions(-)

diff --git a/lisp/org-bibtex.el b/lisp/org-bibtex.el
index b7b7416..5667a4d 100644
--- a/lisp/org-bibtex.el
+++ b/lisp/org-bibtex.el
@@ -1,9 +1,10 @@
 ;;; org-bibtex.el --- Org links to BibTeX entries
 ;;
-;; Copyright (C) 2007, 2008, 2009, 2010 Free Software Foundation, Inc.
+;; Copyright (C) 2007, 2008, 2009, 2010, 2011 Free Software Foundation, Inc.
 ;;
 ;; Author: Bastien Guerry <bzg at altern dot org>
 ;;         Carsten Dominik <carsten dot dominik at gmail dot com>
+;;         Eric Schulte <schulte dot eric at gmail dot com>
 ;; Keywords: org, wp, remember
 ;; Version: 7.5
 ;;
@@ -66,12 +67,30 @@
 ;; =====================================================================
 ;; * READ <== [point here]
 ;;
-;; [[file:/file.bib::dolev83][Dolev & Yao 1983: security of public key protocols]]
+;; [[file:file.bib::dolev83][Dolev & Yao 1983: security of public key protocols]]
 ;;
 ;; Danny Dolev and Andrew C. Yao (1983): On the security of public-key protocols
 ;; In IEEE Transaction on Information Theory, 198--208.
 ;; =====================================================================
 ;;
+;; Additionally, the following functions are now available for storing
+;; bibtex entries within Org-mode documents.
+;; 
+;; - Run `org-bibtex' to export the current file to a .bib.
+;; 
+;; - Run `org-bibtex-check' or `org-bibtex-check-all' to check and
+;;   fill in missing field of either the current, or all headlines
+;;   
+;; - Run `org-bibtex-create' to add a bibtex entry
+;; 
+;; - Use `org-bibtex-read' to read a bibtex entry after `point' or in
+;;   the active region, then call `org-bibtex-write' in a .org file to
+;;   insert a heading for the read bibtex entry
+;;   
+;; - All Bibtex information is taken from the document compiled by
+;;   Andrew Roberts from the Bibtex manual, available at
+;;   http://www.andy-roberts.net/misc/latex/sessions/bibtex/bibentries.pdf
+;;
 ;;; History:
 ;;
 ;; The link creation part has been part of Org-mode for a long time.
@@ -80,12 +99,17 @@
 ;; of Austin Frank: http://article.gmane.org/gmane.emacs.orgmode/4112
 ;; and then implemented by Bastien Guerry.
 ;;
+;; Eric Schulte eventually added the functions for translating between
+;; Org-mode headlines and Bibtex entries, and for fleshing out the Bibtex
+;; fields of existing Org-mode headlines.
+;;
 ;; Org-mode loads this module by default - if this is not what you want,
 ;; configure the variable `org-modules'.
 
 ;;; Code:
 
 (require 'org)
+(require 'bibtex)
 
 (defvar description nil) ; dynamically scoped from org.el
 
@@ -94,20 +118,206 @@
 (declare-function bibtex-parse-entry "bibtex" (&optional content))
 (declare-function bibtex-url "bibtex" (&optional pos no-browse))
 
+\f
+;;; Bibtex data
+(defvar org-bibtex-types
+  '((:article
+     (:description . "An article from a journal or magazine")
+     (:required :author :title :journal :year)
+     (:optional :volume :number :pages :month :note))
+    (:book
+     (:description . "A book with an explicit publisher")
+     (:required (:editor :author) :title :publisher :year)
+     (:optional (:volume :number) :series :address :edition :month :note))
+    (:booklet
+     (:description . "A work that is printed and bound, but without a named publisher or sponsoring institution.")
+     (:required :title)
+     (:optional :author :howpublished :address :month :year :note))
+    (:conference
+     (:description . "")
+     (:required :author :title :booktitle :year)
+     (:optional :editor :pages :organization :publisher :address :month :note))
+    (:inbook
+     (:description . "A part of a book, which may be a chapter (or section or whatever) and/or a range of pages.")
+     (:required (:author :editor) :title (:chapter :pages) :publisher :year)
+     (:optional (:volume :number) :series :type :address :edition :month :note))
+    (:incollection
+     (:description . "A part of a book having its own title.")
+     (:required :author :title :booktitle :publisher :year)
+     (:optional :editor (:volume :number) :series :type :chapter :pages :address :edition :month :note))
+    (:inproceedings
+     (:description . "An article in a conference proceedings")
+     (:required :author :title :booktitle :year)
+     (:optional :editor (:volume :number) :series :pages :address :month :organization :publisher :note))
+    (:manual
+     (:description . "Technical documentation.")
+     (:required :title)
+     (:optional :author :organization :address :edition :month :year :note))
+    (:mastersthesis
+     (:description . "A Master’s thesis.")
+     (:required :author :title :school :year)
+     (:optional :type :address :month :note))
+    (:misc
+     (:description . "Use this type when nothing else fits.")
+     (:required)
+     (:optional :author :title :howpublished :month :year :note))
+    (:phdthesis
+     (:description . "A PhD thesis.")
+     (:required :author :title :school :year)
+     (:optional :type :address :month :note))
+    (:proceedings
+     (:description . "The proceedings of a conference.")
+     (:required :title :year)
+     (:optional :editor (:volume :number) :series :address :month :organization :publisher :note))
+    (:techreport
+     (:description . "A report published by a school or other institution.")
+     (:required :author :title :institution :year)
+     (:optional :type :address :month :note))
+    (:unpublished
+     (:description . "A document having an author and title, but not formally published.")
+     (:required :author :title :note)
+     (:optional :month :year)))
+  "Bibtex entry types with required and optional parameters.")
+
+(defvar org-bibtex-fields
+  '((:address      . "Usually the address of the publisher or other type of institution. For major publishing houses, van Leunen recommends omitting the information entirely.  For small publishers, on the other hand, you can help the reader by giving the complete address.")
+    (:annote       . "An annotation. It is not used by the standard bibliography styles, but may be used by others that produce an annotated bibliography.")
+    (:author       . "The name(s) of the author(s), in the format described in the LaTeX book.  Remember, all names are separated with the and keyword, and not commas.")
+    (:booktitle    . "Title of a book, part of which is being cited. See the LaTeX book for how to type titles. For book entries, use the title field instead.")
+    (:chapter      . "A chapter (or section or whatever) number.")
+    (:crossref     . "The database key of the entry being cross referenced.")
+    (:edition      . "The edition of a book for example, 'Second'. This should be an ordinal, and should have the first letter capitalized, as shown here; the standard styles convert to lower case when necessary.")
+    (:editor       . "Name(s) of editor(s), typed as indicated in the LaTeX book. If there is also an author field, then the editor field gives the editor of the book or collection in which the reference appears.")
+    (:howpublished . "How something strange has been published. The first word should be capitalized.")
+    (:institution  . "The sponsoring institution of a technical report.")
+    (:journal      . "A journal name.")
+    (:key          . "Used for alphabetizing, cross-referencing, and creating a label when the author information is missing. This field should not be confused with the key that appears in the \cite command and at the beginning of the database entry.")
+    (:month        . "The month in which the work was published or, for an unpublished work, in which it was written. You should use the standard three-letter abbreviation,")
+    (:note         . "Any additional information that can help the reader. The first word should be capitalized.")
+    (:number       . "Any additional information that can help the reader. The first word should be capitalized.")
+    (:organization . "The organization that sponsors a conference or that publishes a manual.")
+    (:pages        . "One or more page numbers or range of numbers, such as 42-111 or 7,41,73-97 or 43+ (the ‘+’ in this last example indicates pages following that don’t form simple range). BibTEX requires double dashes for page ranges (--).")
+    (:publisher    . "The publisher’s name.")
+    (:school       . "The name of the school where a thesis was written.")
+    (:series       . "The name of a series or set of books. When citing an entire book, the the title field gives its title and an optional series field gives the name of a series or multi-volume set in which the book is published.")
+    (:title        . "The work’s title, typed as explained in the LaTeX book.")
+    (:type         . "The type of a technical report for example, 'Research Note'.")
+    (:volume       . "The volume of a journal or multi-volume book.")
+    (:year         . "The year of publication or, for an unpublished work, the year it was written.  Generally it should consist of four numerals, such as 1984, although the standard styles can handle any year whose last four nonpunctuation characters are numerals, such as '(about 1984)'"))
+  "Bibtex fields with descriptions.")
+
+(defvar *org-bibtex-entries* nil
+  "List to hold parsed bibtex entries.")
+
+(defcustom org-bibtex-autogen-keys nil
+  "Set to a truthy value to use `bibtex-generate-autokey' to generate keys."
+  :group 'org-bibtex
+  :type  'boolean)
+
+(defcustom org-bibtex-prefix nil
+  "Optional prefix for all bibtex property names.
+For example setting to 'BIB_' would allow interoperability with fireforg."
+  :group 'org-bibtex
+  :type  'string)
+
+\f
+;;; Utility functions
+(defun org-bibtex-get (property)
+  (or (org-entry-get (point) (upcase property))
+      (org-entry-get (point) (concat org-bibtex-prefix (upcase property)))))
+
+(defun org-bibtex-put (property value)
+  (let ((prop (upcase (if (keywordp property)
+                          (substring (symbol-name property) 1)
+                        property))))
+    (org-set-property
+     (concat (unless (string= "CUSTOM_ID" prop) org-bibtex-prefix) prop)
+     value)))
+
+(defun org-bibtex-headline ()
+  "Return a bibtex entry of the given headline as a string."
+  (flet ((get (key lst) (cdr (assoc key lst)))
+         (to-k (string) (intern (concat ":" string)))
+         (from-k (key) (substring (symbol-name key) 1))
+         (flatten (&rest lsts)
+                  (apply #'append (mapcar
+                                   (lambda (e)
+                                     (if (listp e) (apply #'flatten e) (list e)))
+                                   lsts))))
+    (let ((notes (buffer-string))
+          (id (org-bibtex-get "custom_id"))
+          (type (org-bibtex-get "type")))
+      (when type
+        (let ((entry (format
+                      "@%s{%s,\n%s\n}\n" type id
+                      (mapconcat
+                       (lambda (pair) (format "  %s={%s}" (car pair) (cdr pair)))
+                       (remove nil
+			 (mapcar
+			  (lambda (field)
+			    (let ((value (or (org-bibtex-get (from-k field))
+					     (and (equal :title field)
+						  (org-get-heading)))))
+			      (when value (cons (from-k field) value))))
+			  (flatten
+			   (get :required (get (to-k type) org-bibtex-types))
+			   (get :optional (get (to-k type) org-bibtex-types)))))
+                       ",\n"))))
+          (with-temp-buffer
+            (insert entry)
+            (bibtex-reformat) (buffer-string)))))))
+
+(defun org-bibtex-ask (field)
+  (unless (assoc field org-bibtex-fields)
+    (error "field:%s is not known" field))
+  (save-window-excursion
+    (with-temp-buffer
+      (setf (buffer-name) (format "*Bibtex Help %s*" field))
+      (insert (cdr (assoc field org-bibtex-fields)))
+      (fill-paragraph)
+      (pop-to-buffer (current-buffer))
+      ((lambda (result) (when (> (length result) 0) result))
+       (read-from-minibuffer (format "%s " field))))))
+
+(defun org-bibtex-autokey ()
+  "Generate an autokey for the current headline"
+  (org-bibtex-put "CUSTOM_ID"
+                  (if org-bibtex-autogen-keys
+                      (let ((entry (org-bibtex-headline)))
+                        (with-temp-buffer
+                          (insert entry)
+                          (bibtex-generate-autokey)))
+                    (read-from-minibuffer "id: "))))
+
+(defun org-bibtex-fleshout (type &optional optional)
+  "Fleshout the current heading, ensuring that all required fields are present.
+With optional argument OPTIONAL, also prompt for optional fields."
+  (flet ((get (key lst) (cdr (assoc key lst)))
+         (name (keyword) (upcase (substring (symbol-name keyword) 1))))
+    (dolist (field (append
+		    (remove :title (get :required (get type org-bibtex-types)))
+		    (when optional (get :optional (get type org-bibtex-types)))))
+      (when (consp field) ; or'd pair of fields e.g., (:editor :author)
+        (let ((present (first (remove nil
+                                (mapcar
+                                 (lambda (f) (when (org-bibtex-get (name f)) f))
+                                 field)))))
+          (setf field (or present
+			  (intern (org-icompleting-read
+				   "Field: " (mapcar #'symbol-name field)))))))
+      (let ((name (name field)))
+        (unless (org-bibtex-get name)
+          (let ((prop (org-bibtex-ask field)))
+            (when prop (org-bibtex-put name prop)))))))
+  (when (and type (assoc type org-bibtex-types)
+             (not (org-bibtex-get "CUSTOM_ID")))
+    (org-bibtex-autokey)))
+
+\f
+;;; Bibtex link functions
 (org-add-link-type "bibtex" 'org-bibtex-open)
 (add-hook 'org-store-link-functions 'org-bibtex-store-link)
 
-;; (defun org-bibtex-publish (path)
-;;   "Build the description of the BibTeX entry for publishing."
-;;   (let* ((search (when (string-match "::\\(.+\\)\\'" path)
-;; 		   (match-string 1 path)))
-;; 	 (path (substring path 0 (match-beginning 0)))
-;; 	 key)
-;;     (with-temp-buffer
-;;       (org-open-file path t nil search)
-;;       (setq key (org-create-file-search-functions)))
-;;     (or description key)))
-
 (defun org-bibtex-open (path)
   "Visit the bibliography entry on PATH."
   (let* ((search (when (string-match "::\\(.+\\)\\'" path)
@@ -198,6 +408,87 @@
 ;; Finally add the link search function to the right hook.
 (add-hook 'org-execute-file-search-functions 'org-execute-file-search-in-bibtex)
 
+\f
+;;; Bibtex <-> Org-mode headline translation functions
+(defun org-bibtex ()
+  "Export each headline in the current file to a bibtex entry.
+Headlines are exported using `org-bibtex-export-headline'."
+  (interactive)
+  (let ((bibtex-entries (remove nil (org-map-entries #'org-bibtex-headline))))
+    (with-temp-file (concat (file-name-sans-extension (buffer-file-name)) ".bib")
+      (insert (mapconcat #'identity bibtex-entries "\n")))))
+
+(defun org-bibtex-check (&optional optional)
+  "Check the current headline for required fields.
+With prefix argument OPTIONAL also prompt for optional fields."
+  (interactive "P")
+  (save-restriction
+    (org-narrow-to-subtree)
+    (let ((type ((lambda (name) (when name (intern (concat ":" name))))
+                 (org-bibtex-get "TYPE"))))
+      (when type (org-bibtex-fleshout type optional)))))
+
+(defun org-bibtex-check-all (&optional optional)
+  "Check all headlines in the current file.
+With prefix argument OPTIONAL also prompt for optional fields."
+  (interactive) (org-map-entries (lambda () (org-bibtex-check optional))))
+
+(defun org-bibtex-create (type)
+  "Create a new entry at the given level."
+  (interactive
+   (list (org-icompleting-read
+          "Type:"
+	  (mapcar (lambda (type) (symbol-name (car type))) org-bibtex-types))))
+  (let ((type (if (keywordp type) type (intern type))))
+    (unless (assoc type org-bibtex-types)
+      (error "type:%s is not known" type))
+    (org-insert-heading)
+    (let ((title (org-bibtex-ask :title)))
+      (insert title) (org-bibtex-put "TITLE" title))
+    (org-bibtex-put "TYPE" (substring (symbol-name type) 1))
+    (org-bibtex-fleshout type)))
+
+(defun org-bibtex-read ()
+  "Read a bibtex entry and save to `*org-bibtex-entries*'.
+This uses `bibtex-parse-entry'."
+  (interactive)
+  (flet ((keyword (str) (intern (concat ":" (downcase str))))
+         (clean-space (str) (replace-regexp-in-string
+                             "[[:space:]\n\r]+" " " str))
+         (strip-delim (str)	     ; strip enclosing "..." and {...}
+		      (dolist (pair '((34 . 34) (123 . 125)))
+			(when (and (= (aref str 0) (car pair))
+				   (= (aref str (1- (length str))) (cdr pair)))
+			  (setf str (subseq str 1 (1- (length str)))))) str))
+    (push (mapcar
+           (lambda (pair)
+             (cons (let ((field (keyword (car pair))))
+                     (case field
+                       (:=type= :type)
+                       (:=key= :key)
+                       (otherwise field)))
+                   (clean-space (strip-delim (cdr pair)))))
+           (save-excursion (bibtex-beginning-of-entry) (bibtex-parse-entry)))
+          *org-bibtex-entries*)))
+
+(defun org-bibtex-write ()
+  "Insert a heading built from the first element of `*org-bibtex-entries*'."
+  (interactive)
+  (when (= (length *org-bibtex-entries*) 0)
+    (error "No entries in `*org-bibtex-entries*'."))
+  (let ((entry (pop *org-bibtex-entries*)))
+    (flet ((get (field) (cdr (assoc field entry))))
+      (org-insert-heading)
+      (insert (get :title))
+      (org-bibtex-put "TITLE" (get :title))
+      (org-bibtex-put "TYPE"  (downcase (get :type)))
+      (dolist (pair entry)
+        (case (car pair)
+          (:title    nil)
+          (:type     nil)
+          (:key      (org-bibtex-put "CUSTOM_ID" (cdr pair)))
+          (otherwise (org-bibtex-put (car pair)  (cdr pair))))))))
+
 (provide 'org-bibtex)
 
 ;; arch-tag: 83987d5a-01b8-41c7-85bc-77700f1285f5
-- 
1.7.1


[-- Attachment #3: Type: text/plain, Size: 47 bytes --]


-- 
Eric Schulte
http://cs.unm.edu/~eschulte/