From mboxrd@z Thu Jan 1 00:00:00 1970 From: Leonard Avery Randall Subject: Re: Managing bibtex database using org-mode? Date: Mon, 02 Jun 2014 12:29:18 +0100 Message-ID: <538C600E.1000706@gmail.com> References: <5369F4E8.3020807@gmail.com> <587A9279-A389-4A3F-808D-1E090741DD98@agrarianresearch.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------070203080005010401080601" Return-path: Received: from eggs.gnu.org ([2001:4830:134:3::10]:33956) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WrQQk-0006c4-2m for emacs-orgmode@gnu.org; Mon, 02 Jun 2014 07:29:35 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1WrQQf-0003HR-14 for emacs-orgmode@gnu.org; Mon, 02 Jun 2014 07:29:30 -0400 Received: from mail-wg0-x232.google.com ([2a00:1450:400c:c00::232]:56457) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1WrQQe-0003Dn-E6 for emacs-orgmode@gnu.org; Mon, 02 Jun 2014 07:29:24 -0400 Received: by mail-wg0-f50.google.com with SMTP id x12so4893724wgg.21 for ; Mon, 02 Jun 2014 04:29:23 -0700 (PDT) In-Reply-To: <587A9279-A389-4A3F-808D-1E090741DD98@agrarianresearch.org> List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Vikas Rawal , Richard Lawrence Cc: org-mode mailing list This is a multi-part message in MIME format. --------------070203080005010401080601 Content-Type: multipart/alternative; boundary="------------030102000007070608030508" --------------030102000007070608030508 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi Vikas, Sorry I did not reply earlier. Your original email prompted me to develop a more functional workflow. I have developed a few functions that make organizing research much easier. I have functions that reorganize, rename, and add links to pdfs, search for and add links to pdfs that have been organized by other programs (Papers2), import notes from Skim, rename org-bibtex headlines to a format I find more useful (e.g. Author (year) Title), and do a few other things that I find convenient. The renaming functions also look up crossrefs, and the headline renaming function lets you know if they are missing. The functions are partially documented and have a few customizable functions so they can be set up for different systems. Also note that the pdf organizing system uses the conventions of my old research organizing tool (Papers2) but this can be modified without too much work. I have attached a file with the functions. The bottom of the file also contains the variables that I have set and hooks that I have added to make the functions integrate more smoothly into my set up. They may serve as a guide if you choose to use them. Additionally, here is the capture template I use for bibtex entries, it is based largely on Richard's but it has you import more bibtex info during the main capture process, and uses a function to help you find crossrefs. ("r" "Reading" entry (file+olp "~/Google-Drive/Personal-Projects/Bib/Readinglist.org" "RLIST Inbox") "** %^{Todo state|READ|FIND|PRINT|NOTES} [#%^{Priority|A|B|C}] New Reading Entry %? %^{BIB_TITLE}p %^{BIB_AUTHOR}p %^{BIB_EDITOR}p %^{BIB_YEAR}p %^{CUSTOM_ID}p %^g :PROPERTIES: :BIB_BTYPE: %^{Entry type|book|article|inbook|bookinbook|incollection|suppbook|phdthesis|proceedings|inproceedings|booklet} :ENTERED_ON: %U %(my-org-bibtex-crossref) :END:" :prepend t) When I have some time I will follow up to explain how the functions work in more detail. In the mean time let me know if you have any questions. All best, Leonard --------------030102000007070608030508 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit
Hi Vikas,
Sorry I did not reply earlier. Your original email prompted me to develop a more functional workflow.

I have developed a few functions that make organizing research much easier. I have functions that reorganize, rename, and add links to pdfs, search for and add links to pdfs that have been organized by other programs (Papers2), import notes from Skim, rename org-bibtex headlines to a format I find more useful (e.g. Author (year) Title), and do a few other things that I find convenient. The renaming functions also look up crossrefs, and the headline renaming function lets you know if they are missing. The functions are partially documented and have a few customizable functions so they can be set up for different systems. Also note that the pdf organizing system uses the conventions of my old research organizing tool (Papers2) but this can be modified without too much work. I have attached a file with the functions. The bottom of the file also contains the variables that I have set and hooks that I have added to make the functions integrate more smoothly into my set up. They may serve as a guide if you choose to use them. 


Additionally, here is the capture template I use for bibtex entries, it is based largely on Richard's but it has you import more bibtex info during the main capture process, and uses a function to help you find crossrefs.

("r" "Reading" entry
         (file+olp "~/Google-Drive/Personal-Projects/Bib/Readinglist.org" "RLIST Inbox")
         "** %^{Todo state|READ|FIND|PRINT|NOTES} [#%^{Priority|A|B|C}] New Reading Entry %? %^{BIB_TITLE}p %^{BIB_AUTHOR}p %^{BIB_EDITOR}p %^{BIB_YEAR}p %^{CUSTOM_ID}p %^g
:PROPERTIES:
:BIB_BTYPE: %^{Entry type|book|article|inbook|bookinbook|incollection|suppbook|phdthesis|proceedings|inproceedings|booklet}
:ENTERED_ON: %U %(my-org-bibtex-crossref)
:END:" :prepend t)


When I have some time I will follow up to explain how the functions work in more detail. In the mean time let me know if you have any questions. 

All best,
Leonard

--------------030102000007070608030508-- --------------070203080005010401080601 Content-Type: text/plain; x-mac-type="0"; x-mac-creator="0"; name="my-org-bibtex-functions.el" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="my-org-bibtex-functions.el" (setq-default bibtex-dialect 'biblatex) ;; setting a default bibtex-dialect makes `orb-bibtex-read' and ;; relatives work more smoothly, but I still come accross bugs. (defcustom my-papers-directory nil "Directory for filing papers for org-bibtex entries. For optimal lin fuknctionality path should not contain spaces. Include trailing forward slash." :group 'my-org-bibtex :type '(choice (const nil) (string))) (defcustom my-temp-pdf-dir nil "Default directory used by `my-org-refile-supplements' to search for pdfs. Include trailing forward slash." :group 'my-org-bibtex :type '(choice (const nil) (string))) (defcustom my-miscellaneous-file-dir nil "Directory used-by `my-org-refile-pdf' to refile supplements not associated with org-bibtex entries. Include trailing forward slash." :group 'my-org-bibtex :type '(choice (const nil) (string))) (defcustom my-org-bibtex-rename-entry-searches-for-link t "Determines if `my-org-bibtex-rename-entry' and `my-org-bibtex-header-replacer' look for and insert links using `my-org-bibtex-insert-link'" :group 'my-org-bibtex :type 'boolean) (defcustom my-default-skim-notes-dir nil "Default directory used-by `my-org-bibtex-capture-skim-notes'" :group 'my-org-bibtex :type '(choice (const nil) (string))) (defcustom my-org-bibtex-standard-cite-command "\\cite" "Citation Command used by `my-org-refile-skim-notes'" :type 'string :group 'my-org-bibtex) (defun my-org-bibtex-header-replacer () "Replaces headers of all bibtex files in buffer using `my-org-bibtex-header-replacer'." (interactive) (let ((TYPE (concat org-bibtex-prefix org-bibtex-type-property-name))) (save-excursion (goto-char (point-max)) (while (not (or (org-before-first-heading-p) (= (point) (point-min)))) (backward-char 1) (re-search-backward org-outline-regexp-bol nil t) (when (org-entry-get (point) TYPE) (my-org-bibtex-rename-entry)))))) (defun my-org-bibtex-rename-entry () "Replaces header of bibtex entry to the format Author (year) Title. If `my-org-bibtex-rename-entry-searches-for-link' is non-nil it also looks for associated file and inserts link using `my-org-bibtex-insert-link'" (interactive) (let ((TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) (CROSSREF (concat org-bibtex-prefix "CROSSREF")) (AUTHOR (concat org-bibtex-prefix "AUTHOR")) (EDITOR (concat org-bibtex-prefix "EDITOR")) (YEAR (concat org-bibtex-prefix "YEAR")) (DATE (concat org-bibtex-prefix "DATE")) (TITLE (concat org-bibtex-prefix "TITLE"))) (save-excursion (unless (looking-at org-outline-regexp-bol) (re-search-backward org-outline-regexp-bol)) (let* ((content (nth 4 (org-heading-components))) (tags (nth 5 (org-heading-components))) (crossref (org-entry-get (point) CROSSREF)) (creator (if (org-entry-get (point) AUTHOR) (org-entry-get (point) AUTHOR) (if (org-entry-get (point) EDITOR) (concat (org-entry-get (point) EDITOR) " ed") (if crossref (my-org-find-crossref-entry crossref AUTHOR EDITOR) "Unknown")))) (year (if (org-entry-get (point) YEAR) (org-entry-get (point) YEAR) (if (org-entry-get (point) DATE) (org-entry-get (point) DATE) (if crossref (my-org-find-crossref-entry crossref YEAR DATE) "Unknown")))) (title (org-entry-get (point) TITLE)) (nice-title (my-org-bibtex-convert-emphasis-and-quotes title)) (new-header (concat creator ". (" year ") " nice-title))) (save-excursion (end-of-line) (search-backward content) (insert new-header) ;; this slightly indirect method maintains current visibility. (let ((beg (point))) (end-of-line) (delete-region beg (point))) (org-set-tags-to tags) ;; check if file exists and insert (if my-org-bibtex-rename-entry-searches-for-link (my-org-bibtex-insert-link))))))) (defun my-org-find-crossref-entry (crossref field &optional alternate) (let* ((TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) (CROSSREF (concat org-bibtex-prefix "CROSSREF")) (AUTHOR (concat org-bibtex-prefix "AUTHOR")) (EDITOR (concat org-bibtex-prefix "EDITOR")) (YEAR (concat org-bibtex-prefix "YEAR")) (DATE (concat org-bibtex-prefix "DATE")) (TITLE (concat org-bibtex-prefix "TITLE")) (BIBKEY org-bibtex-key-property) (ID (org-id-get nil t)) (marker (org-id-find ID t)) (cbuffer (current-buffer)) (buffer (marker-buffer marker))) (set-buffer buffer) (save-excursion (goto-char (point-min)) (let ((crossrefp (concat "^[ \t]*:" BIBKEY ":[ \t]*" crossref "[ \t]*$"))) (if (re-search-forward crossrefp (point-max) t) (if (org-entry-get (point) field) (org-entry-get (point) field) (if (and alternate (org-entry-get (point) alternate)) (if (string= alternate EDITOR) (concat (org-entry-get (point) alternate) " ed") (org-entry-get (point) alternate)) "Unknown")) "Missing Crossref"))))) (defun my-org-bibtex-insert-link () "Check if corresponding file exists and if no link to file exists create it." (interactive) (let* ((TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) (type (org-entry-get (point) TYPE)) (subdirectory (if (string= type "article") "Articles/" "Books/")) (file-name (concat (my-org-bibtex-find-file-names) ".pdf")) (full-file-path (concat my-papers-directory subdirectory file-name)) (link (concat "file:" full-file-path))) ;; Check that no link to file currently exists, and that file exists. (if type (save-excursion (unless (looking-at org-outline-regexp) (re-search-backward org-outline-regexp)) (if (re-search-forward link nil t) (when (called-interactively-p) (message "Link already exists")) (if (file-exists-p full-file-path) ;; Insert link in drawer with file name as description (my-org-insert-link-in-drawer nil link file-name) (when (called-interactively-p) (error "The file %s does not exist." file-name))))) (error "Not an org-bibtex entry")))) (defun my-org-bibtex-find-file-names () "Create safe informative file names for supplements to org bibtex entries." (let* ((TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) (TITLE (concat org-bibtex-prefix "TITLE")) (YEAR (concat org-bibtex-prefix "YEAR")) (AUTHOR (concat org-bibtex-prefix "AUTHOR")) (CROSSREF (concat org-bibtex-prefix "CROSSREF")) (crossref (org-entry-get (point) CROSSREF)) (type (org-entry-get (point) TYPE)) (author (if (org-entry-get (point) AUTHOR) (org-entry-get (point) AUTHOR) (when crossref (my-org-find-crossref-entry crossref AUTHOR)))) (title (org-entry-get (point) TITLE)) (year (if (org-entry-get (point) YEAR) (org-entry-get (point) YEAR) (when crossref (my-org-find-crossref-entry crossref YEAR)))) (author-surname (if author (replace-regexp-in-string ",.*" "" author) "")) (safe-author (if author (concat (replace-regexp-in-string "[ \t]" "_" author-surname) "_") "")) ;; convert csquotes and biblatex quote commands to straight ;; single quotes and delete emphasis. (safe-title (when title (if (string-match "\\\\\\|{\\|}" title) (my-org-bibtex-convert-emphasis-and-quotes title t) title))) ;; straighten other quotes (sq-title (when title (replace-regexp-in-string "`" "'" safe-title))) (file-title (if title (concat "_" (replace-regexp-in-string "[ \t]" "_" sq-title)) ""))) (concat safe-author year file-title))) (defun my-org-refile-supplements (&optional supplement new-name refile-directory) "Refile various files and place link to them in current entry. If the current entry is an org-bibtex entry, it gives you the option to refile it as the main pdf for the current entry or as a supplement. Primary pdfs are refiled according to the function in function specified in `my-org-bibtex-find-file-names'. The name and location can also be set with the optional arguments NEW-NAME and REFILE-DIRECTORY. NEW-NAME should be a string with the new title of the supplemnt. Valid options for REFILE-DIRECTORY are 'main 'misc 'supp or a string containing a unix path to an existing file directory. Other values will be ignored. The value 'main will refile the supplement as the main pdf of current entry, providing various options if such an entry already exists, 'supp will refile it in the \"Supplements\" subdirectory of either the \"Articles\" or \"Books\" subdirectory of `my-papers-directory'. 'misc will refile the paper in `my-miscellaneous-file-dir'. If the heading is not an org-bibtex entry 'main and 'supp will also fall back on `my-miscellaneous-file-directory'." (interactive) (let* ((old-name (or supplement (read-file-name "Supplement to refile: " my-temp-pdf-dir nil t))) (TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) (type (org-entry-get (point) TYPE)) (subdirectory (if (equal type "article") "Articles/" "Books/")) (file-name (when type (concat (my-org-bibtex-find-file-names) ".pdf"))) (full-file-path (concat my-papers-directory subdirectory file-name)) (enable-recursive-minibuffers t) (p-name (lambda () (or new-name (read-from-minibuffer "Suplement name: ")))) prompt-name misc-name misc-link supp-name supp-link (set-supp-name (lambda () (setq prompt-name (funcall p-name)) (setq supp-name (concat my-papers-directory subdirectory "Supplements/" prompt-name)) (setq supp-link (concat "file:" supp-name)))) (set-misc-name (lambda () (setq prompt-name (funcall p-name)) (setq misc-name (concat my-miscellaneous-file-dir prompt-name)) (setq misc-link (concat "file:" misc-name))))) ;; Check if refile directory is explicitly named. If so refile ;; file in this directory (if (and refile-directory (not (or (equal refile-directory 'main) (and (stringp refile-directory) (file-directory-p refile-directory))))) (list (when (and (stringp refile-directory) (file-directory-p refile-directory)) (let ((prompt-name (funcall p-name)) (file-loc (concat refile-directory prompt-name))) (rename-file old-name file-loc) (my-org-insert-link-in-drawer nil (concat "file:" file-loc prompt-name) prompt-name))) (when (equal refile-directory 'misc) (list (funcall set-misc-name) (rename-file old-name misc-name) (my-org-insert-link-in-drawer nil misc-link prompt-name))) (when (equal refile-directory 'supp) (if type (list (funcall set-supp-name) (rename-file old-name supp-name) (my-org-insert-link-in-drawer supp-link prompt-name)) (funcall set-misc-name) (rename-file old-name misc-name) (my-org-insert-link-in-drawer nil misc-link prompt-name)))) ;; If not, check if file is an org-bibtex entry (if type (if (not (file-exists-p full-file-path)) (if (or (equal refile-directory 'main) (y-or-n-p "Refile as main pdf for current entry? ") (list (rename-file old-name full-file-path) (my-org-bibtex-insert-link))) (when (y-or-n-p) "Refile as supplement? " (funcall set-supp-name) (rename-file old-name supp-name) (my-org-insert-link-in-drawer nil supp-link prompt-name))) (if (y-or-n-p "PDF associated with entry already exists. Replace it? ") (list (when (y-or-n-p "Keep current PDF as Supplement? ") (funcall set-supp-name) (rename-file full-file-path supp-name) (my-org-insert-link-in-drawer nil supp-link prompt-name)) (rename-file old-name full-file-path t) (my-org-bibtex-insert-link)) (when (y-or-n-p "Refile as supplement? ") (funcall set-supp-name) (rename-file old-name supp-name) (my-org-insert-link-in-drawer nil supp-link prompt-name) ;; Make sure there is a link to the original file. (my-org-bibtex-insert-link)))) ;; if not in org-bibtex entry refile file in misc directory (funcall set-misc-name) (rename-file old-name misc-name) (my-org-insert-link-in-drawer nil misc-link prompt-name))))) (defun my-org-insert-links-drawer () "Insert a links drawer near the beginning of the current entry. Inserts drawer near beginning of entry after scheduling, deadline and clocking info and after standard logging, property and clock drawers." (interactive) (org-back-to-heading t) (looking-at org-outline-regexp) (let ((indent (if org-adapt-indentation (- (match-end 0) (match-beginning 0)) 0)) (beg (point)) (re (concat "^[ \t]*" org-keyword-time-regexp)) end hiddenp) (outline-next-heading) (setq end (point)) (goto-char beg) (while (re-search-forward re end t)) (setq hiddenp (outline-invisible-p)) (end-of-line 1) (and (equal (char-after) ?\n) (forward-char 1)) (while (looking-at "^[ \t]*\\(:CLOCK:\\|:LOGBOOK:\\|CLOCK:\\|:PROPERTIES:\\|:END:\\)") (if (member (match-string 1) '("CLOCK:" ":END:")) ;; just skip this line (beginning-of-line 2) ;; Drawer start, find the end (re-search-forward "^\\*+ \\|^[ \t]*:END:" nil t) (beginning-of-line 1))) (org-skip-over-state-notes) (skip-chars-backward " \t\n\r") (if (and (eq (char-before) ?*) (not (eq (char-after) ?\n))) (forward-char 1)) (goto-char (point-at-eol)) (let ((inhibit-read-only t)) (insert "\n:LINKS:\n:END:")) (beginning-of-line 0) (org-indent-to-column indent) (beginning-of-line 2) (org-indent-to-column indent) (beginning-of-line 0) (if hiddenp (save-excursion (org-back-to-heading t) (hide-entry)) (org-flag-drawer t)))) (defun my-org-insert-link-in-drawer (&optional complete-file link-location default-description) "Find or create links drawer then insert link with `org-insert-link'. The arguments COMPlETE-FILE LINK-LOCATION and DEFAULT-DESCRIPTION are passsed on to `org-insert-link'. See its documentation for information on how these arguments are handled." (interactive) ;; make sure that we are currently in an entry. (if (not (org-before-first-heading-p)) ;; look for links drawer. (save-excursion (org-back-to-heading t) (let ((beg (point)) end hiddenp) (outline-next-heading) (setq end (point)) (backward-char 1) (setq hiddenp (outline-invisible-p)) (goto-char beg) (unless (and (re-search-forward "[ \t]*:LINKS:[ \t]*$" end t) (re-search-forward "[ \t]*:END:[ \t]*$" end t)) ;; if no links drawer found create one. (my-org-insert-links-drawer)) ;; go to links drawer. (goto-char beg) (re-search-forward "[ \t]*:LINKS:[ \t]*$") ;; open new line and insert link. (newline) (org-insert-link complete-file link-location default-description) (if hiddenp (hide-entry)))) ;; Give error if before first heading. (error "Before first heading"))) (defun my-org-bibtex-yank () (interactive) (let ((tempbib-file (concat temporary-file-directory "tempbib.bib"))) (delete-file tempbib-file) (with-temp-file tempbib-file (yank) (my-org-bibtex-preparatory-conversions)) (org-bibtex-import-from-file tempbib-file))) (defun my-org-bibtex-preparatory-conversions (&optional delimited start end) (interactive) (save-excursion (beginning-of-buffer) (replace-regexp "^[ \t]*\\([a-z:]*\\)[ \t]*=[ \t]*" "\\1=" delimited start end) (beginning-of-buffer) (replace-regexp "\\(title=\\){\\({.*}\\)},$" "\\1\\2" delimited start end))) (defun add-bibliographic-data () "Hook for adding bibliographic data during capture process. Note, this function places cursor at beginning of buffer to ensure that it is in main header for post capture hooks. This allows for subheaders with notes without messing up capture process. It also makes this function unsuitable for use in main buffer." (message "optionally adding bibliographic data") (when (and (org-entry-get (point) "CUSTOM_ID") (y-or-n-p "Add bibliographic data? ")) (beginning-of-buffer) (save-excursion (if (y-or-n-p "Use Optional Fields?") (org-bibtex-create-in-current-entry 1) (org-bibtex-create-in-current-entry))) )) (defun my-org-search-for-supplements-p () (when (y-or-n-p "Search for supplements?") (my-org-refile-supplements))) ;;** Crossref insertion (defun my-org-bibtex-crossref () (interactive) (if (y-or-n-p "Add a crossref ?") (let ((reftex-cite-format '((?\C-m . "%l"))) (enable-recursive-minibuffers t) crossref) (if (y-or-n-p "Use Reftex?") (with-temp-buffer (insert (concat "\n:" org-bibtex-prefix "CROSSREF: ")) (reftex-citation) (setq crossref (buffer-string)) (message "%s" crossref)) (setq crossref (concat "\n:" org-bibtex-prefix "CROSSREF: " (read-from-minibuffer "Insert Crossref. "))) (message "%s" crossref))) "")) (defun my-org-bibtex-convert-emphasis-and-quotes (string &optional delete-emphasis) "Converts quotes and emphasis from various sorts of tex emphasis, into single straight quotes and org-mode bold markup. If DELETE-EMPHASIS is non nil it deletes emphasis rather than converting it." (with-temp-buffer (insert string) (end-of-buffer) (let* ((emph "\\\\emph{\\|\\\\textit{\\|\\\\textbf{") (quotes "\\\\enquote{\\|\\\\mkbibquote{") (tex-markup (concat emph "\\|" quotes)) (smartparens-mode t) beg end text-begin) (while (re-search-backward tex-markup (point-min) t) (setq beg (point)) (if (looking-at emph) (if delete-emphasis (list (re-search-forward tex-markup) (setq text-begin (point)) (backward-char 1) (sp-forward-sexp) (delete-char -1) (delete-region beg text-begin) (goto-char beg)) (re-search-forward tex-markup) (setq text-begin (point)) (backward-char 1) (sp-forward-sexp) (delete-char -1) (insert "\*") (delete-region beg text-begin) (goto-char beg) (insert "\*")) (re-search-forward tex-markup) (setq text-begin (point)) (backward-char 1) (sp-forward-sexp) (delete-char -1) (insert "\'") (delete-region beg text-begin) (goto-char beg) (insert "\'")))) (buffer-string))) ;; (defun my-org-refile-skim-notes (notes-file &optional arg) ;; "Formats and refiles notes exported as text from Skim.app. ;; If called from an an entry inside an org file, it creates an ;; indirect buffer for this entry, and Prompts the user to search ;; for files in `my-default-skim-notes-dir'. Once file is selected ;; it formats the notes and refiles them to the subtree of the ;; current entry, under the subheading \"Unprocessed Notes\" Be ;; aware that this function only works if entry falls under ;; `org-refile-targets'. ;; It will format notes labeled by skim as highlights, strike outs ;; or underlines as quotes. If the note file contains text ;; containing text formatted \"--**pp**--\" it will treat \"pp\" as ;; the number of the first page and correct the page number markers ;; given by skim. If the current entry is an org-bibtex entry, it ;; will also insert bibtex cite commands at the end of each note ;; that is marked with a page number. ;; Note that if page numbers in skim match the page numbers in the ;; document you should still include a note with the text ;; \"--**0**--\" so that the function knows the page numbers are ;; correct. Otherwise it will not insert cite commands. ;; Additionally, note that if the pdf contains a cover page, you ;; should adjust the mark accordingly. So, if an article begins on ;; p. 299 but also has a cover page, you should include a note ;; containing \"--**298**--\" rather than \"--**299**--\". ;; Usage notes. Skim formats individual notes such that they look ;; like top level headers, so it is possible to put subpoints in ;; your notes while working in skim. Simply open the note and enter ;; \"** \" and whatever header you like at the beginning of a new ;; line. If you include \"page, pp\" in the header where pp is the ;; page number assigned to the page by skim, the note will also be ;; processed like all other headings. Also note that skim allows you ;; to edit the text of highlights, underlines, and strike outs. ;; Accordingly it is possible to add comments to a highlight by ;; selecting the highlight and adding subordinate headlines after ;; the highlighted text. This will ensure that only the highlighted ;; text is formatted as a quote." ;; (interactive (list (read-file-name "Notes file to import: " ;; my-default-skim-notes-dir ;; nil 'confirm))) ;; (when (not (or (equal major-mode 'org-mode) (equal major-mode 'org-agenda-mode))) ;; (error "Not in org or agenda buffer.")) ;; (push-mark) ;; (if (equal major-mode 'org-agenda-mode) ;; (org-agenda-tree-to-indirect-buffer t) ;; (org-tree-to-indirect-buffer t)) ;; (save-buffer) ;; (if (equal org-indirect-buffer-display 'other-window) (other-window 1)) ;; (beginning-of-buffer) ;; (let* ((org-file (buffer-file-name)) ;; (ind-buffer (buffer-name)) ;; (TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) ;; (type (org-entry-get (point) TYPE)) ;; (entry-id (org-id-get (point) t)) ;; (bib-key (org-entry-get (point) org-bibtex-key-property)) ;; (org-refile-use-outline-path nil) ;; (org-refile-use-cache nil) ;; (org-refile-target-verify-function (lambda () (equal (org-id-get) ;; entry-id)))) ;; (set-buffer (find-file-noselect notes-file)) ;; (org-mode) ;; (beginning-of-buffer) ;; (let* ((pnum-p (re-search-forward "--\\*\\*\\([0-9]+\\)\\*\\*--" ;; (point-max) t)) ;; (init-pnum-str (when pnum-p (match-string-no-properties 1))) ;; (init-pnum (if pnum-p (string-to-number init-pnum-str) 0))) ;; (end-of-buffer) ;; (while (not (or (org-before-first-heading-p) (= (point) ;; (point-min)))) ;; (unless (looking-at org-outline-regexp-bol) ;; (re-search-backward org-outline-regexp-bol)) ;; (let* ((heading (nth 4 (org-heading-components))) ;; (headpageref ", page \\([0-9]+\\)") ;; (quoteref "\\*+ \\(Highlight\\|Underline\\|Strike Out\\),") ;; (beg-quote "#+BEGIN_QUOTE") ;; (end-quote "#+END_QUOTE") ;; end-of-entry) ;; (if (and pnum-p (string-match headpageref heading)) ;; (let* ((raw-pnum-str (match-string-no-properties 1 heading)) ;; (raw-pnum (string-to-number raw-pnum-str)) ;; (pnum (+ raw-pnum init-pnum -1)) ;; (pnum-str (number-to-string pnum)) ;; (pnum-cite-str (concat my-org-bibtex-standard-cite-command ;; "[" pnum-str "]{" ;; bib-key "}")) ;; (pnum-head-str (concat ",page " pnum-str)) ;; (pnum-notes-str (concat "Notes related to " pnum-cite-str)) ;; entry-end) ;; (save-excursion ;; (re-search-forward headpageref) ;; (setq end (point)) ;; (re-search-backward headpageref) ;; (delete-region (point) end) ;; (insert pnum-head-str)) ;; (save-excursion ;; (if (looking-at quoteref) ;; (list ;; (end-of-line) ;; ;; (save-excursion ;; ;; (outline-next-heading) ;; ;; (setq entry-end (point))) ;; ;; (when (re-search-forward "[^ \t\n]" entry-end t) ;; ;; (backward-char 1)) ;; ;; (newline) ;; (newline) (insert beg-quote) ;; (outline-next-heading) ;; ;; (re-search-backward "[^ \t\n]" nil t) ;; ;; (forward-char 1) ;; (open-line 2) ;; (when bib-key ;; (insert pnum-cite-str) ;; (newline)) ;; (insert end-quote)) ;; (when bib-key ;; (outline-next-heading) ;; ;; (re-search-backward "[^ \t\n]") ;; ;; (forward-char 1) ;; (open-line 2) ;; (insert pnum-notes-str))))) ;; (save-excursion ;; (when (looking-at quoteref) ;; (end-of-line) (newline) (insert beg-quote) ;; (if (outline-next-heading) ;; (open-line 2) ;; (newline)) ;; (insert end-quote)))) ;; (save-excursion (org-demote)) ;; (unless (equal (point) (point-min)) ;; (backward-char 1)))) ;; (beginning-of-buffer) ;; (save-excursion ;; (replace-regexp "[ \n\t]*#+END_QUOTE" "/n") ;; (save-excursion "#+BEGIN_QUOTE")) ;; (insert "* Unprocessed notes\n") ;; (beginning-of-line 0) ;; (let ((mr mark-ring) (gmr global-mark-ring)) ;; (push-mark (point-max)) ;; (activate-mark) ;; (org-refile 3) ;; (setq mark-ring mr ;; global-mark-ring gmr)) ;;; I seem to have made it safely to here the ;; ;;; problem appears to be with org-refile. ;; (when (y-or-n-p "Save processed notes file? ") ;; (write-file (concat notes-file "-autoconverted.org "))) ;; (set-buffer ind-buffer) ;; (when (y-or-n-p "Refile as supplement? ") ;; (if type ;; (my-org-refile-supplements (concat notes-file "-autoconverted.org") ;; (concat (my-org-bibtex-find-file-names) ;; "-unprocesed-notes.org") 'supp) ;; (my-org-refile-supplements (concat notes-file "-autoconverted.org") ;; nil 'misc))) ;; (when (not (y-or-n-p "Keep original file? ") ;; ;; delete file, or if `delete-by-moving-to-trash' is ;; ;; non-nil move to trash ;; (delete-file notes-file t)))))) (defun my-org-refile-skim-notes (notes-file &optional arg) "Formats and refiles notes exported as text from Skim.app. If called from an an entry inside an org file, it creates an indirect buffer for this entry, and Prompts the user to search for files in `my-default-skim-notes-dir'. Once file is selected it formats the notes and refiles them to the subtree of the current entry, under the subheading \"Unprocessed Notes\" Be aware that this function only works if entry falls under `org-refile-targets'. It will format notes labeled by skim as highlights, strike outs or underlines as quotes. If the note file contains text containing text formatted \"--**pp**--\" it will treat \"pp\" as the number of the first page and correct the page number markers given by skim. If the current entry is an org-bibtex entry, it will also insert bibtex cite commands at the end of each note that is marked with a page number. Note that if page numbers in skim match the page numbers in the document you should still include a note with the text \"--**0**--\" so that the function knows the page numbers are correct. Otherwise it will not insert cite commands. Additionally, note that if the pdf contains a cover page, you should adjust the mark accordingly. So, if an article begins on p. 299 but also has a cover page, you should include a note containing \"--**298**--\" rather than \"--**299**--\". Usage notes. Skim formats individual notes such that they look like top level headers, so it is possible to put subpoints in your notes while working in skim. Simply open the note and enter \"** \" and whatever header you like at the beginning of a new line. If you include \"page, pp\" in the header where pp is the page number assigned to the page by skim, the note will also be given a cite command formatted as a comment. Regardless it will contain the page number and source as properties. Also note that skim allows you to edit the text of highlights, underlines, and strike outs. Accordingly it is possible to add comments to a highlight by selecting the highlight and adding subordinate headlines after the highlighted text. This will ensure that only the highlighted text is formatted as a quote." (interactive (list (read-file-name "Notes file to import: " my-default-skim-notes-dir nil 'confirm))) (when (not (or (equal major-mode 'org-mode) (equal major-mode 'org-agenda-mode))) (error "Not in org or agenda buffer.")) (push-mark) (when (equal major-mode 'org-agenda-mode) (org-agenda-goto)) (save-buffer) (let* ((org-file (buffer-file-name)) (TYPE (concat org-bibtex-prefix org-bibtex-type-property-name)) (type (org-entry-get (point) TYPE)) (entry-id (org-id-get (point) t)) (orig-buffer (current-buffer)) (bib-key (org-entry-get (point) org-bibtex-key-property)) (org-refile-use-outline-path nil) (org-refile-use-cache nil) (org-refile-target-verify-function (lambda () (equal (org-id-get) entry-id))) orig-notes processed-notes) (set-buffer (find-file-noselect notes-file)) (setq orig-notes (buffer-string)) (with-temp-buffer (insert orig-notes) (my-skim-notes-transformations bib-key entry-id) (my-org-regularize-whitespace) (beginning-of-buffer) (my-fullstop-regexp) (beginning-of-buffer) (insert "* Unprocessed notes\n") (beginning-of-line 0) ;; (let ((mr mark-ring) (gmr global-mark-ring)) ;; (push-mark (point-max)) ;; (activate-mark) (org-refile)) ;; (setq mark-ring mr ;; global-mark-ring gmr)) ;;; I seem to have made it safely to here the ;;; problem appears to be with org-refile. ;; (when (y-or-n-p "Save processed notes file? ") ;; (write-file (concat notes-file "-autoconverted.org "))) ;; (set-buffer ind-buffer) ;; (when (y-or-n-p "Refile as supplement? ") ;; (if type ;; (my-org-refile-supplements (concat notes-file "-autoconverted.org") ;; (concat (my-org-bibtex-find-file-names) ;; "-unprocesed-notes.org") 'supp) ;; (my-org-refile-supplements (concat notes-file "-autoconverted.org") ;; nil 'misc))) (org-id-goto entry-id) (org-tree-to-indirect-buffer) (equal org-indirect-buffer-display 'other-window) (other-window 1) (beginning-of-buffer) (when (not (y-or-n-p "Keep original file? ")) ;; delete file, or if `delete-by-moving-to-trash' is ;; non-nil move to trash (delete-file notes-file t)))) (defun my-skim-notes-transformations (&optional bib-key id) (interactive) (org-mode) (beginning-of-buffer) (let* ((pnum-p (re-search-forward "--\\*\\*\\([0-9]+\\)\\*\\*--" (point-max) t)) (init-pnum-str (when pnum-p (match-string-no-properties 1))) (init-pnum (if pnum-p (string-to-number init-pnum-str) 0)) (headpageref ", [pP]age \\([0-9]+\\)") (quoteref "\\*+ \\(Highlight\\|Underline\\|Strike Out\\),") (beg-quote "#+BEGIN_QUOTE") (end-quote "#+END_QUOTE") end-of-entry) (beginning-of-buffer) (while (not (= (point) (point-max))) (if (org-before-first-heading-p) (outline-next-heading)) (let* ((heading (nth 4 (org-heading-components)))) (if (and pnum-p heading (string-match headpageref heading)) (let* ((raw-pnum-str (match-string-no-properties 1 heading)) (raw-pnum (string-to-number raw-pnum-str)) (pnum (+ raw-pnum init-pnum -1)) (pnum-str (number-to-string pnum)) (pnum-cite-str (concat my-org-bibtex-standard-cite-command "[" pnum-str "]{" bib-key "}")) (pnum-head-str (concat ", page " pnum-str)) (pnum-notes-str (concat "# Notes related to " pnum-cite-str)) entry-end) (save-excursion (re-search-forward headpageref) (setq end (point)) (re-search-backward headpageref) (delete-region (point) end) (insert pnum-head-str)) (if (looking-at quoteref) (list (save-excursion (end-of-line) ;; (save-excursion ;; (outline-next-heading) ;; (setq entry-end (point))) ;; (when (re-search-forward "[^ \t\n]" entry-end t) ;; (backward-char 1)) ;; (newline) (newline 2) (insert beg-quote) (if (outline-next-heading) ;; (re-search-backward "[^ \t\n]" nil t) ;; (forward-char 1) (open-line 2) (newline)) (when bib-key (insert pnum-cite-str) (newline)) (insert end-quote)) (org-entry-put (point) "NOTES_PAGE" pnum-str) (org-entry-put (point) "NOTES_PDF_PAGE" raw-pnum-str)) (when bib-key (org-entry-put (point) "NOTES_PAGE" pnum-str) (org-entry-put (point) "NOTES_PDF_PAGE" raw-pnum-str) (save-excursion (if (outline-next-heading) ;; (re-search-backward "[^ \t\n]") ;; (forward-char 1) (open-line 2) (newline)) (insert pnum-notes-str))))) (when (looking-at quoteref) (save-excursion (end-of-line) (newline) (open-line 2) (insert beg-quote) (if (outline-next-heading) (open-line 2) (newline)) (insert end-quote))) (when (and heading (string-match headpageref heading)) (let ((raw-pnum-str (match-string-no-properties 1 heading))) (org-entry-put (point) "NOTES_PDF_PAGE" raw-pnum-str))))) (save-excursion (org-demote)) (if (looking-at quoteref) (org-entry-put (point) "NOTE_TYPE" "quote") (org-entry-put (point) "NOTE_TYPE" "notes")) (if bib-key (org-entry-put (point) "SOURCE_BIB_KEY" bib-key)) (if id (org-entry-put (point) "SOURCE_ENTRY_ID" id)) (org-entry-put (point) "IMPORTED_ON" (format-time-string "%Y-%m-%d")) (outline-next-heading)) (beginning-of-buffer) (while (not (= (point) (point-max))) (if (org-before-first-heading-p) (outline-next-heading)) (let ((cheading (point))) (save-excursion (when (org-up-heading-safe) (let ((ppnum (org-entry-get (point) "NOTES_PAGE")) (ppdfpnum (org-entry-get (point) "NOTES_PDF_PAGE"))) (when ppnum (org-entry-put cheading "NOTES_PAGE" ppnum)) (if ppdfpnum (org-entry-put cheading "NOTES_PDF_PAGE" ppdfpnum))))) (outline-next-heading))))) (defun my-org-regularize-whitespace () (interactive) (save-excursion (beginning-of-buffer) (replace-regexp "\n\\{2,\\}\\(#\\+END_QUOTE\\|#\\+LaTeX\\|\\\\.*cite\\)" "\n\\1") (beginning-of-buffer) (replace-regexp "\\(#\\+BEGIN_QUOTE\\)\n\\{2,\\}" "\\1\n") (beginning-of-buffer) (replace-regexp "\n\\{3,\\}" "\n\n") (beginning-of-buffer) (replace-regexp "[ \t]+$" ""))) ;;* Add capture hooks (add-hook 'org-capture-prepare-finalize-hook 'add-bibliographic-data) (add-hook 'org-capture-before-finalize-hook 'my-org-search-for-supplements-p) (add-hook 'org-capture-before-finalize-hook 'my-org-bibtex-rename-entry) ;;* Keyboard shortcuts (global-set-key "\C-co" 'my-org-refile-skim-notes) (global-set-key "\C-cy" 'my-org-bibtex-yank) (global-set-key (kbd "C-c L") 'my-org-insert-link-in-drawer) (global-set-key (kbd "C-c M-a") 'my-org-bibtex-rename-entry) (global-set-key (kbd "C-c M-l") 'my-org-bibtex-insert-link) (global-set-key (kbd "C-c C-w") 'my-org-refile-supplements) ;;* My variables (setq my-miscellaneous-file-dir "/Users/leotr/Google-Drive/Miscellaneous-Supplements/" my-papers-directory "/Users/leotr/Google-Drive/Papers2/" my-temp-pdf-dir "/Users/leotr/tmp/pdfs/" my-default-skim-notes-dir "/Users/leotr/tmp/notes/") --------------070203080005010401080601--