Managing articles in orgmode and collaboration

Managing articles in orgmode and collaboration

[Christoph Groth]

Hello,

I’d like to keep my library of scientific articles in orgmode, along
with notes, links to external files (mostly PDF), etc.  This has been
discussed repeatedly on this list, for example in the recent thread
http://thread.gmane.org/gmane.emacs.orgmode/78983.

Most solutions seem to be based around a central BibTeX file and take
advantage of RefTeX to navigate between citations to articles (in LaTeX
or org files), the BibTeX file, related entries in an org-file, and
linked external files.  Often the key that connects the various items is
a unique label (in LastnameYear format, for example).  This key is used
as label when citing and in BibTeX, as orgmode CUSTOM_ID, and as the
filename of an associated external file.

This seems to work well for people who have complete control over the
articles they write.  But what about articles with co-authors?  These
must be self-contained, so one needs a separate BibTeX file for each
article project.  Let’s say that a co-author adds a new reference to a
common project, but the cited paper is already in my database under a
different label.  Maybe that very paper is already cited in an older
article with different co-authors using a different \cite label?

Does anyone have a solution that handles such cases nicely?  Perhaps
something along these lines: My (own, i.e. controlled by myself) library
of articles lives in one/several org files.  Each entry there contains
links to external files, URLs, DOIs, etc. and enough information to
create a BibTeX record.  (Such entries could be generated almost
automatically from .bib files found on the internet.)  A custom Emacs
function allows to create BibTeX records for use in project-specific
.bib-files from the orgfile-entries.  Now the last missing piece is to
connect things also in the other direction: allow to jump to the related
org-entry for a paper from a \cite{} in a paper, even if the reference
has been added to the .bib file by a colleague.

This last piece is the most difficult to realize.  It would require
either a rather smart search, or a robust CUSTOM_ID (Perhaps
LastnameYearPagenr would be unique and robust enough?).

I’m interested in discussing these issues.

Re: Managing articles in orgmode and collaboration

[Christoph Groth]

Hi again,

I’m replying to myself, as I think I’ve found a possible solution 
to my
problem.

Most publications have some kind of unique ID that can be present 
in
BibTeX: a DOI number, a ISBN, a preprint ID, or an URL.  One could 
use
one these fields as an org-id unique ID for the publication’s
org-mode-entry.  (If several such fields are present, priorities 
could
be used to choose one.)  Then, org-id can be used to find article
entries by unique ID.

This way, even if a reference has been added to a project-specific
.bib-file by a collaborator (with some label of his choice), it 
should
be possible to find the relevant org-mode-entry if the paper is 
already
anywhere in the library.  In the worst case, a missing DOI id will 
have
to be added to the .bib-file-entry.

The org entry on the article would contain links and notes.  The 
BibTeX
data could be kept in a :BIBTEX: drawer.

Any comments?

Christoph

Re: Managing articles in orgmode and collaboration

[Thorsten Jolitz]

Christoph Groth <christoph@grothesque.org> writes:

Hi,

> Any comments?

do you have a function that automatically fetches bibtex entries for
books from the web, given some info (title, year, author or so)?

-- 
cheers,
Thorsten

Re: Managing articles in orgmode and collaboration

[Christoph Groth]

Thorsten Jolitz wrote:

> do you have a function that automatically fetches bibtex entries for
> books from the web, given some info (title, year, author or so)?

I haven’t yet started to manage literature with org mode, so I do not
have anything so far.

It’s usually easy to find BibTeX records on the web.  For books and
articles in my field (physics) google scholar works quite fine.  For
articles only, arXiv offers powerful search capabilities.

It shouldn’t be too difficult to write a “capture” function in emacs
that turns such records into entries in an org file.  That would be
automatic enough for me.

Re: Managing articles in orgmode and collaboration

[Vikas Rawal]

> 
> It shouldn’t be too difficult to write a “capture” function in emacs
> that turns such records into entries in an org file.  That would be
> automatic enough for me.
> 

There is bibretrieve (https://github.com/pzorin/bibretrieve) and bibfetch (https://github.com/dschoepe/bibfetch/blob/master/bibfetch.el).

Vikas

Re: Managing articles in orgmode and collaboration

[Fabrice Popineau]

[-- Attachment #1: Type: text/plain, Size: 881 bytes --]

I have written a couple of functions to export the Qiqqa library format to
Org if anybody is interested.
Qiqqa has been much more eager and acute to find online references to all
my pdfs papers.

Fabrice


2014-09-09 14:18 GMT+02:00 Vikas Rawal <vikaslists@agrarianresearch.org>:

> >
> > It shouldn’t be too difficult to write a “capture” function in emacs
> > that turns such records into entries in an org file.  That would be
> > automatic enough for me.
> >
>
> There is bibretrieve (https://github.com/pzorin/bibretrieve) and bibfetch
> (https://github.com/dschoepe/bibfetch/blob/master/bibfetch.el).
>
> Vikas
>



-- 
Fabrice Popineau
-----------------------------
SUPELEC
Département Informatique
3, rue Joliot Curie
91192 Gif/Yvette Cedex
Tel direct : +33 (0) 169851950
Standard : +33 (0) 169851212
------------------------------

[-- Attachment #2: Type: text/html, Size: 1607 bytes --]

Re: Managing articles in orgmode and collaboration

[Thorsten Jolitz]

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

>> 
>> It shouldn’t be too difficult to write a “capture” function in emacs
>> that turns such records into entries in an org file.  That would be
>> automatic enough for me.
>> 
>
> There is bibretrieve (https://github.com/pzorin/bibretrieve) and
> bibfetch
> (https://github.com/dschoepe/bibfetch/blob/master/bibfetch.el).

cool, can you recommend one of the two?
Just from the repo looks, I would probably go for bibretrieve.

-- 
cheers,
Thorsten

Re: Managing articles in orgmode and collaboration

[Samuel Loury]

[-- Attachment #1: Type: text/plain, Size: 384 bytes --]

Vikas Rawal <vikaslists@agrarianresearch.org> writes:

> There is bibretrieve (https://github.com/pzorin/bibretrieve) and bibfetch (https://github.com/dschoepe/bibfetch/blob/master/bibfetch.el).
Is it only a result of a google search or did you personally use one of those?

-- 
Konubinix
GPG Key    : 7439106A
Fingerprint: 5993 BE7A DA65 E2D9 06CE  5C36 75D2 3CED 7439 106A

[-- Attachment #2: Type: application/pgp-signature, Size: 472 bytes --]

Re: Managing articles in orgmode and collaboration

[John Kitchin]

Thorsten Jolitz <tjolitz@gmail.com> writes:

I use https://github.com/jkitchin/jmax/blob/master/org/doi-utils.org

It is some code I wrote to build bibtex entries from a doi, and to
download the pdf if you have access to it.

I load it like this:
(org-babel-load-file "doi-utils.org")

You need to define some variables like:

org-ref-pdf-directory

and some features depend on org-ref
(https://github.com/jkitchin/jmax/blob/master/org/org-ref.org).

I use these pretty often.


> Christoph Groth <christoph@grothesque.org> writes:
>
> Hi,
>
>> Any comments?
>
> do you have a function that automatically fetches bibtex entries for
> books from the web, given some info (title, year, author or so)?

-- 
-----------------------------------
John Kitchin
Professor
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
http://kitchingroup.cheme.cmu.edu

Re: Managing articles in orgmode and collaboration

[Thomas S. Dye]

Aloha Christoph,

I don't manage my bibliography references in Org mode.  I am used to
managing a bibtex database and have never found the need to move
everything to Org.

Christoph Groth <christoph@grothesque.org> writes:

> Most solutions seem to be based around a central BibTeX file and take
> advantage of RefTeX to navigate between citations to articles (in LaTeX
> or org files), the BibTeX file, related entries in an org-file, and
> linked external files.  Often the key that connects the various items is
> a unique label (in LastnameYear format, for example).  This key is used
> as label when citing and in BibTeX, as orgmode CUSTOM_ID, and as the
> filename of an associated external file.

Bibtex mode has functions for automatic reference key generation:

http://www.jonathanleroux.org/bibtex-mode.html#0630

You can configure this process.

> This seems to work well for people who have complete control over the
> articles they write.  But what about articles with co-authors?  These
> must be self-contained, so one needs a separate BibTeX file for each
> article project.  Let’s say that a co-author adds a new reference to a
> common project, but the cited paper is already in my database under a
> different label.  Maybe that very paper is already cited in an older
> article with different co-authors using a different \cite label?

Either a separate bibtex file for each article, or separate bibtex files
for each co-author.  A LaTeX document can use any number of bibtex files
per document.

In general, you'll want to have the bibtex file(s) for an article only
contain the references that you'll use in the article, especially if you
intend to distribute the bibtex files as part of a reproducible research
project. There are tools that use the information in your article .tex
files to create this kind of bibtex file from a larger bibtex database.

In my work flow, I have a large legacy bibtex file with about 6,000
references.  When I'm writing an article, I create another bibtex file
just for the article.  I use ebib to open both the legacy bibtex file
and the article bibtex file and copy from one to the other, which ebib
makes very convenient.  Then I autogenerate the reference key in ebib by
pressing 'K'. 

hth,
Tom

-- 
Thomas S. Dye
http://www.tsdye.com

Re: Managing articles in orgmode and collaboration

[Jorge A. Alfaro-Murillo]

Thomas S. Dye writes: 

> I don't manage my bibliography references in Org mode.  I am 
> used to managing a bibtex database and have never found the need 
> to move everything to Org. 

Same here.

> Bibtex mode has functions for automatic reference key 
> generation: 
> 
> http://www.jonathanleroux.org/bibtex-mode.html#0630 
> 
> You can configure this process.

You can also download the .bib from Google Scholar or whatever and 
then clean the entry, so that your database has the same format.

> Either a separate bibtex file for each article, or separate 
> bibtex files for each co-author.  

Or better do both...

#+BEGIN_SRC latex 
  \bibliography{/home/you/references/articles.bib}
  % \bibliography{/home/collaborator_1/references/articles.bib}
  % \bibliography{/home/collaborator_2/references/articles.bib}
  ...
  \bibliography{references}
 
#+END_SRC

When a collaborator_i is working on the file she/he comments the 
first line and uncomments the i-th line AND everybody runs 
reftex-create-bibtex-file (or copy paste the new references for 
the unfortunate non-emacs user) after adding new references and 
finishing editing. Everybody shares a current version of the .tex 
file and the references.bib file.

> In general, you'll want to have the bibtex file(s) for an 
> article only contain the references that you'll use in the 
> article, especially if you intend to distribute the bibtex files 
> as part of a reproducible research project. 

Note that when the article is ready the references.bib is the only 
thing you need to compile, since it has all the references, so you 
erase the all the other \bibliography's

> There are tools that use the information in your article .tex 
> files to create this kind of bibtex file from a larger bibtex 
> database. 

Yes, emacs via reftex-create-bibtex-file =)

Best,

-- 
Jorge.

Re: Managing articles in orgmode and collaboration

[Rasmus]

jorge.alfaro-murillo@yale.edu (Jorge A. Alfaro-Murillo) writes:

> Thomas S. Dye writes: 
>
>> I don't manage my bibliography references in Org mode.  I am 
>> used to managing a bibtex database and have never found the need 
>> to move everything to Org. 
>
> Same here.

Me too.  I use the format

./literature
./literature/article1/
./literature/article1/article1.pdf 
./literature/article1/article1.bib ← one entry
./literature/article1/article1.org ← notes
...

I collect all bib files to create the master bibfile.  I also generate
a nicer overview.org → overview.html for easy search, though most of
the time reftex is enough.  I then use reftex to cite access the
master bib file.

>> There are tools that use the information in your article .tex 
>> files to create this kind of bibtex file from a larger bibtex 
>> database. 
>
> Yes, emacs via reftex-create-bibtex-file =)

That's wicket cool!

—Rasmus

-- 
Enough with the bla bla!

Re: Managing articles in orgmode and collaboration

[Christoph Groth]

Jorge A. Alfaro-Murillo wrote:

> Thomas S. Dye writes: 
>
>> I don't manage my bibliography references in Org mode.  I am used to
>> managing a bibtex database and have never found the need to move
>> everything to Org.
>
> Same here.

My motivation for keeping bibliography in org was to keep all local
information about a paper (including notes and comments) in one place.
This should make it easier to find it.

Meanwhile I found org-bibtex.  It seems to implement just what I had in
mind and is even included in orgmode by default.  The following function
can be used to save a org-bibtex headline into the kill-ring in bib
format, so that it can be yanked into a project-specific .bib-file:

(defun my-org-bibtex-kill-ring-save-headline ()
  (interactive)
  (kill-new (org-bibtex-headline)))

>> Either a separate bibtex file for each article, or separate bibtex
>> files for each co-author.
>
> Or better do both...
>
> #+BEGIN_SRC latex 
>   \bibliography{/home/you/references/articles.bib}
>   % \bibliography{/home/collaborator_1/references/articles.bib}
>   % \bibliography{/home/collaborator_2/references/articles.bib}
>   ...
>   \bibliography{references}
>  
> #+END_SRC
>
> When a collaborator_i is working on the file she/he comments the first
> line and uncomments the i-th line AND everybody runs
> reftex-create-bibtex-file (or copy paste the new references for the
> unfortunate non-emacs user) after adding new references and finishing
> editing. Everybody shares a current version of the .tex file and the
> references.bib file.

I like to keep papers under version control, and the commenting that you
suggest does not seem to fit this way of working very well.

Re: Managing articles in orgmode and collaboration

[Jorge A. Alfaro-Murillo]

Christoph Groth writes: 

> My motivation for keeping bibliography in org was to keep all 
> local information about a paper (including notes and comments) 
> in one place.  This should make it easier to find it. 

In BibTeX if any field contains an entry that is not part of the 
required or optional entries for the field type, that entry is 
ignored.  So you can do want you want in a single bib file, just 
include for every field that you want a "COMMENTS" entry. The 
"NOTE" entry is an optional entry for the field type article and 
book, so you have to use something else, I use "ANNOTE", since it 
is the standard "to be ignored" entry in the emacs BibTeX mode.

> I like to keep papers under version control, and the commenting 
> that you suggest does not seem to fit this way of working very 
> well. 

Then change it to:

#+BEGIN_SRC latex
  \include{personal_references}
  \bibliography{references.bib}
#+END_SRC

And in personal_references.tex, every user has just one line:

#+BEGIN_SRC latex
  \bibliography{the_user_path_to_her/his_references.bib}
#+END_SRC

Add personal_references.tex to the .gitignore file (or the 
equivalent if you are not using git), and then you have version 
control and no more commenting/uncommenting.

Perhaps I am biased because I learned LaTeX and BibTeX before Org, 
but I think that for references BibTeX (plus a little bit of emacs 
configuration) has everything I could need. I guess if you are 
more used to Org, it might be worth to invest time and come up 
with a org-based solution. Please keep us posted, I find this a 
very interesting thread.

Cheers,

-- 
Jorge.

Re: Managing articles in orgmode and collaboration

[Christoph Groth]

Jorge A. Alfaro-Murillo wrote:

> Perhaps I am biased because I learned LaTeX and BibTeX before 
> Org, but I think that for references BibTeX (plus a little bit 
> of emacs configuration) has everything I could need. I guess if 
> you are more used to Org, it might be worth to invest time and 
> come up with a org-based solution. Please keep us posted, I find 
> this a very interesting thread.

This is an update about my experiments so far.  Please comment! 
This is my current setup:

All the articles (mostly PDF files) are kept in one directory. 
There’s also an index.org file in there with headlines like this

--8<---------------cut here---------------start------------->8---
** 
   [[file:vazifeh13-electromagnetic_response_weyl.pdf][Electromagnetic 
   Response of Weyl Semimetals]]
:PROPERTIES:
:TITLE:    Electromagnetic Response of Weyl Semimetals
:BTYPE:    article
:ID:       vazifeh13-electromagnetic_response_weyl
:AUTHOR:   Vazifeh, M. M. and Franz, M.
:JOURNAL:  Phys. Rev. Lett.
(…)
:END:
Some comments/notes with links and in-line formulas.
--8<---------------cut here---------------end--------------->8---

These entries are created by org-bibtex-yank, Emacs’ bibtex is 
configured to generate the unique IDs (authorYY-three_first_words 
should be sufficiently unique in practice).

This setup works pretty well:

- Adding new articles is easy (there's still ample room for 
  improvement).
- Opening associated the associated article files and other links 
  works well.
- It’s easy to add notes (with in-line maths!) and do simple 
  searches.
- Linking from and to other org files (agenda, project-specific 
  notes) is possible.
- Additional attachment files (e.g. whiteboard photos from a 
  discussion) can added with org-attach.
- org-bibtex provides support to put entries into project-specific 
  bibtex files (org-bibtex-export-to-kill-ring).
- Thanks to the non-random unique IDs, it’s possible to jump to 
  articles in the library from references in latex files, even if 
  the key scheme in the latex file differs from the one used in 
  index.org.

But not all is good:

- Scaling: Some simple tests seem to indicate that org mode 
  becomes too sluggish with files of about 50k lines.  This is a 
  dimension that could be easily reached over 10 years if the file 
  grows by 20 lines per day on average.  This is not a problem in 
  the beginning, but if the scheme does not scale to a few 
  thousand entries, this renders the whole idea way less 
  interesting.

- More advanced searching is lacking: AFAIK org mode currently 
  does not support searching for articles of a given author that 
  also contain a given keyword in the notes.

Any insights about these two problems?  Perhaps the scaling could 
be managed by splitting index.org into several files (by year for 
example).  But how to search then?  It's probably not a good idea 
to add all the bibliography org-files into the agenda.  (Is there 
a way to have a secondary list of agenda files?)  Perhaps the 
solution for both problems would be to write a fast commandline 
query tool for such org-databases?  The tool could even use a fast 
cache if necessary.

Re: Managing articles in orgmode and collaboration

[Jorge A. Alfaro-Murillo]

Christoph Groth writes: 

> But not all is good: 
> 
> - Scaling: Some simple tests seem to indicate that org mode  
>   becomes too sluggish with files of about 50k lines.  This is a 
>   dimension that could be easily reached over 10 years if the 
>   file  grows by 20 lines per day on average.  This is not a 
>   problem in  the beginning, but if the scheme does not scale to 
>   a few  thousand entries, this renders the whole idea way less 
>   interesting.
> - More advanced searching is lacking: AFAIK org mode currently  
>   does not support searching for articles of a given author that 
>   also contain a given keyword in the notes. 
> Any insights about these two problems? Perhaps the scaling could 
> be managed by splitting index.org into several files (by year 
> for example). But how to search then? It's probably not a good 
> idea to add all the bibliography org-files into the agenda. (Is 
> there a way to have a secondary list of agenda files?) Perhaps 
> the solution for both problems would be to write a fast 
> commandline query tool for such org-databases? The tool could 
> even use a fast cache if necessary. 

BibTeX provides bibtex-search-entries (in a bib file it is bound 
to C-c C-a), which searches for entries with a certain field 
matching a regexp.  Perhaps you want to take advantage of that 
function to direct your org searches to bib files and back. Check 
also bibtex-files and bibtex-search-entry-globally, since you 
would then be able to split the org files and export into several 
bib files if a single bib file proves too much. My current 
articles.bib is about 14k lines, and it is not sluggish at all, 
but I will have to wait a couple of years until I tell you if a 
50k one would be.

Best,

-- 
Jorge.