My reference data management approach with org and emacs

My reference data management approach with org and emacs

[Marcelo de Moraes Serpa]

[-- Attachment #1.1: Type: text/plain, Size: 1688 bytes --]

Hello list,

I would like to share how I'm keeping my reference data. This includes
articles I write, blog post drafts, braintorms and anything else that we
could fit in the reference category (gtd-wide).

I don't like categories too much. Actually, I find them too strict and
limited. Putting things into folders just makes you loose ]time thinking
about structure. I'm adept of tags, though. I love them. So, my basic idea
was to have a folder (which I right now call wiki/) with a compendium of all
my reference data.

Whenever I need to create a new entry, I press s-r and it triggers dired
with this directory as context. So, I can just type something .org and press
<enter> to create it. Then, at the bottom, I create a * tags item. I tag it
with relevant tags and save. *I don't add it to the agenda list* -- I have a
custom rgrep function to seach over wiki/, which is binded to s-o. When I
want to find something from my reference data, I just press s-o and type a
string, and rgrep does the rest.

It's pretty simple, and, as you could note, doesn't use much of org's
functionalities. Using agenda would be overkill, as I have dozens of files
in the directory, and it would be probably overkill for org-agenda.

Anyways, just thought I'd share. It works great, is very organic, flexible
and simple. The goal was to have a simple storage system which was easy to
search and that wouldn't get on my way, but be easy to access/use when I
needed it.

As a knowledge worker, I find that it works quite well to quickly
brainstorm, draft blog posts or anything else that I want to keep as
reference.

How do you manage reference information? It'd be nice to know :)

Cheers,

Marcelo.

[-- Attachment #1.2: Type: text/html, Size: 1826 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: My reference data management approach with org and emacs

[Jan Böcker]

[The following text has gotten quite long. Sit comfortably and get a cup
of your preferred drink if you want to proceed.]

That is an interesting setup you describe there. I had considered
something similar myself, but found it a hassle to come up with a file
name for every new piece of information (although unix does allow
everything except "/" in a file name, I want my file names to be lower
case, short and without spaces where feasible).

To paraphrase what you said, putting things into files just makes you
loose time thinking about file names (which I also consider structure).

In the end, I settled upon one big org file ("reference.org"). Each
piece of reference information is in its own top-level node. When I want
to find something there, I use isearch and/or search for a specific tag
(in virtually every case, a simple isearch for one or two words is
sufficient).

This is way faster than navigating the file system! I also share your
dislike of categories (which a strictly hierarchical file system would
force me to use).

I have two remember templates to add an entry to reference.org.

The first template asks me for tags ("%^g") and a title for the
headline. After filing it (at the top of reference.org) with C-c C-c,
Org jumps to the location it was just filed ("%&" in the template), in
case I want to use C-c C-c again to readjust the tags.

I use this first template to keep data that can be expressed in plain
text (including all the powerful tools Org gives me to work with plain
text, such as outlines, links and tables).

The second template is a little more complex; it calls a custom elisp
function to do all the work.

When I call this second template, I am asked for the following:
- a title for the headline
- a date (I mostly use this template to file scanned letters, invoices
and the like, so it helps to be able to change the date from the default
of today.)
- a folder name (defaulting to YYYY-MM-DD.S, i.e. the previously
specified date followed by a sequence number to make the folder name
unique). Normally, I do not customize the folder name, because I only
need to find the reference data via Org and do not need to navigate to
it using e.g. the "open" dialog of any other program (and I do not want
to change it in the future, which might make a folder name containing a
date obsolete).
- where the original went (defaults to "Trash"). This is stored as an
attribute in the outline node.

A new subfolder with the specified folder name is created in ~/org/data/
and set up as this node's attachment directory. The ID of the node is
set to data-<folder name>, so I can link conveniently to this entry from
project notes.

The custom elisp function also installs a hook that automatically calls
org-attach-attach-mv if I try to file the template without having added
any attachment, so I do not forget this.

I use this second template when I have to attach a file to the entry,
because it cannot be represented in Org. This mostly applies to scanned
paper of any sort (letters, invoices, etc).

I have a shell script which I use to scan directly to PDF files (I do
not use OCR, the PDF just serves as a container for possibly multiple
scanned pages, so that browsing and printing the whole document is
convenient).

If it was an important document where I might need the original in the
future, I specify "Filed" when asked where the original went, write the
folder name/ID number in the top right corner of the document with a
pencil (in case of very important documents or certificates I use a
post-it note), and file it on top of a normal paper file folder.
When keeping the original, I do not change the folder name from its
default. Should I have to dig out the original for any reason, I can
manually execute a binary search on my  chronologically sorted file
folder(s).

I have been using this system for a few weeks now, and it has worked
great so far. Its main design goal was not simplicity of implementation,
but simplicity of use: it has to be so simple (and require as few
decisions as possible) to file something that I actually do it instead
of postponing it.

The system actually evolved along with the aforementioned shell script
for scanning while I was scanning and filing about 20 exercise sheets
(about four to twelve pages each) from the last semester to access them
conveniently when preparing for the exams.

I also noticed a while ago that very long org files become less
intimidating once you learn to love C-x n s (org-narrow-to-subtree),
which helped with my decision for one big file over many small ones. One
big file also avoids cluttering the buffer list.

- Jan,
  who really should start a blog to do more detailed write-ups of this
and similar things, because they are so much fun to write.

Re: My reference data management approach with org and emacs

[Claus Klingberg]

[sorry for the dupe, forgot to cc the list on first shot]

Jan,

thanks for sharing your setup, which got me intrigued. Is there a
place where a interested person can take a look at your (org-)
configuration-files, esp. the two template-sources you mentioned?

Thanks,
Claus


On Sat, Apr 10, 2010 at 1:31 AM, Jan Böcker <jan.boecker@jboecker.de> wrote:
> [The following text has gotten quite long. Sit comfortably and get a cup
> of your preferred drink if you want to proceed.]
>
> That is an interesting setup you describe there. I had considered
> something similar myself, but found it a hassle to come up with a file
> name for every new piece of information (although unix does allow
> everything except "/" in a file name, I want my file names to be lower
> case, short and without spaces where feasible).
>
> To paraphrase what you said, putting things into files just makes you
> loose time thinking about file names (which I also consider structure).
>
> In the end, I settled upon one big org file ("reference.org"). Each
> piece of reference information is in its own top-level node. When I want
> to find something there, I use isearch and/or search for a specific tag
> (in virtually every case, a simple isearch for one or two words is
> sufficient).
>
> This is way faster than navigating the file system! I also share your
> dislike of categories (which a strictly hierarchical file system would
> force me to use).
>
> I have two remember templates to add an entry to reference.org.
>
> The first template asks me for tags ("%^g") and a title for the
> headline. After filing it (at the top of reference.org) with C-c C-c,
> Org jumps to the location it was just filed ("%&" in the template), in
> case I want to use C-c C-c again to readjust the tags.
>
> I use this first template to keep data that can be expressed in plain
> text (including all the powerful tools Org gives me to work with plain
> text, such as outlines, links and tables).
>
> The second template is a little more complex; it calls a custom elisp
> function to do all the work.
>
> When I call this second template, I am asked for the following:
> - a title for the headline
> - a date (I mostly use this template to file scanned letters, invoices
> and the like, so it helps to be able to change the date from the default
> of today.)
> - a folder name (defaulting to YYYY-MM-DD.S, i.e. the previously
> specified date followed by a sequence number to make the folder name
> unique). Normally, I do not customize the folder name, because I only
> need to find the reference data via Org and do not need to navigate to
> it using e.g. the "open" dialog of any other program (and I do not want
> to change it in the future, which might make a folder name containing a
> date obsolete).
> - where the original went (defaults to "Trash"). This is stored as an
> attribute in the outline node.
>
> A new subfolder with the specified folder name is created in ~/org/data/
> and set up as this node's attachment directory. The ID of the node is
> set to data-<folder name>, so I can link conveniently to this entry from
> project notes.
>
> The custom elisp function also installs a hook that automatically calls
> org-attach-attach-mv if I try to file the template without having added
> any attachment, so I do not forget this.
>
> I use this second template when I have to attach a file to the entry,
> because it cannot be represented in Org. This mostly applies to scanned
> paper of any sort (letters, invoices, etc).
>
> I have a shell script which I use to scan directly to PDF files (I do
> not use OCR, the PDF just serves as a container for possibly multiple
> scanned pages, so that browsing and printing the whole document is
> convenient).
>
> If it was an important document where I might need the original in the
> future, I specify "Filed" when asked where the original went, write the
> folder name/ID number in the top right corner of the document with a
> pencil (in case of very important documents or certificates I use a
> post-it note), and file it on top of a normal paper file folder.
> When keeping the original, I do not change the folder name from its
> default. Should I have to dig out the original for any reason, I can
> manually execute a binary search on my  chronologically sorted file
> folder(s).
>
> I have been using this system for a few weeks now, and it has worked
> great so far. Its main design goal was not simplicity of implementation,
> but simplicity of use: it has to be so simple (and require as few
> decisions as possible) to file something that I actually do it instead
> of postponing it.
>
> The system actually evolved along with the aforementioned shell script
> for scanning while I was scanning and filing about 20 exercise sheets
> (about four to twelve pages each) from the last semester to access them
> conveniently when preparing for the exams.
>
> I also noticed a while ago that very long org files become less
> intimidating once you learn to love C-x n s (org-narrow-to-subtree),
> which helped with my decision for one big file over many small ones. One
> big file also avoids cluttering the buffer list.
>
> - Jan,
>  who really should start a blog to do more detailed write-ups of this
> and similar things, because they are so much fun to write.
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode
>

My reference data management approach with org and emacs

[Marcelo de Moraes Serpa]

[-- Attachment #1.1: Type: text/plain, Size: 6578 bytes --]

Amazing stuff, Jan.

From what I could understand, it aims on being a no-brainer reference
system, and indeed, my aim is capturing quickly *but* with enough
information and structure to be able to find it later. What I find happening
to me time after time, is that even though I do have a quick way to capture,
I end up not using the information later, because it is hard to find. The
reason? I have two reference baskets:
 * The wiki files (under wiki/) -- I press S-w and can type the name of a
file and start typing.
 * My GTDReference.org file. C-c C-r r, title and boby, C-c C-c to capture.

(And the GTD Inbox, but that's not reference, only data that I want to
process later).

The problem, I think, is that I end up with a bunch of files under wiki/ and
a bunch of notes in GTDReference.org. A rgrep search is often enough to find
what I want, but it is definetly not as clean, straightfoward and simple as
your system's implementation.

In an effort to make the wiki stuff more organic, I have setup howm-mode to
work with org, and while it works great to link files, it just added more
complexity. I think I will just get rid of it.

Could you share the custom elisp function and the scan shell script?

Also, how does all of this fit on your overall PIM architecture? There's
more to it? If so, then I'm ready to get another cup of coffee :)

Thank you very much for sharing all these gems.

Marcelo.


On Fri, Apr 9, 2010 at 6:31 PM, Jan Böcker <jan.boecker@jboecker.de> wrote:

> [The following text has gotten quite long. Sit comfortably and get a cup
> of your preferred drink if you want to proceed.]
>
> That is an interesting setup you describe there. I had considered
> something similar myself, but found it a hassle to come up with a file
> name for every new piece of information (although unix does allow
> everything except "/" in a file name, I want my file names to be lower
> case, short and without spaces where feasible).
>
> To paraphrase what you said, putting things into files just makes you
> loose time thinking about file names (which I also consider structure).
>
> In the end, I settled upon one big org file ("reference.org"). Each
> piece of reference information is in its own top-level node. When I want
> to find something there, I use isearch and/or search for a specific tag
> (in virtually every case, a simple isearch for one or two words is
> sufficient).
>
> This is way faster than navigating the file system! I also share your
> dislike of categories (which a strictly hierarchical file system would
> force me to use).
>
> I have two remember templates to add an entry to reference.org.
>
> The first template asks me for tags ("%^g") and a title for the
> headline. After filing it (at the top of reference.org) with C-c C-c,
> Org jumps to the location it was just filed ("%&" in the template), in
> case I want to use C-c C-c again to readjust the tags.
>
> I use this first template to keep data that can be expressed in plain
> text (including all the powerful tools Org gives me to work with plain
> text, such as outlines, links and tables).
>
> The second template is a little more complex; it calls a custom elisp
> function to do all the work.
>
> When I call this second template, I am asked for the following:
> - a title for the headline
> - a date (I mostly use this template to file scanned letters, invoices
> and the like, so it helps to be able to change the date from the default
> of today.)
> - a folder name (defaulting to YYYY-MM-DD.S, i.e. the previously
> specified date followed by a sequence number to make the folder name
> unique). Normally, I do not customize the folder name, because I only
> need to find the reference data via Org and do not need to navigate to
> it using e.g. the "open" dialog of any other program (and I do not want
> to change it in the future, which might make a folder name containing a
> date obsolete).
> - where the original went (defaults to "Trash"). This is stored as an
> attribute in the outline node.
>
> A new subfolder with the specified folder name is created in ~/org/data/
> and set up as this node's attachment directory. The ID of the node is
> set to data-<folder name>, so I can link conveniently to this entry from
> project notes.
>
> The custom elisp function also installs a hook that automatically calls
> org-attach-attach-mv if I try to file the template without having added
> any attachment, so I do not forget this.
>
> I use this second template when I have to attach a file to the entry,
> because it cannot be represented in Org. This mostly applies to scanned
> paper of any sort (letters, invoices, etc).
>
> I have a shell script which I use to scan directly to PDF files (I do
> not use OCR, the PDF just serves as a container for possibly multiple
> scanned pages, so that browsing and printing the whole document is
> convenient).
>
> If it was an important document where I might need the original in the
> future, I specify "Filed" when asked where the original went, write the
> folder name/ID number in the top right corner of the document with a
> pencil (in case of very important documents or certificates I use a
> post-it note), and file it on top of a normal paper file folder.
> When keeping the original, I do not change the folder name from its
> default. Should I have to dig out the original for any reason, I can
> manually execute a binary search on my  chronologically sorted file
> folder(s).
>
> I have been using this system for a few weeks now, and it has worked
> great so far. Its main design goal was not simplicity of implementation,
> but simplicity of use: it has to be so simple (and require as few
> decisions as possible) to file something that I actually do it instead
> of postponing it.
>
> The system actually evolved along with the aforementioned shell script
> for scanning while I was scanning and filing about 20 exercise sheets
> (about four to twelve pages each) from the last semester to access them
> conveniently when preparing for the exams.
>
> I also noticed a while ago that very long org files become less
> intimidating once you learn to love C-x n s (org-narrow-to-subtree),
> which helped with my decision for one big file over many small ones. One
> big file also avoids cluttering the buffer list.
>
> - Jan,
>  who really should start a blog to do more detailed write-ups of this
> and similar things, because they are so much fun to write.
>

[-- Attachment #1.2: Type: text/html, Size: 7472 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: My reference data management approach with org and emacs

[Jan Böcker]

I have published a more detailed description of my setup, including the
source code, here:

http://www.jboecker.de/2010/04/14/general-reference-filing-with-org-mode.html

Thanks to Claus and Marcelo for the (off-list) nudge to do this.

(It's getting late, so I finally stopped fiddling with the layout and
changing phrases.)

- Jan

Re: My reference data management approach with org and emacs

[Alexander Poslavsky]

[-- Attachment #1.1: Type: text/plain, Size: 428 bytes --]

On Wed, Apr 14, 2010 at 10:29 PM, Jan Böcker <jan.boecker@jboecker.de>wrote:

> I have published a more detailed description of my setup, including the
> source code, here:
>
>
> http://www.jboecker.de/2010/04/14/general-reference-filing-with-org-mode.html
>
> Wow, this is a very nice setup, and a good way to see how org-attachments
can be used in practice. Maybe a link to worg/tutorials would be ok?

bye!
alex

[-- Attachment #1.2: Type: text/html, Size: 806 bytes --]

[-- Attachment #2: Type: text/plain, Size: 201 bytes --]

_______________________________________________
Emacs-orgmode mailing list
Please use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: My reference data management approach with org and emacs

[Carsten Dominik]

On Apr 15, 2010, at 12:09 PM, Alexander Poslavsky wrote:

> On Wed, Apr 14, 2010 at 10:29 PM, Jan Böcker  
> <jan.boecker@jboecker.de> wrote:
> I have published a more detailed description of my setup, including  
> the
> source code, here:
>
> http://www.jboecker.de/2010/04/14/general-reference-filing-with-org-mode.html
>
> Wow, this is a very nice setup, and a good way to see how org- 
> attachments can be used in practice. Maybe a link to worg/tutorials  
> would be ok?

Definitely.  Can please someone make that link?

- Carsten

>
> bye!
> alex
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: My reference data management approach with org and emacs

[Matt Lundin]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> On Apr 15, 2010, at 12:09 PM, Alexander Poslavsky wrote:
>
>> On Wed, Apr 14, 2010 at 10:29 PM, Jan Böcker
>> <jan.boecker@jboecker.de> wrote:
>> I have published a more detailed description of my setup, including
>> the
>> source code, here:
>>
>> http://www.jboecker.de/2010/04/14/general-reference-filing-with-org-mode.html
>>
>> Wow, this is a very nice setup, and a good way to see how org-
>> attachments can be used in practice. Maybe a link to worg/tutorials
>> would be ok?
>
> Definitely.  Can please someone make that link?
>

Done.

- Matt

Re: My reference data management approach with org and emacs

[Sandro Giessl]

Hi Jan,

thanks a lot for your description. I've actually been using a similar
setup of one big and unstructured reference/snippet file and a script for
scanning paper documents away without spending too much time thinking
about where to put the resulting files. The problem was that the scanned
documents were accumulating in a scan-inbox directory, waiting to be
filled into a file hierarchy... too much overhead for little additional
gain when digging for reference documents again.

Your setup seems to fill the gap between reference paper and org-mode,
so I'm happy you shared it. =)

One tiny remark to your defcustom declaration:

(defcustom jb/filing-attachment-dir nil
  "The directory in which individual attachment dirs are created."
  :type 'string)

resulted in an error message (custom-variable-mark-to-save: Symbol's
value as variable is void: nilasdf) when trying to save the customize
buffer. Customize was trying to save this value to ~/.emacs without ""
around it. Using "" instead of nil in the declaration fixed this for me.

'C-c r p' also behaves a bit odd as long as jb/filing-attachment-dir
isn't initialized yet, but I don't care.

Best regards,
Sandro