Tips on maintaining history in Org Mode

Tips on maintaining history in Org Mode

[David Masterson]

There are many ways of maintaining history in a group of Org files:
1. Archive within a file
2. Archive to a separate (archive) file
3. Special TODO types for history
4. Special TAG types for history
5. etc.

My question is, if you have meetings/phone calls as TODOs, what is the
preferred way to handle when they move into history so that, *much*
later, you can easily produce a list of all of the meetings/phone calls
with dates and times of them?  The issue (I think) is, when you mark the
TODO as DONE, you lose the info of what the TODO was originally.

Suggestions
-- 
David Masterson

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

David Masterson <dsmasterson92630@outlook.com> writes:
> My question is, if you have meetings/phone calls as TODOs, what is the
> preferred way to handle when they move into history so that, *much*
> later, you can easily produce a list of all of the meetings/phone calls
> with dates and times of them?  The issue (I think) is, when you mark the
> TODO as DONE, you lose the info of what the TODO was originally.

See Org manual :: 5.3 Progress Logging

Re: Tips on maintaining history in Org Mode

[Tim Cross]

David Masterson <dsmasterson92630@outlook.com> writes:

> There are many ways of maintaining history in a group of Org files:
> 1. Archive within a file
> 2. Archive to a separate (archive) file
> 3. Special TODO types for history
> 4. Special TAG types for history
> 5. etc.
>
> My question is, if you have meetings/phone calls as TODOs, what is the
> preferred way to handle when they move into history so that, *much*
> later, you can easily produce a list of all of the meetings/phone calls
> with dates and times of them?  The issue (I think) is, when you mark the
> TODO as DONE, you lose the info of what the TODO was originally.
>

A lot will depend on your requirements.

For me, my TODOs are setup so that they record a date stamp for when
they were added and whenever they change state e.g. started, done,
delegated etc.

For non-TODO items, I will often put an inactive timestamp in the
heading title.

I also make extensive use of the ability to add timestamp entries as
part of capture templates - for exmaple, my notes and even the file of
bookmakrs (RUIs) I have.


--
Tim Cross

Re: Tips on maintaining history in Org Mode

[Eric S Fraga]

My approach is simple.  For TODO items, I archive to separate file when
done.  That file is easily searchable, e.g. using C-c /.

I keep both the original file and the archive file under revision
control, just in case.

> The issue (I think) is, when you mark the TODO as DONE, you lose the
> info of what the TODO was originally.

What info is lost?  In fact, if you log state changes, all the
information is there.

By the way, if you have recurring items, one package that you might find
useful is org-recur which is on ELPA.

-- 
: Eric S Fraga via Emacs 28.0.50, Org release_9.4.4-231-gf46925

Re: Tips on maintaining history in Org Mode

[TRS-80]

On 2021-02-26 01:22, David Masterson wrote:
> There are many ways of maintaining history in a group of Org files:
> 1. Archive within a file
> 2. Archive to a separate (archive) file
> 3. Special TODO types for history
> 4. Special TAG types for history
> 5. etc.
> 
> My question is, if you have meetings/phone calls as TODOs, what is the
> preferred way to handle when they move into history so that, *much*
> later, you can easily produce a list of all of the meetings/phone calls
> with dates and times of them?  The issue (I think) is, when you mark 
> the
> TODO as DONE, you lose the info of what the TODO was originally.
> 
> Suggestions

I agree with what others already said about logging state changes with
timestamps.  I do the same and find this an extremely handy place to
put "metadata about the task, or reason that it changed" and keep it
separate/hidden from the main body of the task (which in my mind
should only contain info about the subject of the headline / task
itself).

However when you say "history" I suspect you mean "archive" and this
is something I have been thinking about a long time as well.  Because
I think the current way that Orgmode handles this is a bit naive /
simplistic.  Well, at least there is the option to create your own
archival function, which I suspect I will do at some point.  In fact I
have been thinking about the best way to do that for some time
already.

Cheers,
TRS-80

Re: Tips on maintaining history in Org Mode

[Samuel Wales]

org does indeed have a lot of related features, maybe too many even.
here is some of what i do.

- if i doneify, it means i will likely not need to search for it.
archived to a file.
  [currently the archiver is so slow i can't use it]
  CLOSED: [2012-11-08 Thu 19:40]
- state logging for repeaters in logbook
- your question: notes to keep around like this
  ***** LOG [2021-02-26 Fri 14:18] talked to bill gates; he uses linux now
  these get sorted at bottom by timestamp using user-defined
  this is the main thing
- bare inactive free form as needed

-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Tips on maintaining history in Org Mode

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson92630@outlook.com> writes:
>> My question is, if you have meetings/phone calls as TODOs, what is the
>> preferred way to handle when they move into history so that, *much*
>> later, you can easily produce a list of all of the meetings/phone calls
>> with dates and times of them?  The issue (I think) is, when you mark the
>> TODO as DONE, you lose the info of what the TODO was originally.
>
> See Org manual :: 5.3 Progress Logging

Interesting, but then how do you get the list?  I mean is there an
agenda to use?
-- 
David Masterson

Re: Tips on maintaining history in Org Mode

[David Masterson]

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson92630@outlook.com> writes:
>
>> There are many ways of maintaining history in a group of Org files:
>> 1. Archive within a file
>> 2. Archive to a separate (archive) file
>> 3. Special TODO types for history
>> 4. Special TAG types for history
>> 5. etc.
>>
>> My question is, if you have meetings/phone calls as TODOs, what is the
>> preferred way to handle when they move into history so that, *much*
>> later, you can easily produce a list of all of the meetings/phone calls
>> with dates and times of them?  The issue (I think) is, when you mark the
>> TODO as DONE, you lose the info of what the TODO was originally.
>
> A lot will depend on your requirements.
>
> For me, my TODOs are setup so that they record a date stamp for when
> they were added and whenever they change state e.g. started, done,
> delegated etc.

So, you use progress logging.

> For non-TODO items, I will often put an inactive timestamp in the
> heading title.

Do your headings become busy?

What would you use to then make a list of all meetings you had last year?

-- 
David Masterson

Re: Tips on maintaining history in Org Mode

[Tim Cross]

David Masterson <dsmasterson92630@outlook.com> writes:

> Tim Cross <theophilusx@gmail.com> writes:
>
>> David Masterson <dsmasterson92630@outlook.com> writes:
>>
>> For me, my TODOs are setup so that they record a date stamp for when
>> they were added and whenever they change state e.g. started, done,
>> delegated etc.
>
> So, you use progress logging.

Yes.

>
>> For non-TODO items, I will often put an inactive timestamp in the
>> heading title.
>
> Do your headings become busy?
>

Some would feel they are 'busy'. I always put the timestamp at the
beginning of the heading, so there is a regular pattern (not much
different from the leading heading stars) and I've just got use to it,
so I don't really see it now.

> What would you use to then make a list of all meetings you had last year?

For me, archiving is about data I'm unlikely to need again, but just in
case I do, it is in the archive. I rarely look at my archives. However,
when I do archive, I will usually archive into a 'year' file. So, to
find all the meetings held in 2015, I would open that archive file and
search for entries with the tag MEETING (I also have a tag for PHONE).

--
Tim Cross

Re: Tips on maintaining history in Org Mode

[David Masterson]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> My approach is simple.  For TODO items, I archive to separate file when
> done.  That file is easily searchable, e.g. using C-c /.

Ah! org-occur! That's something forgot about and looks useful.

> I keep both the original file and the archive file under revision
> control, just in case.

I do that, too.

>> The issue (I think) is, when you mark the TODO as DONE, you lose the
>> info of what the TODO was originally.
>
> What info is lost?  In fact, if you log state changes, all the
> information is there.

I see how that workss now with org-occur.

> By the way, if you have recurring items, one package that you might find
> useful is org-recur which is on ELPA.

Thanks

-- 
David Masterson

Re: Tips on maintaining history in Org Mode

[David Masterson]

Tim Cross <theophilusx@gmail.com> writes:

> David Masterson <dsmasterson92630@outlook.com> writes:
>
>> What would you use to then make a list of all meetings you had last year?
>
> For me, archiving is about data I'm unlikely to need again, but just in
> case I do, it is in the archive. I rarely look at my archives. However,
> when I do archive, I will usually archive into a 'year' file. So, to
> find all the meetings held in 2015, I would open that archive file and
> search for entries with the tag MEETING (I also have a tag for PHONE).

Got it.  Thanks.
-- 
David Masterson

Re: Tips on maintaining history in Org Mode

[Samuel Wales]

note that there is an issue when you try to name your archive files
using years like computer-2000.org_archive.  it can take seconds to
find-file big files so it is understandable to want to name files like
that.

however, if you change the name of an archive file, it will not be
found by org when you tell it to do certain things with archive files.
some code in org derives the basename to search from the basename of
the original file like computer,org.

in principle, maybe org could allow year suffixes/prefixes or it could
search all archive files in dirs that have org files.

perhaps also changing org-archive-file-header-format to allow a format
thingie for a timestamp would allow you to take parts of an archive
file and move them into one per year without having to put the date in
each archived entry.


On 2/26/21, David Masterson <dsmasterson92630@outlook.com> wrote:
> Tim Cross <theophilusx@gmail.com> writes:
>
>> David Masterson <dsmasterson92630@outlook.com> writes:
>>
>>> What would you use to then make a list of all meetings you had last
>>> year?
>>
>> For me, archiving is about data I'm unlikely to need again, but just in
>> case I do, it is in the archive. I rarely look at my archives. However,
>> when I do archive, I will usually archive into a 'year' file. So, to
>> find all the meetings held in 2015, I would open that archive file and
>> search for entries with the tag MEETING (I also have a tag for PHONE).
>
> Got it.  Thanks.
> --
> David Masterson
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

David Masterson <dsmasterson92630@outlook.com> writes:

> Interesting, but then how do you get the list?  I mean is there an
> agenda to use?

Generally yes, you can use agenda. Or you can use sparse tree (more manual).
For agenda, if you customise org-log-done, you can use
org-agenda-log-mode ("v l" or "v L" in an agenda buffer). You can add
archived items as well with "v a" or/and "v A".

Just org-agenda-log-mode will show everything, not just calls. Narrowing
to calls only will depend on how you define a todo, which is a call.

If you use something like PHONE or CALL todo keywords, it might be
slightly tricky. You will need to customise org-todo-keywords, so that
your CALL->DONE changes are recorded (see the org-todo-keywords
docstring). You will also need to filter displayed items in agenda by
regexp involving the keyword you use to define the call.

An easier way could be marking your calls with a tag. Then, you can
filter your org-agenda-log by that tag to show only calls.

Hope it helps.

Best,
Ihor

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

Samuel Wales <samologist@gmail.com> writes:

> perhaps also changing org-archive-file-header-format to allow a format
> thingie for a timestamp would allow you to take parts of an archive
> file and move them into one per year without having to put the date in
> each archived entry.

FYI: I have implemented automatic per-year archiving, which is correctly
handled by other org commands in my personal config:
https://github.com/yantar92/emacs-config/blob/master/config.org#archiving

Best,
Ihor

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

Samuel Wales <samologist@gmail.com> writes:
>   [currently the archiver is so slow i can't use it]

Are your existing archives very big (few Mbs)? If so, you may try to
speed up the archiving using feature/org-fold branch [1]. If that is not
enough, I recommend splitting archives on yearly basis [2] or disabling
font-lock in archive files.

Best,
Ihor

[1] https://github.com/yantar92/org
[2] https://github.com/yantar92/emacs-config/blob/master/config.org#archiving

Re: Tips on maintaining history in Org Mode

[Samuel Wales]

Hi Ihor,

it never occurred to me that bulk archiving could be sped up by
changing anything about the archive files.  i assumed that archiving
worked by appending to those files, so once the initial find-file was
performed, there should be no additional slowness.

yet i tried a new file with no archive file, and bulk archiving took
less than a second for 9 entries.  before it would take idk a minute.

i will keep in mind disabling font lock in archive files.  any
suggested code for that?  your config is impressively [and for me
dauntingly] comprehensive.  the Archiving anchor does not seem to have
a referent.  I searched for archiving, but didn't seem to find what
you were probably pointing me to.

thanks.

Re: Tips on maintaining history in Org Mode

[Ihor Radchenko]

Samuel Wales <samologist@gmail.com> writes:

> Hi Ihor,
>
> it never occurred to me that bulk archiving could be sped up by
> changing anything about the archive files.  i assumed that archiving
> worked by appending to those files, so once the initial find-file was
> performed, there should be no additional slowness.

Disregarding the initial find-file (which may also be very slow on large
archives), the slowness is happening because many org-mode commands are
using line-motion functions that respect visibility. Those commands
become slow when there are many folded elements (see [1] for technical
details why). So, many org commands tend to lag on large archives.

The lags can be solved in several ways:
1. Reduce the archive file size
2. Use optimised folding mechanism [1] (this will speed up org-mode in
general as well)
3. (untested) Put #+STARTUP: showeverything at the beginning of the
   archives, so that nothing is going to be folded

> i will keep in mind disabling font lock in archive files.  any
> suggested code for that?

Note that it will mostly affect find-file performance. To disable
font-lock, you will need to customise font-lock-global-modes, so that it
does not auto-start in org-mode. The value should be something like
'(not org-mode), but check the current value - you may have other
customisation in place.

Then, you will need to have a custom org-mode-hook that activates
font-lock only when the org file is not the archive file. Something like

(defun yant/font-lock-activate-maybe ()
"Activate font-lock in non-archive org buffers."
(when (org-all-archive-files) (font-lock-mode +1)))
(add-hook 'org-mode-hook #'yant/font-lock-activate-maybe)

> your config is impressively [and for me
> dauntingly] comprehensive.  the Archiving anchor does not seem to have
> a referent.  I searched for archiving, but didn't seem to find what
> you were probably pointing me to.

Sorry, the config is actually not yet formatted for public use. You can
search for the code block containing "defun org-archive--compute-location". 
You will need that code block and the following code block.

[1] https://github.com/yantar92/org

Best,
Ihor

Re: Tips on maintaining history in Org Mode

[David Masterson]

Ihor Radchenko <yantar92@gmail.com> writes:

> David Masterson <dsmasterson92630@outlook.com> writes:
>
>> Interesting, but then how do you get the list?  I mean is there an
>> agenda to use?
>
> Generally yes, you can use agenda. Or you can use sparse tree (more manual).
> For agenda, if you customise org-log-done, you can use
> org-agenda-log-mode ("v l" or "v L" in an agenda buffer). You can add
> archived items as well with "v a" or/and "v A".
>
> Just org-agenda-log-mode will show everything, not just calls. Narrowing
> to calls only will depend on how you define a todo, which is a call.
>
> If you use something like PHONE or CALL todo keywords, it might be
> slightly tricky. You will need to customise org-todo-keywords, so that
> your CALL->DONE changes are recorded (see the org-todo-keywords
> docstring). You will also need to filter displayed items in agenda by
> regexp involving the keyword you use to define the call.
>
> An easier way could be marking your calls with a tag. Then, you can
> filter your org-agenda-log by that tag to show only calls.
>
> Hope it helps.

Nice! Thanks.
-- 
David Masterson