emacs-orgmode@gnu.org archives
 help / color / mirror / code / Atom feed
* archiving speed [was Re: Tips on maintaining history in Org Mode]
@ 2021-03-01  4:22 Samuel Wales
  2021-03-01  6:02 ` Ihor Radchenko
  2021-08-11  1:08 ` Samuel Wales
  0 siblings, 2 replies; 13+ messages in thread
From: Samuel Wales @ 2021-03-01  4:22 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson

thank you for your detaild reply.

more below.

On 2/28/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> details why). So, many org commands tend to lag on large archives.

that makes sense.  but why would appending to an archive as the result
of bulk archiving lag?  if the problem is large archive files, which
i'd bet is the case for a lot of users and not just me, then could org
in principle be changed so that all it does is append?  thus not lag?
like, build the entry in a temporary buffer?

as i see it, having more than one archive file per org file is good
for speed, but doesn't work in existing org, because iirc e.g. v A in
the agenda goes org agenda file -> corresponding archive file and will
miss the archive files that do not have a corresponding org file with
exactly the same basename sans extension.

i'd be ok with released org either allowing hte user to make
year-based archives by having all of org recognize them, or my just
append thing above.  maybe i am missing something.

> The lags can be solved in several ways:
> 1. Reduce the archive file size

this implies to me e.g. year-based archives, which would fail the v A
test iiuc.  thus needed extra code.

> 2. Use optimised folding mechanism [1] (this will speed up org-mode in
> general as well)

i look forward to this filtering down to maint.  :]  [i used to follow
master but too much for me now for health reasons.]

> 3. (untested) Put #+STARTUP: showeverything at the beginning of the
>    archives, so that nothing is going to be folded

good idea.  my included-by-agenda archive files do seem to be in
showeveryting mode already for some reason.  but perhaps not when bulk
archiving.

would it be a silly idea for an fr that org make this an option for
bulk archiving?  hmm or for archive files in general?

>> i will keep in mind disabling font lock in archive files.  any
>> suggested code for that?
>
> Note that it will mostly affect find-file performance. To disable

if so, then i figure it's a one-time thing per file so no big deal.
but thanks for hte font lock stuff i didnt' know about.

> Sorry, the config is actually not yet formatted for public use. You can
> search for the code block containing "defun org-archive--compute-location".

firefox find does not seem to find it.

>
> You will need that code block and the following code block.
>
> [1] https://github.com/yantar92/org
>
> Best,
> Ihor
>
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-03-01  4:22 archiving speed [was Re: Tips on maintaining history in Org Mode] Samuel Wales
@ 2021-03-01  6:02 ` Ihor Radchenko
  2021-08-11  1:08 ` Samuel Wales
  1 sibling, 0 replies; 13+ messages in thread
From: Ihor Radchenko @ 2021-03-01  6:02 UTC (permalink / raw)
  To: Samuel Wales; +Cc: emacs-orgmode, David Masterson

Samuel Wales <samologist@gmail.com> writes:

> that makes sense.  but why would appending to an archive as the result
> of bulk archiving lag?  if the problem is large archive files, which
> i'd bet is the case for a lot of users and not just me, then could org
> in principle be changed so that all it does is append?  thus not lag?
> like, build the entry in a temporary buffer?

Looking into the source code, the culprit appears to be the call to
org-show-all inside org-archive-subtree. The call to org-show-all on
master calls org-cycle-hide-drawers, which goes through every single
drawer (= every archived heading in archive file) and folds it manually
(just to unfold it later). Do not ask.

> as i see it, having more than one archive file per org file is good
> for speed, but doesn't work in existing org, because iirc e.g. v A in
> the agenda goes org agenda file -> corresponding archive file and will
> miss the archive files that do not have a corresponding org file with
> exactly the same basename sans extension.

The code from my config takes care about this. v A will work. I plan to
suggest it for master in future, but I am focused on the org-fold branch
for now. If you wish, feel free to propose a patch based on this code.

>> 3. (untested) Put #+STARTUP: showeverything at the beginning of the
>>    archives, so that nothing is going to be folded
>
> good idea.  my included-by-agenda archive files do seem to be in
> showeveryting mode already for some reason.  but perhaps not when bulk
> archiving.
>
> would it be a silly idea for an fr that org make this an option for
> bulk archiving?  hmm or for archive files in general?

Upon reviewing the source of org-archive-subtree, this should not be
needed. Actually, org-mode already disables startup visibility and
various hooks when opening archives. As you can see, it is not
sufficient, since org-archive shoots its own leg later by changing
visibility every time you archive a subtree.

>> Sorry, the config is actually not yet formatted for public use. You can
>> search for the code block containing "defun org-archive--compute-location".
>
> firefox find does not seem to find it.

This is odd. All I can suggest then is cloning the repo and searching
using Emacs. Emacs is more reliable when opening org files ;)

git clone https://github.com/yantar92/emacs-config
emacs emacs-config/config.org

Best,
Ihor


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-03-01  4:22 archiving speed [was Re: Tips on maintaining history in Org Mode] Samuel Wales
  2021-03-01  6:02 ` Ihor Radchenko
@ 2021-08-11  1:08 ` Samuel Wales
  2021-08-11  4:13   ` Ihor Radchenko
  1 sibling, 1 reply; 13+ messages in thread
From: Samuel Wales @ 2021-08-11  1:08 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson

i have not bulk archived in a year or two, because it is too slow for
me.  i am curious why, when appending to an archive file,
append-to-file or write-region are not used, to merely put the entries
there instead of loading the archive file into emacs?

[this is the case even with zero-size archive files; after a few
entries it slows down.]


On 2/28/21, Samuel Wales <samologist@gmail.com> wrote:
> thank you for your detaild reply.
>
> more below.
>
> On 2/28/21, Ihor Radchenko <yantar92@gmail.com> wrote:
>> details why). So, many org commands tend to lag on large archives.
>
> that makes sense.  but why would appending to an archive as the result
> of bulk archiving lag?  if the problem is large archive files, which
> i'd bet is the case for a lot of users and not just me, then could org
> in principle be changed so that all it does is append?  thus not lag?
> like, build the entry in a temporary buffer?
>
> as i see it, having more than one archive file per org file is good
> for speed, but doesn't work in existing org, because iirc e.g. v A in
> the agenda goes org agenda file -> corresponding archive file and will
> miss the archive files that do not have a corresponding org file with
> exactly the same basename sans extension.
>
> i'd be ok with released org either allowing hte user to make
> year-based archives by having all of org recognize them, or my just
> append thing above.  maybe i am missing something.
>
>> The lags can be solved in several ways:
>> 1. Reduce the archive file size
>
> this implies to me e.g. year-based archives, which would fail the v A
> test iiuc.  thus needed extra code.
>
>> 2. Use optimised folding mechanism [1] (this will speed up org-mode in
>> general as well)
>
> i look forward to this filtering down to maint.  :]  [i used to follow
> master but too much for me now for health reasons.]
>
>> 3. (untested) Put #+STARTUP: showeverything at the beginning of the
>>    archives, so that nothing is going to be folded
>
> good idea.  my included-by-agenda archive files do seem to be in
> showeveryting mode already for some reason.  but perhaps not when bulk
> archiving.
>
> would it be a silly idea for an fr that org make this an option for
> bulk archiving?  hmm or for archive files in general?
>
>>> i will keep in mind disabling font lock in archive files.  any
>>> suggested code for that?
>>
>> Note that it will mostly affect find-file performance. To disable
>
> if so, then i figure it's a one-time thing per file so no big deal.
> but thanks for hte font lock stuff i didnt' know about.
>
>> Sorry, the config is actually not yet formatted for public use. You can
>> search for the code block containing "defun
>> org-archive--compute-location".
>
> firefox find does not seem to find it.
>
>>
>> You will need that code block and the following code block.
>>
>> [1] https://github.com/yantar92/org
>>
>> Best,
>> Ihor
>>
>>
>
>
> --
> The Kafka Pandemic
>
> Please learn what misopathy is.
> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11  1:08 ` Samuel Wales
@ 2021-08-11  4:13   ` Ihor Radchenko
  2021-08-11  5:58     ` Samuel Wales
  0 siblings, 1 reply; 13+ messages in thread
From: Ihor Radchenko @ 2021-08-11  4:13 UTC (permalink / raw)
  To: Samuel Wales; +Cc: emacs-orgmode, David Masterson

Samuel Wales <samologist@gmail.com> writes:

> [this is the case even with zero-size archive files; after a few
> entries it slows down.]

Do you get the same behaviour with the following code?

(setq org-archive-save-context-info '(file time))

Best,
Ihor


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11  4:13   ` Ihor Radchenko
@ 2021-08-11  5:58     ` Samuel Wales
  2021-08-11  6:43       ` Ihor Radchenko
  2021-10-17 12:08       ` Ihor Radchenko
  0 siblings, 2 replies; 13+ messages in thread
From: Samuel Wales @ 2021-08-11  5:58 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson

i should clarify.  bulk archiving slows down even with /nonexistent/
(have not tried empty) archives.  as part of normal and expected
operation, bulk creates the archive for the first entry, and then
subsequent entries are added.  those get slower and slower.

i was trying to imply e.g. doing archives by year won't fix it.  only
a few entries are enough to slow it down.

i use (olpath category itags).  i will try (file time) when i can, if
that still applies.  my brain needs to be more operational.

i should mention that i did also find a bug, but was not able to
narrow it down.  it has been a while, but it was something like not
killing the original entry for one of the entries.  i was unable to
figure out what conditions needed to obtain for this to occur.


On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> Samuel Wales <samologist@gmail.com> writes:
>
>> [this is the case even with zero-size archive files; after a few
>> entries it slows down.]
>
> Do you get the same behaviour with the following code?
>
> (setq org-archive-save-context-info '(file time))
>
> Best,
> Ihor
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11  5:58     ` Samuel Wales
@ 2021-08-11  6:43       ` Ihor Radchenko
  2021-08-11 22:23         ` Samuel Wales
  2021-10-17 12:08       ` Ihor Radchenko
  1 sibling, 1 reply; 13+ messages in thread
From: Ihor Radchenko @ 2021-08-11  6:43 UTC (permalink / raw)
  To: Samuel Wales; +Cc: emacs-orgmode, David Masterson

Samuel Wales <samologist@gmail.com> writes:

> i should clarify.  bulk archiving slows down even with /nonexistent/
> (have not tried empty) archives.  as part of normal and expected
> operation, bulk creates the archive for the first entry, and then
> subsequent entries are added.  those get slower and slower.

That's what I suspected. I also see this and my suggestion helped
archiving speed in my case.

> i use (olpath category itags).  i will try (file time) when i can, if
> that still applies.  my brain needs to be more operational.

When you use category, every time you modify the original file (not the
archive!), Org mode re-calculates *all* the categories in the original
Org file. It happens for every single archived heading. If your original
Org file is large, re-calculations make things extremely slow.

Best,
Ihor


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11  6:43       ` Ihor Radchenko
@ 2021-08-11 22:23         ` Samuel Wales
  2021-08-12  0:24           ` Ihor Radchenko
  2021-08-12  3:38           ` Tim Cross
  0 siblings, 2 replies; 13+ messages in thread
From: Samuel Wales @ 2021-08-11 22:23 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson

thanks for the clarification.  are you saying that, for every archived
entry, it calculates teh category property, using the original org
file, in order to add a category property to just one archived entry?

that would certainly slow down more and more, but it sends me back to
my question about whether append to file would work.
i.e. build the single entry in a temporary buffer then write that
region to a file on disk.

On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> Samuel Wales <samologist@gmail.com> writes:
>
>> i should clarify.  bulk archiving slows down even with /nonexistent/
>> (have not tried empty) archives.  as part of normal and expected
>> operation, bulk creates the archive for the first entry, and then
>> subsequent entries are added.  those get slower and slower.
>
> That's what I suspected. I also see this and my suggestion helped
> archiving speed in my case.
>
>> i use (olpath category itags).  i will try (file time) when i can, if
>> that still applies.  my brain needs to be more operational.
>
> When you use category, every time you modify the original file (not the
> archive!), Org mode re-calculates *all* the categories in the original
> Org file. It happens for every single archived heading. If your original
> Org file is large, re-calculations make things extremely slow.
>
> Best,
> Ihor
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11 22:23         ` Samuel Wales
@ 2021-08-12  0:24           ` Ihor Radchenko
  2021-08-12  5:47             ` Samuel Wales
  2021-08-12  3:38           ` Tim Cross
  1 sibling, 1 reply; 13+ messages in thread
From: Ihor Radchenko @ 2021-08-12  0:24 UTC (permalink / raw)
  To: Samuel Wales; +Cc: emacs-orgmode, David Masterson

Samuel Wales <samologist@gmail.com> writes:

> thanks for the clarification.  are you saying that, for every archived
> entry, it calculates teh category property, using the original org
> file, in order to add a category property to just one archived entry?

Nope. It does not just calculate category for the archived entry, but
re-calculates all the category properties in the original Org file
(updating category cache).

> that would certainly slow down more and more, but it sends me back to
> my question about whether append to file would work.
> i.e. build the single entry in a temporary buffer then write that
> region to a file on disk.

Appending can indeed work if your archive location is at the end of the
file. However, it is not necessary the performance bottleneck. Certainly
not when the archive file is small.

Best,
Ihor


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11 22:23         ` Samuel Wales
  2021-08-12  0:24           ` Ihor Radchenko
@ 2021-08-12  3:38           ` Tim Cross
  2021-08-12  5:49             ` Samuel Wales
  1 sibling, 1 reply; 13+ messages in thread
From: Tim Cross @ 2021-08-12  3:38 UTC (permalink / raw)
  To: Samuel Wales; +Cc: Org-mode, Ihor Radchenko, David Masterson

[-- Attachment #1: Type: text/plain, Size: 2869 bytes --]

I think the problem with just using append to file is that it won't
preserve the shape of the file. For example, if I had a file with

* Notes
** Note 1
   blah blah
** Note 2 blah blah

* Tasks
** DONE task 1
** TODO Task 2

and I decide to archive note 1 and task 1, I would like them to both appear
under the same headings and with the same level. If the process just uses
append to file, I can have this for the first archiving i.e.

* Noes
** Note 1

* Tasks
** DONE task 1

but then later, I decide to archive note 2, if append file is used, I will
end up with

* Notes
** Note 1

* Taks
** DONE task 1

* Notes
** Note 2

which is not what I want. I want

* Notes
** Note 1
** Note 2

* Tasks
** DONE Task 1

So, if we want to preserve hierarchies in our archive files and not have
everything jumbled up together, the system has to parse the file. If you
are also using something like Categories, then even more work needs to be
odne to update the category lists.

What I tend to do is mark items with the ARCHIVE tag and leave them in the
file and then every few months, move archived data to archive files.  It
can still get slow, but I don't do it often, so it isn't too much of a
hassle.


On Thu, 12 Aug 2021 at 08:23, Samuel Wales <samologist@gmail.com> wrote:

> thanks for the clarification.  are you saying that, for every archived
> entry, it calculates teh category property, using the original org
> file, in order to add a category property to just one archived entry?
>
> that would certainly slow down more and more, but it sends me back to
> my question about whether append to file would work.
> i.e. build the single entry in a temporary buffer then write that
> region to a file on disk.
>
> On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> > Samuel Wales <samologist@gmail.com> writes:
> >
> >> i should clarify.  bulk archiving slows down even with /nonexistent/
> >> (have not tried empty) archives.  as part of normal and expected
> >> operation, bulk creates the archive for the first entry, and then
> >> subsequent entries are added.  those get slower and slower.
> >
> > That's what I suspected. I also see this and my suggestion helped
> > archiving speed in my case.
> >
> >> i use (olpath category itags).  i will try (file time) when i can, if
> >> that still applies.  my brain needs to be more operational.
> >
> > When you use category, every time you modify the original file (not the
> > archive!), Org mode re-calculates *all* the categories in the original
> > Org file. It happens for every single archived heading. If your original
> > Org file is large, re-calculations make things extremely slow.
> >
> > Best,
> > Ihor
> >
>
>
> --
> The Kafka Pandemic
>
> Please learn what misopathy is.
>
> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
>
>

-- 
regards,

Tim

--
Tim Cross

[-- Attachment #2: Type: text/html, Size: 4293 bytes --]

^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-12  0:24           ` Ihor Radchenko
@ 2021-08-12  5:47             ` Samuel Wales
  0 siblings, 0 replies; 13+ messages in thread
From: Samuel Wales @ 2021-08-12  5:47 UTC (permalink / raw)
  To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson

thank you.

i will give up on archiving categories if needed to make archiving be
practical.  will ahfe to try it as soon as i can.

On 8/11/21, Ihor Radchenko <yantar92@gmail.com> wrote:
> Samuel Wales <samologist@gmail.com> writes:
>
>> thanks for the clarification.  are you saying that, for every archived
>> entry, it calculates teh category property, using the original org
>> file, in order to add a category property to just one archived entry?
>
> Nope. It does not just calculate category for the archived entry, but
> re-calculates all the category properties in the original Org file
> (updating category cache).
>
>> that would certainly slow down more and more, but it sends me back to
>> my question about whether append to file would work.
>> i.e. build the single entry in a temporary buffer then write that
>> region to a file on disk.
>
> Appending can indeed work if your archive location is at the end of the
> file. However, it is not necessary the performance bottleneck. Certainly
> not when the archive file is small.
>
> Best,
> Ihor
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-12  3:38           ` Tim Cross
@ 2021-08-12  5:49             ` Samuel Wales
  2021-08-12  5:56               ` Tim Cross
  0 siblings, 1 reply; 13+ messages in thread
From: Samuel Wales @ 2021-08-12  5:49 UTC (permalink / raw)
  To: Tim Cross; +Cc: Org-mode, Ihor Radchenko, David Masterson

what is the current status of hierarchy in archive files?  surely they
don't deal with updating categories and updating hierarchy structure
[sounds brittle and syncy]?  i'm thinking it isn't hierarchical at
present, except when you have a doneified task with children?


On 8/11/21, Tim Cross <theophilusx@gmail.com> wrote:
> I think the problem with just using append to file is that it won't
> preserve the shape of the file. For example, if I had a file with
>
> * Notes
> ** Note 1
>    blah blah
> ** Note 2 blah blah
>
> * Tasks
> ** DONE task 1
> ** TODO Task 2
>
> and I decide to archive note 1 and task 1, I would like them to both appear
> under the same headings and with the same level. If the process just uses
> append to file, I can have this for the first archiving i.e.
>
> * Noes
> ** Note 1
>
> * Tasks
> ** DONE task 1
>
> but then later, I decide to archive note 2, if append file is used, I will
> end up with
>
> * Notes
> ** Note 1
>
> * Taks
> ** DONE task 1
>
> * Notes
> ** Note 2
>
> which is not what I want. I want
>
> * Notes
> ** Note 1
> ** Note 2
>
> * Tasks
> ** DONE Task 1
>
> So, if we want to preserve hierarchies in our archive files and not have
> everything jumbled up together, the system has to parse the file. If you
> are also using something like Categories, then even more work needs to be
> odne to update the category lists.
>
> What I tend to do is mark items with the ARCHIVE tag and leave them in the
> file and then every few months, move archived data to archive files.  It
> can still get slow, but I don't do it often, so it isn't too much of a
> hassle.
>
>
> On Thu, 12 Aug 2021 at 08:23, Samuel Wales <samologist@gmail.com> wrote:
>
>> thanks for the clarification.  are you saying that, for every archived
>> entry, it calculates teh category property, using the original org
>> file, in order to add a category property to just one archived entry?
>>
>> that would certainly slow down more and more, but it sends me back to
>> my question about whether append to file would work.
>> i.e. build the single entry in a temporary buffer then write that
>> region to a file on disk.
>>
>> On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote:
>> > Samuel Wales <samologist@gmail.com> writes:
>> >
>> >> i should clarify.  bulk archiving slows down even with /nonexistent/
>> >> (have not tried empty) archives.  as part of normal and expected
>> >> operation, bulk creates the archive for the first entry, and then
>> >> subsequent entries are added.  those get slower and slower.
>> >
>> > That's what I suspected. I also see this and my suggestion helped
>> > archiving speed in my case.
>> >
>> >> i use (olpath category itags).  i will try (file time) when i can, if
>> >> that still applies.  my brain needs to be more operational.
>> >
>> > When you use category, every time you modify the original file (not the
>> > archive!), Org mode re-calculates *all* the categories in the original
>> > Org file. It happens for every single archived heading. If your
>> > original
>> > Org file is large, re-calculations make things extremely slow.
>> >
>> > Best,
>> > Ihor
>> >
>>
>>
>> --
>> The Kafka Pandemic
>>
>> Please learn what misopathy is.
>>
>> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
>>
>>
>
> --
> regards,
>
> Tim
>
> --
> Tim Cross
>


-- 
The Kafka Pandemic

Please learn what misopathy is.
https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html


^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-12  5:49             ` Samuel Wales
@ 2021-08-12  5:56               ` Tim Cross
  0 siblings, 0 replies; 13+ messages in thread
From: Tim Cross @ 2021-08-12  5:56 UTC (permalink / raw)
  To: Samuel Wales; +Cc: Org-mode, Ihor Radchenko, David Masterson


It is for me. However, this all depends on how you configure things.

In my case, I have a standard structure for my org files. I have a
different org file for each 'topic' and each org file has headings for

* Tasks

* Notes

* Resources

* Comms

* General

and each of those headings has a property list with an ARCHIVE property
which looks like %s_archive::<heading> i.e. %s_archive::* Tasks, so when
I archive a tree/subtree is is placed under the heading according to the
ARCHIVE property for the tree it comes from.

almost all data I enter into org files comes via a capture template. All
captured data initially goes into a refile.org file and then I refile to
the appropriate topic org file each morning.

For completed tasks, I will usually marked them with an ARCHIVE tag soon
after they have been completed and then every 6 months or so, I will
archive into an archive file (where heading hierarchies are retained).
This is most common with completed tasks. I have a custom agenda which
shows tasks broken up into "Completed", "In Progress", "Next" and
"Backlog". I will archive the DONE items so that the "Completed" list in
the agenda does ot grow too large.

Every 12 months I move the archive files to an archive folder where they
are renamed to include the year in the filename.

I don't find archiving terribly slow. this is mainly because few of my
org files are particularly large (because they are broken up into
topics) and because I move older archives out into an archive directory
after 12 months. It is very rare that I need to go digging around in
archive files (either current or older year archives).

Of course this could all change down the track. My org files are slowly
getting larger and I expect at some point, I will hit a tipping point
where things become slow. So far, even with the larger files (around
5Mb) performance is fine. I also don't put 'everything' in the org file.
If I have another file of data, I will just link to that from the org
file rather than have that data actually reside in the org file. SO for
example, my org-mode.org file has a lot of links to interesting messages
from the org list, but the messages themselves are all in my mu4e
maildir folders.

My main point was that because configuration like mine exist, simply
appending archived items to the archive file simply would not work. I
like having my archive records in a similar 'shape' to my normal org
files because when I do need to dig into the archive, I don't want to
have to go through the whole file looking for something. I generally
know if I'm looking for an old task, note, general entry or comms record
and it is handly to know I only have to look in that section of the
file. 

This is one of the big challenges for org mode. Because it is so
flexible and people take advantage of that flexibility, what may appear
like a simple way to solve an issue often ends up being far more complex
than it initially seemed. If, for exmaple, you could not archive based
on heading, date, etc, just appending entries would probably work fine.
However, as the archviing policy might be more complex, org needs to
examine/parse the archive file to work out where to insert the archived
entry.

Tim

Samuel Wales <samologist@gmail.com> writes:

> what is the current status of hierarchy in archive files?  surely they
> don't deal with updating categories and updating hierarchy structure
> [sounds brittle and syncy]?  i'm thinking it isn't hierarchical at
> present, except when you have a doneified task with children?
>
>
> On 8/11/21, Tim Cross <theophilusx@gmail.com> wrote:
>> I think the problem with just using append to file is that it won't
>> preserve the shape of the file. For example, if I had a file with
>>
>> * Notes
>> ** Note 1
>>    blah blah
>> ** Note 2 blah blah
>>
>> * Tasks
>> ** DONE task 1
>> ** TODO Task 2
>>
>> and I decide to archive note 1 and task 1, I would like them to both appear
>> under the same headings and with the same level. If the process just uses
>> append to file, I can have this for the first archiving i.e.
>>
>> * Noes
>> ** Note 1
>>
>> * Tasks
>> ** DONE task 1
>>
>> but then later, I decide to archive note 2, if append file is used, I will
>> end up with
>>
>> * Notes
>> ** Note 1
>>
>> * Taks
>> ** DONE task 1
>>
>> * Notes
>> ** Note 2
>>
>> which is not what I want. I want
>>
>> * Notes
>> ** Note 1
>> ** Note 2
>>
>> * Tasks
>> ** DONE Task 1
>>
>> So, if we want to preserve hierarchies in our archive files and not have
>> everything jumbled up together, the system has to parse the file. If you
>> are also using something like Categories, then even more work needs to be
>> odne to update the category lists.
>>
>> What I tend to do is mark items with the ARCHIVE tag and leave them in the
>> file and then every few months, move archived data to archive files.  It
>> can still get slow, but I don't do it often, so it isn't too much of a
>> hassle.
>>
>>
>> On Thu, 12 Aug 2021 at 08:23, Samuel Wales <samologist@gmail.com> wrote:
>>
>>> thanks for the clarification.  are you saying that, for every archived
>>> entry, it calculates teh category property, using the original org
>>> file, in order to add a category property to just one archived entry?
>>>
>>> that would certainly slow down more and more, but it sends me back to
>>> my question about whether append to file would work.
>>> i.e. build the single entry in a temporary buffer then write that
>>> region to a file on disk.
>>>
>>> On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote:
>>> > Samuel Wales <samologist@gmail.com> writes:
>>> >
>>> >> i should clarify.  bulk archiving slows down even with /nonexistent/
>>> >> (have not tried empty) archives.  as part of normal and expected
>>> >> operation, bulk creates the archive for the first entry, and then
>>> >> subsequent entries are added.  those get slower and slower.
>>> >
>>> > That's what I suspected. I also see this and my suggestion helped
>>> > archiving speed in my case.
>>> >
>>> >> i use (olpath category itags).  i will try (file time) when i can, if
>>> >> that still applies.  my brain needs to be more operational.
>>> >
>>> > When you use category, every time you modify the original file (not the
>>> > archive!), Org mode re-calculates *all* the categories in the original
>>> > Org file. It happens for every single archived heading. If your
>>> > original
>>> > Org file is large, re-calculations make things extremely slow.
>>> >
>>> > Best,
>>> > Ihor
>>> >
>>>
>>>
>>> --
>>> The Kafka Pandemic
>>>
>>> Please learn what misopathy is.
>>>
>>> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html
>>>
>>>
>>
>> --
>> regards,
>>
>> Tim
>>
>> --
>> Tim Cross
>>



^ permalink raw reply	[flat|nested] 13+ messages in thread

* Re: archiving speed [was Re: Tips on maintaining history in Org Mode]
  2021-08-11  5:58     ` Samuel Wales
  2021-08-11  6:43       ` Ihor Radchenko
@ 2021-10-17 12:08       ` Ihor Radchenko
  1 sibling, 0 replies; 13+ messages in thread
From: Ihor Radchenko @ 2021-10-17 12:08 UTC (permalink / raw)
  To: Samuel Wales; +Cc: emacs-orgmode, David Masterson

Samuel Wales <samologist@gmail.com> writes:

> i should clarify.  bulk archiving slows down even with /nonexistent/
> (have not tried empty) archives.  as part of normal and expected
> operation, bulk creates the archive for the first entry, and then
> subsequent entries are added.  those get slower and slower.

Hi. I just pushed a change that should solve the archiving speed. Feel
free to try it out.

Best,
Ihor


^ permalink raw reply	[flat|nested] 13+ messages in thread

end of thread, other threads:[~2021-10-17 12:08 UTC | newest]

Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-01  4:22 archiving speed [was Re: Tips on maintaining history in Org Mode] Samuel Wales
2021-03-01  6:02 ` Ihor Radchenko
2021-08-11  1:08 ` Samuel Wales
2021-08-11  4:13   ` Ihor Radchenko
2021-08-11  5:58     ` Samuel Wales
2021-08-11  6:43       ` Ihor Radchenko
2021-08-11 22:23         ` Samuel Wales
2021-08-12  0:24           ` Ihor Radchenko
2021-08-12  5:47             ` Samuel Wales
2021-08-12  3:38           ` Tim Cross
2021-08-12  5:49             ` Samuel Wales
2021-08-12  5:56               ` Tim Cross
2021-10-17 12:08       ` Ihor Radchenko

Code repositories for project(s) associated with this inbox:

	https://git.savannah.gnu.org/cgit/emacs/org-mode.git

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).