* archiving speed [was Re: Tips on maintaining history in Org Mode] @ 2021-03-01 4:22 Samuel Wales 2021-03-01 6:02 ` Ihor Radchenko 2021-08-11 1:08 ` Samuel Wales 0 siblings, 2 replies; 13+ messages in thread From: Samuel Wales @ 2021-03-01 4:22 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson thank you for your detaild reply. more below. On 2/28/21, Ihor Radchenko <yantar92@gmail.com> wrote: > details why). So, many org commands tend to lag on large archives. that makes sense. but why would appending to an archive as the result of bulk archiving lag? if the problem is large archive files, which i'd bet is the case for a lot of users and not just me, then could org in principle be changed so that all it does is append? thus not lag? like, build the entry in a temporary buffer? as i see it, having more than one archive file per org file is good for speed, but doesn't work in existing org, because iirc e.g. v A in the agenda goes org agenda file -> corresponding archive file and will miss the archive files that do not have a corresponding org file with exactly the same basename sans extension. i'd be ok with released org either allowing hte user to make year-based archives by having all of org recognize them, or my just append thing above. maybe i am missing something. > The lags can be solved in several ways: > 1. Reduce the archive file size this implies to me e.g. year-based archives, which would fail the v A test iiuc. thus needed extra code. > 2. Use optimised folding mechanism [1] (this will speed up org-mode in > general as well) i look forward to this filtering down to maint. :] [i used to follow master but too much for me now for health reasons.] > 3. (untested) Put #+STARTUP: showeverything at the beginning of the > archives, so that nothing is going to be folded good idea. my included-by-agenda archive files do seem to be in showeveryting mode already for some reason. but perhaps not when bulk archiving. would it be a silly idea for an fr that org make this an option for bulk archiving? hmm or for archive files in general? >> i will keep in mind disabling font lock in archive files. any >> suggested code for that? > > Note that it will mostly affect find-file performance. To disable if so, then i figure it's a one-time thing per file so no big deal. but thanks for hte font lock stuff i didnt' know about. > Sorry, the config is actually not yet formatted for public use. You can > search for the code block containing "defun org-archive--compute-location". firefox find does not seem to find it. > > You will need that code block and the following code block. > > [1] https://github.com/yantar92/org > > Best, > Ihor > > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-03-01 4:22 archiving speed [was Re: Tips on maintaining history in Org Mode] Samuel Wales @ 2021-03-01 6:02 ` Ihor Radchenko 2021-08-11 1:08 ` Samuel Wales 1 sibling, 0 replies; 13+ messages in thread From: Ihor Radchenko @ 2021-03-01 6:02 UTC (permalink / raw) To: Samuel Wales; +Cc: emacs-orgmode, David Masterson Samuel Wales <samologist@gmail.com> writes: > that makes sense. but why would appending to an archive as the result > of bulk archiving lag? if the problem is large archive files, which > i'd bet is the case for a lot of users and not just me, then could org > in principle be changed so that all it does is append? thus not lag? > like, build the entry in a temporary buffer? Looking into the source code, the culprit appears to be the call to org-show-all inside org-archive-subtree. The call to org-show-all on master calls org-cycle-hide-drawers, which goes through every single drawer (= every archived heading in archive file) and folds it manually (just to unfold it later). Do not ask. > as i see it, having more than one archive file per org file is good > for speed, but doesn't work in existing org, because iirc e.g. v A in > the agenda goes org agenda file -> corresponding archive file and will > miss the archive files that do not have a corresponding org file with > exactly the same basename sans extension. The code from my config takes care about this. v A will work. I plan to suggest it for master in future, but I am focused on the org-fold branch for now. If you wish, feel free to propose a patch based on this code. >> 3. (untested) Put #+STARTUP: showeverything at the beginning of the >> archives, so that nothing is going to be folded > > good idea. my included-by-agenda archive files do seem to be in > showeveryting mode already for some reason. but perhaps not when bulk > archiving. > > would it be a silly idea for an fr that org make this an option for > bulk archiving? hmm or for archive files in general? Upon reviewing the source of org-archive-subtree, this should not be needed. Actually, org-mode already disables startup visibility and various hooks when opening archives. As you can see, it is not sufficient, since org-archive shoots its own leg later by changing visibility every time you archive a subtree. >> Sorry, the config is actually not yet formatted for public use. You can >> search for the code block containing "defun org-archive--compute-location". > > firefox find does not seem to find it. This is odd. All I can suggest then is cloning the repo and searching using Emacs. Emacs is more reliable when opening org files ;) git clone https://github.com/yantar92/emacs-config emacs emacs-config/config.org Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-03-01 4:22 archiving speed [was Re: Tips on maintaining history in Org Mode] Samuel Wales 2021-03-01 6:02 ` Ihor Radchenko @ 2021-08-11 1:08 ` Samuel Wales 2021-08-11 4:13 ` Ihor Radchenko 1 sibling, 1 reply; 13+ messages in thread From: Samuel Wales @ 2021-08-11 1:08 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson i have not bulk archived in a year or two, because it is too slow for me. i am curious why, when appending to an archive file, append-to-file or write-region are not used, to merely put the entries there instead of loading the archive file into emacs? [this is the case even with zero-size archive files; after a few entries it slows down.] On 2/28/21, Samuel Wales <samologist@gmail.com> wrote: > thank you for your detaild reply. > > more below. > > On 2/28/21, Ihor Radchenko <yantar92@gmail.com> wrote: >> details why). So, many org commands tend to lag on large archives. > > that makes sense. but why would appending to an archive as the result > of bulk archiving lag? if the problem is large archive files, which > i'd bet is the case for a lot of users and not just me, then could org > in principle be changed so that all it does is append? thus not lag? > like, build the entry in a temporary buffer? > > as i see it, having more than one archive file per org file is good > for speed, but doesn't work in existing org, because iirc e.g. v A in > the agenda goes org agenda file -> corresponding archive file and will > miss the archive files that do not have a corresponding org file with > exactly the same basename sans extension. > > i'd be ok with released org either allowing hte user to make > year-based archives by having all of org recognize them, or my just > append thing above. maybe i am missing something. > >> The lags can be solved in several ways: >> 1. Reduce the archive file size > > this implies to me e.g. year-based archives, which would fail the v A > test iiuc. thus needed extra code. > >> 2. Use optimised folding mechanism [1] (this will speed up org-mode in >> general as well) > > i look forward to this filtering down to maint. :] [i used to follow > master but too much for me now for health reasons.] > >> 3. (untested) Put #+STARTUP: showeverything at the beginning of the >> archives, so that nothing is going to be folded > > good idea. my included-by-agenda archive files do seem to be in > showeveryting mode already for some reason. but perhaps not when bulk > archiving. > > would it be a silly idea for an fr that org make this an option for > bulk archiving? hmm or for archive files in general? > >>> i will keep in mind disabling font lock in archive files. any >>> suggested code for that? >> >> Note that it will mostly affect find-file performance. To disable > > if so, then i figure it's a one-time thing per file so no big deal. > but thanks for hte font lock stuff i didnt' know about. > >> Sorry, the config is actually not yet formatted for public use. You can >> search for the code block containing "defun >> org-archive--compute-location". > > firefox find does not seem to find it. > >> >> You will need that code block and the following code block. >> >> [1] https://github.com/yantar92/org >> >> Best, >> Ihor >> >> > > > -- > The Kafka Pandemic > > Please learn what misopathy is. > https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 1:08 ` Samuel Wales @ 2021-08-11 4:13 ` Ihor Radchenko 2021-08-11 5:58 ` Samuel Wales 0 siblings, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2021-08-11 4:13 UTC (permalink / raw) To: Samuel Wales; +Cc: emacs-orgmode, David Masterson Samuel Wales <samologist@gmail.com> writes: > [this is the case even with zero-size archive files; after a few > entries it slows down.] Do you get the same behaviour with the following code? (setq org-archive-save-context-info '(file time)) Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 4:13 ` Ihor Radchenko @ 2021-08-11 5:58 ` Samuel Wales 2021-08-11 6:43 ` Ihor Radchenko 2021-10-17 12:08 ` Ihor Radchenko 0 siblings, 2 replies; 13+ messages in thread From: Samuel Wales @ 2021-08-11 5:58 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson i should clarify. bulk archiving slows down even with /nonexistent/ (have not tried empty) archives. as part of normal and expected operation, bulk creates the archive for the first entry, and then subsequent entries are added. those get slower and slower. i was trying to imply e.g. doing archives by year won't fix it. only a few entries are enough to slow it down. i use (olpath category itags). i will try (file time) when i can, if that still applies. my brain needs to be more operational. i should mention that i did also find a bug, but was not able to narrow it down. it has been a while, but it was something like not killing the original entry for one of the entries. i was unable to figure out what conditions needed to obtain for this to occur. On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote: > Samuel Wales <samologist@gmail.com> writes: > >> [this is the case even with zero-size archive files; after a few >> entries it slows down.] > > Do you get the same behaviour with the following code? > > (setq org-archive-save-context-info '(file time)) > > Best, > Ihor > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 5:58 ` Samuel Wales @ 2021-08-11 6:43 ` Ihor Radchenko 2021-08-11 22:23 ` Samuel Wales 2021-10-17 12:08 ` Ihor Radchenko 1 sibling, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2021-08-11 6:43 UTC (permalink / raw) To: Samuel Wales; +Cc: emacs-orgmode, David Masterson Samuel Wales <samologist@gmail.com> writes: > i should clarify. bulk archiving slows down even with /nonexistent/ > (have not tried empty) archives. as part of normal and expected > operation, bulk creates the archive for the first entry, and then > subsequent entries are added. those get slower and slower. That's what I suspected. I also see this and my suggestion helped archiving speed in my case. > i use (olpath category itags). i will try (file time) when i can, if > that still applies. my brain needs to be more operational. When you use category, every time you modify the original file (not the archive!), Org mode re-calculates *all* the categories in the original Org file. It happens for every single archived heading. If your original Org file is large, re-calculations make things extremely slow. Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 6:43 ` Ihor Radchenko @ 2021-08-11 22:23 ` Samuel Wales 2021-08-12 0:24 ` Ihor Radchenko 2021-08-12 3:38 ` Tim Cross 0 siblings, 2 replies; 13+ messages in thread From: Samuel Wales @ 2021-08-11 22:23 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson thanks for the clarification. are you saying that, for every archived entry, it calculates teh category property, using the original org file, in order to add a category property to just one archived entry? that would certainly slow down more and more, but it sends me back to my question about whether append to file would work. i.e. build the single entry in a temporary buffer then write that region to a file on disk. On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote: > Samuel Wales <samologist@gmail.com> writes: > >> i should clarify. bulk archiving slows down even with /nonexistent/ >> (have not tried empty) archives. as part of normal and expected >> operation, bulk creates the archive for the first entry, and then >> subsequent entries are added. those get slower and slower. > > That's what I suspected. I also see this and my suggestion helped > archiving speed in my case. > >> i use (olpath category itags). i will try (file time) when i can, if >> that still applies. my brain needs to be more operational. > > When you use category, every time you modify the original file (not the > archive!), Org mode re-calculates *all* the categories in the original > Org file. It happens for every single archived heading. If your original > Org file is large, re-calculations make things extremely slow. > > Best, > Ihor > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 22:23 ` Samuel Wales @ 2021-08-12 0:24 ` Ihor Radchenko 2021-08-12 5:47 ` Samuel Wales 2021-08-12 3:38 ` Tim Cross 1 sibling, 1 reply; 13+ messages in thread From: Ihor Radchenko @ 2021-08-12 0:24 UTC (permalink / raw) To: Samuel Wales; +Cc: emacs-orgmode, David Masterson Samuel Wales <samologist@gmail.com> writes: > thanks for the clarification. are you saying that, for every archived > entry, it calculates teh category property, using the original org > file, in order to add a category property to just one archived entry? Nope. It does not just calculate category for the archived entry, but re-calculates all the category properties in the original Org file (updating category cache). > that would certainly slow down more and more, but it sends me back to > my question about whether append to file would work. > i.e. build the single entry in a temporary buffer then write that > region to a file on disk. Appending can indeed work if your archive location is at the end of the file. However, it is not necessary the performance bottleneck. Certainly not when the archive file is small. Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-12 0:24 ` Ihor Radchenko @ 2021-08-12 5:47 ` Samuel Wales 0 siblings, 0 replies; 13+ messages in thread From: Samuel Wales @ 2021-08-12 5:47 UTC (permalink / raw) To: Ihor Radchenko; +Cc: emacs-orgmode, David Masterson thank you. i will give up on archiving categories if needed to make archiving be practical. will ahfe to try it as soon as i can. On 8/11/21, Ihor Radchenko <yantar92@gmail.com> wrote: > Samuel Wales <samologist@gmail.com> writes: > >> thanks for the clarification. are you saying that, for every archived >> entry, it calculates teh category property, using the original org >> file, in order to add a category property to just one archived entry? > > Nope. It does not just calculate category for the archived entry, but > re-calculates all the category properties in the original Org file > (updating category cache). > >> that would certainly slow down more and more, but it sends me back to >> my question about whether append to file would work. >> i.e. build the single entry in a temporary buffer then write that >> region to a file on disk. > > Appending can indeed work if your archive location is at the end of the > file. However, it is not necessary the performance bottleneck. Certainly > not when the archive file is small. > > Best, > Ihor > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 22:23 ` Samuel Wales 2021-08-12 0:24 ` Ihor Radchenko @ 2021-08-12 3:38 ` Tim Cross 2021-08-12 5:49 ` Samuel Wales 1 sibling, 1 reply; 13+ messages in thread From: Tim Cross @ 2021-08-12 3:38 UTC (permalink / raw) To: Samuel Wales; +Cc: Org-mode, Ihor Radchenko, David Masterson [-- Attachment #1: Type: text/plain, Size: 2869 bytes --] I think the problem with just using append to file is that it won't preserve the shape of the file. For example, if I had a file with * Notes ** Note 1 blah blah ** Note 2 blah blah * Tasks ** DONE task 1 ** TODO Task 2 and I decide to archive note 1 and task 1, I would like them to both appear under the same headings and with the same level. If the process just uses append to file, I can have this for the first archiving i.e. * Noes ** Note 1 * Tasks ** DONE task 1 but then later, I decide to archive note 2, if append file is used, I will end up with * Notes ** Note 1 * Taks ** DONE task 1 * Notes ** Note 2 which is not what I want. I want * Notes ** Note 1 ** Note 2 * Tasks ** DONE Task 1 So, if we want to preserve hierarchies in our archive files and not have everything jumbled up together, the system has to parse the file. If you are also using something like Categories, then even more work needs to be odne to update the category lists. What I tend to do is mark items with the ARCHIVE tag and leave them in the file and then every few months, move archived data to archive files. It can still get slow, but I don't do it often, so it isn't too much of a hassle. On Thu, 12 Aug 2021 at 08:23, Samuel Wales <samologist@gmail.com> wrote: > thanks for the clarification. are you saying that, for every archived > entry, it calculates teh category property, using the original org > file, in order to add a category property to just one archived entry? > > that would certainly slow down more and more, but it sends me back to > my question about whether append to file would work. > i.e. build the single entry in a temporary buffer then write that > region to a file on disk. > > On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote: > > Samuel Wales <samologist@gmail.com> writes: > > > >> i should clarify. bulk archiving slows down even with /nonexistent/ > >> (have not tried empty) archives. as part of normal and expected > >> operation, bulk creates the archive for the first entry, and then > >> subsequent entries are added. those get slower and slower. > > > > That's what I suspected. I also see this and my suggestion helped > > archiving speed in my case. > > > >> i use (olpath category itags). i will try (file time) when i can, if > >> that still applies. my brain needs to be more operational. > > > > When you use category, every time you modify the original file (not the > > archive!), Org mode re-calculates *all* the categories in the original > > Org file. It happens for every single archived heading. If your original > > Org file is large, re-calculations make things extremely slow. > > > > Best, > > Ihor > > > > > -- > The Kafka Pandemic > > Please learn what misopathy is. > > https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html > > -- regards, Tim -- Tim Cross [-- Attachment #2: Type: text/html, Size: 4293 bytes --] ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-12 3:38 ` Tim Cross @ 2021-08-12 5:49 ` Samuel Wales 2021-08-12 5:56 ` Tim Cross 0 siblings, 1 reply; 13+ messages in thread From: Samuel Wales @ 2021-08-12 5:49 UTC (permalink / raw) To: Tim Cross; +Cc: Org-mode, Ihor Radchenko, David Masterson what is the current status of hierarchy in archive files? surely they don't deal with updating categories and updating hierarchy structure [sounds brittle and syncy]? i'm thinking it isn't hierarchical at present, except when you have a doneified task with children? On 8/11/21, Tim Cross <theophilusx@gmail.com> wrote: > I think the problem with just using append to file is that it won't > preserve the shape of the file. For example, if I had a file with > > * Notes > ** Note 1 > blah blah > ** Note 2 blah blah > > * Tasks > ** DONE task 1 > ** TODO Task 2 > > and I decide to archive note 1 and task 1, I would like them to both appear > under the same headings and with the same level. If the process just uses > append to file, I can have this for the first archiving i.e. > > * Noes > ** Note 1 > > * Tasks > ** DONE task 1 > > but then later, I decide to archive note 2, if append file is used, I will > end up with > > * Notes > ** Note 1 > > * Taks > ** DONE task 1 > > * Notes > ** Note 2 > > which is not what I want. I want > > * Notes > ** Note 1 > ** Note 2 > > * Tasks > ** DONE Task 1 > > So, if we want to preserve hierarchies in our archive files and not have > everything jumbled up together, the system has to parse the file. If you > are also using something like Categories, then even more work needs to be > odne to update the category lists. > > What I tend to do is mark items with the ARCHIVE tag and leave them in the > file and then every few months, move archived data to archive files. It > can still get slow, but I don't do it often, so it isn't too much of a > hassle. > > > On Thu, 12 Aug 2021 at 08:23, Samuel Wales <samologist@gmail.com> wrote: > >> thanks for the clarification. are you saying that, for every archived >> entry, it calculates teh category property, using the original org >> file, in order to add a category property to just one archived entry? >> >> that would certainly slow down more and more, but it sends me back to >> my question about whether append to file would work. >> i.e. build the single entry in a temporary buffer then write that >> region to a file on disk. >> >> On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote: >> > Samuel Wales <samologist@gmail.com> writes: >> > >> >> i should clarify. bulk archiving slows down even with /nonexistent/ >> >> (have not tried empty) archives. as part of normal and expected >> >> operation, bulk creates the archive for the first entry, and then >> >> subsequent entries are added. those get slower and slower. >> > >> > That's what I suspected. I also see this and my suggestion helped >> > archiving speed in my case. >> > >> >> i use (olpath category itags). i will try (file time) when i can, if >> >> that still applies. my brain needs to be more operational. >> > >> > When you use category, every time you modify the original file (not the >> > archive!), Org mode re-calculates *all* the categories in the original >> > Org file. It happens for every single archived heading. If your >> > original >> > Org file is large, re-calculations make things extremely slow. >> > >> > Best, >> > Ihor >> > >> >> >> -- >> The Kafka Pandemic >> >> Please learn what misopathy is. >> >> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html >> >> > > -- > regards, > > Tim > > -- > Tim Cross > -- The Kafka Pandemic Please learn what misopathy is. https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-12 5:49 ` Samuel Wales @ 2021-08-12 5:56 ` Tim Cross 0 siblings, 0 replies; 13+ messages in thread From: Tim Cross @ 2021-08-12 5:56 UTC (permalink / raw) To: Samuel Wales; +Cc: Org-mode, Ihor Radchenko, David Masterson It is for me. However, this all depends on how you configure things. In my case, I have a standard structure for my org files. I have a different org file for each 'topic' and each org file has headings for * Tasks * Notes * Resources * Comms * General and each of those headings has a property list with an ARCHIVE property which looks like %s_archive::<heading> i.e. %s_archive::* Tasks, so when I archive a tree/subtree is is placed under the heading according to the ARCHIVE property for the tree it comes from. almost all data I enter into org files comes via a capture template. All captured data initially goes into a refile.org file and then I refile to the appropriate topic org file each morning. For completed tasks, I will usually marked them with an ARCHIVE tag soon after they have been completed and then every 6 months or so, I will archive into an archive file (where heading hierarchies are retained). This is most common with completed tasks. I have a custom agenda which shows tasks broken up into "Completed", "In Progress", "Next" and "Backlog". I will archive the DONE items so that the "Completed" list in the agenda does ot grow too large. Every 12 months I move the archive files to an archive folder where they are renamed to include the year in the filename. I don't find archiving terribly slow. this is mainly because few of my org files are particularly large (because they are broken up into topics) and because I move older archives out into an archive directory after 12 months. It is very rare that I need to go digging around in archive files (either current or older year archives). Of course this could all change down the track. My org files are slowly getting larger and I expect at some point, I will hit a tipping point where things become slow. So far, even with the larger files (around 5Mb) performance is fine. I also don't put 'everything' in the org file. If I have another file of data, I will just link to that from the org file rather than have that data actually reside in the org file. SO for example, my org-mode.org file has a lot of links to interesting messages from the org list, but the messages themselves are all in my mu4e maildir folders. My main point was that because configuration like mine exist, simply appending archived items to the archive file simply would not work. I like having my archive records in a similar 'shape' to my normal org files because when I do need to dig into the archive, I don't want to have to go through the whole file looking for something. I generally know if I'm looking for an old task, note, general entry or comms record and it is handly to know I only have to look in that section of the file. This is one of the big challenges for org mode. Because it is so flexible and people take advantage of that flexibility, what may appear like a simple way to solve an issue often ends up being far more complex than it initially seemed. If, for exmaple, you could not archive based on heading, date, etc, just appending entries would probably work fine. However, as the archviing policy might be more complex, org needs to examine/parse the archive file to work out where to insert the archived entry. Tim Samuel Wales <samologist@gmail.com> writes: > what is the current status of hierarchy in archive files? surely they > don't deal with updating categories and updating hierarchy structure > [sounds brittle and syncy]? i'm thinking it isn't hierarchical at > present, except when you have a doneified task with children? > > > On 8/11/21, Tim Cross <theophilusx@gmail.com> wrote: >> I think the problem with just using append to file is that it won't >> preserve the shape of the file. For example, if I had a file with >> >> * Notes >> ** Note 1 >> blah blah >> ** Note 2 blah blah >> >> * Tasks >> ** DONE task 1 >> ** TODO Task 2 >> >> and I decide to archive note 1 and task 1, I would like them to both appear >> under the same headings and with the same level. If the process just uses >> append to file, I can have this for the first archiving i.e. >> >> * Noes >> ** Note 1 >> >> * Tasks >> ** DONE task 1 >> >> but then later, I decide to archive note 2, if append file is used, I will >> end up with >> >> * Notes >> ** Note 1 >> >> * Taks >> ** DONE task 1 >> >> * Notes >> ** Note 2 >> >> which is not what I want. I want >> >> * Notes >> ** Note 1 >> ** Note 2 >> >> * Tasks >> ** DONE Task 1 >> >> So, if we want to preserve hierarchies in our archive files and not have >> everything jumbled up together, the system has to parse the file. If you >> are also using something like Categories, then even more work needs to be >> odne to update the category lists. >> >> What I tend to do is mark items with the ARCHIVE tag and leave them in the >> file and then every few months, move archived data to archive files. It >> can still get slow, but I don't do it often, so it isn't too much of a >> hassle. >> >> >> On Thu, 12 Aug 2021 at 08:23, Samuel Wales <samologist@gmail.com> wrote: >> >>> thanks for the clarification. are you saying that, for every archived >>> entry, it calculates teh category property, using the original org >>> file, in order to add a category property to just one archived entry? >>> >>> that would certainly slow down more and more, but it sends me back to >>> my question about whether append to file would work. >>> i.e. build the single entry in a temporary buffer then write that >>> region to a file on disk. >>> >>> On 8/10/21, Ihor Radchenko <yantar92@gmail.com> wrote: >>> > Samuel Wales <samologist@gmail.com> writes: >>> > >>> >> i should clarify. bulk archiving slows down even with /nonexistent/ >>> >> (have not tried empty) archives. as part of normal and expected >>> >> operation, bulk creates the archive for the first entry, and then >>> >> subsequent entries are added. those get slower and slower. >>> > >>> > That's what I suspected. I also see this and my suggestion helped >>> > archiving speed in my case. >>> > >>> >> i use (olpath category itags). i will try (file time) when i can, if >>> >> that still applies. my brain needs to be more operational. >>> > >>> > When you use category, every time you modify the original file (not the >>> > archive!), Org mode re-calculates *all* the categories in the original >>> > Org file. It happens for every single archived heading. If your >>> > original >>> > Org file is large, re-calculations make things extremely slow. >>> > >>> > Best, >>> > Ihor >>> > >>> >>> >>> -- >>> The Kafka Pandemic >>> >>> Please learn what misopathy is. >>> >>> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html >>> >>> >> >> -- >> regards, >> >> Tim >> >> -- >> Tim Cross >> ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: archiving speed [was Re: Tips on maintaining history in Org Mode] 2021-08-11 5:58 ` Samuel Wales 2021-08-11 6:43 ` Ihor Radchenko @ 2021-10-17 12:08 ` Ihor Radchenko 1 sibling, 0 replies; 13+ messages in thread From: Ihor Radchenko @ 2021-10-17 12:08 UTC (permalink / raw) To: Samuel Wales; +Cc: emacs-orgmode, David Masterson Samuel Wales <samologist@gmail.com> writes: > i should clarify. bulk archiving slows down even with /nonexistent/ > (have not tried empty) archives. as part of normal and expected > operation, bulk creates the archive for the first entry, and then > subsequent entries are added. those get slower and slower. Hi. I just pushed a change that should solve the archiving speed. Feel free to try it out. Best, Ihor ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2021-10-17 12:08 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2021-03-01 4:22 archiving speed [was Re: Tips on maintaining history in Org Mode] Samuel Wales 2021-03-01 6:02 ` Ihor Radchenko 2021-08-11 1:08 ` Samuel Wales 2021-08-11 4:13 ` Ihor Radchenko 2021-08-11 5:58 ` Samuel Wales 2021-08-11 6:43 ` Ihor Radchenko 2021-08-11 22:23 ` Samuel Wales 2021-08-12 0:24 ` Ihor Radchenko 2021-08-12 5:47 ` Samuel Wales 2021-08-12 3:38 ` Tim Cross 2021-08-12 5:49 ` Samuel Wales 2021-08-12 5:56 ` Tim Cross 2021-10-17 12:08 ` Ihor Radchenko
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs/org-mode.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).