From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id yErLHLy+FGE7AQAAgWs5BA (envelope-from ) for ; Thu, 12 Aug 2021 08:25:00 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id +INwGLy+FGFRCwAAB5/wlQ (envelope-from ) for ; Thu, 12 Aug 2021 06:25:00 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 7F5A5B7F9 for ; Thu, 12 Aug 2021 08:24:59 +0200 (CEST) Received: from localhost ([::1]:33560 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mE49B-0002o5-U7 for larch@yhetil.org; Thu, 12 Aug 2021 02:24:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:36566) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mE479-0007AL-Nd for emacs-orgmode@gnu.org; Thu, 12 Aug 2021 02:22:51 -0400 Received: from mail-pj1-x1031.google.com ([2607:f8b0:4864:20::1031]:52968) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mE476-0001ni-9e for emacs-orgmode@gnu.org; Thu, 12 Aug 2021 02:22:51 -0400 Received: by mail-pj1-x1031.google.com with SMTP id nt11so7743963pjb.2 for ; Wed, 11 Aug 2021 23:22:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=references:user-agent:from:to:cc:subject:date:in-reply-to :message-id:mime-version; bh=ASwAGjuapZHLzMbK1ew5/FOY+PIbnCk5P1uxT8cxhm0=; b=jCMqIZoYO2bkG7iQoh+9017XLFHdYm0t9wRcHPp9GJxffN08sZhosGvUOQdQ916t40 b5QON2UdB0vdplrbbwrIwSqwfqyQRloURL5BSFnluNvkXJDMOhTWejJkRLyHgu4OW8s8 aEMMP8HyYr59kJMSKwhZU6D1J2B6/fM28ET+KWYyUDTtoN/BwXYnDxMpsHwZ/+mgUOm9 9cJYEKHAsrlwEfQer373eYF1ltzLuytBllCisSrKD2I3LB/qKJusiCNCAtL1kH4rwBUN kWpgJ4n6NEwA+eKcFnZLJjd5WAWCEd/gLjWpDTvAtTWYXj3Mefu1S4MhER/CTktFefDH 7ePA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:references:user-agent:from:to:cc:subject:date :in-reply-to:message-id:mime-version; bh=ASwAGjuapZHLzMbK1ew5/FOY+PIbnCk5P1uxT8cxhm0=; b=XuszFHhpRa/wkKkscYc+Przl+saW8zj9PDlOLaZ4ptPZAqMyG+kCX1VEqLmsaky1WV TCSq4qndMIWuD0ygQVHE9skOs+hwVWcKf3i5aPnuSmdmdwoGfGkPBDGocJyN9M/hQzIN Md7TPhdUtbCtvNxXN2NTy6OZrPgWpR27v73CPGdgK/jyKYXDZgZ+8EP3W3h33mWQDhb6 fxacDqO94o8jc+iSEV9BSMkXPxbrlKSQbqDNykv2zpOyyr8MNZMWVnXVW2rKAXj3F69e FtRL9KapvwVTLDJCELcgAcg7qEu2KJgrqGktFXaRTQJIa3grHLcCrY0mqvMqeZOLrX1Y 6bRA== X-Gm-Message-State: AOAM531SwxAC7dvBzWo/PaGkFFb4Etkbiiz3r944SFAwZaX0gZtMLEFm jtfUMIP+ThRwpUdtAeNUYKg= X-Google-Smtp-Source: ABdhPJwR1FBoipcdclmN5SUioF9RVd6Q2xVsOXzKUK1N0j5XCNHznoaLFEsQMI6T42zB4Ylnc4y1IA== X-Received: by 2002:a17:902:bd05:b029:12d:5ce2:1e77 with SMTP id p5-20020a170902bd05b029012d5ce21e77mr2325911pls.10.1628749366800; Wed, 11 Aug 2021 23:22:46 -0700 (PDT) Received: from tim-desktop (106-69-158-200.dyn.iinet.net.au. [106.69.158.200]) by smtp.gmail.com with ESMTPSA id c11sm1536070pji.24.2021.08.11.23.22.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Aug 2021 23:22:46 -0700 (PDT) References: <87wnos8vqi.fsf@localhost> <87tujw8orc.fsf@localhost> User-agent: mu4e 1.6.2; emacs 28.0.50 From: Tim Cross To: Samuel Wales Subject: Re: archiving speed [was Re: Tips on maintaining history in Org Mode] Date: Thu, 12 Aug 2021 15:56:34 +1000 In-reply-to: Message-ID: <87a6lnchcd.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::1031; envelope-from=theophilusx@gmail.com; helo=mail-pj1-x1031.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Org-mode , Ihor Radchenko , David Masterson Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1628749499; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=ASwAGjuapZHLzMbK1ew5/FOY+PIbnCk5P1uxT8cxhm0=; b=VRAuD4VfP5waRQoG7kFVEDEmAziKs7ZHZaERTFkJtXTlc+INYlz0WMz5Cis6YFMU37FIT1 WMSgnGeuEHKH2cccVuQkhhZAcPB1SM0wWQqYoJKuY8OepHTeNnv1Yg2lOXTh7dsRP64w2T 78AN4Q3kIp/J8vBdxJWzTncDNz7WGC55n+AxezJqMY5QOt+r6fOWpqu/xjSRzvdnfgvZ8l aGJQqQtNqJtMKBf7pom03rg8C0p6mTCNGdvxadG3Vrdn0c696VedaunsLBmdegD1DHLTNk vyhoCaB3kld0ai3cMmtgnCG4dig1QzW97nxMINapR7ZEyBc+afQ7uthJPeKSsA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1628749499; a=rsa-sha256; cv=none; b=SEDjls7FBXRu7Ar7krtDeM7W6olB8OccgZVPFn+gRzz203CYI/njsHDF3ipSxh0yNWPpWZ XEGVTmfMyTg0VdEU9d/oKMsZypKgKak1CNFzymQUvqkyct0wypwzUyvrtuEugXbjHE0jRT RhQpGbj5DMrbjdJsHZf6KCCTyrNYmOUa6KcLOgNhF+TOAdwd/6YoJU+0r7xWwRsXg2dLyM iTq2IKeSx2PgaJaO48RH99PVqxTF89PJh+EV+PVcC4W0UIKXRYY6QQV8wYOxikAAdpVlTZ 5tA+MWGlU+lsmwaKLRv8lPx1V/JxkHJsl6QvXeIFRBGN4o3djSw+LMq7cRoDzQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=jCMqIZoY; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Spam-Score: -1.31 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=jCMqIZoY; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Migadu-Queue-Id: 7F5A5B7F9 X-Spam-Score: -1.31 X-Migadu-Scanner: scn0.migadu.com X-TUID: V5lsXs1lRwNH It is for me. However, this all depends on how you configure things. In my case, I have a standard structure for my org files. I have a different org file for each 'topic' and each org file has headings for * Tasks * Notes * Resources * Comms * General and each of those headings has a property list with an ARCHIVE property which looks like %s_archive:: i.e. %s_archive::* Tasks, so when I archive a tree/subtree is is placed under the heading according to the ARCHIVE property for the tree it comes from. almost all data I enter into org files comes via a capture template. All captured data initially goes into a refile.org file and then I refile to the appropriate topic org file each morning. For completed tasks, I will usually marked them with an ARCHIVE tag soon after they have been completed and then every 6 months or so, I will archive into an archive file (where heading hierarchies are retained). This is most common with completed tasks. I have a custom agenda which shows tasks broken up into "Completed", "In Progress", "Next" and "Backlog". I will archive the DONE items so that the "Completed" list in the agenda does ot grow too large. Every 12 months I move the archive files to an archive folder where they are renamed to include the year in the filename. I don't find archiving terribly slow. this is mainly because few of my org files are particularly large (because they are broken up into topics) and because I move older archives out into an archive directory after 12 months. It is very rare that I need to go digging around in archive files (either current or older year archives). Of course this could all change down the track. My org files are slowly getting larger and I expect at some point, I will hit a tipping point where things become slow. So far, even with the larger files (around 5Mb) performance is fine. I also don't put 'everything' in the org file. If I have another file of data, I will just link to that from the org file rather than have that data actually reside in the org file. SO for example, my org-mode.org file has a lot of links to interesting messages from the org list, but the messages themselves are all in my mu4e maildir folders. My main point was that because configuration like mine exist, simply appending archived items to the archive file simply would not work. I like having my archive records in a similar 'shape' to my normal org files because when I do need to dig into the archive, I don't want to have to go through the whole file looking for something. I generally know if I'm looking for an old task, note, general entry or comms record and it is handly to know I only have to look in that section of the file. This is one of the big challenges for org mode. Because it is so flexible and people take advantage of that flexibility, what may appear like a simple way to solve an issue often ends up being far more complex than it initially seemed. If, for exmaple, you could not archive based on heading, date, etc, just appending entries would probably work fine. However, as the archviing policy might be more complex, org needs to examine/parse the archive file to work out where to insert the archived entry. Tim Samuel Wales writes: > what is the current status of hierarchy in archive files? surely they > don't deal with updating categories and updating hierarchy structure > [sounds brittle and syncy]? i'm thinking it isn't hierarchical at > present, except when you have a doneified task with children? > > > On 8/11/21, Tim Cross wrote: >> I think the problem with just using append to file is that it won't >> preserve the shape of the file. For example, if I had a file with >> >> * Notes >> ** Note 1 >> blah blah >> ** Note 2 blah blah >> >> * Tasks >> ** DONE task 1 >> ** TODO Task 2 >> >> and I decide to archive note 1 and task 1, I would like them to both appear >> under the same headings and with the same level. If the process just uses >> append to file, I can have this for the first archiving i.e. >> >> * Noes >> ** Note 1 >> >> * Tasks >> ** DONE task 1 >> >> but then later, I decide to archive note 2, if append file is used, I will >> end up with >> >> * Notes >> ** Note 1 >> >> * Taks >> ** DONE task 1 >> >> * Notes >> ** Note 2 >> >> which is not what I want. I want >> >> * Notes >> ** Note 1 >> ** Note 2 >> >> * Tasks >> ** DONE Task 1 >> >> So, if we want to preserve hierarchies in our archive files and not have >> everything jumbled up together, the system has to parse the file. If you >> are also using something like Categories, then even more work needs to be >> odne to update the category lists. >> >> What I tend to do is mark items with the ARCHIVE tag and leave them in the >> file and then every few months, move archived data to archive files. It >> can still get slow, but I don't do it often, so it isn't too much of a >> hassle. >> >> >> On Thu, 12 Aug 2021 at 08:23, Samuel Wales wrote: >> >>> thanks for the clarification. are you saying that, for every archived >>> entry, it calculates teh category property, using the original org >>> file, in order to add a category property to just one archived entry? >>> >>> that would certainly slow down more and more, but it sends me back to >>> my question about whether append to file would work. >>> i.e. build the single entry in a temporary buffer then write that >>> region to a file on disk. >>> >>> On 8/10/21, Ihor Radchenko wrote: >>> > Samuel Wales writes: >>> > >>> >> i should clarify. bulk archiving slows down even with /nonexistent/ >>> >> (have not tried empty) archives. as part of normal and expected >>> >> operation, bulk creates the archive for the first entry, and then >>> >> subsequent entries are added. those get slower and slower. >>> > >>> > That's what I suspected. I also see this and my suggestion helped >>> > archiving speed in my case. >>> > >>> >> i use (olpath category itags). i will try (file time) when i can, if >>> >> that still applies. my brain needs to be more operational. >>> > >>> > When you use category, every time you modify the original file (not the >>> > archive!), Org mode re-calculates *all* the categories in the original >>> > Org file. It happens for every single archived heading. If your >>> > original >>> > Org file is large, re-calculations make things extremely slow. >>> > >>> > Best, >>> > Ihor >>> > >>> >>> >>> -- >>> The Kafka Pandemic >>> >>> Please learn what misopathy is. >>> >>> https://thekafkapandemic.blogspot.com/2013/10/why-some-diseases-are-wronged.html >>> >>> >> >> -- >> regards, >> >> Tim >> >> -- >> Tim Cross >>