From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id ABHEDFHXIGJyZQAAgWs5BA (envelope-from ) for ; Thu, 03 Mar 2022 15:57:21 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id wJhSCVHXIGJKmAAAauVa8A (envelope-from ) for ; Thu, 03 Mar 2022 15:57:21 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DD8FB3F2FD for ; Thu, 3 Mar 2022 15:57:20 +0100 (CET) Received: from localhost ([::1]:46520 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nPmtL-00053q-Iq for larch@yhetil.org; Thu, 03 Mar 2022 09:57:19 -0500 Received: from eggs.gnu.org ([209.51.188.92]:39628) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPmsm-00053A-Sj for emacs-orgmode@gnu.org; Thu, 03 Mar 2022 09:56:44 -0500 Received: from ciao.gmane.io ([116.202.254.214]:40522) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nPmsl-0007F5-DR for emacs-orgmode@gnu.org; Thu, 03 Mar 2022 09:56:44 -0500 Received: from list by ciao.gmane.io with local (Exim 4.92) (envelope-from ) id 1nPmsh-0004pB-K7 for emacs-orgmode@gnu.org; Thu, 03 Mar 2022 15:56:39 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: emacs-orgmode@gnu.org From: Max Nikulin Subject: Re: profiling latency in large org-mode buffers (under both main & org-fold feature) Date: Thu, 3 Mar 2022 21:56:33 +0700 Message-ID: References: <87fsobpism.fsf@localhost> <87r17to817.fsf@localhost> <87lexy2hrz.fsf@localhost> <87y21wkdwu.fsf@localhost> <87fso0xuaz.fsf@localhost> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.5.0 Content-Language: en-US In-Reply-To: <87fso0xuaz.fsf@localhost> Received-SPF: pass client-ip=116.202.254.214; envelope-from=geo-emacs-orgmode@m.gmane-mx.org; helo=ciao.gmane.io X-Spam_score_int: 28 X-Spam_score: 2.8 X-Spam_bar: ++ X-Spam_report: (2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_ADSP_CUSTOM_MED=0.001, FORGED_GMAIL_RCVD=1, FORGED_MUA_MOZILLA=2.309, FREEMAIL_FORGED_FROMDOMAIN=0.249, FREEMAIL_FROM=0.001, HEADER_FROM_DIFFERENT_DOMAINS=0.249, NICE_REPLY_A=-0.001, NML_ADSP_CUSTOM_MED=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=no autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1646319440; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post; bh=hhwBJD5Ll31M2ZurvAyDjk/sPdXzDsh72TKTN4NpA7o=; b=WuYoT9GAShLkwky/v20b1eOb+8ynI+jID2b6BAFtxl5lIDkqyoZUydGicuUWv3iEScc52y m2SUQkRvjZahvNPlNX3yLt2WsHFT7IQiOlOHLgkSk0vLvEuDA1PlrP072NGA9oMMW5lr8O q8O3b30BNIOp2cxxxMLPZ4GFZZG97JIcy2tJsQgCL+KyV2Wam3I/Baexh8mVBcQjO6DlIz x7+/WN6a7GV9iwpCpRQyIp7cOomEsBHmXliVa5oIlhGaRAJc7HpPjdmxH7V93T5ZIMDlJI aCAxd56nbCADyBNrWwKEH9E+R17nM01X+6CHS6PZ7pYGSy5fZJnOiQxmmVMpIQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1646319440; a=rsa-sha256; cv=none; b=gAwxkTGkifNrDTvWaP492KiNStL2FjijxTLOwflb/Dskk/0VnSk91lKWrjRIYOpZCY831K M4p/wGRYoNGYmZVXb40g6/UYrP2nAbxpbXx9f6GZZAfKaQxMh0t494rCPELxQhvTz/JadP LRV+ZYPPbPkDxxj7dLcyWp3iVAHLuXVoy0roRGXjJBS8F5RHLRclHd3h0ZfhD5XF/m+UuP x00dYGCsQ9lCGE04hDgFdTh3ea/rUSZ+/dwkGn0E9lRkt8nUuMG/8K7Yk7wHI6OuAz2Z44 bO25O3SK/hHPDsPkDD7/zXwk6xS5HsvGur0ej1lnbDGCoYf2tBmDVs7fS+Ej6Q== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: 2.43 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: DD8FB3F2FD X-Spam-Score: 2.43 X-Migadu-Scanner: scn0.migadu.com X-TUID: 0hGOltCVEy0V On 02/03/2022 22:12, Ihor Radchenko wrote: > Max Nikulin writes: > > I tend to agree after reading the code again. > I tried to play around with that marker loop. It seems that the loop > should not be mindlessly disabled, but it can be sufficient to check > only a small number of markers in front of the marker list. The cached > temporary markers are always added in front of the list. I did not try to say that the loop over markers may be just thrown away. By the way, for sequential scan (with no backward searches) single marker might work reasonably well. Some kind of index for fast mapping between bytes and positions should be maintained at the buffer level. I hope, when properly designed, such structure may minimize amount of recalculations on each edit. I mean some hierarchical structure of buffer fragments and markers keeps relative offsets from beginning of the fragment they belong to. Hierarchy of fragments is enough to provide initial estimation of position for byte index. Only markers within the fragment that is changed need immediate update. > I am currently using a custom version of org-ql utilising the new > element cache. It is substantially faster compared to current > org-refile-get-targets. The org-ql version runs in <2 seconds at worst > when calculating all refile targets from scratch, while > org-refile-get-targets is over 10sec. org-ql version gives 0 noticeable > latency when there is an extra text query to narrow down the refile > targets. So, is it certainly possible to improve the performance just > using high-level org-element cache API + regexp search without markers. It is up to you to choose at which level your prefer to optimize the code. And it is only my opinion (I do not insist) that benefits from changes in low level code might be much more significant. I like the idea of markers, but their current implementation is a source of pain. > (note that Nicolas did not use > markers to store boundaries of org elements). E.g. export-related code certainly does need markers. You experienced enough problems with attempts to properly invalidate cache when lower level is not supposed to provide appropriate facilities.