From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id +ADjBRryz15ADAAA0tVLHw (envelope-from ) for ; Thu, 28 May 2020 17:17:14 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id BAPYARryz17ddQAAB5/wlQ (envelope-from ) for ; Thu, 28 May 2020 17:17:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 32292940983 for ; Thu, 28 May 2020 17:17:13 +0000 (UTC) Received: from localhost ([::1]:44514 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jeM9X-0001T8-WC for larch@yhetil.org; Thu, 28 May 2020 13:17:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:48674) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jeM8u-0001Sm-Ng for emacs-orgmode@gnu.org; Thu, 28 May 2020 13:16:32 -0400 Received: from mail-pg1-x52c.google.com ([2607:f8b0:4864:20::52c]:34542) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1jeM8t-0004Vv-3E for emacs-orgmode@gnu.org; Thu, 28 May 2020 13:16:32 -0400 Received: by mail-pg1-x52c.google.com with SMTP id m1so7566049pgk.1 for ; Thu, 28 May 2020 10:16:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:in-reply-to:references:date:message-id:mime-version; bh=zmANKx6u906J8AiT089mVF2NIId1H64stjlZScTpCWE=; b=edY09fS2mEC8mUTbUHTBKWJDZQeGzdr52hN33iNNTPRCqT7lFWgjnuXRjT9S0fh9i9 70rXfpDHqXBD3U+NEeegZP07l6gnR8RfbNvI4lQ48IAVQzvkE8SXL+n4iWm7dJbuDCOn thPS0hUK62HcgkW6fFVoeW56yLm94Eb5PpZ7mQ2DpKy56gEfps66mfyCWxGgKOu6ZFTd 9UIIATOf11CLTS4Ows34HqmxUsq+R06Ngay7Sha/ZftTtksmwMRGrQFI1fhNKscLHt5c Q8DTrg9Ka6KhwIyk/jf2Pd3Z+o5rzvdFR2T3HQtpYahEKJcuwqY6UhB+9qW5esLZASkJ HdQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:in-reply-to:references:date :message-id:mime-version; bh=zmANKx6u906J8AiT089mVF2NIId1H64stjlZScTpCWE=; b=R6/jx/VUqrk3kxp5fOAHunnbTqxZCpqnUfcKgww0o+3MgRBegm4ybS/Gnyre8pVgkt aEoq6PDUmM75sk+jI646G5x3BO/qvr8GYBT4kFWxiZHBoN+HaehJzPmKOu00YPTDaFKl umrJFm+iYcjTMG2N/WRVy7AQwawdiQDVQhjoLrZkpuI2CaT1o93JLuUF9lV5Tdz4n1Ls WHm+R7OIiLltX4kTaJWqJKqZVSFLX6Xx/a5N+jEsZKxMtjtsDYoduAhk258YdzwJuSwY Jkn4RxUODUX1Wq7ltoPyBn2bzOiN00RqlSEFijnGG3T4qcDaC0avMGFLblvFq2ovoLZU ae7Q== X-Gm-Message-State: AOAM532MqsidOhpyrOwhMn47kk/Zo4fgNla1fttWAeiLYPPlhcAQcwsN 3LAhULZ4OMaqIpcQq8X+xDo= X-Google-Smtp-Source: ABdhPJyeYL+LtKwlFwDIsubjty1KoX1vsKFwlK1nuJLLdxgNzCt72xr/LbjX1iWVmb3uw2DcfPWrBg== X-Received: by 2002:a65:40c3:: with SMTP id u3mr4110089pgp.305.1590686189483; Thu, 28 May 2020 10:16:29 -0700 (PDT) Received: from localhost ([104.250.131.79]) by smtp.gmail.com with ESMTPSA id 7sm5306955pfc.203.2020.05.28.10.16.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 28 May 2020 10:16:28 -0700 (PDT) From: Ihor Radchenko To: Matthew Lundin , numbchild@gmail.com, Org Mode Subject: Re: [Feature] add a new org-attach dispatcher command to offline save web page In-Reply-To: <874ks0vxpk.fsf@fastmail.fm> References: <87sgflu2gw.fsf@gmail.com> <87r1v4wyy4.fsf@fastmail.fm> <87r1v4bodg.fsf@localhost> <874ks0vxpk.fsf@fastmail.fm> Date: Fri, 29 May 2020 01:11:49 +0800 Message-ID: <875zcgugq2.fsf@localhost> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::52c; envelope-from=yantar92@gmail.com; helo=mail-pg1-x52c.google.com X-detected-operating-system: by eggs.gnu.org: No matching host in p0f cache. That's all we know. X-Spam_score_int: -17 X-Spam_score: -1.8 X-Spam_bar: - X-Spam_report: (-1.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: "Emacs-orgmode" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=edY09fS2; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of emacs-orgmode-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=emacs-orgmode-bounces@gnu.org X-Spam-Score: -1.21 X-TUID: Tlt8btG4K2Tn > My uneasiness has more to do with the specificity of the dependence on > monolith and the way that is hard-coded into the patch. When it comes to > patches, I think priority should go to those that are configurable, > accessible, and useful for everyone as opposed to those that have > hard-coded work-flows or highly-specific user configurations. Agree. Though I can see a use of having monolith as one of the options to help people discover what kind of tools they can use. I personally had a hard time finding command-line cli like monolith. Actually, it is the first time I heard about some offline tool handling js without a need to write python or ruby code. > The question is: which functionality? A simple downloading tool or a > full archival tool? Achieving similar functionality to org-board or > monolith would a big task, since they aim to download an archival > version of a webpage (including all resources). My view on this is bare-bones download, in a spirit of org-attach itself. There is already 'url method in org-attach-attach, but it is hard-coded to url-retrieve-synchronously. It would be handy if user could configure alternative retrievers (like monolith, wget, curl, or some user-defined function). Note that monolith does not crawl the website. It only collects everything needed to show the page as you see it in browser into single html file. This behaviour is what one expects to obtain when saving a full web-page from browser. > In addition, with > archiving you also quickly run into the complexity of versioning based > on time archived. I guess that org-attach-git can be used for versioning, but I don't think that versioning is within scope of this patch. Monolith does not even support versioning. > There's also the challenge of mapping the downloaded > files to metadata (specifically the original url). Org-board currently > handles both of these very well. org-board is a great package, but it is not built-in. I do not think that all the org-board functionality needs to be included into org-attach. At least not within scope of this patch as I understand it. Best, Ihor Matthew Lundin writes: > Ihor Radchenko writes: > >> It does not mean that attaching URL directly is not worth including >> into org. This sounds pretty common use case, especially considering >> the number of packages providing similar feature. You mentioned >> org-board, but there is also org-download and org-web-tools. > > My uneasiness has more to do with the specificity of the dependence on > monolith and the way that is hard-coded into the patch. When it comes to > patches, I think priority should go to those that are configurable, > accessible, and useful for everyone as opposed to those that have > hard-coded work-flows or highly-specific user configurations. > >> I agree that monolith is completely uncommon tool and I would not expect >> the majority of users to have it installed, but the same functionality >> utilising built-in url.el (as a default) should be acceptable. > > The question is: which functionality? A simple downloading tool or a > full archival tool? Achieving similar functionality to org-board or > monolith would a big task, since they aim to download an archival > version of a webpage (including all resources). In addition, with > archiving you also quickly run into the complexity of versioning based > on time archived. There's also the challenge of mapping the downloaded > files to metadata (specifically the original url). Org-board currently > handles both of these very well. > > I suppose there would be a few options depending on what the aims are: > > 1. At the simple end, include little more than than a quick and dirty > way of downloading a single resource (html, pdf, jpeg) using url.el > or wget (or optionally, monolith) and putting that in the attachment > folder. Those who want full archiving of all resources could use > other tools like org-board or org-web-tools. > > 2. At the (much) more complex end, it would be to code out a robust > archiving solution on top of url.el or wget. > > 3. Another, possibly simpler option... Add a command to the dispatcher > that allows the user to invoke a custom function that is called with > the attachment directory as the default-directory. This would enable > more end-user flexibility, such as the ability to use > wkhtmtoimage/wkhtmltopdf, monolith, phantom.js, archive.is, etc. > > Best, > > Matt -- Ihor Radchenko, PhD, Center for Advancing Materials Performance from the Nanoscale (CAMP-nano) State Key Laboratory for Mechanical Behavior of Materials, Xi'an Jiaotong University, Xi'an, China Email: yantar92@gmail.com, ihor_radchenko@alumni.sutd.edu.sg