From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp12.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms9.migadu.com with LMTPS id MF/ALWX632RwcwAASxT56A (envelope-from ) for ; Sat, 19 Aug 2023 01:10:29 +0200 Received: from aspmx1.migadu.com ([2001:41d0:403:4789::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp12.migadu.com with LMTPS id qGB+LWX632TRWgAAauVa8A (envelope-from ) for ; Sat, 19 Aug 2023 01:10:29 +0200 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 49A7541CEE for ; Sat, 19 Aug 2023 01:10:29 +0200 (CEST) Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=PuOu+EEr; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1692400229; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=EbquYvb1jXJLxX2OBl71dVqGx4JvJ0XD8elXrFQHsJ0=; b=QEgi9DKHy32Iu5fgvCR68gKG2aEnvL3973l0wZvXQABeUwWm9T0nzp8sr4LY4tIr/OrWRB 9KQ8F0QbmgEqJ5WirldkqVbqEslbDTSNFYGHwwXuVuh3uV7KRXz4Xmf4vd6tulJ9Sj/5MU burv8S22Ni4OozcO6W+ULoYz5c+RCPPzSGiIDpq/BRUQDB0dJvR50ZCbOV+0UZlW4asSBb PTovUqnrOYhD0wfTFgc5K3dBXphClvFb6jqQv/ifhzu2R33qfbewFisJr1zUsbZyGQRRnC PcvjoJ91dGoZTrqgqr/dHGm4r673Dndf8bk4M5cBlZmMnc/TSUEckGzje61EmA== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1692400229; a=rsa-sha256; cv=none; b=k4E5c4+qAsSyZTzL3W2gkphKd7e+ibvw8D1Z8NmNn3ckiBjOGDVY+epX0fSu8wX4cNzgyw wB3ACg8PufjVNtpDDZ0ZSRrxpgAxBZaAZ9gRHPSHJNhjgqo/Zt4eK/oaJbD7sn0HVIUMZw p08l6Q7a5mRwUthWbJMODjvXS/RzRBTOkcEvoJejvo4RPPyTkXZPcoLAM73oRwWmDoMvuR ogp8uDGsGQtQdK0icdPWm19b3y8bUdx8muewTMR0XE/W/97FMJ0ngiXUHBqc4VLHbxC1Zd CQWinSKaAt3VSyHqq4wMCKLxX4O/4lm9fIVE4/v6lrovgzrf3ZMsvATTdyUivw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=PuOu+EEr; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=gmail.com Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1qX8an-0002wP-3x; Fri, 18 Aug 2023 19:09:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1qX8ah-0002w2-7y for emacs-orgmode@gnu.org; Fri, 18 Aug 2023 19:09:16 -0400 Received: from mail-oi1-x234.google.com ([2607:f8b0:4864:20::234]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1qX8ae-0003Vb-R2 for emacs-orgmode@gnu.org; Fri, 18 Aug 2023 19:09:15 -0400 Received: by mail-oi1-x234.google.com with SMTP id 5614622812f47-3a7f4f7a8easo650025b6e.2 for ; Fri, 18 Aug 2023 16:09:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1692400148; x=1693004948; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:from:to:cc:subject:date:message-id:reply-to; bh=EbquYvb1jXJLxX2OBl71dVqGx4JvJ0XD8elXrFQHsJ0=; b=PuOu+EErNSiKKK0udu65R2Es6n5bzkD7gxYUJP0pQxQgrTVD6PZqkmi0+Yljd27RU0 086Yu+3umA3I/hqQX0Jg2pCCW2C2RS5MNQpMltuewesX86+SEJ6a41Y6NOhI78vMgiqG Seef42dGNxxzS1H7CHWUBWBFPm0I6Sb/OXRwylYzxCe1kC19sYpAu7ACy5e/GER4HdVd jOebGtxarm2Mt/LcZDkwUITTHIq4dKkdVr1DabhOz5UgxznDcfoGzsPXn71it7rctfMc yiijHgqAS+TO5uktvHijse0lHF8EysJ9JA+7N/MQN2p9QT6DlCNFDVM/YmVn7NuyGnay gJIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692400148; x=1693004948; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=EbquYvb1jXJLxX2OBl71dVqGx4JvJ0XD8elXrFQHsJ0=; b=C7RkEMnCcLgz2cShA1bQf038H+1N0ibVMfJklgw9HlWvCnkyGf/xVQE7cKakvglUVL D8Fl8STUMctiQVkSmMsBcaHy7rofl1BnS5Ge5qdrFvJyR8GHXwX/zXYzODEpkQTl7pZm cTNsiGmsydGvMUDq72CFlNS7EMCtYTuNu/2ZtccZrH+EhWPAPdLkiulB3uJtl7kv22Sb m8dnS1+aNqHaBd5Gnn548ecKC9bfWFmbTUZLFTSB6Xj4/4UNvTdrkx4g9kk9QfTZj8ws dVsZyHfs5090oceKLEmaV2k9R8bvngmUnWXvmQMDaZ93sYtIab/idriZWe30e1Vutn2V A+sg== X-Gm-Message-State: AOJu0Yyfj2vm279ts+LtFFXmSmj700sYnR+jWLvdbvbcAgSMFnKtJZ85 DyCX+NNDEAhhxwoBog3h57M= X-Google-Smtp-Source: AGHT+IGFsrMe/8F7IZAO/PzTsDPdtTRhu/bdu1Z7cJBu55NFu5AwGazUC6SSFtGrPfaXrRgQ6USXfA== X-Received: by 2002:a05:6808:13d3:b0:3a7:49e5:e0da with SMTP id d19-20020a05680813d300b003a749e5e0damr874791oiw.26.1692400148262; Fri, 18 Aug 2023 16:09:08 -0700 (PDT) Received: from localhost (157-131-78-143.fiber.dynamic.sonic.net. [157.131.78.143]) by smtp.gmail.com with ESMTPSA id c3-20020aa78c03000000b00640dbbd7830sm2105702pfd.18.2023.08.18.16.09.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Aug 2023 16:09:07 -0700 (PDT) From: Jack Kamm To: Liu Hui Cc: emacs-orgmode@gnu.org, Ihor Radchenko Subject: Re: [PATCH] ob-python results handling for dicts, dataframes, arrays, and plots In-Reply-To: References: <87a5ur6f7w.fsf@gmail.com> Date: Fri, 18 Aug 2023 16:09:06 -0700 Message-ID: <87edk0aqy5.fsf@gmail.com> MIME-Version: 1.0 Content-Type: text/plain Received-SPF: pass client-ip=2607:f8b0:4864:20::234; envelope-from=jackkamm@gmail.com; helo=mail-oi1-x234.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -9.57 X-Spam-Score: -9.57 X-Migadu-Queue-Id: 49A7541CEE X-Migadu-Scanner: mx1.migadu.com X-TUID: TptyrsT1F32N Liu Hui writes: > Hi, > > Thank you for the patch! Thanks for your feedback, I've incorporated it into https://github.com/jackkamm/org-mode/tree/python-results-revisited-2023 More specifically, here: https://github.com/jackkamm/org-mode/commit/af1d18314073446045395ff7a3d1de0303e92586 > Do we need to limit the table/list size by default, or handle them > only with relevant result type (e.g. `table/list')? Dataframe/array > are often large. I've updated the patch so that Dataframe/Array are converted to table only when ":results table" is explicitly set now. If ":results table" is not set, they will be returned as string by default. So code blocks that return large dataframes/arrays can continue to be safely run. Note I did make an additional change to Numpy array default behavior: Previously, numpy arrays would be returned as table, but get mangled when they were very large, e.g.: #+begin_src python import numpy as np return np.zeros((30,40)) #+end_src #+RESULTS: | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) | ... | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) | (0 0 0 ... 0 0 0) | But now, Numpy array is returned in string form by default, in the same format as in Jupyter: #+begin_src python import numpy as np return np.zeros((30,40)) #+end_src #+RESULTS: : array([[0., 0., 0., ..., 0., 0., 0.], : [0., 0., 0., ..., 0., 0., 0.], : [0., 0., 0., ..., 0., 0., 0.], : ..., : [0., 0., 0., ..., 0., 0., 0.], : [0., 0., 0., ..., 0., 0., 0.], : [0., 0., 0., ..., 0., 0., 0.]]) >> + if isinstance(result, pandas.DataFrame): >> + result = [[''] + list(result.columns), None] + \ > > Here we can use '{}'.format(df.index.name) to show the name of index Patch has been updated to print the index name when it is non-None. > Maybe `org-babel-python--def-format-value' can be evaluated only once > in the session mode? It would shorten the string sent to the python > shell, where temp files are used for long strings. Patch has been updated to evaluate `org-babel-python--def-format-value' once per session.