From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp10.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id yF5QKdEvZGNucgEAbAwnHQ (envelope-from ) for ; Thu, 03 Nov 2022 22:17:05 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp10.migadu.com with LMTPS id YOg0KNEvZGMelwAAG6o9tA (envelope-from ) for ; Thu, 03 Nov 2022 22:17:05 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 4E1ED166D4 for ; Thu, 3 Nov 2022 22:17:05 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oqhZD-0008Vo-SP; Thu, 03 Nov 2022 17:16:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oqhZC-0008VW-Ej for emacs-orgmode@gnu.org; Thu, 03 Nov 2022 17:16:02 -0400 Received: from mail-ua1-x934.google.com ([2607:f8b0:4864:20::934]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1oqhZ7-00057d-29 for emacs-orgmode@gnu.org; Thu, 03 Nov 2022 17:16:02 -0400 Received: by mail-ua1-x934.google.com with SMTP id 97so1679394uam.0 for ; Thu, 03 Nov 2022 14:15:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=8T8Qtp2x83EQEni9AAGAKiKfZ2P/UKEu7rMw5ya919I=; b=hx+hpHrzr8APu3tCfV5rm3OOrvy7OEGDNalvPQ0qCf2gfH5yDiSZOEyc+jEZKx0IRq beklGYQT1N84/S102R99pduGbDZjO3gjzmBAAW79bs0rHkn6b8yiPGr4CcIZ1TnFMIOU YVVwxlvtmES1G1YVLWi+9qNSrkSbTjAPmYUgfj9i3lCfdz5kCX/754hDBJoB4fJOQGBA fSeFmrNhHTiYbPYAIhV+fIb/qeFwicXoWqcftZBK2sYugm99awYB76I4Gi9o74CjMzz0 fgr4giBLafbRA+Vy7ujhSytr2rk3nho6gAK8w0svZw3FQ2Nk5zUV5kBuFo0XG2/RsVF3 efsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8T8Qtp2x83EQEni9AAGAKiKfZ2P/UKEu7rMw5ya919I=; b=WOhSJqEfGVVK9b8xV/9WBHLG6p/KzldCo2YwD+9aoZQhN2l4Zid2Pdz+BpVYs8Q51d GhC986EHnBvEsUzdSsl5eHtz4TC0Q9dQ05gxhNrL47Npi0+Vukg0qjuIKg8yWe8VtPa0 /k2qIMlO/BG3AFmtOeiVHaPopOi87P8m6IF39tkm87qGgETg4nlZn7plOBQclwx5YDBV 7/hLmv4sQDAIruOE9FYu4bm+1jf2VZ27LiLrcqCnZ++7iIh8JwwuN+arJ/p2WNvflnQ2 w+KIiOW0aUzrntYZK3hzFS0LR2WPsH3Oszad/4WBgazLde/W0W2BjTgvusoBMsObp7/s R7RQ== X-Gm-Message-State: ACrzQf1L7lHwUhMluEjPI9fBW4roVeO5cve5PZSXYZRsIKmKjmhYgtkh 27AdbZg+pYZuIBs0hCW4RmTrnKLIKgaoh19p20fDqzEbkFA= X-Google-Smtp-Source: AMsMyM4xyaywnq4Q1abKno4AEJSKI7vGo38O5jda4GP5ATjOGPOBywsx3mMQ8oF4+cePsDANy90NXlmNrQpNnOnE+rU= X-Received: by 2002:ab0:20d2:0:b0:415:715c:1e70 with SMTP id z18-20020ab020d2000000b00415715c1e70mr1726816ual.81.1667510154746; Thu, 03 Nov 2022 14:15:54 -0700 (PDT) MIME-Version: 1.0 References: <878rktluej.fsf@localhost> <87leos7bnp.fsf@localhost> In-Reply-To: <87leos7bnp.fsf@localhost> From: =?UTF-8?Q?Christian_K=C3=B6stlin?= Date: Thu, 3 Nov 2022 22:15:18 +0100 Message-ID: Subject: Re: [PATCH] Ignore flaky ob-python tests To: Ihor Radchenko Cc: emacs-orgmode@gnu.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=2607:f8b0:4864:20::934; envelope-from=christian.koestlin@gmail.com; helo=mail-ua1-x934.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Emacs-orgmode" Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1667510225; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=8T8Qtp2x83EQEni9AAGAKiKfZ2P/UKEu7rMw5ya919I=; b=tlXymO5tdHWch7Uldey3QNKmKjs9z4+8PABzGp278rFBMrdjoHbSjhplTKbJxKihHEHKFz e7+ydBF6F2vsPnOivQnQgkJwn1+x4dnYq0TOInrwTty7nvRKlM7HZtDmCSdUrRnbblA4bT x542qhR1Px51ER/2o+HmRpHR0AgCsOB8luI88xsqe6/gORtqrVkDUKZctbNY5mFFzobhVP Y8fxfqIdR6lFqGI2YwoOL3lwwzsmXKPLlHu/lRD2dd4L/2NHLucXsEmUUG30YEoSCPFGx+ /cZ4GGhLhpDpSyzCnLD9hPgR/datR4Biu2uP4yJAkHUqnIfkJetLS4IPEiO1ug== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1667510225; a=rsa-sha256; cv=none; b=JB+eyu2el+SI7m76ZnsgO8d/KhZCbqxEW4n2GQEC0iWLmnxUVXke7SMSXIKzKnyw7U5Mne QgPZtBmkyQg77IdGYnX1j+F0P3HQUe+syMiezOKlj9BCYsgpI30qK8dBwrFVc23Qbp3BWy BJwxMkuy/KkipVOvWZ00cUfDzxyLY6ZWpqIpZ40JgOvby52plvQhMokSvezxFFc0qd6PPm eqZAxH9JKHnLhaZtnoTZTELqZY9ftRlTaEXjl73kUTreHl1CzohlJdp0EJvaYsKZYoTF+6 8hn/bjEJxcCnru/d7MhFSUKMnE1fJ+thOaORvr/X7OuP9R4AEhPYJolSgHcZNQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=hx+hpHrz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Spam-Score: -4.29 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=hx+hpHrz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-Migadu-Queue-Id: 4E1ED166D4 X-Spam-Score: -4.29 X-Migadu-Scanner: scn0.migadu.com X-TUID: RtwoN/4kQMxE Hi Ihor, I somehow seem to remember that the async tests were also broken with python.el (python-mode.el not installed). at least sometimes? or is this only the underscore test? If it's only the underscore test, I would suggest not installing python-mode for ci and ignoring the really flaky tests (i think the underscore one is a candidate here). I somehow got the async things mixed up with the flaky tests, which is not true for python-mode.el installed. what do you think? I really would like to see python tests pass ... Kind regards, Christian On Thu, Nov 3, 2022 at 8:13 AM Ihor Radchenko wrote: > > Christian K=C3=B6stlin writes: > > > With my patch the tests are still failing but the testsuite continues > > to run and returns > > with 0 as status code (one can see the failed test though). > > FYI, I am using automated scripts to run make test locally on multiple > Emacs versions. 0 status code would do no good for my testing. > > >> Maybe we can instead provide some variable that can turn-on ignoring > >> some problematic tests only during CI tests? > > Yes ... that would also be possible e.g. skip the test or test nothing > > if it's running on the ci. > > (I am talking only about the 3 tests!). > > More like just 1 test with underscore :) > The 3 tests in the patch are real failures because ob-python does not > fully support python-mode.el. > > -- > Ihor Radchenko // yantar92, > Org mode contributor, > Learn more about Org mode at . > Support Org development at , > or support my work at