From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp11.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms5.migadu.com with LMTPS id WCnTFDx/x2NzbAAAbAwnHQ (envelope-from ) for ; Wed, 18 Jan 2023 06:10:20 +0100 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp11.migadu.com with LMTPS id SNLaFDx/x2NqLQAA9RJhRA (envelope-from ) for ; Wed, 18 Jan 2023 06:10:20 +0100 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D809747A79 for ; Wed, 18 Jan 2023 06:10:19 +0100 (CET) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1pI0hY-0000Mz-Ra; Wed, 18 Jan 2023 00:09:32 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pI0hW-0000MV-0P for emacs-orgmode@gnu.org; Wed, 18 Jan 2023 00:09:30 -0500 Received: from sender4-op-o12.zoho.com ([136.143.188.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1pI0hT-0002jl-UL for emacs-orgmode@gnu.org; Wed, 18 Jan 2023 00:09:29 -0500 ARC-Seal: i=1; a=rsa-sha256; t=1674018561; cv=none; d=zohomail.com; s=zohoarc; b=f3jyy6RCmXgQweK/kQL1dS61N3AJb1+AX7KVLhWkn6QtbSdmOcWr4T9rCZuEZgwfm8UqiScGkqKd8lunsJrWz9T1OLatq5VodSObwzTb0RZPK3etB2IWrSXN/esPsJP8f/2ObopNiw+o1MYpuR3vc1H90o1i2q/iQ9xyOYVnlKk= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1674018561; h=Content-Type:Content-Transfer-Encoding:Cc:Date:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:To; bh=HG+KvH6xcjkCqq8gWFzWrLpY8gwHPA+PjMV4r7bw8QY=; b=gETwKtuss1KO7Npezc+2kR0Cb8LDhA1ZFyLOpcyjwVLhL4eu17FFtzsqrhe+qW9zic0jccrQTZ8jZdMOPDVc1NyOEa1zQZTM+N/PkFAqU99M9fxXGSpbp5N8/H1moEhmc6ECINlc8krl+AgPFot+1jqpZfkDW2EvMKiJvqfxpfw= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=excalamus.com; spf=pass smtp.mailfrom=matt@excalamus.com; dmarc=pass header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1674018561; s=zmail; d=excalamus.com; i=matt@excalamus.com; h=Date:Date:From:From:To:To:Cc:Cc:Message-ID:In-Reply-To:References:Subject:Subject:MIME-Version:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; bh=HG+KvH6xcjkCqq8gWFzWrLpY8gwHPA+PjMV4r7bw8QY=; b=OJSliLCob7KpQW7Gk9dJf2rO8xX81n/wQ9J6uaZaUAqBbO9vSYzkCtKBrHlkd9og fIpLDBkU+vGoYJnzqxjdHEk3pozFw0JdCO1EKxrefuSIxz3vZwDe4L43OzbNypL9KZo 37Lx7+IM7kuJOAVnqNW9sAQ3dGBSrHXwV7QdA4zc= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1674018559157576.1102676437126; Tue, 17 Jan 2023 21:09:19 -0800 (PST) Date: Wed, 18 Jan 2023 00:09:19 -0500 From: Matt To: "Osher Jacob" Cc: "emacs-orgmode" Message-ID: <185c34814a4.bcdd7b94578193.327779579906739164@excalamus.com> In-Reply-To: References: <185bc86c826.11ae860d1190626.407710511075709301@excalamus.com> <185bd43e452.bb3f5419220019.823265614059778953@excalamus.com> Subject: Re: [BUG] ob-shell doesn't evaluate last line on Windows (cmd/cmdproxy) [9.6.1 ( @ c:/Users/Osher/AppData/Roaming/.emacs.d/elpa/org-9.6.1/)] MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Importance: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail Received-SPF: pass client-ip=136.143.188.12; envelope-from=matt@excalamus.com; helo=sender4-op-o12.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: emacs-orgmode@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: emacs-orgmode-bounces+larch=yhetil.org@gnu.org Sender: emacs-orgmode-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1674018620; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=HG+KvH6xcjkCqq8gWFzWrLpY8gwHPA+PjMV4r7bw8QY=; b=K2uXX59gRZ772b/cYJXA8ER4OkHcUdP/sFrGXQ+pZ47bgdnKZNOvfgvAW+Ws5DFfq7qFmK smWMkbEmZ4YXy5SzU7Fotc+W9P8WnKKCCV+VLBSOLiyH0pRlnpnwWzgUpDnUwaibT6V6Xi iLPvFHX6aSVVmd0tM44D9masZkd01FgRCGfv2wWk6t43CQwKAD93TYqNRvPabRcV9lys0d +Mvkp94m8E3kAn1GtvpAI8uMavfNyD3C6IZ8WCBl/bhyeOEiJcl0aHgzh0Y3S+i8Mx32g2 sgX0D4D56oe306550/i7JvjM7THfkC0t6Ng0NILHBpcCzrOCjCt3nbkQ9fMmsQ== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=OJSliLCo; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" ARC-Seal: i=2; s=key1; d=yhetil.org; t=1674018620; a=rsa-sha256; cv=pass; b=jqZzlDuIxjRCZlWq4cdBg8G6ltSvGOSwaAnL62FempzwAva2X5qcNfV9GCC368IPpBEq0n 8eLTzEXBDFsHIS315STSHAdlUJPVzDAAQGTlr6Wrong6Baxoi6r6GIixVSBvqvepPxO7Cm pg3N2YthbK77QGhAtlMirw6086080z3VovKkoljZBADimDq0xeVMJgbZnEPFCnCaClkzFF eWYRrBRIHHwFCde9EAwmG9gLGE0GcTMLxsX3lfJnil/k7FTUzSGin1v1nYRoIMF9Ql+ob7 POL8p5TAHQODcrrQBDEc7okOKvt9bG9RPhZ583HjKCYQDC2P/PxXib3Hrh+R4A== X-Migadu-Scanner: scn1.migadu.com X-Migadu-Spam-Score: -6.82 X-Spam-Score: -6.82 X-Migadu-Queue-Id: D809747A79 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=excalamus.com header.s=zmail header.b=OJSliLCo; dmarc=none; arc=pass ("zohomail.com:s=zohoarc:i=1"); spf=pass (aspmx1.migadu.com: domain of "emacs-orgmode-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="emacs-orgmode-bounces+larch=yhetil.org@gnu.org" X-TUID: ta/BT4vUYs8y ---- On Tue, 17 Jan 2023 14:53:39 -0500 Osher Jacob wrote --- > changing shell-command-switch to "/k" or "-k", I get a similar output: Thanks for checking that. > You also mentioned the source code block is being passed through the "-c" flag as a command-line argument.I might be misunderstanding something here, but it seems like it is being passed through the stdin of the shell process when the calls process-file -> call-process are being made. That the block source is passed through "-c" is an educated guess. I keep finding myself in C land where it's less clear to me what precisely happens. You're correct that `call-process' acts on the block source. It's not clear to me whether that's done through stdin or whether being passed through stdin would even help us. Here's what I'm seeing. Evaluating the block goes through 4 lisp commands and then into C: 1. Calls `(org-babel-eval shell-file-name (org-trim body))' This evaluates as... (org-babel-eval "cmdproxy.exe" ; command, whatever `shell-file-name' is (org-trim body) ; query, the block body ) The query parameter (the block body) gets inserted into a temp buffer on which `org-babel--shell-command-on-region' is called. 2. Calls `(org-babel--shell-command-on-region command error-buffer)' on a temp buffer containing block body. This evaluates as... (org-babel--shell-command-on-region command ; "cmdproxy.exe" error-buffer ; # ) This in turn calls `process-file' where INPUT-FILE is a temp file containing the contents of the buffer `org-babel--shell-command-on-region' was called on (that is, the temp file contains the block body). 3. Calls `(process-file shell-file-name input-file (if error-file (list t error-file) t) nil shell-command-switch command)' This evaluates as... (process-file "cmdproxy.exe" ; shell-file-name "/path/to/temp/containing/block/source" ; input-file (t "/tmp/babel jTCHe/ob-error-yHOivA") ; output destination (like call-process's DESTINATION) nil ; display "-c" ; args "cmdproxy.exe" ; args ) Of course, I imagine the paths would be Windows paths when run on Windows. According to the documentation, the args are passed to the process (cmdproxy.exe) "verbatim without filename handling, as `call-process' does." The `call-process' documentation says, "remaining arguments ARGS are strings passed as command arguments to PROGRAM." To me, that sounds like cmdproxy.exe gets "passed into" cmdproxy.exe. This would explain the nested shell calls. If all this were written out, I imagine it would look something like: cmdproxy.exe -c cmdproxy.exe What's not clear to me is how the INPUT-FILE is handled. The `process-file' documentation says "normally". I assume it's how `call-process' handles INPUT-FILE, but the documentation doesn't really address it. Finally, `process-file' finally calls `call-process'. 4. Calls `(apply 'call-process program (or lc infile) (if stderr-file (list (car buffer) stderr-file) buffer) display args)' This evaluates as... (apply 'call-process program ; cmdproxy (or lc infile) ; local-copy of the INFILE, "/path/to/temp/containing/block/source" (if stderr-file (list (car buffer) stderr-file) buffer) ; (t "/tmp/babel jTCHe/ob-error-yHOivA" ) display ; nil args) ; ("-c" "cmdproxy.exe") How this actually works is non-trivial (https://git.savannah.gnu.org/cgit/emacs.git/tree/src/callproc.c#n250) and not something I understand at the moment. We can see here that indeed a call like `cmdproxy.exe -c cmdproxy.exe' is being made. Still, I'm not sure how the INPUT-FILE gets processed. For example, is it passed in the second cmdproxy call? Or, maybe it gets called first and then the second call to cmdproxy happens and hangs? I don't know. > Any ideas on how to proceed from here? I have two ideas. 1. Another naive work around attempt. Again, I'm going from memory, documentation, and what I have previously written. I have in my init a command to open a terminal when working on Windows that looks like: (start-process "cmd" nil "cmd.exe" "/C" "start" "cmd.exe" "/K" "cd" dir) This starts a cmd process with /c which terminates the prompt after the command that follows is run. The command That follows starts another cmd process with /k which changes the directory and leaves the terminal open. I have another command to open QGIS that does the same thing: (start-process "cmd" nil "cmd.exe" "/C" "start" "\"qgis\"" "cmd.exe" "/K" "C:\\Program Files\\QGIS 3.22.3\\bin\\qgis.bat") It's not clear to me why the extra call to a second cmd with /k is needed. Maybe it's copy-pasta or Windows being Windows. Anyway, I mention these because I wonder if we might be able to do something similar. Try changing `shell-file-name' to "cmd.exe" (ideally, the full path) and `shell-command-switch' to "/k" (or maybe "-k"). My hope would be that the final call would look something like, cmd.exe /k cmd.exe /k Given my analysis, I'm not sure if there's a way we could trick the call into being `cmd.exe /c cmd.exe /k input-file'. 2. We could write some ob-shell code to explicitly handle Windows cmd and powershell. For example, call `process-file' similar to the stdin/cmdline branch of `org-babel-sh-evaluate'. I'm open to this, but, like I've said, I don't have a Windows machine to work on right now. I'd not be able to verify it. I, and I'm sure others, would be happy to guide you if that's something you'd like to help implement. Thoughts?