Why Emacs shows ^M at the end of some lines in sh code blocks that use curl?

Why Emacs shows ^M at the end of some lines in sh code blocks that use curl?

[Rodrigo Morales]

[-- Attachment #1: Type: text/plain, Size: 3871 bytes --]


I saved the contents of the code block below to the file =/tmp/a.org=.

#+HEADER: :tangle /tmp/a.org
#+BEGIN_SRC org
,#+HEADER: :tangle no
,#+HEADER: :results verbatim
,#+BEGIN_SRC sh
curl --silent --request GET 'http://localhost:4891/posts'
,#+END_SRC

,#+HEADER: :tangle no
,#+HEADER: :results verbatim
,#+BEGIN_SRC sh
curl --include --silent --request GET 'http://localhost:4891/posts'
,#+END_SRC

,#+HEADER: :tangle no
,#+HEADER: :results verbatim
,#+BEGIN_SRC sh
seq 1 10
,#+END_SRC
#+END_SRC

I started Emacs using the command shown below.

#+BEGIN_SRC sh
emacs -Q  --eval "(require 'ob-shell)" /tmp/a.org
#+END_SRC

I evaluated the three =sh= code blocks in Emacs. Here's an screenshot
that shows the content of the buffer =/tmp/a.org= after the three code
blocks have been evaluated: https://archive.is/RnAD6 (edited for
clarity: https://archive.is/IJzuc ) Note that the character "^M" is
shown at the end of each line that contain response headers shown by
curl (in curl, response headers are shown using the flag
--include). Those who can't open the screenshot can refer to the code
bock below, I have manually replaced the character '^M' with the string
"^M" just so that you see what I am seeing in my computer. The character
^M is only shown for lines showing the response headers, not for lines
showing the response body.

#+BEGIN_SRC org
,#+HEADER: :tangle no
,#+HEADER: :results verbatim
,#+BEGIN_SRC sh
curl --include --silent --request GET 'http://localhost:4891/posts'
,#+END_SRC

,#+RESULTS:
,#+begin_example
HTTP/1.1 200 OK^M
Server: Werkzeug/3.0.4 Python/3.10.12^M
Date: Thu, 17 Oct 2024 14:57:28 GMT^M
Content-Type: application/json^M
Content-Length: 123^M
Connection: close^M
^M
[
  {
    "title": "Title of post 1"
  },
  {
    "title": "Title of post 2"
  },
  {
    "title": "Title of post 3"
  }
]
,#+end_example
#+END_SRC

I then pressed =M-x shell RET= and executed the same commands from the
sh code blocks. Here's an screenshot that shows the buffer =*shell*=
after the three commands have been executed: https://archive.is/XDyIg
(edited for clarity: https://archive.is/A8dkZ ) Note that the character
"^M" is not shown at the end of each line, as happened when the same
command was evaluated in an Org Mode code block.

I wish those ^M were not shown in the sh code blocks in the Org Mode
buffer. Does anyone know how to accomplish this behavior?

You can find the file =/tmp/a.org= (after the three code blocks have
been evaluated) attached in this message.

** Additional information

Below you can find a simple Python application that you can use as an
API to reproduce this behavior. The first code block below shows the
command that you can run to initialize the virtual environment. The
second code block shows the Flask application. The third code block
below shows the command to run the Flask application.

#+HEADER: :results silent
#+HEADER: :tangle no
#+BEGIN_SRC bash
rm -rf /tmp/python \
  && mkdir /tmp/python \
  && cd /tmp/python \
  && virtualenv venv \
  && source venv/bin/activate \
  && pip install flask
#+END_SRC

#+HEADER: :tangle /tmp/python/blog.py
#+BEGIN_SRC python
import flask

app = flask.Flask(__name__)

@app.route('/posts', methods = ['GET'])
def get_posts():
  return [
    {'title': 'Title of post 1'},
    {'title': 'Title of post 2'},
    {'title': 'Title of post 3'}
  ], 200

@app.route('/categories', methods = ['GET'])
def get_categories():
  return [
    {'name': 'Category 1'},
    {'name': 'Category 2'},
    {'name': 'Category 3'}
  ], 200
#+END_SRC

#+HEADER: :tangle no
#+BEGIN_SRC bash
cd /tmp/python && source venv/bin/activate && python -m flask --app blog --debug run --port 4891
#+END_SRC

** System information

Emacs version: GNU Emacs 29.4 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.33, cairo version 1.16.0) of 2024-10-12

OS: Ubuntu 22.04.5 LTS

Org mode version: 9.6.15


[-- Attachment #2: a.org --]
[-- Type: text/plain, Size: 912 bytes --]

#+HEADER: :tangle no
#+HEADER: :results verbatim
#+BEGIN_SRC sh
curl --silent --request GET 'http://localhost:4891/posts'
#+END_SRC

#+RESULTS:
#+begin_example
[
  {
    "title": "Title of post 1"
  },
  {
    "title": "Title of post 2"
  },
  {
    "title": "Title of post 3"
  }
]
#+end_example

#+HEADER: :tangle no
#+HEADER: :results verbatim
#+BEGIN_SRC sh
curl --include --silent --request GET 'http://localhost:4891/posts'
#+END_SRC

#+RESULTS:
#+begin_example
HTTP/1.1 200 OK
Server: Werkzeug/3.0.4 Python/3.10.12
Date: Thu, 17 Oct 2024 15:02:54 GMT
Content-Type: application/json
Content-Length: 123
Connection: close

[
  {
    "title": "Title of post 1"
  },
  {
    "title": "Title of post 2"
  },
  {
    "title": "Title of post 3"
  }
]
#+end_example

#+HEADER: :tangle no
#+HEADER: :results verbatim
#+BEGIN_SRC sh
seq 1 10
#+END_SRC

#+RESULTS:
#+begin_example
1
2
3
4
5
6
7
8
9
10
#+end_example

Re: Why Emacs shows ^M at the end of some lines in sh code blocks that use curl?

[Bruno Barbier]

Hi Rodrigo,

Rodrigo Morales <rodrigo-mailing-lists@morales.pe> writes:

> I saved the contents of the code block below to the file =/tmp/a.org=.
>
>
> I wish those ^M were not shown in the sh code blocks in the Org Mode
> buffer. Does anyone know how to accomplish this behavior?
>

Your result is mixing different "end-of-line" conventions.  If you
redirect the result into a file and open it with Emacs, you will get the
same result: it's not related to Org (HTTP headers use '\r\n', your JSON
'\n').

You could include a translation step. For example, using 'tr':
   #+begin_src shell :results raw
     curl --include --silent https://filesampleshub.com/download/code/json/sample1.json | tr -d '\r'
   #+end_src
   
If you are fetching arbitrary data, you do not want to do that though.
You may ask curl to dump the headers in a file, and translate only
that file (using --dump-header).

Note that getting any JSON file out there should give the same result;
i.e. no need to install Flask :)


HTH,

Bruno