Insert link with "foreign" character - cannot save

Insert link with "foreign" character - cannot save

[Mattias Jämting]

(I'm using English Windows Vista x64, Emacs 23.1 and Org-mode 6.32b)

So i'm doing C-u C-c C-l to browse for a file in order to insert a link to it.

The path and/or the filename contains for instance an ö (an o with two
dots above it, also the swedish word for "island"), which gets
translated in my org-file as \366.

When I try to save the file I see the message:

 These default coding systems were tried to encode text  in the buffer
`jwd.org':
   (utf-8-dos (79 . 4194294))
 However, each of them encountered characters it couldn't encode:
   utf-8-dos cannot encode these:  These default coding systems were
tried to encode text  in the buffer `jwd.org':
   (utf-8-dos (79 . 4194294))
 However, each of them encountered characters it couldn't encode:
   utf-8-dos cannot encode these:  \366

Next I tried to hack myself a fix :-)

I added (?\366 . "%F6") to org-link-escape-chars and ran make on it
again, but it didn't seem to work.

So what can I try next?

Best regards,
Mattias

Insert link with "foreign" character - cannot save

[Mattias Jämting]

(I'm using English Windows Vista x64, Emacs 23.1 and Org-mode 6.32b)

So i'm doing C-u C-c C-l to browse for a file in order to insert a link to
it.

The path and/or the filename contains for instance an ö (an o with two dots
above it, also the swedish word for "island"), which gets translated in my
org-file as \366.

When I try to save the file I see the message:

 These default coding systems were tried to encode text
 in the buffer `jwd.org':
   (utf-8-dos (79 . 4194294))
 However, each of them encountered characters it couldn't encode:
   utf-8-dos cannot encode these:  These default coding systems were tried
to encode text
 in the buffer `jwd.org':
   (utf-8-dos (79 . 4194294))
 However, each of them encountered characters it couldn't encode:
   utf-8-dos cannot encode these:  \366

Next I tried to hack myself a fix :-)

I added (?\366 . "%F6") to org-link-escape-chars and ran make on it again,
but it didn't seem to work.

So what can I try next?

Best regards,
Mattias 

Re: Insert link with "foreign" character - cannot save

[Carsten Dominik]

Hi Mattias,

I tried that, and my buffer swiched to unicode encoding automatically.

Unfortunately I don't know much about coding systems, and so I do
not know how to fix this.

Anyone????

- Carsten

On Nov 12, 2009, at 10:28 AM, Mattias Jämting wrote:

> (I'm using English Windows Vista x64, Emacs 23.1 and Org-mode 6.32b)
>
> So i'm doing C-u C-c C-l to browse for a file in order to insert a  
> link to
> it.
>
> The path and/or the filename contains for instance an ö (an o with  
> two dots
> above it, also the swedish word for "island"), which gets translated  
> in my
> org-file as \366.
>
> When I try to save the file I see the message:
>
> These default coding systems were tried to encode text
> in the buffer `jwd.org':
>   (utf-8-dos (79 . 4194294))
> However, each of them encountered characters it couldn't encode:
>   utf-8-dos cannot encode these:  These default coding systems were  
> tried
> to encode text
> in the buffer `jwd.org':
>   (utf-8-dos (79 . 4194294))
> However, each of them encountered characters it couldn't encode:
>   utf-8-dos cannot encode these:  \366
>
> Next I tried to hack myself a fix :-)
>
> I added (?\366 . "%F6") to org-link-escape-chars and ran make on it  
> again,
> but it didn't seem to work.
>
> So what can I try next?
>
> Best regards,
> Mattias
>
>
>
>
>
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten

Re: Insert link with "foreign" character - cannot save

[Sebastian Rose]

Carsten Dominik <carsten.dominik@gmail.com> writes:

> Hi Mattias,
>
> I tried that, and my buffer swiched to unicode encoding automatically.
>
> Unfortunately I don't know much about coding systems, and so I do
> not know how to fix this.
>
> Anyone????
>
> - Carsten
>
> On Nov 12, 2009, at 10:28 AM, Mattias Jämting wrote:
>
>> (I'm using English Windows Vista x64, Emacs 23.1 and Org-mode 6.32b)
>>
>> So i'm doing C-u C-c C-l to browse for a file in order to insert a link to
>> it.
>>
>> The path and/or the filename contains for instance an ö (an o with two dots
>> above it, also the swedish word for "island"), which gets translated in my
>> org-file as \366.
>>
>> When I try to save the file I see the message:
>>
>> These default coding systems were tried to encode text
>> in the buffer `jwd.org':
>>   (utf-8-dos (79 . 4194294))
>> However, each of them encountered characters it couldn't encode:
>>   utf-8-dos cannot encode these:  These default coding systems were tried
>> to encode text
>> in the buffer `jwd.org':
>>   (utf-8-dos (79 . 4194294))
>> However, each of them encountered characters it couldn't encode:
>>   utf-8-dos cannot encode these:  \366
>>
>> Next I tried to hack myself a fix :-)
>>
>> I added (?\366 . "%F6") to org-link-escape-chars and ran make on it again,
>> but it didn't seem to work.


It's true, you cannot encode the bytes with dec. values above 127 in
utf-8 (see `man utf-8', unicode.org, whatever).



Seems your filenames are not utf-8 encoded.

Here it works, because on current Linux distros (and MAC OS??) filenames
are utf-8 encoded:

  C-u C-c C-f
  nä TAB RET

  [[file:nächtes-nötiges.org][Umlaute in Dateinamen]]


Maybe, if the file is from an old system, rename it (twice, to give it
the original name) and try again. Would that work?




There's an interesting discussion going on on emacs-devel, that might be
related (but it's not about filenames). You may read the entire thread
here:

  http://lists.gnu.org/archive/html/emacs-devel/2009-11/msg00661.html


The fazit so far, as I understood it, is, that Emasc 23 distinguishes
single and multibyte strings. Better not use array functions to handle
strings (which are multibyte internaly) in Emacs 23.

The OP did

  (setq nl "\n")
  (aset nl 0 ?ñ
  (insert nl)

which sets the first _byte_ of an array to 241, which in turn has no
valid representation on screen as character in Unicode (see `man urf-8'
and unicode.org). Thus Emacs insert \361 - for some reason :)



   Sebastian

Re: Insert link with "foreign" character - cannot save

[Sebastian Rose]

OK, this is a severe Emacs 23 bug.


I can reproduce the errors like this:


sh$ mkdir test
sh$ cd test
sh$ touch ümläute.org
sh$ ls
ümläute.org
sh$ convmv --notest -f utf-8 -t iso-8859-1 ümläute.org 
Your Perl version has fleas #37757 #49830 
mv "./ümläute.org"	"./�ml�ute.org"
Ready!
sh$ ls
?ml?ute.org


Now I open an Org-file in Emacs 23, and do

  C-u C-c C-l

Navigate to that file and

BAMM - cannot save my work anymore!!!



Another nice test: Try to recode an UTF-8 encoded file name with German
Umlauts unsing

  M-x recode-file-name 

It does not work either.



I'll file an Emacs bug report as this is no bug in Org-mode.




  Sebastian





Sebastian Rose <sebastian_rose@gmx.de> writes:
> Carsten Dominik <carsten.dominik@gmail.com> writes:
>
>> Hi Mattias,
>>
>> I tried that, and my buffer swiched to unicode encoding automatically.
>>
>> Unfortunately I don't know much about coding systems, and so I do
>> not know how to fix this.
>>
>> Anyone????
>>
>> - Carsten
>>
>> On Nov 12, 2009, at 10:28 AM, Mattias Jämting wrote:
>>
>>> (I'm using English Windows Vista x64, Emacs 23.1 and Org-mode 6.32b)
>>>
>>> So i'm doing C-u C-c C-l to browse for a file in order to insert a link to
>>> it.
>>>
>>> The path and/or the filename contains for instance an ö (an o with two dots
>>> above it, also the swedish word for "island"), which gets translated in my
>>> org-file as \366.
>>>
>>> When I try to save the file I see the message:
>>>
>>> These default coding systems were tried to encode text
>>> in the buffer `jwd.org':
>>>   (utf-8-dos (79 . 4194294))
>>> However, each of them encountered characters it couldn't encode:
>>>   utf-8-dos cannot encode these:  These default coding systems were tried
>>> to encode text
>>> in the buffer `jwd.org':
>>>   (utf-8-dos (79 . 4194294))
>>> However, each of them encountered characters it couldn't encode:
>>>   utf-8-dos cannot encode these:  \366
>>>
>>> Next I tried to hack myself a fix :-)
>>>
>>> I added (?\366 . "%F6") to org-link-escape-chars and ran make on it again,
>>> but it didn't seem to work.
>
>
> It's true, you cannot encode the bytes with dec. values above 127 in
> utf-8 (see `man utf-8', unicode.org, whatever).
>
>
>
> Seems your filenames are not utf-8 encoded.
>
> Here it works, because on current Linux distros (and MAC OS??) filenames
> are utf-8 encoded:
>
>   C-u C-c C-f
>   nä TAB RET
>
>   [[file:nächtes-nötiges.org][Umlaute in Dateinamen]]
>
>
> Maybe, if the file is from an old system, rename it (twice, to give it
> the original name) and try again. Would that work?
>
>
>
>
> There's an interesting discussion going on on emacs-devel, that might be
> related (but it's not about filenames). You may read the entire thread
> here:
>
>   http://lists.gnu.org/archive/html/emacs-devel/2009-11/msg00661.html
>
>
> The fazit so far, as I understood it, is, that Emasc 23 distinguishes
> single and multibyte strings. Better not use array functions to handle
> strings (which are multibyte internaly) in Emacs 23.
>
> The OP did
>
>   (setq nl "\n")
>   (aset nl 0 ?ñ
>   (insert nl)
>
> which sets the first _byte_ of an array to 241, which in turn has no
> valid representation on screen as character in Unicode (see `man urf-8'
> and unicode.org). Thus Emacs insert \361 - for some reason :)
>
>
>
>    Sebastian
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Remember: use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

Re: Insert link with "foreign" character - cannot save

[Sebastian Rose]

Sebastian Rose <sebastian_rose@gmx.de> writes:
> OK, this is a severe Emacs 23 bug.
>


Hm -- it's not that severe maybe. I can now save my work encoded as raw
text. 

Also, I can follow the link, that is created. It terribly ugly, but
working.


  Sebastian

Re: Insert link with "foreign" character - cannot save

[Mattias Jämting]

Hello again,

I've now got it to work. Not really sure why. This is what i did:

I removed

(set-terminal-coding-system 'utf-8)
(set-keyboard-coding-system 'utf-8)
(prefer-coding-system 'utf-8-dos)

from my init.el

And replaced it with

(modify-coding-system-alist 'file "\\.org\\'" 'utf-8-dos)
(modify-coding-system-alist 'file "\\.org_archive\\'" 'utf-8-dos)

And everything works great! Thanks anyway.

/Mattias

Mattias Jämting, Jämting Web & Design
www.jamting.se | +46 (0)70 6760182



On Sun, Nov 22, 2009 at 02:32, Sebastian Rose <sebastian_rose@gmx.de> wrote:
> Also, I can follow the link, that is created. It terribly ugly, but
> working.

Re: Insert link with "foreign" character - cannot save

[Carsten Dominik]

Hi Mattias,

thanks for telling us what worked.

If someone can figure out what is going on here, a FAQ entry would be  
MUCH appreciated.

- Carsten

On Dec 8, 2009, at 12:36 AM, Mattias Jämting wrote:

> Hello again,
>
> I've now got it to work. Not really sure why. This is what i did:
>
> I removed
>
> (set-terminal-coding-system 'utf-8)
> (set-keyboard-coding-system 'utf-8)
> (prefer-coding-system 'utf-8-dos)
>
> from my init.el
>
> And replaced it with
>
> (modify-coding-system-alist 'file "\\.org\\'" 'utf-8-dos)
> (modify-coding-system-alist 'file "\\.org_archive\\'" 'utf-8-dos)
>
> And everything works great! Thanks anyway.
>
> /Mattias
>
> Mattias Jämting, Jämting Web & Design
> www.jamting.se | +46 (0)70 6760182
>
>
>
> On Sun, Nov 22, 2009 at 02:32, Sebastian Rose  
> <sebastian_rose@gmx.de> wrote:
>> Also, I can follow the link, that is created. It terribly ugly, but
>> working.
>
>
> _______________________________________________
> Emacs-orgmode mailing list
> Please use `Reply All' to send replies to the list.
> Emacs-orgmode@gnu.org
> http://lists.gnu.org/mailman/listinfo/emacs-orgmode

- Carsten