encoding problem

encoding problem

[Julien Cubizolles]

I'm having a very strange problem with character encoding. I write all
my text files with emacs, with non-ascii characters (I'm french). I keep
a copy of many files (latex/org/...) on separate machines using
unison. Very often after a synchronization, the non-ascii charaters are
completely displayed wrong (à for à, ç for ç) in the org files, but
never in the latex files.

I guess it's more an Emacs than org files but I can't see what's special
in the org files that makes them more prone to such errors.

Is there a way to *fix* easily these corruptions on a file, ie searching
for all "weird" characters to replace ?

How could I prevent this from happening again (checking/changing
character encoding maybe ?)

Thanks for your help,

Julien.

Re: encoding problem

[Bernt Hansen]

Julien Cubizolles <j.cubizolles@free.fr> writes:

> I'm having a very strange problem with character encoding. I write all
> my text files with emacs, with non-ascii characters (I'm french). I keep
> a copy of many files (latex/org/...) on separate machines using
> unison. Very often after a synchronization, the non-ascii charaters are
> completely displayed wrong (à for à, ç for ç) in the org files, but
> never in the latex files.
>
> I guess it's more an Emacs than org files but I can't see what's special
> in the org files that makes them more prone to such errors.
>
> Is there a way to *fix* easily these corruptions on a file, ie searching
> for all "weird" characters to replace ?
>
> How could I prevent this from happening again (checking/changing
> character encoding maybe ?)
>
> Thanks for your help,
>
> Julien.

Hi Julien,

I get prompts for encoding when saving/exporting (on Windows only) so I
put the following at the top of my org-files

--8<---------------cut here---------------start------------->8---
# -*- coding: utf-8 -*-
--8<---------------cut here---------------end--------------->8---

which seems to fix the problem for me.  Maybe this will help?

Bernt

Re: encoding problem

[Eric S Fraga]

Bernt Hansen <bernt@norang.ca> writes:

> Julien Cubizolles <j.cubizolles@free.fr> writes:
>
>> I'm having a very strange problem with character encoding. I write all
>> my text files with emacs, with non-ascii characters (I'm french). I keep
>> a copy of many files (latex/org/...) on separate machines using
>> unison. Very often after a synchronization, the non-ascii charaters are
>> completely displayed wrong (à for à, ç for ç) in the org files, but
>> never in the latex files.
>>
>> I guess it's more an Emacs than org files but I can't see what's special
>> in the org files that makes them more prone to such errors.
>>
>> Is there a way to *fix* easily these corruptions on a file, ie searching
>> for all "weird" characters to replace ?
>>
>> How could I prevent this from happening again (checking/changing
>> character encoding maybe ?)
>>
>> Thanks for your help,
>>
>> Julien.
>
> Hi Julien,
>
> I get prompts for encoding when saving/exporting (on Windows only) so I
> put the following at the top of my org-files
>
> # -*- coding: utf-8 -*-
>
> which seems to fix the problem for me.  Maybe this will help?

I used to have this problem and it was incredibly annoying.  I also
started adding the line Bernt suggests but I kept forgetting for new
files.  I finally solved this problem by adding the following lines to
my emacs initialisation:

#+begin_src emacs-lisp
(prefer-coding-system 'utf-8)
(set-charset-priority 'unicode)
(setq default-process-coding-system '(utf-8-unix . utf-8-unix))
#+end_src

I couldn't tell you which of these matter or whether they are all
necessary but I don't have these problems any longer so I haven't
investigated any further!


-- 
: Eric S Fraga (GnuPG: 0xC89193D8FFFCF67D) in Emacs 24.1.50.1
: using Org release_7.8.10-630-g4144c5.dirty

Re: encoding problem

[Bernt Hansen]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:

> Bernt Hansen <bernt@norang.ca> writes:
>
>> Julien Cubizolles <j.cubizolles@free.fr> writes:
>>
>>> I'm having a very strange problem with character encoding. I write all
>>> my text files with emacs, with non-ascii characters (I'm french). I keep
>>> a copy of many files (latex/org/...) on separate machines using
>>> unison. Very often after a synchronization, the non-ascii charaters are
>>> completely displayed wrong (à for à, ç for ç) in the org files, but
>>> never in the latex files.
>>>
>>> I guess it's more an Emacs than org files but I can't see what's special
>>> in the org files that makes them more prone to such errors.
>>>
>>> Is there a way to *fix* easily these corruptions on a file, ie searching
>>> for all "weird" characters to replace ?
>>>
>>> How could I prevent this from happening again (checking/changing
>>> character encoding maybe ?)
>>>
>>> Thanks for your help,
>>>
>>> Julien.
>>
>> Hi Julien,
>>
>> I get prompts for encoding when saving/exporting (on Windows only) so I
>> put the following at the top of my org-files
>>
>> # -*- coding: utf-8 -*-
>>
>> which seems to fix the problem for me.  Maybe this will help?
>
> I used to have this problem and it was incredibly annoying.  I also
> started adding the line Bernt suggests but I kept forgetting for new
> files.  I finally solved this problem by adding the following lines to
> my emacs initialisation:
>
> #+begin_src emacs-lisp
> (prefer-coding-system 'utf-8)
> (set-charset-priority 'unicode)
> (setq default-process-coding-system '(utf-8-unix . utf-8-unix))
> #+end_src
>
> I couldn't tell you which of these matter or whether they are all
> necessary but I don't have these problems any longer so I haven't
> investigated any further!

Thanks Eric!

I'll try this and drop my mode line setting in each org file.  I still
encounter this when archiving for the first time to a new file -- since
I'm archive utf-8 content and the new target org file prompts for
encoding with my current setup.

Regards,
Bernt

Re: encoding problem

[Nick Dokos]

Bernt Hansen <bernt@norang.ca> wrote:

> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
> 
> > Bernt Hansen <bernt@norang.ca> writes:
> >
> >> Julien Cubizolles <j.cubizolles@free.fr> writes:
> >>
> >>> I'm having a very strange problem with character encoding. I write all
> >>> my text files with emacs, with non-ascii characters (I'm french). I keep
> >>> a copy of many files (latex/org/...) on separate machines using
> >>> unison. Very often after a synchronization, the non-ascii charaters are
> >>> completely displayed wrong (à for à, ç for ç) in the org files, but
> >>> never in the latex files.
> >>>
> >>> I guess it's more an Emacs than org files but I can't see what's special
> >>> in the org files that makes them more prone to such errors.
> >>>
> >>> Is there a way to *fix* easily these corruptions on a file, ie searching
> >>> for all "weird" characters to replace ?
> >>>
> >>> How could I prevent this from happening again (checking/changing
> >>> character encoding maybe ?)
> >>>
> >>> Thanks for your help,
> >>>
> >>> Julien.
> >>
> >> Hi Julien,
> >>
> >> I get prompts for encoding when saving/exporting (on Windows only) so I
> >> put the following at the top of my org-files
> >>
> >> # -*- coding: utf-8 -*-
> >>
> >> which seems to fix the problem for me.  Maybe this will help?
> >
> > I used to have this problem and it was incredibly annoying.  I also
> > started adding the line Bernt suggests but I kept forgetting for new
> > files.  I finally solved this problem by adding the following lines to
> > my emacs initialisation:
> >
> > #+begin_src emacs-lisp
> > (prefer-coding-system 'utf-8)
> > (set-charset-priority 'unicode)
> > (setq default-process-coding-system '(utf-8-unix . utf-8-unix))
> > #+end_src
> >
> > I couldn't tell you which of these matter or whether they are all
> > necessary but I don't have these problems any longer so I haven't
> > investigated any further!
> 
> Thanks Eric!
> 
> I'll try this and drop my mode line setting in each org file.  I still
> encounter this when archiving for the first time to a new file -- since
> I'm archive utf-8 content and the new target org file prompts for
> encoding with my current setup.
> 
> Regards,
> Bernt

Isn't the setting of LANG used during initialization to set these things?
I have LANG set to en_US.UTF-8  and new buffers are in utf-8-unix
(except for mail composition buffers: they are in undecided-unix).
I'm pretty sure I'm not mucking with coding systems anywhere in my emacs
initialization otherwise.

Nick

Re: encoding problem

[Bernt Hansen]

Nick Dokos <nicholas.dokos@hp.com> writes:

> Bernt Hansen <bernt@norang.ca> wrote:
>
>> Eric S Fraga <e.fraga@ucl.ac.uk> writes:
>> 
>> > Bernt Hansen <bernt@norang.ca> writes:
>> >
>> >> Julien Cubizolles <j.cubizolles@free.fr> writes:
>> >>
>> >>> I'm having a very strange problem with character encoding. I write all
>> >>> my text files with emacs, with non-ascii characters (I'm french). I keep
>> >>> a copy of many files (latex/org/...) on separate machines using
>> >>> unison. Very often after a synchronization, the non-ascii charaters are
>> >>> completely displayed wrong (à for à, ç for ç) in the org files, but
>> >>> never in the latex files.
>> >>>
>> >>> I guess it's more an Emacs than org files but I can't see what's special
>> >>> in the org files that makes them more prone to such errors.
>> >>>
>> >>> Is there a way to *fix* easily these corruptions on a file, ie searching
>> >>> for all "weird" characters to replace ?
>> >>>
>> >>> How could I prevent this from happening again (checking/changing
>> >>> character encoding maybe ?)
>> >>>
>> >>> Thanks for your help,
>> >>>
>> >>> Julien.
>> >>
>> >> Hi Julien,
>> >>
>> >> I get prompts for encoding when saving/exporting (on Windows only) so I
>> >> put the following at the top of my org-files
>> >>
>> >> # -*- coding: utf-8 -*-
>> >>
>> >> which seems to fix the problem for me.  Maybe this will help?
>> >
>> > I used to have this problem and it was incredibly annoying.  I also
>> > started adding the line Bernt suggests but I kept forgetting for new
>> > files.  I finally solved this problem by adding the following lines to
>> > my emacs initialisation:
>> >
>> > #+begin_src emacs-lisp
>> > (prefer-coding-system 'utf-8)
>> > (set-charset-priority 'unicode)
>> > (setq default-process-coding-system '(utf-8-unix . utf-8-unix))
>> > #+end_src
>> >
>> > I couldn't tell you which of these matter or whether they are all
>> > necessary but I don't have these problems any longer so I haven't
>> > investigated any further!
>> 
>> Thanks Eric!
>> 
>> I'll try this and drop my mode line setting in each org file.  I still
>> encounter this when archiving for the first time to a new file -- since
>> I'm archive utf-8 content and the new target org file prompts for
>> encoding with my current setup.
>> 
>> Regards,
>> Bernt
>
> Isn't the setting of LANG used during initialization to set these things?
> I have LANG set to en_US.UTF-8  and new buffers are in utf-8-unix
> (except for mail composition buffers: they are in undecided-unix).
> I'm pretty sure I'm not mucking with coding systems anywhere in my emacs
> initialization otherwise.
>
> Nick


Maybe - I'm having this issue on Windows...

-Bernt

Re: encoding problem

[Nick Dokos]

Bernt Hansen <bernt@norang.ca> wrote:

> Nick Dokos <nicholas.dokos@hp.com> writes:
> 
> > Isn't the setting of LANG used during initialization to set these things?
> > I have LANG set to en_US.UTF-8  and new buffers are in utf-8-unix
> > (except for mail composition buffers: they are in undecided-unix).
> > I'm pretty sure I'm not mucking with coding systems anywhere in my emacs
> > initialization otherwise.
> >
> > Nick
> 
> 
> Maybe - I'm having this issue on Windows...
> 

Ah - OK: forgive my parochialism. I guess the question is whether
whatever version of emacs you are using can access the locale
information on Windows and do the equivalent of what it does on POSIX-y
systems.

Nick

Re: encoding problem

[Julien Cubizolles]

Eric S Fraga <e.fraga@ucl.ac.uk> writes:


> I used to have this problem and it was incredibly annoying.  I also
> started adding the line Bernt suggests but I kept forgetting for new
> files.  I finally solved this problem by adding the following lines to
> my emacs initialisation:
>
> #+begin_src emacs-lisp
> (prefer-coding-system 'utf-8)
> (set-charset-priority 'unicode)
> (setq default-process-coding-system '(utf-8-unix . utf-8-unix))
> #+end_src

I'll give it a try, thanks for your suggestion. I think my encoding
problems come from the many copy-paste I made from web pages (wikipedia
mostly) to org-files. I didn't think it would mess anything since
wikipedia pages are encoded in utf-8, or so I thought.

Julien.