From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sebastian Rose Subject: Re: [bug] org-link-escape and (wrong-type-argument stringp nil) Date: Mon, 27 Sep 2010 00:43:12 +0200 Message-ID: <87bp7ki0sf.fsf@gmx.de> References: <87tylkwpq0.fsf@mundaneum.com> <87mxrc1bwj.wl%dmaus@ictsoc.de> <87sk14rz3t.fsf@gmx.de> <87sk128cuk.wl%dmaus@ictsoc.de> <87fwx1yhw5.fsf@gmx.de> <87aan8xpzy.wl%dmaus@ictsoc.de> <87d3s48c8c.fsf@gmx.de> <87bp7kicuh.wl%dmaus@ictsoc.de> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Return-path: Received: from [140.186.70.92] (port=53327 helo=eggs.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1OzzwK-0003m2-1H for emacs-orgmode@gnu.org; Sun, 26 Sep 2010 18:43:25 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.69) (envelope-from ) id 1OzzwI-0000q1-NR for emacs-orgmode@gnu.org; Sun, 26 Sep 2010 18:43:23 -0400 Received: from mailout-de.gmx.net ([213.165.64.23]:53224 helo=mail.gmx.net) by eggs.gnu.org with smtp (Exim 4.69) (envelope-from ) id 1OzzwI-0000pt-Ba for emacs-orgmode@gnu.org; Sun, 26 Sep 2010 18:43:22 -0400 In-Reply-To: <87bp7kicuh.wl%dmaus@ictsoc.de> (David Maus's message of "Sun, 26 Sep 2010 20:22:46 +0200") List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: David Maus Cc: =?utf-8?Q?S=C3=A9bastien?= Vauban , emacs-orgmode@gnu.org --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable David Maus writes: > Sebastian Rose wrote: >>David Maus writes: >>>> sh$ man utf-8 >>> >>> Thanks! I finally get a grip on one of my personal nightmares. > > >>It's not that bad, is it? :D > > Even better: It makes sense ;) > >>> The attached patch is the first step in this direction: It modifies >>> the algorithm of `org-link-escape', now iterating over the input >>> string with `mapconcat' and escaping all characters in the escape >>> table or are between 127 and 255. > >>Between 128 (1000 0000) and 255 ?? > >>The binary representation of 127 is 0111 1111 and valid ascii char. DEL >>actually (sh$ man ascii) > > Right, and that's why it is encoded: No control characters in a URI. > > The final algorithm for the shiny new unicode aware percent encoding > function would be: > > - percent encode all characters in TABLE > - percent encode all characters below 32 and above 126 > - encode the char in utf-8 > - percent escape all bytes of the encoded char > > The remaining problem is keeping backward compatibility. There are Org > files out there where "=C3=A1" is encoded as "%E1" and not "%C3A1". The > percent decoding function should be able to recognize these old > escapes and return the right value. There is no chance to do it in a secure way. But here's what's possible. These all work as expected: (org-protocol-unhex-string "%E1") ; =C3=A1 (org-protocol-unhex-string "%A1") ; =C2=A1 (org-protocol-unhex-string "%E1%A1") ; =C3=A1=C2=A1 (org-protocol-unhex-string "%C3%B6") ; still german =C3=B6 Also, capturing text from this page still works: http://www.jnto.go.jp/jpn/ --=-=-= Content-Type: text/x-diff; charset=utf-8 Content-Disposition: inline; filename=org-protocol-unhex-compound-single-byte.patch Content-Transfer-Encoding: quoted-printable diff --git a/lisp/org-protocol.el b/lisp/org-protocol.el index 21f28e7..f37ce1c 100644 --- a/lisp/org-protocol.el +++ b/lisp/org-protocol.el @@ -305,7 +305,7 @@ part." =20 (defun org-protocol-unhex-string(str) "Unhex hexified unicode strings as returned from the JavaScript function -encodeURIComponent. E.g. `%C3%B6' is the german Umlaut `=C3=BC'." +encodeURIComponent. E.g. `%C3%B6' is the german Umlaut `=C3=B6'." (setq str (or str "")) (let ((tmp "") (case-fold-search t)) @@ -321,7 +321,11 @@ encodeURIComponent. E.g. `%C3%B6' is the german Umlaut= `=C3=BC'." =20 =20 (defun org-protocol-unhex-compound (hex) - "Unhexify unicode hex-chars. E.g. `%C3%B6' is the German Umlaut `=C3=BC'= ." + "Unhexify unicode hex-chars. E.g. `%C3%B6' is the German Umlaut `=C3=B6'. +Note: this function also decodes single byte encodings like +`%E1' (\"=C3=A1\") if not followed by another `%[A-F0-9]{2}' group. +Singlebyte decoding is not secure though, since we could have +two single byte characters above 128 in a row." (let* ((bytes (remove "" (split-string hex "%"))) (ret "") (eat 0) @@ -353,9 +357,22 @@ encodeURIComponent. E.g. `%C3%B6' is the german Umlaut= `=C3=BC'." (setq val (logxor val xor)) (setq sum (+ (lsh sum shift) val)) (if (> eat 0) (setq eat (- eat 1))) - (when (=3D 0 eat) + (cond + ((=3D 0 eat) ;multi byte (setq ret (concat ret (org-protocol-char-to-string sum))) (setq sum 0)) + ((not bytes) ; single byte(s) + (let ((bytes (remove "" (split-string hex "%"))) + (ret "")) + (message "bytes: %s" bytes) + + (while bytes + (let* ((b (pop bytes)) + (a (elt b 0)) + (b (elt b 1))) + (setq ret + (concat ret (char-to-string + (+ (lsh a 4) b))))))))) )) ;; end (while bytes ret )) =20 --=-=-= Content-Type: text/plain Best wishes Sebastian --=-=-= Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Emacs-orgmode mailing list Please use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode --=-=-=--