From mboxrd@z Thu Jan 1 00:00:00 1970 From: Christopher League Subject: [PATCH] fix org-feed when retrieve-method is curl or wget Date: Tue, 30 Jun 2009 11:59:20 -0400 Message-ID: Mime-Version: 1.0 (Apple Message framework v935.3) Content-Type: multipart/mixed; boundary=Apple-Mail-31--365095689 Return-path: Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MLfk3-0006kP-An for emacs-orgmode@gnu.org; Tue, 30 Jun 2009 11:59:31 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MLfjy-0006Yg-GS for Emacs-orgmode@gnu.org; Tue, 30 Jun 2009 11:59:30 -0400 Received: from [199.232.76.173] (port=54907 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MLfjy-0006YU-CO for Emacs-orgmode@gnu.org; Tue, 30 Jun 2009 11:59:26 -0400 Received: from contrapunctus.net ([207.210.219.173]:60084) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1MLfjx-0006vE-LP for Emacs-orgmode@gnu.org; Tue, 30 Jun 2009 11:59:26 -0400 Received: from sagan.home.lan (pool-173-56-122-34.nycmny.fios.verizon.net [173.56.122.34]) by contrapunctus.net (Postfix) with ESMTPSA id 0819982AF for ; Tue, 30 Jun 2009 11:59:20 -0400 (EDT) List-Id: "General discussions about Org-mode." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org Errors-To: emacs-orgmode-bounces+geo-emacs-orgmode=m.gmane.org@gnu.org To: Emacs-orgmode@gnu.org --Apple-Mail-31--365095689 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Hi Carsten and everyone. I love using org-feed, to gather various collection points (delicious, starred in google reader, dial2do, etc) into org-mode. I tried switching the org-feed-retrieve-method to curl or wget, and encountered some bugs. The fixes were simple, and the full details are in the attached git patch. The problem I was trying to solve, however, was that delicious.com would return a "500 server error" sometimes with url.el, and I'm not sure why. It returns RSS content anyway, with the error message as an . Org-feed doesn't notice the HTTP response status, and processes the error as if it were a legit item (which means that next time the error occurs, it is silent). Now that I got curl working, the 500 doesn't seem to happen anymore. My hypotheses so far: maybe it has something to do with the user- agent, or with mangling special characters in the URL. The delicious URL contains the '&' argument separator, and when the error message comes back, it appears with something like '&' in it.. as if it were replaced twice. I haven't traced further, to determine if fault lies with url.el or with delicious.com. Best wishes Chris --Apple-Mail-31--365095689 Content-Disposition: attachment; filename=0001-fix-org-feed-when-retrieve-method-is-curl-or-wget.patch Content-Type: application/octet-stream; x-unix-mode=0644; name="0001-fix-org-feed-when-retrieve-method-is-curl-or-wget.patch" Content-Transfer-Encoding: quoted-printable =46rom=200fe1cac25d1d189881a136028e3b2fca5ec6f377=20Mon=20Sep=2017=20= 00:00:00=202001=0AFrom:=20Christopher=20League=20= =0ADate:=20Tue,=2030=20Jun=202009=2011:34:50=20= -0400=0ASubject:=20[PATCH]=20fix=20org-feed=20when=20retrieve-method=20= is=20curl=20or=20wget=0A=0A-=20test=20on=20line=20312=20failed=20because=20= these=20methods=20returned=20a=20string=20instead=0A=20=20of=20a=20= buffer=0A=0A-=20requesting=20'wget=20actually=20executed=20"curl",=20= with=20bad=20parameters=0A=0A-=20curl=20needs=20--silent,=20so=20that=20= progress=20messages=20don't=20interrupt=20content=0A=0A-=20atom=20parser=20= had=20code=20to=20skip=20HTTP=20headers,=20but=20these=20are=20present=20= only=0A=20=20when=20using=20url-retrieve-synchronously;=20caused=20= errors=20with=20curl/wget.=0A=20=20Instead,=20remove=20HTTP=20headers=20= right=20after=20feed=20buffer=20is=20populated.=0A---=0A=20= lisp/org-feed.el=20|=20=20=2022=20++++++++++++++--------=0A=201=20files=20= changed,=2014=20insertions(+),=208=20deletions(-)=0A=0Adiff=20--git=20= a/lisp/org-feed.el=20b/lisp/org-feed.el=0Aindex=20edf49d1..5e14e00=20= 100644=0A---=20a/lisp/org-feed.el=0A+++=20b/lisp/org-feed.el=0A@@=20= -309,7=20+309,7=20@@=20it=20can=20be=20a=20list=20structured=20like=20an=20= entry=20in=20`org-feed-alist'."=0A=20=09=20=20feed-buffer=20inbox-pos=20= new-formatted=0A=20=09=20=20entries=20old-status=20status=20new=20= changed=20guid-alist=20e=20guid=20olds)=0A=20=20=20=20=20=20=20(setq=20= feed-buffer=20(org-feed-get-feed=20url))=0A-=20=20=20=20=20=20(unless=20= (and=20feed-buffer=20(bufferp=20feed-buffer))=0A+=20=20=20=20=20=20= (unless=20(and=20feed-buffer=20(bufferp=20(get-buffer=20feed-buffer)))=0A= =20=09(error=20"Cannot=20get=20feed=20%s"=20name))=0A=20=20=20=20=20=20=20= (when=20retrieve-only=0A=20=09(throw=20'exit=20feed-buffer))=0A@@=20= -549,18=20+549,28=20@@=20If=20that=20property=20is=20already=20present,=20= nothing=20changes."=0A=20=09=20=20=20=20=20=20=20(org-split-string=20s=20= "\n")=0A=20=09=20=20=20=20=20=20=20(concat=20"\n"=20(make-string=20n=20= ?\=20)))))=0A=20=0A+(defun=20org-feed-skip-http-headers=20(buffer)=0A+=20= =20"Remove=20HTTP=20headers=20from=20BUFFER,=20and=20return=20it.=0A= +Assumes=20headers=20are=20indeed=20present!"=0A+=20=20= (with-current-buffer=20buffer=0A+=20=20=20=20(widen)=0A+=20=20=20=20= (goto-char=20(point-min))=0A+=20=20=20=20(search-forward=20"\n\n")=0A+=20= =20=20=20(delete-region=20(point-min)=20(point))=0A+=20=20=20=20buffer))=0A= +=0A=20(defun=20org-feed-get-feed=20(url)=0A=20=20=20"Get=20the=20RSS=20= feed=20file=20at=20URL=20and=20return=20the=20buffer."=0A=20=20=20(cond=0A= =20=20=20=20((eq=20org-feed-retrieve-method=20= 'url-retrieve-synchronously)=0A-=20=20=20=20(url-retrieve-synchronously=20= url))=0A+=20=20=20=20(org-feed-skip-http-headers=20= (url-retrieve-synchronously=20url)))=0A=20=20=20=20((eq=20= org-feed-retrieve-method=20'curl)=0A=20=20=20=20=20(ignore-errors=20= (kill-buffer=20org-feed-buffer))=0A-=20=20=20=20(call-process=20"curl"=20= nil=20org-feed-buffer=20nil=20url)=0A+=20=20=20=20(call-process=20"curl"=20= nil=20org-feed-buffer=20nil=20"--silent"=20url)=0A=20=20=20=20=20= org-feed-buffer)=0A=20=20=20=20((eq=20org-feed-retrieve-method=20'wget)=0A= =20=20=20=20=20(ignore-errors=20(kill-buffer=20org-feed-buffer))=0A-=20=20= =20=20(call-process=20"curl"=20nil=20org-feed-buffer=20nil=20"-q"=20"-O"=20= "-"=20url)=0A+=20=20=20=20(call-process=20"wget"=20nil=20org-feed-buffer=20= nil=20"-q"=20"-O"=20"-"=20url)=0A=20=20=20=20=20org-feed-buffer)=0A=20=20= =20=20((functionp=20org-feed-retrieve-method)=0A=20=20=20=20=20(funcall=20= org-feed-retrieve-method=20url))))=0A@@=20-610,10=20+620,6=20@@=20The=20= `:item-full-text'=20property=20actually=20contains=20the=20sexp=0A=20= formatted=20as=20a=20string,=20not=20the=20original=20XML=20data."=0A=20=20= =20(with-current-buffer=20buffer=0A=20=20=20=20=20(widen)=0A-=20=20=20=20= (goto-char=20(point-min))=0A-=20=20=20=20;;=20Skip=20HTTP=20headers=0A-=20= =20=20=20(search-forward=20"\n\n")=0A-=20=20=20=20(delete-region=20= (point-min)=20(point))=0A=20=20=20=20=20(let=20((feed=20(car=20= (xml-parse-region=20(point-min)=20(point-max)))))=0A=20=20=20=20=20=20=20= (mapcar=0A=20=20=20=20=20=20=20=20(lambda=20(entry)=0A--=20=0A1.6.3.3=0A=0A= --Apple-Mail-31--365095689 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit --Apple-Mail-31--365095689 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode --Apple-Mail-31--365095689--