I modified the scheme and the function. I think, that these 2 if-s just complicate the code for comprehension: we have 3 cases,
for each of them we should return the appropriate value.

Here is my variant of the function, including the fix, that you suggested:

(defun org-read-prop (prop)
      "Convert the string property PROP to a number if appropriate.
Otherwise if prop looks like a list (meaning it starts with a
'(') then read it as lisp, otherwise return it unmodified as a
string."
      (if (and (stringp prop) (not (equal prop "")))
            (let ((out (string-to-number prop)))
                  (if (equal out 0)
                        (cond
                              ((or
                                     (equal "(" (substring prop 0 1)) 
                                     (equal "'" (substring prop 0 1)))

                                    (condition-case nil
                                          (read prop)
                                          (error prop)))
                              ((string-match "^\\(+0\\|-0\\|0\\)$" prop)
                                    0)
                              (t  
                                    (set-text-properties 0 (length prop) nil prop)
                                    prop))
                        out))
            prop))


All the tests work fine with the new version.
When I tried

(org-read-prop "(1 2 3))") -> (1 2 3)

It gave me (1 2 3), ignoring the last ')'. Seems to be the read function bug.

At last I got rid of these nasty little squares on the scheme :)