I forgot a little piece in the last patch. This one is probably right.
It is based off of https://github.com/jkitchin/org-mode/blob/colored-link-3.