I think the problem is general. If you work with any timestamp that is agenda like, you can't check using this function if it's active or inactive. The one solution would be to remove parameter "agenda" and consider every timestamp as a agenda like (the "timestamp" in "org-at-timestamp-p" suggest that there is time information in it anyway) by default, or keep it as it is but extend parameter list with support of named parameters where agenda-like can be activated with :agenda t, inactive timestamp with :inactive t, with default values nil.

Thanks,

Adam