We focus on CREATE-Tag, which provides 210K training [Data Download] and 5K test [Data Download] samples. The training and test sets share 2,795 tags. For evaluation, we split these into 705 common and 2,090 rare tags.

We also build PEXEL-Tag, a newly collected dataset from the Pexels platform , comprising 162K training [Data Download] and 5K test [Data Download] samples. The dataset contains 28,094 unique tags, with 5,669 shared between training and testing. We further categorize these into 1,627 common and 4,042 rare tags for performance analysis

Open-set Metric (Tag Gain)

To evaluate the model’s ability to generate novel and informative tags, we introduce an open-set evaluation protocol based on Tag Gain, which jointly considers tag novelty, visual relevance, and semantic diversity.

Given a generated tag set \(T_{gen} = \{t_1, t_2, \dots, t_k\}\) and the corresponding visual input \(v\), we construct a filtered subset \(\hat{T}_{gen}\) by enforcing two criteria:

• Visual relevance: Each candidate tag \(t_j \in T_{gen}\) must be sufficiently aligned with the image content. We measure this using the CLIP-based similarity between the tag and the visual input.
  \[ \mathrm{sim}_{\text{CLIP}}(t_j, v) > \tau_{\text{rel}}. \]
• Semantic diversity: To avoid redundancy, each selected tag should be semantically dissimilar to the tags already included in \(\hat{T}_{gen}\). This is enforced by requiring:
  \[ \forall t' \in \hat{T}_{gen},\quad \mathrm{sim}_{\text{BGE}}(t_j, t') < \tau_{\text{div}}. \]

We apply a greedy filtering process: starting from an empty set \(\hat{T}_{gen} = \emptyset\), we iterate through the candidate tags (optionally sorted by visual relevance), and sequentially add a tag \(t_j\) to \(\hat{T}_{gen}\) only if it satisfies both conditions above. Typically, we set \(\tau_{\text{rel}}=0.3\) and \(\tau_{\text{div}}=0.8\). Formally:

\[ \hat{T}_{gen} = \left\{ t_j \in T_{gen} \;\middle|\; \mathrm{sim}_{\text{CLIP}}(t_j, v) > \tau_{\text{rel}} \land \forall t' \in \hat{T}_{gen},\ \mathrm{sim}_{\text{BGE}}(t_j, t') < \tau_{\text{div}} \right\}. \]

Based on the filtered tag set \(\hat{T}_{gen}\), we define two variants of Tag Gain to assess open-world tagging:

• Known Tag Gain (\(\Delta_{\text{known}}\)): The proportion of relevant but unannotated tags that were seen during training:
  \[ T_{\text{known}} = \left\{ t \in \hat{T}_{gen} \;\middle|\; t \notin T_{\text{gt}},\ t \in T_{\text{train}} \right\}, \] \[ \Delta_{\text{known}} = \frac{ \left| T_{\text{known}} \right| }{ \left| T_{\text{gt}} \right| }. \]
• Novel Tag Gain (\(\Delta_{\text{novel}}\)): The proportion of generated tags that are unseen in training:
  \[ T_{\text{novel}} = \left\{ t \in \hat{T}_{gen} \;\middle|\; \ t \notin T_{\text{train}} \right\}, \] \[ \Delta_{\text{novel}} = \frac{ \left| T_{\text{novel}} \right| }{ \left| T_{\text{gt}} \right| }. \]