The paper, titled 'OVS Meets Continual Learning: Towards Sustainable Open-Vocabulary Segmentation', points out that when additional training data becomes available, existing fine-tuning, retraining and continual learning approaches do not sufficiently extend the recognition capability of OVS models. It also notes that, unlike foundation models pre-trained on large-scale data, current OVS models do not perform well enough to be used as they are.

To resolve this, the team proposes ConOVS, a continual learning framework for OVS. ConOVS uses a mixture-of-experts strategy that dynamically merges decoder weights according to the input sample. Each time a new dataset is added, the decoder is lightly fine-tuned, and weight interpolation/merging between the original model and the expert decoder is performed per sample, extending capability without performance degradation.

In experiments ConOVS consistently outperformed a range of comparison methods. In particular, the team observed that as datasets were progressively expanded, performance also improved on zero-shot test sets containing classes not included in training. This shows that continual expansion can strengthen rather than harm generalization performance.

Aimed at the practical limitations of OVS, the research presents a training procedure that simultaneously considers the ability to absorb additional data, cost efficiency and generalization. It is expected to provide a direct reference point for building sustainable OVS in industry and academia.

- Conference page: https://neurips.cc/

- Paper link: https://arxiv.org/abs/2410.11536

- Code link: https://github.com/dongjunhwang/ConOVS