The research team of Professor Nang Jong-ho, Department of Computer Science & Engineering,

Wins an Outstanding Presentation Paper Award at the Korea Software Congress (KSC) 2023

▲ (Top row, from left) Professor Nang Jong-ho, Department of Computer Science & Engineering; Lee Jung, master's student on the DHE track of the Graduate Department of Artificial Intelligence

(Bottom row, from left) Choi Young, master's student, Department of Computer Science & Engineering; Song Jin-ha, doctoral student

The research team of Professor Nang Jong-ho of the Department of Computer Science & Engineering (master's student Lee Jung on the DHE track of the Graduate Department of Artificial Intelligence, master's student Choi Young and doctoral student Song Jin-ha of the Department of Computer Science & Engineering) won an Outstanding Presentation Paper Award at the Korea Software Congress (KSC) 2023.

Deep learning-based image generation models, led by Stable Diffusion, have recently drawn great attention. User interaction is a key factor determining the quality of the results produced by such generative models and the usability of the models themselves. Existing research, however, has focused solely on generating images in various ways, giving rise to several problems. First, many users find it difficult to compose a prompt with the right vocabulary to obtain the result they want. Second, most models tend to provide only a single result for a given input. Third, models are limited in how effectively they reflect user feedback.

To address these problems, the team proposes a new method that expands the user's input prompt into a variety of semantically related prompts to generate diverse images, and then reflects the user's feedback. By introducing a thesaurus-based test time augmentation (TTA) technique, the method analyses the structure of the prompt and varies it using synonyms for words of designated parts of speech, thereby generating outputs from a range of perspectives and giving users a wider choice. In addition, by asking the user to select a preferred image at each stage in which the input prompt is varied and then using the synonyms that were changed in the prompts of the selected images, user feedback could be reflected effectively.

▲ The model architecture of the paper

Experiments using the Stable Diffusion model confirmed that the team's approach is of great help in narrowing the gap between user intent and model output. Qualitative analysis showed how the reflection of user feedback and the augmented prompts affect image generation, and quantitative evaluation using BERT Score between the input prompt and the augmented prompts confirmed that diverse outputs can be generated while preserving the meaning of the input prompt.

▲ Example scenario of the method proposed in the paper

▲ Qualitative and quantitative evaluation of the method proposed in the paper

The team said: "While researching image generation models, we noticed that accessibility for ordinary users is poor from a real service perspective, which led us to write this paper. We expect our method — which can generate diverse outputs without departing from user intent — to improve the usability of generative models." They added: "Although our experiments used the Stable Diffusion model, our method can be applied to any text-to-image generative model. In future we intend to apply it to a variety of text-to-image generative models while also strengthening user feedback through the introduction of negative prompts."

The KSC 2023 programme committee announced the list of Outstanding Presentation Paper Award recipients on Friday 2 February; papers from three teams at the University received awards, including Professor Nang's research team.

▶ Paper title: A Method for Semantic Expansion and Generation of Stable Diffusion User Prompts Using a Thesaurus and TTA

Source: Sogang People https://www.sogang.ac.kr/research/res_01.html