I obtained my PhD in computer science from the University of Illinois at Chicago under the guidance of Bing Liu. Currently, I work at Accenture as an Advanced AI Research Scientist. During PhD, I studied the development of an AI system with the ability to detect unknown instances (out-of-distribution detection) and continually learn new knowledge (continual learning). Prior to joining UIC, I completed my undergraduate studies in economics and statistics (double major) at Washington University in Saint Louis (WashU) and earned my master's degree in mathematics from New York University (NYU).

Papers

• Multi-Modal Continual Pre-training for Audio Encoders. Gyuhak Kim, Ho-Hsiang Wu, Luca Bondi, Bing Liu. ICASSP 2024. link tl;dr

Existing audio pre-training approaches are constrained to a single pretext task, preventing their adaptability to multi-modal interactions beyond the modalities provided in training data. Existing continual learning (CL) approaches are limited to learning downstream tasks such as classification. In this work, we propose to combine CL methods with several audio encoder pre-training methods. The audio encoders, when pre-trained continually over audio-visual and audio-text, exhibit improved performance across various downstream audio tasks due to knowledge accumulation. The audio encoders are also capable of performing cross-modal tasks of all learned modalities.

• Continual Learning: Learnability and Algorithm. Gyuhak Kim^*, Changnan Xiao^*, Tatsuya Konishi, Bing Liu. ICML 2023. link tl;dr

In this paper, we show that class-incremental learning (CIL) is in fact learnable. The learnability arises from the fact that OOD detection is learnable (Fang et al., 2022) and CIL problem can be solved by out-of-distribution detection, as demonstrated in our NeurIPS 2022 publication. Our theory also shows that by training a continual learning algorithm with out-of-distribution (OOD) detection capabilities for each task, the problem can be reduced to multi-task learning. Based on the theoretical analysis, we propose a novel CIL method that outperforms the existing state-of-the-art CL methods.

• Parameter-Level Soft-Masking for Continual Learning. Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, Bing Liu. ICML 2023. link tl;dr

Existing research on task-incremental learning (TIL) primarily focuses on preventing catastrophic forgetting. In this paper, we introduces a novel technique called soft-masking of parameter-level gradient flow (SPG). The proposed method enjoys knowledge transfer while effectively prevents forgetting. We show the effectiveness of the proposed method on both similar and dissimilar tasks.

• Open-World Continual Learning: Unifying Novelty Detection and Continual Learning. Gyuhak Kim, Changnan Xiao, Tatsuya Konishi, Zixuan Ke, Bing Liu. Preprint 2023. link tl;dr

In order to make an AI agent thrive in the open world, it has to detect novelties and learn them incrementally to make the system more knowledgeable over time. The former is called out-of-distribution (OOD) detection and the latter is called continual learning. In the research community, the two are regarded as two completely different problems. This paper theoretically unifies them by proving that good OOD detection is, in fact, necessary for class-incremental learning (CIL). Thus, a good CIL algorithm can naturally be used in the open world to perform both problems.

• Continual Pre-Training of Language Models. Zixuan Ke^*, Yijia Shao^*, Haowei Lin^*, Tatsuya Konishi, Gyuhak Kim, Bing Liu. ICLR 2023. link tl;dr

This paper proposes a novel method to continually domain-adaptive pre-train (DAP) a language model (LM) with a sequence of unlabeled domain corpora to adapt the LM to these domains to improve their endtask performances. Several techniques are proposed. Extensive experiments show that the method not only overcomes catastrophic forgetting, but also achieves knowledge transfer to improve end-task performances.

• A Theoretical Study on Solving Continual Learning. Gyuhak Kim^*, Changnan Xiao^*, Tatsuya Konishi, Zixuan Ke, Bing Liu. NeurIPS 2022. link tl;dr

We provide a theoretical guidance on how to solve class-incremental learning (CIL). It connects CIL, task-incremental learning (TIL), and out-of-distribution (OOD) detection. The theoretical analysis is general as it is agnostic to specific implementations and is applicable to any CIL problem settings (e.g., blurry, online, offline, etc.). Based on the analysis, we design several CIL methods and show superior performances to the state-of-the-art baselines.

• A Multi-Head Model for Continual Learning via Out-of-Distribution Replay. Gyuhak Kim, Zixuan Ke, Bing Liu. CoLLAs 2022. link tl;dr

We propose a novel replay-based class-incremental learning (CIL) method. Instead of using the replay samples to prevent forgetting as the existing methods, we use them as out-of-distribution (OOD) samples to train the network for OOD detection. The proposed method is evaluated using a recently proposed transformer architecture. It outperforms the baselines and is effective with few replay samples. Its algorithm design is inspired by the theory in our NeurIPS 2022 publication.

• Continual Learning Based on OOD Detection and Task Masking. Gyuhak Kim, Sepideh Esmaeilpour, Changnan Xiao, Bing Liu. CVPR Workshops 2022. link tl;dr

We propose a novel class-incremental learning (CIL) method based on out-of-distribution (OOD) detection and task-incremental learning (TIL). CIL and TIL methods have their own advantages, and they are developed independently. A continual learning method intended for CIL is weaker in TIL, and vice versa. The proposed method is superior in both TIL and CIL. We achieve it by leveraging a TIL method with contrastive learning and building OOD detectors.

• Partially Relaxed Masks for Knowledge Transfer Without Forgetting in Continual Learning. Tatsuya Konishi, Mori Kurokawa, Chihiro Ono, Zixuan Ke, Gyuhak Kim, Bing Liu. PAKDD 2022. link tl;dr

We propose a novel task-incremental learning (TIL) method with positive forward transfer. Recent TIL methods suffer little or no forgetting, but are weak in knowledge transfer. We overcome this limitation by relaxing the protected parameters. This allows forward transfer and thus, results in better performances.

• Continual Learning Using Pseudo-Replay via Latent Space Sampling. Gyuhak Kim, Sepideh Esmaeilpour, Zixuan Ke, Tatsuya Konishi, Bing Liu. Preprint 2021. link tl;dr

Existing continual learning (CL) methods train the model from scratch despite the fact that strong pre-trained models are available. We propose a simple, efficient, yet effective method based on a pre-trained model. The proposed method generates pseudo feature vectors from the latent space to prevent forgetting. The system outperforms the baselines by large margins.

• Continual Learning via Principal Components Projection. Gyuhak Kim and Bing Liu. Preprint 2019. link tl;dr

We propose a novel task-incremental learning (TIL) method based on an orthogonal projection of parameters. We project the parameters of each task onto a task-specific space orthogonal to the previous tasks. By construction, the system does not suffer from forgetting. To our knowledge, this approach was the first in the family of orthogonal weight modification methods in TIL, which now became popular.

(* equal contribution)