Chan Hee (Luke) Song송찬희

I am a computer science doctoral student at The Ohio State University advised by Yu Su. My research focus is on multi-modality, specifically on embodied AI and multimodal foundation models. I received BS Cum Laude in Computer Science from University of Notre Dame in May 2020. In Notre Dame, I was a part of Notre Dame Natural Language Processing Group.

Email  /  Twitter  /  LinkedIn  /  Google Scholar

News

  • 02/2024: Two papers on biology vision foundation model (oral) and web-navigation (poster) have been accepted to CVPR 2024!
  • 02/2024: I will be interning at Nvidia Learning and Perception Research Group this summer. Catch me in Seattle!
  • 07/2023: Our paper on using large language models for vison-and-language navigation accepted to ICCV 2023. Check the paper here.
  • 03/2023: Our SalsaBot work has been accepted to the Embodied AI Workshop at CVPR 2023! Check the (short version) paper here.
  • 03/2023: I will be interning at Adobe Research this summer. Catch me in San Jose!
  • 03/2022: Our paper on long-horizon vison-and-language navigation accepted to CVPR 2022. Check the paper here.
  • 12/2021: I will lead the OSU team to participate in the Alexa Simbot challenge! Check us out here. Go Bucks!

Research

Currently my interests are embodied AI and multimodal foundation models. I am also interested in AI for science and multimodal knowledge base.

BioCLIP: A Vision Foundation Model for the Tree of Life
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo, Chan Hee Song, David Edward Carlyn, Li Dong, Wasila M Dahdul, Charles Stewart, Tanya Berger-Wolf, Wei-Lun Chao, Yu Su
[preprint] [website] CVPR 2024 (Oral)

Dual-View Visual Contextualization for Web Navigation
Jihyung Kil, Chan Hee Song, Boyuan Zheng, Xiang Deng, Yu Su, Wei-Lun Chao
[preprint] CVPR 2024

LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models
Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, Yu Su
[paper] ICCV 2023
[short version] Embodied AI Workshop at CVPR 2023
[website]

SalsaBot: Towards a Robust and Generalizable Embodied Agent
Chan Hee Song, Jiaman Wu, Ju-Seung Byeon, Zexin Xu, Vardaan Pahuja, Goonmeet Bajaj, Samuel Stevens, Ziru Chen, Yu Su
[short version] Embodied AI Workshop at CVPR 2023
[long version] Alexa Prize SimBot Challenge Proceedings 2023

One Step at a Time: Long-Horizon Vision-and-Language Navigation with Milestones
Chan Hee Song, Jihyung Kil, Tai-Yu Pan, Brian M. Sadler, Wei-Lun Chao, Yu Su
[paper] CVPR 2022

Work Experiences

Adobe Research
Research Scientist Intern
05/2023 -> 08/2023

- I worked on an video editing assistant, with emphasize on open domain code generation problem with large language models. Worked with Gang Wu, Haoliang Wang, and Uttaran Bhattacharya.

Human Language Technology Center For Excellence, Johns Hopkins University
Visiting NLP/ML Researcher, SCALE program
05/2019 -> 08/2019

- I implemented various state-of-the-art neural architectures and adapted to named entity recognition task. Introduced a lexical feature generated from gazetteers to the named entity recognition (NER) system. As an independent project, developed a novel named entity recognition system achieving state-of-the-art F1 score on Chinese datasets using glyph features generated from convolutional neural network.

Kyndi, San Mateo CA
Software Engineering Intern
01/2019 -> 05/2019

- I migrated in-memory graph to external database by building a GraphQL server to interact with the external database using Apollo and Typescript. Also improved accuracy and representation of the knowledge graph by writing unique graph operations on a large knowledge graph.

Undergraduate Research

prl

Using Chinese Glyphs for Named Entity Recognition
Chan Hee Song, Arijit Sehanobish
09/2019
extended abstract, full paper, AAAI-20 student abstract

Used Chinese glyph information to augment neural named entity system. Introduced a convolutional neural network (CNN) architecture that achieves a state-of-art performance on Chinese Weibo NER dataset and sets a new baseline F1 score on Chinese OntoNotes v5.0 dataset. Easy to implement, train and uses less data. Robust to non-Chinese characters in the dataset.

prl

Gazetteer Generation for Neural Named Entity Recognition
Chan Hee Song, Dawn Lawrie, Tim Finin, Jim Mayfield
09/2019
paper, FLAIRS-33

Introduced a simple method to generate gazetteers from the Web. Generated gazetteer is used as a feature to augment neural named entity recognition system, showing improvement over baseline. Developed a method to create new training data using gazetteer entity replacement. Presented new Russian NER corpus gathered from Reddit.

prl

Sentence Representation for Neural Machine Translation Systems
Chan Hee Song, David Chiang
06/2018 -> 05/2019

Independent undergraduate researcher under supervision of professor David Chiang. I investigated the impact of sentence representation on neural machine translation to develop an optimal encoding mechanism that has a better trade-off between time and accuracy. I am using C, Python and PyTorch library to build encoder/decoder, modify neural machine translation system, and implement new equations.

prl

Verba Volant, Scripta Manent: Automatic Transcription of Medieval Latin Manuscripts
Undergraduate Research Assistant
01/2018 -> 05/2018
poster

Research assistant to the project which aims to find a novel method to process Vatican Secret Archives to transfer early Medieval Latin handwritten papers into a computer readable format. I analyzed various character recognition algorithm and used Python to build an auto analyzer tool for Optical Character Recognition systems

Teaching
ND

Undergraduate Teaching Assistant, Theory of Computing Spring 2020
Undergraduate Teaching Assistant, Fundamentals of Computing Fall 2018



(borrowed from here)