UTCS Colloquia/AI - Serge Belongie/University of California, San Diego, "Visual Recognition with Humans in the Loop", ACES 2.402

Contact Name: 
Jenna Whitney
Feb 11, 2011 11:00am - 12:00pm

There is a sign-up schedule for this event that can be found at



Type of Talk: UTCS Colloquia/AI

Speaker/Affiliation: Serge

Belongie/University of California, San Diego

Date/Time: Friday, Febr

uary 11, 2011, 11:00 a.m.

Location: ACES 2.402

Host: Kristen Gr


Talk Title: Visual Recognition with Humans in the Loop

We present an interactive, hybrid human-computer method for o

bject classification. The method applies to classes of problems that are di

fficult for most people, but are recognizable by people with the appropria

te expertise (e.g., animal species or airplane model recognition). The cla

ssification method can be seen as a visual version of the 20 questions game

, where questions based on simple visual attributes are posed interactivel

y. The goal is to identify the true class while minimizing the number of qu

estions asked, using the visual content of the image. Incorporating user i

nput drives up recognition accuracy to levels that are good enough for prac

tical applications; at the same time, computer vision reduces the amount

of human interaction required. The resulting hybrid system is able to handl

e difficult, large multi-class problems with tightly-related categories. W

e introduce a general framework for incorporating almost any off-the-shelf

multi-class object recognition algorithm into the visual 20 questions game

, and provide methodologies to account for imperfect user responses and unr

eliable computer vision algorithms. We evaluate the accuracy and computatio

nal properties of different computer vision algorithms and the effects of n

oisy user responses on a dataset of 200 bird species and on the Animals Wit

h Attributes dataset. Our results demonstrate the effectiveness and practic

ality of the hybrid human-computer classification paradigm.

This work

is part of the Visipedia project, in collaboration with Steve Branson, Ca

therine Wah, Florian Schroff, Boris Babenko, Peter Welinder and Pietro P


Speaker Bio:
Serge Belongie received the B.S. degree (with ho

nor) in Electrical Engineering from the California Institute of Technology

in 1995 and the M.S. and Ph.D. degrees in Electrical Engineering and Comput

er Sciences (EECS) at U.C. Berkeley in 1997 and 2000, respectively. While

at Berkeley, his research was supported by a National Science Foundation G

raduate Research Fellowship. He is also a co-founder of Digital Persona, I

nc., and the principal architect of the Digital Persona fingerprint recogn

ition algorithm. He is currently an associate professor in the Computer Sci

ence and Engineering Department at U.C. San Diego. His research interests i

nclude computer vision and pattern recognition. He is a recipient of the NS

F CAREER Award and the Alfred P. Sloan Research Fellowship. In 2004 MIT Tec

hnology Review named him to the list of the 100 top young technology innova

tors in the world (TR100).