Abstract
The internet is a vast and rapidly changing landscape. For every new algorithm
invented to increase the security and reliability of our society, an adversarial group
coalesces in reaction. Some may argue that this constant arms race between defenders
and attackers is currently being won by the latter. To shift the tides, we need scalable
approaches to counter their attacks, whatever they may be, at the scale and speed
with which the internet moves. This dissertation focuses on developing computer
vision algorithms to help ensure security within three main areas:
1) In the video surveillance chapter (Ch. 2), we focus on the problem of ReIdentification, or finding all appearances of a given subject across a given camera
system. Most all existing ReID approaches employ local and global body features
(e.g., clothing color and pattern, body symmetry, etc.). These ‘body ReID’ methods
implicitly assume that facial resolution is too low to aid in the ReID process. We
assert that faces, even when captured in low resolution environments, may contain
unique and stable features for ReID. Such ‘facial ReID’ approaches are relatively
unexplored in the literature. We explore facial ReID using a new dataset that was
collected from a real surveillance network in a municipal rapid transit system
2) In the facial recognition chapter (Ch. 3), we focus on the problem of off pose faces. If a Convolutional Neural Network (CNN) is intended to tolerate facial pose, then we face an important question: should this training data be diverse in
its pose distribution, or should face images be normalized to a single pose in a preprocessing step? To address this question, we evaluate a number of facial landmarking
algorithms and a popular frontalization method to understand their effect on facial
recognition performance.
3) In the Image Forensics and retrieval chapters (Ch. 4, 5, and 6), we focus on
building novel image retrieval algorithms that can retrieve and trace modified images
back to their origins. Images from social media can reflect diverse viewpoints, heated
arguments, and expressions of creativity. They are often composites, borrowing content from many different sources to create a single image. Taking into account these
attributes introduces new complexity to image search tasks. We propose multiple
new frameworks for image retrieval that model object-level regions, allowing for fine grained object level retrieval results. These retrieval results can help aid in the task
of verifying the forensic integrity of an image. We then utilize these algorithms to
perform real-world analysis on data scraped from Twitter and Instagram from the
2019 Indonesian Election.
