David Jacobs' app redefines bird watching for the 21st Century

COLLEGE PARK, Md. - Was that a catbird or a mockingbird you just saw? The answer is now just a smartphone tap away. Birdsnap—a new app developed with the help of a University of Maryland computer scientist—can identify birds from photos, using methods borrowed from facial recognition software. Simply snap a photo of a bird, and the app can tell you what species it is and all about it.

Birdsnap is the latest in a series of electronic field guides that David Jacobs, professor ofcomputer science at UMD and a part of UMD's Brain and Behavior Initiative, has helped to develop with his colleague Peter Belhumeur, professor of computer science at Columbia University. Their developments are based on the idea that the same techniques used by computers to recognize faces could also be used to identify plants and animals.

In 2011, Jacobs and Belhumeur launched Leafsnap, a free iPhone app that identifies trees from pictures of their leaves, in partnership with W. John Kress, research scientist and curator in the Department of Botany at the Smithsonian's National Museum of Natural History.

Encouraged by their success with plants, Jacobs and Belhumeur decided to try identifying animals. They originally chose to look at dogs and released the corresponding app, Dogsnap, in 2012. However, there's not a huge need to identify dogs from photos, according to Jacobs, so he and Belhumeur sought a more in-demand use for their techniques. Birdsnap, which debuted in Apple's App Store in May, extends the methods of the earlier apps to identifying birds.

To identify a bird from a photo, the app first identifies the different parts of the bird, such as the beak, based on thousands of examples of what each body part looks like. It also considers the beak's position relative to the other parts of the bird, such as the tail and wings. Next, each feature is compared against a database of the 500 most common bird species in North America. The algorithm calculates the likelihood of each species matching all of the features and returns the most likely candidates.

One way in which Birdsnap improves on Leafsnap and Dogsnap is by incorporating available data on bird sightings from the Cornell Lab of Ornithology. Users specify the date and place where a photo was taken, and Birdsnap looks up which birds are most common in that location at that time of the year. This helps the algorithm to narrow the list of possible matches.

However, Birdsnap goes beyond simple visual recognition. The app, along with its associated website (www.birdsnap.com), contains a wealth of information about the birds in its database. Birds can be sorted according to how common they are in a particular place and time or according to their evolutionary relationships. The app also shows which birds are arriving, departing or migrating through, based on the location and time of year. Clicking on a bird will show you photos, a description (sourced from Wikipedia and independently verified), and maps showing the bird's range and likelihood of sighting in different places throughout the year, as well as play audio files of the bird's song. It also shows side-by-side comparisons of similar species that highlight the most distinguishing features.

Although Birdsnap is geared toward recreational bird watchers, Jacobs notes that the app, and others like it, could be useful for serious research. "There's a lot of interest in studying how the distribution of species is varying over time," Jacob says. "Are we seeing birds in parts of the country where we didn't typically see them?" Such information could be useful in studying the effects of climate change, for instance.

Jacobs' next steps include looking at other animals and other parts of the world (a version of Leafsnap for the United Kingdom was released May 16, 2014). Jacobs hopes to create a system that automatically generates new field guides, so that "if someone wanted to create a field guide for beetles, that person could take images of different species of beetles and we could automatically generate a field guide that included those species."

Published June 1, 2014