C. Virtual Identification of Images: Increasing Taxonomic Identification Capacity. (Andre Poremski and Neil S. Cobb)
In North America, only 3-4% of non-lepidopteran insect species can be reliably identified from images [31], whereas an estimated 80% of butterflies, 70% of “macros” and 55% of micros can be identified to species from images (~6,870 total species). Identification of butterflies and moths based on external characters is feasible based largely on wing shape, venation, and scale coloration patterns [56-58]. For Lepidoptera, identifying undetermined specimens is primarily limited by a lack of taxonomic experts. Thus, increased capacity to identify insect species from images will greatly aid taxonomists. if one includes Lepidoptera then approximately 10% of the 142,000 species of arthropods in North America may be identified to species based on image, location and date.
To further enhance image-based identifications, LepNet is collaborating with a Google-funded project (Visipedia) [59]on a computer vision project called LepSnap(Figure 4). LepSnap will facilitate taxonomic identifications of images and tools for improved image searching. Visipedia and its back-end (Vibe) have many operational similarities with Leafsnap [60]; they are open source platforms allowing communities to build their own X-snap kind of apps [61-63]. LepSnap is an insect extension of the Visipedia collaboration with the Cornell Laboratory of Ornithology [64]. In a pilot project that included 62 species of arctiine moths from the Pacific Northwest, over 75% of the tested images were correctly identified to the respective species with 80-100% probability based solely on appearance. The remaining 23% exhibited correct identification probabilities higher than 60%. Accuracy will undoubtedly improve with more training images per species, as well as additional ecological filtering (e.g., phenology, distribution). The capacity to identify specimens quickly and efficiently by simply taking a photo and submitting it to LepSnap for identification will greatly aid the LepNet contributor and user communities in improving data quality, by focusing on taxa that are problematic or cannot be identified by wings.
LepNet will collaborate with the Visipedia team to: 1) provide images to assemble a taxonomically comprehensive LepSnap reference image library; 2) solicit support from taxonomic experts to help train computer vision algorithms; 3) provide data-driven ecological filters, including geographic boundaries and flight time dates to narrow choices (e.g., DiscoverLife ID guides) [65]; and 4) build a web interface and phone apps (for both Android and iPhone) for submitting images of collection specimen to Visipedia and have the LepSnap-provided identification results directly integrated with the TCN portal. To interact with this novel, automated identification service, LepNet participants will have the ability to upload images to LepNet through either an app (handheld resolution images) or the web (high-resolution images) identification interface. All images will be available for processing with Visipedia to yield identifications while also tracking the associated specimen information. Identification results are confirmed by LepNet participants, supporting the development of LepSnap identification for lesser known taxa.
We will create Darwin Core (DwC)-compliant Android and iPhone apps that allow registered LepNet users to apply their handheld devices to image cataloged specimens, submit the images through LepSnap, and receive novel identifications or confirmations of existing identifications. With the high quality cameras available on most smart phones, the images that are identified to genus or species will be valuable contributions to the LepNet image library. Thus, phone apps produced from LepSnap will provide a means to rapidly image lepidopteran specimens as well as a resource to quickly identify specimens. Identified and unidentified images will be processed through LepSnap and then uploaded directly to LepNet. We will engage the 67 LepNet taxonomists to review unidentified images and help train computer algorithms for problematic taxa. Additionally all identified images added to LepNet will be made accessible to Visipedia to further develop its algorithms in identification. The LepSnap apps will only accept images that have a color chart, scale bar and meet similar requirements that we have set for the imaging protocol (Table 1, protocols 2, 6, 7, 8, 9, 10 and 15).
Each LepNet collection will establish a smartphone workflow that is best suited for its respective student and volunteer workforce, and then train/supervise members to image the bulk of the projected > 160,000 specimens. Over 4,100 volunteers are distributed among the 29 collections (3,300 adult volunteers, 478 student researchers/interns, and 428 K-12 volunteers), and thus LepSnap provides a strong, inquiry-based activity for the contributing collection members (See Section 9). The priority for imaging will be butterflies (serving ButterflyNet), distinctive macromoths and larger micromoths, and especially species that exhibit geographic variation. The LepSnap project will build on the existing efforts of the Moth Photographers Group (MPG) website, which has images for 9,862 lepidopteran species.
Figure 4. LepSnap workflow. Tan cubes and black arrows indicate processes/content embedded in LepNet (iDigBio server) and the Blue cube and arrow indicate activity occurring through Visipedia (Caltech server). LepSnap results will be stored in LepNet and be provided to the end user (e.g., smartphone). The Visipedia processing involves comparison of unidentified images to the LepNet library of images and filtering based on specimen sex, location, date, and other ecological data (LepNet and other databases).
31. Cobb, N., K.C. Seltmann, and N.M. Franz, The current state of arthropod biodiversity data: Addressing impacts of global change. 2014.
56. Grinter, C., Personal communication. 2012.
57. Opler, P.A., Personal communication. 2014.
58. Wikle, D., Personal communication. 2014.
59. Pietro Perona’s Vision Group at Caltech and Serge Belongie’s Vision Group at UCSD. Visipedia. 2014; Available from: http://www.vision.caltech.edu/visipedia/.
60. Kumar, N., et al., Leafsnap: A computer vision system for automatic plant species identification, inComputer Vision–ECCV 2012. 2012, Springer. p. 502-516.
61. Branson, S., et al., The ignorant led by the blind: A hybrid human-machine vision system for fine-grained categorization. International Journal of Computer Vision, 2014: p. 1-27.
62. Matera, T., et al. A User Friendly Crowdsourcing Task Manager.
63. Wah, C., et al. Similarity Comparisons for Interactive Fine-Grained Categorization.
64. Ornithology, C.L.o. CCUB NABirds 700. 2014; Available from:http://info.allaboutbirds.org/nabirds.
65. DiscoverLife, DiscoverLife Identification Guides. 2014.