© 2013 IBM Corporation

Efficient Multi-stage Image Classification for Mobile Sensing in Urban Environments

Presented by

Shashank Mujumdar

IBM Research, India

© 2013 IBM Corporation

Point and Shoot

Smartphones enable easy image capture. Growing number of smartphone users opens up possibility for real-world applications in

image classification.

© 2013 IBM Corporation

Overview of GHMC

GHMC has a vision to make Hyderabad a citizen friendly, well-governed and environmental friendly city by providing high quality services.

Ensure the city is clean by monitoring the trash collection on a daily basis.

Third party supervisors use smartphones to capture images of the dumpsters through mobile application and submit them to an online server where they are manually analyzed.

Need:• Provides a transparent interface to the citizens.• Allows for validation of submitted feedback and to take corrective actions if

required.

© 2013 IBM Corporation

The Task

We automate the process the of identifying the state of the dumpster bins. Perform binary image classification over the dumpster images to classify

into one of the following categories.– Clean (trash is not visible from the bin opening)– Unclean (trash is visible from the bin opening)

Unique Problem:

• Classification between the two states of the same object.

• In literature, focus is around retrieval and recognition tasks for mobile imagery.

• Challenging imaging conditions, background clutter in images and complex urban environment.

© 2013 IBM Corporation

The Proposed Framework

Region of Interest (ROI)Image Data

Stage 1: Detection

Feature Computation Train and Test with SVM Classifier

Stage 2: Classification

We proposed a simple multi-stage pipeline to perform image classification.

Data Collection:• Utilizing a web-crawler we downloaded the images

from the publically accessible web portal.• We excluded images that are ambiguous

- contain multiple dumpsters- dumpster lid area is not visible

• A total of 1710 images were collected.• Manual labels for the images served as ground

truth.

Challenges:• Varying illumination conditions

• Image background clutter

• Different scales and viewing angles of the dumpsters.

© 2013 IBM Corporation

Cropped Images Compute SIFT Features Cluster SIFT Features

Generate Visual Vocabulary

Match Visual Words

Frequent Visual Words

Find Visual Words

Extract Region of Interest (ROI)

Step 1:

Training to Generate

Frequent Visual Words

Step 2:

Finding Frequent Visual

Words to Extract ROI

Generate Bounding BoxImage Data

• Typically a sliding window approach is utilized for object localization (computationally expensive).• We use Bag of Words (BoW) approach for detection (typically used for recognition/classification).• Dumpster is present and identifiable in every image. Identify visual words that represent the dumpster.• Match local features (SIFT) with the visual words to obtain the region of interest (ROI).

The Detection Stage

© 2013 IBM Corporation

The Classification Stage

Image Data

Train Data

Test Data

Detect ROI

HOG Feature Computation

k-Fold Cross

Validation

SVM Classifier

Predict Labels

Learning

Training (Step 1)

Testing (Step 2)

• Classifier Training:- identify and extract the ROI.- compute the HOG features over ROI.- reduce dimensionality with Fisher’s

LDA.- train kernel SVM with a RBF kernel.- k-fold cross validation to estimate

optimal classifier parameters.

• Classifier Testing:- classification performed using training

parameters.- extract ROI -> compute HOG ->

perform LDA -> classify with SVM.

• Data of 1710 images divided into training (90%) and testing (10%) set.

© 2013 IBM Corporation

Results and Performance Evaluation

An accuracy of 80.59% was achieved on the unseen test data. ROC curves were generated to assess the performance and AUC was computed. Area under the curve (AUC) was computed to be 0.8710. Comparison with conventional single stage classification pipelines

HOG, LBP and Haralick’s texture features were used in single stage SVM. Proposed multi-stage approach outperforms all of the single stage variants.

© 2013 IBM Corporation

Conclusion

Developed and implemented a multi-stage image classification system.

Efficient and robust for challenging imaging conditions in real-world mobile sensing applications.

Demonstrated the effectiveness for the real-world images of dumpsters captured with mobile phones.

Achieved an accuracy of 80.59% on a challenging (public) dataset.

Shown to outperform conventional single-stage image classification techniques.

The proposed pipeline can be extended to other real-world applications in mobile sensing by experimenting with other features suitable to the task at hand.

© 2013 IBM Corporation

Thank You