ABSTRACT
The aim of this thesis is to explore different methods for helping computers interpret the real world visually, investigate solutions to those methods offered by the open-sourced computer vision library, OpenCV, and implement some of these in a Raspberry Pi based application for detecting and keeping track of objects. The main focus rests on the practical side of the project.
The result of this thesis is a GNU/Linux based C/C++ application that is able to detect and keep track of objects by reading the pixel values of frames captured by the Raspberry Pi camera module. The application also transmits some useful information, such as coordinates and size, to other computers on the network that send an appropriate query. The source code of the program is documented and can be developed further.
RASPBERRY PI
Figure 1. RPi model B and CSI camera module.
Table 1. RPi hardware specifications.
Taking into account the relatively high performance requirements of image processing in general and the equipment currently available to the faculty, as a relatively inexpensive and powerful embedded platform the Raspberry Pi was an obvious choice. The hardware specifications taken into consideration for this work can be seen in Table 1.
Figure 2. sudo raspi-config.
The easiest way to overclock the Raspberry Pi model B is to do it via Raspberry’s configuration interface, which appears on every start-up, or can be opened using the command sudo raspi-config, which brings up a menu for various configuration possibilities as seen in Figure 2.
OPEN CV INTRODUCTION
Figure 4. Partial overview of what OpenCV includes,
The application written for this thesis relies heavily on computer vision, image processing and pixel manipulation, for which there exists an open source library named OpenCV (Open Source Computer Vision Library), consisting of more than 2500 optimized algorithms. Uses range from facial recognition, object identifying, classifications of human actions in videos, achieved with filters, edge mapping, image transformations, detailed feature analysis and more (Figure 4).
Figure 5. HSV colour cylinder.
The reason for using the HSV pixel format here instead of RGB, is because the HSV colour space separates color information (saturation) from light intensity (value). When ignoring value in the thresholding parameters, it is still possible to detect any colour from an image, given the right hue and saturation values.
Another important characteristic is the circular progression (Figure 5) of hue values for different colours, which means that the highest hue value (179 in OpenCV to fit into an unsigned char) is simultaneously the lowest, because the space is circular not linear, and should be kept in mind when using image thresholding algorithms in an OpenCV-based application.
DEVELOPMENT OF THE APPLICATION
Figure 7. Image after thresholding.
Figure 7. Image after thresholding. The selected hue range in this case was between 0 and 38, which is roughly orange or yellow (colour of door on image). The lower boundary of saturation was also raised to accommodate current lighting conditions.
The instances of this structure are essentially the objects which the program will be handling. From the creation of these copies up to the removal or drawing of markers and transmitting information, the objects go through three phases (illustrated in Figure 10).
Figure 10. Three levels of object existence described using counters as criteria.
Figure 11. Drawing markers on detected objects. Function seen in Listing 8 of Appendix 1.
This is where life counters are utilized (not exclusively), as only objects with this counter’s value over a specified amount (which is higher than the removal counter, by default) get their centroids and area-equivalent red circles drawn on the live camera feed (example shown in Figure 11). This is mainly for aesthetic purposes, but can be used to check if the user has provided correct parameters (e.g. correct HSV range).
USING THE APPLICATION ON RASPBERRY PI
The application can be run on a Linux operating system, in this case a Raspberry Pi Linux distribution (Debian Wheezy). It is preferrably run from command line and can be provided parameters by the user (currently implemented arguments described in Table 4). It is recommended the user try using the application without any parameters first to get a glimpse of how these settings change what the program is doing, especially using the colour range sliders.
Figure 13. Hue values as a range from 0 to 179.
Figure 13. Hue values as a range from 0 to 179. The numbers are rough estimations as to where one colour ends and another starts, while the „pipe“ represents its continuous nature, which means that if the lower boundary is set to be higher than the upper boundary on the trackbar, the range will not end at 179, but rather progresses through 0.
IDEAS FOR FURTHER DEVELOPMENT
The first idea, which is not really a different method but an extenstion of what is already implemented, was to have multiple simultaneous colour ranges used to threshold multiple binary images, which would then be thrown to the contour finding algorithm. This would provide the application the capability to look for various differently coloured objects and would come in handy when trying to detect objects of colours that are not next to each other (e.g. blue and orange).
Implementing this would require the use of similar methods as seen in the current object detection and tracking functions, but could be extremely CPU-intensive, which a Raspberry Pi might not be able to handle, but could provide more flexibility to a colour-based object detection program.
CONCLUSIONS
Identifying objects via filtering of colours (pixel HSV values) is only one of many different methods that can be used for such a system. Colour-based object detection using colours is definitely an effective method, especially when dealing with objects that generally have no constant distinguishable features or corners. Such objects can be balloons and other round coloured objects, or even spots of paint, to provide a few examples.
The main problem with using colours, or more accurately, pixel values, for this purpose is the effects of inconsistent lighting, which a computer can be very sensitive towards, while a human eye can only detect a slight difference. The main cause of this problem is computers’ lack of an inherent learning ability, which prevents adapting to new conditions such as a spike in the amount of yellow and white pixels on an image due to natural (varying amounts of daylight) and/or synthetic lights (flickering).
Considering the aim of this thesis was to create a practical C/C++ application for use in a Raspberry Pi based system that detects objects depending on their colour and attempts to ensure linear and realistic movement tracking, the results are satisfactory enough, while the existing code can and should be optimized further, using a wider array of different functions available in either OpenCV or C++ standard libraries, as well as improved style-wise, towards using more consistent standards and practices of programming in C++.
Source: Tallinn University
Authors: Carl-Martin Ivask