Smart Cameras Get Smarter

How Actionable Data Drives Today’s Imaging Technology

Cameras are the eyes of the modern world. The devices equip today’s cars and machines, and real-time imaging capabilities support law enforcement, businesses, and homeowners. Thus far, cameras have been just that: simple cameras, eyes without a brain. Their sole purpose has been, primarily, to output images. Nowadays, however, cameras are “smarter.” In this article, we will review how imaging technology’s ability to intelligently process data will support new insights and applications.

A camera detects a cardboard box. For security and surveillance purposes, object detection capabilities can be performed in specific zones, such as airports, trains, and bus stations. (Image Credit: IntelliVision Technologies)

‘Intelligence’

What makes a camera intelligent is its ability to understand and interpret images or videos, and to take action with that information. A smart camera can send event alerts, track objects, and intelligently search through volumes of visual data. In order to achieve such capabilities, the CPU or GPU inside the camera must constantly receive video and process it simultaneously — to understand it. When a camera has the ability to take action, to send alerts, or convey information or metadata in a meaningful way to the user, this is called actionable intelligence.

In an effort to process real-time video and information at the very edge of a connected IP network, the camera has become the focal point for new innovation and development. Just ten years ago, the camera was simply a device for recording video or producing images. Today, the camera has become highly intelligent, mainly through the advent of advances in semiconductors, software, and math algorithms.

The IP cameras are not only accessible over the Internet, but now they also represent a visual edge of a communications network. Having the camera “sit at the edge” goes beyond simple connections between devices; today’s devices are connected in an intelligent network. With the addition of powerful and cost-effective computers embedded into smart cameras, computer technology has been pushed to the edge, or into the camera itself.

Today’s cameras can recognize individual faces, as shown in the above image. (Image Credit: IntelliVision Technologies)

Emerging Application Areas

While smart cameras can be used in a way that benefits a variety of applications or industries, here are three emerging areas:

  • Smart Home
  • Smart Retail
  • Smart Cars

For a home device to detect intruders, for a retailer to spot customers, and for a car to find pedestrians, the camera must first understand the background of a scene and then continuously look for changes, or objects in motion. When a valid change is detected, the camera's software and algorithm must perform classification. For example, is the shape a person? A vehicle? An animal?

Finally, “recognition” intelligence allows cameras to “learn” faces or objects, often through a user supplying his or her own data to “train” the system. Based on a database of “learned” faces, the latest artificial intelligence, or deep learning, software can then recognize a detected individual as a son, daughter, or spouse.

Heat maps display customer traffic and areas with the greatest amount of activity. Such "hot zones" are shown above in red and yellow. (Image Credit: IntelliVision Technologies)

Smart Home Cameras

Home cameras are becoming increasingly more common. Featuring built-in intelligence, the small, Wi-Fi-enabled devices can be used for indoor, outdoor, doorbell, or roadside use. The “brain” inside the camera can look for people, intelligently detect motion, spot a human entering a given zone, track a passing car, find an individual’s face, or even recognize the license plate of a car and check it against a watch list.

All of this smart processing can be done inside the camera. Soon, smart cameras will be able to send parents a notification when their child arrives home, when a stranger or a parcel delivery appears at a doorstep, or when someone has been loitering around the property. This processing is all done instantly, within the camera’s internal CPU, by video content analysis; the output, or alert, is then passed directly to the user.

The advancements in video content analytics have now made it possible to monitor many situations in real time, and to take action accordingly. In this market, the camera has become one of the primary sensors in the home. With such a connected environment, the IoT, or Internet of Things, cloud is now an ideal place to extend camera intelligence.

Smart Retail Stores

Additionally, many “smart” retail outlets are taking full advantage of smart cameras and cloud-based services. The technology not only provides security, surveillance, and peace of mind to store owners, but the data generated by the smart cameras also creates important insights for businesses.

The intelligent cameras perform many retail data extractions, such as customer counts, queue lengths, report service times at the teller, and customer interest in specific sales and/or products.

A dashboard displays a retail store's consumer data, including percentage of male/female customers, age group, and time spent shopping. (Image Credit: IntelliVision Technologies)

Business intelligence can be gathered to monitor point-of-sale (PoS) activity, queue length/time and duration, customer demographics, or traffic maps, showing in graphical format where high-volume traffic is concentrated in a store. Such business intelligence can ultimately be used to directly drive productivity and profits in stores.

Smart Cars

Your new car probably already has multiple cameras. The devices are part of a vehicle’s Automated Driver Assistance Systems (ADAS), which send critical information to operators behind the wheel: lane change warnings, front collision alerts, pedestrian detection, and reverse/back-up collision signals, for example. Many vehicles use smart cameras to provide alerts and actionable intelligence to the driver, thus saving lives in many areas.

Many cars today have more than one camera; some luxury models have up to nine imaging devices. The new “eyes” of the vehicle are not only recording video; they are also using the data to physically initiate a car’s actions — an essential capability as fully autonomous vehicles appear on the road.

The camera is the most widely used sensor for automated driver assistance. The safety and information systems use the data extracted from a smart camera’s video to guide a vehicle through potentially dangerous situations, often taking control from the driver to prevent accidents.

All of the imaging advances are pushing the boundaries of science, demanding the latest computer vision technology and deep learning techniques. The car, for example, is essentially beginning to react like the human brain.

The above image shows how a camera supports the detection of humans and vehicles. The different color bounding box highlights the separate “objects.” (Image Credit: IntelliVision Technologies)

Deep learning has been the subject of scientific and engineering research for many years now, and will be critical for the automotive industry as increasingly autonomous vehicles are “trained” to learn new environments and obstacles. The machine learning technology is driven ultimately by advances in smart cameras.

A Snapshot of the Future

The traditional security and surveillance market has transitioned from a slow-moving industry to a very fast-paced one. Where cameras were previously used to simply monitor a building or traffic junction, new technologies are being deployed in metropolitan areas to create “smart” cities, with large numbers of cameras monitoring vast areas and crowds of people.

The advances in the cameras’ computational power allow the devices themselves to take actions and raise alarms without human intervention. The wide-scale use of video content analytics — or the ability to process, recognize, detect, sense motion, and analyze video in real time — has opened up many new possibilities. A camera can be used to automatically detect people, recognize faces and car license plates, spot suspicious objects, create zones of interest, protect virtual areas, look for objects left or removed, and also provide intelligent alerts or information back to users — as opposed to simply being a video recording device.

We are only now in the early stages of these developments. The smart camera, via a combination of computing power and advanced computer vision technology, is only getting smarter and smarter. Similarly to how we saw the cell phone evolve into the smart phone — where the device’s overall capabilities became more important than its ability to make simple phone calls — cameras are quickly evolving in the same way and taking on more advanced functions.

Welcome to the burgeoning new world of the “smart” and aware camera, always processing the visual world and delivering actionable data directly to the user. No longer just a device that takes video and images, a camera’s value will soon be determined by its ability to extract data and take action. Moving beyond the confines of security and surveillance, the camera is now becoming a primary development platform for some of the most advanced research, from smart phones and smart cities to artificial-intelligence efforts that allow machines like cars, robots, and drones to “think.”

This article was written by David Jones, Vice President of Marketing, and Vaidhi Nathan, President and CEO, at IntelliVision Technologies (San Jose, CA). For more information, Click Here .