The world of photography made a major pivot today. With COVID-19 delaying new products and seriously impacting revenues and profits, most photography and camera manufacturers are operating in a wait and see, hanging by their finger tips approach today. Some big names may not survive this pandemic. So, change is inevitable.

However, today we had some positive news that foretells what is to come. Sony announced two new sensor chips – the IMX500 and the IMX501. Both are just 12.3 megapixels, so they are not aimed at the professional or advanced amateur photography crowd. They are ideal however as superb sensors for closed circuit cameras (CCTV) and other cameras that do not demand the highest pixel count. What makes these sensors special is that they have artificial intelligence built-in onboard the sensor chip.

Over the past few years, devices ranging from smartphones to surveillance cameras have benefited from the integration of AI. Machine learning can be used to not only improve the quality of the pictures we take, but also understand video like a human would; identifying people, and objects in frame. The applications of this technology are huge (and sometimes worrying), enabling everything from self-driving cars to automated surveillance.

But many applications rely on sending images and videos to the cloud to be analyzed. This can be a slow and insecure journey, exposing data to hackers. In other scenarios, manufacturers have to install specialized processing cores on devices to handle the extra computational demand, as with new high-end phones from Apple, Google, and Huawei.

But Sony says its new image sensor offers a more streamlined solution than either of these approaches.

“There are some other ways to implement these solutions,” said Sony vice president of business and innovation Mark Hanson, referencing edge computing, which use dedicated AI chips not attached directly to the image sensor. “But I do not believe they will be anywhere close to as cost effective as us shipping image sensors in the billions.”

Sony’s huge presence in the image processing market will certainly push this technology to clients at a huge scale. Hanson notes that the company has more than 60 percent market share, and shipped about 1.6 billion sensors last year, including for all three cameras in Apple’s iPhone 11 Pro.

This first-generation AI image sensor, though, is unlikely to end up in consumer devices like smartphones and tablets, at least to begin with. Instead, Sony will be targeting retailers and industrial clients, with Hanson referencing Amazon’s cashierless Go stores as a potential application.

With AI built-in, some basic machine learning functions normally associated with sophisticated and highly expensive cloud-based AI platforms can be performed instantly right on the sensor. This opens up many interesting and powerful possibilities for CCTV cameras.

If a CCTV camera is monitoring your driveway, this AI can know the difference between a person versus a cat or dog walking around. So, it will eliminate many alerts that are determined not to be a meaningful threat.

To be clear, these image sensors will not replace the work of sophisticated cloud-based machine learning tools, which can go far beyond just recognizing objects. But they are a step towards a future where cameras can function more like self-contained computers.

And they can also assist more complex computer vision systems. For example, in a cashier-less store, something that may be needed following COVID-19, where tons of video data is being captured and processed by a slew of cameras, Sony’s sensors could detect exactly where customers are located. If nobody is by the milk aisle, there is no reason to flip those 3D cameras on, right?

Onboard AI also means you can get information processed in real-time. A camera at the front of a store could count the number of people entering, and even alert security if someone is not wearing a face mask (a major health risk as we are still dealing with the COVID-19 pandemic). Another camera could keep track of item stock on store shelves, while others could monitor the flow of foot traffic to determine shopping “hot spots.”

Other systems can collaborate with the lack of motion detection reported from the cameras, such as dimming lights to save electricity. Or, the inverse is true too, if shoppers are detected, then local announcements can be played to promote specials, or digital signage can be altered to direct physical distancing if the cameras detect shoppers too closely clustered.

Another big application is industrial automation, where image sensors are needed to help so-called co-bots — robots designed to work in close proximity to humans — from bashing their flesh-and-blood colleagues. Here the main advantage of an integrated AI image sensor is speed. If a co-bot detects a human where they shouldn’t be and needs to come to a quick stop, then processing that information as quickly as possible is paramount.

The big bottleneck, though, is the ability of the IMX500 to handle more complex analytical tasks. Right now, says Hanson, the image sensor can only work with pretty “basic” algorithms. That means that more sophisticated and varied tasks, like driving an autonomous car, will certainly require dedicated AI hardware for the foreseeable future. Instead, think of the IMX500 as a simple, single-application device.

The opportunities are rich and exactly which algorithms work best will be known in time. The sensors start shipping in June 2020 and products can be expected in late 2020 and in early 2021. Exactly how manufacturers leverage these new Sony sensors will be exciting to watch.

But this is only the first generation, and the technology will undoubtedly improve in future. Right now, cameras are smarter because they send their data to computers. In the future, the camera itself will be the computer, and all the smarter for it.

Test samples of the IMX500 have already started shipping to early customers with prices starting at ¥10,000 ($93).

————————–MJM ————————–

References:

Hardawar, D. (2020). Sony’s new image sensors will make cameras smarter with onboard AI. Engadget, Verizon Media. Retrieved on May 14, 2020 from, https://www.engadget.com/sony-image-sensor-ai-smart-cameras-070036055.html

Unknown. (2020). Sony’s first AI image sensor will make cameras everywhere smarter. TechZimo. Retrieved on May 14, 2020 from, https://techzimo.com/sonys-first-ai-image-sensor-will-make-cameras-everywhere-smarter/

————————–MJM ————————–

About the Author:

Michael Martin has more than 35 years of experience in systems design for applications that use broadband networks, optical fibre, wireless, and digital communications technologies.

He is a business and technology consultant. He offers his services on a contracting basis. Over the past 15 years with IBM, he has worked in the GBS Global Center of Competency for Energy and Utilities and the GTS Global Center of Excellence for Energy and Utilities. He is a founding partner and President of MICAN Communications and before that was President of Comlink Systems Limited and Ensat Broadcast Services, Inc., both divisions of Cygnal Technologies Corporation (CYN: TSX).

Martin currently serves on the Board of Directors for TeraGo Inc (TGO: TSX) and previously served on the Board of Directors for Avante Logixx Inc. (XX: TSX.V). 

He has served as a Member, SCC ISO-IEC JTC 1/SC-41 – Internet of Things and related technologies, ISO – International Organization for Standardization, and as a member of the NIST SP 500-325 Fog Computing Conceptual Model, National Institute of Standards and Technology.

He served on the Board of Governors of the University of Ontario Institute of Technology (UOIT) [now OntarioTech University] and on the Board of Advisers of five different Colleges in Ontario.  For 16 years he served on the Board of the Society of Motion Picture and Television Engineers (SMPTE), Toronto Section. 

He holds three master’s degrees, in business (MBA), communication (MA), and education (MEd). As well, he has three undergraduate diplomas and five certifications in business, computer programming, internetworking, project management, media, photography, and communication technology. He has earned 15 badges in next generation MOOC continuous education in IoT, Cloud, AI and Cognitive systems, Blockchain, Agile, Big Data, Design Thinking, Security, and more.