How-To

Add Image and Video Analysis with the Amazon Rekognition API

Photo analysis holds huge potential, but is only just beginning to mature.

• By Brien Posey
• 02/20/2018

Image recognition isn't really anything new. We've all seen facial recognition features on social media sites. More recently, various mobile computing devices, such as the Microsoft Surface Book 2, have begun using facial recognition for device authentication. There are even digital cameras available that not only recognize the subject being photographed, but can tell if that person is smiling.

Photo analysis is one of those technologies that holds huge potential, but is only just beginning to mature. Photo recognition could conceivably be used for everything from content moderation, to the protection of intellectual property.

Amazon has begun offering both photo and video recognition services through the Amazon Rekognition API. The service is still quite new, but right now Amazon has a number of online demos that illustrate just how powerful Amazon Rekognition really is.

For example, Amazon Rekognition can perform object and scene detection. If you look at Figure 1, for instance, you can see that Rekognition was able to automatically tag this image with words such as human and skateboard.

If you're skeptical that Amazon has optimized its software for this particular image, then try uploading your own image. Admittedly, the service is not yet perfect. While experimenting with image recognition, the service crashed on me a few times. I also tried uploading a picture from my commercial astronaut training, and Amazon Rekognition thought the orange spacesuit was a lifejacket. Uploading a picture in which I was wearing a flight suit returned tags such as military and officer, as shown in Figure 2.

Amazon Rekognition also has an image moderation service. The basic idea behind this feature is that you can upload a picture, and Amazon Rekognition will tell you whether or not the picture contains anything explicit. In Figure 3, for example, Amazon Rekognition was able to determine that the sample picture (provided by Amazon) was of a woman in a swimsuit. I spent a bit of time trying to trick the image moderation software with a variety of image types, many of which were G-rated, and found that the software is pretty accurate.

Amazon Rekognition is also able to perform facial analysis. As I mentioned earlier, facial recognition has been around for quite some time. However, Amazon Rekognition takes it to the next level. As you can see in Figure 4, the software tries to figure out gender, age, mood and more. If you look at the screen capture, you'll notice that the software was even smart enough to ignore the woman's reflection in the mirror.

If you look at the bottom of the previous figure, you'll notice that there's a second image containing multiple people. I tried this one out, too, and found that when multiple people appear in a picture, AWS provides a separate analysis for each.

I was curious how accurate the facial recognition software was, so I tried uploading my own picture (the same one shown in Figure 2). Amazon Rekognition said that the picture is of a male, age 35-52, who is happy and wearing glasses, all of which is correct. You can see the results in Figure 5.

Like I already mentioned, Amazon's software has enormous potential, but it isn't perfect. Here is one especially amusing result. I tried uploading a few of my spacesuit pictures to the Facial Analysis demo. When the spacesuit visor was closed, Amazon Rekognition could not even recognize that a face appeared in the photograph. Next, I tried uploading a picture with the visor open. Here's where things got weird. The software automatically took me from Facial Analysis into Image Moderation, and indicated that the picture contained graphic female nudity. As you can see in Figure 6, I'm not female, nor am I naked in the picture. I am pretty sure that the placement of the mission patch and a Velcro pad confused the software. Incidentally, the software worked correctly the second time that I tried to analyze the picture.

Besides the features that I've talked about, Amazon Rekognition also has the ability to recognize text in images, pictures of celebrities and even perform a facial comparison. As previously noted, the software can perform similar analysis on videos, identifying things such as people, objects and celebrities. Unfortunately, I didn't get to try this out for myself. The software is still a bit buggy and it kept crashing on me. Even so, I do plan to revisit Amazon Rekognition in the future once it has time to mature.

In retrospect, Amazon Rekognition holds enormous potential, but Amazon still has quite a bit of work to do before it will be suitable for production use. In the meantime, the Amazon Rekognition demo makes for a great source of entertainment.