Machine learning and AI tools can support broadcast productions in many applications. One common example is the use of AI tools in the archive where text analysis and image recognition are helping to enrich media assets with additional metadata and thus enable quick search and access to material. In the following example, we tried to show use cases how AI tools may create a benefit on the end user side, as well as in professional journalistic investigations.
We built a browser extension that analyses the running video in the active browser window, tries to detect faces and recognizes the person. A browser overlay then shows additional information about that person. We created a sample data set on German association football players and used Wikidata to retrieve and show information like age, height and the position played on the team by the player. One requirement we wanted to fulfil is that all the detection is done completely in the browser, so the user doesn’t leave a trace about what video content he watched.
To recognize the football players, we gathered a relatively small set of training data, about 10 to 20 pictures per person. From these we extracted the embeddings of the person’s faces. An embedding is a vector representation of the detected face with a vector size of 128. These embeddings together with the respective label, in our case the Wikidata-ID, is bundled together with the extension. In the following clip you can see the position of all the created embeddings reduced to a 3D space. The embeddings from the same person (same color) are very similar and therefore are very close to each other. By positioning another embeddings vector in this plane und determining the shortest distance to the next point you can infer the person label.
When you activate the extension, it hooks on the first <video>-Tag on the active browser tab and extracts single frames from the video. For the face detection and recognition task in the browser we used face-api.js. This framework already provides different models for face detection, face landmark extraction and face recognition. An API lets you apply all these models on an input image. These extracted embeddings are then compared to our predefined embeddings and the most likely label inferred. The following block diagram shows the steps from extracting a frame from the <video>tag up to receiving the face embeddings.
We developed this demo use case along the idea to recognize football players and giving the user additional information about the player on the screen. But in principle this could be used for any other case where you want to add information about the persons currently appearing on-screen. Further examples are to find actors and showing other roles or movies they appeared in. Or to add information about other persons of interest like politicians for example. Especially, recognition and adding of further background information about regional and local persons of interest could add a lot of value for end-users. In practical use, a challenge exists in collecting of the embeddings of the persons of interest, i.e. in assembling the training material. For this purpose it would be beneficial to establish a framework for collecting and sharing such data sets between partners, such as a group of broadcasters. A next step in our work is to evaluate how such a framework could look like and what technical issues need to be solved. There are also legal issues that need to be examined, like how can archive data be used for machine learning and who holds the rights of the resulting model. And in the case of face recognition how can personal rights be protected.
Demo Extension
You can download our demo extension here.
Be aware that you need to enable the developer mode on your Google Chrome browser to
load the extension. Here you find some example Videos, but feel free to use
the extension on different content
· Elfmeter
(Min 1:45)
· Marco
Reus
· Pfiffe gegen DFB
· Lina
Magull (Min 3:50)
· Klara
Bühl
Note: The extension is a development sample and is provided for
demo purposes only, you may discover bugs and errors. In general, we’d like to
hear your feedback on the extension and the use case in general.