Feature Articles: Creating Immersive UX Services for Beyond 2020
Real-time Moving Object Detection Technology and Trial of Stone Location Delivery at a Curling Venue
NTT Service Evolution Laboratories is researching and developing tracking technology combined with video understanding as part of efforts to develop advanced video services. This technology features both high recognition performance and a short processing time and shows promise for application to public event viewings and stadium solutions. This article introduces a trial conducted at the 2017 Sapporo Asian Winter Games in which this technology was applied to deliver stone location information to spectators at a curling venue.
Keywords: moving object detection, image processing, curling
NTT Service Evolution Laboratories is researching and developing tracking technology for real-time recognition and tracking of objects in video with the objective of developing more advanced stadium solutions and public viewing of events. The aim here is to achieve tracking technology that has both a short processing time and high-accuracy object recognition to provide in real time meaningful information to spectators of competitive matches, audiences of theatrical performances, and other event spectators. We can envision various types of applications using tracking technology combined with a video understanding system, for example, recognizing and tracking an athlete or the ball during a sports match and overlaying information as in augmented reality, and recognizing and tracking the motion of an actor’s hands and adding stage effects. We can expect new video services to be offered in this way.
In this article, we outline the real-time moving object detection technology that we developed targeting curling stones and describe the trial we conducted at the curling competition held at the 2017 Sapporo Asian Winter Games using a system incorporating this technology.
2. Demonstration experiments on providing new spectator experiences at Asian Winter Games
The City of Sapporo and NTT held demonstration experiments on providing new spectator experiences using advanced information and communication technology at the 2017 Sapporo Asian Winter Games held in Sapporo, Hokkaido, from February 19 to 26. The targeted sports were figure skating, ice hockey, and curling, and these demonstration experiments used an official smartphone application (app) for the 2017 Sapporo Asian Winter Games to showcase new spectator services that could not be experienced by the traditional means of watching television or observing the sports at the venue.
When viewers watch a curling match, their ability to accurately understand the stone’s current location is tied to having an accurate understanding of the state of the match. However, since it is impossible to obtain an overhead view of the ice-based curling sheet (playing surface) and house (the circular target on the sheet) from spectator seats inside the venue, it has been difficult to accurately grasp the locations of the stones. To address this problem, we aimed to provide a new spectator experience service enabling a spectator inside the curling venue to grasp the state of the current match by delivering stone location information to the smartphones of spectators.
To deliver stone location information that is useful in understanding the state of the match, the stone location must be obtained during the match as the match progresses. We therefore developed technology for detecting in real time the location of a stone as a moving object and conducted a trial on the use of moving object detection technology to deliver stone location information to spectators at a curling venue (Fig. 1).
3. Real-time moving object detection technology targeting curling stones
To achieve low-cost detection of the locations of all stones present within the effective area between the hog line and back line without making modifications to players or their equipment, we decided to make use of existing venue stones and cameras. The camera above the sheet of a match in progress has a limited shooting range and cannot be controlled freely, so we used a camera above a sheet not currently in use and oriented it to overlook and shoot the effective area (Fig. 2).
We developed technology for inputting video taken in this way and for detecting in real time the team color and location coordinates of each stone. The process flow for each video frame is shown in Fig. 3 and described below:
(1) Projection transformation: A projection transformation is applied to the oblique view of the input image shown in Fig. 1 to achieve an overhead image. Subsequent processing is limited to the effective area after this projection transformation to reduce computational complexity.
(2) Foreground extraction: A background image less any stones or athletes is shot, the foreground is extracted by background difference processing, and the result is passed to the next process as stone candidate areas.
(3) Extraction by color: Only two colors—red and yellow—were used for the colors of stone surfaces in this trial, so two types of color patterns were recorded from stone images prepared beforehand. This process compares these red and yellow patterns with the stone candidate areas, extracts those areas with similar colors, and passes the result to the next process.
(4) Extraction by shape: The stone candidate areas input to this process would include athlete uniforms if their colors were reddish or yellowish. To therefore exclude such candidate areas and detect only stones, this process detects the outline of each input stone candidate area and uses its shape and size to judge whether that candidate area corresponds to a stone.
(5) Coordinate correction: The above projection transformation assumes a flat target, so any object having height as in the case of a stone surface will be transformed to a location offset from the actual location. This process corrects each detected location to estimate a more accurate location and finally outputs sets of stone colors and stone coordinates as program results for this frame.
4. Stone location delivery system for spectators at curling venue and trial
We constructed a stone location delivery system based on real-time moving object detection technology that we developed. This system delivers stone location information to a smartphone app every time a throw is completed, enabling spectators at a curling venue to understand where the stones in a match are currently located. In this trial, a function for recognizing the time that a throw is completed (the time at which all stones are stationary) was not implemented as part of the program but simply achieved by having the operator determine that time visually.
The basic configuration of this system for one curling sheet is shown in Fig. 4 and summarized below.
As previously described, video signals from existing camera facilities in the curling venue are distributed and used as input video to the proposed system. In curling, however, the stone-throwing direction changes every end*, so the effective area enclosed by the input image must be switched for each end. The system therefore installs a switcher so that the video input to the video analysis personal computer (PC) can be selected from the video output from two cameras.
Now, once the operator visually determines that a throw has been completed and operates the system accordingly, the video analysis PC detects the location of that stone from the input video using real-time moving object detection technology and passes that result to the delivery control PC.
Next, the operator at the delivery control PC visually checks the received stone location information for errors, corrects errors if any exist, and uploads the result to the delivery server. The smartphone app accesses the delivery server every few seconds. If the app detects an update on the stone location information at that time, it obtains that information and updates its display content.
This system trial was conducted during the curling competition held February 18–24. The system operated without problems during this period and received a favorable response from users and curling associations.
5. Accuracy of stone location detection results in trial
We evaluated the accuracy of the stone location detection results using video of 21 matches of the men’s and women’s curling competitions at the 2017 Sapporo Asian Winter Games. The stone location after being visually checked and corrected if necessary by the operator of the delivery control PC was taken to be correct location information.
For each match, the number of stones corrected by the operator of the delivery control PC, the total number of stones placed in the effective area and targeted for location detection, and the stone correction rate are listed in Table 1. The stone correction rate per match fell in the range from 1.12% to 25.98%, resulting in a value of 11.12% for all matches. This means that the probability of having no correction needed by the operator of the delivery control PC was approximately 90%.
Plots of corrected stone locations on the curling sheet are shown in Fig. 5 by stone color. A comparison of the two sets of plots indicates that the number of corrections is higher for red stones and that many of these red-stone corrections appear on the red circle of the house. We surmise that the close resemblance of the house color and the red stone color led to errors in detection.
6. Future development
In this article, we described real-time moving object detection technology targeting curling stones, a stone location delivery system incorporating this technology, and a system trial conducted at the 2017 Sapporo Asian Winter Games. In the trial, the stone locations were detected with an accuracy rate of about 90%. We were able to provide a new spectator experience service that delivered stone location information to spectators at the curling venue without any problems during the period of these games.
Going forward, we plan to continue our research and development efforts to enhance tracking accuracy using moving-object locus information and to expand the number of recognizable moving objects. In this way, we seek to provide new video services featuring, for example, new ways of viewing sports and new types of staging in entertainment.