Video captioning is known as a method to summarize or explain a video. However, it is very difficult to create sports commentaries, which are running scene-by-scene descriptions, for sports videos by using conventional video captioning. This is the case because sports commentary requires not only specific and varied information about every scene, such as player action descriptions in baseball, but also background knowledge and dynamic at-bats statistics that are not found in the video. We propose a new system to automatically generate commentary for baseball games. In our system, given real-time baseball videos, suitable descriptions are relayed using four deep-learning models (i.e., a scene classifier, player detector, motion recognizer, and pitching result recognizer) integrated with domain ontology. Using these four deep-learning models, pieces of information about “who is doing what in which area of the field” and “what results are expected” are obtained. This approach is used to select an appropriate template, which is combined with baseball ontology knowledge for the generation of commentary. We evaluate our system using baseball games from the KBO (Korea Baseball Organization) League’s 2018 season. As a result of the evaluation, we identify that our system can act as a commentator for baseball games.
| [New] Demo (with NAO Robot, 5:11) |
[New] Demo (with English, 6:47) |
[Old] Demo 1 (with pitching result recognition, 2:27) |
|---|---|---|
| Onedrive | Onedrive | Onedrive |
| Ground to 1st Base | Double Play |
 |
 |
| Outfield Single Hit | Outfield Two-Base Hit |
 |
 |
| Type | Method | Example | in English |
| A | Casting using *Web Data, Pitching Result Recognition and Ontology(knowledge) | 볼 인가요? 스트라이크! 구석에 꽂히는 직구에 타자가 제대로 속았습니다. 유희관 투수 이번 시즌 높은 확률로 초구 스트라이크 잡아내고 있습니다. | Is it Ball? Strike! The fastball in the corner. Yoo Hee-kwan is making his first pitch this season with a high chance. |
| B | Casting using Scene Data and Ontology(knowledge) | 오지환 타자와 유희관 투수 사이에 긴장이 흐르는 가운데 오지환 타자 이번 시즌 2할 7푼의 타율을 기록하고 있습니다. | Amid tensions between Oh and Yoo, Oh has recorded a .27 batting average this season. |
| C | Casting using Motion and Position of Player, and Ontology(knowledge) | 유희관 투수 공을 던졌습니다. | Yoo Hee-kwan pitched. |
| About. BatterBox | ||||||
| BatterBox | Batter | Close up | ||||
 |
 |
 |
||||
| About. Field | ||||||
| 1st. Base | 2nd. Base | 3rd. Base | Right Outfield | Center Outfield | Left Outfield | ShortStop |
 |
 |
 |
 |
 |
 |
 |
| Etc. | ||||||
| Coach | Gallery | etc. | ||||
 |
 |
 |
||||
It is difficult to get a similar amount of scene data for each class because of the relatively infrequent number of plays near third base and right field. To solve this problem, we take advantage of the symmetric features of a baseball stadium. The flipped image of the 1st base area and the image of the 3rd base area look alike, despite the different stadiums. We supplement insufficient data for third base and right field by flipping first base and left field data. This is also applied in the opposite direction.
| Pitcher | Batter | Catcher | Fielder | Referee |
 |
 |
 |
 |
 |
| ex) Scene of 1st Base | |
| Detected one field player. | Detected many field players. |
 |
 |
| Only one player who is in 1st Base scene => 1st Baseman |
The player who is in significant area of 1st Base scene => 1st Baseman |
| Pitching | Waiting | Swing | Catching - Catcher |
 |
 |
 |
 |
| Throwing | Walking | Running | Catching - Fielder |
 |
 |
 |
 |
- Strike
- Ball
- Foul
- Hit
- Ground out
- Fly out
- Etc.
| Knowledge | the recent record between the hitter A and the pitcher B |
|---|---|
| Query | SELECT ?date ?stadium ?inning ?result WHERE {?s ?toHitter ?A . ?s ?fromPitcher ?B . ?s ?hasTimeStamp ?t . ?s ?hasResult ?result . ?s ?inInning ?inning . ?s ?inGame ?g . ?g ?inDate ?date . ?g ?hasHomeTeam ?h . ?h ?hasHomeStadium ?stadium} order by desc(?t) |
| Template | KOR) B 투수와 A 타자의 최근 전적은 {date} {stadium}에 진행한 경기 {inning}회 {result}입니다
ENG) The batter "A" {result} in the {inning} inning of the latest game against the pitcher "B" on {date} at {stadium} |
| Commentary | KOR) B 투수와 A 타자의 최근 전적은 2018년 10월 10일, 잠실에 진행한 경기 8회 스트라이크 아웃입니다
ENG) The batter "A" struck out in the 8th inning of the latest game against the pitcher "B" on October 10th, 2018 at Jamsil |
- In N company, there is a KBO Text Broadcasting site.
- Each events can intelligible with Ontology in real-time.
#download pretrained four models
mkdir _model
cd _model
wget -O https://1drv.ms/u/s!AtL3vScJgk8Fkcpcdo4SbT09PKJlvg?e=dq4uHV
unzip -j _model.zip
#download the web data files (about result "{videoname}.txt", about ball "{videoname}_ball.txt") to _data/{videoname}/
#download the video "{videoname}.mp4" to _data/{videoname}/
#ex) 2018. 09. 06 LG vs NC game
mkdir _data
cd _data
wget https://1drv.ms/u/s!AtL3vScJgk8FkcpryZAWuKuF18hiDQ?e=cOAJov
#modify GAME_DATE, GAME_NO, FILE_NAME in settings.py to find video and web data
#modify FIRST_BATTERBOX_START_IN_VIDEO, FIRST_BATTERBOX_START_IN_OWL, START_FRAME to make the same time point of the web data and the video
#you can set the FIRST_BATTERBOX_START_IN_VIDEO, FIRST_BATTERBOX_START_IN_OWL using frame number and pitchID of {videoname}.txt of the first pitching judgement
vi settings.py
#run the Automatic-sports-commentary-framework
python kbo.py
#run the simulation with predicted bounding boxes and labels
vi kbo.py
#modify isSimulation=False to isSimulation=True
python kbo.py
#run the simulation with predicted bounding boxes and labels
vi kbo.py
#modify isSimulation=False and isAPI=True
#modify host and port in run_server method
python kbo.py