Abstract
In human performance, even if a piece is played according to the score, it will not be played according to the score due to fluctuations in tempo, changes in intensity, deliberate insertions and other variations related to performance expression, as well as pure performance errors. Therefore, even if the score is known, it is not easy to track in real time which part of the score a performance is playing.In this research, we have developed a mechanism to track the position of a performance in real time by using a probabilistic model to represent the degree to which a performance varies, and a mechanism to predict the position of the next performance based on the local tempo of the performance.
Application 1: Automatic accompaniment playback synchronized with violin performance
We have developed an automatic accompaniment system that takes the acoustic signal of a musical instrument performance as input and the other parts (system) follow. By preparing a score (MIDI data) for the human performance and the accompaniment parts, a human-machine ensemble can enjoy any kind of music. The video below shows how the system robustly follows the tempo changes and performance errors of a violin performance. In addition to violin, we have also confirmed that the system works with piano and guitar.
Application example 2: Live demonstration of automatic accompaniment playback synchronized with violin performance
At Interaction 2013 in March 2013, we presented a live demonstration of our automatic accompaniment playback system.The conference place was crowded with exhibitors and visitors, so we were very concerned about the effect of ambient noise, but the system performed reasonably robustly.
Arm-type robot that follows piano playing
Since September 2015, Denso Corporation, Denso Wave Corporation and the SOKEN and our laboratory have been collaborating to develop Denmaiko, an arm-type robot that dances to musical performances. The results were exhibited at the Denso booth at the International Robot Exhibition 2015 in December 2015. Cobotta is a small 6-axis arm robot newly developed to enable industrial robots to be used in familiar places, such as the home and education. Cobotta is a small 6-axis arm robot, newly developed to bring industrial robots closer to people at home and in education. Robot technology will be introduced into the home in the future, but in order for robots to work with humans, they must be able to understand human intentions correctly. With this in mind, we came to the conclusion that it would be possible to use performance tracking technology that follows human performance, which led to the joint development of this robot.
During the exhibition period, live demonstrations were regularly held for many visitors to see. We were lucky enough to be able to find a video of the demos on YouTube, courtesy of Mr. Kazumichi Moriyama.
Publications
- Shinji Sako, Ryuichi Yamamoto, and Tadashi Kitamura, "Ryry: A Real-Time Score-Following Automatic Accompaniment Playback System Capable of Real Performances with Errors, Repeats and", Active Media Technology (AMT) Lecture Notes in Computer Science, LNCS 8610, pp. 134–145, Aug. 2014. [DOI]
- Ryuichi Yamamoto, Shinji Sako, and Tadashi Kitamura, "Robust On-line Algorithm For Real-time Audio-to-score Alignment Based on A Delayed Decision and Anticipation Framework", International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 191–195, May. 2013. [DOI: DOI]
- Ryuichi Yamamoto, Shinji Sako, and Tadashi Kitamura, "Accurate and Low Computational Audio-to-score Alignment Using Segmental CRF with An Explicit Continuous Tempo Model", International Workshop on Nonlinear Circuits, Communications and Signal Processing (NCSP), pp. 345–348, Mar. 2013.
- Ryuichi Yamamoto, Shinji Sako, and Tadashi Kitamura, "Real-time Audio to Score Alignment Using Semi-Markov Conditional Random Fields and Linear Dynamical System", The Music Information Retrieval Evaluation eXchange (MIREX2012), Oct. 2012. [PDF]