Choreographed and Orchestrated media refers to a relatively new concept involving different aspects of a coordinated media presentation coming from different networked devices. In this case, COVA will allow video annotations such as audio descriptions to be played from a personal device such as a smartphone, while a primary video presentation is being played on a completely separate device such as a projector in a theater.
The initial COVA prototype will be based on a purely audio identification and synchronization approach, made famous by such services as Shazam and SoundHound. The smart phone will use its microphone to sample the sound track of the video being played and perform a rapid matching algorithm that identifies the video from a database. The smartphone app can then automatically synchronize a separate, private description channel through the smartphone.
A technology such as COVA will be especially useful in educational situations such as assemblies or presentations by third-party educators to large student groups. The presenters may not be aware of the availability of, or need for, the description, but COVA will allow individual students or teachers to quickly and seamlessly find the available description and synchronize to it through the student’s own smart phone. The descriptive and synchronization information can be stored and distributed through an open-source framework such as DVX.
The research and development associated with COVA focuses primarily on the applicability of existing listening and matching algorithms to educational video streams. Focus group feedback will be used to help shape our idea of the potential uses and benefits of such a technology, as well as some of the potential problems with it. Preliminary algorithms and prototype software are being developed in collaboration with the IDEAL Group, a DLN Partner, and the viability of the approach will be evaluated from the standpoints of feasibility and usability. As with all software developed under the VDRDC, the COVA prototype will be made available as free and open-source software (FOSS) at the conclusion of the project.
Get Involved
If you are interested in vision science or want to learn more about low vision and blindness, there are many opportunities to get involved at The Smith-Kettlewell Eye Research Institute.
