Overview

EXPERImonitor is a framework focused on the management of experiment content that allows developers to explore the relationship between QoS and QoE in complex distributed multimedia systems. The tool is specifically designed to support the observation of systems where user-centricity, mobility, ad hoc participation and real-time access to information are critical to success.

EXPERImonitor uses a hybrid data model that combines formal low level metric reporting with semantic provenance information. The hybrid approach provides the ability to collect large quantities of measurement data (e.g. service response times, network latency, user satisfaction, etc) whilst allowing for exploration of causation between observations within such data (e.g. user satisfaction in relation to service response time).

The ability to efficiently traverse experiment content between QoS and QoE is an essential capability for evaluation of complex socio-technical systems. Data exploration can provide indications of factors that influence each other and is used to segment data for further investigation and analysis. With ever growing big data sets generated by Internet systems, EXPERImonitor can significantly reduce the time from observation to insight.

EXPERImonitor is a web service with a web-based admin interface and a REST API which connects to clients via RabbitMQ in order to receive high-volume monitoring data. Client APIs are available in multiple languages (Java, Android, C#, C++, Ruby). The web interface offers a live view of incoming metric data and a data explorer view for completed experiments. Data may also be exported to CSV files for further analysis in more specialised tools.

The software was developed primarily in the EXPERIMEDIA project to support distributed multimedia experiments. The 3DLive and STEER projects also contributed to and used the software.

License

The software is Copyright University of Southampton IT Innovation Centre. It is licensed under the LGPL v2.1 license.

Licenses for the Java and web application components of this software and of the third-party software distributed with and/or used in those components may be found in the top-level “licenses” folder.

Licenses for the Android, C# and C++ libraries and their dependencies may be found in the “IPR.txt” file.

Contact

For further information on collaboration, support or alternative licensing, please contact:

Website:http://www.it-innovation.soton.ac.uk Email:info@it-innovation.soton.ac.uk Telephone:+44 (0)23 8059 8866

Screenshots

Participant QoE summary

Live metric dashboard

Overlaying activities on a QoS graph

Documentation Summary

The documentation provides:

• An introduction to the EXPERIMEDIA test-bed context
• An overview of the EXPERImonitor experiment process and metric model
• Instructions on how to set up EXPERImonitor within an experiment test-bed
• Instructions on how to use EXPERImonitor dashboard to execute and monitor an experiment
• A brief review of each of the EXPERImonitor’s internal components
• Guidance on how to write an EXPERImonitor software client

EXPERIMEDIA Venue-Based Experimentation

EXPERIMEDIA was an FP7 funded project that was focussed on developing and operating unique facilities for large-scale, Future Media Internet experimentation. The experiments were conducted on test-beds situated in advanced and exciting real-world venues that host large communities of people interacting with each other in physical and on-line contexts using new FMI technologies and infrastructures.

Within these venue based communities, EXPERIMEDIA developed a test-bed environment that offers a range of cutting-edge FMI ‘baseline technologies’ (including 3D and augmented reality systems; media streaming services and pervasive gaming) upon which technologists and experiments can build innovative new systems. The overall framework for this environment can be seen in the figure below.

Initial venues chosen by EXPERIMEDIA represent environments that enjoy an advanced set of infrastructure services that are suitable for FMI experimentation. EXPERIMEDIA’s baseline technologies are a set of components (themselves composed of multiple sub-systems) that offer the FMI services including integration with (and analysis of) social networks; pervasive services for mobile users (including user tracking; QoE sampling; and real-world gaming); high quality media/meta-data streaming and user generated content management systems; and 3D human motion acquisition and analysis.

For further information on this architecture and the content components, please see the EXPERIMEDIA deliverables.

EXPERImonitor was designed to support the evaluation of FMI systems developed in the EXPERIMEDIA, STEER and 3DLive projects, by:

• Providing a metrics based experimentation process
• Providing an Internet based experiment management support system, including:
  • Semi-automatic deployment of experimentation systems
  • Metrics monitoring and visualisation
  • Experiment metrics data management
• Providing an API intended to support the instrumentation of an extensible range of technologies

An indicative deployment of an EXPERImonitor system and its integration with baseline and new, experimental FMI technologies is depicted in the diagram below. In this illustration, we see two distinct content pathways. First of these is the FMI Content data flow (blue arrows) depicting data interchanging between the technologies used to provide the novel FMI services and experiences to the end user within what is referred to as the EXPERIMEDIA content lifecycle. Second, we see the experimental instrumentation integration (orange arrows) between some (not necessarily all) of the technologies delivering services. Metrics creation and management is controlled by the EXPERImonitor via a two-way communication protocol operating with the EXPERIMEDIA experiment lifecycle. This protocol is described in further detail in section ECC monitoring protocol.

In the following sections, the reader is introduced to the EXPERImonitor and its principal components; the experimental process that it supports and the data model that underpins experimental metric capture.