ELIAS

Although the speed of computers and volume of information has grown exponentially since the first search engines were created in the 1950s, nearly 6 decades on, the methods used to test state of the art information access systems have changed little. However, there is growing scientific evidence of a need for a complete testing overhaul: producing novel evaluation methods that take a user-oriented perspective on assessing the effectiveness of information access systems

The proposed network will establish a research and training programme for the evaluation of information access systems which will define a new measurement paradigm based on so-called living laboratories. This paradigm involves:

• Exploitation of novel market places and forums where large numbers of users are recruited into early stage evaluation experiments to test a particular aspect of an information access system
• using operational systems as experimental platforms on which to conduct user-based experiments at scale.

The network will achieve an evaluation methodology directed at user-oriented interactive evaluation, a new and tested set of interactive evaluation metrics, infrastructure and test suites based on these and an ongoing collaborative forum with evaluation cycles with a focus on evaluating information access systems, as well as producing a European network of new researchers trained in the improved methodologies.