ECAI 2016

EGPAI 2016 - Evaluating General-Purpose AI

The Hague, The Netherlands, Tuesday 30th August 2016

A workshop held in conjunction with ECAI 2016

We are organising the 2nd edition! EGPAI 2017 @ IJCAI 2017

Workshop Programme

(Room: South America, World Forum, The Hague)

Keynote Talks


"How to Benchmark Gradual and Guided Learning: Designing the Right Learning Tasks to Teach the Right Skills""


Each panel member will have about 3-5 min. to say a few words about the general panel topic and then we will allow the audience to participate and ask questions to the panel.

Call for Papers

The aim of this workshop is to bring to bear on the expertise of a diverse set of researchers to progress in the evaluation of general purpose AI systems. Up to now, most AI systems are tested on specific tasks. However, to be considered truly intelligent, a system should exhibit enough flexibility to be able to learn how to perform a wide variety of tasks, some of which may not be known until after the system is deployed. This workshop will examine formalisations, methodologies and test benches for evaluating the numerous aspects of this type of general AI systems. More specifically, we are interested in theoretical or experimental research focused on the development of concepts, tools and clear metrics to characterise and measure the intelligence, and other cognitive abilities, of general AI agents.

We are interested in questions such as: Can the various tasks and benchmarks in AI provide a general basis for evaluation and comparison of a broad range of such systems?, Can there be a theory of tasks, or cognitive abilities, that enables a more direct comparison and characterisation of AI systems? How much does the specificity of an AI agent relate to how fast it can approach the optimal performance?

We welcome regular papers, demo papers about benchmarks or tools, and position papers, and encourage discussions over a broad list of topics (not exhaustive):

Important dates

Workshop format

The workshop will begin with a short presentation, followed by four sessions (two in the morning, two in the afternoon) of around 80 minutes, with breaks between them. Technical sessions will consist of a keynote speaker followed by short paper presentations, devoting an important share of time to discussion and interaction. The demo session will present real platforms and ways to evaluate AI systems for several tasks in these platforms. The discussion session will include a panel and a more open discussion about the research challenges around the workshop topics, continuation of the workshop, future initiatives, etc.

Submission of Papers

We welcome submissions describing work in progress as well as more mature work related to AI evaluation.

Submitted papers must be formatted according to the camera-ready style for ECAI'16, and submitted electronically in PDF format through

Papers (technical, demos, position) are allowed a maximum eight (8) pages. An additional page containing the list of references is allowed.

Authorship is not anonymous (single-blind review). Papers will be reviewed by the program committee.

Accepted Papers

Presentation and publication

Authors of accepted papers will be asked to prepare a presentation (short or long) during the workshop.

Pre-proceedings containing all accepted papers will be provided electronically on the workshop web page. The final workshop proceedings will be distributed electronically together with the ECAI conference proceedings. According to a late notice from the ECAI organisation, papers will not be included in the USB sticks that will be given at at the registration desk but will be published on the ECAI conference website instead.

After the workshop, a special journal issue is considered (journal to be discussed), to which, in case, contributing authors would be invited to submit a paper.

Program Committee

Workshop Chairs

Related links

Competitions that aim at general-purpose AI systems: Benchmarks and platforms: Papers, tutorials and books: Related workshops: