Registering and analysis of a corpus of natural language assisting requests

Various studies have shown that ordinary users are still unsatisfied with the traditional help systems available for computer software (Help files, searches by keyword or long FAQ). We consider the possibility to enhance their experience by using Embodied Conversational Agents (which benefits in Human-Computer Interaction have been proven) embedded to the software applications. In order to offer an efficient assistance to those ordinary users’ problems, we have to conceive Rational Assisting Agents able to understand both the problems users are facing and the way they express their need for assistance. If at least some of the problems can be forecasted by analyzing software applications, the second point clearly can’t be dealed with without a corpus of real people’s requests collected in situ.

The Daft corpus is a set of 11.000 isolated assistance requests (in French), built up in three successive and complementary steps:

1. Original collection: for two years, about 100 human subjects have been asked to use several applications (from simple applets to dynamic websites) embedding an ECA powered by the first version of the DAFT system, providing 5.000 original typed requests.
2. Cover enhancement: to improve the cover of the assistance language domain, the corpus size has been significantly increased by adding different formulations of original requests inspired from thesauri.
3. Application domain expansion: by adding 3.000 questions from FAQ of text processing systems, we handled the concern raised by the “simplicity” of some of the applications used in the original collection process and reinforced the proportion of questions.

Corpora comparison

In order to characterize the Daft corpus, and indirectly to validate its specifity, it has been compared to similar task oriented ones, from a speech acts point of view. We used three other corpora for this comparative study:

• the Switchboard corpus: 200.000 manually annotated utterances from task-oriented phone talks,
• the MapTask corpus: 128 dialogues in which one person has to draw a map following another person instructions,
• the Bugzilla corpus: 1.200.000 comments from 128.000 bug reports created during the development of the Mozilla Foundation’s suite.

We used the Searle classical taxonomy of speech acts as the reference, converting the original corpora taxonomies into to this one to be able to compare their repartition: Despite the difficulty to convert some acts (which explains the "unknown" additional category), the Daft corpus clearly appears as different from the other ones. We can especially note:

• A majority of directives (57%), explained by the high number of orders and questions. This phenomenon can be explained by the fact users are more direct when interacting with computer than with other humans (as it’s the case for the 3 other corpora).
• A rather low proportion of assertives (13%), as users seem to prefer to express their feelings and states of mind (29%) rather than more objective and neutral facts.
• Very few commissives (1%), which is explainable by the nature of the user-agent relationship, where the latter is subordinate to the former.

Conversational activities of the Daft corpus

During the corpus collection phase, human subjects were informed they had some tasks to do for which they could ask some help (if needed) from an artificial assistant agent. However, they were completely free to do what they wanted, and particularly could type whatever they wanted without any constraint. Various behaviors were observed, some users ending up abandoning their original task, and hence the collected corpus reflects this diversity.
After having randomly extracted two independent subsets of one tenth of the corpus size, we manually gathered requests by similar activities, which allowed us to distinguish four main conversational activities:

1. Control activity: direct controls to make the agent interact directly with the application software by himself.
2. Direct assistance activity: explicit help requests from the user.
3. Indirect assistance activity: user’s judgments about the software/agent, revealing an actual need of assistance.
4. Chat activity: where the user generally focuses more on the agent than the application itself, and which can itself be divided into:
   • reactions to an agent’s answer: a set of way to agree/disagree, expressions of incredulity (“doubt it”), lack of understanding (“you lost me”) or insistence (“please answer”).
   • communicative functions: forms used to start/end the communication, as well as some phatic acts (“are you there?”).
   • dialogue with the agent: from orders (“shut up”) to questions (“do you have a soul?”) and from threats (“don’t force me to kill you!”) to compliments (“you look cute”).
   • comments on the application: without any assistance value (“this page is nice”).
   • others : “I’m an ordinary user”, “I want to do a cognitive gift”...

The existence of chat and control subcorpora reveals that users actually expect the agent to be able not only to bring them assistance, but also to do things in the application by himself (and for them) as well as to have capacities to react to comment not related to the global task they are working on (phenomenon certainly reinforced by the use of a visible embodied agent).

For more information, see:

• F. Bouchet, Caractérisation d’un Corpus de Requêtes d’Assistance, RECITAL'07, Toulouse, June 5-8th 2007
• F. Bouchet, J-P. Sansonnet, Étude d’un corpus de requêtes en langue naturelle pour des agents assistants, WACA 2006, Toulouse, October 26-27th 2006
• F. Bouchet, Conception d'un langage de requêtes pour un agent conversationnel assistant, Master Research Thesis (in french), September 2006