Understanding Relevance Assessments for Test Collection Development in E-Discovery

Title: Understanding Relevance Assessments for Test Collection Development in E-Discovery

Guest speaker: William Webber, University of Maryland

Abstract:

The established method of document production in e-discovery (information retrieval for civil litigation) is large-scale manual review by a team of junior attorneys working under the instruction of a supervising senior attorney.

High levels of inter-assessor disagreement, however, make the reliability of such manual review questionable. In this talk, I present the results of an experiment into whether more detailed review instructions can reduce the incidence of assessor disagreement, and hence increase the reliability of review.  Surprisingly, the experiment found no increase in assessor agreement with more detailed review instructions, either amongst experimental assessors or between experimental and official assessors.  We conclude that a conception of relevance is not reliably transferable from one person to another by written instructions.  This finding suggests that the use of automated text classification tools may not only be cheaper, but may be as or more reliable in applying the supervising attorney's conception of relevance as written instructions implemented by human reviewers.

William Webber received his Ph.D. from the University of Melbourne in Australia.  His dissertation topic was on evaluation in information retrieval.  He is currently a research associate at the University of Maryland, working on e-discovery.