Gary A. Olson 
Experimental Comparisons of Usability Evaluation Methods [PDF ebook] 
A Special Issue of Human-Computer Interaction

This special issue contains essays regarding the CHI ”95 conference, which featured a panel titled, Discount or Disservice? Discount Usability Analysis: Evaluation at a Bargain Price or Simply Damaged Merchandise? Wayne Gray, who organized the panel, presented a controversial critique of studies that had evaluated various usability evaluation methods (UEMs). The level of interest in this discussion led Gray to propose a review article that dealt with the issues in a more systematic fashion. The resulting essay, written by Gray and his collaborator Marilyn Salzman, conducted an in-depth review of a series of influential studies that used experimental methods to compare a variety of UEMs. Gray and Salzman”s analysis was framed using Cook and Campbell”s (1979) well-known discussion of various forms of validity. They used this to evaluate numerous details of these comparative studies, and they concluded that the studies fell short on the criteria by which good experimental studies are designed and interpreted.