Map
Index
Random
Help
Notes
Related
th

ThesaHelp: a preliminary test of directed search

Thesa topic root > ThesaGroup: search with Thesa


--related--
search with Thesa
how to find everything relevant to some topic or question
KMS questions for testing directed search
needle-in-a-haystack test of directed search
read-everything test of directed search
why does directed search succeed for Thesa

--referenced--
all Thesa topics
map of the Thesa web site
Thesa - a thesaurus of ideas

Summary

Barber performed an informal test of directed search in October 1989 with several subjects, and presented the results in a research seminar. The experiment should be redone in a formal setting.

The results indicate that directed search is better than keyword search or read-everything search.

Procedure

One person selected and wrote 10 questions for passages ("needles") in (QuoteRef: akscRM7_1988 ). The questions did not use keywords from the passages. The subjects were computer science graduate students. They were given a short introduction to Thesa and to the task. The subjects searched for relevant material for each question. There was three conditions: reading a preprint of the article, using a keyword index, or using directed search in Thesa.

See notes below for further details.

Read-everything Search

question@ @#1@ @#2@ @#3@ @#4@ @#5@ @#6@ @#7@ @#8@ @#9@ @#10@ @Total

found@ @1@ @8@ @15@ @9@ @7@ @2@ @6@ @1@ @4@ @1@ @54

missed@ @0@ @2@ @6@ @0@ @2@ @0@ @1@ @0@ @1@ @0@ @12

searched@ @220@ @220@ @220@ @220@ @220@ @220@ @220@ @220@ @220@ @220@ @2200

found needle@ @1@ @1@ @1@ @1@ @0@ @1@ @1@ @1@ @1@ @n/a@ @8

results@ @

recall@ @0.82

precision@ @0.02

needles@ @8 of 9

Keyword Search

question@ @#1@ @#2@ @#3@ @#4@ @#5@ @#6@ @#7@ @#8@ @#9@ @#10@ @Total

found@ @1@ @2@ @7@ @3@ @3@ @1@ @3@ @0@ @1@ @0@ @21

missed@ @0@ @8@ @14@ @6@ @6@ @1@ @4@ @1@ @4@ @1@ @45

searched@ @1@ @13@ @17@ @5@ @32@ @42@ @31@ @3@ @8@ @17@ @169

found needle@ @1@ @0@ @0@ @1@ @n/a@ @1@ @1@ @0@ @0@ @n/a@ @4

results@ @

recall@ @0.39

precision@ @0.12

needles@ @4 of 8

Thesa Directed Search

question@ @#1@ @#2@ @#3@ @#4@ @#5@ @#6@ @#7@ @#8@ @#9@ @#10@ @Total

found@ @1@ @10@ @21@ @9@ @8@ @2@ @5@ @1@ @5@ @1@ @63

missed@ @0@ @0@ @0@ @0@ @1@ @0@ @2@ @0@ @0@ @0@ @3

searched@ @17@ @44@ @38@ @13@ @18@ @18@ @28@ @5@ @25@ @5@ @211

found needle@ @1@ @1@ @1@ @1@ @n/a@ @1@ @1@ @1@ @1@ @n/a@ @8

results@ @

recall@ @0.95

precision@ @0.30

needles@ @8 of 8

Notes up

Found measures the number of relevant quotations found. Missed is the number of quotations found in some other search. Searched is the number of quotations viewed. Recall is 'found/(found+missed)' . Precision is 'found/searched'. Needles is the number of needles (i.e., pre-selected passages) found.

The passage for question #5 was from a figure of the preprint. It was not in the electronic copy of the article.
The subjects decided that the passage for question #10 was not relevant to the question. So the directed and keyword searches had eight needles to locate, and the read-everything searches had nine needles to locate.

In the read-everything case, Searched is the total number of quotations identified for the directed and keyword searches plus the number of tables and figures. Subjects were explicitly asked about the relevance of unfound needles. Other quotations were deemed relevant if one of the subjects deemed it relevant.

Related up

ThesaHelp: how to find everything relevant to some topic or question
ThesaHelp: KMS questions for testing directed search
ThesaHelp: needle-in-a-haystack test of directed search
ThesaHelp: read-everything test of directed search
ThesaHelp: why does directed search succeed for Thesa

Updated barberCB 10/1997
Copyright © 2002-2023 by C.B. Barber
Thesa, Avev, and thid-... are trademarks of C.B. Barber