Relevancy rankings: Pay for performance search engines in the hot seat

Pay for performance (PFP) search engines provide search services for documents on the Web, but unlike traditional search engines, they rank documents not on content characteristics, but according to the amount of money the owner of a Web site is willing to pay if a user visits the Web site through the search results pages. A study was conducted to compare the retrieval effectiveness of Overture (a PFP search engine) and Google (a traditional search engine) using a test suite of general knowledge questions. A total of 45 queries, based on a popular game show, "Who wants to be a millionaire?", were submitted to each of these search engines and the first ten documents returned were analysed using different relevancy criteria. Results indicated that Google outperformed Overture in terms of precision and number of queries that could be answered. Implications for this study are also discussed.