1. Formulate five criteria for the evaluation of search engines
To effectively evaluate three different search engines from the perspective of an advanced web user, the following criteria were established:
1. Relevance and accuracy of search results
2. Search speed
3. Advanced search options
4. Other services
5. Site design and layout
More information about these criteria and how they were used is available below.
2. Test three search engines against your criteria
For this evaluation, the following three search engines were tested:
1. AltaVista – http://www.altavista.com/
2. Google – http://www.google.com/
3. DirectHit – http://www.directhit.com/
3. Explain your methodology
The criteria outlined in Part 1 were reached by considering the things that would be important to an advanced user. Of course, factors like Relevance and Speed are critical to such a user (particularly in a work environment). The search options criteria is particularly important to advanced users who are after a specific page and do not want to waste time ‘wading’ through results from a ‘standard’ search. The other criteria are important to most users and mean that the search engines are more usable and useful.
The search results from the three test search engines (Part 2) were evaluate according the criteria, explained it more detail below.
Criteria 1 – Relevance and Accuracy of Results
This criterion was to evaluate the usefulness of a particular search engine according to the accuracy of its results. I defined the relevance, according to personal standards, after browsing the pages in the results, for three separate search terms. The Relevance Percentage is calculated by counting the number of relevant pages that appear in the first 20 search ‘hits’ and then multiplying by 20 (for a percentage).
Criteria 2 – Search Speed
This criterion was to evaluate which of the test search engines returns results quicker.
However, it is important to note that the speed test is both a test of the Internet connection to the remote machine, as well as the time taken by the search engine to perform the search. Furthermore, tests of Internet speeds are very variable based on time of day, other internet traffic etc and are therefore of only limited accuracy. For this reason, no ‘hard numbers’ were calculated (ie timed searches). Instead, I rated each search engine based on 5 searches per engine at the same time, same day. The possible Speed Ratings are as follows: Excellent, Good, Poor
Criteria 3 – Search Options
This criteria was to evaluate the search engines advanced search options. This is important for advanced web users because it allows them to track down specific pages, and weed out irrelevant pages from their searches, meaning they find the most relevant pages quickly.
In evaluating this criteria, the three test search engines were tested to see if they had the following search options:
Variable number of results per page
Search for “phrases”
Boolean Searches (AND, OR, NOT)
Wildcard (*) searches
Search for text in only title or body
Search for pages in different languages
Search for pages located in specific countries
Search for pages under a certain domain
Search for pages containing links to a URL (eg link:http://www.yahoo.com/)
Search for pages LIKE a certain page (eg like:http://www.yahoo.com/)
The number of search options each search engine had available was calculated and then recorded out of 10.
Criteria 4 – Additional Services
This criteria was to evaluate the other services offered by the search engine to make web searching and browsing easier for advanced users. These features make the search engines more useful and more of a ‘one-stop destination’ for finding information, including news, sport results, TV guides, weather reports and other information in a ‘start page’ design to allow the user to easily access the information he needs quickly. These services are particularly relevant to the advanced and regular web users because they use the web frequently, making such a page very convenient.
In evaluating this criteria, the three test search engines were tested to see if they offered the following additional services/features:
Online shopping guide
Suggested searches. Searches based on what you have searched for
Other searches – whitepages, yellowpages, maps/directions etc
Hand-picked categories or channels for web browsing
Personalisation of search results, layout, information etc
The number of additional services offered by each of the test search engines was calculated and then recorded out of a score of 5.
Criteria 5 – Page Design and Layout
This criteria was to evaluate the usefulness and usability of the page