At least there is a way around this using PHP: Removing HTML Tags from Strings and PHP: Avoiding mySQL Injections. There are also a few other tricks up our sleeves to combat this sort of activity but more investigation and research needs to be carried out on the effective mechanisms for securing input data. One possibility would be to allow only text based input but that could ultimately prove difficult in a multi-lingual environment.

Bots are ideal for this repetitive automated activity by taking a work queue (a given list of URLs) and producing a set of information that can be passed to the next stage in the submission process. Even though these techniques are called bots they are essentially just software programs that run either at a scheduled time or in a loop constantly checking the work queue.

In addition to the usual information a bot can periodically extract additional words from body text itself by parsing out html tags, removing stop words and building a word frequency table. A word frequency table is simply a list of words with the number of times a word appears in the given text.

Recently I have been working to find a way of automatically extracting information for a given URL by accessing the html from a submitted URL, parsing the relevant information and using this information as input to the submission process. Information is often placed as meta tags in the HEAD section of html but I have found this process so far to somewhat hit and miss as some sites include the information and others do not. For those that do not an additional ways needs to be identified to provide a suitable link title and description - it may be the case that this will always require some user intervention but further analysis should identify more.

The bot also needs to be aware of the period of time since the last change of a web page to ensure that valuable information or updates is not missed and that processing power isn’t wasted on a link that does not change very much. The way we can address this is by producing a hash for a page and comparing it with a previously stored hash.

A hash, simply put is the result of a consistent algorithm applied to information. MD5 is a well known hash mechanism that I may investigate in a later article. For now a simple example to demonstrate a hash would be to calculate a check digit by taking letter values of a phrase, adding them together then finding the modulus using a number say 27 (allowing 26 letters and the space), in the example the first two phrases give the same answer however the third phrase with one letter different gives a much different check digit.

I have a working program up and running - but not in bot mode yet -but can be initiated manually. I need to add the loop code to continually check the work queue for submitted URLs.

Stop words are a mix of short, common and everyday words such as “a”, “when”, “besides”, and “to” that are sometimes ignored by search engines when a user is searching for information. There are positive and negative elements to this, the positive element is that it makes word lists much shorter making calculations quicker but negatives can impact the context of the intended search. This can be compounded by stemming phrases (We will cover stemming in a later post but perhaps besides may be stemmed to beside).

An example of the negative element could be if I were searching for images of “plants beside water”, if the word besides is stemmed to beside which is filtered as a stop word then the search engine may think I am interested in “plant water” or “water plants” or something else which would likely impact the results receive.

An example portion of a stop word list may be:

b ba back be became because become been before began beget behind being beside best bet between big bin both but by

Stemming could increase the size of the list to effectively make the following variations of the word “back” into stop words: backed, backing, backs, backend, backy.

There does not appear to be a universal list of stop words, which complicates matters, and an important factor is language variations which could make some words “as typed” stop words in one language but not in other languages.

Our initial view is that we may generate a stop word list however not implement it - or even make it optional. This way we could assess the difference in the results gained from using stop words and not using stop words for a particular search phrase.

This methods appears to be: a web page is presented to the user with a search box, we type a keyword (or more) into the search box then click the search button and the relevant matching results are returned.

However, whilst it may seem that the process is pure information retrieval there is far more going on behind the scenes and what is going on behind the scene is information collection.

Information collection is a valuable part of any search engine project. Simply returning results based on a keyword pattern match is only half (or maybe even less) of the story and is unlikely to contribute to the success of a search engine project which is why we are looking at it as an important factor of the Eggnchips project.

With information collection, results can be more intelligently returned based on a number of factors including popularity, occurrence, relevance, perhaps even context and user profile information such as geographical region and browser/computer information.

With the application of information collection the search engine can generate its own information based on user habits which provides a more dynamic and usable platform.

The primary elements of the Search Manager include:

Match keyword

Begin the process with a simple keyword match as before

Results Profiling

Modify the results based on previous results returned, and user actions based on those results, such as click-through rate, number of back links to results, and peer grading information.

User Profiling

Modify the results, as appropriate, based on information detected from the user such as previous search history, browser information, geographic information. This area can be made an opt-out depending on user preferences.

Update Manager

Update the User Profiling and Results profiling stores based on this information. The Update Manager also collects popular search terms and results and also watches for new added content that matches these terms.

The green areas indicate interactive web pages, blue areas are components. The blue components identified are URL Submission Validation, Keyword Validation, Update Manager, Mail Manager and Search Handler and Presentation/Formatting Component.

Initially I used a html table arrangement for the web pages however it would make sense to switch to CSS which I aim to do before development progresses any further.

• The component that submits web sites (I call it the Finder)
• The component that Gets the web sites and stores them in the Database (I call it the Getter)
• The component that handles the submitted search query and returns the results (I call it the Searcher)

A breakdown of the components follows:

Finder

The Finder receives work requests using a web based form. Once submitted this information (URL,title,description,keywords) is added to the work queue. The processor sub-component analyses the work queue and checks various parts of the information submitted then submits the results to the Getter. Some of the checks include whether the URL is well formed and valid, and obtaining manual authorisation to include the web site.

The Form submission may benefit from an image based input mechanism.

Getter

The Getter retrieves the information from its work queue, gets any other information that is required then places it into the various database structures. Some of the jobs of the Getter would be to retrieve counts of keywords, store the date of the submission, and categorise the web site.

The Getter needs to assess the impact of Spammers who may try to trick the Finder, possibly the addition of a Spam list to ensure that no site gets through. Also, a manual submission authorisation is required before a site goes live. There are a whole bunch of different ways that Spammers could try and add sites to the search engine so this is likely to become a topic of its own.

Searcher

The searcher is driven by a web based form. Once submitted the form checks the keywords against those submitted previously by the Getter and returns the necessary results to the end user.

The first stage of the search engine is up and running with the ability to perform some simple keyword searches. The engine shows the basic screen (the idea here is to keep the homepage short and sweet). An option is included to make EggnChips the home page. A search term can be entered and I have included a few other buttons to make the engine more versatile. Also included is a running count of the number of approved web sites listed.

When someone enters a keyword (or multiple keywords) the term is validated (to ensure it is safe) then checked against the URL database and appropriate entries listed. I have added a keyword highlighting mechanism so the searched phrase appears in green.

The next stage I plan to add is a URL submission facility.

Master Categories: #cats#

Sub-Categories: #subcats#

Total Links: #url-count#

Total Links Pending: #url-pending#

Link Submissions Today: #url-pending-today#

In this project a form will be presented the user with a familiar text input box followed by a button for performing a search. To make the system usable and to provide a mechanism for comparison several other buttons will be in operation to redirect searches to other reference sites.

Upon a user entering a keyword and clicking on the search button, a simple lookup will be made in a back-end database against the keyword and the matching results presented.

The next step will be to enable the use of multiple keywords in an and/or mechanism. Simply entering two keywords will give assume an OR whilst prefixing each keywords with a plus symbol will force an ADD decision to be made. Enclosing a series of keywords in quotes will turn the keywords into a phrase entity which will itself become a keyword for search purposes.

In order to conduct a search the back-end database will be populated with a number of popular web site address URL together with the site Title, Description and Keyword Tags. A facility to add a URL to the database will be added in the next phase.

At present the form is working together with the alternate buttons however the back-end database has yet to be populated. This will be performed over the next few weeks.

The project progress can be viewed at http://www.eggnchips.com and is always to be considered a ‘work in progress’.