Full Featurs of version 3.3

Sphider Pro version 3.3 has been developed with a host of features. The features have been listed below and a brief explanation of each feature.

Easy installation

Sphider Pro has been developed with an easy installation interface. Once you have your server ready you can use the installation interface to configure your Sphider Pro installation with just a few clicks. This ensures that sphider pro installs correctly with the minimum of effort.

Secure login

We have implemented a secure login feature for sphider pro compared to all other versions of sphider. The login feature also has a remember me option so that you do not have to enter your username and password each visit. Your password is stored and encrypted using the strongest encryption currently available (md5).

Password reset

Because Sphider Pro has implemented a secure md5 login this means that password recovery is impossible. Just incase you do forget your password we have added a password reset feature. This will allow you to request a new password which will be emailed to your email address used for your admin control.

Multiple language support

Sphider Pro has been developed to allow full support to index websites in various languages such as Arabic, Russian and Chinese. Search queries can also be made in multiple languages returning results based on the search query. Websites results will be shown in their native language.

Periodic indexing

Website can be set so that they can be periodically re-indexed. These can be set to monthly, daily or every four hours. In order to use this feature cron jobs are required to be activated on the server.

Default do not includes

Indexing can return un-wanted url’s from time to time, adding the do not includes for each website can be time consuming, when adding websites the include default lists feature allows you to add the most common do not index url if includes list at the click on a button. This feature allows you to add websites faster.

Must and must not includes

Allows you to edit the lists of what and what should not be indexed in a url includes such words or terms.

Spider Bot options

Allows you to select index full website or to what depth of the website.

Sphider Bot can leave domain

This feature allows Sphider Bot to index other websites once it has found links on the current website that it is indexing. This allows for a greater database to be produced without the need to add each website.

Enter sitemap locations

Enter sitemap path locations for individual websites. These should be relative to the main url path, for example   sitemap.xml   or  maps/sitemapMain.xml

Follow sitemaps

Sphider Pro will index xml sitemaps. Sitemaps locations can be specified for individual domain names if know. If no sitemap is know Sphider Pro will try to search for the main location and default file name of a sitemap. If no valid sitemap is found Sphider Pro will default to index links found. Follow sitemaps must be turned on in the admin control settings page for this feature to work

Easily edit website details via admin control interface

Website details can be updated with a simple click. Edit a websites url, category (if category search is enabled) title, description, must include, must not include and periodic cron indexing.

Site categories

Sphider Pro search function allows users to search via categories as part of the advanced search, (if enabled in admin control settings) you can easily edit or delete site categories from within admin control.

View site details

View your website basic details in one place, this includes website url, website title, website short description, its index status (either not indexed or last indexed date) and website category if enabled.

Re-index individual websites

Re-index an individual website direct from the admin control interface.

Index individual websites

Index an individual website direct from the admin control interface

Continue indexing

If indexing has been interrupted you can continue suspended indexing direct from the admin interface.

Browse Pages

Browse the pages of on individual website that has been indexed, use the filter function to find individual pages. Option to delete individual pages of a website that has been indexed.

View stats

View the statistics of a website that has been indexed, this includes its indexed state, (either not indexed or last indexed date, how many pages have been indexed for the website, the total index size of all pages, the cached text size in kb, total number of key words indexed for the website and website size total in kb.

Delete the entire site

The option to delete the entire site and all its associated keywords.

Delete and re-index

Completely delete a website and all its associated keywords, re-add the website with its last known settings to the database ready for indexing all at the click of a button.

Search for websites

As your site grows finding a website to pull up its statistics can be hard, with the search function you can easily search for individual websites from within the admin control. Searches can be preformed by complete url or keyword based on the websites url, title and description.

Database check status and repair

As your website grows larger from time to time databases can become corrupt. This is normally only on large database where sites are being added and removed from the indexing list. To cover this issue we have implemented a check database status and auto repair function in the admin control.

The function will check all tables within your database, and if any table is found to be corrupt or damaged these will automatically be repaired.

For very large databases this option can also be set as a cron job so that this is periodically done from time to time.

Database optimisation

Where databases are being re-index continually and links removed it is beneficial from time to time to optimise your database tables. From within the admin control you can optimise your database when you feel the need arises. This feature can also be set as a cron so this is done periodically for you.

Index pdf, doc, xls and ppt files

Sphider Pro allows indexing of all pdf, doc, xls and ppt files. There are many uploadable converter scripts available and other software packages claim to have them installed, (you just have to configure them). We have opted to leave these poor scripts out and use the scripts already tried tested and 100% working. These scripts are installed or can be installed on all good servers. To index these files you need to activate the settings in admin control.

Show Thumbnails in search results

This feature selectable from within the admin control settings allows you to show a thumbnail of the website url in the search results. This feature allows a small thumbnail of the website url to be shown on the left hand side of the page next to each search result.

Delete and re-index

You can delete the entire website and start a re-index. All data for the website is removed from the database, all links associated with the website are removed from the links tables. Your website is the added back to the website table with the original settings ready for re-indexing. All this is done with a click of a button.

Random Re-index

With this function Sphider pro will randomly select three website from your sites list and re-index these. this is helpful for large websites.

Random Un-indexed

This feature is mainly used for new installations of Sphider Pro. When large amounts of website are added to the database quickly you will produce a large list of un-indexed websites. Sphider Pro can select three of these at random and index them.

Re-index all

For those who wish to re-index / index the whole database in one go.

Cron 1

indexing every four hours. Once this is set you can edit websites in your admin interface to re-index websites every four hours. To do this select the website and then edit. Click cron indexing every four hours, Update site and that's it. next time you cron runs your website will be included in the indexing procedure.

Cron 2

indexing every day. Once this is set you can edit websites in your admin interface to re-index websites every day. To do this select the website and then edit. Click cron indexing daily, Update site and that's it. next time you cron runs your website will be included in the indexing procedure.

Cron 3

indexing monthly. Once this is set you can edit websites in your admin interface to re-index websites every month. To do this select the website and then edit. Click cron indexing monthly, Update site and that's it. next time you cron runs your website will be included in the indexing procedure.

Image Indexing

Sphider Pro will index images on websites url's while indexing. This then populates the database for image searching.

Remove Link

This feature allows you to instantly block a webpage url, not the whole site but if a page contains material that you do not want including you can block a particular url.

Show reported links

This feature can be turned on via the admin control settings panel. once activated users will have the option to report a website url that they feel should not be included within your search results.

Once a url has been reported it will show in your admin area and you can either reject the report or approve. Once approved the url will then be moved to your blocked url tables and will no longer be indexed.

Show Suggested links

This feature can be turned on via the admin control settings panel. Once activated users will be able to click a link and show a simple form to suggest a domain name. These will then show in your admin control area and you can either reject the website or approve it. Once approved this will then move the website to your sites list ready to be indexed. This gives users the ability tpo help build your search engine database.

Image management

As Sphider Pro can index many thousands of images we have incorporated an image management feature from with the admin control. This allows you to view all images stored in the database, filter the image search down and remove unwanted images.

Once you have removed unwanted images from the database they will automatically be added to the removed img table and subsequently will not be included with indexing again.

This is a useful feature to make sure that there are no inappropriate images stored within your database.

Clean Keywords

This feature cleans any keywords that are not associated with links currently in the database. With large databases that have links re-indexed or removed this is a feature that helps keeps table size down and helps speed up search results.

Clean Links

This feature removes any links that is not associated with any website. If a website is removed from the admin control delete site feature that with automatically remove the site and any associated links. Although this feature is not required we have included it as a percussion.

Clear temp tables

When indexing runs temps tables are automatically cleared. If indexing has been interrupted or stopped these tables will not have been emptied. With this feature you can clear the temp tables from previous uses of Sphider Pro Bot

Clear Search Log

Each time a search is preformed these are stored in the search log. The users IP address, the search query, results and time taken are all stored. Over time this can produce a large log file and you may want this to be cleared. Say after initial installation and testing.

Remove Blocked url's

When indexing completes Sphider Pro will run through a series of checks. One of these is to check the against the blocked url's table and make sure that now websites url's have been indexed that are on the blocked list. If indexing was interrupted or stopped these checks would not have completed. This feature allows you to quickly ran a check against the database and check for any blocked urls and remove them.

Remove Blocked images

When indexing completes Sphider Pro will run through a series of checks. One of these is to check the against the blocked images table and make sure that now websites images have been indexed that are on the blocked list. If indexing was interrupted or stopped these checks would not have completed. This feature allows you to quickly ran a check against the database and check for any blocked images and remove them.

Remove 404 url's

This is something that we have all come across. When searching we have clicked a link that has returned a 404 not found message. This happens as people change their website structure. When indexing Sphider Pro will just see that the page is present and that it has some text. This feature allows you to clean your database and remove these urls from your search feature.

Clear Empty Searches

Even when a user clicks the search button and no search term has been entered Sphider Pro still records this data. To combat this we have added the clear empty searches function. This will remove these from the search log and keep only relevant results in the log.

Overall Statistics

The overall statistics is the main page under the statistics feature of the admin control panel. This gives you a general outlook of the amount of sites, links, keyword relations, images keywords, total sites size and the cached text size.

Top Keywords

This feature shows the most commen keywords and how often that key word appears from all the pages that have been indexed by Sphider Pro.

Largest Pages

This feature shows a list of the largest pages that Sphider Pro has indexed. This is displayed by a link to to url and the size of the url in kb.

Most popular searches

This feature shows the most popular search. These are displayed in a table displaying the search query, the amount of times searched (count), the average results and when it was lasted search for.

Search Log

This feature allows you to have a more in depth look at the search queries that are being preformed and by what IP address. This table displays the search query, the type of search; either image or web search, the amount of result returned by the search query. When the search query was made, the time taken to perform the search query and also the IP address of who performed the search query.

Sphidering Logs

If save logs is set in the admin control settings panel all log files will be saved. This feature allows you to access your saved log files. From here you can view or delete your saved indexing log files.