3F_Crawler

The 3F Crawler loads a page, with the assumption that it is perfect, giving it a score of 100%.%20Of course, it then starts checking the server for points lost, for example, does the server's HTTP header include "Server: <product>"? If so, take away 2 points for insecurity.

Next, the content is analyzed for poor HTML quality, for example does the page contain <script> without type or language specified, then it deducts 1 point per instance.

A page with a score below threshold is not added to the search engine database. Even though the page may have the greatest content in the world, a poorly designed page and possible security vulnerability site - will be rejected from our database.

The 3F Crawler also contains a database for a few statistics, like Number of Images, Number of Links, Number of DIVs, Number of Scripts, Contains Flash, etc.

HTTP DEDUCTIONS

1. -2 Points - Server: variable returned in header.
2. -2 Points - X-Powered-By: variable returned in header.
3. -2 Points - X-AspNet-Version: variable returned in header.
4. -1 Point - Missing P3P: variable in header.
5. -1 Point - Missing Content-Length: variable in header.
6. -1 Point - Missing Date: variable in header.
7. -1 Point - Missing Content-Type: variable in header.
8. -1 Point - Missing Content-Language: variable in header.

So depending upon your server software and setup, the best your page could get is 100%%20or 89%.

HTTP STATISTICS

URL
Content-Language
Expires
Date
Server
HTTP Status
Last-Modified
Cache-Control

We do track what languages are being presented, when this page will expire, what was the date on the server when we collected this page and what server software are they running.

HTML HEADER DEDUCTIONS

1. -1 Point - Blank lines before the content
2. -1 Point - Excessive blank lines within the content
3. -2 Points - Missing or Multiple "<!DOCTYPE" as the first line
4. -5 Points - Missing or Multiple "<TITLE>" in the header
5. -1 Point - Missing META TAG KEYWORDS
6. -1 Point - Missing META TAG DESCRIPTION
7. -10 Points - Empty "<SCRIPT></SCRIPT>" section found
8. -1 Point - <SCRIPT> Missing Type or Language attribute, per script found
9. -1 Point - for every comment in the code (outside of scripts/styles)
10. -1 Point - for Inline and Link Style Sheets
11. -1 Point - No Meta codes for ROBOTS
12. -2 Points - For <!-- comments --> in header

HTML HEADER STATISTICS

URL
Document Type
Length of Title
Language
Number of Meta tags
Words in Meta KEYWORDS
Length of Meta DESCRIPTION
Number of Style Sheets
Script Names
Number of Comments
Header Bytes (no cr/lf counted)

Menu Shortcuts Latest Company News Our Wiki/Content Management System Premium Web Hosting Service Premium Email Hosting Service Amazing Custom Web Design Service 24 Hour - Real-Time Monitoring Real-Time Directory of our site

44610 visitors. Built using 3F Wiki Engine Copyright © 2010 by 3F, LLC. All rights reserved, wordlwide.

SSL