Leading spider technology /

SiteRay's own spidering technology was designed specifically to discover pages that users would find, even where search engines like Google cannot.

Flash

SiteRay understands Flash animations, and can read the links and content inside them. If content is accessible only via a Flash animation, SiteRay can still find it, and highlight it as inaccessible.

Javascript

Many badly designed websites only work when Javascript is enabled. SiteRay can interpret and discover links within Javascript on a page, including those in dynamic HTML generation.

Cookies

Built-in intelligence analyses the use of cookies, and will automatically include or exclude those that contribute to the significance of a site. For example, cookies that must be set to discover new pages, or that change the language will be explored, while tracking cookies are discarded. This allows SiteRay to test an otherwise potentially infinite number of cookies and return a practical list of pages.

Forms, logins, manual steps

Submit any form on a website, or perform any other manual step you wish (set a cookie, POST to a URL) at any time during the spidering process. This allows you to train SiteRay to go through the stages of a website that otherwise render content inaccessible, such as logging in, or registering via an online form.

Complex & infinite URLs

Many websites use web addresses that include a potentially infinite number of variations, only a fraction of which are significant. For example, an online calendar can display an infinite number of pages (one for each day, month, year etc). Some websites complicate this further by adding a unique tracking identifier for each person who visits the site.

SiteRay uses built-in intelligence to detect and circumvent these scenarios automatically. Variations in query parameters are identified and resulting pages compared for significance. Technical users can choose to override these settings and forcefully include or exclude sections if they wish.