Building a coldfusion based spider - Crash course

So you are interested in building a spider in coldfusion. Well i will try to explain the basic principles of building your own spider. First we will have to access data on the web. Below a simple example of downloading data through the cfhttp tag.

<cfhttp method="get" timeout="30" redirect="no" getasbinary="yes" url="http://coldfusion9.blogger.com" charset="utf-8" userAgent="Mozilla/5.0 (Windows; U; Windows NT 6.1; nl; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15">
 <cfhttpparam type="header" name="Accept-Encoding" Value="no-compression"> 
 <cfhttpparam type="header" name="Cache-Control" value="no-cache"> 
</cfhttp>

You might have noticed that i set the getasbinary argument to yes. The reason for this is to be able to convert ISO-8859-1 (or any other character set) to utf-8. One (of the many) problems you will encouter is determining which character set the spidered page is using. In many cases the cfhttp tag won't return the proper character set. The code below finds the proper character set and if necessary converts the content to utf-8.

<cfif cfhttp.errorDetail eq ''>
 <cfset mycharset = "">
 <cfset mycharset = #cfhttp.Charset#>
 <cfset filec = #ToString(cfhttp.fileContent)#>
 <cfif len(mycharset) eq 0>
  <cfset mycharset = "ISO-8859-1">
  <cfset pat = "(?i)<META(.*)charset=\s*([^\s|^""|^']*)"> 
  <cfif #refindnocase(pat,filec)# gt 0>
   <cfset local_re = #refindnocase(pat,filec,1,true)#>
   <cfif local_re.len[3] gt 0>
    <cfset mycharset = #Mid(filec,local_re.pos[3],local_re.len[3])#>
   <cfelse>
    <cfset mycharset = #Mid(filec,local_re.pos[2],local_re.len[2])#>
    <cfset mycharset = trim(rereplacenocase(mycharset,'.*charset=([^>^' & "'" & '^"]+).*','\1','one'))>
   </cfif>
  </cfif>
 </cfif>
 <cfif mycharset neq 'utf-8'>
  <cftry>
   <cfset filec = CharsetEncode(cfhttp.fileContent, mycharset)>
   <cfset filec = #ToString(filec)#>
   <cfcatch type="any">
    <cfset filec = #ToString(cfhttp.fileContent)#>
   </cfcatch>
  </cftry>
 </cfif>
</cfif>

Okay we have downloaded content from an url and converted this content to utf-8. The next step is applying patterns to the content. This is where the fun begins. As of coldfusion version 8 you can use the REMatch function to find and extract patterns. However i will advice you not to use this tag because is has some limitations. Use the following function instead:

<cfset objPattern = CreateObject("java","java.util.regex.Pattern").Compile('#yourpattern#') />
<cfset objMatcher = objPattern.Matcher(filec) />
<cfloop condition="objMatcher.Find()">
#objMatcher.Group(1)#
</cfloop>

Now you have learned some basic principles about spidering and extracting data from an webpage. If you like this information and like to know more about building a web spider in coldfusion please leave a comment.

Convert a relative url to an absolute url in coldfusion

To my surprise there is not a default function in coldfusion that converts relative urls to absolute ones. So i created one. This function relies heavily on regular expressions. If you are not familiar with regular expression i suggest you do not change any of the functions that use it (like rereplace and refind). So use the function simply input the relative url and the absolute url the relative url comes from. E.g.
Relative url = "/example.html"
Absolute url = "http://www.example.com/a/b/c/index.html"
Result = http://www.example.com/example.html

Example call function

#regex_UrlRelToAbs('/example.html','http://www.example.com/a/b/c/index.html')#

Complete function

<cffunction name="regex_UrlRelToAbs">
 <cfargument name="RelativePath">
 <cfargument name="AbsolutePath">
 <cfset var tempstruct = structnew()>
 <cfset var AbsolutePath_arr = ''>

 <cfset RelativePath=ReplaceNoCase(RelativePath,"&apos;","'","ALL")>
 <cfset RelativePath=ReplaceNoCase(RelativePath,"&quot;","""","ALL")>
 <cfset RelativePath=ReplaceNoCase(RelativePath,"&lt;","<","ALL")>
 <cfset RelativePath=ReplaceNoCase(RelativePath,"&gt;",">","ALL")>
 <cfset RelativePath=ReplaceNoCase(RelativePath,"&amp;","&","ALL")>


 <cfif isvalid('url',RelativePath)><cfreturn RelativePath></cfif>
 <cfif refind('^\/\/',RelativePath)>
  <cfset RelativePath = rereplace(RelativePath,'^\/\/','','one')>
  <cfreturn "http://#RelativePath#">
 <cfif>

 <cfset tempstruct.domain =  reReplace(AbsolutePath,"(^\w+://)([^\/:]+)[\w\W]*$","\1\2","one") />

 <cfset AbsolutePath_arr = #refind("^((http[s]?|ftp):\/)?\/?([^:\/\s]+)((\/\w+)*\/)([\w\-\.]+[^##?\s]+)(.+)?(##[\w\-]+)?$",AbsolutePath,1,true)#>
 <cftry>
  <cfif AbsolutePath_arr.len[5] gt 0>
   <cfset tempstruct.folders =  #trim(mid(AbsolutePath,AbsolutePath_arr.pos[5],AbsolutePath_arr.len[5]))#>
  <cfelse>
   <cfset tempstruct.folders =  ''>
  </cfif>
  <cfcatch type="any">
   <cfset tempstruct.folders =  ''>
  </cfcatch>
 </cftry> 

 <cfif left(RelativePath,1) eq '/'>
  <cfreturn "#tempstruct.domain##RelativePath#">
 <cfelseif left(RelativePath,1) neq '.'>
  <cfreturn "#tempstruct.domain##tempstruct.folders##RelativePath#">
 <cfelse>
  <cfset regex_UrlProperties_oc = #arraylen(rematch('(\.\./|\./)',RelativePath))#>
  <cfset regex_UrlProperties_newF = ''>
  <cfloop from=1 to=#listlen(tempstruct.folders,'/')-regex_UrlProperties_oc# index="i"><cfset regex_UrlProperties_newF = "#regex_UrlProperties_newF##listgetAt(tempstruct.folders,i,'/')#/"></cfloop>
  <cfreturn "#tempstruct.domain#/#regex_UrlProperties_newF##rereplace(RelativePath,'(\.\./|\./)','','all')#">
 </cfif>
</cffunction>