Problem

How can I exclude session cookies from URL in the robot?

Solution

Use this "rules" file:

user-agent      +http://www.egothor.org/nonexistentpage.html
loop            2

valid           http://www\.egothor\.org(:\d+)?/.*
replace         (\?|&)\w+=[^&\?]{8,}($|&)      <EMPTY>
replace         ;\w+=[^&\?]{8,}(\?|$)          <EMPTY>
replace         &&                             <EMPTY>
replace         \?&                            \?
replace         \?$                            <EMPTY>

-- LeoGalambos - 04 Apr 2004