Config (egothor2 4.0.5-SNAPSHOT API)

java.lang.Object
- org.egothor.robot.Config

```
public class Config
extends Object
```
The main configuration object of the robot

Author:

Leo Galambos

Nested Class Summary

Nested Classes
Modifier and Type Class and Description

static class Config.TypeURIRepository

Nested Classes
Modifier and Type	Class and Description
`static class`	`Config.TypeURIRepository`

Field Summary

Fields
Modifier and Type	Field and Description
`static int`	`BACKUPRESPONSES` Backup all in-content-table-specified documents into a gzipped stream?
`static int`	`BUCKETSCACHE` Number of cached `Table` objects.
`static int`	`BUCKETSIZE` Size of buckets (in bytes).
`static int`	`CAPACITY` How many servers will be kept in memory for processing (implies up to 4 IO handles for each).
`static boolean`	`DEBUG` Print more details about robot work into logs.
`static int`	`DNSRETRIES` How many times we restart DNS query?
`static String`	`DNSSERVER` IP address of our DNS server.
`static int`	`DNSTIMEOUT` How long we wait for DNS response.
`static long`	`DOCDELAY` How often we may ask for the same URI? [msec]
`static long`	`DOCDELAYONFAIL` How often we may ask for the same URI if the request failed? [msec]
`static long`	`DOCINITDELAY` How often we may ask for the same URI on start (the first two fetches)? [msec]
`static int`	`FETCHQSIZE` Number of ready servers for gathering (number of pages gathered concurrently).
`static int`	`IMGBUCKETSCACHE` Number of cached `Table` objects when storing IMG-SRC URIs.
`static int`	`IMGNODESCACHE` An internal cache which is used in `Directory` when storing IMG-SRC URIs.
`static int`	`IMGSCHUNKLEN` Chunk length of `IMGSLISTCHUNKFILENAME` and `IMGSSTRUCTCHUNKFILENAME` reporters.
`static String`	`IMGSLISTCHUNKFILENAME` Filename of log file with all new IMG URIs accepted.
`static String`	`IMGSSTRUCTCHUNKFILENAME` Filename of log file with the occurencies of images on pages.
`static int`	`IMGURISCACHE` An internal cache which is used in `Table` when storing IMG-SRC URIs.
`static int`	`INDEXCHUNK` How large chunks are created by the indexer.
`static int`	`IP_PAUSE` Delay between two connections to the same IP (msec).
`static int`	`IP_VIRTUAL` Number of requests to one IP concurrently (in case of virtual hosts).
`static int`	`MAXCONNECTIONSINSEQ` One server may be stay in the processing queue up to this value of connections, then is must give a chance to others.
`static int`	`MAXFAILURESINBATCH` How many times do we allow to restart DNS or HTTP requests during a batch (sequence)?
`static int`	`MAXIMUSTOTALUS` What's the maximum size of a document we accept?
`static int`	`MAXITEMSINQUEUE` If a queue has more than this number of URIs in a queue, then new URIs are not planned.
`static long`	`MAXPAGES` Initial number of URIs we will collect.
`static long`	`MAXSERVERS` Initial number of servers robot will scan.
`static int`	`MAXURILEN` Max length of a URI.
`static int`	`MAXURISFORBLINDHASH` Size of a hash table (bitmap) for all URIs in the system.
`static int`	`NEWURISCACHE` An internal cache which is used in `LinksCollector`.
`static int`	`NODESCACHE` An internal cache which is used in `Directory`.
`static int`	`POSTQUEUESIZE` How many `Response` objects are in post-process queues.
`static int`	`PREFSORTED` Number of items sorted preferably.
`static int`	`RESOLVEQSIZE` Number of ready servers for resolving (number of concurrent request our DNS server can handle).
`static int`	`RESPONSESIZE` What's the maximum size of a defragmented response we accept?
`static String`	`ROBOTID` Identification string of our robot.
`static long`	`ROBOTSTXT` How old can the robots.txt specification be (in milliseconds)?
`static int`	`SAVEINTERVAL` New URIs are saved at these intervals (msec) when possible.
`static boolean`	`SCRUB` Scrub all URI to a standard format?
`static long`	`SRVEMPTY` How long is a server parked when its queue is empty?
`static long`	`SRVIPLOCKED` How long is a server parked when its IP is locked currently?
`static long`	`SRVRECYCLE` How long is a server parked when it is exhaused?
`static long`	`SRVTCPERROR` How long is a server parked when it is unreachable?
`static long`	`SRVUNRESOLVED` How long is a server parked when it cannot be resolved?
`static String`	`STATCHUNKFILENAME` Filename of log file with statistics values of pages processed successfully.
`static int`	`STATCHUNKLEN` Chunk length of `STATCHUNKFILENAME` reporter.
`static int`	`TRANSMITTERPORT` UDP SAX events transmitter port.
`static int`	`TURNUPWHEEL` How many requests are parsed in synchronous mode.
`static Config.TypeURIRepository`	`URIREPOSITORY` What URI repository is used inside `T0` to assign ids to URIs.
`static int`	`URISCACHE` An internal cache which is used in `Table` or `BijectInt2StringAppender`.
`static String`	`URISCHUNKFILENAME` Filename of log file with all new URIs accepted.
`static int`	`URISCHUNKLEN` Chunk length of `URISCHUNKFILENAME` reporter.
`static String`	`USERAGENT` User agent string of our robot.
`static String`	`VR` Version number of the robot.
`static int`	`WWWRETRIES` How many times we restart HTTP connection?
`static int`	`WWWTIMEOUT` How long we wait for data.

Constructor Summary

Constructors
Constructor and Description

Config()
This is an empty constructor - used by test scripts only.

Constructors
Constructor and Description
`Config()` This is an empty constructor - used by test scripts only.

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static Escape`	`acceptToDownload(String contentType, String contentLength)` Tests whether we have some interest to download a document of a given content-type and suggested content-length (both are HTTP headers sent to us).
`static boolean`	`allowedPage(String name)`
`static boolean`	`allowedServer(String name)`
`static boolean`	`allowedToBackup(String contentType)`
`static String`	`explain(URI uri)`
`void`	`exportTo(DataOutputStream dos)` Save this configuration into an output stream.
`boolean`	`hasSomeBackupConditions()` Return true if this configuration defines some rules for backup.
`ArrayList<String>`	`initialize(String filename)` Load a configuration from a file.
`URI`	`normalize(URI uri)` This function first normalizes the URI and then tests whether such a URI is valid with a configuration specified by this class.
`void`	`shutdown()` Close any resources this object may held.

Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - DEBUG
```
public static boolean DEBUG
```
    Print more details about robot work into logs. This option is for programmers only.
  - USERAGENT
```
public static String USERAGENT
```
    User agent string of our robot.
  - ROBOTID
```
public static String ROBOTID
```
    Identification string of our robot.
  - VR
```
public static final String VR
```
    Version number of the robot.
    
    See Also:
    
    Constant Field Values
  - DNSSERVER
```
public static String DNSSERVER
```
    IP address of our DNS server.
  - RESPONSESIZE
```
public static final int RESPONSESIZE
```
    What's the maximum size of a defragmented response we accept?
    
    See Also:
    
    Constant Field Values
  - MAXIMUSTOTALUS
```
public static int MAXIMUSTOTALUS
```
    What's the maximum size of a document we accept?
  - IP_PAUSE
```
public static int IP_PAUSE
```
    Delay between two connections to the same IP (msec).
  - IP_VIRTUAL
```
public static int IP_VIRTUAL
```
    Number of requests to one IP concurrently (in case of virtual hosts).
  - MAXURILEN
```
public static int MAXURILEN
```
    Max length of a URI.
  - CAPACITY
```
public static int CAPACITY
```
    How many servers will be kept in memory for processing (implies up to 4 IO handles for each).
  - RESOLVEQSIZE
```
public static int RESOLVEQSIZE
```
    Number of ready servers for resolving (number of concurrent request our DNS server can handle).
  - FETCHQSIZE
```
public static int FETCHQSIZE
```
    Number of ready servers for gathering (number of pages gathered concurrently).
  - DNSTIMEOUT
```
public static int DNSTIMEOUT
```
    How long we wait for DNS response.
  - WWWTIMEOUT
```
public static int WWWTIMEOUT
```
    How long we wait for data.
  - DNSRETRIES
```
public static int DNSRETRIES
```
    How many times we restart DNS query?
  - WWWRETRIES
```
public static int WWWRETRIES
```
    How many times we restart HTTP connection?
  - MAXFAILURESINBATCH
```
public static int MAXFAILURESINBATCH
```
    How many times do we allow to restart DNS or HTTP requests during a batch (sequence)?
    
    See Also:
    
    MAXCONNECTIONSINSEQ
  - ROBOTSTXT
```
public static long ROBOTSTXT
```
    How old can the robots.txt specification be (in milliseconds)?
  - DOCDELAY
```
public static long DOCDELAY
```
    How often we may ask for the same URI? [msec]
  - DOCINITDELAY
```
public static long DOCINITDELAY
```
    How often we may ask for the same URI on start (the first two fetches)? [msec]
  - DOCDELAYONFAIL
```
public static long DOCDELAYONFAIL
```
    How often we may ask for the same URI if the request failed? [msec]
  - SRVEMPTY
```
public static long SRVEMPTY
```
    How long is a server parked when its queue is empty?
  - SRVUNRESOLVED
```
public static long SRVUNRESOLVED
```
    How long is a server parked when it cannot be resolved?
  - SRVIPLOCKED
```
public static long SRVIPLOCKED
```
    How long is a server parked when its IP is locked currently?
  - SRVTCPERROR
```
public static long SRVTCPERROR
```
    How long is a server parked when it is unreachable?
  - SRVRECYCLE
```
public static long SRVRECYCLE
```
    How long is a server parked when it is exhaused?
    
    See Also:
    
    MAXCONNECTIONSINSEQ
  - NEWURISCACHE
```
public static int NEWURISCACHE
```
    An internal cache which is used in LinksCollector.
  - BUCKETSCACHE
```
public static int BUCKETSCACHE
```
    Number of cached Table objects.
  - URISCACHE
```
public static int URISCACHE
```
    An internal cache which is used in Table or BijectInt2StringAppender.
  - NODESCACHE
```
public static int NODESCACHE
```
    An internal cache which is used in Directory.
  - IMGBUCKETSCACHE
```
public static int IMGBUCKETSCACHE
```
    Number of cached Table objects when storing IMG-SRC URIs.
    
    See Also:
    
    ImageLinksExtractor
  - IMGURISCACHE
```
public static int IMGURISCACHE
```
    An internal cache which is used in Table when storing IMG-SRC URIs.
    
    See Also:
    
    ImageLinksExtractor
  - IMGNODESCACHE
```
public static int IMGNODESCACHE
```
    An internal cache which is used in Directory when storing IMG-SRC URIs.
    
    See Also:
    
    ImageLinksExtractor
  - SAVEINTERVAL
```
public static int SAVEINTERVAL
```
    New URIs are saved at these intervals (msec) when possible.
  - URISCHUNKFILENAME
```
public static String URISCHUNKFILENAME
```
    Filename of log file with all new URIs accepted.
    
    See Also:
    
    T0, Reporter
  - URISCHUNKLEN
```
public static int URISCHUNKLEN
```
    Chunk length of URISCHUNKFILENAME reporter. Valid values are:
    - less than 0 = the reporter is not created at all
    - 0 = the reporter is never rotated
    - above 0 = maximum number of records in a chunk (x100,000)
    See Also:
    
    T0
  - STATCHUNKFILENAME
```
public static String STATCHUNKFILENAME
```
    Filename of log file with statistics values of pages processed successfully. The filename is rooted into t5 subdirectory.
    
    See Also:
    
    T5, Reporter
  - STATCHUNKLEN
```
public static int STATCHUNKLEN
```
    Chunk length of STATCHUNKFILENAME reporter. Valid values are:
    - less than 0 = the reporter is not created at all
    - 0 = the reporter is never rotated
    - above 0 = maximum number of records in a chunk (x100,000)
    See Also:
    
    T5
  - IMGSLISTCHUNKFILENAME
```
public static String IMGSLISTCHUNKFILENAME
```
    Filename of log file with all new IMG URIs accepted.
    
    See Also:
    
    ImageLinksExtractor, Reporter
  - IMGSCHUNKLEN
```
public static int IMGSCHUNKLEN
```
    Chunk length of IMGSLISTCHUNKFILENAME and IMGSSTRUCTCHUNKFILENAME reporters. Valid values are:
    - less than 0 = the reporter is not created at all
    - 0 = the reporter is never rotated
    - above 0 = maximum number of records in a chunk (x100,000)
    See Also:
    
    ImageLinksExtractor, Response
  - IMGSSTRUCTCHUNKFILENAME
```
public static String IMGSSTRUCTCHUNKFILENAME
```
    Filename of log file with the occurencies of images on pages.
    
    See Also:
    
    ImageLinksExtractor, Reporter
  - MAXSERVERS
```
public static long MAXSERVERS
```
    Initial number of servers robot will scan.
  - MAXPAGES
```
public static long MAXPAGES
```
    Initial number of URIs we will collect.
  - MAXCONNECTIONSINSEQ
```
public static int MAXCONNECTIONSINSEQ
```
    One server may be stay in the processing queue up to this value of connections, then is must give a chance to others.
  - BUCKETSIZE
```
public static int BUCKETSIZE
```
    Size of buckets (in bytes).
    
    See Also:
    
    Bucket
  - INDEXCHUNK
```
public static int INDEXCHUNK
```
    How large chunks are created by the indexer.
  - POSTQUEUESIZE
```
public static int POSTQUEUESIZE
```
    How many Response objects are in post-process queues.
  - TURNUPWHEEL
```
public static int TURNUPWHEEL
```
    How many requests are parsed in synchronous mode. If this value is too low, then the robot may start the crawling process slowly because it will require new links before they are really parsed from the previous documents.
    
    See Also:
    
    T5
  - BACKUPRESPONSES
```
public static int BACKUPRESPONSES
```
    Backup all in-content-table-specified documents into a gzipped stream?
    
    See Also:
    
    StorageEscape
  - MAXURISFORBLINDHASH
```
public static int MAXURISFORBLINDHASH
```
    Size of a hash table (bitmap) for all URIs in the system.
    
    See Also:
    
    FastBlindAppender, T0
  - URIREPOSITORY
```
public static Config.TypeURIRepository URIREPOSITORY
```
    What URI repository is used inside T0 to assign ids to URIs.
  - SCRUB
```
public static boolean SCRUB
```
    Scrub all URI to a standard format?
    
    See Also:
    
    Scrub
  - PREFSORTED
```
public static int PREFSORTED
```
    Number of items sorted preferably.
    
    See Also:
    
    SequentialQueue
  - MAXITEMSINQUEUE
```
public static int MAXITEMSINQUEUE
```
    If a queue has more than this number of URIs in a queue, then new URIs are not planned.
    
    See Also:
    
    PageSequentialQueue
  - TRANSMITTERPORT
```
public static int TRANSMITTERPORT
```
    UDP SAX events transmitter port. Must be greater than 1024, else the feature is not used.
    
    See Also:
    
    Transmitter
- Constructor Detail
  - Config
```
public Config()
```
    This is an empty constructor - used by test scripts only.
- Method Detail
  - exportTo
```
public void exportTo(DataOutputStream dos)
              throws IOException
```
    Save this configuration into an output stream.
    
    Parameters:
    
    dos - the output stream
    
    Throws:
    
    IOException - when the data cannot be written
  - initialize
```
public ArrayList<String> initialize(String filename)
                             throws IOException
```
    Load a configuration from a file. Be careful, if the rules related to URI rewriting were invalid and buggy, they could lead to unnormalized URIs.
    
    Parameters:
    
    filename - the filename
    
    Returns:
    
    Throws:
    
    IOException - on I/O error
    
    See Also:
    
    normalize(java.net.URI)
  - normalize
```
public URI normalize(URI uri)
```
    This function first normalizes the URI and then tests whether such a URI is valid with a configuration specified by this class.
    
    Parameters:
    
    uri - the entry URI
    
    Returns:
    
    null if and only if the URI cannot be normalized (i.e. http://some.where/../../) or it does not pass the rules of this class
    
    See Also:
    
    valid(java.net.URI)
  - explain
```
public static String explain(URI uri)
```
  - acceptToDownload
```
public static Escape acceptToDownload(String contentType,
                                      String contentLength)
```
    Tests whether we have some interest to download a document of a given content-type and suggested content-length (both are HTTP headers sent to us).
    
    Parameters:
    
    contentType -
    
    contentLength -
    
    Returns:
  - allowedServer
```
public static boolean allowedServer(String name)
```
    Parameters:
    
    name -
    
    Returns:
  - allowedPage
```
public static boolean allowedPage(String name)
```
    Parameters:
    
    name -
    
    Returns:
  - allowedToBackup
```
public static boolean allowedToBackup(String contentType)
```
    Parameters:
    
    contentType -
    
    Returns:
  - hasSomeBackupConditions
```
public boolean hasSomeBackupConditions()
```
    Return true if this configuration defines some rules for backup.
  - shutdown
```
public void shutdown()
```
    Close any resources this object may held.

Class Config

Nested Class Summary

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

DEBUG

USERAGENT

ROBOTID

VR

DNSSERVER

RESPONSESIZE

MAXIMUSTOTALUS

IP_PAUSE

IP_VIRTUAL

MAXURILEN

CAPACITY

RESOLVEQSIZE

FETCHQSIZE

DNSTIMEOUT

WWWTIMEOUT

DNSRETRIES

WWWRETRIES

MAXFAILURESINBATCH

ROBOTSTXT

DOCDELAY

DOCINITDELAY

DOCDELAYONFAIL

SRVEMPTY

SRVUNRESOLVED

SRVIPLOCKED

SRVTCPERROR

SRVRECYCLE

NEWURISCACHE

BUCKETSCACHE

URISCACHE

NODESCACHE

IMGBUCKETSCACHE

IMGURISCACHE

IMGNODESCACHE

SAVEINTERVAL

URISCHUNKFILENAME

URISCHUNKLEN

STATCHUNKFILENAME

STATCHUNKLEN

IMGSLISTCHUNKFILENAME

IMGSCHUNKLEN

IMGSSTRUCTCHUNKFILENAME

MAXSERVERS

MAXPAGES

MAXCONNECTIONSINSEQ

BUCKETSIZE

INDEXCHUNK

POSTQUEUESIZE

TURNUPWHEEL

BACKUPRESPONSES

MAXURISFORBLINDHASH

URIREPOSITORY

SCRUB

PREFSORTED

MAXITEMSINQUEUE

TRANSMITTERPORT

Constructor Detail

Config

Method Detail

exportTo

initialize

normalize

explain

acceptToDownload

allowedServer

allowedPage

allowedToBackup

hasSomeBackupConditions

shutdown