Monitoring and Restricting Access on Web Pages

The log files

The HTTP daemons at FNWI are configured to keep both an access log file and an error log file. The files are accessible for all users with an account on the machines. The logs are here:

/home/www/WWW/httpd/logs/access_log
/home/www/WWW/httpd/logs/error_log

In the same directory, log files for the last two months are kept in gzip format.

The format of the access log file

hostname - - [date string] "request" status bytes_sent

hostname

The name of the connecting client. If there are is an IP number here, the nameserver had a problem figuring out the name.

date string

For example: 08/Nov/1994:16:34:53 +0100

request

The request as passed to the server. It usually has the text GET followed by a space, a document name, another space and the string HTTP/1.0 or HTTP/0.9. The HEAD request is also supported. The PUT request is not.

status

Possible values:

200   DOCUMENT_FOLLOWS
302   REDIRECT
304   USE_LOCAL_COPY
400   BAD_REQUEST
401   AUTH_REQUIRED
403   FORBIDDEN
404   NOT_FOUND
500   SERVER_ERROR
501   NOT_IMPLEMENTED
6992  NO_MEMORY		please report

bytes_sent

Bytes sent, excluding the bytes sent for the mime type.

For example:

zijde.science.uva.nl - - [08/Nov/1994:16:34:53 +0100] "GET /pict/uvaheader.gif HTTP/1.0" 200 4252

Note:

Currently reverse name lookup using IDENT is disabled. A name lookup per connection introduces transfer delays that are too large for comfort.

The format of the error log file

[date string] error

text preceding this error

CGI scripts often produce errors on standard out. This error output is appended to the error log. It is always followed by an error message from the HTTP daemon (httpd).

date string

For example: Tue Nov 8 15:11:07 1994

error

There is no set syntax for the error. Also, it is often unclear what caused the error. For example:

httpd: malformed header from script

What script did that? Combined with the access log file, it could turn out to be a CGI script that didn't produce any output at all, not even a mime type, due to some error.

Publicly available access logs

The awstats script is run each hour on the complete two-month access_log on all the servers. You can view the access logs as a web page .

Restricting access

Access to directories can be restricted using the Limit directive in a .htaccess file. The restrictions or capabilities set in the .htaccess file also affect all the subdirectories. They can, of course, be overridden again with other .htaccess files.

Please read this entire section, before asking questions. More documentation can be found at NCSA.

Directives

AllowOverride controls...

Defines what may be changed in the subdirectories. It does not restrict what permissions may be set in this access file.

• None - Nothing may be overridden. Other .htaccess files cannot contain relevant settings. All subdirectories have exactly the same access permissions as this directory.
• Options - Permit Options directive.
• FileInfo - Permit use of the AddType and AddEncoding directives.
• Indexes - Permits changes in automatic generation parameters of directory indexes, such as AddDescription.
• AuthConfig - Permit use of the AuthName, AuthType, AuthUserFile, and AuthGroupFile directives.
• Limit - Permit use of the Limit directive.
• All - Permit anything to be overridden.

Options controls...

Set permissions for retrieval by the HTTP daemon.

• None - Permit normal file retrieval only.
• Indexes - Permit automatic generation of directory indexes. This is done when the default HTML file is not present.
• Includes - Permit server side include files.
• IncludesNoExec - Permit server side include files, but exec is disabled.
• FollowSymLinks - Permit the server to follow symbolic links.
• SymLinksIfOwnerMatch - Permit the server to follow symbolic links when the owner of the link is the same as the owner of the file is points to.
• ExecCGI - Permit execution of CGI scripts.
• All - Permit everything. Almost never required.

<Limit GET> controls... </Limit>

Defines which hosts may connect.

• order - set the evaluation order of allow and deny.
  deny,allow - deny first
  allow,deny - allow first
  mutual-failure - same as deny,allow followed by: deny from all
• deny from host1 host2 ... hostn
• allow from host1 host2 ... hostn
• require entity en1 en2 ... enn
  entity can be: user, group, and valid-user
  en are entity names
• Example: (dir: /*/public_html)

        AllowOverride Limit AuthConfig FileInfo Indexes
        Options Indexes FollowSymLinks IncludesNoExec
        <Limit GET>
        order allow,deny
        allow from all
        </Limit>

• To just limit access to the FNWI domain, copy the following to the .htaccess file in your directory

        order deny,allow
        deny from all
        allow from .science.uva.nl
        

AddType type/subtype extension

Example: AddType text/plain doc

AddEncoding type extension

Example: AddEncoding x-gzip gz

AddDescription "description" filename

Example: AddDescription "My dog Fido" fido.gif

AuthName name

Example: AuthName PhysicsCollab

AuthType type

Example: AuthType Basic

AuthUserFile path

Example: AuthUserFile /usr/local/etc/httpd/conf/.htpasswd

AuthGroupFile path

Example: AuthGroupFile /usr/local/etc/httpd/conf/.htgroup