Installing Globus 4.0

0. Notes:

Installed at wgsara1 and wgsara2 nodes. vangogh0 is the master.

Please note: This installation guide is out-of-date. It was for a beta version of Globus 4, 3.9.2 to be exact, and the layout of this HowTo is inferior to the one at Globus3Installation

# means "run as root";
% means "run as user" (typically the globus user)

Expect was often used to run a command on all nodes.

october 2004
Using Globus 3.9.2 (beta of 4.0) for Debian

Debian tricks found at http://www.dcl.hpi.uni-potsdam.de/research/grid/testbed/∞

1. Create user globus

user: globus; group: globus
# su globus


2. Installing software requirements

% su -

2.1 Installing Java

Already done. Version 1.4.2 required.
# export JAVA_HOME=/usr/local/j2sdk

[Note: not done using apt-get, deb is in bad shape]

2.2 Installing ant

Already done. Version 1.5 or higher required.
# export ANT_HOME=/usr/local/apache-ant

2.3 Installing JUnit

# apt-get install junit
% ls -l /usr/share/java/junit.jar
/usr/share/java/junit.jar -> junit-3.8.1.jar

2.4 Install Debian base packages

# apt-get install libc6-dev libgdbm-dev libdb4.2-dev


3. Set Environment

3.1 Set Global environment for all users

Added to /etc/csh.cshrc and /etc/bashrc.user :
# export JAVA_HOME=/usr/local/j2sdk
# export ANT_HOME=/usr/local/apache-ant
# export GLOBUS_LOCATION=/usr/local/gt4.0

3.2 Set user-specific environment

Note: User who want to use globus, should:
Add this line to .cshrc (or .login):
source $GLOBUS_LOCATION/etc/globus-user-env.csh
Add this line to .bashrc (or .profile):
source $GLOBUS_LOCATION/etc/globus-user-env.sh

% export CLASSPATH=.:/usr/share/java/junit.jar
Add this line to .bashrc for globus user

I added this for user globus and user freek


4. Download Globus Toolkit 4.0 package

We took the source installer:
# wget http://www-unix.globus.org/ftppub/gt3/3.9/3.9.2/gt3.9.2-wsrf-source-installer.tar.gz∞


5. Install globus toolkit

# mkdir /usr/local/gt4.0
# chown globus:globus /usr/local/gt4.0

# mkdir /space/user/globus
# chown globus:globus /space/user/globus/

% cp gt3.9.2-wsrf-source-installer.tar.gz /space/user/globus/

% tar xzf gt3.9.2-wsrf-source-installer.tar.gz
% cd gt3.9.2-wsrf-source-installer
% ./install-wsrf /usr/local/gt4.0


7 Set up trust certificates

7.1 Install DutchGrid CA certificates

Went to http://certificate.nikhef.nl/gridhelp.html∞
# wget http://certificate.nikhef.nl/medium/cacert.pem∞
# mv cacert.pem /etc/grid-security/certificates/16da7552.0
# wget http://certificate.nikhef.nl/medium/cacrl.pem∞
# mv cacert.pem /etc/grid-security/certificates/16da7552.r0
# created /etc/grid-security/certificates/16da7552.signing_policy with:
# EACL - NIKHEF medium-security X.509 authority
access_id_CA X509 '/C=NL/O=NIKHEF/CN=NIKHEF medium-security certification auth'
pos_rights globus CA:sign
cond_subjects globus '"/C=NL/O=NIKHEF/CN=NIKHEF medium-security certification auth"

7.2 Create a host certificate/key

Used the DutchGrid CA signed certificate:

Went to http://certificate.nikhef.nl/request/∞
Filled in form for vangogh0..vangogh8,
printed PDF file and filled in
modified makerequest.sh script to allow it to run on all machines
% ./makerequest.sh <hostname> <hostname>
created directories ~globus/<hostname>/ with in it the proper certificates and request

Filled in form and mailed it to ca@nikhef.nl.
Got 9 mails back, saved them in ~globus/<hostname>/ as usercert.pem
converted to unix line breaks

Copied ~globus/<hostname>/ to /etc/grid-security
made aliases:
# ln -s usercert.pem hostcert.pem
# ln -s userkey.pem hostkey.pem
Checked permissions:
# chmod 400 userkey.pem
# chmod 644 usercert.pem

Note: if we wanted to have used the UCSD certificate, we should have run:
# export GLOBUS_LOCATION=/usr/local/gt3.2
# /usr/local/gt3.2/bin/grid-cert-request -host vangogh0

7.3 Install UCSD certificate

% download globus_simple_ca_3d7853ec_setup-0.17.tar.gz to ~globus/
% $GLOBUS_LOCATION/sbin/gpt-build globus_simple_ca_3d7853ec_setup-0.17.tar.gz
% $GLOBUS_LOCATION/sbin/gpt-postinstall

# export GLOBUS_LOCATION=/usr/local/gt3.2
# /usr/local/gt3.2/setup/globus_simple_ca_3d7853ec_setup/setup-gsi

Note: do NOT use the "-default" tag behind the setup-gsi, unless that is really what you want!

7.4 Added users to grid-mapfile

Added file /etc/grid-security/grid-mapfile
with contents:
"/O=Grid/OU=OptIPuterTest/OU=simpleCA-csag-frontend-0.local/OU=local/CN=Nut Taesombut" nut
"/O=dutchgrid/O=users/O=uva/OU=wins/CN=Freek Dijkstra" freek

Distributed file over all nodes.

7.5 Change ownerships and access permissions

# /usr/local/gt3.2/bin/setperms.sh

8.1 Registed gsigatekeeper

As root, add the following line to the file /etc/services:
gsigatekeeper 2119/tcp # Globus Gatekeeper

8.2.1 Setup GRAM job manager

% /usr/local/gt3.2/setup/globus/setup-globus-gram-job-manager

8.2.2 Added gatekeeper service

Added file /etc/xinetd.d/globus-gatekeeper with contents:
service gsigatekeeper
{
}

# /etc/rc.d/init.d/xinetd restart

8.2.3 Startup MDS server

As globus user, start on all nodes:
% globus-mds start

note: should we indeed start this on all nodes, if we only want 1 information server?

9 Configure globus

9.1 DNS

Made sure that all hosts have DNS and reverse DNS.
Checked that "hostname -f" returns the correct full hostname

In the vangogh configuration files, "vangogh*.saradomain" was listed as the hostname. Replaced that with "vangogh*.uva.netherlight.nl" by running:
% perl -pi -e 's/saradomain/uva.netherlight.nl/gi' /usr/local/gt3.2/etc/*.conf
% perl -pi -e 's/saradomain/uva.netherlight.nl/gi' /usr/local/gt3.2/setup/globus/*.conf
verified replacement with
% grep -rsi saradomain /usr/local/gt3.2/*

9.2 Virtual Organisation name

For now, support only the "OptIPuter" VO:
Changed all occurances of the "site" VO (which is the default name) with "OptIPuter". Leave the "local" VO occurances intact:
Mds-Vo-name=local

% perl -pi -e 's/Mds-Vo-name=site/Mds-Vo-name=OptIPuter/g' /usr/local/gt3.2/etc/*.conf
% perl -pi -e 's/Mds-Vo-name=site/Mds-Vo-name=OptIPuter/g' /usr/local/gt3.2/setup/globus/*.conf
% grep -rsi site /usr/local/gt3.2/* | grep -i VO

9.3 Configure information server

on vangogh0,
in /usr/local/gt3.2/etc/grid-info-site-policy.conf, change the following line
policydata: (&(Mds-Service-hn=localhost.localdomain)(Mds-Service-port=2135))
to
policydata: (&(Mds-Service-hn=*)(Mds-Service-port=2135))

Note: maybe "*" could be stricter. Perhaps *.uva.netherlight.nl is better.

Substitute the following to etc/grid-info-resource-register.conf:

dn: Mds-Vo-Op-name=register, Mds-Vo-name=OptIPuter, o=grid
regtype: mdsreg2
reghn: vangogh*.uva.netherlight.nl
regport: 2135
regperiod: 600
type: ldap
hn: vangogh*.uva.netherlight.nl
port: 2135
rootdn: Mds-Vo-name=local, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 30
bindmethod: ANONYM-ONLY

dn: Mds-Vo-Op-name=register, Mds-Vo-name=OptIPuter, o=grid
regtype: mdsreg2
reghn: vangogh0.uva.netherlight.nl
regport: 2135
regperiod: 600
type: ldap
hn: vangogh*.uva.netherlight.nl
port: 2135
rootdn: Mds-Vo-name=local, o=grid
ttl: 1200
timeout: 20
mode: cachedump
cachettl: 30
bindmethod: ANONYM-ONLY

Note: for vangogh0, only one of the two was added, since they would otherwise be duplicate.

9.4 Errors

9.4.1 class not found: org.globus.ogsa.impl.core.se

No such file was found; this should probably be something in org.globus.ogsa.impl.core.services
I attributed this to a bug in the binary installer, and used the source installer instead.

9.4.2 file not found: etc/rips-service-config.xml

Error: java.io.FileNotFoundException: etc/rips-service-config.xml

% perl -pi -e 's/="etc\="\/usr\/local\/gt3.2\/etc\g' /usr/local/gt3.2/*.wsdd

9.4.3 file not found: etc/globus-host-providers.conf

not solved. must be similar though!

9.4.4 The available CRL has expired

If grid-proxy-init gives this error (only show with -verify):
% grid-proxy-init -verify -debug

Gave error:
ERROR: Couldn't verify the authenticity of the user's credential to generate a proxy from.
grid_proxy_init.c:948:globus_credential: Error verifying credential: Failed to verify credential
globus_gsi_callback_module: Could not verify credential
globus_gsi_callback_module: Could not verify credential
globus_gsi_callback_module: Invalid CRL: The available CRL has expired

Then the Ceritificate Revocation List (CRL) of the CA has expired. You should periodically renew them.
You can find the CRL's using
% ls /etc/grid-security/certificates/*.r0

You can either download CRL's manually, or install a script called edg-fetch-crl. See 12

10. Install Grid-FTP

See http://www-unix.globus.org/toolkit/docs/3.2/installation/install_config_gridftp.html∞

10.1 Add gsiftp entry
As root, add the following entry to /etc/services:
gsiftp 2811/tcp

10.2 Add xinitd entry

Create file /etc/xinetd.d/grid-ftp with content:
service gsiftp
{

instances = 1000
socket_type = stream
wait = no
user = root
env = LD_LIBRARY_PATH=/usr/local/gt3.2/lib
server = /usr/local/gt3.2/sbin/in.ftpd
server_args = -l -a -G /usr/local/gt3.2
log_on_success += DURATION
nice = 10
disable = no

}

10.3 Restart xinetd

# /etc/init.d/xinetd reload

10.4 Test

% grid-proxy-init -verify
% pico -w /tmp/file1
Create test file

% globus-url-copy gsiftp://localhost/tmp/file1 file:///tmp/file2
Gave error:
globus-url-copy: relocation error: /usr/local/gt3.2/lib/libglobus_xio_gcc32dbg.so.0: undefined symbol: gxx_personality_v0

Fixed (see 10.4.1)

% globus-url-copy gsiftp://localhost/tmp/file1 file:///tmp/file2
Gave another error:
error: an end-of-file was reached
globus_xio: An end of file occurred

The file /tmp/file2 was created but has no data in it.



10.4.1 relocation error

globus-url-copy gave error:
globus-url-copy: relocation error: /usr/local/gt3.2/lib/libglobus_xio_gcc32dbg.so.0: undefined symbol: gxx_personality_v0

Somehow, an other library compiled from source did not have this problem:
# cd /usr/local/gt3.2/lib/
# mv libglobus_xio_gcc32dbg.so.0.0.0 libglobus_xio_gcc32dbg.so.0.0.0.orig
# cp /usr/local/gt3.2src/lib/libglobus_xio_gcc32dbg.so.0.0.0 /usr/local/gt3.2/lib/
Fixed the problem.

I can only assume that because of some odd reason the version of libglobus_xio_gcc32dbg.so.0.0.0 in /usr/local/gt3.2/lib/ was incorrectly compiled.

10.4.2 End-of-file error

% globus-url-copy gsiftp://localhost/tmp/file1 file:///tmp/file2

error: an end-of-file was reached
globus_xio: An end of file occurred

The file /tmp/file2 was created but has no data in it.

The following mail gave a possible solution:
http://www-unix.globus.org/mail_archive/discuss/2004/05/msg00118.html∞


/usr/local/gt3.2/sbin/gcc32dbg/shared/in.ftpd
/usr/local/gt3.2/sbin/in.ftpd

# cp /usr/local/gt3.2src/sbin/in.ftpd /usr/local/gt3.2/sbin/
# /etc/init.d/xinetd restart

This did NOT solve the problem. :-(

11. Install RFT (state machine for GridFTP)

See http://www-unix.globus.org/toolkit/docs/3.2/installation/install_config_rft.html∞
Note that we use /var/lib/pgsql/data instead of /usr/local/pgsql/data

11.1 Install postgresql

Postgresql was already installed (see 1.4 above)


11.2 Create ogsa database

% su - postgresql
% echo $PGDATA ; already set

Note that /var/lib/pgsql/data is still empty! So we can safely call initdb.
% initdb

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale en_US.UTF-8.
fixing permissions on existing directory /var/lib/pgsql/data... ok
creating directory /var/lib/pgsql/data/base... ok
creating directory /var/lib/pgsql/data/global... ok
creating directory /var/lib/pgsql/data/pg_xlog... ok
creating directory /var/lib/pgsql/data/pg_clog... ok
selecting default max_connections... 100
selecting default shared_buffers... 1000
creating configuration files... ok
creating template1 database in /var/lib/pgsql/data/base/1... ok
initializing pg_shadow... ok
enabling unlimited row size for system tables... ok
initializing pg_depend... ok
creating system views... ok
loading pg_description... ok
creating conversions... ok
setting privileges on built-in objects... ok
creating information schema... ok
vacuuming database template1... ok
copying template1 to template0... ok
Success. You can now start the database server using:

/usr/bin/postmaster -D /var/lib/pgsql/data

or

/usr/bin/pg_ctl -D /var/lib/pgsql/data -l logfile start

% createdb ogsa

Gives error:
createdb: relocation error: createdb: undefined symbol: get_progname

/usr/local/gt3.2/share/multirft/rft_schema_ogsa.sql

12. Install edg-fetch-crl

edg-fetch-crl is a script to periodically fetch the Certifacte Revocation Lists.

You can find it at
http://datagrid.in2p3.fr/cvsweb/edg-utils/cert/∞
http://datagrid.in2p3.fr/distribution/autobuild/i386-rh7.3/wp6/RPMS/edg-utils-system-1.6.1-1.noarch.rpm∞

Note that edg-crl-upgrade is a script to call edg-fetch-crl with default options,
and that edg-gridmapfile-upgrade is a script to update the gridmapfile based on a
LDAP server with current users. This latter script is not used (also the LDAP-based
architecture will be replaced by a VOMS-based architecture)

I downloaded the RPM:
% wget http://datagrid.in2p3.fr/distribution/autobuild/i386-rh7.3/wp6/RPMS/edg-utils-system-1.6.1-1.noarch.rpm∞
% rpm -i --prefix=/tmp/rpm edg-utils-system-1.6.1-1.noarch.rpm
% su -
# cp /tmp/rpm/sbin/edg-fetch-crl /usr/local/sbin/

Now I still need to make a configuration file for each CA I want to renew the CRL for.
For example, to renew the Dutchgrid CA, I have this file:
echo "http://certificate.nikhef.nl/medium/cacrl.pem∞" > /etc/grid-security/certificates/16da7552.crl_url

Now I need to run this command to renew all CRLs:
# edg-fetch-crl -o /etc/grid-security/certificates/

13. Install on Debian

Installing Globus 3.2 on Debian system
22 september 2004
Jeroen van der Ham

I installed debian on our test machine and installed globus on it. Didn't
really encounter big problems and I've included the recipe for it. This is
probably not completely correct since I didn't document everything I did, but
it's not missing all that much I think.


Installing Globus Toolkit 3.2.1 on Debian:

Installing J2SDK: (Or use kaffe, dunno if it works)

Copy j2sdk to /usr/local/j2sdk
ln -s /etc/alternatives/java /usr/local/bin/java
sudo update-alternatives --install java java /usr/local/j2sdk/bin/java 400
sudo update-alternatives --auto java
sudo update-alternatives --display java (to see that it worked)

Installing Ant:

Copy apache-ant to /usr/local/apache-ant
/usr/local/bin/# ln -s ../apache-ant/bin/ant ant

Installing JUnit:

apt-get install junit
ln -s /usr/share/java/junit.jar /usr/local/apache-ant/lib/junit.jar

Installing C-compiler:

sudo apt-get install gcc

Installing YACC (or Bison):

sudo apt-get install bison

Installing GNU tar: done


Things not listed by globus:

sudo apt-get install make
sudo apt-get install libc6-dev

Installing Globus Toolkit 3.2.1:

add user globus:globus
mkdir /usr/local/gt3.2.1/
chown globus:globus /usr/local/gt3.2.1/
sudo su - globus
export JAVA_HOME="/usr/local/j2sdk/" ANT_HOME="/usr/local/apache-ant/"
cd /home/globus/gt3.2.1-all-source-installer/
./install-gt3 /usr/local/gt3.2.1/
./install-gt3-mmjfs /usr/local/gt3.2.1/

After that, follow:
http://www-unix.globus.org/toolkit/docs/3.2/installation/install_config_req.html#requirements∞



NOT TO FORGET:

set /etc/service to correct port
set /etc/hosts.allow with name of service (not port number!)



Errors while connecting to Globus 3.9.2 server:

FROM 3.92 CLIENT:

debug: reading into data buffer 0x4108f008, maximum length 1048576
debug: data callback, error an authorization operation failed, buffer 0x4108f008, length 0, offset=0, eof=true
debug: response from gsiftp://wgsara2.uva.netherlight.nl:2323/tmp/abc1:
500-Command failed. : globus_xio_gsi: gss_accept_sec_context failed.
500-globus_gsi_gssapi: Error during delegation: Delegation protocol violation
500 End.

debug: operation complete

(This is what the client said)
error: an authorization operation failed
globus_gsi_gssapi: Authorization denied: The name of the remote entity (/O=dutchgrid/O=hosts/OU=netherlight.nl/CN=wgsara2.uva.netherlight.nl), and the expected name for the remote entity (/O=dutchgrid/O=users/O=uva/OU=wins/CN=Freek Dijkstra) do not match
(server kept silent)


FROM 3.2 CLIENT:

(This is what the server said)
500-Command failed. : an authorization operation failed
500-globus_xio_gsi: The peer authenticated to /O=dutchgrid/O=users/O=uva/OU=wins/CN=Freek Dijkstra. Expected the peer to authenticate as /O=dutchgrid/O=hosts/OU=netherlight.nl/CN=wgsara2.uva.netherlight.nl
500 End.