Automatic Startup on the Online Cluster

This How-To describes the steps necessary to run a process in the online cluster in an automatic and reliable mode

A step-by-step guide by EnricoBonaccorsi

Prerequisites

• Very basic knowledge of shell scripting
• A process to start-up
• A writeable directory where you will put your startup script

Overview

There are many ways to start-up a process automatically in Linux, for example:

• you can create an init script
• you can create a xinit script
• you can add your program to /etc/rc.local

But all this methods do not allow to restart the program if it fails.

As many of you know, Domenico Galli wrote a suite of daemons called FMC. The FMC (Farm Monitoring and Control) System has been designed to monitor and control each PC of the Online Cluster (not only of the farm) of the LHCb Experiment at CERN.

One of this daemon is the Process Controller. The FMC Process Controller is a tool in charge of keeping a dynamically manageable list of applications up and running on the farm nodes.

We strongly suggest you to use this method to start your programs on the farm nodes. ("Strongly" means that you will have to convince us that you want use any other method :-))

Steps

To add your process to FMC, you should write a script (you can take inspiration from the following one), and be sure that with this script your process runs well. After you tested it, and you are happy with it, you should open a ticket via OTRS and request the automatic start-up of this script in a certain node. The System Manager on duty will add it via quattor

Remember to specify in the request:

• The path of your script
• The name of the node where you need to have this script in automatic start-up
• The user and the group which the script should run (therefore your executable as child of the script that you wrote)

Example of a script

Gareth Rogers made a great script to run the camera process on r2ecs01, you can use it as an example

 
#!/bin/bash                                                                                                                        

# This setup script is to be used with the LHCb online enviroment to
# start the CAMERA server when the computer boots.
# It is a simple script that setups the enviroment to use,
# moves to the directory containing the etc directory that contains
# the server config files and then starts the executable with
# the exec command. This allows the LHCb online scripts to maintain
# control over the server.

# Define a couple of variables that will point to the CAMERA executable
# and config files.

Panoptes_version=Panoptes_v2r2
Panoptes_release=/group/rich/sw/cmtuser/$Panoptes_version/Rich/Panoptes

# This sources the appropriate script to setup the LHCb environment.
source /group/rich/sw/scripts/setup.sh
# The official LHCb recommended mechanism for setting up a project.
# Setting up the online project gives us access to the CAMERA executable.
source `which SetupProject.sh` Online v4r20

# Change to the directory containing etc/camserv.conf
# This means the executable that we are about to run can find its config.
cd $Panoptes_release/scripts
# Start the executable.
exec -a CameraServer `which CameraServ.exe`


# Can do things this way but it relies on knowing where the install area
# is for the LHCb software. Using SetupProject as above to give us the appropriate
# environment seems more reliable.
#cmtconfig=slc4_ia32_gcc34
#Online_version=ONLINE_v4r20
#Online_install_area=/sw/lib/lhcb/ONLINE/$Online_version/InstallArea/$cmtconfig/bin
#exec -a CameraServer $Online_install_area/CameraServ.exe

Recommendations

• Remember to define your environment
• If you plan to change the executable often, try to use variables in your script: Every time that you need to change something you will need to change only the value of variable.
• One script = One process to execute (If you need more process, create more scripts)
• To start the executable use the exec command

  exec -a WhateverNameToRecognizeTheProcess FullPathOfTheExecutable

• You do not have to worry about the logs of your executable. Do not redirect any output anywhere. Your System Manager will add your program also to the FMC Message Logger, then you can analyze the logs using logViewer
• Make always a backup copy of your script when it works

Summary

1. Create your script to execute your process
2. Test your script and be sure that it work properly
3. Open a ticket via OTRS

Hint

After the process is started, if you want to change some things in your script (for example you could want to change the environment), you just have to change your script and kill the previous process, the Process Controller will restart it again immediately using the modified script.

Before do changes to your script, remember to backup it.

-- NikoNeufeld - 05 May 2009