Tags:
create new tag
, view all tags

 

How to Recover an unresponsive system

First, note that the system will not respond to FSM commands if any of the checked parameters (foil temperatues, BCMs, vacuum) are out of limits. In this case the voltages should not be ramped up. The checked parameters are described at HV Checks.

Depending on the symptoms that are seen the possible interventions that can be made are:

 

  1. try restarting the FSM
  2. try the instructions below for an OPC server crash
  3. try stopping and restarting the WinCC project on vedcs03
  4. try controlling the voltages with the manual override.
  5. try rebooting vedcs03
  6. try resetting the HV crate, using the reset button on its front panel
 

How to recover from an OPC Server Crash (part 1)

 

  • Do rdesktop -g 1200x850 vecc0103w and log in as velo_user, then go to the WinCC:console window on vecc0103w and select the OPC DA Client 8 (must be number 8!). If you cannot connect, try to ping vecc0103w. If the machine doesn't answer it must be rebooted. This can be done from the Big Brother panel, after clicking the"Fieldbus status" button.
HVOpcServer.png
  • Click on the OPC DA client 8 and then use the forced kill icon (the red on/off type button) on the right column of the window. confirm this kill in the pop up window.
 
  • also check that the process is no longer running with the windows task manager, and if it is running kill this also.
do this by going Start -> TaskManager -> processes -> look for isegHV~1.exe, and if it is there stop the process.

 

  • restart the process on the console window by clicking on OPC DA client 8 and pressing the top green start icon on the right column of the window.
 
  • Now check that the OPC server is running, see "How to check the OPC server is running"
 

How to recover from an OPC Server Crash (part 2)

If "How to recover from an OPC Server Crash (part 1)" does not fix your problem it might be that the Iseg Crate is stuck. Symptoms are: OPC server looks fine, OPC client looks fine, but channels don't respond to any command. Then try the following:

 

  • go to the WinCC:console window and scroll down to find the OPC DA Client 8 (must be number 8!).
HVOpcServer.png
  • Click on the OPC DA client 8 and then use the forced kill icon (the red on/off type button) on the right column of the window. confirm this kill in the pop up window.
 
  • also check that the process is no longer running with the windows task manager, and if it is running kill this also.
do this by going Start -> TaskManager -> processes -> look for isegHVOPCServer.exe, and if it is there stop the process.

* Open the IsegCANHVcontrol software in Start->AllPrograms->iseg HV control; you will get the following window:

IsegCANHV_1.JPG

Often this window shows an error in German (helpfully). Unfortunately this means that the machine needs to be rebooted.

Reboot the machine and re-login and kill the OPC server again in the same way before opening the IsegCANHVControl program.

If all the Banks are empty and the Bus Status (bottom right) does not show the message "OK", then it means that the crate is stuck; click on the leftmost button to access the crate window:

IsegCANHV_2.JPG

Reset the crate (red button with a "1"), then click on the OFF button and finally on the On button; close this window and the previous one, then re-open it. You should now see the following:

IsegCANHV_3.JPG

Now the first two Banks show all the boards installed and the Bus Status says "OK". Close this software and restart the OPC server

 

How to check the OPC server is running (Part 1)

The VELOA_HV:TOP and VELOC_HV:TOP both display the "CAN HeartBeat". If the heartbeat has stopped the OPC server has died. However, this is not conclusive the heartbeat can still continue even if the OPC server has died. The instructions to restart the OPC server should be followed. The hearbeat should be a continuosly increasing number.

 

How to check the OPC server is running (Part 2)

These instructions show how to check the OPC server is running WITHOUT using the FSM

From the Device Editor Navigator window (for example on Vedcs03w)

Go to Hardware Tab -> VEDCSHV -> ISEG ->CAN0, click to diplay a new window, that is shown here

Device_Module.JPG

check that the "OPC server heartbeat" is constantly increasing number.

If not, the OPC server has crashed and needs to be restarted.

 

How to restart the OPC server

Do rdesktop -g 1200x850 vecc0103w and log in as velo_user, machine LHCb. .If you cannot connect, try to ping vecc0103w. If the machine doesn't answer it must be rebooted. This can be done from the Big Brother panel, after clicking the"Fieldbus status" button. In the WinCC console, select the OPC client
(manager num 8) :

HVOpcServer-console.png

Click on to kill the OPC client; the manager will try to restart once, kill it a second time and wait for the manager to stop (the green number 2 becomes a blue 0).

Open the task manager and kill the OPC server isegHVOPCServer.exe:

HVOpcServer-taskmanager.png

Go back to the WinCC console and restart the OPC client by selecting it and clicking on . The OPC client will automatically restart the OPC server too, so you must not start it manually.

 

How to access the Device Editor and Navigator

From the command line type vehv_den or use the menus "Linux Start Button" > VELO_EXPERT > VEHV > den

From the Device Editor and Navigator the VELO A or c side HV top panels can be accessed directly by selecting the FSM tab -> VELOA_HV or VELOC_HV, right click and select view

 

How To recover from the values not updating on the screen

If the current values are not updating on the display, or if the values for all channels are reading the same value for the current then restart the OPC server. This is true even if the OPC server is still running and its heartbeat is still going, try restarting the OPC server. for instructions see "How to recover from an OPC server (parts 1 and 2)"

 

How to recover from a current trip

Do not attempt to recover from a trip unless you understand what caused it. If in doubt call an expert.

If a VELO HV channel draws a current above the software limit set for this channel it will undergo an I trip.

To recover this

  1. open the panel of the corresponding channel, a red Itrip alarm will be displayed
  2. Turn the channel off and wait till its status has changed to off.
  3. Press the Clear Alarm button
This is shown on the following screenshot. HV_trip_Error.JPG

If there are multiple trips, the buttons on the VELO:TOP panel allow all alarms to be cleared.

How to know which channel you are controlling (naming conventions)

The VELOA_HV panel lists the channels in two numbering conventions. It gives the slot number and the high voltage channel number.

The slot numbers are for example PU01_AB for Pile-up (PU) 1st (01) Aside (A) Bottom repeater board (B) VL06_AT for Velo (VL) 6th (06) Aside (A) Top repeater board (T). There are some spare channels on the HV which are listed as "undefined" in this convention

The high voltage channel numbers are listed by board number and channel number for example VELOC_HV_BOARD02 ch 03 The channels are numbered 00 to 07 (8 channels) on each board.

Physically there are two boards on each power supply module, i.e. each module has 16 channels. Board00 and Board01 belong to the same power supply module (similiarly 02/03,04/05 separately for either A or C side) The 6 power supply module sfor the whole velo are in the same crate in D3.

The full connectivity is described in the "VELO ECS/LV/HV connectivity" document and the full HV chain in the "HV connectivity" note which are linked on the main Twiki HV page.

Topic attachments
I Attachment Action Size Date Who Comment
Unknown file formatJPG Console_Panel.JPG manage 57.8 K 2015-02-27 - 17:04 KarolHENNESSY  
Unknown file formatJPG Device_Module.JPG manage 39.2 K 2015-02-27 - 17:03 KarolHENNESSY  
PNGpng HVOpcServer-console_1.png manage 78.3 K 2015-02-27 - 17:05 KarolHENNESSY  
PNGpng HVOpcServer-taskmanager_1.png manage 78.1 K 2015-02-27 - 17:05 KarolHENNESSY  
PNGpng HVOpcServer.png manage 496.8 K 2015-02-27 - 17:09 KarolHENNESSY  
Unknown file formatJPG HV_trip_Error.JPG manage 28.2 K 2015-02-27 - 17:03 KarolHENNESSY  
Unknown file formatJPG IsegCANHV_1.JPG manage 22.1 K 2015-02-27 - 17:03 KarolHENNESSY  
Unknown file formatJPG IsegCANHV_2.JPG manage 20.9 K 2015-02-27 - 17:03 KarolHENNESSY  
Unknown file formatJPG IsegCANHV_3.JPG manage 26.8 K 2015-02-27 - 17:03 KarolHENNESSY  
Unknown file formatJPG KillManager.JPG manage 1.1 K 2015-02-27 - 17:04 KarolHENNESSY  
Unknown file formatJPG StartManager.JPG manage 1.1 K 2015-02-27 - 17:04 KarolHENNESSY  
Unknown file formatJPG TaskManager.JPG manage 87.5 K 2015-02-27 - 17:04 KarolHENNESSY  
Unknown file formatJPG VELOA_HV_BOARD_TOP.JPG manage 106.1 K 2015-02-27 - 17:01 KarolHENNESSY  
Unknown file formatJPG VELOC_HV_BOARD02_LOAD.JPG manage 103.8 K 2015-02-27 - 17:01 KarolHENNESSY  
Unknown file formatJPG VELO_HV.JPG manage 41.7 K 2015-02-27 - 17:01 KarolHENNESSY  
PNGpng VELO_HV_TOP.png manage 44.0 K 2015-02-27 - 17:01 KarolHENNESSY  
Edit | Attach | PDF version | Print version |  | Raw View | Raw edit | Backlinks | History: r5 < r4 < r3 < r2 < r1 | More topic actions
Topic revision: r5 - 2018-05-31 - ThomasEdwardLatham
 

TWIKI.NET
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2022 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback