HOW TO TELL1 board [information for the calo piquet]
This page contains the information about the Tell1 boards (DAQ) for the calo piquet.
The Basics
The Tell1 boards are part of the DAQ. These boards:
- collect the data sent by the CROC via optical fibers,
- process these data in the PP-FPGA (zero suppression (PSSPD) or compression (EHCAL)),
- merge all links in MEPs (Multi Event Packets) in the Synclink FPGA,
- send the MEPs to the Event Builder via the network in IP frames.
Data flow: [CROC --->] Optical Receivers (
ORx) ---> PP-FPGA ---> Synclink FPGA ---> Gigabit Ethernet (
GBE) [---> Event Builder]
- Optical mezzanine (ORx): Optical fibbers from CROCs [only up to 8 on 12 links are used per ORx]
- Credit Card PC (CCPC): Connection with ECS
- PP-FPGA: User data compression
- Synclink FPGA: Gathers PP-FPGA information and Build Multiple Event Packet (MEP)
- Giga Bit Ethernet (GBE): Transmits MEP to Event Builder
- TTC Receiver: Clock, L0 Trigger, Reset (L0evID – BX-ID) and DAQ IP destination address
Tell1 PVSS/WINCCOA Project
This panel summarizes the information about the status of the Tell1.
TELL1 overall status:
|
|
Becomes Red |
Action |
CCPC |
go to Board Status tab (CCPC: check memory and CPU usage) |
TTC |
go to TTC & FlowCtrl Mon tab (expert panel) |
ORx |
go to ORx tab |
GBE |
go to GBE tab |
Process mon |
go to Proc. Mon tab (expert panel) |
Throttles |
go to TTC & FlowCtrl Mon and Buffer Mon tabs (expert panels) |
Memory mon |
go to Buffer Mon tab (expert panels) |
|
|
|
Versions:
Server Version: if 0-0-0, it may be because of ccserv dead [see below]
- Evt cnt: wr: events sent from the GOLs (on CROC mezzanine) are WRITTEN in the ORx input synchronization RAM
- Evt cnt: rd: events stored in the ORx input synchronization RAM are READ by the PP-FPGAs (TELL1)
Rack location
The tell1 boards for the calorimeters are located in D3 barrack:
- rack D3B01L for PSTELL1
- rack D3B01U for HCTELL1
- rack D3B02L for ECTELL1 C-side
- rack D3B02U for ECTELL1 A-side [ IMPORTANT: TWO L0calo Tell1 boards are also in this rack (crate slots: 17 and 18) ]
CROC to Tell1 connections
|
|
PSSPD C-side |
PSSPD A-side |
Tell1: name |
Tell1: Crate slot |
Feb Crate |
Tell1: name |
Tell1 Crate: slot |
Feb Crate |
pstell01 (2 ORx) |
02 |
C0 Feb 1-14 |
pstell05 (1 ORx) |
06 |
C4 Feb 1-7 |
pstell02 (2 ORx) |
03 |
C1 Feb 4-11 |
pstell06 (1 ORx) |
07 |
C5 Feb 1-7 |
pstell03 (1 ORx) |
04 |
C2 Feb 1-7 |
pstell07 (2 ORx) |
08 |
C6 Feb 4-11 |
pstell04 (1 ORx) |
05 |
C3 Feb 1-7 |
pstell08 (2 ORx) |
09 |
C7 Feb 1-14 |
pstell09 (1 ORx) |
10 |
C3 Feb 8-14 |
pstell10 (1 ORx) |
11 |
C4 Feb 8-14 |
pstell11 (1 ORx) |
12 |
C2 Feb 8-14 |
pstell12 (1 ORx) |
13 |
C5 Feb 8-14 |
|
|
|
|
|
HCAL C-side |
HCAL A-side |
Tell1: name |
Tell1: Crate slot |
Feb Crate |
Tell1: name |
Tell1 Crate: slot |
Feb Crate |
hctell01 |
06 |
C22 Feb 2-7 |
hctell05 |
10 |
C24 Feb 1-3 |
hctell02 |
07 |
C22 Feb 8-12 |
hctell06 |
11 |
C24 Feb 8-14 |
hctell03 |
08 |
C23 Feb 1-7 |
hctell07 |
12 |
C25 Feb 2-7 |
hctell04 |
09 |
C23 Feb 8-11 |
hctell08 |
13 |
C25 Feb 8-12 |
hctell09 |
14 |
C23 Feb 12-14 |
hctell10 |
15 |
C24 Feb 4-7 |
|
|
|
|
|
ECAL C-side |
ECAL A-side |
Tell1: name |
Tell1: Crate slot |
Feb Crate |
Tell1: name |
Tell1 Crate: slot |
Feb Crate |
ectell01 |
04 |
C8 Feb 1-7 |
ectell14 |
04 |
C15 Feb 2-7 |
ectell02 |
05 |
C8 Feb 8-13 |
ectell15 |
05 |
C15 Feb 8-13 |
ectell03 |
06 |
C9 Feb 1-7 |
ectell16 |
06 |
C16 Feb 2-7 + Feb1 |
ectell04 |
07 |
C9 Feb 8-13 |
ectell17 |
07 |
C17 Feb 0-7 |
ectell05 |
08 |
C10 Feb 0-7 |
ectell18 |
08 |
C17 Feb 8-13 |
ectell06 |
09 |
C10 Feb 8-15 |
ectell19 |
09 |
C18 Feb 0-7 |
ectell07 |
10 |
C11 Feb 2-7 |
ectell20 |
10 |
C18 Feb 8-13 |
ectell08 |
11 |
C11 Feb 9-15 |
ectell21 |
11 |
C19 Feb 0-7 |
ectell09 |
12 |
C12 Feb 2-7 + Feb1 |
ectell22 |
12 |
C19 Feb 8-15 |
ectell10 |
13 |
C12 Feb 8-15 |
ectell23 |
13 |
C20 Feb 1-7 |
ectell11 |
14 |
C13 Feb 2-7 |
ectell24 |
14 |
C20 Feb 8-13 |
ectell12 |
15 |
C13 Feb 8-13 |
ectell25 |
15 |
C21 Feb 1-7 |
ectell13 |
16 |
C14 Feb 2-7 |
ectell26 |
16 |
C21 Feb 8-13 |
ectell27 |
17 |
C14 Feb 8-13 |
ectell28 |
19 |
C16 Feb 8-13 |
|
|
|
Note: C12 Feb1 (ectell09) and C16 Feb1 (ectell16) are for crate timing
These boards are configured in the Tell1 recipes as PIN boards.
For Calo Piquet
The main problems you may encounter are:
- hardware: a board is broken and must be replaced
- flash a firmware
- change a recipe
- Tell1 didn't boot up after switching on the crate
- data process (corrupted or missing events, Destination IP from ODIN, overflow Buffers, etc.) [these problems will most of the time require an expert]
In case you need to replace a Tell1 board, here are the steps to do so:
1) find the Tell1 board to be replaced:
- either with PVSS Tell1 panels --> tell1 name (for example: ectell10)
- or calo presenter --> sourceID. IMPORTANT: tell1 name = SourceID + 1 [ex: if "ECAL sourceID = 9" then the tell1 board to exchange is ectell10 ]
2) Take a Tell1 board from the elec lab (2nd floor above control room)
- Use the red bubble wrap to protect the board during the transport.
- Take the sticker with the MAC address to be given to the Online piquet
3) go to D3
with a pen , and your piquet GSM phone
4) Replace the board:
- switch off the crate
- find the Tell1 board to be replaced (see above for the slot in the Tell1 crate: Rack location)
- unplug with care:
- front: 12-ribbon connector
- back: ECS, TTC fiber, throttle, Gbe
IMPORTANT: the Gbe Ethernet cables must be replugged in order --> you should mark with a pen: 0 ,1 ,2 ,3 for port 0 (up) to port 3 (down) [hence the pen in 3)]
5) Extract the board with extreme care (watch out not to bend the board and the mezzanines (ORx, CCPC, Gbe)
6) Insert the new board (again with extreme care, watch out the mezzanine)
7) Replug all cables (ORX, ECS, TTC, Gbe (in proper order), Throttle)
8) Switch on the crate
9) Try first the web-service for changing TELL1 boards:
https://lbdokuwiki.cern.ch/online_user:using_the_tell1_changer_website
(==> You need a valid Online account and the page is only visible from within the Online network.)
If it fails, contact the online piquet.
10) bring the broken Tell1 board to the electronic pool (2nd floor, on top of the control room) [see:
https://edms.cern.ch/document/1083357 ]
11) Fill an entry to the CALO logbook! and send an email to the expert.
To load the Tell1 firwmware to a given Tell1 board, the FSM should be in
NOT READY for the given Tell1.
First option with
WinCCOA (PVSS):
- Click on NOT READY to popup a window [which contains |Configure|Reset|LoadFrmwr|] then LoadFrmwr.
- The FSM state switches to CONFIGURING.
- It takes about 2 minutes to upload the firmware.
- Once done, the FSM state switches back to NOT READY.
- Do Configure the given Tell1 board to check that everything is ok.
Second option on the CCPC:
- logon on the Tell1 CCPC (e.g., ssh ectell01)
- execute the following command: "EPC16Handling -e -p 1 file.pof" where "file.pdf" is located in /group/calo/Tell1Firmware (Choose EHCAL for ECAL and HCAL, PSSPD for PSSPD!)
- wait that the flashing is done (about 2 minutes) and execute the command "reset_tell1"
- Use PVSS/WinCCOA panels. Configure your board to go to "READY" state.
- Go to the panel where you want to change something (e.g., ORx panel)
- Make your change (e.g., enable or disable a link on the ORx panel)
- Go to the last panel named Recipes
To check if the Tell1 board has booted up, execute "ping tell1name" (ex: ping pstell08).
- It sould give "64 bytes from pstell08.lbdaq.cern.ch (10.130.36.18): icmp_seq=0 ttl=63 time=0.702 ms" ("time=0.702 ms" may vary).
- If the ping command is stuck to "PING pstell08.lbdaq.cern.ch (10.130.36.18) 56(84) bytes of data.", it means that the Tell1 board did not boot up.
- Power off and on again the crate. Wait that all Tell1 booted (about 5 minutes). Then configure the Tell1 boards with PVSS/WinCCOA.
- If switching off than on the crate failed, go to D3 barrack and press on "Reset CCPC" button (front cover of the board).
FOR EXPERTS ONLY (you must know what you are doing!)
Information can be obtained about the tell1 board by running binaries on the Tell1 CCPC (Credit Card PC).
PLEASE CALL THE EXPERT FIRST
The CALO Tell1 firmware contains spy memories in the PP-FPGA [size: 256 words - 1 CROC event: 34 words (1 header - 1 ctrl word and 32 data words)] on the ORx outputs which allow to read what was received by the CROC before any treatment of the data.
The base addresses are
[PP0]: 0x4000000,
[PP1]: 0x5000000,
[PP2]: 0x6000000 and
[PP3]: 0x7000000
Each PP-FPGA treats 4 internal input links (after multiplexer). The address for each link is:
- Internal link0: 0x508000
- Internal link1: 0x50A000
- Internal link2: 0x50C000
- Internal link3: 0x50E000
Two additional spy memory give access to the information received by TTC (
TriggerType,
L0EvtID, BXID) and the Event Information (used in the PP-FPGA and Error Bank):
- TTYPE,L0 and BXID: 0x510000 [Format: |31-16: L0EvtID| 15-4:BXID|3-0:TriggerType|]
- Event Information: 0x512000
Examples:
- to read PP0 link2: lbread 0x450C000 34 [1st arg: address - 2nd arg: number of words] which gives
0450c000: df2003fe 9f203fd3 5f300101 5f200100
0450c010: 5f3000ff 5f2000fd 5f300104 5f200105
0450c020: 5f3000ff 5f3000ff 5f300101 5f200100
0450c030: 5f200100 5f200103 5f200100 5f300102
0450c040: 5f200100 5f2000fe 5f300102 5f300104
0450c050: 5f300102 5f200100 5f300101 5f200100
0450c060: 5f300102 5f200103 5f300101 5f2000fe
0450c070: 5f3000ff 5f200105 5f2000fe 5f300102
0450c080: 5f200100 5f200100
- lbread 0x4510000 1 gives 0x00413fe6: L0EvtID=0x0041, BXID=0x3fe, TTYPE=0x6)
- lbread 0x4512000 15 gives
04512000: 000243fe 00000000 aaaaaaaa 000243ff
04512010: 00000001 aaaaaaaa 00024400 00000002
04512020: aaaaaaaa 00024401 00000003 aaaaaaaa
04512030: 00024402 00000004 aaaaaaaa
Input Format (CROC)
34 words: 1 header word + 1 control word + 32 data words
|
|
11 |
FE Crate(5b) |
FE Card(4b) |
P(1b) |
L0EvtID(10b) BX ID(10b) |
10 |
FE Crate(5b) |
FE Card(4b) |
P(1b) |
Control word(20b) |
01 |
FE Crate(5b) |
FE Card(4b) |
P(1b) |
Data 0(20b) |
01 |
FE Crate(5b) |
FE Card(4b) |
P(1b) |
... |
01 |
FE Crate(5b) |
FE Card(4b) |
P(1b) |
Data 31(20b) |
|
|
|
EHCAL Data (20b):
PSSPD Data (20b):
|
|
Trigger 0(2b) |
ADC 0(8b) |
Trigger 1(2b) |
ADC 1(8b) |
|
|
|
MEP Data Format
EDMS documents:
EHCAL ZS Data Format
PSSPD ZS Data Format
Control word(9b): (General Error = Error Link OR Error Synchro)
|
|
CTRL(2b) |
Error Link(1b) |
Error Synchro(1b) |
TTYPE(4b) |
General Error(1b) |
|
|
|
TTYPE (ODIN board):
|
|
Trigger Type |
Encoded |
Priority |
Reserve |
000 (0x0) |
- |
Physics |
001 (0x1) |
1 |
Auxiliary |
010 (0x2) |
2 |
Random |
011 (0x3) |
3 |
Periodic |
100 (0x4) |
4 |
NZS |
101 (0x5) |
- |
Timing |
110 (0x6) |
5 |
Calibration |
111 (0x7) |
6 |
|
|
|
Error Bank Format
|
|
Header |
Bank Length(16b) |
Bank ID(0x8E00)(16b) |
|
|
|
|
|
Link0 |
0 |
Error Synchro(1b) |
Error Link(1b) |
FE Crate(5b) |
FE Card(4b) |
FE L0EvtID(10b) |
FE BXID(10b) |
Link1 |
0 |
Error Synchro(1b) |
Error Link(1b) |
FE Crate(5b) |
FE Card(4b) |
FE L0EvtID(10b) |
FE BXID(10b) |
Link2 |
0 |
Error Synchro(1b) |
Error Link(1b) |
FE Crate(5b) |
FE Card(4b) |
FE L0EvtID(10b) |
FE BXID(10b) |
Link3 |
0 |
Error Synchro(1b) |
Error Link(1b) |
FE Crate(5b) |
FE Card(4b) |
FE L0EvtID(10b) |
FE BXID(10b) |
|
|
|
bank_length : 168[0xA8] bytes,42[0x2A] words
PRS Threshold RAM (also used to mask a PRS Channel)
L0061 to L0124 in Tell1 cfg file. These RAM are used to define the ADC threshold for each channel.
They can also be used to mask a PRS channel by putting max value for the threshold (0x3FF == 0b1111111111).
The threshold RAM is ordered:
- PP0 to PP3
- ADC channel 0 to 3 [ie "internal optical link" after multiplexer]
- Word 0 to 31 (Word 0 at MSB on L0061), only bits [19-0] are valid
- Each word contains TWO channels [19-10] (first channel) and [9-0] (second channel).
Each PP-FPGA receives up to 4 optical links.
|
|
Feb Number |
ORx input link |
Feb Number |
ORx input link |
Feb Number |
ORx input link |
Feb Number |
ORx input link |
1 ---> |
0 [PP0] |
6 ---> |
0 [PP1] |
9 ---> |
0 [PP2] |
14 ---> |
0 [PP3] |
3 ---> |
1 [PP0] |
X (no GOL) |
1 (Disable) |
11 ---> |
1 [PP2] |
X (no GOL) |
1 (Disable) |
0 ---> |
2 [PP0] |
X (no GOL) |
2 (Disable) |
8 ---> |
2 [PP2] |
X (no GOL) |
2 (Disable) |
2 ---> |
3 [PP0] |
4 ---> |
3 [PP1] |
10 ---> |
3 [PP2] |
12 ---> |
3 [PP3] |
X (no GOL) |
4 (Disable) |
7 ---> |
4 [PP1] |
X (no GOL) |
4 (Disable) |
15 ---> |
4 [PP3] |
X (no GOL) |
5 (Disable) |
5 ---> |
5 [PP1] |
X (no GOL) |
5 (Disable) |
13 ---> |
5 [PP3] |
|
|
|
console_tell1 is a binary which is run after logging on the tell1 CCPC. After executing
console_tell1, you have the following menu:
=======Running Console on TELL1======
ACTION : ini_GLUECARD_PLX [ OK]
ACTION : read_TELL1_CONSTANT [ OK]
************************************
* C : TELL1 Control *
* M : TELL1 Monitor *
* T : TELL1 Test *
* P : TELL1 Programming *
* F : TELL1 export Files *
*----------------------------------*
* D : Detector specific *
*----------------------------------*
* A : About *
* Q : Quit *
* ? : show this manual *
************************************
CMD---> [?]
The
allowed menus are "M" (TELL1 Monitor), "T" (TELL1 Test) and "D" (Detector specific)
when LHCb does NOT TAKE DATA!
[
if you mess up entering "M" for example, the delete key (or Backspace key) does not work, use instead "CTRL+h" to delete ]
TELL1 Monitor
After entering "m" and "return":
CMD---> [?]m
ACTION : read_TELL1_CONSTANT [ OK]
================================
2 : GBE status
3 : Read temperature
4 : Read BER counters
5 : Read GBE phy chips registers
6 : Dump the initial counter data in PLL reconfig module
8 : Read ttc monitor counters
---------------------------
r : Read rate statistic
a : Read memory max usage
d : Read event monitor registers
f : Read flow_control information
h : Read Ethernet and IPv4 header information
---------------------------
ESC : CCPC DAQ routine control
Q : Quit
? : Show this manual
MONITOR--->[?]
where you can access various status of the tell1 board.
TELL1 Test
After entering "t" and "return":
CMD---> [c]t
ACTION : read_TELL1_CONSTANT [ OK]
=================================================
ESC : CCPC DAQ routine control
-------------------------------------------------
s : Tell1 full function test for ST or VELO, including tell1 program.
g : GBE check with MEP packet changing.
l : Test lbus
t : Test trigger info bus
e : Test ECS bus
m : Test MEP bus (QDR)
-------------------------------------------------
Q : Quit
? : show this manual
TESTING--->[?]
to test various hardware problems, like "Monitor QDR parity errors" (see:
https://lbtwiki.cern.ch/bin/view/CALO/LHCbCaloTell1#Proc_Mon_tab) with "m : Test MEP bus (QDR)".
Detector specific
If you press "d" then "return": (this menu is a bit different between EHCAL and PSSPD)
CMD---> [m]d
ACTION : read_TELL1_CONSTANT [ OK]
------------------------------------------------------------------------
Comments for the conditions of command execution:
(Stop daq_tell1) means daq_tell1 must be paused or killed.
(Run daq_tell1) means daq_tell1 must be running.
(Run daq_tell1 with Data genrator) means daq_tell1 must be running with data generator enabled.
(Data genrator enabled) means data generator must be enabled.
Otherwise, the command can be executed under any condition.
=============================================
a : Dump PSSPD specific settings.
b : Dump PSSPD ADC threshold RAM.
---------------------------
o : Read link probe/sync registers
r : Read data generator ram
s : The Last 128 Mep Location
t : Read Mep Buffer
l : Read and Parse MEP during run(Except L0DU!!!)
p : Parse MEP (Stop daq_tell1)
u : Parse all 128 MEPs (Stop daq_tell1)
m : Extract raw data from newest event in the MEP
w : Write raw data from newest event into a file. (Run daq_tell1)
n : Check newest Non_zerosuppressed bank. (Data genrator enabled)
z : Check newest Zero suppressed bank. (Data genrator enabled)
i : Export Present Setting of Tell1 Board to a new cfg file(Stop DAQ)
---------------------------
ESC : CCPC DAQ routine control
Q : Quit
? : show this manual
PSSPD--->[?]
IMPORTANT: the only menu allowed during data taking is
o : Read link probe/sync registers [Read the ORx status]. Everything else may crash the data acquisition.
Logon the tell1 ccpc. Stop the ccserv daemon [sudo service ccserv stop]. Then execute: "ccserv -d -f- -vN" where N is the verbose level (the higher the number, the higher the verbosity) [a typical value is N=6]
To update a cfg file with the new tell1lib release, use daq_tell1 old_cfg.cfg (daq_tell1 must be the binary from the new tell1lib)
Example: convert EHCAL3.v23.cfg (tell1lib v2.3) to EHCAL3.v26.cfg (tell1lib v3.0)
daq_tell1 EHCAL3.v23.cfg
CFG: Read EHCAL3.v23.cfg
>>>>>>>>>>>>
A new DB version is used now. We will convert to this new DB format.
All your settings will be kept, but the comment you add will lost.
CFG: Write EHCAL3.v26.cfg
then stop daq_tell1 with CTRL+C
PVSS Tell1 projects are running on (linux machine):
- hcdaq01 for PSTELL1 and HCTELL1
- ecdaq01 for ECTELL1
The projects are in "autostart" mode and are handled under Linux as a service [/etc/init.d/pvss_mp {start|stop|restart|status}]
After logging on either hcdaq01 or ecdaq01:
- to know the status of a project: execute "sudo /sbin/service pvss_mp status" [can be "running" or "stopped" or "unknown"]
- to restart a project: execute "sudo /sbin/service pvss_mp restart"
- to stop a project: execute "sudo /sbin/service pvss_mp stop" ( BEWARE: you must know what you are doing!)
The calo pvss shortcuts are available at: /group/online/ecs/Shortcuts311/CALO
- The TELL1 PVSS projects are installed:
- PSTELL1: /data/pvss/PSTELL1
- HCTELL1: /data/pvss/HCTELL1
- ECTELL1: /data/pvss/ECTELL1
- The tell1 components are installed in: /group/calo/pvss/fwComponents_ECTELL1, fwComponents_HCTELL1, fwComponents_PSTELL1
The PVSS console can also be accessed via a browser. Enter
http://localhost:port in a browser where the pmon of a project runs. [the default pmon port is 4999 but is generally modified in the config file]
- PS: port = 28300 [on hcdaq01, run firefox with http://localhost:28300; config file in /data/pvss/PSTELL1/config]
- HC: port = 30300 [on hcdaq01, run firefox with http://localhost:30300; config file in /data/pvss/HCTELL1/config]
- EC: port = 29300 [on ecdaq01, run firefox with http://localhost:29300; config file in /data/pvss/ECTELL1/config]
There is also a (Windows) link in the CALO shorcut directory (XXTELL1_Console).
Generally after (re)starting a PVSS project, FSM is in a DEAD status or stuck.
- To put it in a "NOT READY" status, you need to run a DEN (Device Editor and Navigator)
- Then go to "FSM" tab and finally press "Stop All" followed by "Start/Restart All".
To run a DEN, you can use the shortcuts [HCTELL1_UI_DEN, PSTELL1_UI_DEN, ECTELL1_UI_DEN] (See below: Tell1 PVSS Shortcuts).
Another way is to restart the FSM from top layer FSM (Restart FSM)
PVSS commands are sent to the Tell1 board via dim (PVSS manager
PVSSdim) and are received on the Tell1 with
ccserv (and vice versa). If you encounter a problem of "CONFIGURING" the Tell1 with PVSS, it may be:
- ccserv is dead
- the tell1 board did not boot up (e.g., after powering on the crate)
ccserv problem:
To check which servers are running, log on either hcdaq01 or ecdaq01, and execute
did. Then click on
View --->
All Servers. You should have one entry per ccserv (ie, per tell1 board). If one (or more) is missing, log on the missing tell1 (ex: ssh ectell01) and execute "sudo /sbin/service ccserv status". If the answer is:
- running: it is OK.
- ccserv dead but subsys locked. In this case, execute "sudo /sbin/service ccserv restart" and then check with did that the entry is back to the ccserv list.
Note: if
ALL ccservs are missing after running did (View ---> All Servers), it may be because the crate is off.
did on ecdaq01 (order may be different)
did on hcdaq01 (order may be different)
(example below: pstell08 is missing)
--
StephaneTJampens- 17 Jun 2009