Checking Stations Into GLOW Mode

Wait for scheduled LOCAL mode time

An e-mail from ILT operations should come, informing us that DE601 has been set to local mode operations. Check the DE601 monitoring software at https://glow-control.mpifr-bonn.mpg.de/cacti/graph_view.php to check that the station is in swlevel 0.

Log into the interface computer (lofarx)

ssh -X observer@glow-control

Set up SSH stuff with the following commands

ssh-agent /bin/tcsh -l

ssh-add

You will need the password for the observer account.

Log into the gateway machine

(E.g. for DE601:)

ssh -X glow601

Log into the DE601 LCU

ssh -X de601c

Check that no-one else is on the station

w

If there is more than one login active (you), then check who else is logged into the station.  If people from ASTRON are still logged in, contact ASTRON.  Their processes may need to be killed (sudo access required).

Check that no more remote mounts are there

mount

If there is a line (usually the last) starting like:

KIS001:/net/home on /misc/home ...

then there is an external mount, which does not work in GLOW mode and will cause huge delays. You'll have to ask ASTRON to remove that.

Check software level (should be 0)

swlevel

If not in software level 0, bring it down. 

Verify that no station processes are running.  A report of swlevel 0 from the swlevel program does not necessarity mean that everything is stopped.

( ps -AF )

(We need some scripts that generate easier to read output! For the time being, just use: )

ps -AF | grep lofarsys

This should print out nothing or one line containing "grep lofarsys"

Check the station configuration files in /opt/lofar/etc/

less /opt/lofar/etc/RSPDriver.conf

Check that the RSP lanes are sending data to the correct recording computers.  For recording beamlet data on lofarBN, this should have

RSPDriver.LANE_00_BLET_OUT= 0
RSPDriver.LANE_00_XLET_OUT= 5
RSPDriver.LANE_00_SRCMAC  = 00:22:86:06:01:00
RSPDriver.LANE_00_SRCIP   = 10.211.1.1
RSPDriver.LANE_00_DSTMAC  = 00:19:99:ba:42:cb   # lofarB1  
RSPDriver.LANE_00_DSTIP   = 10.211.1.3

RSPDriver.LANE_01_BLET_OUT=8
RSPDriver.LANE_01_XLET_OUT=5
RSPDriver.LANE_01_SRCMAC  = 00:22:86:06:01:01
RSPDriver.LANE_01_SRCIP   = 10.212.1.1
RSPDriver.LANE_01_DSTMAC  = 00:19:99:e2:21:46   # lofarB2
RSPDriver.LANE_01_DSTIP   = 10.212.1.3

RSPDriver.LANE_02_BLET_OUT=4
RSPDriver.LANE_02_XLET_OUT=5
RSPDriver.LANE_02_SRCMAC  = 00:22:86:06:01:02
RSPDriver.LANE_02_SRCIP   = 10.213.1.1
RSPDriver.LANE_02_DSTMAC  = 00:19:99:e2:21:ba   # lofarB3
RSPDriver.LANE_02_DSTIP   = 10.213.1.3

RSPDriver.LANE_03_BLET_OUT=12
RSPDriver.LANE_03_XLET_OUT=5
RSPDriver.LANE_03_SRCMAC  = 00:22:86:06:01:03
RSPDriver.LANE_03_SRCIP   = 10.214.1.1
RSPDriver.LANE_03_DSTMAC  = 00:19:99:e2:20:ce   # lofarB4
RSPDriver.LANE_03_DSTIP   = 10.214.1.3
 

When you are done type "q" to exit less.

If the computers that the data is being sent to needs to be changed (depends on the whishes of the observer), you need tp copy the correct version of the file. As of 2013 May 13, current versions of the RSPDriver.conf file can be found at

~user9/StationConfigs/RSPDriver.conf.local_lofarAN_20130513      for lofarAN, copy that with:

cp ~user9/StationConfigs/RSPDriver.conf.local_lofarAN_20130513 /opt/lofar/etc/RSPDriver.conf

~user9/StationConfigs/RSPDriver.conf.local_lofarBN_20130423      for lofarBN, copy that with:

cp ~user9/StationConfigs/RSPDriver.conf.local_lofarBN_20130423 /opt/lofar/etc/RSPDriver.conf

less /opt/lofar/etc/BeamServer.conf

Check that the HBA update interval is reasonable.  For pulsar observations, where the noise from communicating with the HBA antennas is a bad thing, set this to a large value such as 300 seconds.  For other observations, check the reasonable default values

BeamServer.HBAUpdateInterval= 300

Start up the station to test LBA and HBA antennas

swlevel 1

This starts up the ServiceBroker.  Wait 10 seconds, then continue

swlevel 2

This starts up the RSPDriver and TBBDriver.  This takes quite a long time.  Follow the instructions for the next command to see when this finishes.

tail -f /log/RSPDriver.log

This continually checks the last log entries for the RSPDriver.

Wait till no more lines: "port ??? has not yet completed sync or had errors, trying to continue..." show up for at least 5 seconds.  This may take a minute or two.  Then enter <CTRL-C> to kill the tail program.

If the RSPDriver terminates at this point, then there is a problem with the hardware, please contact an expert. (Or go back to swlevel 0 and try again.)

Do some simple tests

Allow for a few seconds after each rspctl command to allow the settings to be distributed to and accepted by the hardware!

rspctl --rcumode=3

rspctl --rcuenable=1

This switches on all LBAs.

rspctl --rcu

This gives a list of all rcus. They should all be in rcumode 3 (or whatever you set). If there are some stuck in rcumode -1 (i.e. fail), then try again with setting the rcumode. If that doesn't help go back to swlevel 0 and try again.

Start up the beam-server

rspctl --bitmode=8

This switches the station to the 8-bit mode.

swlevel 3

This starts up the CalServer and BeamServer software. Check if all services are running.

Some more tests

You can do some more tests, they are not really needed.

beamctl --antennaset=LBA_OUTER --rcus=0:191 --rcumode=3 --beamlets=0:243 --subbands=100:343 --digdir=0,1.5708,J2000&

This starts up the beam software to make a beam pointing at zenith usign RCUMODE 3 (LBA, 10--30 MHz).  This will be used to test antennas.  Wait until the message "All pointings sent and accepted" appears (< 3 seconds).

(Need to fix the "stopped in background" issue!)

rspctl --rcu

This checks that the RCU boards have been set in RCUMODE 3.  Check that the RCUs are enabled (look for "=>  ON" for all RCUs).  Check that all RCUs are in RCUMODE 3 (look for "mode:3" for all RCUs).

If some RCUs are OFF or have the wrong RCUMODE, wait a few seconds and try again.  If the RCUs still are not correct, something is wrong. If that is the case first try going to "swlevel 0"  and back to this point. If it still doesn't work, call the Bonn people...

(We need to explain that better!)

rspctl --statistics  --integration=3

This checks the spectrum (0--100 MHz) for each RCU, with all RCU spectra plotted in a single gnuplot window.

When the program starts up, check the initial log report.  It should print out:

Taking subscription on the clockvalue
Current clockvalue is 200 Mhz
Taking subscription on the state of the splitter
The splitter is currently OFF
Taking subscription on the bitmode
The bitmode is currently 16

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!
 

rspctl --statistics=beamlet  --integration=3

This checks the beamlet spectrum, using the specified subbands.

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!

killpointing

This turns off the beam command. (If you hit <enter> or at the next command, you'll get a line that beamctl was killed.)

beamctl --antennaset=HBA_JOINED --rcus=0:191 --rcumode=5 --beamlets=0:243 --subbands=100:343 --anadir=0,1.5708,J2000 --digdir=0,1.5708,J2000&

This checks out the HBA tiles, making a beam pointing at zenith usign RCUMODE 5 (HBA, 110--190 MHz).  This will be used to test tiles.  Wait until the message "All pointings sent and accepted" appears (< 3 seconds).

rspctl --rcu

This checks that the RCU boards have been set in RCUMODE 5.  Check that the RCUs are enabled (look for "=>  ON" for all RCUs).  Check that all RCUs are in RCUMODE 5 (look for "mode:5" for all RCUs).

If some RCUs are OFF or have the wrong RCUMODE, wait a few seconds and try again.  If the RCUs still are not correct, something is wrong.

rspctl --statistics  --integration=3

This checks the spectrum (100--200 MHz, but incorrectly plotted by the software as 0--100 MHz) for each RCU, with all RCU spectra plotted in a single gnuplot window.

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!

rspctl --statistics=beamlet  --integration=3

This checks the beamlet spectrum, using the specified subbands.

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!

Check that beamlet data are arriving

Using another terminal, log into the recording computers and check that data are arriving.  For example, do

(use a VNC for all of this?)

ssh observer@lofarb1

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

ssh observer@lofarb2

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

ssh observer@lofarb3

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

ssh observer@lofarb4

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

In the original terminal do:

Send the e-mail to ASTRON and our users that we have taken over the station

(We have to set up the script, so that the operators can run it.)

Turn the station over to the users

If a user is going to immediately use the station, you can leave it in swlevel 3 (nothing needs to be done in this case). Otherwise, put it in swlevel 0.

(Need to describe that better!)