Checking Stations Into GLOW Mode

Wait for scheduled LOCAL mode time

An e-mail from ILT operations should come, informing us that DE601 has been set to local mode operations. Check the DE601 monitoring software at https://glow-control.mpifr-bonn.mpg.de/cacti/graph_view.php to check that the station is in swlevel 0.

Log into the interface computer (lofarx)

ssh -X observer@lofarx

Set up SSH stuff with the following commands

ssh-agent /bin/tcsh -l

ssh-add

You will need the password for the observer account.

Log into the DE601 LCU

ssh -X de601c

Check that no-one else is on the station

w

If there is more than one login active (you), then check who else is logged into the station.  If people from ASTRON are still logged in, contact ASTRON.  Their processes may need to be killed (sudo access required).

Check software level (should be 0)

swlevel

If not in software level 0, bring it down. 

Verify that no station processes are running.  A report of swlevel 0 from the swlevel program does not necessarity mean that everything is stopped.

( ps -AF )

(We need some scripts that generate easier to read output! For the time being, just use: )

ps -AF | grep lofarsys

This should print out nothing or one line containing "grep lofarsys"

Set the MAC version to the supported software version

MAC_revert -v  LOFAR-Release-1_13_1-20130514T093126-lofarBN-Official

(This doesn't seem to work right, we need to fix this.)

Note that to get the old (1_7) software version, use:

MAC_revert -v  LCU_MAC_LOFAR-Release-1_7-20130514T092921-lofarBN-Official

Check the station configuration files in /opt/lofar/etc/

less /opt/lofar/etc/RSPDriver.conf

Check that the RSP lanes are sending data to the correct recording computers.  For recording beamlet data on lofarBN, this should have

RSPDriver.LANE_00_BLET_OUT= 0
RSPDriver.LANE_00_XLET_OUT= 5
RSPDriver.LANE_00_SRCMAC  = 00:22:86:06:01:00
RSPDriver.LANE_00_SRCIP   = 10.211.1.1
RSPDriver.LANE_00_DSTMAC  = 00:19:99:ba:42:cb   # lofarB1  
RSPDriver.LANE_00_DSTIP   = 10.211.1.3

RSPDriver.LANE_01_BLET_OUT=8
RSPDriver.LANE_01_XLET_OUT=5
RSPDriver.LANE_01_SRCMAC  = 00:22:86:06:01:01
RSPDriver.LANE_01_SRCIP   = 10.212.1.1
RSPDriver.LANE_01_DSTMAC  = 00:19:99:e2:21:46   # lofarB2
RSPDriver.LANE_01_DSTIP   = 10.212.1.3

RSPDriver.LANE_02_BLET_OUT=4
RSPDriver.LANE_02_XLET_OUT=5
RSPDriver.LANE_02_SRCMAC  = 00:22:86:06:01:02
RSPDriver.LANE_02_SRCIP   = 10.213.1.1
RSPDriver.LANE_02_DSTMAC  = 00:19:99:e2:21:ba   # lofarB3
RSPDriver.LANE_02_DSTIP   = 10.213.1.3

RSPDriver.LANE_03_BLET_OUT=12
RSPDriver.LANE_03_XLET_OUT=5
RSPDriver.LANE_03_SRCMAC  = 00:22:86:06:01:03
RSPDriver.LANE_03_SRCIP   = 10.214.1.1
RSPDriver.LANE_03_DSTMAC  = 00:19:99:e2:20:ce   # lofarB4
RSPDriver.LANE_03_DSTIP   = 10.214.1.3
 

When you are done type "q" to exit less.

If the computers that the data is being sent to needs to be changed (depends on the whishes of the observer), you need tp copy the correct version of the file. As of 2013 May 13, current versions of the RSPDriver.conf file can be found at

~user9/StationConfigs/RSPDriver.conf.local_lofarAN_20130513      for lofarAN, copy that with:

cp ~user9/StationConfigs/RSPDriver.conf.local_lofarAN_20130513 /opt/lofar/etc/RSPDriver.conf

~user9/StationConfigs/RSPDriver.conf.local_lofarBN_20130423      for lofarBN, copy that with:

cp ~user9/StationConfigs/RSPDriver.conf.local_lofarBN_20130423 /opt/lofar/etc/RSPDriver.conf

less /opt/lofar/etc/BeamServer.conf

Check that the HBA update interval is reasonable.  For pulsar observations, where the noise from communicating with the HBA antennas is a bad thing, set this to a large value such as 300 seconds.  For other observations, check the reasonable default values

BeamServer.HBAUpdateInterval= 300

Start up the station to test LBA and HBA antennas

swlevel 1

This starts up the ServiceBroker.  Wait 10 seconds, then continue

swlevel 2

This starts up the RSPDriver and TBBDriver.  This takes quite a long time.  Follow the instructions for the next command to see when this finishes.

tail -f /log/RSPDriver.log

This continually checks the last log entries for the RSPDriver.

Wait till no more lines: "port ??? has not yet completed sync or had errors, trying to continue..." show up for at least 5 seconds.  This may take a minute or two.  Then enter <CTRL-C> to kill the tail program.

swlevel 3

This starts up the CalServer and BeamServer software.

beamctl --antennaset=LBA_OUTER --rcus=0:191 --rcumode=3 --beamlets=0:243 --subbands=100:343 --digdir=0,1.5708,J2000&

This starts up the beam software to make a beam pointing at zenith usign RCUMODE 3 (LBA, 10--30 MHz).  This will be used to test antennas.  Wait until the message "All pointings sent and accepted" appears (< 3 seconds).

(Need to fix the "stopped in background" issue!)

rspctl --rcu

This checks that the RCU boards have been set in RCUMODE 3.  Check that the RCUs are enabled (look for "=>  ON" for all RCUs).  Check that all RCUs are in RCUMODE 3 (look for "mode:3" for all RCUs).

If some RCUs are OFF or have the wrong RCUMODE, wait a few seconds and try again.  If the RCUs still are not correct, something is wrong. If that is the case first try going to "swlevel 0"  and back to this point. If it still doesn't work, call the Bonn people...

(We need to explain that better!)

rspctl --statistics  --integration=3

This checks the spectrum (0--100 MHz) for each RCU, with all RCU spectra plotted in a single gnuplot window.

When the program starts up, check the initial log report.  It should print out:

Taking subscription on the clockvalue
Current clockvalue is 200 Mhz
Taking subscription on the state of the splitter
The splitter is currently OFF
Taking subscription on the bitmode
The bitmode is currently 16

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!
 

rspctl --statistics=beamlet  --integration=3

This checks the beamlet spectrum, using the specified subbands.

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!

killpointing

This turns off the beam command. (If you hit <enter> or at the next command, you'll get a line that beamctl was killed.)

beamctl --antennaset=HBA_JOINED --rcus=0:191 --rcumode=5 --beamlets=0:243 --subbands=100:343 --anadir=0,1.5708,J2000 --digdir=0,1.5708,J2000&

This checks out the HBA tiles, making a beam pointing at zenith usign RCUMODE 5 (HBA, 110--190 MHz).  This will be used to test tiles.  Wait until the message "All pointings sent and accepted" appears (< 3 seconds).

rspctl --rcu

This checks that the RCU boards have been set in RCUMODE 5.  Check that the RCUs are enabled (look for "=>  ON" for all RCUs).  Check that all RCUs are in RCUMODE 5 (look for "mode:5" for all RCUs).

If some RCUs are OFF or have the wrong RCUMODE, wait a few seconds and try again.  If the RCUs still are not correct, something is wrong.

rspctl --statistics  --integration=3

This checks the spectrum (100--200 MHz, but incorrectly plotted by the software as 0--100 MHz) for each RCU, with all RCU spectra plotted in a single gnuplot window.

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!

rspctl --statistics=beamlet  --integration=3

This checks the beamlet spectrum, using the specified subbands.

(We need to put in a plot of the spectra!)

If the spectra looks O.K., then end by typing <CTRL-C> in the (text-)terminal. Due to a "feature" in the software this might produce lots of lines of output in the terminal, ignore that!

Check that beamlet data are arriving

Using another terminal, log into the recording computers and check that data are arriving.  For example, do

(use a VNC for all of this?)

ssh observer@lofarb1

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

ssh observer@lofarb2

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

ssh observer@lofarb3

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

ssh observer@lofarb4

sudo tcpdump -pni eth1

stop it with <CTRL-C>

exit

In the original terminal do:

Send the e-mail to ASTRON and our users that we have taken over the station

(We have to set up the script, so that the operators can run it.)

Turn the station over to the users

If a user is going to immediately use the station, you can leave it in swlevel 3 (nothing needs to be done in this case). Otherwise, put it in swlevel 0.

(Need to describe that better!)