Doing TBB and pulsar observations from lofarx

1) Prepare the shell on lofarx with a running ssh-agent:

• log in on lofarx with "ssh observer@lofarx" from a machine in the mpifr LDAP network (e.g. your desktop or portal or so.)
  • if it asks for a password, don't bother, it won't work. You can try running "kinit" on you local machine before doing the ssh.
• ssh-agent /bin/tcsh -l
• ssh-add
  • give passphrase

2) Boot up the station:

• log into the LCU with "ssh de601c"
• Check the destination of the beam-formed (pulsar) data:
  • grep lofar /opt/lofar/etc/RSPDriver.conf

  • The result should look like this for the lofarAN computers:

    RSPDriver.LANE_00_DSTMAC  = 00:30:48:34:74:5f   # lofarA1  
    RSPDriver.LANE_01_DSTMAC  = 00:30:48:34:74:73   # lofarA2
    RSPDriver.LANE_02_DSTMAC  = 00:30:48:34:74:69   # lofarA3
    RSPDriver.LANE_03_DSTMAC  = 00:30:48:bb:90:2f   # lofarA4

    and like this for the lofarBN computers:

    RSPDriver.LANE_00_DSTMAC  = 00:19:99:ba:42:cb   # lofarB1  
    RSPDriver.LANE_01_DSTMAC  = 00:19:99:e2:21:46   # lofarB2
    RSPDriver.LANE_02_DSTMAC  = 00:19:99:e2:21:ba   # lofarB3
    RSPDriver.LANE_03_DSTMAC  = 00:19:99:e2:20:ce   # lofarB4 

  • (The destination for the TBB data is chosen later byt thedump-script, and does not require change of the swlevel)
• swlevel 2
  • RSPDriver and TBBDriver need to be running
• tail -f /log/RSPDriver.log
  • wait till no more lines: "port ??? has not yet completed sync or had errors, trying to continue..." show up for several seconds
  • If anything strange appears, write it down in the observing logbook and tell the station team: lofar-ops@mpifr-bonn.mpg.de
  • end it with <CTRL-C>
• tbbctl --version
  • all 12 TBB-boards should show up with their firmware and software versions, if one is "board not active", then please contact the station team for a "48V reset"
• swlevel 3
  • In addition to RSPDriver and TBBDriver also BeamServer needs to be running
• beamctl --antennaset=HBA_JOINED --rcus=0:191 --rcumode=5 --beamlets=0:243 --subbands=100:343 --anadir=0,1.5708,J2000 --digdir=0,1.5708,J2000&
• beamctl --antennaset=LBA_OUTER --rcus=0:191 --rcumode=3 --beamlets=0:243 --subbands=100:343 --digdir=0,1.5708,J2000&
  • This should reach "All pointings sent and accepted" pretty fast.
  • hit <enter> to get a clean commandline
  • Wait several seconds for all tiles to start up
• rspctl --rcu
  • all 192 RCUs should be "ON" and in "mode:5"
  • if some are "OFF", wait a few seconds and try again
  • if some don't become "ON" then go bach to swlevel 1 and try again
• Check if the beam-formed data arrives at the correct computers:
  • in another window, log into the data recording-computer and run:
    • sudo tcpdump -pni eth1
  • you should see lots of packets arriving, if not see note 3) below
• rspctl --stati --int=3
  • check if the spectrum of all antennas look O.K.
• Check if the TBB-data correctly arrives at the correct computers:
  • in another window, log into the TBB-data recording-computer and run:
    • sudo tcpdump -pni eth1
  • then on the LCU again run:
    • setup_tbbs.sh
    • dumpall-lofarAN-sm.sh
    • (or dumpall-lofarBN-sm.sh)
  • you should see lots of packets arriving, if not see note 3) below
• killpointing
  • hit <enter> to get the confirmation that beamctl terminated

3) Observe

• Go back to lofarx
• Run one of the scripts:
  • tbb-psr-HBAsingle-LBA-observe.py                                 (Works for rcumodes 3, 5 and 7.)
  • tbb-psr-HBAsingle-LBA-observe-rcumode6test.py      (Should work for rcumode 6 and also 3,5 and 7.)
• The scripts should manage everything you need for a TBB dump with interleaved pulsar observations.  Call it with "-h" for help.
• You can put many calls to these scripts into one script to perform continous observations.
• During an observation you can check with "ps -fluobserver" or "top" if the data-writers are running.
• After an observation you can check the logfiles with "tail PSR_Logs/udpdump_pulsar.out" or "less /media/scratch/observer/TBB-dumps/TBBraw2h5.log"

4) Finish up:

• check with "ps -fluobserver" if there are lingering jobs on the lofarN machines. If there are, please kill them.
• shut down the hardware of the station with "swlevel 0"

Comments:

1. If at any time RSPDriver, TBBDriver, or BeamServer crash ("swlevel" show them as "DOWN") you need to go back to swlevel 1 and boot the station up again. (And note the fact in the logbook.)
2. If nothing fails, there is no need to change the swlevel on the station when observing multiple pulsars. Just leave it in "swlevel 3"
3. If you don't see any data coming with the "sudo tcpdump -pni eth1" command, then you can try "sudo tcpdump -ni eth1" (i.e. without the "-p" option). If you see the data then, this means that the destination-addresses for the data is set wrong. If you don't know how to fix this, ask someone from the station team.