VLBI - PV Support

January 2019

• DBBC3 installation: DBBC3 was shipped back to PV. First checks indicate that the system has survived the transport without damage. For the EHT dress rehearsal the DBBC3 will be used for processing the USB, whereas the R2DBEs will process the LSB.
• BDC firmware upgrade: The BDC firmware has been upgraded to 3.2a to allow remote monitoring and control.
• EHT preparation: for details see the EHT wiki page.

December 2018

R2DBE#2 Repaired of a broken power supply.

BDC 1(sn: 007) refurbished to the "230GHz Mode" with the hardware kit supplied by NOVA. Four bands 5-9 GHz are processed in two subbands 5-7 and 7-9 GHz.  BDC 2(sn:006) was shipped to Haystack.

October 2018

• Preparation of the 0.8mm test.

April 2018

• Preparation for the EHT session (see EHT-wiki for details).
• Configured automatic backup of mrt-vlbi (fs, home). See: mrt-vlbi-backup.txt
• Updated the LO-Calculator to include 5-9 GHz IFs (see above under General instructions)
• Updated the EHT schematics (see above under Overview)
• Updated the EHT related instructions for system setup and doing obervations (see above under General instructions)
• Switched to the new script by Pablo to write the CAL information of all 8 IFs to the FS logs. Call getcal in th FS.

Jan 2018

• Preparation for the EHT dress-rehearsal. Deployment of 64 Gbps R2DBE System was finished and tested (see EHT-wiki for details). Due to bad weather PV was unable to particiapte in the dress rehearsal.
• New maser was delivered in installed. Performance of the maser was verified agaist the Osilloquartz crystal and was up to the nominal performance as specified by T-Science.

29.3.2017

• GMVA schedules: downloaded all schedules (c171a,c171b,c171c,mg002,mb007,ma008) in the latest version to mrt-vlbi2. Drudged all schedules and produced snap & procedure files. The procedure files for the ALMA part (mg002,mb007,ma008) needed manual changes! Executed script /usr2/sched/dist_sched on mrt-vlbi to copy all vex,snp and prc files to mrt-vlbi, pv-mark6-1 and pv-mark6-2. On the mark6 machines translated the vex schedules to xml format to be run by the mark6 scheduler.
• Mark6: Installed new fans and ducts into all 4 Mark6 machines.

28.3.2017

• VLBI backup PC: Prepared backup vlbi-PC (mrt-vlbi2) to be used for the upcoming session. Field system and all relevant files were brought in sync with mrt-vlbi. The PC was moved out of the equipment room. For the upcoming session it is planned to use mrt-vlbi to run the field system only. All other control should be done on mrt-vlbi2.
• GMVA scripts: Two scripts now exist on pv-mark6-1 to do a short recording  and display the auto-correlations automatically.  For PFB mode use plotpfb_m6.sh; for ALMA mode use plotfull_m6.sh. Resiults are also saved under /home/oper/linecheck/DATE_TIME

23.3.2017

• DBBC3: Adjusted the config file of the DBBC3 to increase the target  IF power level to 37000. Verified that this still yields proper sampling statistics. Also shifted the configuration for the Valon frequency to 4GHz (will yield a true frequency of 8 GHz).
• Documentation: Updated the instructions page for DBBC3 operations.
• Documentation: Updated the instructions page for R2DBE operations.

20.1.2017 - 2.2.2017

• DBBC3 commissioning
• GPS antenna: a small GPS antenna was installed in the VLBI room (currently taped to the window). The GPS signal is split and serves the DBBC3 and FiLA10G for time synchronization. Despite the non-ideal location of the antenna GPS satellites were picked by the antenna at most times (but not always). It should be discussed with IRAM staff whether the boxes can be connected to the rooftop antenna.
• FiLA10G installation: In order to be ready for the DBBC2 fullband mode a FiLA10G box was installed. The internal FiLA10G card was removed and a second VSI cable was connected to the output board of the DBBC2. Both VSI cables from the DBBC2 have been connected to the FiLA10G box. The mark5b+ is now served by a cable from the FiLA10G VSI output connector.  The first ethernet port (eth0) of the FiLA10G was connected to eth2 of pv-mark6-2 by glass fibre. During the tests the FiLA10G has repeatedly shown problems, e.g. the network activity on eth0 has randomly stopped/restarted to work. The suspicion is that this is due to an overheating issue. As a test the top cover was removed which apparently could solve the problem. Possibilty to install an additional fan should be discussed with Michael.
• FiLA10 testing: Recorded data has shown "ghost lines" in the autocorrelations when recording on the Mark6. In a discussion with Gino it has later turned out that the wrong firmware version has been used for the test. After repeating with the correct version the lines have dissapeared.
• DBBC2 standard operations: Due to the installation of the FiLA10G there are changes in the way the DBBC2 is operated during the standard GMVA observations (in particular loading firmware and configuring the FiLA10G unit). The new procedure has been documented. A final test was done by doing a recording on the mark5b+ via the FiLA10G box with a line injected at 700 MHz. The line was recovered in the correct location in the autocorrelations in the DBBC2 flexmode (see line700_flex.png ) and the single pol modes (see line700_spol.png )
• Mark6 overheating: Two of the Myricon network cards have failed to work after a few days of continous operations. In both cases these were located at the far-out right side (close to the chasis) which makes overheating due to a bad air flow a likely cause. To investigate we have monitored the temperature of the interface board using an IR camera. As suspected the temperatures are around 85°C which is critcal (see IR picture). As a workaround we have unscrewed one of the backside fans ( the leftmost) and have installed it in the hole of the backplane separating the modules from the computer backside (see picture). In addition the two rightmost fans were unscrewed and tilted. This was done using cable binders (see picture). With these modifications the board temperature dropped significantly to about 42°C (see IR picture)

12.12.2016 - 16.12.2016

• Commissioning of R2DBE equipment.
• Field System: Upgraded Field System to Version 9.11.18. Prepared a Makefile that will now build all of the station code. Updated the instructions page.
• Phase noise: Measurement of the phase noise at 3mm. Results show around 30° phase jitter in both polarizations which is the nominal value.
• DBBC2: Full system verification test
• DBBC2: calibrated PFB V16 software.
• Mark5B+: Attempt to upgrade the OS from Debian Etch (V4) to Squeeze (V6) has failed. When trying to boot from an installation or live DVD the system hangs immediately. Attempts were repeated with 3 different DVD drives (internal, external USB and internal SATA) and 3 different installation DVDs. Succesfull Debian Squeeze (32bit) installation using pxe boot from pv-mark6-2. Updated to SDK9.4 and the latest jive5ab.
• Disk conditioning: Modules MPH%016, MPH%017, MPH%018, MPH%019, MPH%020, MPH%021 were conditioned with hammer. Information and plots were uploaded to the EHT wiki.

22.9.2016

DBBC: upgraded PFB firmware to version 16.

18.5.2016 - 21.5.2016

• DBBC: upgraded PFB firmware to version v15_1. Redid calibration and updated the configuration file with the new values (calibration_v15_1.pdf)
• DBBC: upgraded DDC firmware to version v105. Redid calibration and updated the configuration with the new values.(calibration_v105.pdf)
• Upgraded the field system to version 9.11.8
• Attempted to establish new hardware setup that uses an external FiLA10G box. Aborted efforts due to apparent hardware problems either with the FiLA10G unit or the VSI cables. The autocorrelation when using the external FILA10G box shows corrupted/distorted channels (FILA10G.pdf). When directly connecting the Mark5 recorder to the VSI-1 port of the DBBC the autocorrelation looks OK (directVSI.pdf). Removing and cleaning of the VSI cables has not solved the problem.

12.3.2015 - 20.3.2015

• Preparation of the EHT session. Various procedures have been written up on the EHT wiki public area (http://eht-wiki.haystack.mit.edu/Public_area)

13.12.2014  - 18.12.2014

• Mark6 recorders: 4 mark6 systems were shipped and installed in the VLBI racks. These are named pv-mark6-1 to pv-mark6-4. One of the systems (pv-mark6-4) was missing the complete set of miniSAS cables which are being send to Granada now but probably will not arrive in time to be installed during this mission. Salvador will install the cables once available. One issue discovered was that the order of the two Host Bus Adapter Cards is swapped with respect to the Haystack documentation. This means that 3&4 should go into the left-most connectors and 1&2 into the right-most. The correct order was verified by issuing:

   mod_init=1:8:ABC_0001:sg:new
  

  and then watching the bus activity on the disk modules. In this case LEDs should light up for module 1 (if module 3 is active the cables need to be swapped). Verification of the cabling was done for all mark6 systems except pv-mark6-4 due to missing cables. Basic verification of the general functionality was done (forming groups, mounting groups, closing groups, module initialisation etc.).
• Mark6 test recording: Recording with mark6 recorders was verified to work (pv-mark6-1 to pv-mark6-3). Due to the missing miniSAS cables on pv-mark6-4 recording could not be tested on that machine. Verification will be done by Salvador once the cables have been delivered and installed. Running the mark6 software turned out to be workable but not very well suited for regular operations yet. Some issues have been identified and communicated to Chet. The new Mark6 software release 1.2 has not arrived in time to be installed and tested yet. Mark6 control from the field system (making use of the mk5cn) module has been tried but was unsuccessful.
• Mark5B+ OS upgrade: The OS (Debian etch) on pv-mark5b-1 turned out to be largely corrupted (damaged package lists etc.) which prevented installation of the latest streamstor/dimino debian packages. A fresh installation of Debian 4rev9 (=Etch) was done on that machine.
• Mark5B+ streamstore SDK & mark5 software upgrade: The latest packages:

mark5_2.2.1-i386.deb
mk5bio_1.0.6-i386.deb
streamstor_9.2.1-i386.deb

were obtained from the Haystack website and were installed following the Haystack documentation. Following the installation the latest SDK firmware was flashed onto the Amazon board using:

ssflash -u SDK9.2.ssf

The xbar version was verified to be 2.038.Using a 16TB module it was confirmed that large modules are accepted and can be filled to 100% when recording.

• Mark5C software upgrade: The latest drs software package (0.9.19-1) was installed on pv-mark5c. Verified correct version of Xbar firmware (2.038) on the amazon board.
• jive5ab installation: jive5ab version 2.5.1 was installed on pv-mark5b-1 and on pv-mark5c-1. Wrote small client program (jive5abclient) to talk to jive5ab without the FS. Verfified basic operations together with FS (recording on/off scan_check etc.).
• Field system upgrade on mrt-vlbi: Field system on mrt-vlbi was upgraded to version 9.11.6. Minor modifications needed to be done to equip.ctl due to new DBBC features in the FS.
• Flexbuf: Two 256 GB SSD disks were installed in pv-mark6-4 to build up a small flexbuf system for automated parallel recording of scans that will be used for fringe checks in the beginning of a session. the vlbi_streamer deamon was compiled successfully and is configured to start up at boot time. The 10Gbs network port of pv-mark6-4 was connected to the eth0 port of the FILA10G board. FILA10G was then configured to send test data (vdif format) to the eth2 port of pv-mark6-4. The data stream was succesfully recorded at 2 Gbps and 4 Gbps. The packet loss rate @ 2Gbps was marginal (0.003%) but significantly increased @4Gbps (12%).  For details of the installation and testing see: Notes.
• Create backup field system computer: A secondary field system computer (mrt-vlbi2) has been set up. It is meant to serve as a second FS in case we will be do parallel observations or to serve as a backup in case the normal field system computer (mrt-vlbi) will break during/close to the session. Installation was done using OpenSuse13.2. The latest field system was installed (9.11.6) and tested using a fake schedule. Station code was confirmed to work. Sending of the source string to the telescope system is only allowed from mrt-vlbi in order to prevent dual reception of a source command.
• DBBC calibration: Calibration for the latest DDC  firmware (104) was done and the appropriate values were inserted in the configuration file. Unclear why the peaks go into saturation (report to Gino). Calibration plot  for v104. Calibration was repeated for DDC v102. Results are consistent with the last calibration. Slightly improved Values were inserted in the configuration file.

• Upgrade DBBC DDC firmware to version 104: The firmware was obtained from the HatLab website and installed onto the DBBC. Calibration was performed (see above). The  links on the DBBC desktop were updated accordingly.  The DBBC version in the field system configuration was set to v104 (equip.ctl)
• Upgrade of the FILA10G firmware to version 3.3.2_0: Latest firmware received by Gino and installed on the DBBC. This version was activated by default when starting DBBC versions: DDC102, DDC104 or PFBv14.
• Update of PV Station page
  

   

19.9.2013 - 26.9.2013

• Changed the default recorder for the Field System to be Mark5C (pv-mark5c-1). This was done because of the 16TB modules delivered to PV for the upcoming session. The Mark5B+ can currently only record up to 8TB due to the outdated SDK (8.2). Wrote two scripts (snap2mark5c, proc2mark5c) that transform the snap and proc files created by drudg into mark5c format. The scripts are located in /usr2/sched and /usr2/proc.
• Updated the DBBC DDC firmware to the latest version v104_2. However it has turned out that this is too recent for the FS 9.10.5. Therefore the DBBC DDC firmware v102 was installed on the DBBC and will be used for the Sep13 session.
• DBBC calibration was performed for the DDC v102 firmware and the PFB v14 firmware. The corresponding control files were updated with the new calibration values.
• Verification that the DBBC on PV has ADB1 boards installed. Fixed an error in the DBBC configuration for the PFB v14 firmware which had assumed ADB2.

10.5.-16.5.2013

• Salvador has updated equip.ctl. rack type is now correctly set to dbbc. FS now properly recognizes and executes the bbcXX commands.
• Updates the time.ctl file with a new rate, following the procedure outlined in /usr2/fs/misc/fstime.txt. This has solved the issue of the FS time running away quickly.
• Configured and tested the autoftp feature of the FS. An autoftp script was created under  /usr2/fs/autoftp/autoftp that will handle the transfer to the Bonn FTP server. Installed ncftp onpv-mar5b-1 (required by autoftp). Setup password-less login from the FS computer to the mark5b machine for user oper.

17.3.-22.3.2013

• Recording test with Mark5C failed (corrupted data). Jan Wagner suggested to load different firmware for the amazon daughterboard (see notes). Afterwards recording was successful (tested with m5test).
• Wrote scripts to automatically download and drudg all vex schedules on mrt-vlbi and pv-mar5c-1.See instructions for usage.
• Prepared instructions for extracting data from a  scan recorded on mark5c and mark5b for ftp fringe test.

12.2.-19.2.2013

Rottmann travels to Pico to support  the 2GB/s test experiment (TW004) and to prepare the system for the upcoming 4GB/s 1mm session in march 2013.

The following work was done on this trip:

• a new Mark5c recorder has been shipped to Pico Veleta and was installed in the VLBI-rack. The recorder was setup with the latest SDK (9.2), DRS and other auxillliary packages (see notes )
• The internal FILA10G board already present (but disconnected) was connected to the DBBC (see notes).
• The Mark5C was connected to the FILA10G card by optical fibre / Glapper / Copper.
• The PFB firmware on the DBBC was upgraded to the latest version (v14).
• The FILA10G firmware was upgraded to the latest version v2.0 (see notes)
• The latest documentation (DBBC hardware, PFB, FILA10G) was obtained from HatLab, printed out and stored in the DBBC folder in the VLBI room.
• The new field system computer brought up in 2011 was finalized and used for the 2GB/s recording on 14.2.
• The field system was upgraded to 9.10.5. DBBC support was confirmed and tested. PV station code was recompiled and verified to work.
• Test recordings were done with the Mark5C and Mark5B+
• After the 2GB/s test 8s of data were transfered to Bonn. Correlation done by Bertarini confirmes fringes between Pv, Ef and On (see fringe plot)

• The actual procedure for the upcoming  4Gbps experiment in march using 2 recorders was established and verified to work. Step-by-step instructions were documented (see document).

23.1.2012

Fringes with PV DBBC found. Project phase 1 finished.

24.11. - 25.11.2011

Rottmann and Wunderlich travel to Pico. The following work was done:

• Replace defective boards in the DBBC. Verified functionality of the DBBC.
• OS upgrade on the Mark5A to Debian Etch (2.6.18 kernel)
• Upgrade of the Mark5A software on the Mark5A computer.
• SDK upgrade to version 8.2  on the Mark5A system.
• Installed field system version 9.10.4 on the new and the old field system computers. New field system computer tested.
• Perfomed test observations with Effelsberg  @2Gbps PFB mode. Due to extremely bad weather only 5 minutes of good data.

30.9.2011

Rottmann travels to Pico. The following work was done:

• Installed the DBBC hardware. Stability and reliabilty not optimal. Must be improved.
• Brought and installed a new field system computer. No time to fully test. Recomend to stay with the old system for now.
• Plans to upgrade SDK version on the Mark5A recorder postponed due to oudated kernel version. OS needs to be upgraded first to more recent version.