NOEMA April 2019

    Temporary page for filling in details on the NOEMA April 2019 setup.

    Observing log:

    f191a (sched files)

    • weather about 3-4mm PWV
    • f191a pretest 094-1150 : ok
    • 094-1200 : 2013+370 : adjustment of R2DE clk power level 2.0 dBm to 2.5 dBm just before scan, seems to have distrupted R2DBE so it reports huge clock offset; NOEMA correlator also suffered a hang-up; scan probably not good; re-ran start_backend.py and M6_CC after this scan
    • 094-1211 : 2013+370 : started ok : vdiftimeUDP ok, recording ok, power levels slightly high i.e. a bit of clipping, adjusting analog levels down after the scan : relatively good
    • 094-1218 : 2013+370 : ok, powers levels not adjusted yet, local fringes most stable 12:11:40--12:18:10, good
    • 094-1231 : sun avoidance : recorded but no source
    • 094-1237 : sun avoidance : recorded but no source
    • 094-1248 : sun avoidance : recorded but no source
    • 094-1255 : switched to R-Cas SiO : recorded, first 20sec are off-source, rest are ok, local test scan

    f191b (sched files)

    • weather better, stable atm phase, much better than 2 hours ago
    • 094-1330 : ok, also r2dbe levels
    • 094-1345 : ok but 45 sec late
    • 094-1401 : not observed, operator needs to go to antenna to clear out water and 1cm of snow
    • 094-1415 : ok, snow cleared
    • 094-1430 : ok
    • 094-1445 : started 1min 20sec late
    • 094-1500 : started 1min 30sec late
    • 094-1517 : on time, ok
    • 094-1530 : started 55sec late
    • 094-1545 : started 12sec late

    FTP transfer log

    FTP scan 094-1345 timerange 13:46:20 to 13:46:30, from scan start 13:45:00 i.e. 80 seconds into scan. The VDIF frame rate is 125000 frames per second, frame size 8224 byte. Copy 12 seconds instead of 10 for some extra.

    oper@Mark6-4124:~$ dd if=/mnt/fuse/34/f191b_Na_094-1445.vdif \
       of=/home/oper/f191b_Na_094-1445.if1.vdif.v2 \
       bs=8224 count=$((12*125000)) \
       skip=$(( (60+20) * 125000 ))
    
    [oper@cc-noema zerobaseline]$ m5time recorder1_oper/f191b_Na_094-1445.if1.vdif.v2 VDIF_8192-8192-1-2
    MJD = 58577/14:46:20.00
    
    oper@Mark6-4124:~$ vdifcontinuitycheck.py /home/oper/f191b_Na_094-1445.if1.vdif.v2
    Thread 1 Second 8088380 : 124925 frames : #0--#124999 : 75 lost, 0 out-of-order, 75 invalid, 0 dup, of 125000 total
    ...
    Thread 1 Second 8088390 : 124774 frames : #0--#124999 : 226 lost, 0 out-of-order, 226 invalid, 0 dup, of 125000 total
    
    [root@cc-noema VLBItxfrToGre]# sshfs oper@recorder1:/home/oper/ /root/recorder1_oper
    [root@cc-noema VLBItxfrToGre]# cp -v /root/recorder1_oper/f191b_Na_094-1445.if1.vdif.v2 /VLBItxfrToGre/f191b_Na_094-1445.if1.vdif
    '/root/recorder1_oper/f191b_Na_094-1445.if1.vdif.v2' -> '/VLBItxfrToGre/f191b_Na_094-1445.if1.vdif'
    
    

    Second scan for hand-carry: 094-1517  + 1min for 20sec plus a bit, use oposite if0 (slot 12):

    oper@Mark6-4124:~$ dd if=/mnt/fuse/12/f191b_Na_094-1517.vdif \
       of=/home/oper/f191b_Na_094-1517.if0.vdif \
       bs=8224 count=$((21*125000)) \
       skip=$(( 60 * 125000 ))
      
    [oper@cc-noema zerobaseline]$ m5time recorder1_oper/f191b_Na_094-1517.if0.vdif VDIF_8192-8192-1-2
    MJD = 58577/15:18:00.00
    
    [jwagner.JW-X230t]> scp -v ehtcc:/home/oper/zerobaseline/recorder1_oper/f191b_Na_094-1517.if0.vdif .
    
    
    Further scans to hand-carry on one disk on the first module:
    location /mnt/disks/1/0/handcarry/if{0,1} = /dev/sdb1 with serial number ZA25Z3VG
    has the first 6 out of 10 scans if f191b in dual pol
    root@Mark6-4124:/home/oper# mount |grep sdb1
    /dev/sdb1 on /mnt/disks/1/0 type xfs (rw)
    
    root@Mark6-4124:/home/oper# udevadm info --query=all --name=/dev/sdb1 | grep ID_SERIAL
    E: ID_SERIAL=ST10000NM0016-1TT101_ZA25Z3VG
    E: ID_SERIAL_SHORT=ZA25Z3VG
    
     
     

    1. Systems Check

    1.1 RF-over-Fiber to R2DBE

    The RFoF receiver and IF processor unit currently receives Antenna 4, 2 polarizations x IF 4-12 GHz.

    The RFoF has an internal dual-LO chain (9.0 GHz +5.4 dBm,  4.0 GHz +16.9 dBm) that splits the RFoF signal into a selected 2 pol x 2 sideband x 2 GHz. In case of April 2019, the RX LSB is mixed LSB(1) LSB(2) so is Net LSB.

    R2DBE receives lower 2 GHz from boths pols into if0, if1.

    R2DBE power levels were checked to be within +-1 dBm of ideal after semi on-sky test; antenna stowed pointing at horizon due to snowfall i.e. ground contribution to Tsys, levels likely to drop a bit when actually on sky.

    R2DBE 2048 MHz clocking and the dual-LO clocking is somewhat adventurous. Timetech freq dist amp (in: H-maser 5 MHz) in the correlator room, one output sent to a stack of three synthesizers (R&S SMA 100B for 9 GHz, SMA 100A for 4 GHz and 2.048 GHz) with ext.ref. from this 5 MHz line shared by BNC-T. Oscilloscope shows distortion of sine. Spectrum analyzer shows still good phase noise. 

    1.2 GPS/Maser 1PPS

    R2DBE : HB_sma <-- GPS 1PPS; 1pps_sma <-- H-maser 1PPS

    GPS--Maser 1PPS offset currently +53.28 milliseconds by oscilloscope.

    Internal r2dbe post-sync 1pps vs. external GPS 1PPS very stable, zero to a few 256 MHz clock cycles.

    1.3 LO offsets

    On the RFoE receiver side in the correlator room it is readibly apparent from the synthesizer settings that there are no LO offsets.

    The 1st LO tuning is based on a R&S SMA 100A synthesizer in the correlator room. Its output feeds the "NOEMA LO system III" which is a coax-based LO distribution system to the ndividual antennas, and compensates for coax temperature etc related delay/phase changes.

    Olivier has taken care that there are no potential rounding-error induced LO offsets for the 1st LO synthesizer. Tested were 1823.0 MHz in engineering tests. And 1910.500 MHz chosen for GMVA VLBI and tested and verified, including inspection of freq setting on synthesizer screen when commanded by the NOEMA system. Hence it is extremely likely that there is no 1st LO offset

    1.4 Walsh switching

    The Walsh switching cycle is 32 Hz. New phases are loaded aligned to H-Maser 1PPS rather than GPS 1PPS due to drift/offset issues of GPS 1PPS relative to H-maser -derived frequencies.

    Olivier & co confirmed on the NOEMA LO System III bench directly on the inputs&outputs that for the selected Reference antenna (in VLBI it is Antenna 4) the Walsh switching is stationary (walshing is essentially off) and does not alter the phase of the reference antenna.

    1.5 Doppler correction

    Need to double check with operators but Doppler was off during Oct 2018 and is known required to be off for VLBI.

    The LO synthesizer in the H-maser housing is not connected to the Doppler control system, hence when the correlator room LO synthesizer output is swapped against the H-maser room LO synthesizer, it is guaranteed that Doppler is off.

    1.6 Coherence test by tone injection

    Cannot test full RX chain coherence. Not possible at the moment at any antenna, also not the reference antenna. Considerations to in the future perhaps to have either a beacon system for a sky-freq tone for the whole array. Or injection at one antenna. Currently tone injection into 1st IF after the receiver is technically not possible. A further issue is how to transport the necessary "pristine" 5 MHz maser standard to the reference antenna instead of using derived frequencies from the phase/delay-compensated 1st LO distribution system.

    Tested however a partial coherence test was possible: LO2/LO3/R2DBE -chain coherence test. This does not verify "NOEMA LO System III" behaviour of 1st LO distribution to the telescopes i.e. receiver coherence and whatever the phase rotator might be introducing. 

    coherence test synthesizers forTone LO2 LO3 Clk.jpg coherence test RFoF analog tone coupling.jpg coherence test R2DBE spectrum.png m5tone_example.png

    Coherence test setup 04April2019:

    • R&S SMB 100A synthesizer used for a tone at 5.512 GHz -30 dBm
    • Tone coupled at IF1-level into the analog output of the RFoF receiver in the VLBI rack.
    • Noise comes from the actual antenna signal also present.
    • R&S SMA 100B for 9 GHz LO2
    • R&S SMB 100A for 4 GHz LO3
    • Marconi 2024 for R2DBE 2048 MHz (not enough SMB 100A synths available so had to use this one)
    • Recorded 30 seconds on Mark6 from R2DBE if0 & if1
    • m5tone.py on EHT-CC server used to extract tone phase and tone coherence
    • 100% coherence in 10 milliseconds
    • -120deg average phase with some erratic drifts within +-5 deg over the 30 sec

    Recordings:

    • tone512M : brief 30 sec recording for testing m5tone.py
    • tone512Mnoref.vdif : 30 sec recording with 5 MHz ExtRef disabled for tone synthesizer
    • tone512Mlong.vdif : 15 minutes recording to catch room thermal effects (7min cycle), 5 MHz is shared via BNC-T'ees
    • tone512Mdaisy5MRefShort.vdif : brief 30 sec recording, all 5 MHz refs chained via synthesizer Ext Ref In-->Ext Ref Out
    • tone512Mdaisy5MRef.vdif : 15 minutes recording with all 5 MHz, 5 MHz chained via synthesizer Ext Ref In-->Ext Ref Out

    1.7 H-maser rate

    Michael Bremer has checked the 1pps offset data of 30 past days.

    The rate is linear and approximately +4.52e-15 sec/sec.

    (TODO: plot)

    2. Pending checks

    1. Run DiFX correlation on the 30sec dual-pol recording: ok

    2. Brief observation of an SiO maser source from the VEX schedule to make sure the R2DBE backend data contains the maser lines i.e. have correct sideband selections in all three mixing stages: schedule f191a.vex has 

    1. P-V1111OPH   -32.00 km/s LSR
    2. P-RCAS    ( SIO-RCAS )  +24.00 km/s LSR

    but these are scheduled to be observed only with the VLBA and only the 43 GHz SiO lines. Perhaps http://www.apex-telescope.org/observing/pointing/spectra/ something falls into the 86 GHz observed band (?)

    04Apr2019 at end of f191a got a ~4min scan on R-Cas. Line width likely 10 km/s for the v=1 J=2->1 86243.37 GHz transition (1998A%26A...329..219P), line width in the spectrum then about 3.3 MHz (rule of thumb).

    Spectrum was confirmed in the data: ok!

    3. Michael Bremer has suggested a 1st LO synth output comparison. One synth ("VLBI synthesizer"; R&S SMA 100A) is located jointly with the H-maser in a climate controlled housing and has about 10-15 meters of cable run to the correlator room to the LO Distribution units. Next to the LO Distribution sits another 1st LO synthesizer for normal NOEMA operation (R&S SMA 100A) that receives maser 5 MHz over a similar 10-15 meter cable run. Bremer suggests mixing both LO (with small tuning offset) to see thermal stability. Given that 5 MHz cable and "VLBI" LO cable run the same below-floor-tile route the comparison will probably compare phase stability of synthesizers between climate controlled vs correlator room aircon cooled.

    3. System setup details

    3.1 R2DBE setup

    Clock  2048 MHz clk, 0 +- 3 dBm 50 ohm

    Driven by R&S SMB 100A that is 5 MHz H-maser locked.

    "HB" SMA input from GPS 1PPS

    "1PPS" SMA ref input from Maser 1PPS. Connected in order to utilize internal counter-comparator in addition to separate external 1PPS GPS-MASER unit. This functionality seems however to have issues in the new r2dbe software (?).

    3.2 EHTC Control Computer (re-)customizations

    Enabled ip4 NAT/masquerade to allow Mark6's to use further NTP, and access the internet for installing DiFX for zero baseline tests.

    Added jwagner as publickey-only auth (/etc/ssh/sshd_config section like "Match User daan").

    Installed DiFX-2.6 under /home/oper/difx/ in order to have m5tone, m5spec.py and source code of some important VDIF utilities (fuseMk6, vdiftimeUDP) that can be compiled on Mark6.

    The /home/oper/bin/vex2xml.py is broken for python 2.7. Modified it.

    Initially, reverted back to the R2DBE scripts from SAO-EHT https://github.com/sao-eht rather than Radbout written-from-scratch "vlbicontrol". We have not tested it, especially not for GMVA, and no earlier experience since neither NOEMA nor APEX were in the EHT dress rehersal. Testing 03Apr2019 showed that r2dbe_monitor.py does not work.

    Switched then to the new R2DBE scripts. Needed a new config file for it. Here, r2dbe_monitor.py works.

    Files: dnsmasq.conf ntp.conf vex2xml.py m5tone.py

    3.3 Mark6 setup

    Installed vdifstream git repository utilities:
    vdiftimeUDP   continuous monitoring of realtime VDIF timestamp vs UTC/NTP
    vdifsnapshotUDP  burst mode capture to memory then disk without needing dplane/cplane setup
    vdifcontinuitycheck.py analyse VDIF file for continuity, thread misalignment, lost/duplicate/reordered frames
    fuseMk6
    (/mnt/fuse/mount_all.sh)
    mount the disk module as readable proper gatheredVDIF files
     
    Installed Mark6 disk spin-down prevention script (fixed for Debian Squeeze on NOEMA Mark6's: disk-toucher.sh) and CRON job as described in https://eht-wiki.haystack.mit.edu/Event_Horizon_Telescope_Home/Observing/0_General/Recorders/Disk_Module_Spin-Down_Prevention.
     

    4. Practical details for future NOEMA internal reference

    4.1 FUSE-Mounting Mark6 modules for reading VDIF files

    oper@recorder1:~> sudo mkdir -p /mnt/fuse/12
    oper@recorder1:~> sudo mkdir -p /mnt/fuse/34
    oper@recorder1:~> sudo chown oper:oper /mnt/fuse/12
    oper@recorder1:~> sudo chown oper:oper /mnt/fuse/34
     
    oper@recorder1:~> fuseMk6 -r "/mnt/disks/[1-2]/[0-7]/data/" /mnt/fuse/12
    oper@recorder1:~> fuseMk6 -r "/mnt/disks/[3-4]/[0-7]/data/" /mnt/fuse/34
    oper@recorder1:~> fusermount -u /mnt/fuse/12
    oper@recorder1:~> fusermount -u /mnt/fuse/34
     
    Note, a lot of utilities (e.g., m5spec, m5tone) could not be installed on the Mark6 because the ancient Debian Squeeze archive key is expired and keys cannot be refreshed by apt-key, so cannot install new packages such as the FFT library fftw  (maybe there is a way to get apt-get working again?).

    4.2 Access on EHT-CC server to Mark6 FUSE-mounted modules

    [root@cc-noema ~]# pacman -S sshfs
    [oper@cc-noema zerobaseline]$ mkdir -p zerobaseline/recorder1_fuse_12
    [oper@cc-noema zerobaseline]$ mkdir -p zerobaseline/recorder1_fuse_34
     
    [oper@cc-noema zerobaseline]$ sshfs recorder1:/mnt/fuse/12/ /home/oper/zerobaseline/recorder1_fuse_12/
    [oper@cc-noema zerobaseline]$ sshfs recorder1:/mnt/fuse/34/ /home/oper/zerobaseline/recorder1_fuse_34/
    [oper@cc-noema zerobaseline]$ m5spec recorder1_fuse_34/test_Na_tone512M.vdif VDIF_8192-8192-1-2 65536 2000 r2dbe_if1_65kpt.spec
    [oper@cc-noema zerobaseline]$ gnuplot
    > plot "r2dbe_if1_65kpt.spec" using 1:2 with lines
     

    4.3 Correlating a zero baseline test 2-pol recording

    Scripts for DiFX zero baseline correlation of two VDIF files are under https://bitbucket.org/jwagner313/apex-tools in Correlation/DiFX_zerobaseline_generator/r2dbe_r2dbe/. The scripts use the program 'vsum' from DiFX library vdifio, DiFX vex2difx, and DiFX mpifxcorr. Additionally, HOPS fourfit is required for the fringe fitting and fringe plot.
     
    Alternatively can use DiFX zerocorr (todo)

    4.4 Coherence test tone phase extraction

    Tone phase extraction script m5tone.py is part of DiFX library mark5access Python extensions and its examples.
    Prerequisite libraries, installation:
    [root@cc-noema ~]# pacman -S python-numpy
    
    [oper@cc-noema python]$ cd /home/oper/difx/svn/libraries/mark5access/python
    [oper@cc-noema python]$ python setup.py build
    [oper@cc-noema python]$ python setup.py install --prefix /home/oper/difx --install-lib=/home/oper/difx/lib/python/
    [oper@cc-noema python]$ cp examples/*.py $DIFXROOT/bin/
    Then with a test recording make 10 millisec integrations with tone expected at 512 MHz and using 8192k-point FFT:
    [oper@cc-noema zerobaseline]$ m5tone.py recorder1_fuse_34/test_Na_tone512M.vdif VDIF_8192-8192-1-2 \
        tone512m.m5s 1   0.010 512e6 $((8192*1024))
    Tone at 512000.000 kHz in the 2048.000 MHz wide band lands in 0.488 kHz-wide bin 1048576.
    Integrating for 10.24 milliseconds with 5 spectra per integration.
    58576.907001684 mjd : 0.005120 sec : 3.327136e-02 /_ -120.35 deg : r=1.000
    58576.907001802 mjd : 0.015360 sec : 3.352154e-02 /_ -120.04 deg : r=1.000
    58576.907001921 mjd : 0.025600 sec : 3.325227e-02 /_ -120.06 deg : r=1.000
    58576.907002039 mjd : 0.035840 sec : 3.340412e-02 /_ -119.87 deg : r=1.000
    58576.907002158 mjd : 0.046080 sec : 3.321879e-02 /_ -120.00 deg : r=1.000
    ...
    
    # Plot phase and coherence against time
    [oper@cc-noema zerobaseline]$ gnuplot
      set yrange [-185:185] ; set y2range [0:1.05] ; set xlabel 'Time (s)'
      set ylabel 'Tone phase (deg)' ; set y2label 'Coherence'
      plot "tone512m.m5s" using 2:4 title 'tone phase' axes x1y1, \
         "tone512m.m5s" using 2:5 title 'tone coherence' axes x1y2    
    
     
    [oper@cc-noema zerobaseline]$ octave
    Tint = 10.24e-3;  % set it to the integration time used during m5tone.py
    fname = 'tone512m.m5s';
    dd = dlmread(fname);
    phases = dd(:,4); P = abs(fft(phases)); PfirstNyq = P(1:floor(numel(P)/2));
    FfirstNyq = linspace(0, 1/Tint, numel(PfirstNyq-1));
    startbin = 1; % keep DC?
    %% plot(FfirstNyq(startbin:end), PfirstNyq(startbin:end)); grid on;
    semilogy(FfirstNyq(startbin:end), PfirstNyq(startbin:end)); grid on;
    xlabel('Frequency (Hz)'); ylabel('Amplitude of phase-rate');
    title(['Fourier transform of ' num2str(numel(phases)) ' phase data points each at ' num2str(Tint*1e3,'%.2f') 'msec integration']);
     
    Tone phase and coherence against time in 10.24ms APs for 30sec Spectrum of 10.24ms AP phase time series (phaserate spectrum)
    m5tone_example.png m5tone_30sec_512M_extRef_FFTphase.png
    # Plot phase for BNC-T shared 5 MHz and phase of Ext RefOut shared 5 MHz against time
    [oper@cc-noema zerobaseline]$ gnuplot
      set yrange [-185:185] ;  set y2range [-185:185] ; set xlabel 'Time (s)'
      set ylabel 'Tone phase BNC-T (deg)' ; set y2label 'Tone phase RefIn/Out chain (deg)'
      plot "tone512m.m5s" using 2:4 title 'tone phase BNC-T' axes x1y1 pointtype 1, \
         "tone512Mdaisy5MRefShort.m5s" using 2:4 title 'tone phase ExtRefOut' axes x1y2 pointtype 1

     
    Tag page (Edit tags)
    • No tags
    Pages that link here
    Page statistics
    1594 view(s), 88 edit(s) and 24177 character(s)

    Comments

    You must login to post a comment.

    Attach file

    Attachments

    FileSizeDateAttached by 
     coherence test R2DBE spectrum.png
    No description
    65.55 kB01:47, 4 Apr 2019jwagnerActions
     coherence test RFoF analog tone coupling.jpg
    analog tone 5.512 GHz coupled after RFoF RX into the 4-9 GHz IF
    110.19 kB01:41, 4 Apr 2019jwagnerActions
     coherence test synthesizers forTone LO2 LO3 Clk.jpg
    synthesizer config for coherence test with post RFoF RX tone coupling
    456.92 kB01:44, 4 Apr 2019jwagnerActions
     dhclient.conf
    dhcp client config for NOEMA Mark6's
    419 bytes13:25, 4 Apr 2019jwagnerActions
     disk-toucher.sh
    fixed copy of Mark6 disk module spin-down prevention script from EHTC
    400 bytes09:52, 4 Apr 2019jwagnerActions
     dnsmasq.conf
    dnsmasq config for NOEMA EHT CC
    2.43 kB00:16, 4 Apr 2019jwagnerActions
     fuse_mount_all.sh
    fuseMk6 mounting of all modules on Mark6
    315 bytes13:27, 4 Apr 2019jwagnerActions
     IMG_6711_w.jpg
    Three LO as used for RFoF Rx and R2DBE
    173.48 kB21:30, 3 Apr 2019jwagnerActions
     IMG_6716_w.jpg
    5 MHz at end of BNC-T chain, 500 kHz BW, with a spurious 90 dB down from 5 MHz at about -93 kHz offset
    298.79 kB21:43, 3 Apr 2019jwagnerActions
     m5tone.py
    m5tone.py fixed for Python 2.7/3 and EHT-CC pythonpath
    6.97 kB01:06, 4 Apr 2019jwagnerActions
    m5tone_30sec_512M_extRef.png
    full 30 seconds of data, tone extracted at 10ms integrations, tone synthesizer on Ext Ref
    69.09 kB09:46, 4 Apr 2019jwagnerActions
     m5tone_30sec_512M_extRef_FFTphase.png
    FFT of phase data from tone512m_BNC_Tees.m5s
    112.67 kB10:49, 4 Apr 2019jwagnerActions
     maser 5MHz sharing between synths.jpg
    maser 5 MHz sharing between synths by BNC-T
    267.96 kB02:11, 4 Apr 2019jwagnerActions
     ntp.conf
    NTP config for NOEMA EHT-CC
    572 bytes00:11, 4 Apr 2019jwagnerActions
     r2dbe1_monitor-03apr2019-13h18m.png
    r2dbe levels
    69.69 kB21:44, 3 Apr 2019jwagnerActions
     r2dbe1_monitor-03apr2019-21h45m.png
    r2dbe levels after horizon-pointing stow and RFoF adjust
    95.7 kB21:46, 3 Apr 2019jwagnerActions
     tone512m_BNC_Tees.m5s
    tone extracted of 30sec scan with tone at 512MHz and synths run off BNC-T -shared 5 MHz
    145.88 kB10:23, 4 Apr 2019jwagnerActions
     vex2xml.py
    vex2xml.py fixed to work in Python 2.7/3 of EHT CC
    23.64 kB00:09, 4 Apr 2019jwagnerActions
     vlbi rack.JPG
    VLBI rack
    2.65 MB02:14, 4 Apr 2019jwagnerActions