C212A

Data completness

• Ef: missed scans: No0038, No0039, No0041, No0042, No0046
• Mh missed scans: No0001
• Ys: No0008 - No0014 corrupted data files

Ky: File /data/c212a/ky/c212a_KVNYS_274071501.vdif VDIF summary failed with return value -5

Kt: File /data/c212a/kt/c212a_KVNTN_274070001.vdif VDIF summary failed with return value -5

NOEMA: LCP time shift

Timeserver on one of the polyfix units was not running leading to a timeshift between LCP and RCP in the beginning of the session including the fringe test. The issue was fixed on DOY 274  09:51:40.0 UT. Last affected scan was No202 of c212a.

GBT: corrupted data

Issue: Strong GBT fringes in LCP found but no detection in RCP (tested on scans 189 and 191).

Diagnostics: Inspection of the VDIF structure shows 8 threads as expected. However the threads with odd IDs (representing LCP channels) are offset considerably (> 150 frames). The suspicion was that RCP frames possibly get dropped by DiFX due to these gap sizes exceeding the read buffer sizes (even though this seems unlikely because of the even-threaded data yielding fringes).

Test 1:

To check whether the RCP data in principle is OK the v2d setup for GBT was split into two separate datastreams; one for LCP and one for RCP:

ANTENNA GB {
   datastreams=GBRCP,GBLCP
}
DATASTREAM GBRCP {
    file=/mark6-06_fuse/1/C212A_GB_No0762
    format=INTERLACEDVDIF/0:2:4:6/5032/2
}
DATASTREAM GBLCP {
    file=/mark6-06_fuse/1/C212A_GB_No0762
    format=INTERLACEDVDIF/1:3:5:7/5032/2
}

Correlation still yielded no RCP fringes (all WEIGHTS 0) with .difxlog summary:

mark6-06 INFO  VDIF multiplexing statistics: nValidFrame=153 nInvalidFrame=0 nDiscardedFrame=0 nWrongThread=18421482 nSkippedByte=0 nFillByte=0 nDuplicateFrame=0 bytesProcessed=92697667320 nGoodFrame=38 nCall=8661
mark6-06 INFO  VDIF multiplexing statistics: nValidFrame=9938486 nInvalidFrame=0 nDiscardedFrame=359095 nWrongThread=153 nSkippedByte=0 nFillByte=0 nDuplicateFrame=8851360 bytesProcessed=92697667320 nGoodFrame=2482554 nCall=23029
mark6-06 ERROR  One or more wrong-threads were identified.  This may indicate a correlator configuration error.
mark6-06 ERROR  One or more wrong-threads were identified.  This may indicate a correlator configuration error.
mark6-06 ERROR  One or more duplicate data frames (same time and thread) were found.  This may indicate serious problems with the digital back end configuration.

WEIGHTS for LCP are around 0.94. This might indicate also problems in LCP thread(s).

Inspection with vdifcontinuitycheck shows duplicate thread IDs for the LCP file indicating that RCP threads were written with wrong (odd RCP) thread IDs:

Threads alignment       :  0[+0] 1[+156] 2[+1] 3[+157] 4[+2] 5[+158] 6[+3] 7[+159] frames
Thread 1 Second 8106300 :  25580 frames : #0--#12799 : 0 lost, 639 out-of-order, 0 invalid, 12780 dup, of 12800 total
Thread 3 Second 8106300 :  25580 frames : #0--#12799 : 0 lost, 639 out-of-order, 0 invalid, 12780 dup, of 12800 total
Thread 5 Second 8106300 :  25580 frames : #0--#12799 : 0 lost, 639 out-of-order, 0 invalid, 12780 dup, of 12800 total
Thread 7 Second 8106300 :  25580 frames : #0--#12799 : 0 lost, 639 out-of-order, 0 invalid, 12780 dup, of 12800 total

Test 2:

The VDIF file was split into two separate files using modifyVDIF:

modifyVDIF --extract=6,4,2,0 /mark6-06_fuse/1/C212A_GB_No0762 /data/c212a/gb/C212A_GB_No0762_RCP
modifyVDIF --extract=7,5,3,1 /mark6-06_fuse/1/C212A_GB_No0762 /data/c212a/gb/C212A_GB_No0762_LCP

The produced files vary largely in size:

1066784        11. Mär 09:56 C212A_GB_No0766_RCP
154581691424   11. Mär 10:09 C212A_GB_No0766_LCP

Using the separated files for correlation still results in 0 WEIGHTS and no fringes in RCP. Same results concerning duplicate thread IDs as described in test1.

Conclusion: (even) RCP threads apparently have been configured with the wrong thread IDs (odd LCP threads) for some portion of the session. This leads to duplicate threads even thread IDs. The extended header carries no informaton that permist to separate the correct from the incorrect threads. Even though LCP correlation has shown fringes it consists of a mixture of LCP and RCP threads. This is also refelcted by the fairly large SNR in the LR cross correlations.

Problem seems to have been fixed from scan C212B_GB_No0631 onwards. All scans prior will have to be discarded.

Residuals

FITS completeness (pclist)