Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
[Type here] 1 [Type here]
VLBI Session Pre-Checks for Mark IV, VLBA and DBE Systems
See also: “Basic Checklists” section of Mark IV Field System manual
Sample checklist in Operations workshop notes
Draft VGOS set-up in Operations workshop notes
Please note that all references to Mark IV apply to Mark III as well unless specific separate
instructions are given for Mark III. The same applies to VLBA and VLBA4.
Experiment SNAP and procedure files
• Run DRUDG on the session (.skd or .drg) file to create the SNAP and procedure file.
Print hardcopies of the schedule and session notes.
• Edit the session procedure file to insert station-specific parameters such as IF
distributor attenuator settings.
• Be sure to create a new procedure file for each session from the .skd or .drg file. Do
not rely on default set-up procedures in the station procedure file in case system set-up
parameters have changed such as the frequency sequence.
Computer control
• Check that the FS is controlling the VLBI terminal and antenna.
o On a Mark IV terminal, check that the remote/local switches on all the
modules are set to remote.
• Set up the VLBI terminal with the session set-up procedure (e.g., setupa).
• Refer to the FS VGOS Operations notes, which follow the plots at the end of this
section..
Station timing
• Check that the formatter time agrees with GPS time at the one-second level. If it does
not, use ‘fmset’ to adjust the formatter time. For Mark III you must set the time
manually.
• If NTP is reliably available, the FS computer should run on NTP (with both local and
remote stratum one servers). The FS time.ctl control file should be set to use the
"computer" model. Experiment operations should not begin after a computer reboot
until NTP is sync'd. Use the check_ntp SNAP procedure to verify that NTP is
sync’d; one server must have an asterisk ‘*’ in front of it. Please see notes for the TOW
lecture Impact of Operations on Data Analysis for details about set-up and use of
check_ntp.
[Type here] 2 [Type here]
• If NTP is not reliably available, check that the FS clock is correct to <1 second. If it is
not, enter the command sy=run setcl offset &. This command, should also
be issued, whenever the formatter time is reset. If ‘fmset’ is used to set the formatter
time, the FS time, will automatically be reset when ‘fmset’ exits.
• If a computer other than the FS computer controls the antenna pointing, check that its
timing agrees with GPS to <1 second.
• For RDBEs, to check time and offsets use time SNAP procedure. Run 'fmset' if
necessary. Cycle through bands a,b,c,d (Avoid "s"ync) and exit.
• Using a counter with time interval measurement accuracy <100 ns, measure the time
interval between the 1pps signals from
o formatter clock and GPS receiver
o maser and GPS receiver.
The formatter should be within a few microseconds of GPS.
• Check that the formatter/GPS/maser 1pps offsets are consistent with other recent
measurements.
Feed polarization
• Ensure that the feed polarization is correct.
• For geodetic S/X observations, the correct polarization is RCP.
• For VGOS use linear X and Y.
Receiver
• Check that the physical temperatures of the 20 K and 70 K stations in the Dewar are
within their normal ranges and are not rising.
• Check that other receiver parameters that can be monitored, such as FET currents, LO
and phase cal temperatures, power levels, power supply voltages, etc., are within their
normal ranges.
• Check that the receiver LO is set to the correct frequency.
• If there is any uncertainty about the LO frequency, measure it.
o If the LO is accessible, measure its frequency directly with a counter.
o Alternatively, radiate a coherent test tone at a frequency in the RF passband
toward the antenna and measure its frequency in the IF or at baseband. Suitable
test tones can be generated up to a few GHz by using an amplifier and frequency
multiplier on the LO monitor output of a BBC or VC.
[Type here] 3 [Type here]
IF distributor/IF3 module/IF upconverter or downconverter
• Check that the inputs to these modules are connected to the correct IF signals.
• In a Mark IV DAT, check that the VC inputs are connected to the correct IF signals
(IF1/IF2 low/high or IF3). The IF switch box controlled through the IF3 module can be
used to switch the VC03 and VC10 inputs automatically between low and high bands.
• For RDBEs all 'sigma' values should be close to 20.
Video or baseband converters
• Measure the frequency of each VC/BBC LO with a frequency counter and compare
against the correct value.
• Check that the baseband bandwidth in each VC/BBC is correct.
• In a VLBA DAR, check that the IF input to each BBC is correct.
• Check that the baseband power levels are in the proper range.
o In a Mark IV DAT, adjust the IFD/IF3 attenuation to give USB and LSB power
readings of 0.1–2.0 on the front-panel meter of each VC, with the 10-dB
attenuators in. Use the largest amount of IFD/IF3 attenuation that keeps all
channels being recorded above 0.1 volt.
▪ If readings of 0.1–2.0 cannot be achieved in all VCs, as a temporary
measure the lower limit can be relaxed to 0.05, or small (1–3 dB) SMA
attenuators can be added to the inputs to the stronger VCs.
▪ Do not use large (≥10 dB) attenuators on the VC inputs, as those VCs
without attenuators are liable to saturate.
o In a VLBA DAR, check that the baseband power readings are all approximately
16000 counts. The AGC gain in each BBC should be between –10 dB
and +10 dB.
o The proper long-term solution to strongly frequency-dependent VC power
readings or BBC gains is to flatten the IF passband with appropriate filtering.
Please find this describe in more detail in the Ops Impact on Analysis chapter.
• For a Mark IV system, the command ‘ifadjust’ may be used to set the attenuators
automatically, provided the VC TPI readings are working and RFI is not severe. The
DAT must first be configured with the set-up procedure from the experiment procedure
library. The antenna should be pointed near zenith (or toward any direction that gives
nearly the minimum system temperatures), and the weather should be clear. If ‘ifadjust’
converges, the results should be the optimum attenuation for that mode. (No changes
should be necessary during the experiment as the source elevation or the weather
changes.) The resulting values should be edited into the IFD set-up procedure for the
mode. If there is more than one mode in an experiment, this procedure should be
repeated for each mode.
• For VGOS RDBE use FS 'setupbb' and 'auto'.
[Type here] 4 [Type here]
Antenna focus
• For geodetic and astrometric observations, lock the focus at the position that gives
peak response at mid-elevations.
• For astronomy-only (i.e., source mapping) observations, the focus may be adjusted
with elevation angle to give maximum gain.
Antenna pointing
• Check antenna pointing on two or more widely-separated sources.
• Pointing errors in both axes should be smaller than beamwidth/10 at the highest
observation frequency.
• The FS command ‘fivept’ may be used to automate the measurement.
System Equivalent Flux Density (SEFD)
• Measure the SEFD in each frequency band and compare against the nominal value.
o Using a strong radio source of known flux density, measure Tsource
/Tsys
.
o Convert to SEFD by dividing Tsource
/Tsys
into the source flux density
(corrected for size effect if the source angular extent is comparable to, or larger
than, the antenna beamwidth).
o The FS command ‘onoff’ may be used to automate the measurement.
System temperatures
• With the antenna pointed near zenith, measure the system temperatures in all
VC/BBC and broadband IF channels, and compare against nominal values.
Phase calibration signal
• Use an oscilloscope and 10–kHz viewing filter to examine the baseband phase cal
signal in each USB channel. (This section assumes that phase cal is present at 10 kHz
in each baseband USB channel, as will be the case with a 1 MHz phase cal pulse rate
and a total LO frequency of nnn.99 MHz.) Trigger the oscilloscope with a timing signal
derived from the maser (e.g., the frame sync VS-14 module).
o Phase cal should be present with the proper amplitude.
o Phase cal should be stable in phase relative to the maser timing signal.
• Use the same equipment to examine each LSB channel.
o The LSB phase cal level should be >20 dB weaker than in USB.
[Type here] 5 [Type here]
• A good end-to-end system test is to check for phase cal in the bit stream from each
USB data track.
o For a Mark IV system, connect one of the decoder DATA outputs to the 10–
kHz viewing filter input. Enter the commands ‘enable=s1’ and ‘st=for,0,on’.
Select USB tracks with the ‘repro=byp,<trackA>,<trackB>,,<rate>’ command
and check for phase cal on the oscilloscope.
o For a VLBA system, use the ‘repro’ and ‘dqa’ commands to measure the
phase
cal amplitude in each USB channel.
o For VGOS RDBE run RDBE monitor program.
Cable calibration
• The cable counter should be operating in single–sample mode, not averaging.
• The cable counter reading should be in the normal range.
• Single–sample readings should be stable to ±1 μsec.
• Check that the cable calibration system measures cable length changes correctly.
o Log the cable reading.
o Insert a short cable length in the cable from the ground unit to the antenna
unit, and log the cable reading again.
o Remove the short cable and log the cable reading again.
o Compare the change in readings with the nominal value.
Mark 5A
Please note that Mark-5A operational procedures are in the TOW07 notebook. Mark-6 will be in the
TOW15 notebook. Additional guidelines for both the Mark-5 and Mark-6 can be found in the Systems
failure chapter of the TOW15 notebook.
• 8-Pack Installation –
1. Remove the 8-pack from the shipping box and their shipping covers.
2. Inspect for damage.
3. Pull front lever down so that it is horizontal.
4. Slide 8-pack easily into Mark5A machine until it stops.
5. Lift front lever which will insert the 8-pack into the connector.
6. Turn the key switch to the right until it stops.
7. Repeat with second 8-pack.
• Conditioning the 8-Pack –
If there is time each 8-pack that is received at a station should be conditioned before it
[Type here] 6 [Type here]
is used to verify there has been no shipping damage. This can be done in any appropriate
length gap in recording (4 hours or more) or left running overnight or over a weekend.
Please see ftp://web.haystack.mit.edu/pub/mark5/Conditioning.txt for more
information. A copy of this document follows in this section. Please note that the
SSErase program is run outside the Mark 5A. If the 8-pack is write protected, you will
be prompted to remove the write protection, and you should do so. When the
conditioning is complete, please check the results to confirm that the 8-pack is okay
before using it.
• Mark5A Start-Up –
1. From the Mark5A terminal, login as oper.
2. Enter the password.
3. From the prompt, start a script running by entering ‘script -f anyname’.
4. To start the Mark5A running, enter ‘Mark5A -m0 -f0 &’.
5. To stop the Mark5A, enter ‘EndM5’.
• Field System Checks –
1. Test the communications with the Mark5A by entering ‘disk_serial’ at the Field
System prompt. Sample output:
/disc_serial/VNVF11G6G9BS9L,VNVF11G6G9BW6L,VNVF11G6G99V7L,VNVF11G6G9BVDL,VNVF11G6G9BUVL,VNVF11G6G9A2HL,NVF11G6G99XZL,VNVF11G6G9A2XL,
2. Enter ‘disk_pos’. This will give the position of the disk pointers. Sample
output: 2003.245.16:42:53.42/disc_pos/81952,0,
3. Enter ‘mk5=VSN?’. This will check if a VSN is assigned to the 8-pack.
4. Erase the disk with ‘mk5=reset=erase’. Be very careful that the disk installed
should be erased before entering this command.
5. Enter ‘disk_pos’. This will respond with the disk pointers at 81952,0 (or it may
be 0,0, depending on the version of erase).
6. Set up the procedure file that will be used for the session, e.g., ‘proc=r1087wf’.
7. Issue the set-up command from the session, e.g., ‘setup4f’.
8. Enter ‘disk_record=on’.
9. Enter ‘disk_pos’. Repeat this command many times to confirm recording.
10. Enter ‘disk_record=off’. This stops the recording.
11. Enter ‘scan_check’. Sample output:
/scan_check/2,r1172_0552+398_115-1741,mark4,32,2005y115d17h41m1.080s,49.1s,8.000000,0
12. Erase the disk with ‘mk5=reset=erase’.
[Type here] 7 [Type here]
Meteorological sensors
• Check that the values of barometric pressure, temperature, and relative humidity
reported by the ‘wx’ command are reasonable and repeatable.
‘Ready’ message
• Approximately an hour before the session is to start, or as soon thereafter as possible,
send a “ready” email message containing the following information:
o comments on any unusal conditions
o pointing offsets and source az/el
o SEFDs and source az/el
o system temperature for each IF
o formatter offset from GPS
o source and epoch of first scheduled observation
o weather and sky conditions
See the Operations workshop notes for more information.
More extensive testing
• After any equipment changes or if a long period of time has elapsed since the last VLBI
session, more extensive testing of the VLBI systems should be carried out. More
rigorous testing should also be conducted on a periodic basis (monthly to annually).
Such tests include:
o Measure pointing offsets over the full sky.
o Check for RFI and for jumps in the phase cal phase and cable cal as the
antenna is slewed across the sky.
o Calibrate the meteorological sensors, especially the barometer.
o Measure the DC level and AC ripple in the DAT/DAR power supplies.
o Measure the strength of the phase cal signal by observing how much the IF or
baseband power level changes when phase cal is turned off.
o Measure the levels of spurious phase cal signals:
▪ with phase cal turned off via the ground unit switch
▪ with phase cal turned on and the receiver LO unlocked
▪ with the cable driving the antenna unit disconnected from the ground
unit.
[Type here] 8 [Type here]
o For each VC/BBC:
▪ Check that the LO will lock over the full design frequency range (100–
500 MHz for a VC, 500–1000 MHz for a BBC).
▪ Measure the LO phase noise at 2-3 frequencies spanning the design
range.
▪ Measure the USB/LSB image rejection at the upper and lower ends and
middle of the baseband frequency range.
▪ Check each sideband signal on a spectrum analyzer for anomalous shape.
• Monthly execution of the ‘overnite’ procedure is recommended to check the stability of the
VLBI systems. The procedure should be run for at least 12 hours, and preferably 24 hours,
with the antenna stationary. At a minimum, the values of the following quantities should
logged at least every 15 minutes:
o wx
o cable
o formatter-GPS time interval
o system temperatures in all channels
o phase cal amplitude and phase in at least 1 channel per frequency band
o receiver parameters: dewar temperatures, phase cal temp, etc.
o other station-specific parameters
[Type here] 9 [Type here]
IVS Operations workshop TOW 2017
1.0 Starting a schedule
The basic steps in starting a schedule are:
1.1. Reporting problems when you need urgent help.
An e-mail list has been set up for items needing urgent attention by the IVS Coordinating
Center. The address is: [email protected]
This list should be used by anyone needing urgent attention by or advice from the
Coordinating Center. You can expect an answer normally within a few hours, if the
message is sent during US East Coast working hours. Messages sent on weekends or after
hours may take a little longer.
Please use this address rather than sending messages to ivs-ops or ivs-stations. If the reply
from the Coordinating Center is of interest to a broader audience, we will copy the
appropriate lists on the response.
Anyone can post a message to this list. The only recipients for this list are the Coordinating
Center staff: Dirk Behrend, Ed Himwich, Cynthia Thomas, Mario Bérubé, and John
Gipson.
Examples of the types of messages that should be sent to this list are: problems with
schedules, problems running DRUDG or the Field System, schedules that seem to be made
for the wrong date, schedules that do not include the right stations, operational problems
during observing, and any urgent issue that impacts operations and you need advice on how
to handle.
1.2. Review session notes
This provides one last opportunity to verify that nothing special has been missed in the set-
up for this experiment.
1.3. Correct cable wrap for AZ/EL antennas
For AZ/EL antennas it is important to verify that the antenna is on the correct cable wrap
for the first source. If the antenna starts on the wrong wrap it is all but guaranteed that at
some point in the schedule one or more scans will be lost because the antenna will have to
unwrap unexpectedly.
The wrap for each source is shown in the listings and in the "source=…" commands for
each source. You can perhaps get the antenna onto the right wrap by either driving it
manually or giving it a series of source commands that will "walk it around." Please be sure
to look into this issue far enough before when you start the schedule so that
[Type here] 10 [Type here]
you will have time to swing the antenna around if it is needed.
1.4. Make sure there is no old test log on disk
If you made a test running of the schedule be sure to delete the test log file before starting
the schedule for real.
If Mark5B is being used, load both decks with an empty eight-pack and verify the Mark-5B
system is operational. The Mark-5B verification and use notes can be found in this handbook.
The Mark-6 check-out can be found in the VGOS Draft at the end of this chapter.
1.6. Start schedule with "schedule=…"
The preferred form of the "schedule" command uses the line number to start with, e.g.,
"schedule=na225gc,#1", to start at line 1. Start the schedule as early in the
pre-session checklist as possible. This will allow you to record as many of the checks as
possible, directly in the log, with comments.
1.7. Check cable sign. The key points are:
1.7.1. Do the cable check before or after the experiment in the experiment log. Do not do
it during the experiment as this is likely to cause problems.
1.7.2. Check that the cable counter is not using averaging
Averaging should not be needed. The cable counter should be stable at the 1 μsecond
(about 1 mm of one-way cable delay) level without averaging; which is more than
sufficient. If it isn't stable at this level there is probably some problem that needs to
be corrected. Using averaging would only obscure a problem that needs to be
corrected instead of providing any useful improved accuracy.
1.7.3. Do a "cable" with the normal cable.
This establishes the "normal" reading.
1.7.4. Install a short calibrated extension cable
You should have a short cable with a male N connector on one end and a female N
connector on the other end. A cable of 30 cm will give a reading change of about 600
microseconds.
1.7.5. Do a "cablelong" command once the cable calibrator has stabilized again
[Type here] 11 [Type here]
This establishes the "long" reading.
1.7.6 Do a cablediff command.
This will automatically find the difference and report the sign. If the difference was
as expected, enter a comment to that effect. A suggested format for the comment is
"change of xxx usecs is nominal, longer cable increases/decreases reading".
If the difference was not as expected, you should troubleshoot the cable cal system if
you have time.
1.7.7. Remove the extension cable
Return to normal cable length. If you leave the extension cable in by mistake, please
do not remove it after the experiment starts. Leaving it in will cause no harm, taking it
out is likely too.
1.7.8. Do a "cable" to make sure everything is back to normal
This is final check that everything is back to normal. You should get approximately
the same reading as before.
1.8. Finish start check-list
Complete any remaining items in the pre-experiment checklist. A copy of a sample
checklist is included at the end of this workshop write-up as a reference.
1.9 Send the Ready message. The FS utility msg will send the information in a standard format
and is recommended.
This message summaries your status and lets the world know you are ready. It should be
sent to [email protected]. Be sure to include the session name in the subject line.
Please include the following information.
1.9.1 Comments about anything that may affect the data quality. If you have more than
one polarization available it is recommended that you check the polarization to make
sure it is correct and note that it has be checked.
1.9.2 The source and start time for the first scan. This assists the coordinator center in
determining if you have the right schedule if an update was issued.
1.9.3 Formatter to GPS clock offset. This is needed by the correlator to find fringes. If the
Formatter 1 PPS starts the measurement please report this as the Formatter to GPS
offset; if GPS 1 PPS starts, please report it as the GPS to Formatter offset.
1.9.4 WX: temperature, pressure, humidity, and sky conditions. This assists the analysts in
understanding the atmospheric conditions and interpreting the SEFD and Tsys data.
[Type here] 12 [Type here]
1.9.5 Cable difference. Please report whether a longer cable makes the reading larger or
smaller. Please also report the difference you observed in the reading when you lengthened the cable and whether it was nominal (as expected) or not.
1.9.6 Pointing values: Please report both X and S SEFDs, source used for measurement, its
azimuth and elevation when measured, and the pointing offsets on each axis. You
can include as many sources as you think are useful.
1.9.7 Tsys values. Please report the system temperature for each IF: X (lower X), S and X2
(upper X). If you have interference you may need to report the values from a video
(base-band) channel for the effected IFs.
You can refer to the Summary of Ops Messages near the end of the notes for the Operations
workshop. That summary includes an example of the recommended format for Pointing and
Tsys data.
If your FS PC has sendmail installed you can use the graphical FS utility msg to send
ready, Start, and Stop messages. It can be configured to add all the information to the
experiment log. Please note there are reports of problems with msg utility if you use the
“keypad” (typically on the right side of keyboards, sometimes used for numeric entry). This
appears to be due to a bug in the operating system or its utilities.
1.10. Send the Start message
This lets the world know that you started and whether there are any problems. Any
additional useful information for the record doesn't hurt either. The message should be sent
to [email protected]. Be sure to include the session name in the subject line.
1.10.1. When started and first source
1.10.2. Any problems
2.0 Making comments in the log Making comments in the log when problems occur is one of the most important functions
of an operator. It is the primary way of information about problems to the correlators (for
some correlators, in particular the VLBA, it is the only way). Please provide as full a
description of a problem and its consequences as possible. Many of the data analysts are not
familiar with the acquisition hardware. It is important to describe the implications in high
level terms, such as which channels are affected. For example, don't just say "A and B cable
switched", say "A and B cables switched; cables were wrong from the beginning of the
experiment until now; the IF channels had the wrong signals; X and S band were reversed;
no valid data in any channel until now.”
[Type here] 13 [Type here]
Entering comments as a problem is being trouble-shot is desirable since it creates a record
of what happened. Once a problem is resolved or by the end of the experiment in any event,
please make sure the following information has been entered in comments:
2.1. The nature of the problem
2.2. How it was resolved if it was
2.3. The consequences of the problem: how the data was affected.
2.4. The time period for which data was affected.
2.5. TOW Failure Recover class will cover the most common failures.
3.0 Periodic monitoring I: each scan correct:
For each scan the following information should be verified:
3.1. source
3.2. mode (configuration), frequency sequence, formatter mode, IF patching
3.3. on-source
4.0 Periodic monitoring II: hourly (at least)
These items need to be checked less frequently than every scan. Every hour is a reasonable
interval.
4.1. Sky conditions entered as comments in log Cloud cover, wind, precipitation, etc.
4.2. Phase-cal present in all VCs
Visual checks of the phase-cal with 10 kHz viewing filter is desirable. Displaying the
values from a phase-cal extraction system is helpful if they are available as well. The phase
should be stable and the amplitude at the expected level. An occasional check of the
sidebands that are not supposed to have a signal is desirable to verify that the image
rejection mixer hasn't failed. Use the phase-cal check for VGOS.
4.3. Formatter time/date correct
A comparison of the formatter to GPS is desirable. If the formatter doesn't have a display, it
is recommended that you NOT run "fmset" to do this check. Instead check that the error
reported by "setcl" (last entry of the first line of output) is small. And compare the FS time
displayed by "monit2" (System Status Display) agrees with GPS. You should also check
the pointing computer's time it if has one and is used for pointing calculations. The "setcl"
error output is useful for tracking jumps of integer seconds in the formatter time.
[Type here] 14 [Type here]
4.4. Formatter-GPS offset reasonable
The Formatter-GPS offset should not show any significant jumps in value. If you are
monitoring other clock offsets, e.g., Formatter-Maser, Maser-GPS, or Maser-Other clock,
these should be checked too. It is desirable to have at least one loop, e.g., Formatter-GPS,
GPS-Maser, and Maser-Formatter, to help isolate the guilty party if a jump is detected.
Since the clock offsets are based on 1 PPS signals they are useful for detecting jumps of
less than one second.
4.5. Cable and WX readings reasonable
These readings should be checked to make sure the displayed values haven't jumped outside
a reasonable range.
4.6. Tsys reasonable in all VCs
Monitored Tsys values give some indication of whether the system temperature is
remaining stable and hence whether a problem with the receiver may be developing. One
way to think about these values is that they provide a less complete, but still useful, check
on the sensitivity of the system, when a full SEFD measurement is not possible, such as
during observations in a schedule. The interpretation of the Tsys data requires a little care
since it may become elevated because of weather conditions (particularly at X-Band) or
because of ground pick-up on low elevation scans.
4.7. Disk pack full?
Check get_stats? output to see whether a disk pack has filled up and if so, swap it out with a
fresh one. The disk pack that is not being recorded should always be a fresh one and a full
disk pack should always be swapped out as soon as possible. This applies even if the one
currently being recorded “should” be the last.
[Type here] 15 [Type here]
5.0 Each shift change III: (at least)
This section describes some things that should be monitored even less frequently than once
an hour. Mostly this is because it is necessary to develop some history of data to look for
trends and outliers. It is useful to run "logpl" to plot these values. With a command file,
"logpl" can generate all the plots automatically for you to view when they are ready. See
the section on "logpl" for more information on running it.
5.1. WX parameters
5.2. Cable
5.3. Tsys, all IFs
5.4. Clock offsets
5.5. Other station specific values
5.6. For disk, check remaining disk capacity
6.0 Rack electronic failures
This may include failure of power supplies, BCC OR VC modules, formatters and Dewars,
disc packs and more. All impact data collection and analysis. The decision to continue
observing or stop and repair the system depends on the nature of the failure. Dewar failures
and Base Band or Video converter problems cause data loss and is described in detail in the
Ops Impact on Analysis chapter.
7.0 Clock Jumps
Handling clock jumps is a somewhat delicate operation. The handling depends on which
clock jumped. If the formatter jumped, but appears now to be stable, there is normally no
point in resetting the formatter since this will just introduce another clock jump. Except in
specific circumstances, described in the last paragraph of this section labeled as LARGE
FORMATTER CLOCK JUMPS, the formatter should not be reset. If the Maser or GPS
or some other clock jumps, it is even more obvious that it is probably best to not make any
changes to the formatter time. In any event the jump, which clock is affected, and the span
of data affected by the jump, should be noted in comments in the log.
It is useful, mandatory really, for identifying clock jumps to be comparing at least three
different clocks, e.g., formatter, Maser, and GPS. It is unlikely that two will jump at the same
time. Of course monitoring even more clocks will reduce the probability further of being
fooled by such an event.
To remove the error message generated by setcl for a clock that doesn’t need to be
corrected, please refer to the /usr2/fs/misc/fstime.txt file. The section on HANDLING
[Type here] 16 [Type here]
PROBLEMS and SETCL (the case for “a positive or negative integer or zero”) are useful in
this regard. This document contains other useful information about how time is used in the
FS and is recommended reading.
LARGE FORMATTER CLOCK JUMPS: If the formatter has jumped and the new
offset results in the sub-second portion of the Formatter to GPS offset (as reported by gps-
fmout or fmout-gps) greater than ±500 milliseconds or the integer second portion (as
reported by sy=run setcl &) is greater than ±5 seconds, the formatter clock should be
reset. A sub-second offset of greater than ±500 milliseconds will cause confusion about the
actual second. Integer second offsets greater than ±5 seconds may cause loss of SNR.
However, if the jump is not as large as described in this paragraph, the formatter should not
be reset, but monitored closely instead.
8.0 Power Failure recovery
Power failure (or general disaster) recovery can only be learned by practicing it and this
highly recommended. A detailed discussion of the basics for doing it is included in the
Experiment Operations manual in Vol. 1 of the FS Manuals. Some key concepts are
presented here.
8.1. Determine the next source
The most important issue is to pick a source that will give you enough time to get
everything set-up, and the antenna positioned. You should probably get all the equipment
up and working before trying to pick the next source, unless your antenna moves very
"majestically", i.e., slowly. In that case the time required to move the antenna may
dominate everything else. Consider the following factors when selecting the next source to
observe:
8.1.1. Where is the antenna?
Consider how long will it take the antenna to slew from its current position to the new
source. If you have an AZ/EL antenna be sure to make sure that you have considered
whether you will need to force the antenna to a different cable wrap.
8.1.2. Mark 5A Setup
Mark5A presents some special considerations after a power failure. The following
guidelines and Mark5A commands will help save pre-recorded data and allow recording the
session to continue after restarting the mark5A program.
If recording is aborted because of some problem before the "disk_record=off" command is
executed by the Mark 5A, then all data recorded since the previous "disk_record=on" may
be lost. If you are using the "2004y351d" (16December2004) version, or a later version, of
the Mark5A program, then as described below, in most of these cases, it is possible to
update the record pointer (but not the directory) on the disk pack to reflect the data that had
been recorded. This procedure will prevent the data from being overwritten by new
recordings and allow the data from the aborted recording to be correlated. This command
[Type here] 17 [Type here]
seems to work well in at least two cases: (1) the Mark5A program is ended prematurely by
crash or some other problem or (2) the power is lost to the Mark5A unit. It does not seem
work as well in the case where the key switch is turned off during recording (care should be
exercised of course to avoid turning the key switch on the selected disk pack).
Regardless of the cause of the aborted recording, the best recovery procedure is to execute
the following FS commands (after re-starting the Mark5A program):
mk5=bank_set=<bank with aborted recording>;
mk5=recover=0;
If you do not know which bank has the aborted recording, it is probably the one that is
neither empty nor full. You can use the monit5 'disk space remaining' display and the
'bank_status' command to see which bank is neither empty (100% remaining) nor full
(0.x% remaining). If recording aborted during the first scan, then that bank would show
100% remaining.
It is probably not worthwhile to recover short recordings particularly if doing so would
prevent other scans from being recorded. However, for longer recordings, particularly if
several scans have been recorded "continuously" and a problem prevented a proper
"disk_record=off", recovery may be helpful. Since at this time directory information is not
recovered properly, this won't be helpful for scans transferred off-line after an experiment
with software that depends on the directory information. In that case, the transfer will not
work, but this will at least preserve the data on disk.
The Mark5A "recover" command is available with the "2004y351d" (16December2004) or
later versions of the Mark5A program. It is documented in the "Mark 5A command set"
document:
http://www.haystack.mit.edu/tech/vlbi/mark5/docs/command5a.pdf
8.2. Position antenna.
Once it has been determined which source to use, position the antenna in preparation for
restarting.
8.3. Restart schedule with “schedule=...,#n,1” line number form
The value of "n" should be the starting line from the listing of the intended next
observation. The effect of using "1" as the number of lines to execute will be to only
command the source. While the antenna is moving, you can use the "list" command to see
if there are any commands coming that you want to avoid. If you find there are no
commands you want to skip, you can restart the schedule with a "cont" command. If you do
want to skip some commands, then restart the schedule at the line number where you want
to restart.
[Type here] 18 [Type here]
9.0 Post experiment
The post-experiment checks provide an opportunity to "close-the-loop" on the
pre-experiment checks and verify that some things that cannot be checked during the
experiment, such as the SEFDs, are still okay.
9.1. Check cable sign again
This is repeated for good measure in case either there was some problem with the initial
check. This also verifies that the cable system still detects the calibrated cable correctly and
hence has not failed during the experiment. Please see the "Start schedule" section for more
details on how to do the check.
9.2. Complete end check-list
There are several other items on the end of experiment check-list. This should all be
completed. The results of some of the tests will go in the End Message. A sample post-
experiment checklist is included at the end of this write-up for reference.
9.3. Send the Stop Message
The End Message lets the world know you have completed the experiment and your status
at the end. This is especially useful for the correlator since it will help them prepare for
correlating the data. This information should all be reported in comments in the log as well.
It should be sent to [email protected]. Be sure to include the session name in the
subject line. The End Message should have the following components:
9.3.1. Scans missed
List the scans missed, not by line numbers, but by the time spans that data was lost for.
9.3.2. Comments
Please describe any unusual conditions that affected data collection. Please describe any
equipment problems. This might not be limited to things that affected the data collection,
but also include items that require maintenance.
Please include any other information that you think might be useful for the correlator or the
data analysts.
9.3.3. Stop time
Indicate the time data collection stopped.
[Type here] 19 [Type here]
9.3.4. Clock offset information
Report the final clock offset information. Use the same format as described above for the
Ready message. Reporting the clock offset a second time may seem redundant but it has
two purposes. First in case there was an error in reporting the first value, it provides a back-
up. Secondly if there was no error in the first value it can be used to a rough estimate of the
clock drift rate. Please also include any information about clock jumps and the time span of
data affected by any jumps.
9.3.5. 8-pack inventory
Report the module inventory in the ‘End” message.
9.4. Close log
As with the pre-experiment tests, the post-experiment tests should be done as much as
possible in the experiment log. Once the post-experiment tests have been completed, you
can close the log, either using the "schedule=…" command with a null or
non-existent schedule, by changing the log file with a "log=station" command, or by
terminating the FS.
9.5. Delog parameters for the record
Use "logpl" to delog the ancillary data, cable, WX, clock offsets, comments, etc. It is best to
do this before sending the end message so that any problems that are discovered from the
plots can be reported in the End Message. If you do discover problems, open the log again
with the "log=…" command and add explanatory comments. DO NOT EDIT THE LOG
MANUALLY. (The log is intended to be a record of what happened during the experiment.
If you edit it, and change or delete anything, you defeat this purpose. No one minds if there
are some start-up or other errors in the log. It is more important for the log to be a faithful
record of what happened than for it to be neat.) Save the hardcopy plots along with the
listings you printed which might include handwritten notes. See the section below about
running "logpl".
9.6. Transfer log to BKG/OPAR/CDDIS/aspen
The log needs to be placed on the appropriate server depending on which correlator will
handle the data.
9.7. Send 8-pack(s) to correlator
Prepare the 8-pack(s) for shipping.
[Type here] 20 [Type here]
9.8 Enter Mark 5 module shipping information into the TRACK program. A new program,
PACKTRACK is currently being developed to also handle Mark 6 modules.
9.9. Send the Wrap-up Message
This message contains the last few items related to an experiment. You can optionally include
this information in the Stop message. It should be sent to
[email protected]. Be sure to include the session name in the subject line.
9.9.1. Log Placement Info
Report where the log was placed: server, directory, and file name.
9.9.2. Additional Shipping Information
If you have any additional information about the shipping such as the routing, this would be a
good place to send it.
9.9.3 Additional Experiments Comments
If you have become aware of any additional problems or if there is any special information you
either have discovered since the experiment ended or forgot to place in the Stop message, this
is a good place to send it. You can of course send follow-up information as often as necessary.
[Type here] 21 [Type here]
10.0 Sample Checklist
Station experiment check list, Experiment Date
START
1. Pointing: Sky Conditions
Source Az/El Offsets SEFDs: X/S
/ / /
/ / /
/ / /
2. TSYS IF 1(A) IF 2(B) IF 3(C)
3. Cable sign with comments in the log
normal
long
difference, nominal , okay?
longer cable: increases decreases reading
extra cable removed?
4. WX temperature Pressure Humidity
5. Formatter time
6. formatter-GPS offset Counter Starts: GPS FMOUT
7. MASER -GPS offset Counter Starts: GPS MASER
8. VCs phase-cal Levels Frequencies
9. Local checks
10. Send "ready" e-mail with TSYSs, SEFDs, pointing offsets, source, az-el, weather, sky
conditions, gps-fmout offset, first source and time, recorder humidity, any problems,
confirmation of any unusual set-up, equipment condition
11. send "start" message
EVERY HOUR or more often
1. enter WX and sky condition comments
2. Check Phase-cal all VCs
3. Check VC signal levels
4. Check system temperatures
5. check formatter time
6. formatter offset reasonable
7. Check cable value reasonable
8. Check WX reasonable
9. Parity errors okay
END
1. Pointing: Sky Conditions
Source Az/El Offsets SEFDs: X/S
/ / /
2. TSYS IF 1(A) IF 2(B) IF 3(C)
3. Cable sign with comments in the log
normal
long
difference, nominal , okay?
longer cable: increases decreases reading
extra cable removed?
4. WX temperature Pressure Humidity
5. Formatter time
6. formatter-GPS offset Counter Starts: GPS FMOUT
7. MASER -GPS offset Counter Starts: GPS MASER
8. VCs phase-cal Levels Frequencies
9. Local checks
10. Send "end" message with log location
[Type here] 22 [Type here]
11.0 Running "logpl"
This section describes the basics of running "logpl", a graphically oriented program for plotting
data from the log. This program is a more modern version of the old "logex" program (which is
still available). Please refer to the "logpl" manual available in PostScript form on the FS servers
in the doc sub-directory.
11.1. Interactive Use
"logpl" can be started from a shell prompt by just entering "logpl". The program uses the X
display, so the starting used must have access rights to the X display. If you are using X server
and open new window (C-S-W) and type "logpl" at the prompt, it should start okay.
11.1.1 Selecting a log file
If the FS is running, the log that is open will be selected automatically. If the FS isn't running
or if you want to use a different log, use the "File" menu item and select the "New Log File"
option. Enter the name of the log file you wish to examine. The directory will default to
"/usr2/log" and the extension to ".log".
11.1.2 Plotting Data
You can plot up to four "channels" of data simultaneously. The menu items "Channel1",
"Channel2", and so on are pull-down menus that let you select the data to be plotted for each
channel. The "Options" menu item allows you select whether the plot are superimposed or
shown separately and whether to connect the points with a line.
11.1.3 Entering New Parameters
You can enter new parameters to plot either by updating the "logpl.ctl" control file and
restarting "logpl" or by interactively using the "Options" menu item and selecting "Edit
Selections" options. Any changes that are made interactively are not preserved when "logpl" is
terminated.
11.1.4 Printing Plots
Once you have a plot you would like to print, you can print it as PostScript either to a printer or
to a file. Use the "File" menu item and select the "Print" option to print the file.
11.2 Non-interactive Use
"logpl" also has a non-interactive, or command mode, for automated processing. This is
particularly useful for automatically generating plot of ancillary data for inspection at shift
changes and after the experiment is over. To use "logpl" in command mode use the "-cmd"
command line option: "logpl -cmd". You can start processing from a command file
automatically by entering: "logpl -cfile xxx" where "xxx" is the name of the command file.
You can even select the log file to process on the command line by entering "logpl -log lll -
cfile xxx", where "lll" is the name of the log file.
[Type here] 23 [Type here]
11.2.1 Sample Experiment Delogging Command File
The following is a sample command file that will delog some general ancillary experiment
data. If you want to use this command file you should customize it for your station.
output=temp,overwrite
channel=1
command=wx/
parm=1
channel=2
command=wx/
parm=2
channel=3
command=wx/
parm=3
channel=4
command=cable/
parm=1
SCALE=.038308,.038365
plot
output=temp,append
channel=1
command=tsys1/
parm=9
scale=55,75
channel=2
command=tsys1/
parm=10
scale=50,70
channel=3
command=tsys2/
parm=7
scale=70,100
channel=4
select=0
plot
output=temp2,overwrite
command="
glist
[Type here] 24 [Type here]
11.2.2 Sample Output
The list below and first two following pages of plots show sample outputs from
"logpl" using the sample command file from the previous sub-section. The list of
comments is given here and the plots are shown after the page titled “Summary of Ops
Messages”, although that is not the order they would normally appear in.
------------------------------------------------
Filename: /usr2/log/ca034gc.log Selected data: Cmd=" Parm=7
9811117293172:" CA034 1998 GILCREEK A Gc
9811117293172:" A GILCREEK XYNS 7.3152 60.0 0 -86.0 86.0 60.0 0 -73.5 73.5 25.9 Gc 101
9811117293172:" Gc GILCREEK -2281547.29145 -1453645.07155 5756993.15313 40476601
9811117293172:" 101 MOJ-VLBA 1217640
9811117300862/newtape/"to continue, use label command"
9811117320970;"now doing cable length check
9811117323316;"adding extra cable
9811117334902;"removing extra cable
9811117345251;"shorter cable is higher measurement
9811117351006;"gps - maser 11.23 micros.
9811117352465;"gps - formout 11.8 micros.
9811117595001&preob/"nsource
9811118022476;" mostly clear skys
9811119010572;" scattered cumulus clouds
9811120003368;" scattered cumulus clouds
9811121001114;" scattered cumulus clouds
9811122150632;" mostly cloudy skys
9811123031029;" mostly cloudy skys
9811200001880;" mostly cloudy skys
9811201012244;" mostly cloudy skys
9811202000276;" mostly cloudy skys
9811203012995;"weather rpt.partly cloudy
9811204040763;"weather rpt.partly cloudy
9811205025448;"weather rpt.partly cloudy
9811206035266;"weather rpt.partly cloudy
9811207275178;"weather rpt.partly cloudy
9811208070338;"weather rpt.partly cloudy
9811209000615;"weather rpt.partly cloudy
9811210001989;"weather rpt.partly cloudy
9811211015214;"weather rpt.partly cloudy
9811211553262;"weather rpt.partly cloudy
9811213111449;"weather rpt.partly cloudy
9811214050071;"weather rpt.partly cloudy
9811215041480;" partly cloudy skys
9811216035476;" partly cloudy skys
9811217000980;" thin high stratus clouds
9811218003191;" thin high stratus clouds
9811218025818;"now doing cable length check
9811218031886;"adding extra cable
9811218041157;"removing extra cable
9811218050979;"shorter cable is higher measurement
9811218052061;"gps - maser 11.26 micros.
9811218053098;"gps - formout 11.8 micros.
9811218054115&unlod/"**dismount this tape now**" Glist: Time from
9811117293170 to 9811218065785. Data points listed: 684.
1998.237.17:22:55.12 1998.238.05:49:07.06 1998.238.18:15:19.01
1: Temperature3: Humidity
2: Pressure4: Cable−length
Station: GILCREEKFilename: /usr2/log/ca043gc.log
17.1
7.6
964.3
959.1
89.6
31.8
0.038365
0.038308
17.1
7.6
964.3
959.1
89.6
31.8
0.038365
0.038308
1998.237.17:22:55.12 1998.238.05:49:07.06 1998.238.18:15:19.01
1: Cmd=tsys1/ Parm=93: Cmd=tsys2/ Parm=7
2: Cmd=tsys1/ Parm=10Station: GILCREEKFilename: /usr2/log/ca043gc.log
75
55
70
50
100
70
75
55
70
50
100
70
75
55
70
50
100
70
[Type here] 25 [Type here]
12.0 Mark 5A Issues
12.1 When to change disk packs
During an experiment, the FS will use a pack until it is full. It will then attempt to switch to the
other bank. The full pack should be swapped with a fresh pack as soon as is practical. Great
urgency is obviously not important here since it should be hours until the new pack is needed.
Mounting a fresh pack as soon as possible will minimize the chances that data is lost if the
other pack has problems that causes it to be filled early. In that case data may be lost if there is
not an empty pack in the other bank. A problem such as this should be rare. However,
periodically monitoring the predicted time for next pack change may help to discover a
problem before it occurs. In addition, the output of the “mk5=get_stats?” commands in the
“postob_mk5a” procedure may show symptoms that may alert you to a problem. These
statistics should be periodically monitored. Both of these checks should be included in the
hourly, at least, checks.
Currently there is no automatic method to predict the time when a disk pack will be full and
ready to be changed. You can however get a pretty good idea by
comparing the GB count on the schedule summary print-out with the capacity of the disk and
using a little math. This technique is pretty accurate except in the case where there may be a
problem with a pack. In that case, the pack will fill sooner than expected. Additional
information is available from Monit-5 (Control-Shift-5) “Mark5 Remaining Capacity” display.
This display shows which pack is active, the remaining capacity of each pack (if it is known) in
three different units, and the UT time when this information was most recently determined for
each pack. The display of the remaining capacity of the active pack is updated during the set-up
for each observation. The remaining recording time in the current mode that is displayed is not
the remaining clock time. The clock time will depend on the recording duty cycle. For
example, for geodesy, the recording duty cycle is typically about 50%. So the active pack may
last about twice as long as the amount of time shown. This will vary from session to session
and station to station: your mileage may vary.
12.2 Changing disk packs
When the FS detects that the current pack is full, it will attempt to switch to the other pack. A
pop-up window will be displayed asking the operator to change the full pack (the display
includes information on which pack to remove). The full pack can be removed at anytime.
However, the Mark5 will become unresponsive to the FS for up to about 20 seconds when the
mount switch is turned to (lock) when the new pack is inserted. In order to avoid having this
impact the session, please be sure to do this only when the FS system will be quiescent for at
least 20 seconds, either between scans or while recording. In either case, it should be done 20
seconds or more before the next scheduled event. After completing the pack change, dismiss
the pop-up window to avoid confusion in the future.
12.3 Starting Mark5A and Recording Mark5A debug output
[Type here] 26 [Type here]
In order to help debug problems with the FS, Mark5A, and their interaction, we would like to
ask all stations to record the debug output generated by Mark5A. In order to this we
recommend that every time you start Mark5A, you use the following commands on the Mark5:
cd
script -af <name> Mark5A -m0 |& LLogr &
#(running of experiment occurs here)
EndM5 exit
The sequence above shows that you need to execute an "exit" (or Control-D) to terminate
"script" recording. Please be sure to terminate Mark5A and exit the "script" recording after
each experiment.
Please note the commands above assume you are using csh (or tcsh) as the shell.
You will need to adjust this accordingly if you are using a different shell, but you can always
start a csh (or a tcsh) and enter the commands as shown.
The "script" command shown above will cause all screen output to be put in a file with the
name "<name>" in your home directory. For "<name>", we suggest you use the name of
experiment with ".txt" appended, e.g. "r4174wz.txt" or if you are not running a schedule, you
might use the date for the part before the ".txt", e.g., "040224.txt".
If a problem is encountered with Mark5 control please send us the ".txt" file and the
corresponding FS log. This will aid greatly in debugging.
Periodically, you should delete old debug output files on the Mark5 so that the file system
13.0 Running "plotlog"
This section describes running "plotlog", an non-interactive program for generating plots from
log files. It was developed for analyzing the logs collected for the "overnite" and short-term
stability tests in "systests" . Basically, it plots any of the typically recorded ancillary data in a
log file. It plots everything it finds including: wx/, cable/, gps-fmout/, setcl#time/, tsys/,
receiver parameters, and phase-cal from the Mark IV decoder. Other items to be plotted can
easily be added, please let Ed ([email protected]) know if you want to include plots of
other data or want other command line options.
After the section and after the two example ‘logpl’ plots, there are five pages of sample output
from the ‘plotlog’ program.
[Type here] 27 [Type here]
13.1 Basic Command-line Use
The "plotlog" program is only run from a shell command-line or from a script. The basic usage
is:
cd /usr2/log
plotlog log_file
where log_file is the name of the log file, with the path if needed, including any suffix,
such as the typical ".log". This will display the plots on your terminal (press return to step
through the pages until your reach the last).
13.2 Plot Output to a File
The plots can be sent to a file for printing and/or other purposes, use:
cd /usr2/log
plotlog -f file/device log_file
where "log_file" is above, and "file" is the name of the output file, with the path if needed,
including any suffix such as the typical ".eps". The "device" should typically be vps (for
portrait orientation PostScript). You can use -f \? to be prompted for the device and at that point
there is an option to display a list of available devices.
See the "Sample Command and Output" subsection below for an example.
13.3 Command-line Options
There are several command-line options available to tailor the output. You can use "plotlog -h"
to display a help page with a full listing of options. You may not need any of the command-line
options, but the most useful ones are:
-c value -- delete certain cable values
This command allows you to delete the cable readings that were generated when the
extra cable was added for the sign check if "cablelong" wasn't used. If "value" is
negative, all points that far below the maximum value are deleted. If "value" is positive,
all points that far above the minimum value are deleted. The units for "value" are the
the ones on the vertical axis of the plot and so are affected by whether the "-r" option is
selected.
-p -- also plot phase-cal amplitude versus phase for available tones
-r -- show cable in raw units, rather than relative delay
-t value -- edit tsys values
All tsys points of zero or below and all tsys points greater than"value" are deleted.
[Type here] 28 [Type here]
-z value -- deletes PCal amplitudes less than "value"
See the "Sample Command and Output" subsection below for an example of the "-t" and "-c"
options.
13.4 Sample Command and Output
The command:
plotlog -t 500 -c -1000 -f /tmp/ca043gc.eps/vps ca043gc.log
was used to generate sample out from the same log as in section 11, for "logpl". The output is
included after the sample "logpl" plots at the end of this write-up. Typically, it should not be
necessary to use the "-t" or "-c" options, but they were needed for this old (1998) log. No phase
cal plots appear because there was no Mark IV decoder phase-cal data in the log.
13.5 Description of Plots
The subsection describes the basic features of "plotlog" output. The plots appear as up to six
plots per page. All time plots on all pages have the same horizontal time axis for easier
comparison. The reference time for "0 hours" appears in the title of the plot along with the
station name and the log name. No data editing is applied unless specified by command-line
options. The plots are auto-scaled to show all included data. All of the included data appears
within the box of the plot, never along the edges. Points that were not numeric or were
excluded by editing criteria appear along the top edge of _that_ plot. The pages are numbered.
On the last page the text "Last Page" is included so it is clear how many pages there should be.
Summary of Ops Messages
Send all messages to [email protected]. Please include the session name, station
name and type of message in the subject line. Please avoid the use of tab character and HTML
for these messages. The msg utility available now the FS has forms for Ready/Start/Stop
messages (In the initial version, the clock offset in the stop message must be entered as
comment).
1. Ready message
Comments
First source and time formatter to gps clock offset
WX: temp, pres, humid, sky conditions
Cable difference: longer cable makes reading larger/smaller change in microseconds and is/is
not nominal
Pointing values (as many as appropriate):
SEFD X/S Source Az/El Offset1 Offset2 707/705 Cas-A 352/34 -.004 .008
Tsys (x1/s/x2): 123/77/109
(The suggested format of the pointing and Tsys entries is shown above.)
[Type here] 29 [Type here]
1. Start message
After starting, report source and start time
2. Stop message
Comments
End formatter to gps clock offset list of problem scans
Module inventory on site
Observation end time
All module shipping information (VSN, AWB, shipper) should be placed in
TRACK/PACKTRACK. Please wait until you have the AWB numbers available to do
this unless it would cause an unreasonable delay. This item is not part of the message,
just a reminder to enter the information into TRACK/PACKTRACK.
3. Wrap-up message
Optionally this information may be included in the Stop message, you can of course send as
many of these follow-up messages as you need. The wrap-up message may also contain log
placement information, any additional shipping information.
Additional information about problems during the session should be sent as a wrap-up message
at any time after discovery post observing.
VGOS Operators Notes
2017-03-07
Contents
1 Introduction 1
2 Pre-setup 2A DRUDG experiment files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
3 Experiment setup 2A Verify NTP is sync’d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2B Start FS log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2C Check RDBE Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3D Check Mark 6 Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3E Check MCI Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3F Mount Mark 6 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3G Verify RDBE time, offsets, and VDIF epochs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4H Initialize pointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5I Set mode and attenuators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6J Check RDBEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6K Check pointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7L Make test recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Start experiment 8A Start non-FS multi-cast logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8B Send “Ready” message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8C Start schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9D Send “Start” message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
5 Monitor experiment 9A Monitor scan_check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9B Check RDBE Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
6 Post experiment 9A Stop the schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9B Stop multicast logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10C Check pointing and SEFDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10D Send “End” message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10E Send test scan data files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11F Remove the module for shipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11G Transfer log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
7 Appendix 13A Setting Up Password-less SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13B Setting Up Password-less Log Transfers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13C Manually uploading log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13D Manually processing schedules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14E Schedule rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14F Module conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14G Setup Field System PC from Cold start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15H Setup of RDBEs from a cold start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15I Set-up of Mark 6 server from a cold start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16J Setup MCI server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17K Connecting RDBE IF inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
1. IntroductionThese notes cover basic procedures for running VGOS experiment operations with the Field System (FS). Theyare described in a logical order for the tasks that need to be completed. If another order of operations make
1
more sense for some locale that is fine of course.A basic assumption is that all the equipment is already up and running. The procedures check this and refer tothe appendix for set-up instructions for a device if it isn’t already running. In addition to these setup procedures,the appendix also contains instructions for some other useful operations.As written here, the procedures are mostly station independent. However, some station dependencies arenoted. These only cover GGAO and KPGO for now but Westford can, and will, be added. Some of the stationdependencies can be removed by writing some additional scripts that hid the dependencies. This is also a goodidea because the scripts can reduce the complexity of the operations.
EH: The etransfer instructions need to be checked.
2. Pre-setupA. DRUDG experiment files
This step is first for two reasons. First, the schedule file is typically available before the day of the experiment. Secondly, the number of modules in a recording group may eventually depend on the experiment. When that is the case, the number needed will conveyed through the listing that is made in this step.To create the station specific SNAP and procedure files from the session schedule, fetch the schedule from IVS and DRUDG it by entering, in a FS PC shell, the commandfesh -d <sched>
where <sched> is the schedule name, eg v16033. If you have manually received the schedule, follow the instruc-tions in the Manually processing schedules section of the appendix.Along with the .snp and .prc files for the FS, this will create the listings file <sched><stn id>.lst in /usr2/sched which shows a human readable summary of the schedule. Here <stn id> is the two letter station code, eg “gs” for GGAO. To view this on the FS PC, open in a text viewer; eg, in a FS shellless /usr2/sched/<sched><stn id>.lst
Review the information in this listings file to ensure you have sufficient storage for the observation.
3. Experiment setupA. Verify NTP is sync’dIn the FS enter:
check_ntp
The output is of the form
remote refid st t when poll reach delay offset jitter==============================================================================*192.168.1.20 18.26.4.105 2 u 444 1024 377 0.208 -0.336 0.396+192.168.1.21 18.26.4.105 2 u 167 1024 377 0.215 -0.822 0.153
The offsets should be small and there must be a server with an asterisk * in the first column. It may take a fewminutes to get an *.
B. Start FS log file
In the FS, open the experiment log so the set-up will recorded in that log:log=<schedule><stn id> # eg v16033gs
2
C. Check RDBE Status
From the Field System, check the RDBEsrdbe_status
The response values should be 0x0f41.
Old Server: The response values should be 0x0941.
If the response is to not correct for any RDBEs, refer to Setup of RDBEs from a cold start in the appendix.
D. Check Mark 6 Status
From the Field System, check the Mark6smk6=dts_id?
You should receive a sensible response similar to!dts_id?0:Mark6-4605:1.0.24-1:1.2;
If the response is to not correct for any Mark6s, refer to Set-up of Mark 6 server from a cold start in theappendix.
E. Check MCI Status
This is specific to GGAO.
You can test whether this is needed by using the FS SNAP procedure:dewar
This is specific to KPGO.cryo
If it is working, you will see the readouts for the vacuum pressure and 20K and 70K stages. If not, see the SetupMCI server section in the appendix.
F. Mount Mark 6 Modules
This step is included just after initial verification that all the devices are working, but before the more detailedpre-experiment checks. This allows any issues with getting the modules mounted resolved as soon as possible.However, these steps can be done as part of the making the test recording if that is preferred. All commandsare entered into the FS unless otherwise noted.1. This is to help with debugging, display and clear the Mark 6 message queue:
mk6=msg?
If an unexplained error happens during the following procedure, please use this command again to get moreinformation.
2. Initialize module; create, mount, and open moduleCheck status:mk6=mstat?all
After the two fields: return code and cplane status (hopefully mstat?0:0) there are 10 fields per group:group:slot:eMSN:#disks found:#disks nominal:free space:total space:status1:status2:type
It may be easier to read if individual groups are queried; eg. for group 1:mk6=mstat?1
3
If the module has already been initialized, ie status1 is initialized, and the data is no longer needed (becertain first), erase it:mk6=group=unprotect:<group>mk6=group=erase:<group>
If the module has not been initialized (status1 is “unknown” and no eMSN?), initialize it:mk6=mod_init=<slot#>:<#disks>:<MSN>:<type>:<new>
For examplemk6=mod_init=1:8:HAY%0001:sg:new
Note: due to a current incompatibility, the FS–Mark 6 connection will timeout during long runningcommands such as this.
Until this is fixed, you can optionally run this command directly on the Mark 6 from a shell promptwithssh root@mark6ada_clientmod_init=<slot#>:<#disks>:<MSN>:<type>:<new>;
note the final semicolon is necesseary in da_client but is automatically added by the FS.
Create, open and mount the group:mk6=group=new:<slots>mk6=group=mount:<slots>mk6=group=open:<slots>
(Slots is a list of slot numbers included in the group, without any seperators, eg <slots>=12)To query if the group is created properly:mk6=group?
Print out should have the group number at the end. If it is -, something has gone wrong.
G. Verify RDBE time, offsets, and VDIF epochs
In the FS, check RDBE time, offsets, and VDIF epochs:time
This will display the pps_offset, dot, and gps_offset:2016.320.18:11:39.28/rdbed/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.28/rdbec/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.29/rdbea/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.29/rdbeb/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.29/rdbeb/!dbe_dot?0:2016-320-18-11-39.296s:-0.007s:33;2016.320.18:11:39.29/rdbea/!dbe_dot?0:2016-320-18-11-39.296s:-0.018s:33;2016.320.18:11:39.29/rdbed/!dbe_dot?0:2016-320-18-11-39.296s:-0.024s:33;2016.320.18:11:39.29/rdbec/!dbe_dot?0:2016-320-18-11-39.296s:0.000s:33;2016.320.18:11:39.29/rdbeb/!dbe_gps_offset?0:-3.758203125e-05;2016.320.18:11:39.29/rdbed/!dbe_gps_offset?0:-3.758203125e-05;2016.320.18:11:39.29/rdbea/!dbe_gps_offset?0:-3.758203125e-05;2016.320.18:11:39.29/rdbec/!dbe_gps_offset?0:-3.758203125e-05;
The offsets should be small (GPS typically ± a few tens of µs, PPS less than ±0.1 µs) and the DOT times shouldbe the same the FS log timestamps, within about 0.1 seconds. The third field of ‘dot’ output is the VDIF epoch.All RDBEs must have the same value, (33 in this case). The VDIF epochs of all RDBEs are also shown in theRDBE status window.
For old server, the VDIF epochs are not displayed
4
If any of the DOT times are not correct, the ones that are wrong must be set with ‘fmset’. To do this from theFS console, press <Control><Shift>T to start fmset, or typefmset
in an FS PC shell.You can select the RDBE to set by letter: a, b, c, or d.With that RDBE’s time being displayed, verify that the time is correct by comparing it to the FS/Computer time.If it is off by a lot, use ‘.’ to get it close — within a few seconds. Once it is close, you can use ‘+’ and/or ‘-’ toincrement and/or decrement the RDBE by a second at time until it agrees with the FS time.If the PPSoffset is greater inmagnitude than ±1e-7 (±0.1 µs) for an RDBE, it must be resync’d. You can try usingthe ‘s’ command in ‘fmset’ for each RDBE that has too large an offset. This command will take approximately45 seconds per RDBE to complete. If as a result, the PPS offset comes within the tolerance, please check thatall the RDBEs are sending data withmk6in
If an interface is not showing receipt of data, reinitilize the corresponding RDBE with:rdbe_init<id>
where <id>=a, b, c, or d for the interface that is not recieving data, eth2, eth3, eth4, or eth5 respec-tively.If this does not solve the data transmission problem. or the ‘s’ did not make the offset small enough, restartthe RDBE, see Setup of RDBEs from a cold start in the appendix.
For the old server if the PPS offset is too large, you will need to restart the RDBE to re-sync, see Setupof RDBEs from a cold start in the appendix.
If the displayed VDIF epochs are not the same, use the ‘;’ command for each RDBE to set the epoch to thenominal one.
For the old server, the only way to verify this is to note that the Mark 6 will not record a test scanwhen you make one (but there could be other causes besides this one). If this happens and you candetermine which RDBEs do not have the current nominal VDIF epoch set its time explicitly. If you don’tknow which is wrong just set them all explicitly. Use the procedure described below for after June 30or December 31 to set the time(s).
If this is the first experiment since December 31 and June 30, and the RDBEs have not had their epoches resetsince that date, they should be reset. Use the ‘;’ command in ‘fmset’ for each RDBE to set it to the nominalVDIF epoch. The third field in the ‘dot’ output above must be the same for all the RDBEs. The VDIF epochs ofall RDBEs are also shown in the RDBE status window.
For the old server, this requires setting the time explicitly for each RDBE with ‘fmset’ even if it lookscorrect. Use at least one of ‘.’, ‘+’, ‘-’, or ‘=’ commands and then it is necessary to verify/set the timefor each RDBE. The third field of the ‘dot’ output above will be missing for the old server and the VDIFepoch will not be in the RDBE status window.
If any changes were necessary due to the above considerations, check the values again with:time
to verify they are correct. If not, follow the above instructions again and/or seek assistance.
H. Initialize pointing
In the FS, initialize pointing configuration and send antenna to a test source:proc=pointinitpcasa
The following sources are the most reliable for these small antennas:
5
Source Approximate L.S.T. of transitTaurus A 05:30
Virgo A 12:30Cygnus A 20:00
Cas A 23:30
This is KPGO specific:
verify Az and El for source are acceptableantenna=operate
If not, try a different source first.
This following information is site specific.
Local apparent sidereal time (L.A.S.T) is displayed in the antenna monitor window (monan) at GGAO and KPGO.Cas A is always up at GGAO and Westford, but another source may be more appropriate at times.
I. Set mode and attenuators
While waiting for the antenna to move to the test source, setup the experiment mode and adjust attenuators.In the FS,proc=<schedule><stn id> # eg. 'v16033gs'setupbbifdbbmk6bbauto # sets the attenuatorsproc=point
J. Check RDBEs
Locate the RDBE Monitor window (monit6) or start it by pressing <Control><Shift>6. Noting that for somefields the display switches between IF0 and IF1 every second, check for each RDBE that:1. DOT ticking and correct time2. All RDBEs have the same epoch
Not applicable to old server
3. DOT2GPS value small (±a few µs) and stable (varies by 0.1 µs or less)4. DOT2PPS value small (±0.1 µs) and stable (varies by 0.004 µs or less).5. RMS values close to 20. They may be higher if the antenna has reached the source since the “auto” command
above. They may be higher or lower if the elevation has changed significantly since the “auto” or there isvariable RFI.
For the old server, RMS values close to 32.
6. Tsys IF0 and IF1 about 50-100, may be jumping a bit7. Phase-cal amplitude about 10-100, phase stable to within a few degreesLeave the window open for later monitoring.
EH: Is this still needed?
Check multicast for all 4 bands in FS shell prompt:mon<id>
where <id>=a, b, c, or d (eg. mona etc.)
Check RDBE data connectons
6
rdbe=data_connect?
and verify that band a,b,c,and d equal 0,1,2, and 3, respectively.
K. Check pointing
Check that the antenna is now on the source we selected earlier with the FS commandonsource
The result should be TRACKING. If the antenna status is still SLEWING wait until you see an onsource message inthe FS log window or onsource indicates tracking.Once the antenna is onsource, start the pointing check withfivept
This will take a few minutes. Once complete fivpt will give you output in the form:Az El xEl_offs El_offs
xoffset 99.4469 30.8190 0.01417 -0.00806 0.00452 0.00801 1 1 01d0 virgoa
The xEl_offs and El_off values (ie. the 3rd and 4th columns) are the beam spaces offsets of the pointing fit.The absolute value of these should be less that ~0.02 degrees in each coordinate. There should also be theflags ‘1 1’ in the 3rd and 4th columns from the end.Next, measure the SEFDs on test sourceonoff
This will also take a few minutes. Once complete onoff will give you output in the form:source Az El De I P Center Comp Tsys SEFD Tcal(j) Tcal(r)
VAL virgoa 170.9 63.0 15a0 1 l 3016.40 0.9943 52.03 2895.6 55.657 1.67VAL virgoa 170.9 63.0 15a1 2 r 3016.40 1.0088 47.93 2549.8 53.201 1.60VAL virgoa 170.9 63.0 15b0 3 l 5256.40 0.9946 49.58 2742.3 55.306 1.66VAL virgoa 170.9 63.0 15b1 4 r 5256.40 1.0148 41.57 2549.6 61.331 1.84VAL virgoa 170.9 63.0 15c0 5 l 6376.40 0.9831 42.57 2294.2 53.891 1.62VAL virgoa 170.9 63.0 15c1 6 r 6376.40 0.9862 44.09 2248.1 50.992 1.53VAL virgoa 170.9 63.0 15d0 7 l 10216.40 1.0121 51.91 3009.5 57.979 1.74VAL virgoa 170.9 63.0 15d1 8 r 10216.40 0.9870 53.64 3084.2 57.496 1.72
source Az El De I P Center Comp Tsys SEFD Tcal(j) Tcal(r)
Verify SEFDs for eight bands are reasonable. They should be in the range ~2000-3000.Finally, zero the offsets:azeloff=0d,0d
(note there is no space between arguments)
L. Make test recording
1. Check Mark 6 inputs withmk6in
which will show the Gb/s by interface in the FS log. For example, a rate of 2 Gb/s should should look like#popen#mk6in/eth2 2.078 eth3 2.079 eth4 2.079 eth5 2.079 Gb/s
If one or more interfaces are not showing the approximate nominal data rate (initially 2 Gb/s per interface),it is likely that the corresponding RDBEs needs to be reconfigured.To see more details of the Ethernet ports state for the Mk6, usemk6=input_stream?
Sample output:
7
mk6a/!input_stream?0:0:rdbeB:vdif:8224:42:66:eth3:127.0.0.1:12000:0:rdbeC:vdif:8224:42:66:eth4:127.0.0.1:12000:0:rdbeA:vdif:8224:42:66:eth2:127.0.0.1:12000:0:rdbeD:vdif:8224:42:66:eth5:127.0.0.1:12000:0;
which shows rdbeA going to eth2, rdbeB going to eth3, rdbeC going to eth4, and rdbeD going to eth5.2. The modules and groups should have been set-up already. If not refer to Mount Mark 6 Modules above.3. In FS, record some test data:
mk6=record=on:30:30
verify that lights on the mk6 flash appropriately. You can check recording status with:mk6=record?
It should progress starting as “recording”, then transitioning to “off”.If the status stays “pending”, it may be that not all the RDBEs are sending data. You can check this by usingthe FS SNAP proceduremk6in
as shown in step #1 above.If all interfaces are receiving data at the correct rate, it may be that the VDIF epochs of all the RDBEs don’tagree. See Verify RDBE time, offsets, and VDIF epochs for detail on how to verify/set.You can also check if the disk is full withmk6=rtime?
4. Once recording ends, check quality:mk6=scan_check?
Results should show vdif, the time when recording was started, 30 seconds of data, 30 GB of data with an8 Gbps data rate.
EH: data rate will eventually go to 16 and 32 Gbps.
4. Start experimentA. Start non-FS multi-cast logging
EH: Once we have InfluxDB logging of data, maybe we can get rid of this logging,
From the FS enter:start_mlog
AB: Doesn’t seem to work at Wf
If there are no errors reported and the “Done” message is printed the logging has been started.
B. Send “Ready” message
From FS console window enter Control-Shift-G.
EH: Jason will eventually move vgos-msg-gui.py to the FS machines so we can have better function-ality. Maybe the placement of the window should be controlled locally by .Xresources.
At this point a GUI window should pop up. Enter the session name, station code (lower case) and select thetype of message from the drop down list.
• Click the update values button. This collects the information in real time and the SEFDs from the pointingcheck in the log file.
• Complete the maser offset value by looking at the maser counter in the maser room.
8
• In the “to” email address field, send it to [email protected]
• Enter a brief comment, include weather information.• Click the send message button when finished.
C. Start schedule
In a Linux shell (xterm), look at the list file <schedule><stn id>.lst created in the DRUDG step (eg. v16033gs.lst). Find the first observation and note line number ‘nnn’ after scan name at start of line.
Now, in the FS, start schedule:schedule=<session><stn id>,#<nnn>
(Note: the pound sign (#) is required and there should be no spaces in the command)
D. Send “Start” message
Send “Start” message using the same procedure as in Send “Ready” message.
5. Monitor experimentA. Monitor scan_check
To display scan_check results as they come in (and the old ones so far) open the ‘scnch’ window (Control><Shift>K).
Results should show vdif, reasonable record start time, about equal seconds and GBs of data (typically 30+),and 8 Gbps data rate. Be aware scan_checks occasionally fails and the data is okay.Position and size window for convenient viewing, new output will follow any changed size. You can stop thiswith <Control>-C
EH: The placement and size of the window can be controlled by .Xresources.
B. Check RDBE Monitor
Check the display for reasonable values periodically, note some fields alternater between IF0 and IF1 everysecond:1. DOT ticking and correct time2. VDIF epoches for all RDBEs agree, if there is any disagreement some will be in inverse video
For old server, not applicable.3. DOT2GPS value small (±a few µs) and stable (varies by 0.1 µs or less).4. DOT2PPS value small (±0.1 µs) and stable (varies by 0.004 µs or less).5. While recording RMS values are close to 20 for the new server, sometime RFI can cause the value to be off,
but it should always be between 10 and 40 during recording. If valeus are outside the nominal ragnget theywill be shown in inverse video.
For the old server, RMS values close to 32 and inverse video is not used.6. Tsys IF0 and IF1 about 50-100 (may lower at Wf due to use of a preliminary cal value), may be jumping a bit7. Phase-cal amplitude about 10-100, phase stable to within a few degrees.
6. Post experimentA. Stop the schedule
In the FS, runschedule=
9
B. Stop multicast logging
In the FS, runstop_mlog
C. Check pointing and SEFDs
Select procedure ‘point’ libraryproc=point
If the FS has been restarted since the initial check, you will need to re-initialize the pointing set-upinitp
Site Specific for KPGO:
antenna=off (to allow verification of Az and El for source before antenna moves)
Try Cas-A as a source:casa
Site Specific for KPGO:
If necessary, try other sources from table in Initialize pointing until one with a good position is found,then:
antenna=operate (restart antenna)
If you re-ran ‘initp’ above, you will need to restore the experiment set-up:proc=<schedule><stn id> # eg. 'v16033gs'setupbbifdbbmk6bbautoproc=point
Wait until the antenna is on source. You can either watch the log or check withonsource
The result should be “tracking”.As in the Check pointing section in pre-experiment, run a pointing checkfivept
and check the “xoffset” offset values are small. Then check the SEFDsonoff
and verify SEFDs for eight bands are reasonable, ~2000-3000.Finally, zero the offsetsazeloff=0d,0d
Site specific for KPGO:
source=stow (wait until stow is reached) antenna=off
D. Send “End” message
Send “End” message using the same procedure as in Send “Ready” message. Include details such as the stoptime and the current weather conditions on-site.
10
E. Send test scan data files
Chris: This section is completely different for KPGO, due to our e-transfer Mk6 being a different unitthan our operational Mk6. Also for actually sending entire experiments not just test scans. Below theoriginal steps provided in this procedure are the KPGO site specific steps highlighted.
In a terminal, log in to the Mark 6
EH: this is the part I know the least about and I suspect it different for different stations, maybe usingsomething besides “gather”, have tried it at GGAO?
ssh mark6agator <group> <filename>.vdif ~/dqa –d <filename>.vdifscp <filename>_*.vdif evlbi1.haystack.mit.edu:/data-st12/vgos/<exp>/
EH: Maybe put into a script, or something, to minimize typing? As it is, it is definitely too much typing
F. Remove the module for shipping
In the FSmk6=group=close:<slots>mk6=group=unmount:<slots>
Before removing, check the modules are unmounted with
key off disk before doing mk6=mstat?all
mk6=mstat?all
Chris: KPGO site specific send test scan and e-transfer procedure
Close and unmount disk module(s) and prepare for e-transferring a scan or experiment.mk6=group=close:<slots>mk6=group=unmount:<slots>
turn keys off, remove module(s)mk6=mstat?all
(to clear module info and check the modules are unmounted)Insert Mark6 modules into the e-tranfer Mark6From the da-client mount the modules and verify all disks are seen:da-clientgroup=mount:<slots>;mstat?all;
(if you get “6:0:1” restart cplane)group=open:<slots> list?
From another xterm window gather the scan(s) to your RAID disk, and de-thread if necessary:For test scan that needs to be de-threaded:gator <slots> <scan name>.vdif /mnt/raiddqa -d <scan name>.vdif
(this will create 4 files with thread ID on scan name)For scans where you intend to transfer the entire experiment use gather464:gator -t <slots> "scan name".vdif /mnt/raid
11
Start tsunami server specifying the scans of the session to transfertsunamid <scan_name>_*.vdif
You will see the available scans to be pulledAt another xterm window (in “oper”, not “root”)Ssh to Haystack storage nodes:ssh evlbi1.haystack.mit.edu (password is oper password)cd /data-st12/vgos
Run tsunami, setting the transfer rate, error free, and connecting back to your machinetsunami set rate 100M set error 0 connect 146.88.148.18
Make sure needed files are there to be pulleddir
Pull filesget *
Once transfer is complete exit tsunami client to get prompt backexitls <scan_name>* (verify all scans were copied)
After last scan has copied logoutlogout<Ctrl-C> (to quit server)
From da-client unmount the disk and prepare for shippinggroup=unmount:<slots>;
turn keys off, remove modulesmk6=mstat?all
(clears module info and checks the modules are unmounted)
G. Transfer log file
This may be done before transferring the test scan.
In FS, close experiment log:log=station
In a terminal, copy the log to CDDIS and Haystack with plog. If you are transferring the most recent log you canuseplog -l
Otherwise useplot <session> # v16033
Or if the log file does not conform to the standard naming conventionplog <log file path> # eg /usr2/log/v16033gs.log
If this is not successful, see Manually uploading log files in the appendix
12
7. AppendixA. Setting Up Password-less SSH
It is convenient to setup password-less login for local devices from the Field System PC. You can do this withSSH using public-key cryptography. To generate public/private key pair with SSH (if you don’t already have one),runssh-keygen
Accept the defaults and enter a blank password when prompted.For each computer you want to enable password-less login, append your public key to .ssh/authorized_keyson the remote host. On a recent versions of the Field System OS (i.e., FSL9 based on Debian Wheezy) use thecommand (use the target host node name or IP address in place of $host):ssh-copy-id $host
If this is not available usecat ~/.ssh/id_rsa.pub | ssh $host 'cat >> ~/.ssh/authorized_keys'
If you do not wish to have completely password-less login, an alternative is to encrypt your ssh key with apassword and use ssh-agent to unlock it for your session. The upshot is you still have the convenience ofpassword-less login, you just have to enter your password once after you login to the FS computer. Howeverthis is not recommended for devices that FS procedures use ssh to connect to, including, RDBEs, Mark 6s, MCI,and backend-PCs, which will need to have no prompt for smooth operations.This is also more secure since the ssh key is encrypted on disk and if anyone ever takes your key, they can notgain access to your systems.This is a good idea for remote terminals, although is slightly more cumbersome for local access.To encrypt your private key, enter a password when you generate it. To encrypt an old key, or change its pass-word, usessh-keygen -p -f ~/.ssh/id_rsa
The process for adding your public key to a login to a remote host is the same as above.Now, when you want to use your ssh key, add it to your ssh-agent withssh-add ~/.ssh/id_rsa
This will decrypt your private key and allow any ssh clients the current login session to use it without a password.
B. Setting Up Password-less Log Transfers
EH: replace with new CDDIS transfer set-up procedure, use URL reference
Currently we are using FTP to transfer log files to CDDIS. This is will change in the future. For now, add thefollowing line to the ~/.netrc file:machine cddisin.gsfc.nasa.gov login <username> password <password>
replacing the appropriate fields with your username and password.
C. Manually uploading log files
EH: must be re-written for new CDDIS procedure
In a terminal, copy the log to CDDIScd /usr2/logftp cddisin.gsfc.nasa.govuser <your cddisin username>password <your cddisin password>put <session><stn id>.log # eg 'v16033gs'quit
13
And to Haystack:scp <session><stn id>.log evlbi1.haystack.mit.edu:/data-st12/vgos/logs
D. Manually processing schedules
1. Put schedule in /usr2/sched on FS PC2. Run drudg, from FS PC shell:
cd /usr2/scheddrudg <schedule>.skd
Now, in drudg, give the following commands (ignoring text after # and filling in the variables)<stn id> # the two-letter station id (eg. 'gs' at GGAO)3 # make .snp12 # make .prc9 # change printer output destination<schedule><stn id>.lst # destination file, eg 'v16033gs.lst'
# three more <Return>s5 # print summary0 # exit DRUDG
AB: Should use /exper/ directory?
E. Schedule rotation
1. Start DRUDG with original schedule2. Pick option 10 in DRUDG.3. Specify the full fine name for the output file, i.e., include the .skd. I suggest you call it hYYDDD.skd. The “h”
is to avoid confusing it with real schedules.4. Pick the start time. YYYY MM DD HH MM SS5. Pick the duration — usually 24 hours6. End DRUDG7. Restart DRUDG with the new file to make the normal output (or skip step #6 reselect schedule, option #8)
F. Module conditioning
1. Load modules and enter da-clientssh mark6a da-client
2. In da-client, initialize the modules with the same mod_init command used for experiment set up:mod_init=<slot#>:<#disks>:<MSN>:<type>:<new>;
3. Create a new group with the modules you want to condition. If more than 1 module is being conditioned,group them together.group=new:WXYZ;group=mount:WXYZ;group=open:WXYZ;
WXYZ=slot 1, 2, 3, or 4. Only enter slots with modules in them.4. Check to the status. It should say “open:ready” for the modules included in the group.
mstat?all;
5. Leave da-client and navigate to bin.<Control>+Ccd /home/oper/bin
14
6. Run the hammer script. After, all 8 lights on each module should be lit.nohup hammer.sh &
7. To break the group, do the mod_init command on each module and reassign groups. If recording on all themodules simultaneously, no further action is needed before an observation besides a test recording.
G. Setup Field System PC from Cold start• Start FS computer• Login as user oper• Check NTP by runningntpq -np
The output is of the formremote refid st t when poll reach delay offset jitter
==============================================================================*192.168.1.20 18.26.4.105 2 u 444 1024 377 0.208 -0.336 0.396+192.168.1.21 18.26.4.105 2 u 167 1024 377 0.215 -0.822 0.153
The offsets should be small and there must be a server with an asterisk * in the first column. It may take afew minutes to get an *.
• Other local devices that use NTP (antenna, RDBE, Mark6, etc) can now be started• Start the Field System in the login shell
fs
H. Setup of RDBEs from a cold startPower up RDBEs
Use the power switch to start or cycle the power of each RDBE to be started.
New server:
Check the status of all RDBEs with the FS commandrdbe_status
When the RDBEs respond with a status value 0x0f41, skip to Check/Set RDBE times
Start RDBE server
From FS PC shell prompt, login to each RDBE:ssh root@rdbe<id>
use <id>=a, b, c, or d to log into RDBE-<id>.Then, on each RDBE runrbinnohup ./rdbe_server 5000 6 &exit
Repeat this for each RDBE that has been restarted. You can verify when all the RDBEs have started from the FSwith:rdbe_status
There will be an error for each RDBE that is not ready. When all RDBEs respond with a status value 0x0941,proceed to the next step.
15
Load Firmware
To load the firmware on all RDBEs, use the FS command:rdbe_fpga
If you want to load the firmware individually for RDBE-<id>, you can use the FS commandrdbe_fpga<id>
Again, use <id>=a, b, c, or d as necessary.There will be an error for each RDBE, since it will not respond right away and will time-out. You verify when thisis finished from the FS with again usingrdbe_status
This time, the RDBEs should respond with status 0x0f01
EH: is 0x0f01 correct?
When all status values reach the correct value, proceed to next step
Configure RDBEs
To initialize the configuration on all RDBEs, use the FS command:rdbe_init
You should get four “success” messages.If you need initialize an RDBE individually use, the FS commandrdbe_init<id>
You should get a “success” message.
Check/Set RDBE times
It is necessary to check/set the time with fmset for an RDBE every time it is restarted. The time only needs tobe set if it is not correct already.To do this from the FS console, press <Control><Shift>T to start fmset, or typefmset
in an FS PC shell.You can select the RDBE to set by letter: a, b, c, or d.With that RDBE’s time being displayed, verify that the time is correct at the one second level (± a few tenths) bycomparing it to the FS/Computer time. If it is off by a lot, use “.” to get it close, within a few seconds. Once it isclose, you can use + and/or - to increment and/or decrement the RDBE by a second at time until it agrees withthe FS time.Be sure to exit with <Escape>.
I. Set-up of Mark 6 server from a cold start
Use the power switch to start or cycle the power of each Mark 6 to be started.
Check Mark 6 connection (New server)
From the Field System, check the Mark 6 connectionmk6=dts_id?
You should receive a sensible response similar to!dts_id?0:Mark6-4605:1.0.24-1:1.2;
If so, you can skip the rest of this section.
16
Starting Mark 6 servers
If you receive an error, check that the Mark 6 servers are running. The programs cplane and dplane need to berunning on the Mark 6. These should startup after boot.To check check if they are running performssh root@mark6aps aux | grep plane
If they are not, start them/etc/init.d/dplane start/etc/init.d/cplane start
J. Setup MCI server
This is specific to GGAO.
GGAO should be set-up agree with KPGO FS approach below.
You can test whether this is needed by using the FS SNAP procedure:dewar
If it is working, you will see the readouts for the 20K and 70K stages. If not, or if more MCI parameters aredesired, use the following from a shell window, login to the MCI computer and run the MCI clientssh mci./tcpip_client mci 10000
a prompt should come up. To display all the MCI data use the commandmci_data?
If the server is not running, start it with./startmciserver
This is specific to Westford
Log in to the MCI node PC from a new window on the FSPCssh 192.52.63.139
Confirm the server is running (called ‘fenode_server’)ps aux |grep server
If it is not running, start as followscd node-software/V0/./mci_fenodesrvr 192.52.63.139
At this point you should see data points all the way through DI345 scroll once per minute. If you close thiswindow, the server will quit.
This is site specific to KPGO
You can test whether this is needed by using the FS SNAP procedure:This is specific to KPGO.cryo
If it is working, you will see the readouts for the vacuum, 20K, and 70K stages. If not, start the server from theFS:startmci
17
If ‘cryo’ still doesn’t work, then use log into the Hub PC in new xterm window of FSssh oper@hubpcps aux|grep mci (to see if mci server is running)startmciserver (to start the server if not running)
To display more MCI date from the FS, enter:mci_data
If that doesn’t work, log into the Backend PC in new xterm window on FSssh oper@backend-pcmci_client.py 128.171.102.237 5000 (opens mci client on backend pc)mci_data? (displays all mci data points current state including dewar temperatures)
K. Connecting RDBE IF inputs
To avoid possibly overloading, and damaging, the RDBE analog-to-digital (A2D) converters used to sample thesignal, it is recommended to make sure that the attenuators are at the maximum setting (31.5)c when connect-ing cables to the IF inputs. Use the command:rdbe_attenX=<if>,31.5
where:‘X’ is the RDBE ‘a’, ‘b’, ‘c’, ‘d’ or null for all. ‘<if>’ is the IF channel ‘0’, ‘1’, or ‘both’ (or null) for bothfor example to set IF1 attenuator for RDBE-B to the maximum use:rdbe_attenb=1,31.5
to set both IF attenuators for RDBE-C to the maximum use:rdbe_attenc=,31.5
or to set all IF attenuators for all RDBEs to the maximum, use:rdbe_atten=,31.5
18
Westford VGOS Setup Checklist
123456789
10111213141516171819202122232425262728293031323334353637383940414243444546
A BDate ____________
fesh -d <sched> Automatically gets files and creates session files
check_ntp Verify the NTP is sync’d
log=<schedule name><station id> vt7011wf
rdbe_status 0x0f41
mk6=dts_id? !dts_id?0:Mark6-4605:1.0.24-1:1.2;
Confirm MCI is running by viewing display and confirming clock is running
mk6=msg? clears mark6 message que
mk6=mstat?1 Query for group 1
If needed erase module:
mk6=group=unprotect:<group>
mk6=group=erase:<group>
If needed initialize module:
mk6=mod_init=<slot>:<#disks>:<msn>:<type>:new mk6=mod_init=1:8:HAY%0001:sg:new
Create, Mount, and Open the module for operations
mk6=group=new:<slots> mk6=group=new:1
mk6=group=mount:<slots>
mk6=group=open:<slots>
mk6=group? Confirms if group is created properly
time This will display the RDBE pps,dot, and gps
If there needs to be a time correction use the fmset command
mk6in then RDBE=data_connect? Confirms RDBE data is streaming and cabling correct If needed to re-initialize the RDBE:
rdbe_init<id> id is either a,b,c,d
time run this to confirm status of any changes
Run mona,monb,monc,mond from new window to confirm phase cal tones then Start pointing checks:
proc=point
initp
casa
proc=<schedule name><station id> vt7011wf
setupbb
ifdbb
mk6bb
auto
proc=point
onsource
fivept Westford normal offsets less than .025 degrees
onoff Sefd numbers 2000-5000 are normal
azeloff=0d,0d
mk6=input_stream?
mk6=record=on:30:30
mk6=record? Do this command while recording to show status
mk6=rtime? Displays recording space left on disk
mk6=scan_check?start_mlog This starts the multicast data
Send Ready Message
Westford VGOS Setup Checklist
47484950515253545556575859606162636465666768697071727374757677787980818283848586
A BStartremote This starts the Haystack remote terminal monitorSchedule=<session><station id>,#<nnn>
Send Start message Then run Cable cal procedure and record valueControl Shift K This opens a window displaying scan_checks from active log file
Monitor session and enter hourly WX, and operational comments
End of Session:
schedule=
stop_mlog
Send Stop message
Do pointing checks:
proc=point
initp
casa
proc=<schedule name><station id> vt7011wf
setupbb
ifdbb
mk6bb
auto
proc=point
fivept
onoff
azeloff=0d,0d
Send tests scan to the correlator:
From the Mark6 terminal:
gator <slots> <scanname>.vdif ~/
Wait till the prompt reappears: ~5 minutes
dqa –d <scanname>.vdif
Wait till prompt reappears ~ 5 minutes
Send files to Haystack:
scp <scanname>_*.vdif evlbi1.haystack.mit.edu:/data-st11/vgos/
Wait for confirmation from correlator that all data look good
Procedure to remove module for shipping:
mk6=group=close:<slots>
mk6=group=unmount:<slots>
Turn key OFF on disk module
mk6=mstat?all
Transfer log session file
log=station
Open terminal window
plog –l This is an “L”