A system with a state-side control center and wide bandwidth link to the operations support base near the array is optimal for remote access to the instrument by astronomers and engineers. The center will serve as the central hub for interaction with the instrument and the archival database. Almost all observing will be done remotely through the center. The overall computer system architecture will be a distributed hierarchical system, similar to many modern telescope systems (GBT, Gemini, VLT). This design would utilize both embedded microprocessors and more general purpose machines to produce an integrated system. It should be possible to present any aspect of the hardware remotely to allow full fault diagnosis and trouble shooting.
The MMA has the capability to produce data at a rate that would exceed data reduction capabilities. This potential problem has been addressed with data rate limitations, data condensation, and an imaging pipeline. Archive space and data processing resources are treated as finite resources, much like present VLBI tapes. Integration times of around 10 seconds will be the norm, but higher time resolution will be available at the expense of channels and baselines. The 10 second integration time will provide the capability for data editing, self-calibration, and other calibration procedures with reasonable time resolution. The data will flow from the realtime system into a calibration and condensation routine that will apply some calibration and will average the data into frequency bins that match the science requirements. Some of these operations are irreversible. Visibility data will then be written to a database and, in parallel, put into a pipeline to produce images. Dirty images from a subset of the data will be sent to the control center in realtime to aid in data evaluation. The MMA will differ from other previous synthesis imaging instruments in that a concerted effort will be made to automatically produce images that satisfy the scientific requirements for as many of the projects as possible. This will increase the scientific output of the instrument and contribute toward ease of use.
The proposed instrumental specifications that affect computing follow. See Appendix I for details of assumptions and their implications.
The state-side control center serves as the terminus for the data link, the location of the archival database and as the hub for access to the array. The only technical requirement to access the array or the archival database will be an adequate bandwidth link to this center. The links to the center may be the Internet or perhaps an intranet between NRAO and some universities and research centers. Dial-up digital links to the control center with adequate bandwidth will certainly be an option at some point in the operation of the MMA.
Time critical processor intensive chores will be handled by embedded microprocessors closely integrated with the hardware. Higher level tasks of coordination, integration, data reduction and human interface will be handled by workstation style technology. Parallel processing will be used to handle requirements of spatial diversity and raw processing power (see the pipeline requirements). Communication and interfacing in a distributed system can be an important design decision that must be made early.
Maintaining high efficiency from the array will require continuous monitoring of the hardware and evaluation of fault conditions. Diagnosis of problems can be aided by software that understands the interconnections and flow of signals within the instrument, including reconfiguration.
The ability to configure different portions of the array into sub-arrays that can use different observing modes is useful for several types of experiments and essential for testing. An extreme possibility is using the array as 40 independent single-dish telescopes, so there should be no restrictions on sub-arrays from the standpoint of the software. Any sub-array can be used for any of the 6 different modes of observing. Another configuration concept is a telescope cluster which is defined as a group of telescopes where no correlation products are formed within the cluster. There will be no limits imposed on the number of clusters. Clustering is only valid for the two synthesis observing modes.
The minimum interferometric integration time of 70 msec is driven by solar and flare star observations but constrained by the need to have a full Walsh phase switch cycle and to do the FFT's on every integration. At short integration times, the number of channels available is greatly reduced to stay within the maximum data rate. It is possible to fine tune this number if necessary, as phase switching details are not yet complete.
The minimum single dish integration time of 3 msec is driven by on-the-fly (OTF) continuum mapping. The goal is to Nyquist sample the beam at the targeted mapping rate of a degree per second. At 300 GHz this will be satisfied by samples every 3 msec. Spectral line OTF experience at Kitt Peak shows that there are still gains to be made by sampling faster than 100 msec. Therefore OTF mapping can be done in either spectral line and/or total power allowing a trade off to be made of the sample rate for channels to stay within the maximum data rate of the instrument.
The peak data rate was chosen to satisfy the science requirements specified at Tucson. This rate of 10MB/sec (36GB/hr, 0.9TB/day) will pose no problem at the data recording stage as it can be done with today's technology. The observer must again trade off time resolution for channels and baselines to stay within this limit. This limit is soft and can be increased if warranted. Parallel techniques can be used for the FFTs, calibration and recording if necessary.
The average data rate specification of 1MB/sec was chosen to provide average data sets of a size that have a reasonable chance of being reduced. For example, this rate would be met by experiments with a 10 second integration time and 3200 channels. A 4 hour project collecting data at the average rate would produce a 14GB data set which is equivalent to a 700MB data set using today's technology. Administrative implementation of the average data rate may require that data rate be considered in the proposal evaluation stage. The software will have to enforce the awarded data rate as an integral part of the approved project specification. The limitations imposed by this average rate will encourage wise use of the correlator, conserve archive space, and are consistent with the goal of having the array produce a stack of usable images. Increasing this instrumental specification will have a strong effect on the already difficult problem of the pipeline and data reduction.
A large reduction in data volume can be achieved if the data are channel averaged from the raw correlator resolution to match the proposed science. Several of the calibration steps (some irreversible) must first be performed on the data. This requires treating the software errors as if they were hardware errors - if they seriously compromise the data then the observations must be repeated. The calibrations to be applied are:
Work on the atmospheric phase correction is still in progress, but it is possible that corrections will be needed in realtime on timescales of 1 second. These may be applied to the LO or alternately to the baseline visibilities as they are being integrated. It may be necessary to consider recording both the atmospheric corrected data and the uncorrected data and allow the observer to choose from the two data sets on a per baseline basis in the reduction. This would potentially double the data rate specification.
Phase calibration from the phase reference source will be applied to the data before recording and the correction recorded; this is a reversible step. Fast phase calibration cycles can be done faster than the data recording integration time, if necessary, with the constraint that there be an integral number of cycles per recorded integration.
The minimum setup time required is driven by the fast phase calibration scheme, band changes, single dish frequency switching, and wideband frequency synthesis within a band. Both fast calibration and band changes require slewing the telescope and are therefore limited to a fraction of a second. Frequency switching is desired at rates of several Hz so reasonable efficiency requires a setup time of around 50 msec. The wideband synthesis (stepping the LO by many GHz on successive integrations to cover the entire receiver band) is useful for solar work and could benefit from running as fast as possible. A 50 msec setup time gives a 60% duty cycle at the fastest sample rate.
The magnitude of the pipeline requirements has been estimated by benchmarking the pipeline tasks on simulated single channel MMA data. The benchmarking machine was a mid-level workstation, a 60 MHz Sparc 20. To scale the benchmarks to the full MMA pipeline, processor power has been scaled using the 20 fold increase previously mentioned and the number of channels and integration times chosen based on data rate limitations and on the brightness sensitivity of the array. The results are given in Appendix II and show that the pipeline will require approximately 40 workstations or their multiprocessor equivalents. The total pipeline cost of about $800k is obtained by scaling the $20k cost of the benchmarking machine. The overall computing demand is very high largely because of multiple spectral channels. Imaging algorithms operate on a single channel at a time, and the single channel imaging problems are quite modest and well suited to individual processors. Hence, the pipeline requires a data server that makes the appropriate spectral channels' data available to each processor.
The pipeline specifications are quite preliminary and are likely to change. The performance-to-cost ratio may improve more than the conservative assumption of 20 before the MMA is operational, and mosaicing and deconvolution algorithms may become more efficient. At this time, the computing requirements placed on the pipeline do not seem to be too extreme, and the entire pipeline will likely cost under one million dollars.
The most traditional method of single dish mm observing, interactive, will give control of the array to an observer for a fixed time interval. The observer uses feedback from the observations to determine the next observation. The second scheduling method is fixed queue observing which simply steps through a set of pre-planned observations. These may be fixed in sequence or fixed in absolute or sidereal time. In the context of the MMA, this mode would be used when observing must be done at specified times (solar system events, VLBI, etc.) or when a specific set of observations needs to be done independent of other factors. It is also the simplest form of automated observing and will therefore play a role as the array is brought online. The scheduling method that can take maximal advantage of changing weather conditions is flexible scheduling. It uses the current atmospheric monitors (seeing and opacity) and instrument status information to select from the queue the project which makes the best use of telescope time under these conditions.
Remote observing (either from a state-side control center or local 2500m operations center) is a requirement for any of the scheduling modes as it will reduce travel and is essential for flexible scheduling. The transition from the flex mode into interactive will require that interactive projects be elevated into a "standby" state while awaiting appropriate atmospheric conditions and that the observer be notified to be prepared to observe.
There are innovative yet simple ways to extend the flexible queue that will provide all of the features of the traditional mode as well as some interesting advantages. On most occasions the interactive user simply requires one or more `checkpoints' during the observations to enable a decision on the further direction of the project. Such requirements for feedback can be incorporated into the flexible queue so that it will stop a project at a predetermined point, continue with the rest of the queue, and make the project ineligible for rescheduling until there is feedback from the observer. This allows indefinite time (minutes, hours, or even days) for head scratching and reflection without pressure. A simple yet powerful example of the the use of checkpoints occurs while trying to integrate down to an adequate S/N. After spending only half of the projected integration time, a checkpoint can be scheduled, and spectral or spatial smoothing will allow a decent evaluation of the data. Cases in which either the science target has already been met or which will never achieve their science goal within the alloted time should be obvious. This thus argues for a system of 'mid-way checkpoints' for a whole class of observations, with the potential of greatly enhancing the efficiency of the telescope. The exact mixture of classical interactive and the enhanced flex queue will depend on the success of the implementations and will probably vary over time.
A number of the points raised above may seem fairly unconventional, but most of these techniques (flexible scheduling, remote observing, checkpoints) are in use, being planned or in the process of being implemented at telescopes such as the JCMT, BIMA, OVRO, and the SMA. By the time the MMA begins operation, remote observing using flexible queues should be common for mm telescopes.
The complexity of the MMA and heavy reliance on flexible remote operation create a very challenging software design problem for control and monitoring. A long and iterative design analysis period must be anticipated and begun as early as possible. Ideally, the MMA project will assemble a design team with full-time participation of people from the several most recent telescope control software projects. Experience is the most transferable software product, and the best experience is that which has used modern software design and management techniques on closely related projects. The result will draw heavily on industry standards and incorporate the best features of existing systems and packages such as CORBA, EPICS, and glish. However, each instrument and its requirements are different and new enough to require a fresh design. This must be recognized in the project cost.
One lesson learned from other projects is the importance of a thorough analysis of the data flow from the outset of the project. This will ensure a smooth interface between major components of the system. The VLT is a good example of the use of this technique. The flow should begin at proposal submission and include proposal review, observing schedules, actual observing logs, data products, and archival data. Implementation should be done with a single database to maintain a consistent view of the data as it moves through different stages in its life cycle.
At the Tucson meeting in Oct, `96 the science groups presented their requirements which are now summarized.
| Min int time | Channels | Data Rate | |
|---|---|---|---|
| Solar flares & Flare stars | 70msec | 224 | 10MB/sec |
| Single dish OTF mapping | 3msec | 420 | 10MB/sec |
| OTF mosaicing | 1sec | 3200 | 10MB/sec |
| Fast phase calibration | 10sec | 32000 | 10MB/sec |
More detailed justification for these estimates and for the pipeline will be given in an upcoming MMA memo "Computing Requirements of an MMA Imaging Pipeline" (Holdaway, 1996).
| Array | FFT Size | CPU Time c.1995 for 1 Channel |
Number of Channels (N) |
CPU Time c.2005 for N channels |
|---|---|---|---|---|
| 10 s int time: dominated by reweighting and gridding | ||||
| D (80m) | 64x64 | 226 s | 3200 | 10.0 h |
| C (200m) | 256x256 | 237 s | 3200 | 10.5 h |
| B (800m) | 512x512 | 257 s | 1600 | 5.7 h |
| A (3000m) | 2048x2048 | 558 s | 400 | 3.0 h |
| 240 s int time: dominated by FFT | ||||
| D (80m) | 64x64 | 10.3 s | 3200 | 0.46 h |
| C (200m) | 256x256 | 16.6 s | 3200 | 0.74 h |
| B (800m) | 512x512 | 33.4 s | 1600 | 0.75 h |
| A (3000m) | 2048x2048 | 389 s | 400 | 2.2 h |