First and foremost, the MMA will be a wonderfully powerful instrument and it should be constructed and exploited. The technical concept is fundamentally sound and the technical innovations seem to be within reach. The current MMA plan is the product of more than a decade of serious study by the NRAO. An archive of about 200 technical memoranda, now being produced at the rate of about one a week, lists the main technical considerations and the approaches to their solution. The many innovations will be demonstrated in production hardware within the Design and Development phase of the project. The MMA project has made considerable progress since our first review. The Project Book (version 1) and draft Management Plan represent good initial steps in forming the necessary baseline definition and management structures. This review report recommends a number of steps to continue this development. Our report uses the NSF Charge to structure our response.

At its second meeting the MMAOC is requested to consider the following issues:

* NRAO has submitted the report Recommended Site for the Millimeter Array to NSF, requesting that the Foundation approve the selection of Llano de Chajnantor as the site for the Millimeter Array. Does the MMAOC concur with the NRAO's recommendation of the Chajnantor site?

The MMAOC concurs with the NRAO's recommendation.

The site selected, the Atacama desert in northern Chile, is a marvel. It is adjacent to the driest known site in the world. For radio astronomy it has superior atmospheric conditions relative to all other tested sites, except perhaps for the South Pole. It is readily accessible by a paved road, and is only 20 hours from the US mainland by plane and car. It has recently been designated as a special scientific reserve by the Chilean President, and it is clear that the MMA has strong support within the Chilean community. Other organizations like the ESO have found it possible to build marvelous telescopes like the VLT in Chile, so clearly the logistical problems can be solved.

Very few, if any, new observatory sites have been so well studied for their adequacy as the Chajnantor site. In addition, NRAO has done an exemplary job of preparation of the site and observatory operation by its intensive contacts with Chilean government and local entities.

The chapters in the Management Plan on the site and the planned operational mode of the MMA are excellent. They provide confidence that the MMA can be built and operated in this remote and deserted part of the world. The approach to the operation, based on the 'turno' principle, is sound. It is indicative of the care with which the broader issues of establishing the MMA in Chile are handled, that Eduardo Hardy has been hired already now to foster the necessary contacts between NRAO and Chile.

* Please provide a critique of MMA project planning and organization, as proposed by NRAO in the Millimeter Array Management Plan, the Millimeter Array Project Book (Version 1.0), and as described at today's MMAOC meeting. In doing so, please provide answers to the following questions:

1. Does the MMAOC believe that the proposed management plan and work breakdown structure will lead to the successful completion of the MMA project's design and development goals?

By funding the 3 year long D&D phase, NRAO is given the opportunity to provide proof of its capability to bring the ultimate and entire MMA project to a fruitful completion. This entails not only solving technological challenges and obtaining economic bids for the antennas and possibly other parts of the equipment, but also setting up a necessary and sufficient management structure for the entire project and to 'practice' this structure from the start, during the D&D phase. This perspective motivated the first report from this Committee. NRAO has made a significant start in responding to the MMAOC recommendations.

The draft Management Plan includes a statement of goals and objectives, top level specifications, a plan for the project's organization and the relationship of key institutions responsible for MMA, information about key staff, and an initial Work Breakdown Structure (WBS) for the entire MMA project. The MMAOC was informed that work is continuing in development of a detailed WBS, a WBS Dictionary, and a bottom up cost estimate based upon the WBS. These are all essential for successful completion of the D&D phase and the entire project.

The MMA project team should capitalize on its good start and proceed to vigorously develop the WBS description and cost estimate for the entire MMA Project. MMA management should continue to place high priority on this activity to insure that adequate staff attention is directed to complete this important exercise.

The Project Book will develop into the guiding document for the MMA team's work plan. This is clearly stated in the Management Plan.

The NRAO is wisely planning to maintain the Project Book on-line, under the control of the System Engineer, as the constant and definitive source of baseline information, and including an audit trail of changes. However, the process by which the MMA configuration will evolve and be revised under control is not yet defined.

For this formative period in the MMA effort, the Project Book should provide a clear statement of the technical baseline, as opposed to a listing of alternatives without definition of the default baseline. This technical baseline would guide the remaining technical R&D and the selection among options, the establishment of the schedule, and what is included and treated in the cost estimate.

The Project Book (version 1.0) reviewed by the MMAOC covers all technical subsystems and outlines many of the remaining options. The considerable effort that has gone into assembling this Project Book should continue so that at the next review, the MMAOC can review the technical baseline definition for its clarity and adequacy to support the functions mentioned above. For each major technical option, the Project Book should define the working technical baseline. Subsequent selection of an alternate, after development work is completed, should be governed by a change management process defined in the Management Plan.

2. Are there issues which have been incompletely treated or omitted from the Plan or Project Book?

Both the Management Plan and the Project Book are rapidly developing drafts. We list below issues and topics to be included in subsequent drafts. In many cases, the NRAO has stated its intention to develop these areas and to include them in the documents.

The Management Plan should include:

** a. A clear list of top level milestones that represent commitments to the NSF. These milestones should be reported on in all regular reports to NSF. Projected delays in achieving these milestones should be matters of significant discussion with the NSF.

** b. A summary of the complete, bottom-up cost estimate for the entire MMA Project baseline.

** c. A configuration management plan that includes how the technical baseline is defined, documented and controlled, and how technical, schedule and cost baseline changes are initiated, documented, reviewed and incorporated in the controlled project configuration.

** d. A statement of how major subcontracts are planned, entered into, reviewed and managed.

** e. A statement of how the project management will coordinate this project with its many sites in the United States and Chile.

** f. A statement of how responsibility for environment, safety, health and quality assurance will be assured and managed.>

The Project Book should include:

** a. For each subsystem, a clear statement of the technical baseline that underpins design, planning and cost estimating. This should include key controlled parameters and specifications.

** b. For each subsystem, a clear statement of remaining design options to the baseline, the criteria governing the possible choice of the option and the schedule point at which the choice must be made.

3. Is the proposed management structure matched to the project's goals?

The proposed management structure is generally well matched to the project's goals. The MMAOC feels that two issues warrant attention.

** a. The System Engineer is critical to assuring the technical functionality, to the successful integration and to establishing, controlling, and resolving interface and configuration definitions. To these ends, the System Engineer should be placed in the organization above all of the subsystem leaders and report directly to the Project Manager.

** b. The geographic dispersion of the MMA in the US and at several sites in Chile is a substantial management challenge. It has been accomplished successfully by prior projects such as the VLBA and projects outside of astronomy. But MMA presents its own unique challenges and the MMA team should focus on developing the necessary methods of coordination. The Project Manager's "presence" at and insight into each of the sites on a continuous basis is critical.

The sharing of information and goals between teams at the several sites and the development of a common sense of teamwork is also essential. MMA management should develop management techniques (such as electronic support of meeting and group work), plan for frequent travel and exchange of team members between sites and express continuous management priority in support of the overall MMA success.

4. Has NRAO responded satisfactorily to the MMAOC's previous recommendations?

The NRAO has taken positive steps to respond to many of our recommendations. Tests of two prototype antennas do not appear to fit into the D&D phase. Flowdown requirements are being defined. WBS, cost and schedule baselines are in the very early stages of assembly. The antenna procurement process is under discussion with potential suppliers and the NRAO is attempting to deal with the complexity of this overarching procurement. The antenna procurement should be reviewed again at our next meeting. International partnership is being addressed. Management planning has been initiated, but as this report indicates, much remains to be defined. The Project Book has begun to encapsulate the flowdown and baseline definition but this area has much work ahead. The recommendation to treat the end of the 3 year D&D phase as a review point, and not a pause between project phases that interrupts or postpones progress appears to be recognized. The NRAO should work to plan the smooth passage through this milestone as uneventful as possible.