The deliverables are crucial because they should provide a basis for the National Science Board's review and consideration of whether to proceed into the construction phase of the project. NSF explained to the Committee that it is essential that the NSF be confident that the cost estimate and schedule for the MMA construction is accurate and that an organization and plan be in place to assure predictable execution of the construction phase.

The Committee notes that it is particularly important during the design and development phase to establish cost and schedule credibility, so that there can be a basis for confidence during the construction phase.

The list of deliverables does not include results from a completed test of a two antenna interferometer. The NRAO explained that delivery of the second prototype antenna could not be accommodated in the planned funding profile in time to carry out the correlated test. The Committee very strongly recommends that the NSF and NRAO work together to identify ways in which to advance the schedule for this test and make it a deliverable of D&D. Money to accelerate delivery of a second antenna by at least 6 months can also act as some of the extra contingency that the Committee desires to see.

There should be a central tabulation of the scientific requirements, with justification for each item, and a description of the consequences if they are not met. This should lead to a central tabulation of the engineering requirements derived from the scientific requirements, with a rationale for the derivation made explicit.

There should be a simple, compact tabulation of the error sources, with formulae given for the impact of each one on the system performance.

A useful tool would be a Design Reference Mission (DRM) similar to that developed for the Next Generation Space Telescope and other NASA-missions, spelling out a representative set of observations to be made, with all observing parameters noted, for use in modeling the system performance. Not every observation requires the utmost in performance of every subsystem, and a DRM allows an evaluation of trade offs between achievable science and technical compromise. (The NGST DRM is described by Eric P. Smith et al. in a paper for the SPIE Kona meeting (March 1998), and a hard copy, Word document or postscript file is available on request.)

The Committee should think more about what constitutes supporting documents for the deliverables, especially as this affects the cost for the development phase. As we proceed, it may be worthwhile to assemble a more specific list of what the Committee wants to see and to make sure that NRAO is clearly aware of what is expected of them

The review package and presentations did not include much information about the basis of cost estimation (except that it was estimated by working engineers), the basis of the schedule, the plans for contingency funds, descope options in the event of problems, PERT charts, critical path analysis, or other tools for project management. The plan for the construction phase is clearly to be the result of the upcoming study, but the plan for D&D does not include these items and it should do so as soon as possible.

Before we can comment on the adequacy of the project management team, we need more information about the track record on prior projects, and the lessons the team has learned from them. We have not seen any job descriptions, and the distinction in roles among the project director, project manager, and project scientist was not explained. Usually a proposal has this sort of information or a project of this magnitude writes a formal project management plan which delineates these items.

The planned staffing estimates are just level with time, and do not reflect detailed consideration of the project needs or the availability of staff as they move off other projects. A new level of planning is required.

Development of a detailed, low level work breakdown cost estimate and schedule for the entire MMA project should be completed prior to the next Committee review.

There should be a detailed Work Breakdown Structure, Cost Estimate and Schedule for the entire 9 year MMA Project, including D&D and the Construction Phase. While a definitive version of these deliverables of D&D, the project definition and organization are sufficiently mature to warrant initial preparation of these plans now. They will provide a guide to the work of D&D, support the smooth transition to the subsequent phase, and by early preparation, offer the opportunity to maintain and perfect these plans during D&D.

The WBS should be prepared down to approximately level 5 or lower for all deliverables. As the WBS will be the basis for the structure of the cost estimate, it should be product-oriented and support rollup of all costs associated with delivery of each subsystem. The WBS should be accompanied by a WBS Dictionary which will serve to define the scope of each costed element and support development of interface definitions. There should be a cost estimation book, providing the description of each item in the WBS, the basis for its cost estimate, and a justification of the appropriate level of contingency for it.

A detailed cost estimate should be prepared for each element at the lowest feasible level of the WBS with a rollup to higher levels. For each costed element, the estimate basis should be documented so as to support audit of the costed elements. Project management should establish uniform estimate procedures for all estimators.

A schedule should be developed following the WBS. Schedule milestones should be developed for several hierarchical levels. Top level milestones define the most significant program events and should be documented for use by the NSF in assessing program schedule performance. Lower level milestones should support oversight by project management and detailed milestones should define measurement of performance for subsystem managers. Critical paths in the project schedule should be identified. In addition to the Gannt methods presented for the Committee's review, a schedule logic network (Pert network) should be developed in order to define task dependencies. Each subsystem must develop detailed schedules showing clearly the critical path.

The Committee felt that the cost estimate and planned funding provide the necessary resources with only minimal reserve for management of technical cost and schedule risk. The minimal level of contingency included in the Plan may require reductions in the scope of the Plan (scope contingency). The Committee explored possible scope modifications with the NRAO concern and found that the options are limited. This is a serious concern. For each estimated element in the WBS, an assessment should be made of the technical, cost and schedule risks. This assessment should be the basis for estimated contingency funds for each element to be rolled up into a contingency fund to be held and allocated by the project management. An adequate contingency fund should be a part of the cost estimate presented for the Committee's next review. They should increase the cost reserve of the program to 15 % minimum of the total contract value (which was given as $26M).

There must also be schedule reserve; a minimum of 1-2 months schedule reserve in each of the subsystems (e.g., Antenna, Receivers, ...) would be an example, although NRAO should determine the complexity of the subsystem and allocate the appropriate reserve. They should also have program schedule reserve of 2-3 months which does not include the subsystem schedule reserve. Part of contingency planning is also an awareness of descope options, and Project Management must develop these options as the baseline program becomes clearer.

We are currently depending on the NRAO to hold its own technical review boards and to request community participation for technical input. We anticipate reviewing their process for doing so at a future date.

The single largest cost driver for the MMA project is the procurement of the antennas. It is not economically feasible to significantly upgrade antennas during the life of the instrument, while the electronics becomes better and relatively less expensive. The D&D Program Plan defines a procurement process which consists of final antenna design by the NRAO, competitive selection of a single vendor for a performance based contract to certify or modify the design and produce a single prototype antenna with an option for a second prototype, followed by a negotiated and "build-to-print" phase to fabricate the entire set of antennas. This procurement strategy can be successful but it contains a number of technical and cost/schedule risks. The NRAO design may not adequately incorporate the capabilities of industry and may yield a design with constructibility issues, leading to cost and schedule escalation. The early selection of a single source narrows the field of potential vendors for the full fabrication at the outset and during a comparatively low cost phase of the procurement. This leads to dependence upon a single source for the later phase, the project's major cost driver. Though the NRAO intends to purchase the design in the initial phase, passing that design to other sources risks additional constructibility and cost and schedule issues. The list of experienced sources is short and has recently contracted through merger.

The Committee feels that the managers of the MMA project should consider ways in which to include early industry participation in the antenna design and to maintain the competitive commercial atmosphere later into this procurement. It is worth involving as many people as possible in the design reviews, particularly the engineers at the firms that could bid on the antennas. It is unlikely that the antenna bidders will have the manpower to do a significantly different design from the NRAO design, but they could well suggest improvements to the design that would be particularly helpful in reducing manufacturing costs. NRAO could involve the few most capable prospective bidders (as demonstrated by earlier performance in similar projects) in the last phases of their own "conceptual" design It would be all right to spend some money on that (provided the company puts in at least as much itself). One should work on very "practical" issues like repetition and mass production, an area where NRAO's structural design team has little, if any, experience. The money spent could be returned many-fold in the construction of the antennas, if they were more optimized towards the advantages of mass production.

There is some unease about the balance of the D&D on receivers in which the concentration of effort will be at frequencies where NRAO has many strengths; there is no doubt that NRAO can do both the HFET receivers at the many lower bands and 230 GHz SIS receivers even with the incremental improvements in side-band rejection and so forth that they have earmarked.

It seems important that NRAO get an early start on the higher frequency bands that they currently do not have experience in, namely 500, 60O, and most challengingly, 900 GHz. As currently constituted, this expertise is not included in the MDC. Since the cost of the receivers would be an important item for production, during D&D it is crucial to understand the scope and the many technical hurdles for these bands. With the SMA, the procedure is to concentrate on designing and constructing the lower frequency receivers first while doing the research on the higher frequency receivers. If NRAO really wants to nail down the cost of the high frequency receivers during the D&D phase. they will need to budget more effort for them and build up their receiver group.

The mean time before failure (MTBF) for such a large system should be estimated, and the system should be designed to accommodate it conveniently. There is a large observational penalty to having equipment that is broken down, and there should be a specification on the amount of tolerable down time.

As stated earlier in this report, the Program Plan contingency provides minimal resources for response to unforeseen program issues. As previously emphasized, we also recommend the development of a full WBS, cost estimate and schedule for all phases of the project in order to support the timely transition to construction.

Discussions with potential international partners, or with non-NSF domestic support sources, call the Program Plan and the technical cost and schedule baselines into question. All of these elements of the Program Plan are potentially subject to substantial change at a later date as a result of the development of one or more of these partnerships. This may lead to schedule stretchout and cost escalation which may offset some of the advantages derived from the partnership.

The Committee is greatly concerned with the program complexity introduced by the negotiation of partnerships in the Project. The NSF and the project managers must advance the negotiation process in order to bring it to a conclusion at the earliest possible time in order to minimize delays. As far as Committee members understand the situation, the Europeans, and ESO in particular, are quite serious about their plans for a large millimeter array and see the collaboration with NRAO as a good thing, certainly politically. The goal for "their" array surpasses the MMA size significantly, which means that ESO will not likely accept being "junior" partner. Clearly, if a collaboration at the 50-50% level would emerge, this would lead to an instrument (60-70 elements) of truly impressive performance. On the other hand, the challenges of assembling such a major collaboration add great risk to the entire program. NSF should actively support the NRAO in negotiation on collaboration, probably even take a lead in that area, regardless of whether it is an international or domestic (presumably NASA) collaboration. The process for achieving a mutually agreeable plan with (foreign or domestic) partners needs to be made explicit. with a schedule for the joint activities made clear. There should be definite decision points on the calendar, and regularly scheduled meetings to work out the details. NSF should also be very careful that arbitrary requirements for a collaboration do not increase eventual costs to NSF for comparable scientific accomplishments and do not increase out of proportion the leverage of the collaborator. By contrast, major NASA science programs now being planned (NGST, SIM, etc.) encourage but do not require collaborations outside of NASA.

The Committee wants to gain a clear understanding as soon as possible as to just how any collaboration will be assembled and what would be the conditions and consequences. The technical and management plan for the construction phase will be very dependent on the results and even the scientific goals of the MMA are at stake.

The configuration management should be formal enough that any changes in the MMA specification after the MMA project decides to put the specs under management control, be visible and/or under the control of upper management at NRAO and the MMA project. Hopefully, by deciding which science goals drive which specifications, some trade-offs in costs can be made. It is important to understand how the flow-down from science requirements to instrument specification follows a cost/benefit analysis.

The management structure reflects the deliverables from the D&D Program Plan. The Committee discussed the need to free the MMA managers from other NRAO responsibilities in order to assure management focus on this program. The Committee learned that NRAO management does plan to make appropriate reassignments to realize this organizational clarity. Once a complete WBS and program plan is prepared for the later project phases, the organization will require adjustment to reflect the later phase deliverables. It should be very clear who gets to decide major items, and the path to resolution of dispute must be clear.

The management structure must be bullet-proof, in the sense that large scientific projects have been canceled at the Congressional level because the project management was accused of being out of control. This means many things. Budget status, progress on the agreed schedule; and staffing levels must be available instantaneously on request from outside; there must be a traceable derivation of requirements; there must be clear control of the contractors, with sound auditing procedures in place; there must be strong, recognizable central leadership; there must be firm configuration management, so that requirements creep is well managed, and "the better is the enemy of the good" is well understood (Voltaire, 1764: "le mieux est l'ennemi du bien.") This is a matter of management and organizational culture and must be established by high level leadership.

NSF should make sure that the MMA project understands the ground rules under which a project could be canceled for cost or schedule overrun; for NASA, a projected overrun of 15% is sufficient for cancellation. This is an extremely important point for agencies whose very survival depends on credibility with Congress. It is also important in giving negotiating power to the MMA project with its contractors.

The importance of clearly defined flow-down from science requirements to technical specifications was emphasized in comments for the first question. Some members of the Committee would be more comfortable with a management structure for the MMA project more formally and rigorously defined than has been the NRAO norm. Such a structure would require, for example, a Systems Engineer (SE) who has custody of the MMA Top Level Specification and should produce this spec. An analogous science spec. would be produced by the Project Scientist. The MMA System Specification (sometimes called Contract End Item) would also be produced by the Systems Engineer. Either SE or the Subsystems should produce Subsystem Specifications; if the Subsystems produce them, SE should approve. Because several organizations are producing subsystems, it would be essential that interface control documents be developed for the program by SE. The Committee realizes that details of the management structure are the responsibility of NRAO, but we also caution that the experience of several Committee members with projects in the cost and complexity range of the MMA has led to an expectation of well defined management and interface mechanisms.

The NSF appears to see this as a self-contained D&D Project, at the completion of which a decision about the construction will be made. The NRAO on the other hand clearly considers the D&D phase as a first episode of the entire MMA Project. It has structured the management of the project accordingly and may have difficulties accommodating the NSF boundary conditions. It would be in the interest of the Project if the NSF could be more flexible and slacken the strict separation between the D&D and the Construction phases. This could be beneficial to the creation of a more complete and detailed management and personnel plan of the Project, and by implication of the D&D phase proper.

It is appropriate that there be some flexibility about the boundary between the D&D phase and the construction phase. Many items acquired in D&D will be used in the final instrument, and investments made in equipment to be used in the fabrication process will need to be made before the construction phase starts.