Office of Public Outreach’s 
Online Interactive Educational Resources:

The Review Processes for Amazing Space

– Carol Christian 12 October 1998

 The review of OPO's educational resources specifically aimed at providing curriculum support progresses in a number of stages. During the authoring of OPO resources during the core Amazing Space summer workshop, progress reports are made to an executive committee of diverse expertise (science, education, design, programming, management).  Storyboards, key concepts and in some cases, prototypes are created and reviewed. The core authoring phase takes five weeks with at least three checkpoints and numerous interactions between the executive committee and author teams.

The production phase follows the authoring phase, and in general, involves only the OPO personnel required to bring the educational resources to completion and readiness for public release as well as management oversight. In production, there is a series of checkpoints to review content, texture, graphic design, readability, etc. By the time the resources are ready for a Beta testing phase, they should be fairly robust, having completed the review process according to published guidelines. Revisions are implemented on the basis of Beta review and then resources are released for public use and further testing, followed by further minor revisions, if necessary.

In summary, the reviews conducted are:

    1. Progress reports to executive committee
    2. Numerous consultations between development teams and

      3. executive committee
    3. Presentation to entire Amazing Space team
    1. Review and clarification of storyboards and concepts
    2. Review of prototype pages - content, texture, readability, usability, graphic treatment, design of interactives
    3. Review of design by experts in a variety of areas (graphics, writing, science, multimedia, instructional technology)
The internal review is the checkpoint at which the resources are assessed by the developers and other personnel with particular expertise specifically chosen to participate. The result of the review is an assesment of the resource according to criteria that are similar to those used for Beta testing. In addition, developers have the responsibility to insure web integrity. Amazing Space in OPO developers conduct four tests before resources are posted on the internet. Internal review may result in revisions to the resources.

Once the revisions recommended by internal review are completed, the resources are submitted to Beta testing by external participants. The participants include teachers across the country and participants in local and regional workshops. The workshops by trained personnel who are familiar with the resources and who
can provide feedback to the Amazing Space project regarding the utility of the resources. The review/testing
matrix is the framework for this activity. In situ testing is minimal at this stage, except for ocassions where
resources are demonstrated in a workshop or seminar setting. In the future, Beta test participants may include the Aerospace Educational Services Project (AESP) interns, NASA Education Resource Centers (ERC) hosts, and other agencies.

Required information regarding Internal Review and Beta Testing conditions:

Certain information is required by the developers to understand the feasibility of using AS in diverse environments. However, internal review concentrates less on the technologies required than beta testing, especially connectivity although some testing of this sort occurs during the web integrity tests. According to the guidelines, the following information is required:

Criteria used for Internal Review and Beta Testing:

Keeping in mind the review/testing conditions above, internal review and/or Beta testing should result in, first, assessment of the resources according to five criteria documented in greater detail in the guidelines:

  1. Content
  2. Science and Technology
  3. Pedagogy
  4. Useability
  5. Multimedia Design
Classification resulting from Internal Review and Beta Testing:

The review and testing cycles also result in identifying the resource within a classification matrix: first, a "content classification" of the resources according to a system adopted within the NASA education and public outreach community (both Earth Science and Space Science). Additionally the resources are classified in terms of recommeded technology and also connectivity for use. Classifications also are used for cataloging and archiving resources for user search criteria. Each column in the following table is independent. Classification does not necessarily result in a revision cycle, but is informative in nature.
 
 
Content Classification
(recommended by reviewer)		 Computer Technology Classification 
(recommended by reviewer)		 Connectivity Classification
(recommended by reviewer)
Turn key - use as is Simple; Netscape 3, 486 cpu, or other  Simple; 14.4 bandwidth or other 
Requires training to use Netscape 4, 90MHz processor minimum  28.8 bandwidth
Good resource, needs some improvement (see results of design review) Freeware plug-ins required, 133MHz cpu minimum T-1 or ISDN required
Resource needs significant improvement (see results of design review) Specialized software or plug ins, 200MHz processor High speed network required
Resource unusable

 

Field testing is accomplished through a network of teachers on the World Wide Web after public release of the resources. When appropriate, field testing with students is also conducted via an outcome evaluation. The purpose of an outcome evaluation is to determine the impact an Amazing Space activity has upon students’ learning. Specifically, a team composed of an internal and external evaluator field tests the lessons with students and makes recommendations based upon their performance. Revisions are then implemented on the basis of these recommendations.
 
 

Amazing Space: Nominal Timeline

 
Dates Activity
October–November Draft and publish teacher recruitment notice 
Contact science supervisors to articulate purpose of program and publish teacher recruitment notice 
November– December Formulate changes in Amazing Space based on lessons learned from previous year 
Program cost accounting
Jan–early Spring Deadline for teacher applications 
Review applications 
Recruit and select teachers
Jan–early Spring Plan topics to be covered 
Arrange for facilitator training 
Recruit Institute participants 
Collect examples of "good resources"
May Conduct teacher orientation 
Finalize materials 
Negotiate teacher contracts
End of June–July Conduct teacher workshop
August Clarify and review storyboards 
Produce prototype graphics, programming
September–
October		 Beta release of resources 
Beta testing of resources
November Plan products 
Complete Beta testing and revision of modules 
Plan public release
December–
January		 Finalize public release 
Continue testing 
Make revisions 
Plan national workshops
March Showcase resources and release products at professional meetings for educators (e.g., National Science Teachers Association, National Council of Teachers of Mathematics)


 

Updated 15 July, 1998 by C. Christian, Head, Office of Public Outreach, Space Telescope Science Institute, carolc@stsci.edu.