Alternate Realities

Standard operating procedures are at the heart of spacecraft operations. Executing these procedures on the International Space Station (ISS) is heavily dependent upon ground controllers assisting crewmembers in performing routine operations and maintenance as well as responses to off-nominal situations. This close collaboration becomes more difficult in exploration missions that take human crews beyond the easy reach of Mission Control, so crewmembers will need to have more autonomy. Augmented and/or virtual reality technology can help replace some of the guidance that ground controllers offer to crewmembers during procedure execution [1]. In support of NASA and in collaboration with the Georgia Institute of Technology Augmented Environmental Lab, TRACLabs is working to integrate its PRIDE system [2] with augmented and virtual reality technologies in a system called PRIDE Augmented and Virtual Reality (PRIDE-AVR).

Augmented reality is a live direct or indirect view of a physical, real-world environment whose elements are augmented (or supplemented) by computer-generated sensory input such as sound, video, graphics or GPS data. Virtual Reality is a realistic and immersive simulation of a three-dimensional environment, created using interactive software and hardware, and experienced or controlled by movement of the body [3]. Hybrid reality, sometimes known as mixed reality [4], is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time. With PRIDE-AVR TRACLabs is investigating all three alternative realities in support of NASA.

Our PRIDE-AVR design is shown in the figure below. It builds on several existing components, including the PRIDE electronic procedure platform and a system ontology.  The former includes a procedure authoring tool (PRIDE Author) and a server (PRIDE View Server) that shows procedures as web pages. The latter includes an ontology editor (PRONTOE [5]) and an ontology server. Newly developed components are an alternative reality (AR) Server and new PRIDE View clients for the different alternative reality systems shown on the right of the figure.  As the user executes a procedure, the AR Server tracks the progress and looks for references to individuals in the domain ontology. The AR server queries the ontology server for any alternative reality attributes of those individuals then sends the information to the AR, VR or HR system involved in the procedure execution.

1 Arthur Tang, Charles Owen, Frank Biocca, and Weimin Mou. Comparative effectiveness of augmented reality in object assembly. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 73–80. ACM, 2003.

2 Izygon, Michel; Kortenkamp, David; Molin, Arthur. (2008). A procedure integrated development environment for future spacecraft and habitats. Space Technology and Applications International Forum (STAIF). Albuquerque, NM, American Institute of Physics.

3 http://www.dictionary.com/browse/virtual–reality

4 de Souza e Silva, Adriana; Sutko, Daniel M. (2009). Digital Cityscapes: merging digital and urban playspaces. New York: Peter Lang Publishing, Inc.

5 Bell, Scott; Bonasso, R. Peter; Boddy, Mark; Kortenkamp, David (2013). PRONTOE: A case study for developing ontologies for operations. 5th International Conference on Knowledge Engineering and Ontology Development (KEOD 13). Algarve, Portugal.

6 Blair MacIntyre, Alex Hill, Hafez Rouzati, Maribeth Gandy, and Brian Davidson. The argon ar web browser and standards-based ar application environment. In Mixed and Augmented Reality (ISMAR), 2011 10th IEEE International Symposium on, pages 65–74. IEEE, 2011.