How can visitors engage with more than their sense of sight?
How can an immersive environment foster more engagement?
How can a designed system adapt to changing behavior and environments?
Today, many museums are facing the challenge of transitioning from rigid institutions to experiential and flexible spaces. This is driven by such factors as expanding collections, increased competition for visitors, and visitor expectations for greater engagement. Museums are turning to virtual reality, apps, and interactive experiences to keep tech-savvy visitors engaged.
Upon arrival, visitors receive their ticket, smart AR glasses, and a palm stamp. Smart glasses reveal AR content and the palm stamp serves as an image anchor for an AR virtual object.
During onboarding, visitors are introduced to Aurum and different gestures that will help them navigate through AR content.
As visitors explore the museum, floor makers indicate the start of AR-enabled exhibits. At the start of the Mineral Hall, visitors are introduced to the narrative of the exhibit (conflict minerals) and can choose a virtual mineral to guide their experience.
Visitors can explore mineral geology, atomic structure, and properties. Properties include fracture, cleavage, radioactivity, fluorescence, and hardness.
After learning about mineral science, visitors can explore human impact stories in a more immersive experience. Stories follow miners, supply chains, or global efforts around conflict minerals.
As visitors navigate through the space, they can discover their virtual object’s physical form and continue exploring more AR-enabled content.
At the end of the exhibit, visitors can engage in a participatory experience by sharing or listening to the reflections of others.
Visitors can navigate through other AR-enabled exhibits.
Aurum uses hand-tracking as an input modality. Visitors can navigate digital content using six gestures.
reveals virtual object
reveals system menu
switch between items
up and down
The Aurum logo was inspired by mineral geometry and Da Vinci’s Vitruvian Man. The Vitruvian Man was created during a time when people shifted their focus from God to humanity. We now hope to shift focus to humanity's relationship with the Earth.
Our research began at the Carnegie Museum of Natural History. We had the opportunity to speak with Rebecca Shreckengast, Director of Exhibition Experience. She gave us valuable insight into the current museum experience from a curatorial and exhibition design perspective.
We also spent time exploring the museum space to observe how museumgoers, including ourselves, interacted with exhibits. After some time exploring, we found ourselves drawn to the Hillman Gallery of Minerals and Gems. This space houses thousands of minerals from across the globe, each on display in large cases. Although the space showcases the beauty of each artifact, minerals feel decontextualized.
On our next few visits to the museum, we had the chance to speak with a few visitors in the Mineral Hall. We approached several different types of people to help us understand how they felt about their experience.
When thinking about increasing engagement within museums, we read through Nina Simon's The Participatory Museum. This book discusses how cultural institutions can become more dynamic, relevant, essential places. Visitor participation is essential when enhancing a museum's experience.
To begin understanding the technology we would be designing for, we took a look at mixed reality and virtual reality tools and experiences. Tools included the Oculus Quest and Hololens headsets and various AR prototyping tools such as Reality Composer, Aero, and Dimension. Experiences included teamlab’s Planets, Punchdrunk’s The Oracles, iart’s Hansel+Gretel, and The British Museum’s Museum of the World.
In our research, we uncovered four key insights. These insights became the foundation for us to begin defining our future experience.
Minerals are decontextualized. The environments, ecosystems, and human relationships within which they are situated are not well represented.
This space requires little participation and engagement on the part of the viewer. Visitors only look at objects, as opposed to interacting with them.
The number of objects and accompanying visual content is a lot to digest, and can easily lead to cognitive overload and fatigue. Further, the content within the Mineral Hall is filled with scientific jargon that visitors may be unfamiliar with.
A few successful experiences with the museum have strong, clear narratives structured around them. How can we create a stronger connection between viewer, artifact, and space by connecting to a bigger picture narrative? How can the space be more relevant to visitors?
Based on our insights, we developed a guiding question. We continued to refine and specify this question as we pushed our concept forward.
With our research insights in mind, we developed 5 design principles to help us better define our experience.
Before diving into ideation, we discussed a larger system framework. Based on our experience navigating through the Mineral Hall, we wanted three guiding phases for our experience:
Visitors will be introduced to the exhibit with relevant background knoweldge.
Visitors will expeirence guided wayfinding and exploration through space.
Visitors can engage in a participatory activity at the end of their experience.
We also discussed leveraging AR smart glass technology. Smart glasses have the ability to create a more immersive, multi-sensory experience while still allowing users to navigate through space.
We then started generating rough concept ideas around our principles, framework, and technology.
We had the opportunity to present our system framework and concepts to our peers and instructors. Based on feedback, we decided to scope down, refine, and dive deeper into the details of our project. We headed into another round of research, ideation, and prototyping.
We began by narrowing our narrative. After discussion, we landed on the idea of a personalized AR object. At the start of each exhibit, visitors will be given a virtual object specific to that exhibit. They can use the object to unlock more immersive learning experiences or content. We liked the idea of this object living in a user's palm, with gestures controlling the experience. This experience would leverage AR image recognition technology with the placement of an image marker on a visitor's hand.
storyboard of our user journey
We also wanted our narrative to focus on human impact to increase the connection between museumgoers and natural history. We hoped to give users an introduction to natural history through an understanding of science, and then allow them to explore natural history within the context of their own lives.
We had a highly iterative design and prototyping process. Researching existing paradigms and physical/digital prototyping were fundamental in helping us understand and push our concept forward.
To make our narrative concrete, we looked into existing interaction paradigms for AR and VR. We looked at Oculus’ hand-tracking design guidelines and at LeapMotion and Hololens gesture-based apps. This helped us think about proximity, gesture controls, and digital signifiers.
Oculus' hand-tracking paradigms: proximity, gesture controls, digital button signifiers/affordances
LeapMotion & Microsoft's Mixed Reality Toolkit: Handtracking, UX/UI Patterns
We started exploring our AR palm object through physical prototyping. This helped us quickly understand how our object might interact within a physical environment. We also started thinking about specific hand gestures that would allow visitors to interact with this object. We then moved into Cinema 4D and After Effects, creating and sourcing OBJ files and pairing them with live footage.
test in museum environment
cinema 4d render
pairing cinema 4d with live footage in after effects
final render test
Users can unlock AR content using digital buttons. We decided to make our system buttons 3D to leverage self-haptic feedback. We explored various button shapes, sizes, and colors by sketching, working in Figma, and prototyping in Reality Composer. We landed on a diamond shape because it aligned with our visual system and signified a pointer.
Our buttons are largely based on proximity. As a user approaches AR content, AR content will reveal itself by scaling up. As users move away, AR content will scale down or disappear.
reality composer prototype
Drawing inspiration from existing AR/VR patterns, we started by hand sketching our UI and detailed interaction flows for both primary and micro-interactions. We moved into Figma to continue exploring higher fidelity mockups on static museum images. This helped us get a better sense of space and proximity. Once we decided on a final UI form, we created all assets in Cinema 4D and After Effects.
initial explorations and user flows
final explorations + proximity test
We wanted to end each exhibit's journey with a participatory activity, allowing museum visitors to reflect on their experience. After various iterations, feedback, and discussions, we landed on the idea of visitors navigating through space, sharing and hearing the reflections of others. We decided to make this experience voice-based for engagement, ease, and consistency.
initial storyboard sketch
cinema 4d prototype
This project was challenging, but a lot of fun. We had to consider 3D digital objects and how these objects would interact with visitors and within the museum's environment. Research, trial, error, and critique helped us figure out different ways to prototype our ideas. Along the way, we learned about paradigms, affordances, and signifiers for emerging AR and VR technology (haptic feedback, proximity, gesture control).
Digital prototyping was a meticulous process that required workflows between various tools. Through the six weeks, we used over 20 different tools:
Cinema 4D, SketchUp, SketchFab
Oculus Quest, Hololens, LeapMotion
Aero, Reality Composer, Unity, Dimension
Figma, XD, Illustrator
After Effects, Photoshop, Lightroom, Audition, Premier Pro, GoPro