Design Beyond a Screen: A Primer for VR, AR and the Multiverse

Posted 6 months ago by Graeme Fulton

Since the first iPhone, standard screens have been the medium for which we’ve been designing for most – usually apps and websites. As technology has advanced though, new avenues have slowly emerged (e.g. Augmented Reality and Virtual Reality) through which we can consume media and information in different ways.

“VR and AR are just 2 realms we can design for today – there’s actually lots more.”

Many of these mediums involve blending the real world with digital information in quite unique ways. A popular AR example for instance, is IKEA’s upcoming Place app, which lets us overlay our living rooms with items of furniture to see how they’d look (or even if they fit in our house).

Augmented Reality (AR) and Virtual Reality (VR) are just 2 of the realms we can design for today though – there’s actually lots more, with each bringing a host of opportunities for app developers. Each realm is made up of a different combination of Time, Space and Matter – check them out in the contents below:

Contents: Realms of opportunity

  1. Reality: Time, Space and Matter
  2. Augmented Reality: Time, Space, No-Matter
  3. Augmented Virtuality: No-Time, No-Space, Matter
  4. Alternate Reality: No-Time, Space, No-Matter
  5. Physical Virtuality: Time, No-Space, Matter
  6. Mirrored Virtuality: Time, No-Space, No-Matter
  7. Virtual Reality: No-Time, No-Space, No-Matter

A lot of the time, all these realms tend to fall under the umbrella phrase, ‘Mixed Reality’, but each term actually has its own meaning depending on the amount of reality and ‘virtual reality’ involved.

“Each different combination of Time, Space and Matter brings different opportunities.”

In this article we’ll explore each one, and uncover ideas and inspiration for creating apps and experiences that combine Time, Space and Matter in different ways. It might give you some new avenues to explore when coming up with your new product! Let’s start by understanding this multiverse:

8 realms of opportunity

Reality and Virtual Reality sit at complete opposite ends of the multiverse, with a range of others realms fitting in between. They can be visualized on this octant, from the book, ‘Infinite Reality: Creating Customer Value on the Digital Frontier’ by B. Joseph Pine and Kim C. Korn, which this article is inspired by:


Eight realms of the Universe, by B. Joseph Pine and Kim C. Korn



Reality to Virtual Reality

The main difference between all of these realms of experience is the relationship between Time, Space and Matter. It’s more straight forward than it looks – let’s start by understanding the 2 extremes, Reality and Virtual Reality:

Reality consists of time, space and matter.

Reality is the realm against which we can benchmark and compare everything with. In reality, all 3 components are present: Time, Space and Matter. It’s the world we live in where we can move through space and touch physical objects.

“Reality is the realm against which we can benchmark and compare everything with.”

The complete opposite realm is Virtual Reality, where Time, Space and Matter as we know in reality don’t exist. All 3 of these components are created depending on the virtual world being explored. It’s a bit like the Matrix – none of it is actually real.

In between these 2 extremes lie many realms, often simply summed up as Mixed Reality. However, each different combination of Time, Space and Matter brings different opportunities. For example, removing ‘Matter’ from Space and Time results in one of the most popular experiences today – Augmented Reality, where digital objects overlay the physical world. Let’s look into it:

Augmented Reality

Augmented Reality: Time, Space and No-Matter

We’re most familiar with AR these days. It’s when we have Time, Space, but the ‘matter’ doesn’t exist. We use the time and space from reality but overlay it with something digital, which presents us with a host of opportunities as outlined below:

Augmented Reality Opportunities

We’ve seen loads of AR over the last couple of years, with games like Pokemon Go being the most popular. Therefore, here’s just a couple examples since we’re already so familiar with these:

Online Retail

A popular example has always been to see how furniture would look in your house. Both Shopify and IKEA have been experimenting to make this a real possibility:

Tools for the World

As seen in both IKEA’s and Shopify’s AR apps, being able to recognise space and surfaces with your phone’s camera makes AR experiences much more powerful. Using Apple’s AR Kit, we can:

These features make it possible to develop apps that can replace physical tools such as rulers:


TapMeasure, an AR App for measuring things


Gaming
As well as opportunities for tools and productivity, Augmented Reality provides a massive playground for games too. Unless you’ve been living under a rock, you’ll have heard of a few so I won’t go into detail. Think Pokémon Go for instance.

Where to start with AR

Overall, we’re augmenting what we can see in reality, with whatever technology we have available. At the moment, that’s often our phones, and therefore usually involves the camera of our mobile device. Therefore when it comes to building your own AR app, you can start with Apple’s AR Guidelines which has a host of tips for designing for AR using a phone’s camera.

Another great starting point is this article by Designlab:

What Does the iPhone X’s AR Engine Mean for UX Designers?We also explore what AR could mean for UX designers, and, finally, our take on the $162bn question: are people going to use it?

Augmented Virtuality

Augmented Virtuality: No-Time, No-Space, Matter

Augmented Virtuality can be seen as an opposite to Augmented Reality in a way. Instead of overlaying the real world with digital stuff, we augment the virtual world with something from the real world. There’s less examples of this than Augmented Reality, but there’s still plenty of opportunities. One of the most prevalent examples of Augmented Virtuality are input devices like remotes and game controllers. A couple are the Nintendo Wii Remote, and more recently, Virtual Reality Gloves:

Nintendo Wii Remote

Augmented Virtuality Example: Nintendo Wii

In it’s time, the Nintendo Wii control was a gamechanger, bringing a new level of interactivity to games that wasn’t previously possible. Instead of pressing buttons to play a tennis game, we could actually carry out the motions of a tennis player and have it transfer to the digital realm.

Analog to digital

A key part that made this possible was the use of an accelerometer to detect direction and force applied to the Wii remote. That’s done by measuring movement along the X, Y and Z axis, as well as rotation. These analogue forces from the real world are then translated into digital data and sent to the Wii via bluetooth.

“Analogue forces from the real world are translated into digital data…”


Sensors that enable Augmented Virtuality


Watch this video to find out exactly how it works:

Virtual Reality Gloves

Moving on from the Nintendo Wii, and into a Virtual Reality setting, the same type of concept is present for VR gloves. However, the gloves not only detect the orientation of each hand, but also the bending of each finger using ‘flex sensors’. The number of sensors required increases with the amount of movement needed to be detected, which makes them much more complex. All the data collected from the gloves provide enough information to build a skeletal model of the hand.

“Data can be transferred from the digital world to the gloves, so users can feel items via haptic feedback.”


VR Glove from Manu


Going further than that though, data can also be transferred from the digital world back to the gloves, so users can feel, touch and hold items via haptic feedback.

Manus VR have designed their gloves to be used specifically for virtual reality, but they could actually be used for any digital environment. Check out the video below to see the hand tracking in action through a Surgeon Simulator game:

There’s more information here on how the sensors work:

Getting started with Augmented Virtuality

To get started with Augmented Virtuality, you don’t really need any complex hardware. For example, here’s a guide to building your own Wii controller in just 10 minutes. And here’s a couple more projects from students, using every day objects from the physical reality to control digital worlds:

Alternate Reality

Alternate Reality: No-Time, Space, No-Matter

Alternate Reality only interacts with the spaces of reality (e.g. fields, roads, surfaces). It’s quite similar to Augmented Reality, but in addition to a digital overlay on our world, time becomes fictional. It’s often based upon a storyline with it’s own concept of time. It’s defined pretty well in Wikipedia too, like so:

“An alternate reality game (ARG) is an interactive narrative that uses the real world as a platform, often involving multiple media and game elements, to tell a story that may be affected by participants’ ideas or actions.”

Essentially, events that normally take place online are brought to the real world across multiple channels. Popular examples from the past have often been marketing campaigns for product launches. Two notable ones are below:

  1. The Dark Knight promotion for the Batman movie

    The Batman

    The Dark Knight ARG immersed over 10 million people in a ‘real world Gothem City’, where the story was played out through an array of different channels. From playing casual games, to receiving text messages and phone calls, players could unlock pieces of the larger story, experiencing an alternate reality. Check out this case study video that sums up the entire thing:

  2. Another similar example is the I Love Bees promotion for Microsoft’s Halo 2Read about the story and game here.

Difference between ARG and AR

Since Alternate Reality Games are played out in real-world environments, they can be confused with Augmented reality games today, such as Pokémon Go or The Walking Dead. Pamela Hogle writes about the differences between the two in her article for Learning Solutions Magazine. A summary of those differences are outlined below:

Alternate Reality

Augmented Reality

Quick starting tip for Alternate Reality

As summarised in the table above, the key components for alternate reality games are a good storyline, and a network of users to play with. A good starting point could be this inspirational list of popular augmented reality games and software.

Physical Virtuality

Physical Virtuality: Time, No-Space, Matter

Physical Virtuality is the combination of real time and real matter, but the space used is that from the digital realm (no space). It’s actually really common, enabling us to take advantage of digital environments to produce things for the real world. Notable examples are shops like Vistaprint, TShirt Studio, and even Moo. They all enable us to try out different designs and see exactly how they’d look before creating and purchasing the real thing:

Physical Virtuality example: Vistaprint

“Items can be customised, and only actually made when a customer has paid for the design.”

Physical Virtuality Advantages

Combining time and matter with no-space through physical virtuality brings lots of advantages that are useful in many contexts, especially within e-commerce:

Mirrored Virtuality

Mirrored Virtuality: Time, No-Space, No-Matter

Mirrored Virtuality comes about when we map real world scenarios (that are happening in real-time) to the virtual world of no space and no matter. The best visualizations give us a compelling overview of information, making fast-paced information easy to consume. A great example is the EarthWindMap site, letting us see weather around the world:

Weather in the UK, from EarthWindMap

“The best visualizations give us a compelling overview, making fast-paced information easy to consume.”

A simpler example is the ‘People living on Earth’ counter:

It provides a simple counter of people on earth, which is quite strange to actually see in this way, yet easy to consume.

Essentially though, mirrored virtuality is offering a mirrored perspective of what’s happening in the real world. To get started creating one of these, you just need to find a data source – available from a host of APIs these days, and think of a way to visualize it. Check out this article for more inspiration.

Overall, Get Experimenting

Overall, we’ve covered a range of opportunities available to design across 7 different realms, and it’s clear to see that there’s plenty of scope for creating something totally new for your users to experience.

“There’s really no restrictions – combining different realms is also a possibility for new opportunities.”

There’s really no restrictions – even combining the different realms is a possibility. For example, the VR Gloves by Manus combine Virtual Reality with Augmented Virtuality to create an even more immersive VR experience.

When thinking about this multiverse, it might also be worth to step right back and begin with the activity or experience you want to create, and then think about different tools that can help get you there. Maybe something as simpler as a smaller screen is enough to create a more immersive experience. As we saw with smartwatches, putting screens on our wrists changed the experience of consuming information quite drastically:

As shown in the diagram by Luke Wroblewski, smartwatches provided a new context to design for, allowing us to get updates and track health etc, whilst staying more connected with our surroundings. The experience was no longer focused around the screen, but also our activities and the world around us. The form factor of the watch leads us to only glance at its screen rather than getting totally lost in it like you would a mobile.

“When it comes to experimenting with VR environments, prototyping tools are a great place to start.”

When it comes to experimenting with VR environments though, prototyping tools can be a great way to explore. For instance, it’s even possible to prototype VR experiences directly in Marvel – check out this article by Maxime on prototyping for VR with Marvel, and get started today!

VR Prototyping with Marvel

Design and prototyping for everyone

Design and prototyping for everyone

Thousands of individuals and teams use Marvel to design and prototype ideas.

Get Started, it's Free!

Designer, Writer and Developer at Marvel! Say hello on Twitter.

Related Posts

Nowadays, more and more product teams are adopting Agile processes to manage projects and create more user-centric products. As a result, we’ve seen a bunch of new jargon pop up alongside the various design methodologies that emerged to help teams bring design to Agile development. “From Design Sprints to Agile UX, this is my attempt at breaking down industry jargon… Read More →

Between 2012 and 2017, we’ve seen the biggest tech companies in the world hire thousands of designers to push the value of design through their organisations. For example, IBM has gone from just 1 designer to every 72 engineers, to a ratio of 1 in 8. Intercom lead the way in terms of numbers, with 1 designer to every 5… Read More →

As individuals, and as designers, we are all looking for those special qualities in our jobs. It can sometimes be overwhelming stepping into the job market. We question: Will the role be challenging enough? What can I expect to work on? How do my own values match to that of the business? After a while the answers just become guess… Read More →