Integrating Extended Reality (XR) With OT Practice Podcast

Editor's note: This text-based course is a transcript of the Integrating Extended Reality (XR) With OT Practice Podcast, presented by Robert Ferguson, MHS, OTR/L, and Dennis Cleary, MS, OTD, OTR/L.

**Please use the handout to complete the exam.

Learning Outcomes

• After this course, participants will be able to:
  • Examine the variability in the taxonomy of extended reality (XR) technologies and platforms and the importance of clarification of terms when using XR for treatment planning and research.
  • Apply the tangible and virtual contexts of activities of daily living to the integration of XR into rehabilitation goals.
  • Examine the barriers and facilitators to the integration of XR in OT practice.

Podcast Discussion

Dennis: Hello, everyone, and welcome. My name is Dennis Cleary, an occupational therapist with 25 years of experience. I am a senior researcher and assistant professor at Cincinnati Children's Hospital Medical Center in beautiful Cincinnati, Ohio. I am happy today to be joined by my good friend, Rob Ferguson, an occupational therapist at the University of Michigan. Rob, could you introduce yourself and tell us a little bit about you and why we are interviewing you today about technology and virtual reality?

Rob: I am a clinical specialist at the University of Michigan Hospital in therapeutic technology and neuro-rehabilitation in inpatient rehab. I run our therapeutic technology program, which encompasses our computer therapy treatment lab, virtual reality program, and pod maker space. I have been a therapist for about 26 years.

In 2008, the computer therapy treatment lab was started at the university. Initially, I was completely against using gaming and technology in therapy because of my dogmatic practice mindset at the time. However, I was convinced over some time that technology was a great way to combine treatment strategies with technology to meet treatment goals.

At the time, this was an early version of virtual reality and considered as such in the literature. I became interested in the immersive part of virtual reality in 2017. We started utilizing it with therapists at Mott hospital. Before this, some technology specialists were using this for distraction with kids during procedures. Once we saw some of the benefits of incorporating it into therapeutic intervention, we were sold and began to research it. I also did some return on investment write-ups for my boss, who was also convinced of the therapeutic benefit once she tried it out.

Dennis: You said you were dogmatic at the beginning? Could you tell a story or two about a patient who benefited from virtual reality?

Rob: Almost all of my clients benefit from virtual reality. Doug Rakoski, now at Loma Linda University, started the lab here at the University of Michigan. When he first started, we talked, and I told him that technology was not an appropriate medium. I could not grasp the concept of using assistive technology therapeutically. And as a neuro-rehabilitation clinical specialist, I was stuck in some of the ways I was mentored and trained. It was not until I started working with Doug in the computer therapy treatment lab that I realized I had no idea what I was doing. It was very eye-opening for me. He will tell the story that it took a conversation at a local restaurant for us to get through our differences. After that, we worked together for over ten years. I learned how to use the technology therapeutically and expanded it into immersive virtual reality.

Dennis: You have now been involved in this for a while. What are some of the significant changes you have seen in technology over the 15 or so years?

Rob: The understanding of virtual reality has changed because technology has changed and will continue to change in the future. Another thing that has opened my eyes and many others is how the context of occupation changes over time. Occupation may not be revolutionary, but it is very evolutionary. Technology, therapeutic uses of technology, virtual reality, and extended realities are part of that evolution of what occupation means to people.

Dennis: Would you say that assistive technology and virtual reality are on a continuum? Or are they different?

Rob: As I always say about everything, that depends. They are very different but use the same technologies and tools. What makes them different is the clinician's intent. If you intend to be adaptive and fill in the gap between a person's abilities and the demands of the activity or task, you will use assistive technology. Technology allows the person to complete the activity.

When you use virtual reality or therapeutic technology as an intervention, the point is not necessarily to fill a gap. The difference is that when you are done using the technology, you have changed the person's capacity to participate and do that activity. It is a remedial approach. The whole point is to change their occupational performance through changes in their capacities to do the activity. It is a very different approach, even if you use the same technologies.

Dennis: In a sense, assistive technology could help you better engage in virtual reality. Correct?

Rob: That is correct. Assistive technology allows you to have access to virtual reality. Accessibility is one of the big barriers to using virtual reality. There are a lot of groups now that are working on improving accessibility to extended reality technologies.

Dennis: We will be talking about that later on, and there will be some resources provided about some of the great occupational therapists doing some great work, such as yourself, with this technology. Could you now talk about some of the different types of technology that you would consider extended reality and virtual reality?

Rob: The taxonomy of extended reality can be confusing, and it changes over time with the technology. Extended reality is an umbrella term for anything that includes a digitally-produced environment that interacts with us in the physical world. There is a broad spectrum of how that occurs, from virtual reality to virtuality and everything in between.

There are terms like virtuality, mixed reality, augmented reality, and other subcategories; yet, everybody calls these extended or virtual reality. The truth is that there are no standard definitions, although the terms are becoming closer. The important thing is to have a reference point when using that terminology with others. 

Dennis: We are currently working on a project at Cincinnati Children's looking at some of those issues, so it has been good for us to clarify what the terms mean. I use technology, but I am not a hands-on, day-to-day expert in terms of some of our technologies. If you look at virtual reality versus augmented reality, how do you differentiate those two?

Rob: I think the way to look at it is first to understand the common definitions and how they have changed over the past few years. I will give you an example. Video games are a type of virtual reality. If you define virtual reality as a computer-generated environment, then anything with digital content would be considered that. Virtuality also encompasses 360-degree video. Even though it represents real life, it is digitally recorded and played back. So that is still considered a virtual component because there is some digitization of the information. However, the content is not created digitally, so people would argue that 360-degree video is not virtual reality, even though you are immersed in it. As you can see, people can confuse different kinds of technologies because people start calling them the same thing.

Dennis: A real estate website may be confused as virtual reality then. You can augment the video by changing the colors and ensuring there is always a sunset in every window.

Rob: Right. Let's frame some of these terms and start with reality. Reality is the environment that we exist in and contains real physical objects. It is interactive, tactile, and created by nature, humans, or other animals. It is a physical world, and we are constrained by the laws of physics in our world and universe. For this discussion, I am going to include 360 videos as part of reality because you are still viewing the real world and the consequences of the physics involved on a display, whether it is on a computer screen, a phone, or a head-mounted display. Reality is anything that reflects the real world. 

Virtual reality would be the synthetic environment made and generated through a computer containing virtual objects. Virtual reality does not have to simulate our physics, but it can have its own physics. You can create something that should not behave the way it does in our reality. It is something digital that you are looking at on a display, or if you do not have a visual display, it can include things like haptic feedback. This feedback can consist of things like different smells and senses as long as they are generated digitally or through a computer interface. All those things would be considered virtuality because they are not part of the real world that we see.

When you get into augmented reality, this includes the spectrum of reality and the other end of virtuality, or a wholly digital environment. In between, the names for some of these have changed over the past few years as technology has changed. I am going to start with mixed reality. Mixed reality used to be visually dominant, where digital information is overlayed over the real environment. For example, this includes the PlayStation EyeToy or Pokemon Go. The digital information is layered over what you see on your device, camera, or through augmented reality glasses. 

Augmented virtuality is the opposite. You have a digital world, and you bring in video or in-realtime activity to that digital display. Think about the PlayStation EyeToy or the Connect. A digital game is displayed on a screen, but it then captures you and puts you into the game. You can see yourself moving and interacting with that digital world. This term does not get used very often because it is just too confusing between augmented reality and augmented virtuality.

Now, what they do is break up augmented reality and mixed reality. Augmented reality is like a two-dimensional layering. Nowadays, mixed reality is still overlaying digital content over the real world, but like the Microsoft HoloLens, the two-dimensional overlay does not size, rescale, and recolor. You can reach out, interact, and move that digital object as if you are there, or you could move around to the other side and view the backside. You cannot do that with augmented reality. In that sense, it could be like viewing a hologram. Mixed reality is interacting with the digital content, not just viewing it or interacting with it on a two-dimensional plane. Technology keeps changing, and I would be willing to bet all this gets changed in the next five to 10 years anyway.

Dennis: As occupational therapists, how would this fit into our practice? Is it mentioned in the occupational therapy practice framework (OTPF)? I know you are an OT who is undoubtedly using many of these virtual realities in your practice. What advice do you have for OTs that are interested in it?

Rob: This is where I get to be critical of my profession and other professions. There is a considerable delay or a gap in defining a virtual context and how it relates to occupation and occupational therapy. If you look at the most up-to-date practice framework in OT, it still defines "virtual" as communication that happens digitally, but they do not get into what it is. They only define it as a context of where occupation occurs in communication. A virtual environment would be a Zoom call, FaceTime, or a telephone call, where you do not have to have that face-to-face component. There are so many areas for virtual reality. For recreation, you have gaming, and education uses in-realtime education. An example is watching a Udemy course in a virtual context. The same thing occurs in other work and leisure activities that can happen in a virtual context. The OTPF limits virtual reality to only digital communication as a definition. This definition is too narrow and does not allow you to evolve the practice and understand what occupation is. Our occupations are increasingly becoming more virtual and less tangible in today's world, or a big mix of the two.

Dennis: Let's say that you have convinced the next generation or the current generation of occupational therapists to start looking at adopting these technologies. How do you learn about the new tech technologies? And, how do you decide which one you think is something that you would like to adopt or do not? Where do you go for your new information?

Rob: It is hard because we view technology as part of occupation in a very narrow sense as a profession. We also tend to focus on just what is available commercially. You learn how to use a computer or a tablet and the operating system. I am trying not to use commercial names of different products because I do not want to endorse them, but there is a big difference between the operating systems, and you have to learn how to do that. It sometimes comes down to preference.

When it comes to therapeutic technologies, you may become aware of them via research. You then have to figure out where to learn about them. More often than not, it is assistive technology within our professions that we know about through continuing education, YouTube, TikTok, OccupationalTherapy.com, et cetera. There are not many opportunities to learn how to specialize in therapeutic technologies or apply them therapeutically. This is a clinical reasoning conundrum.

Once you figure out a technology that you would like to use therapeutically, you do online research or go to the technology manufacturers. This is somewhere where occupational therapy is becoming savvier. If a device is made for treatment and the manufacturer advertises the technology, they will probably give you a demo. Then, you have to analyze it and determine whether it is something you can use in practice. It is an adult learning approach because there is nothing out there right now to help people get basic therapeutic technology training.

Dennis: Can I quote my wife?

Rob: Absolutely.

Dennis: The lovely Dr. Claire Cobain from the University of Notre Dame in South Bend, Indiana, has talked about the process of adopting technology. She uses the Three Es. Is it engaging, efficient, and effective? Most of the time, when we think about technology, it will increase engagement, and sometimes it can be more efficient. Could you talk about the practical part? Because I believe as a therapist, we have a finite amount of time to work with each client or patient that we are seeing. How do we determine whether a technology will be effective in helping one meet this particular therapy goal?

Rob: There are a lot of variables. The most significant barrier to that will be if you are not familiar with the technology, as it will not be engaging for you as a therapist. I know we often talk about the engagement for the client that we are working with, but we also need to think about the therapist. If you are not engaged because you do not know about the technology, it can get frustrating. Then, you cannot get to the other Es. If you are going to be frustrated by the technology, you will not use the technology in practice or see the efficiency it can bring.

There is also a big transition occurring in our clinicians. We have an older group of clinicians, and I include you, Dennis, and myself in this group. A lot of this generation is not tech-savvy. We did not grow up with a great deal of many of the technologies that are available now.

Dennis: Do you remember Pong?

Rob: Pong was such a great game. And now, it is available in three-dimensional virtual reality.

Back to the transition, the younger generation may be tech-savvy, but they may not have the same clinical reason experience. They are learning how to think and practice as a clinician. More experienced clinicians can teach about clinical reasoning, but not necessarily clinical reasoning using the technology. Thus, there is a gap in mentorship and understanding. Then you have newer clinicians who are very tech-savvy and well versed in different technologies but are not mentoring us older people about how to use the tech thoughtfully.

We are at a transition phase, and the mentorship has to happen both ways. You have to have experienced clinicians mentoring in the clinical reasoning component. Still, they have also to be willing to be mentored by less experienced therapists who have a technological skillset. Together, we can do some amazing things. In another five or ten years, it will not be this way. 

Dennis: As a fieldwork coordinator at OSU, I encouraged experienced therapists such as myself to try to take students on, specifically for the mentorship they could provide to the student in the clinical reasoning piece. But as you said, the mentorship that the students can provide to us in terms of technology can be valuable. During COVID, we saw some exceptional leadership in some of our occupational therapy students to help support Zoom or Google Classroom platforms. 

Not to digress, but do you remember the Blackberry?

Rob: I do. I had a Blackberry.

Dennis: See? There you go. I remember when the iPad came out, and we will not mention the tech company, but I used to do a lot of training on using the iPad therapeutically. Around the same time, I was at the Atlanta airport, and there was a Blackberry store. I walked in and asked them about universal design, and they looked at me as if I was green. They did not understand the concept of somebody with a difference needing to access the equipment differently. I was pretty concerned then that Blackberry was not the future, and I was right.

How has universal design provided some opportunities for occupational therapists to help individuals access virtual reality?

Rob: It used to be a consultant role for therapists in tech companies. It used to be that they would have occupational therapists, physical therapists, speech-language pathologists, and rehab engineers involved in the rehab process to consult and give advice. Over the past five years, therapists are becoming part of those tech companies and taking the consultancy role further into the design process. The reality is that engineers and therapists are not that far off in how they think and problem solve, but they just use a different perspective.

The last missing piece is that many tech companies are starting to use universal design to involve people who are not in the middle of the curve but at each end of the bell curve. When you design for someone who has limited access to your product, it makes it easier and inclusive in a broad way. 

Dennis: Yes, because it is so much easier not to have to adapt something if it is designed for a wide variety of abilities.

We are now going to talk about how we integrate virtual reality into our daily practice. Do all the therapists tend to use your lab? Do you trade off patients? How exactly do you go through that?

Rob: The therapeutic technology program is a new emerging kind of programming. It started with the computer therapy treatment lab. Initially, the lab was covered by a technology specialist. The primary therapist and the person running the lab would collaborate on treatments. Sometimes, there would be collaborations in the lab where both would work together, but only one person could document and bill. From an administrative perspective, this was not the best use of time.

Over the past seven years, it has morphed a little bit. We have been trying to get more people comfortable with the technology and create a mentorship between tech uncomfortable people and those who are tech comfortable. Then, also layering in the clinical reasoning piece with the technology.

Dennis: How do you connect virtual reality or other technology platforms to ADL goals, like putting on pants or wiping your butt?

Rob: A lot of it depends on the kind of the level of immersion. Whether it is a computer screen-based virtual interaction or VR immersion, if you want to teach somebody to reach back and pull their pants up or take care of their hygiene, you can adapt the placement of the switches as an example. It depends on the activity analysis of all the games and activities that you research. By research, I mean play. You have to play the game to do your activity analysis. And no matter what my wife says, it's not just playing; it is research.

It takes some time to connect the different virtual contexts to the goals. Knowing whether it is a cognitive, visual, or movement-based strategy is key. How do you break it down to connect to goals? You may have people reaching behind them to access switches or assume different positions, like squatting.

Immersive virtuality provides many opportunities to do things in the hospital or clinic environment that you cannot normally do. I can take somebody out mountain climbing in immersive virtual reality, and they feel like they are actually out on the mountain. Or, they can do archery or a rhythm-based game. Often, they say, "I forgot that I was here. I was immersed and engaged in what I was doing." You can modify the technology to get what you want out of it, whether it is balance, strength, coordination, or range of motion to enable them to build that capacity to meet their goals.

Dennis: Do you feel like, as a therapist, that changes your relationship with the patient? Do they look forward to working with you a little more than a more typical therapy session?

Rob: Yes, absolutely. As a matter of fact, a patient came into the lab today with their primary therapist. She had a very painful shoulder. I put her in a mobile arm support and gave her switch access to a familiar computer game. She went from not moving her shoulder to performing horizontal abduction and adduction to activate the switches in a 2-3 foot range. She was thrilled because it did not hurt, and she drastically improved her range of motion.

The original plan was not for her to do the lab today, but she convinced her therapist to bring her here. Her therapist was able to advance her into reaching overhead in two days with much less pain. They have agreed to try it without the mobile arm support and reintroduce more gravity tomorrow. With the technology, she could complete several hundred repetitions without pain. She was then able to put her shirt on this morning without the pain or an adaptive strategy. She was thrilled by that progress.

Dennis: When a physician, a physical therapist, a speech-language pathologist, nurses, or other healthcare professionals see progress like that, do they respect the work you do a little more than, "They are going and playing with Rob?"

Rob: It has progressed over the years from going down to therapy and playing video games to understanding the connection. Patients tell their nurses and other therapists what they are doing and how "climbing mountains" has helped them do some of their basic ADLs. This also means that patients understand the connection between what they are doing virtually with what they are doing tangibly for their goals. And I think patients understand the connection a little better than a rote clinic activity like stacking cones.

Dennis: You leave those cones alone. I am just kidding.

Rob: If I had ice cream and a bunch of cones, it would be very meaningful for me.

Dennis: How does this work in terms of carryover as we see inpatient rehab stays are getting typically shorter? Are you giving folks homework once they leave? How do you look to do some of that carry over once they have been discharged?

Rob: I give them resources for some of the activities, but the part that becomes a problem is accessibility. I do not mean it from a sense of personal accessibility but rather therapeutic accessibility. And if where they are going for continued therapy does not have the technology, it is a moot point. For example, I will not recommend somebody buy a $1,500 mobile arm support for something they may only use for a few months. If it is something long-term and it will be an assisted device, I can show them how to use it therapeutically.

Additionally, many outpatient centers do not have some of this technology. As it became such a demand from patients going from inpatient to outpatient and was so effective, our outpatient centers started building mobile computer therapy carts. Now, they are beginning to work towards incorporating virtuality because their patients are demanding it. 

Dennis: Great. In terms of goal writing for this, what kind of goals are you writing? Are they specific to the modalities, or are you writing more general occupation-based goals, and this is a modality that you're using?

Rob: The goals are always occupationally-based, and the goals do not change whatsoever.

Dennis: So it is not like getting a 50,000 on a Super Mario Kart.

Rob: No, the score never comes into play except maybe with someone competitive. They may create their own goal, but typically the technology is only the tool. This is the same way we would write about any other therapeutic activity in the clinic. Again, the key is your clinical reasoning. The goal never changes. Virtual reality is only one component and is typically best used as an additional therapy versus a replacement. It is not intended to teach somebody to put a shirt on without teaching them how to put it on. The technology helps build their capacity, but they still have to have a tangible activity. 

Dennis: Great. In terms of a leisure goal specifically, do you often write goals for leisure, or do you try to stay away from those? And if so, does virtual reality play a part in some of those goals?

Rob: I would love to write more leisure-based goals, but it depends on the environment. Someone in a different environment may have recreational and leisure-based goals. In inpatient rehab, we do not typically focus on leisure because we have recreational therapy that works on recreational-based goals. From reimbursement and goal standpoints, our main function is to get people as independent as they can be and get them out of the hospital as soon as possible so they can be in a different environment, whether that is in the home or outpatient clinic. Inpatient rehab is the stepping point.

Even though virtual reality is considered a recreational activity and pursuit, it can also be used for education and communication. We can write assistive technology-based goals on how to use the assistive technology. However, we probably would not write a goal on the independent use of assistive technology. Therapeutically, we use it to improve their occupational performance skills.

Dennis: In terms of your work at the University of Michigan, is it primarily with occupational therapists? Do you support physical therapists and speech-language pathologists as well? 

Rob: It depends on the therapist. We have a lot of therapists, and it comes down to the pragmatics of the work environment. A lab is often in a separate space, and "out of sight, out of mind" is sometimes a thing. They may refer patients, but they may not use it themselves. I have recently provided a mobile cart to the physical therapy gym because there is a very tech-savvy therapist, which has increased their utilization. This has also led to about two or three other physical therapists on the team starting using the technology too. They see how it is being used, and their interest is peaked. When your space is separated physically, it is often hard for them to interact with the technology.

Dennis: As an occupational therapist, would you say virtual reality is a "context"? How would you define this context in terms of your larger job as a therapist in general?

Rob: Virtual reality is a tool that allows you to work in a virtual context. Doing occupational therapy at home versus doing it in inpatient rehab in a clinic are two different environmental contexts, and people behave differently based on those contexts. Virtual reality provides a context to try other things that they might not be able to do in a real-world context. For instance, Fruit Ninja is an oldie but a goody virtual reality game. It originated on your tablets and phones, and you interacted via a screen swipe.

Once immersive virtuality came to the scene, somebody created a three-dimensional fruit version. It defies the laws of physics, and the person can hit pause and freeze them in midair. I can either move work with them in sitting or standing to work on their balance. I can also have them walk around it. They believe that fruit is there, and they will purposefully avoid it because their perception is that it is real. Their perceptual abilities override their cognitive abilities during that particular moment, and it is interesting to watch and see.

If somebody has visual neglect, we may say, "What are the fruits in front of you? Correct, the leftmost fruit is the watermelon. I am going to start this activity again. As the watermelon starts to fall, make sure to swipe it. I want you to miss everything else and just get the watermelon." This is different from how we usually think about interacting with the environment, and it has a powerful effect. I have even had people have a snowball fight with a snowman.

Dennis: If you are working with a gamer who has an injury or an illness, and now they are having difficulty doing their meaningful occupation, do you see more disappointment than maybe our generation that would focus on more traditional ADLs?

Rob: It is very much the same. It depends on their definition of gamer, my definition of gamer, and your definition of a gamer. The reality is that two-thirds of the population of the United States play video games. That is a couple hundred million people who play video games. And the number one genre of video games is for leisure versus the shooting types of games. 

I have seen gamers like the shooting-type genre be very frustrated because they cannot play as they did before. However, I have also seen patients who cannot do their Words with Friends the same way due to language difficulty and become frustrated. I also had an engineer who could not problem solve a simple online activity to create virtual electronic switches. "I don't know why I can't do it." This is a similar sentiment to those who cannot play Call of Duty the same way they did. It is just a different context. We can have them play through some of this frustration that may help them deal with later irritants.

Dennis: Do you have any advice for a therapist that has been reluctant to move into this space? What might be a good first step?

Rob: The key is to learn about it based on their preferred learning style. They can read, watch videos, or try things out. If they have a PC, they can play just about any game on the internet. There are also many apps. They can use all of these virtual contexts to do activity analysis.

The way you approach it is no different from any other activity that you analyze as a therapist. You will see how you can manipulate and change the options of whatever it is that you are playing or doing. This is where the excitement comes in. Once you learn it, then, as a therapist, you can grade their performance and interaction.

I tell therapists the simplest way to do it is to have a treatment model that works for them. The PEO model is an excellent in-realtime thought process that is not complicated. I look at the person, the activity, and the technology that is the interface between the three. "Am I getting what I want? If I need to change something, where in these three areas do I need to focus and change on?" The first thing that most people do wrong is once they set up somebody to play a game, they watch the game because it is interesting and fun. They watch them play on the screen but not engage in the task. Their eyes need to be focused on the patient, the technology, and the interface between the two. Are you using mobile arm support, a standing frame, or an overhead sling for balance? How do you make adjustments to get the behavior and performance and either increase or decrease the challenge? This is the clinical reasoning thought process. It all has to start with learning how to play the game. Once a clinician learns to play the game, they can analyze and apply it to a patient's goals. This is when virtual reality becomes valuable. However, as it said, it was a two-year journey for me to come to this realization.

Dennis: With that final note, Rob Ferguson from the University of Michigan, thank you so much for your time, and thanks to all the listeners. We hope you have a great day.

References

Please refer to the outline and handout.