“While you are in SoundSelf, you dwell in a place which isn’t sleep and isn’t wakefulness. A level of consciousness wraps you which is both extraordinary, and utterly normal. SoundSelf might be the ground floor of an Oculus-biofeedback field to be, or simply an aid to a meditative practice already established.”
When you scuba-dive, a good snorkel mask makes all the difference in the world. Your eyes are your the most finely tuned and vulnerable perceptual instrument, and the job of a mask is both to protect them and to provide an environment in which they can function. If water leaks in, if the masks fog up, if it presses too firmly into your head, you are no longer immersed in a magical underwater world. Instead you are uncomfortably aware of your land-lubber body as it falters in an environment it didn’t evolve for.
VR has been around for a while, but what Oculus has done is the equivalent of creating a better scuba mask: one that doesn’t constantly spray salt-water into your eyes. And in my opinion their most important innovations have been in head-tracking. This means the in-game camera gracefully matches the movement of your head. It’s a complex challenge, and that Oculus has finally gotten it right is the reason so many of us in the tech world are confident that VR is here to stay.
This isomorphic treatment of head movement makes perfect sense for a symbolic virtual reality – one in which the imagined self on the other side of the VR veil is a binocular animal in a universe with three spatial dimensions. But the same assumptions cannot be made for a truly abstract virtual reality like SoundSelf. In SoundSelf, the participant isn’t submerged like a scuba diver into a magical vision of of our own universe – they are instead invited to leave their body behind and forget that they’re human at all. The trouble with an isomorphic head-tracking treatment for SoundSelf is that it makes the participant aware of their body, aware of the visualization as a three dimensional “space” in which they “occupy.” And while it doesn’t totally break the experience, it’s like a bungee connecting the abstract perceptual flow to the memory of being human.
So the question remains: what do we do with head tracking?
By far the most common support inquiries for SoundSelf’s alpha surround our lack of support for head tracking. For better or worse, people expect head tracking in virtual reality experiences, abstract or not, and when head movements don’t provoke a virtual response they think something is broken. So ditching head tracking is not a serious option.
The most attractive technique we explored mapped geometry movement and color to head acceleration. Like rosary beads or whirling dervishes, repeated physical actions can deepen a trance, and I loved the idea of encouraging rhythmic head rocking. These experiments felt great while I tested it for about half an hour: there is a gentle intimacy to feeling the world spin and brighten as your head moves from side to side. But there was one disastrous flaw: upon taking off the VR headset and returning to reality, I felt dizzier than I’ve ever felt in my life. I couldn’t see straight or drive safely for about five hours. Disastrous.
The solution we finally settled on is to keep the visualization stationary in the center of vision, but wrap it in a sphere that *does* respond to head movements isomorphically. SoundSelf begins with the sphere entirely visible, but as the experience gets under way it slowly fades out, coming back into prominence again only when the participant moves their head. It communicates to the participant, “yes, I’m here, nothing is broken” without compromising the abstraction of the virtual space. Players have reported feeling the world respond to their head movement, but not being able to put their finger on what it is that gives that impression.
You’ll have to look closely at the video below to see it. It’s quite subtle outside of the head mounted display.
There’s no getting around the fact that SoundSelf’s implementation of head tracking is in place exclusively to acknowledge user expectations. Head tracking does not directly enhance the experience as it does for most virtual worlds. I know the question on the tip of many of your tongues is “why then call it VR?” The simple answer is that while SoundSelf works without a VR peripheral like the Oculus Rift, it achieves its unique goals with tremendously greater force when it fills the visual field. To me, what makes the Oculus Rift an attractive device for SoundSelf is that it offers a much higher bandwidth passage from image to mind – a distinction that is mostly cosmetic for experiences structured around goals or story, but core to an experience that is structured around sense-perception directly.
I’ve observed two modes of playing SoundSelf. First there’s a playful mode: The player asks the system “what can you do, how do I play with you?” They dance with their voice, push SoundSelf around, explore its limits. They’re having *fun*. Players always begin in this space, but SoundSelf’s magic happens when they transition into a second mode that I’d describe as “surrender”: Their breathing slows down, their voice falls into repeating rhythms, and they stop thinking.
This is the “trance” I’m always talking about. SoundSelf’s interaction is designed to distract your “inner voice” for long enough that you temporarily fall out of the habit of listening to and identifying with it, thus leaving your sense of identity open to being hacked and expanded. However, not everyone makes it over the hump into the surrender phase of the experience, or they’ll surrender for a few minutes before the inner voice returns.
In zen meditation, when the mind inevitably wanders, you actively reign it in again by deliberately drawing your focus back to your breath. Unfortunately, such a mechanism is not directly available to me as a designer without introducing inelegant symbols and instruction into the experience, which would change the nature of SoundSelf from play-partner to teacher, and a teacher is not what I’m interested in making.
So here’s the challenge: How can SoundSelf slowly seduce you into a deeper and deeper trance, but also catch you when you wander back into a playful frame of mind? I think it comes down to respecting the frame of mind the player is currently in – letting SoundSelf respond intuitively to your voice when you’re in the playful mode, but slowly and subtly leading you and moving with you once you’ve surrendered.
Ideally, SoundSelf would be monitoring the player’s brainwaves or heart beat variance and using that data to change the program. For better or worse this technology isn’t commonly available on commercial peripherals. But SoundSelf does have indirect access to a powerful biometric: your breath.
Imitone (the pitch detection algorithm SoundSelf runs on, which is the pride and joy of our programmer Evan Balster) is very sensitive to tonal sounds like your voice, but it’s not designed for atonal sounds like wind and breath. This is a feature, as it effectively ignores background noise. So while SoundSelf can’t know when you’re breathing or how deep and long your breaths are, it can make an educated guess based on the length of your tones and the space between your tones. Combining a two minute rolling average of four elements…:
… and we get a pretty decent heuristic of how entranced you are, and thus how SoundSelf should behave.
It’s not perfect, and it’s quite sensitive to false positives (imagine if in the middle of a period of long low tones from the player, SoundSelf suddenly interprets a distant bicycle bell as a short high pitched tone), but smearing the measurement out over about two minutes gives me a pretty decent high-latency measurement of where your head’s at, and what SoundSelf should do to gently nudge you deeper.
The original motivation behind Deep Sea was a dirt simple question: how do I maximize immersion? It was a curiosity drive! I started out knowing from my own experience that fear can short-cut the rational mind and touch players at a pre-cognitive level. But all the design decisions, like blinding the player, or playing back their breathing to obscure the critical information, all of that was me blindly reaching into the darkness and holding onto what seemed to work. I’m very fortunate to have stumbled onto some ideas that worked incredibly well, but the great irony of Deep Sea’s development is that I didn’t know why they worked. It took about two years of watching people play Deep Sea for me to reverse-engineer my own game and figure out the why.
SoundSelf is built on those understandings. I’ve since come to see immersion as a function of trance. In other words, immersion is in the same family of experience as hypnosis, meditation, and Pentecostal possession. So while SoundSelf is a radically different game from Deep Sea, as a designer they are both knots in the same thread. Only now, instead of accidentally stumbling into a hypnotism design-space, I see that what I’m doing is literally hypnosis. This is tremendously freeing because I don’t have to depend on crutches like the fear response any more, and I can use these hypnosis techniques to induce ecstasy instead.
Games that reject visuals are rare because most games are about handling data… I think that’s what a lot of people think a game is! Handling data and making information-based decisions is as much a part of the paradigm of this medium as words are to literature. And humans are visual creatures, which means that we can process a lot ofdata in images. Games like Deep Sea and SoundSelf are about getting away from data.
But in terms of accessibility, it’s often thought that Deep Sea is a game for the blind, which it emphatically is not. Deep Sea is a game about weakness and dis-empowerment, and you get that by losing your primary sense. The blind, with their hyper-acute sense of hearing, can “see” straight through the brain-trickery that makes Deep Sea frightening. I definitely think there’s a huge untapped market for games for the blind. If I were doing this stuff for money, I’d be making games for the blind.
I think until we get past this paradigm of games being about interpreting and managing data, controllers will still be based around the dexterous hands and fingers. There are a ton of biofeedback technologies that are mature in their development. The only reason these haven’t been integrated into controllers yet is that market leaders think players more of the same – games about navigating data and making decisions. That’s not what players want, that’s just what the edges of this particular skybox look like. What players actually want are experiences that take them on a journey. Systems for navigating information, we can call those systems “games” if you like, are just a familiar tool for getting there.
What’s exciting though is that VR is already shattering that paradigm. If I were in the console business right now, I’d be looking for a way to get ahead of the curve by integrating heartbeat sensors, breath-tracking, and EEGs into peripherals. The next generation of what we call games will not be about using the body as a means of control. It will be defined by experiences that blur the lines between self and software. This leap is right around the corner.
We hope that these words will be burnt into your memory superimposed over the backdrop of a four-hundred-thousand dollar train wreck.
They were addressed to me last Thursday by Matti Leshem, lights hot and cameras rolling, on the set of “GAME_JAM” – a YouTube show in the style of “Iron Chef” created with the intention of showcasing the magical world of indie game development to a public audience. Only instead of making meals, we’d have four days to prototype and demo a game idea. From casual conversations with the crew, I picked up that “GAME_JAM” was one of Polaris’ most ambitious and most expensive project to date – costing about $400,000 to produce. It came crashing to the ground after a single day of shooting.
We are lucky that reporter Jared Rosen was present to document an offensive assault on everything I and my friends in this community hold dear. This article is a response and personal elaboration of his much more comprehensive story. My intention is to show you the dark cloud’s silver lining: how powerful we are when we let our actions be guided by our integrity.
“GAME_JAM” set out to demonstrate the magic of “indie.” It was to be a public platform to showcase, not only our work and our personalities, but the ups and downs of the creative process, the bonds that we build to support ourselves and one another, and the challenge of staying true to ourselves under pressure. In the end it succeeded more elegantly than I could have formerly imagined.
It would have been naive to expect “GAME_JAM” to be a perfect window into our process. There were four teams, and I’d been brought on to lead one of them. Two would be developers of my choosing and one would be a surprise youtuber. From the onset I was prepared to make compromises for showmanship. The game-show structure seemed forced, but a worthwhile compromise to reach a wider audience. The team casting, with me arbitrarily chosen as a leader, was definitely a little weird but made sense because telling four stories is easier than telling sixteen. I saw each of these things as an exciting leadership challenge. I saw each compromise as a worthwhile trade-in to reach a wider audience both for the show itself, and for my in-development game SoundSelf. At the very least, I’d spend the weekend hanging out with and making a game with my friends. The stakes were low.
The same could not be said for my “GAME_JAM” competitor and real-life dear friend, Zoe Quinn. Zoe had stakes that were more real than the competition itself: As the only female team-leader and one of only two women in the whole competition, if she kicked ass she could be a beacon of strength to those who needed it. But if she failed, or if the show were edited to make her look like a failure, her performance would resonate with existing stereotypes at the root of those inequalities.
If you’ve read Jared’s article, you know what happens next. Anything true to what it is that I do was merely incidental to a production that was quickly and deliberately swan-diving into “The Real Housewives of Indie Games.”
The hours that I’d expected we’d be able to spend making something we could stamp our names upon were instead siphoned off to polish a lie. “Back to one, look more surprised and excited for the grand prize of a year supply of Mountain Dew.” Meanwhile, actual game design drama happened with the cameras turned off.
My team had settled upon two wildly different game ideas. There were interesting conversations, compromises, and decisions discussed as we weighed and respected one another’s priorities. But that wasn’t the kind of drama that “Pepsi Consultant” Matti Leshem had in mind.
Matti: followed by a cameraman and a boom operator.
Matti: ordering us around like he owned us.
Matti: banning drinks that weren’t Mountain Dew from the work environment.
Matti: “Do you think you’re at an advantage because you have a pretty lady on your team?”
I declined to answer. He pressed. Adriel curtly told him that his question was offensive.
The day rolled on and I felt a fog of misery and confusion quicken around us. My team was not happy. I was not happy. Surrounded by lights, lawywers, professional entertainers, cameras, humiliation, I felt like a rat in a cage. I didn’t know that the feeling was shared amongst all eleven developers.
On the one hand this is the story of one asshole bringing months of hard work crashing to the ground. It’s an incomplete picture of course: Leshem had a reputation of being an asshole before the show began. But the production’s failings are only part of the story. “GAME_JAM” didn’t crumble because of Leshem, and it didn’t crumble because of the dishonest production.
It crumbled because we, the developers, killed it.
I’ve come to think of ethics as a useful system for making complex scenarios easier to cut through. Ethics are like simple “If, then” statements that you program yourself with. Imagine yourself in a scenario where lying would help you avoid a difficult and unpleasant conversation. It’s one thing to think that lying is generally wrong, but if you’ve previously committed to a code like I do not mislead people, you’ve given yourself a tool to make doing the hard thing easier.
When you’re in the thick of something difficult and confusing, as we were on the set of “GAME_JAM,” it can be hard to know what the right thing to do is. On the one hand, none of us were having a good time. On the other hand, so what? I had an expectation about what being on a game show would be like, and that expectation wasn’t met? Oh how very sad, poor me.
With hindsight, we have the benefit of knowing how powerful we actually were. But at the end of that first shooting day, all any of us had were our private and mostly uncommunicated discontentments.
Adriel had quietly decided to leave the show. She’s outlined that decision in her own lengthy and powerful article on the subject. But I speak here for the bravery I observed in Zoe Quinn because contractual obligations prevent her from speaking for herself. Which is honestly all for the better, because she’s too modest to do herself justice anyway:
If Zoe left the production, she could easily be painted to look like a weak-willed coward for “GAME_JAM”’s potentially massive viewership. It would compromise what she came onto the show for in the first place. But if she stayed, she’d be standing behind and supporting a production operated by a man who tried to isolate and humiliate women for the sake of entertainment.
A particularly useful ethical code is knowing where your loyalties lie. Zoe’s loyalties lay with the young girls she teaches game-making to. She could be beacon for a safe and expressive community if she were publicly shamed as a coward, but she could not do that as an actual supporter of misogyny, lies, and the unsafe creative environment she claims to be fighting.
I think her code went something like this:
If your actions will directly support an unsafe space…
Then jack out. That’s it. No matter what. Abort additional consideration. You’ve found the right thing to do. Leave.
Exhausted from the production’s pageantry, all eleven developers, the show’s creative leads (as disappointed as we were), and the liaison between Polaris and the developers came together to hear one another.
There was a lot on the line for everybody. Most of us had traveled from out of town for a valuable opportunity. Zoe and Adriel were already checked out, but was there a way to walk out of this with something of value? There was debate about what to do, and counter-debate over what degree we could even trust Maker Studio (the YouTube giant that owns Polaris – now a subsidiary of Disney) to follow through on our demands. Possible solutions were aired, and while we had an honest space to communicate, we didn’t get a clear solution until one of the developers asked for a show of hands of who was definitely leaving.
Zoe was out because her ethical code wouldn’t indulge the hippocracy of staying. Adriel was out because she didn’t feel she could safely be herself on the set. Davey Wreden (creator of The Stanley Parable, and a member of Zoe’s “GAME_JAM” team) was out because he was there to support Zoe, and wasn’t interested in participating without her.
Being asked if I was definitely out made me query myself, and there was a split-second of fog. On the one hand, if I left the show it could be edited to look like I was ashamed of my thus far poor performance in the competition. I risked burning bridges with YouTube institutions. And because Adriel was already leaving, if I left we’d be abandoning my one remaining team-mate who’d flown in from Toronto.
If ethics are a personal line of code that helps us stay true to who we want to be, then leadership is articulating that line of code in a way that inspires people to adopt it as their own. Zoe’s impassioned articulation of the problem as she saw it reminded me of my own values and gave me the courage to make them a priority. She turned a vague discomfort of mine into a personal line of code that I could reference to cut through that split-second of fog.
I had been worried about the lack of women among the contestants from the beginning. So my code was easy:
If you will be standing by a representation of my community without gender diversity…
Then do not stand. Leave. Abort additional consideration. You’ve found the right thing to do.
The four hands alone would have hindered the production. But together, they were unified by a shared cause: an unsafe space for smart female developers. From this point there would be no stepping back to redefine what the problem was about, and there would be no compromise. Matti had secured the sponsorship with Pepsi, so even if he was fired, choosing to stay would help save his reputation. The movement to quit production had a flag and it had a leader. It took only a few minutes for a dissenting minority to become a production without a cast.
When we trust our community, we trust that others hold that community in their loyalties. The integrity of a community is a function of how true its members stay to that code. What made Zoe’s ethical code contagious was that it resonated with the community-serving values of those who were present.
This last year has seen me lose faith in myself. I’ve lost track of who I am and why I deserve my own love. I’ve seen my bad habits alienate the people I’m close to. Writing this article has been for me a meditation on my struggle with self love and self respect. I wonder if knowing oneself is an act of building oneself by means of codes like these ones. By extension, I wonder if trusting oneself is an act of being true to the code you wrote.
Developers don’t just create games – we create an ecosystem in which it is safe to be expressive. As this ecosystem grows, it will inevitably fracture. The work we do now to set a standard of holding ourselves individually accountable to the health of the whole defines the seed from which future healthy communities will grow. In the last 48 hours I saw the true colors of my friends who are a part of that seed. They’re good colors. They speak for a bright future.
Until the facts bubble up to the surface, all we have are incomplete personal stories like my own. Each one shows a different angle and shines a light from a different perspective:
Jared Rosen was the only person present with the intent of reporting. With a long history of relationships both with Polaris and with the developers present, Jared’s article is the most comprehensive telling of this story currently available.
Adriel Wallick, the programmer on my team who this story is in many ways about, knows what a game jam looks like at its best because she’s the organizer of one. She’s written about holding those responsible for this disaster accountable.
Zoe Quinn is restricted by contractual obligations in what she can say, but this event is just one chapter for her in a long story of activism. She’s written about the personal cost of balancing her loyalty to the community with a loyalty to her own work.
Between our four stories, there are people whose role in coming out of this safely could not be given the nods that they deserve. I tried to fit them into my story above, but could not do so without compromising its flow and readability:
Aaron Umetani was the director of “GAME_JAM” – it was his baby, supported by producer Jason Serrato. As the project grew in scale, they made compromises that they shouldn’t have, but when those compromises were manifest on set as the unsafe and unfriendly environment the four of us have described, they were as horrified as anyone else. One of the difficult parts about being on set was not knowing who we could trust, or to what extent. Over the course of the weeks leading up to the shoot, the single shooting day, and the days since, these two were the only beacons at Maker/Polaris who I believed I could trust. They have assumed personal responsibility for what went wrong and repeatedly demonstrated their integrity. I would happily work with them again, and sincerely hope that one day Aaron’s baby will be realized as it was imagined.
Akira Thompson is a living demonstration of the supportive and inclusive power of this community. His M.O. is to support our abilities to support one another. He was hired by Polaris to act as a liaison with developers, and his first job was to recruit four team leaders. When the production was clearly no longer a service to us, he assembled the the developers with Jason Serrato and Aaron Umetani present so that we could all hear one another, in a conversation that culminated in killing the project. Without Akira’s actions, we would have remained in the dark about the extent of the production’s toxicity for much longer. Without his reminder of our power as a unified group, and without his private encouragement to be true to our values, this could have ended very differently.