Sketch 3: Frame Difference
Image: Still image from playtesting YMOC Sketch 3: Frame Differencing
Frame Differencing is a two-person sketch. Each person is represented with their webcam and their pose keypoints from PoseNet. Frame differencing—a computational technique that only shows pixels that have changed from frame to frame—is used on each webcam. The technique in effect shows movement from frame to frame. The differenced images are layered directly on top of each other.
The posenet keypoints decipher how close or far apart the two people are. The opacity of the people’s images changes based on proximity. When the users are close to one another, the images are visible, as they move apart their images fade. Thus, in order to see your own image you must move and be close to the other person.
Images: Still images from playtesting YMOC Sketch 3: Frame Differencing. The opacity of the participants’ images change based on proximity. When the users are close to one another, the images are visible (top image), as they move apart their images fade (bottom image).
Image: Still image from playtesting with Renee. She is in Argentina. I am in the US.
I playtested this sketch with Renee Carmichael. We spent approximately five minutes on the site together. She’s located in Argentina, and I’m in Vermont, USA. We both experienced little to no visible latency. She was using a MacBook M1/Chrome. I was using an MSI gaming PC/Chrome. Here are some notes from her feedback:
- The experience was easy to understand and she was able to play quickly.
- The experience was not mirrored, which added to the strangeness of the experience (this is a bug).
- She did not hear the sound in the sketch (also a bug).
- Compared to the sketches 1 and 2, which use contour shapes and fills to represent the body, this sketch inspired her to be more in her body and move more “normally” or freely.
- There was more visual presence in this experience.
- The experience made her want to differentiate herself from me (the other person in the experience) rather than mirroring.
- The interaction made her feel like twirling, and pausing.
- It was nice to see the sketch go quiet when she paused, then to start moving again. This interaction impacted the timing of her movement and the experience.
Image: Still image from playtesting with Renee. In reviewing the documentation I noticed interesting moments of perceived intimacy. For example, in this still I appear to be holding Renee by the waist. My original thought about the sketch was that it would encourage intimacy, although I didn’t feel that it did during testing. Because of the quick fading of the pixels on screen, these “intimate” moments are lost to the users when interacting in real time.
This sketch was originally created as a sketch with only audio interaction. I added the images as an afterthought. Interestingly the sound didn’t work during testing and the images were much more successful than I had expected. The original idea behind the sketch was to use sound and imagery to encourage two people to overlap images. I thought that this would bring out a feeling of intimacy. However the quick fading of the images and the need to continue moving encouraged movement and less time for reflection.
The sketch was meant to mirror the image of the user. However, I reversed the canvases and instead the sketch was not mirrored. In effect this meant that when you would try to move one direction on screen, you would go the other. Interestingly for me this meant that I kept bumping into Renee when I would be trying to give her space. This added an interesting element to the interaction, but was not intentional. I’d like to test with the images properly mirrored.
- Fix the mirroring so that people’s images are properly mirrored.
- Remove sound entirely from the sketch.
- Slow down the fade out of the image so that users have more time to reflect on the imagery and experience.