Concept

We envisioned a wall projected seascape that would become activated by the presence of a person. We wanted to support both passive and active engagement with the installation. Each person that moves past the installation is pursued by a school of fish. If the person is in a hurry they may not even notice their contribution to the seascape, as their fish "shadow" follows behind them. A more curious individual can engage actively with their school of fish. Standing in front of the aquarium environment allows your school of fish to catch up. They circle curiously around you but are startled by sudden changes in direction. Touching the seascape generates a small electric shock. Any nearby fish will be electrocuted. The escaping fish will flee from the electricity. Of course, fish have no memory. They'll soon return to your side so you can shock them again.

Inspiration

Implementation

The wall fish installation uses processing and the Kinect to accomplish blob tracking and touch detection using OpenCV. A big thanks to Asim Mittal who helped enormously with a nice piece of code for detecting touch with the Kinect. Eli worked on creating the art, the fish behaviors, and the interactions. The behaviors and interactions were developed on the computer in response to mouse movements. Eli adapted a flocking algorithm by Daniel Shiffman. Meanwhile I developed the framework for tracking multiple people's movement across the installation and for detecting a touch. I used Shiffman's openKinect library. I then adapted Eli’s code, replacing mouse movement with blob tracking, and replacing the mouse click with a touch on the projected display.

Challenges

One of the challenges was detecting touch on a wall surface with a skewed perspective and needed calibration with the Kinect and the Projection. We decided not to use OpenNI for detection of people so it was difficult to track multiple people in cases of collision and overlap, so we ended up using probabilistic detection of blobs nearest to the previous blob position.

Trying to accomplish an interesting set of behaviors for the school of fish was also a challenge. Working only with human position and speed, we wanted to create a personality for the fish that was both curious and tentative. In order to create this effect we experimented with a number of behaviors and interactions.