Train to Dubai by Alex Wuqi Zhang and Ross Jiang

Our final project was inspired by a recent Korean zombie movie called “Train to Busan”. We localized the title and changed the storyline.

One day Ross receives a text message from Alex: Alex is stuck in a train and surrounded by zombies. The player, as Ross, has to pick up a gun and save his friend from dying from zombies.

Alex was mainly in charge of the game mechanics  and Ross was mainly in charge of the art design and hardware.

The making of the game wasn’t easy. We had to keep multiple versions/drafts in the process of making this game because we want to work on the game at the same time on two different machines. After Alex finishes part of the code, he has to copy and paste onto the code on Ross’ computer to make sure it works. Sometimes it can be confusing as a single line of code missing would result in the whole program not working.

Thanks to Aaron’s help with the ps3eye library and the setup of the camera, the computer vision part was not very difficult. We just had to adopt the existing program to our game.

The hardware and communications between arduino and processing proved to be quite frustrating. Sometimes we had to unplug and plug the USB to make sure processing reads input properly. It was probably simply because of the faulty port on Ross’ computer. The button was broken quite a few times as players stepped on the wire or push too hard on the button. It would be ideal if we had longer wires or made it wireless.

The gameplay could be improved as well. If we had more time, we would have designed more levels with different kinds of zombies as well as a more proper ending to the story.

Here are the codes:

class Zombie{
  
  boolean direction;
  int counterImg;
  float locx, locy;
  PImage img1; 

  Zombie(float x, float y, boolean dir){
    locx = x;
    locy = y;
    direction = dir; 
  }
  
  void displayImages(){
    this.update();
    if(frameCount % 10 == 0){
      counterImg++;
    }
    pushMatrix();
    //if(!direction){
    //  translate(img1.width,0);
    //}
    if(gameover == false){
      img1 = zombieImages1[counterImg % 4]; 
      if(!direction){
        scale(-1, 1);
        image(img1,-locx, locy);
      }else{
        //scale(1, -1);
        image(img1, locx, locy);
      }
    }
    popMatrix();
  }
  
  void update(){
    if (frameCount % 10 == 0){
      if(direction){
        this.locx += 10; 
      }else{
        this.locx -= 10;
        
      }
    }
    if (frameCount % 120 == 0){
      this.direction = !direction;
    }
  }
}
class ZombieFast{
  
  boolean direction;
  int counterImg;
  float locx, locy;
  PImage img1; 

  ZombieFast(float x, float y, boolean dir){
    locx = x;
    locy = y;
    direction = dir ; 
  }
  
  void displayImages(){
    this.update();
    if(frameCount % 10 == 0){
      counterImg++;
    }
    pushMatrix();
    //if(!direction){
    //  translate(img1.width,0);
    //}
    if(gameover == false){
      img1 = zombieFastImages[counterImg % 4]; 
      if(!direction){
        scale(-1, 1);
        image(img1,-locx, locy);
      }else{
        //scale(1, -1);
        image(img1, locx, locy);
      }
    }
    popMatrix();
  }
  
  void update(){
    if (frameCount % 10 == 0){
      if(direction){
      this.locx += 40; 
      }else{
        this.locx -= 40;
        
      }
    }
     if(frameCounter % 25 == 0){
       tempVar++;
       println(tempVar);
       if(tempVar % 4 == 0){
         this.direction = !direction; 
       }
     }
  }
}
class ZombiePeek{
  
  boolean direction;
  int counterImg;
  float w, h, locx, locy;
  PImage img1; 

  ZombiePeek(float x, float y, float a, float b, boolean dir){
    locx = x;
    locy = y;
    w = a;
    h = b;
    
    direction = dir; 
  }
  
  void displayImages(){
    this.update(); 
    //img1 = loadImage("zombie4f1.png");
    //image(img1, locx, locy, width/w, height/h);
    if(frameCount % 20 == 0){
      counterImg++;
    }
    pushMatrix();
    if(gameover == false){
      img1 = zombiePeekImages[counterImg % 4]; 
      image(img1, locx, locy, 128/w, 128/h);
    }
    popMatrix();
  }
  
  void update(){
     if(frameCounter % 25 == 0){
       tempVar++;
       println(tempVar);
       if(tempVar % 4 == 0){
         this.direction = !direction; 
       }
     }
     if(direction){
       this.w += 0.05;
       this.h += 0.05; 
     }else{
       this.w -= 0.05; 
       this.h -= 0.05;
     }
  }
}

 

import processing.video.*;
import processing.sound.*;
PFont title;
PFont typewriter;
PFont instructionFont;

Serial myPort; 

int currentButtonState = 0; 
int prevButtonState = 0; 

int menu;
int substr_cnt;
int substr_cnt2;
int substr_cnt3;
boolean msg2Display = false;
boolean msg3Display = false;
boolean moviePlay = false;
boolean SoundPlayed = false;
Movie myMovie;
SoundFile typingSound;
SoundFile vibratingSound;
SoundFile gunshot;
SoundFile slimeSound;
PImage dubai;
PImage instruction;

import java.util.ArrayList;
import processing.serial.*;

int alreadyAdded = 0;
int numOfZombies = 4; 
int numOfJumpingZombies = 6; 
int numOfFastZombies = 4;
int numOfPeekZombies = 1;
//Zombie zombie = new Zombie(100, 100, 1, true); 
Bullet bullet = new Bullet(mouseX,mouseY);
String[] explosionImageNames = {"explosionf1.png", "explosionf2.png", "explosionf3.png", "explosionf4.png", "explosionf5.png", "explosionf6.png", "explosionf7.png", "explosionf8.png"};
PImage[] explosionImages = new PImage[explosionImageNames.length];
String[] zombieImageNames = {"zombie1f1.png", "zombie1f2.png", "zombie1f3.png", "zombie1f4.png"};
PImage[] zombieImages1 = new PImage[zombieImageNames.length];
String[] zombieJumpingImageNames = {"zombie2f1.png", "zombie2f2.png", "zombie2f3.png", "zombie2f4.png"};
PImage[] zombieJumpingImages = new PImage[zombieJumpingImageNames.length];
String[] zombieFastImageNames = {"zombie3f1.png", "zombie3f2.png", "zombie3f3.png", "zombie3f4.png"};
PImage[] zombieFastImages = new PImage[zombieFastImageNames.length];
String[] zombiePeekImageNames = {"zombie4f1.png", "zombie4f2.png", "zombie4f3.png", "zombie4f4.png"};
PImage[] zombiePeekImages = new PImage[zombiePeekImageNames.length];
//Zombie[] zombies = new Zombie[numOfZombies];
ArrayList<Zombie> zombies = new ArrayList<Zombie>(4*numOfZombies);
ArrayList<ZombieJump> zombieJump = new ArrayList<ZombieJump>(numOfJumpingZombies);
ArrayList<ZombieFast> zombieFast = new ArrayList<ZombieFast>(numOfFastZombies);
ArrayList<ZombiePeek> zombiePeek = new ArrayList<ZombiePeek>(numOfPeekZombies);
PImage background; 
PImage cursor; 
boolean skipInstruction = false;

import gab.opencv.*;
import com.thomasdiewald.ps3eye.PS3EyeP5;
import java.awt.Rectangle;
PS3EyeP5 ps3eye;
OpenCV opencv;
int blobSizeThreshold = 1;

void setup(){
  fullScreen(); 
  
  title = createFont("BankGothicBold.ttf",32);
  instructionFont = createFont("BrookeS8.ttf",24);
  textFont(title);
  typewriter = createFont("type.ttf",22);
  menu = 1;
  background(0, 0, 30);
  myMovie = new Movie(this, "intro.mov");
  typingSound = new SoundFile(this, "typewriter.mp3");
  vibratingSound = new SoundFile(this, "vibratingSound.mp3");
  slimeSound = new SoundFile(this, "slimeSound.mp3");
  gunshot = new SoundFile(this,"gunshot.wav");
  dubai = loadImage("dubai.jpg");
  instruction = loadImage("instruction.png");
  
  printArray(Serial.list()); 
  String portname = Serial.list()[4];
  println(portname); 
  myPort = new Serial(this,portname,9600);
  ps3eye = PS3EyeP5.getDevice(this);
  ps3eye.start();
  opencv = new OpenCV(this, 640, 480);
  
  
  background = loadImage("trainbackground.png");
  background(background); 
  
  cursor = loadImage("cursor.png"); 
  image(cursor, width/2, height/2);
  
  //preload bullet images
  for(int i = 0; i < explosionImageNames.length; i++){
    String imageName = explosionImageNames[i];
    explosionImages[i] = loadImage(imageName); 
  }
  
  //preload zombie images
  for(int i = 0; i < zombieImageNames.length; i++){
    String imageName = zombieImageNames[i];
    zombieImages1[i] = loadImage(imageName); 
  }
  for(int i = 0; i < zombieJumpingImageNames.length; i++){
    String imageName = zombieJumpingImageNames[i];
    zombieJumpingImages[i] = loadImage(imageName); 
  }
  for(int i = 0; i < zombieFastImageNames.length; i++){
    String imageName = zombieFastImageNames[i];
    zombieFastImages[i] = loadImage(imageName); 
  }
  for(int i = 0; i < zombiePeekImageNames.length; i++){
    String imageName = zombiePeekImageNames[i];
    zombiePeekImages[i] = loadImage(imageName); 
  }
}


void draw(){
  
  println(currentButtonState);
  
    if (menu == 1 && currentButtonState == 49 ) {
    menu = 2;
    substr_cnt = 0;
    gunshot.play();
  }
  
  
   if (menu == 1) {
    //ellipse (random (width), random (height), random (4), random (4));
    image(dubai,0,0, width, height);
    textAlign(CENTER); 
    textSize(78);
    text ("Train to Dubai", width/2,height/2);

    textSize(32);
    text("Shoot to Continue", 350, 325);
    textSize(20);
    text("Powered by Ross Jiang and Alex Wuqi Zhang",1100,700);
  }
  if (menu == 2) {
    textFont(typewriter);
    background (0, 0, 30);
    fill(255, 255, 255);
    String msg1 = "Today, you find the streets of Dubai unusually quiet ..";
    String msg2 = "Suddenly, your phone starts vibrating ..";
    String msg3 = "It is from your stupid friend Alex ..";
    
    text (msg1.substring(0,constrain(int(substr_cnt/5),0,msg1.length())), 380, 160);
    if(!SoundPlayed){
      typingSound.play();
      SoundPlayed=true;
    }
    substr_cnt++;
      
    if( substr_cnt == 6 * msg1.length() ){
      msg2Display = true;
      vibratingSound.play();
    }
    
    if(msg2Display){
      text (msg2.substring(0,constrain(int(substr_cnt2/5),0,msg2.length())), 295, 230);
      substr_cnt2++;
    }
    
    if( substr_cnt2 == 6 * msg2.length() ){
      msg3Display = true;
      
    }
    
    if(msg3Display){
       text (msg3.substring(0,constrain(int(substr_cnt3/5),0,msg3.length())), 275, 300);
       substr_cnt3++;
    }
    
    if( substr_cnt3 == 6 * msg3.length() ){
      moviePlay = true;
    }
    
    if(moviePlay){
     
      myMovie.play();
      image(myMovie, 0, 0,width,height);
      
    }
    
    if(myMovie.time() == myMovie.duration()){
      menu = 3;      
    }
    
  }
  
  if(menu == 3){
    textFont(instructionFont);
    image(instruction, 0,0, width,height);
    
  }
  
  if(menu ==3 && currentButtonState ==49){
    gunshot.play();
   menu =4; 
    
  }
  
     if ( menu == 4 ) {
      textFont(typewriter);
      background(background);
      textSize(32);
      opencv.loadImage(ps3eye.getFrame());
      opencv.threshold(100);
      noFill();
      stroke(255, 0, 0);
      strokeWeight(3);
  for (Contour contour : opencv.findContours()) {
    //contour.draw();
     Rectangle r = contour.getBoundingBox();
     
    stroke(255, 0, 0);
    fill(255, 0, 0, 150);
    strokeWeight(2);
    println(r.getCenterX()); 
    clickedX = (int)map((float)r.getCenterX(),640,0,width,0);
    clickedY = (int)map((float)r.getCenterY(),480,0, 0, height);
    ellipse(clickedX,clickedY,5,5);

  }
  
  if(currentButtonState == 49){
    
    gunshot.play();
  
   for(int i = 0; i < zombies.size(); i++){
     if(collisionCheck(zombies.get(i))){
       zombies.remove(i);
       kills++; 
       slimeSound.play();
     }
   }
   
   for(int i = 0; i < zombieJump.size(); i++){
     if(collisionCheckJump(zombieJump.get(i))){
       zombieJump.remove(i);
       kills++; 
       slimeSound.play();
     }
   }
   
   for(int i = 0; i < zombieFast.size(); i++){
     if(collisionCheckFast(zombieFast.get(i))){
       zombieFast.remove(i);
       kills++;
      slimeSound.play();
     }
   }
   
   for(int i = 0; i < zombiePeek.size(); i++){
     if(collisionCheckPeek(zombiePeek.get(i))){
       zombiePeek.remove(i);
       kills++; 
       slimeSound.play();
     }
   }
    
   click = true; 
   bullet = new Bullet(clickedX, clickedY);  
  }
  
  //zombie wave 1
  if(wave == 1 && alreadyAdded == 0){
    for(int i = 0; i < numOfZombies; i++){
      zombies.add(new Zombie((210*(i+1)), (650), true)); 
    }
    
    zombieJump.add(new ZombieJump(300, 500)); 
    zombieJump.add(new ZombieJump(500, 500));
    zombieJump.add(new ZombieJump(700, 500));
    
    zombieJump.add(new ZombieJump(300, 300)); 
    zombieJump.add(new ZombieJump(500, 300));
    zombieJump.add(new ZombieJump(700, 300));
    
    
    zombiePeek.add(new ZombiePeek(1050, 200, 0.75, 0.75, true));
    alreadyAdded++;
  }
  
  for(int i = 0; i < zombies.size(); i++){
  zombies.get(i).displayImages(); 
  }
  for(int i = 0; i < zombieJump.size(); i++){
  zombieJump.get(i).displayImages(); 
  }
  for(int i = 0; i < zombieFast.size(); i++){
  zombieFast.get(i).displayImages(); 
  }
  for(int i = 0; i < zombiePeek.size(); i++){
  zombiePeek.get(i).displayImages(); 
  }
  imageMode(CENTER);
  image(cursor, clickedX, clickedY, 320, 320);
  //Bullet bullet = new Bullet(-100,-100);
    if(frameCount % 4 == 0){
    frameCounter++;
    }
  if(click == true){
     bullet.displayBullet();
  }
  deathCounter(zombies.size() + zombieJump.size() + zombieFast.size() + zombiePeek.size()); 
  waveCounter(zombies.size() + zombieJump.size() + zombieFast.size() + zombiePeek.size()); 
  killsCounter(); 
  
  

  //zombie wave 2
  if(wave == 2 && alreadyAdded == 1){
    zombieFast.add(new ZombieFast(100, (100), true)); 
    zombieFast.add(new ZombieFast(100, (300), true)); 
    zombieFast.add(new ZombieFast(100, (500), true)); 
    zombieFast.add(new ZombieFast(100, (700), true)); 
    zombieFast.add(new ZombieFast(1220, (100), false)); 
    zombieFast.add(new ZombieFast(1220, (300), false)); 
    zombieFast.add(new ZombieFast(1220, (500), false)); 
    zombieFast.add(new ZombieFast(1220, (700), false)); 
    alreadyAdded++; 
  }
  
  //zombie wave 3
  if(wave == 3 && alreadyAdded == 2){
    for(int i = 0; i < numOfZombies; i++){
      zombies.add(new Zombie((210*(i+1)), (650), true)); 
    }
    for(int i = 0; i < numOfZombies; i++){
      zombies.add(new Zombie((210*(i+1)), (500), true)); 
    }
    for(int i = 0; i < numOfZombies; i++){
      zombies.add(new Zombie((210*(i+1)), (350), true)); 
    }
    for(int i = 0; i < numOfZombies; i++){
      zombies.add(new Zombie((210*(i+1)), (200), true)); 
    }
    alreadyAdded++; 
  }
  
  gameWin(); 
  gameOver();
     }
}
  
  


void serialEvent(Serial myPort){
  currentButtonState = myPort.read(); 
}


void movieEvent(Movie m) {
     m.read();
}

// reference:
//https://forum.processing.org/two/discussion/26427/how-to-incorporate-the-typewriter-effect-into-my-game

int clickedX, clickedY, kills, tempVar;
int frameCounter = 1;
boolean click = false; 
boolean gameover = false; 
int gameCounter = 6; 
int wave = 1;
boolean exitGame = false; 



class Bullet{
  
  int bulletX, bulletY;
  int bulletCounter = 0; 
  PImage img; 
  int time;
  
  
  Bullet(int x, int y){
    bulletX = x; 
    bulletY = y; 
  }
  
  void displayBullet(){
    if(click == true){
      int frame = (frameCounter % 8) / 1 + 1;
      img = explosionImages[frame-1]; 
      image(img, bulletX, bulletY);
      if(frame == 8){
        click = false;
      }
    }
  }
}

  void deathCounter(int remainingZombies){
    if(gameover == false && frameCount % 60 == 0){
      gameCounter--;
    }
    fill(255); 
    textSize(60); 
    text(gameCounter, width/2-20, 100);
    if(gameCounter == 0 && remainingZombies > 0){
      gameover = true; 
      //textSize(40); 

    }
  }
  
  void killsCounter(){
    fill(255); 
    textSize(60); 
    text("kills: " + kills, width - 300, 100); 
  }
  
  void waveCounter(int remainingZombies){
    if(gameCounter == 0 && remainingZombies == 0){
      wave++; 
      gameCounter = 6; 
    }
  }
  
  void gameWin(){
    if(wave == 4){
      fill(255); 
      textSize(80); 
      text("YOU WIN!!", width/2-150, height/2-100); 
      textSize(40);
      text("Shoot to Play Again", width/2-200, height/2 +200); 
      if(clickedX > 0 && clickedX < width && clickedY > 0 && clickedY < height && click == true){
        println("CLICKEDD");
        //gameover = false;
        wave = 1;
        alreadyAdded = 0;
        kills = 0; 
        frameCounter = 1;
        tempVar = 0;
        gameCounter = 6;
      }
    }
  }
  
  void gameOver(){
    if(gameover == true){
        if(zombies.size() != 0){
          for(int i = 0; i < zombies.size(); i++){
            zombies.remove(i); 
          }
        }
        if(zombieJump.size() != 0){
          for(int i = 0; i < zombieJump.size(); i++){
            zombieJump.remove(i); 
          }
        }
        if(zombieFast.size() != 0){
          for(int i = 0; i < zombieFast.size(); i++){
            zombieFast.remove(i); 
          }
        }
        if(zombiePeek.size() != 0){
          for(int i = 0; i < zombiePeek.size(); i++){
            zombiePeek.remove(i); 
          }
        }
      fill(255); 
      textSize(80); 
      text("game over", width/2-200, height/2-100);
      textSize(40); 
      text("Shoot to Play Again", width/2-200, height/2 + 200); 
      if(clickedX > 0 && clickedX < width && clickedY > 0 && clickedY < height && click == true){
        println("CLICKEDD");
        gameover = false;
        wave = 1;
        alreadyAdded = 0;
        kills = 0; 
        frameCounter = 1;
        tempVar = 0;
        if(zombies.size() != 0){
          for(int i = 0; i < zombies.size(); i++){
            zombies.remove(i); 
          }
        }
        if(zombieJump.size() != 0){
          for(int i = 0; i < zombieJump.size(); i++){
            zombieJump.remove(i); 
          }
        }
        if(zombieFast.size() != 0){
          for(int i = 0; i < zombieFast.size(); i++){
            zombieFast.remove(i); 
          }
        }
        if(zombiePeek.size() != 0){
          for(int i = 0; i < zombiePeek.size(); i++){
            zombiePeek.remove(i); 
          }
        }
        gameCounter = 6;
      }
    }
  }
  
  

  boolean collisionCheck(Zombie z){
    println("used");
    if((z.locx - 56) < clickedX && (z.locx + 56) > clickedX && (z.locy - 56) < clickedY && (z.locy + 56) > clickedY){
       //z = null; 
       println("hit");
       return true;
    }
    return false;
  }
  
  boolean collisionCheckJump(ZombieJump z){
    println("used");
    if((z.locx - 56) < clickedX && (z.locx + 56) > clickedX && (z.locy - 56) < clickedY && (z.locy + 56) > clickedY){
       //z = null; 
       println("hit");
       return true;
    }
    return false;
  }
  
  boolean collisionCheckFast(ZombieFast z){
    println("used");
    if((z.locx - 56) < clickedX && (z.locx + 56) > clickedX && (z.locy - 56) < clickedY && (z.locy + 56) > clickedY){
       //z = null; 
       println("hit");
       return true;
    }
    return false;
  }
  
    boolean collisionCheckPeek(ZombiePeek z){
    println("used");
    if((z.locx - 56) < clickedX && (z.locx + 56) > clickedX && (z.locy - 56) < clickedY && (z.locy + 56) > clickedY){
       //z = null; 
       println("hit");
       return true;
    }
    return false;
  }

 

Final Project

Initial Idea

For my final project, I started off with a very vague idea that I wanted to combine colours and sounds and somehow recreate synesthesia. I initially thought of making it as small, physically, as having a melodica and a screen but after talking with Aaron, I increased the size of the project by using a piano instead.

 

Stress Inducing Wires

The process of attaching the wires inside the piano was a series of struggles.

After many nights of being stressed, having mental breakdowns and making stupid mistakes in the Art Centre, I was finally able to get every wire in place. I never thought sticking some wires could cause someone so much trouble. At least now I can confidently say that I can solder.

 

Code

Many thanks to Aaron and Daniel Shiffman for “helping me write” the code.

Arduino:

int noteC = 0;
int noteD = 0;
int noteE = 0;
int noteF = 0;
int noteG = 0;
int noteA = 0;
int noteB = 0;
int noteCC = 0;
int inByte = 0;

void setup() {
  Serial.begin(9600);
  pinMode(2, INPUT);
  pinMode(3, INPUT);
  pinMode(4, INPUT);
  pinMode(5, INPUT);
  pinMode(6, INPUT);
  pinMode(7, INPUT);
  pinMode(8, INPUT);
  pinMode(9, INPUT);
  establishContact();
}

void loop() {
  if (Serial.available() > 0) {
    inByte = Serial.read();
    noteC = digitalRead(2);
    noteD = digitalRead(3);
    noteE = digitalRead(4);
    noteF = digitalRead(5);
    noteG = digitalRead(6);
    noteA = digitalRead(7);
    noteB = digitalRead(8);
    noteCC = digitalRead(9);

    Serial.print(noteC);
    Serial.print(",");
    Serial.print(noteD);
    Serial.print(",");
    Serial.print(noteE);
    Serial.print(",");
    Serial.print(noteF);
    Serial.print(",");
    Serial.print(noteG);
    Serial.print(",");
    Serial.print(noteA);
    Serial.print(",");
    Serial.print(noteB);
    Serial.print(",");
    Serial.println(noteCC);
  }
}

void establishContact() {
  while (Serial.available() <= 0) {
    Serial.println("0,0,0,0,0,0,0,0");
  }
}

Processing:

import processing.serial.*; 
import codeanticode.syphon.*;

SyphonServer server;
Serial myPort;  

import toxi.geom.*;
import java.util.*;

PGL pgl;

Emitter emitter;
Vec3D gravity;
float floorLevel;

PImage particleImg;
PImage emitterImg;

int counter;

boolean noteState[]={false, false, false, false, false, false, false, false};
int noteCounter[] = {0, 0, 0, 0, 0, 0, 0, 0};
int notes[]={0, 0, 0, 0, 0, 0, 0, 0};
int countTarget = 10;

boolean ALLOWGRAVITY;    // add gravity vector?
boolean ALLOWPERLIN=true;     // add perlin noise flow field vector?
boolean ALLOWTRAILS;     // render particle trails?
boolean ALLOWFLOOR;      // add a floor?

color c;

void setup() {
  size( 600, 600, P3D );
  server = new SyphonServer(this, "Processing Syphon");
  smooth(4);
  colorMode( RGB, 1.0 );

  pgl         = ((PGraphicsOpenGL) g).pgl;  

  // Loads in a particle image from the data folder. Image size should be a power of 2.
  particleImg = loadImage( "emitter.png" );
  emitterImg  = loadImage( "emitter.png" );

  emitter     = new Emitter();
  gravity     = new Vec3D( 0, .35, 0 );    // gravity vector
  floorLevel  = 400;

  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[1], 9600);
  myPort.clear();
  myPort.bufferUntil('\n');
}

void draw() {
  background( 0 );
  perspective( PI/3.0, (float)width/(float)height, 1, 5000 );

  pgl.depthMask(false);

  pgl.enable( PGL.BLEND );
  pgl.blendFunc(PGL.SRC_ALPHA, PGL.ONE);

  emitter.exist();
  }
  
  float total=0;
  for (int noteNum = 0; noteNum < notes.length; noteNum++) {
    total+= notes[noteNum];
  }
  total/=notes.length;
  for (int noteNum = 0; noteNum < notes.length; noteNum++) {

    if (notes[noteNum]==1 ) {
      noteCounter[noteNum]++;
    } else if (notes[noteNum]==0) {
      noteCounter[noteNum]=0;
    }
    if (noteCounter[noteNum]>countTarget)
      noteState[noteNum]=true;
    else if (noteCounter[noteNum]==0)
      noteState[noteNum]=false;
  }
  
  int numParticles = 2;
  
  if (noteState[0] == true) {
    emitter.pos.x= 290;
    emitter.pos.y= 600;
    c = #B22222;
    emitter.addParticles( numParticles );
  }

  if (noteState[1] == true) {
    emitter.pos.x= 302;
    emitter.pos.y= 600;
    c = #FF3910;
    emitter.addParticles( numParticles );
  }

  if (noteState[2] == true) {
    emitter.pos.x= 314;
    emitter.pos.y= 600;
    c = #FF6347;
    emitter.addParticles( numParticles );
  }

  if (noteState[3] == true) {
    emitter.pos.x= 322;
    emitter.pos.y= 600;
    c = #90EE90;
    emitter.addParticles( numParticles );
  }
  
  if (noteState[4] == true) {
    emitter.pos.x= 332; 
    emitter.pos.y= 600;
    c = #87CEFA;
    emitter.addParticles( numParticles );
  }
  
  if (noteState[5] == true) {
    emitter.pos.x= 342;
    emitter.pos.y= 600;
    c = #2A51FF;
    emitter.addParticles( numParticles );
  }
  
  if (noteState[6] == true) {
    emitter.pos.x= 353;
    emitter.pos.y= 600;
    c = #8A2BE2;
    emitter.addParticles( numParticles );
  }
  
  if (noteState[7] == true) {
    emitter.pos.x= 360;
    emitter.pos.y= 600;
    c = #FF4B60;
    emitter.addParticles( numParticles );
  }

  counter ++;
  server.sendScreen();
}

void serialEvent(Serial myPort) {
  String myString = myPort.readStringUntil('\n');
  myString = trim(myString);
  int _notes[] = int(split(myString, ','));
  if (_notes.length==8) {
    notes=_notes;
  }
  myPort.write("A");
}
void renderImage(PImage img, Vec3D _pos, float _diam, color _col, float _alpha ) {
  pushMatrix();
  translate( _pos.x, _pos.y, _pos.z );
  tint(red(_col), green(_col), blue(_col), _alpha);
  imageMode(CENTER);
  image(img,0,0,_diam,_diam);
  popMatrix();
}
class Emitter{
  Vec3D pos;
  Vec3D vel;
  Vec3D velToMouse;
  
  color myColor;
  
  ArrayList particles;
  
  Emitter(  ){
    pos        = new Vec3D();
    vel        = new Vec3D();
    velToMouse = new Vec3D();
    
    myColor    = color( 1, 1, 1 );
    
    particles  = new ArrayList();
  }
  
  void exist(){
    setVelToMouse();
    findVelocity();
    setPosition();
    iterateListExist();
    render();
    
    pgl.disable( PGL.TEXTURE_2D );
    
    if( ALLOWTRAILS )
      iterateListRenderTrails();
  }
   
  void findVelocity(){
    vel.interpolateToSelf( velToMouse, .35 );
  }
  
  void setPosition(){
    pos.addSelf( vel );
    
    if( ALLOWFLOOR ){
      if( pos.y > floorLevel ){
        pos.y = floorLevel;
        vel.y = 0;
      }
    }
  }
  
  void iterateListExist(){
    for( Iterator it = particles.iterator(); it.hasNext(); ){
      Particle p = (Particle) it.next();
      if( !p.ISDEAD ){
        p.exist();
      } else {
        it.remove();
      }
    }
  }
  
  void iterateListRenderTrails(){
    for( Iterator it = particles.iterator(); it.hasNext(); ){
      Particle p = (Particle) it.next();
      p.renderTrails();
    }
  }

  void addParticles( int _amt ){
    for( int i=0; i<_amt; i++ ){
      particles.add( new Particle( pos, vel, c ) );
    }
  }
}
class Particle {
  int len;            // number of elements in position array
  Vec3D[] pos;        // array of position vectors
  Vec3D startpos;     // just used to make sure every pos[] is initialized to the same position
  Vec3D vel;          // velocity vector
  Vec3D perlin;       // perlin noise vector
  float radius;       // particle's size
  float age;          // current age of particle
  int lifeSpan;       // max allowed age of particle
  float agePer;       // range from 1.0 (birth) to 0.0 (death)
  float bounceAge;    // amount to age particle when it bounces off floor
  boolean ISDEAD;     // if age == lifeSpan, make particle die
  boolean ISBOUNCING; // if particle hits the floor...
  color col;

  Particle( Vec3D _pos, Vec3D _vel, color _c ) {
    radius      = random( 10, 50 );
    len         = (int)( radius );
    pos         = new Vec3D[ len ];
    col = _c;
    startpos    = new Vec3D( _pos.add( new Vec3D().randomVector().scaleSelf( random( 5.0 ) ) ) ); 

    for ( int i=0; i<len; i++ ) {
      pos[i]    = new Vec3D( startpos );
    }

    vel   = new Vec3D( _vel.scale( .5 ).addSelf( new Vec3D().randomVector().scaleSelf( random( 1 ) ) ) );
    vel.x = random(-.1, .1);
    vel.y = random(-5, -1);

    perlin      = new Vec3D();

    age         = 0;
    bounceAge   = 2;
    lifeSpan    = (int)( radius )*10;
  }

  void exist() {
    if ( ALLOWPERLIN )
      findPerlin();

    findVelocity();
    setPosition();
    render();
    setAge();
  }

  void findPerlin() {
    float xyRads      = getRads( pos[0].x, pos[0].z, 10.0, 30.0 );
    float yRads       = getRads( pos[0].x, pos[0].y, 10.0, 30.0 );
    perlin.set( cos(xyRads), -sin(yRads), sin(xyRads) );
    perlin.scaleSelf( .5 );
  }

  void findVelocity() {
    if ( ALLOWGRAVITY )
      vel.addSelf( gravity );

    if ( ALLOWPERLIN ) {
      if (agePer<.75 && agePer > 0 ) {
        Vec3D spread = new Vec3D(random(-1, 1), 0, random(-.5, .5)); 
        vel.addSelf(spread);
      }
    }
  }

  void setPosition() {
    for ( int i=len-1; i>0; i-- ) {
      pos[i].set( pos[i-1] );
    }
    pos[0].addSelf( vel );
  }

  void render() {
    renderImage(particleImg, pos[0], radius * agePer, col, 1.0 );
  }

  void renderTrails() {
    float xp, yp, zp;
    float xOff, yOff, zOff;
    beginShape(QUAD_STRIP);
    for ( int i=0; i<len - 1; i++ ) {
      float per     = 1.0 - (float)i/(float)(len-1);
      xp            = pos[i].x;
      yp            = pos[i].y;
      zp            = pos[i].z;

      if ( i < len - 2 ) {
        Vec3D perp0 = pos[i].sub( pos[i+1] );
        Vec3D perp1 = perp0.cross( new Vec3D( 0, 1, 0 ) ).normalize();
        Vec3D perp2 = perp0.cross( perp1 ).normalize();
        perp1 = perp0.cross( perp2 ).normalize();

        xOff        = perp1.x * radius * agePer * per * .1;
        yOff        = perp1.y * radius * agePer * per * .1;
        zOff        = perp1.z * radius * agePer * per * .1;

        fill( per, per*.25, 1.0 - per, per * .5);
        noStroke();
        vertex( xp - xOff, yp - yOff, zp - zOff );
        vertex( xp + xOff, yp + yOff, zp + zOff );
      }
    }
    endShape();
  }

  void setAge() {
    if ( ALLOWFLOOR ) {
      if ( ISBOUNCING ) {
        age += bounceAge;
        bounceAge ++;
      } else {
        age += .25;
      }
    } else {
      age ++;
    }

    if ( age > lifeSpan ) {
      ISDEAD = true;
    } else {
      // When spawned, the agePer is 1.0.
      // When death occurs, the agePer is 0.0.
      agePer = 1.0 - age/(float)lifeSpan;
    }
  }
}

 

Visual

I continuously changed around the png being emitted, the location of emission and the colour of the emitters. In the end, I settled with a png with one small bubble rather than the bulky bubble groups that I initially used. The location of the emission was in line with the keys.

 

Showcase

The showcase, personally, was quite a stressful event as I was constantly worried about my wires when watching people bang on the keys. It was equally stressful when the bubbles didn’t come up because people were playing too fast – maybe I should have had Adagio projected on the screen for subtle warning.

I got a lot of interesting feedback but the two comments that were the most interesting were:

1. it would be useful to apply this to music education / piano education for children.

2. “I would actually get something like this for my future children. Like seriously.”

But overall, it was very pleasing and satisfying to watch people ooh and ahh as they play the piano.

Improvements

If I were to improve this project, I could extend the effect to all keys on the piano and find a way to make switches work even when the keys are pressed fast.

Final Project Documentation.

Background

My final project was a, “DJ Mixer” where the user could use his/her own hands to control and mix two songs together to play at the same time. For example, Ed Sheerans’s shape of you background music could be played with Justin Bieber singing love yourself.

 

Timeline

  1. Used the Minim Processing library to simply play 2 songs at the same time, one kareoke and one acapella, and was working on a volume controller which looked like this initially and worked with the cursor on the computer:

2. Integrated it to the kinect using  a library by Dan Shiffman and added pause buttons.

3. Added many more songs, removed pause buttons, instead the currently playing song could be hovered on to pause.

 

4. I then made many improvements to the previous version:

  • I improved the design a bit.
  • Cropped the mp3 files of the songs to go directly to the music rather than a pause or an intro as many of the youtube videos had them.
  • Made the webcam visible in a small area at the bottom.
  • I also made the volume be always visible to the user as it wasn’t clear to them that that functionality was possible.

5. The final update was mainly a huge UI and Design bump which I will go over in the next section. I also added a, “DJ Training” stage in order to train users on how to move the hand around the screen to select stuff. (In the video below I use the computer cursor as the pointer for demo purposes)

Technical Aspect

The base of my project was not hard to make. I am glad I went with the object oriented approach as I’m sure it saved me TONS of time as I started making many upgrades to it. I created a Song class that included the functionality of instantiating the song with its song file, displaying the song on the screen, having on hover functionality for that song, playing and pausing functions. I then made an array of Song objects in a songs array and instantiated them all. This made it trivial to add new songs. Using an object oriented approach allowed me to implement the volume controller class and add in many design improvements such as font and on hover shading easily.

 

The controlling of the cursor was perhaps the most important part technically as it is what gives the user the ability to interact with my project so it had to be as seamless as possible which is why I focused on it a lot. The kinect integration was simple as I got the code from Dan Shiffmans website. However, the detection of the hand was VERY buggy initially and I had to make many changes to optimize it specifically for my requirement:

  • The first important thing I did was make the cursor return to a specific spot if there is no depth greater than the threshold value. This reduced some unintentional clicking.
  • The second is that I optimized Dan Shiffman’s Kinect class to suit my needs by adding 2 counters: one totalArea counter which increments in the for loop and one handArea counter which also increments in the for loop but only when the depth is greater than the threshold value. I then only allowed the movement of the cursor if handArea/totalArea=0.065 as after some testing found that the hand is less than 6.5% of the total kinect screen.
  • I made everything hover based, any song, pause button and volume control has its own counter which I use to make it function only when the cursor hovers over that part for some time. This dramatically reduced unintentional clicks.

Design and Interactivity

As you have seem there was a bump in the design from the 90% ready project and the 100% ready project. Let me take you through some of my decisions.

  • The adding of icons was important as professor suggested, as it gave the users a clear idea of what would happen when clicked which is why I used icons for pause, and a sound icon near the volume controllers. This worked well in practice as I didn’t have to explain much to the users, they simply knew what to do.

 

 

 

 

  • I added a hover shade effect when a user hovers over a song name and a color change to the pause buttons when hovered. I did this so that first time users would know that the song is indeed clickable (or pause button). It almost gives the, ‘hyperlink’ feel to the user which is familiar to them and hence serves my project better in terms of interaction.
  • I worked hard on the volume controller, which is why it has its own class. I wanted to design it just like a volume controller in a dj rack where the user could move a knob along a slider in order to control the volume. I did this so the user could relate what they see to a volume knob in a sound mixer in order to know what the volume controller does.  The background of the volume controller initially were horizontal, equally spaced lines but I decided to make it random constantly moving lines to give it a live feel so that the user knows that something can be done by using it.

 

 

  • I needed a good font as the one I selected initially wasn’t appealing to many users. I wanted a font that would be:
    • Easy to read
    • Had some type of old-school tech feel to it (like how text is displayed on a small LED display)
    • Seemed intriguing to catch some passer-by’s attention

After some considerable research, I found a font called “underwood”, which seemed like a good choice as it fit my criteria.

  • The background image of a DJ rack that I found on google images was a nice addition as it fit my project well. The two racks seemed cohesive with the two sides (kareoke and acapella) that the user could choose songs from. Many users actually started hovering over the DJ racks themselves thinking it would do something.

The code

Main code:

import ddf.minim.*;
import ddf.minim.signals.*;
import ddf.minim.analysis.*;
import ddf.minim.effects.*;
import org.openkinect.freenect.*;
import org.openkinect.processing.*;
import processing.video.*;
KinectTracker tracker;
Kinect kinect;
Minim minim;
Movie myMovie;
AudioPlayer splatterSound;
PVector v1, v2;
float vol2, vol3;
int CurRad=30;
boolean lPaused=false;
int lCounter=0;
boolean rPaused=false;
int rCounter=0;
float pX, pY;
float maxX=0; 
float maxY=0;
int kS=99;
int aS=99;
PFont sFont, bFont;
boolean debugMode=false;
PImage hand,splatter,bg,slider,djBg,bHand;
int rVolCounter=0;
int lVolCounter=0;
int rSongCounter=0;
int r2SongCounter=0;
boolean randomSongMode=false;
int stage=0;
boolean djTestPassed=false;
int djTestMarks=0;
Song[] songs = new Song[14];
float[] djHandPosX = new float[3];
float[] djHandPosY = new float[3];
float[] djCounter = new float[3];
boolean[] djCircleClicked = new boolean[3];
boolean randomHover=false;
int djTrainingFinishedCounter=0;
boolean djTrainingReached=false;
VolumeController lVolController,rVolController;
void setup() {
  //size(640, 480);
  fullScreen();
  background(0);
  kinect = new Kinect(this);
  tracker = new KinectTracker();
  minim = new Minim(this);
  splatterSound = minim.loadFile("splatter.mp3", 1024);
  sFont =  createFont("underwood.ttf", 24); //cool.ttf
  bFont =  createFont("capture.ttf", 30);
  hand=loadImage("hand.png");
  bHand=loadImage("bHand.png");
  splatter=loadImage("splatter.png");
  slider=loadImage("slider.png");
  slider.resize(50,60);
  bg=loadImage("bg.jpg");
  djBg=loadImage("djBg.jpg");
    bg.resize(width,height);
  djBg.resize(width,height);
  songs[0] = new Song("k_billy.mp3", false, "billy jean", width/5, 100, 0);           songs[2] = new Song("a_mnm.mp3", true, "lose yourself", width*0.55, 100, 2);
  songs[1] = new Song("k_shape.mp3", false, "shape of you", width/5, 160, 1);         songs[3] = new Song("a_rolling.mp3", true, "roll in the deep", width*0.55, 160, 3);
  songs[5] = new Song("k_strange.mp3", false, "stranger things", width/5, 220, 5);    songs[4] = new Song("a_baby.mp3", true, "baby", width*0.55, 220, 4);
  songs[6] = new Song("k_beat.mp3", false, "beat", width/5, 280, 6);                  songs[7] = new Song("a_yodel.mp3", true, "yodel", width*0.55, 280, 7);
  songs[8] = new Song("k_human.mp3", false, "human", width/5, 340, 8);                songs[9] = new Song("a_love.mp3", true, "love yourself", width*0.55, 340, 9);
  songs[10]= new Song("k_fancy.mp3", false, "fancy", width/5, 400, 10);               songs[11]= new Song("a_closer.mp3", true, "closer", width*0.55, 400, 11);
  songs[12]= new Song("k_fur_elise.mp3", false, "fur elise", width/5, 480, 10);       songs[13]= new Song("a_logic.mp3", true, "logic", width*0.55, 480, 11);
  kS=0;
  aS=3;
  songs[kS].play();
  songs[aS].play();
  smooth(10);
  djHandPosX[0]=width/random(1,1.5); djHandPosX[1]=width/2; djHandPosX[2]=width/6;
  djHandPosY[0]=height/1.5; djHandPosY[1]=height/5; djHandPosY[2]=height/1.8;
    djCircleClicked[0]=false; djCircleClicked[1]=false; djCircleClicked[2]=false;
    djCounter[0]=0; djCounter[1]=0; djCounter[2]=0;
    lVolController= new VolumeController(width/7-50,250,50,250);
    rVolController= new VolumeController(width-width/7,250,50,250);
    tracker.setThreshold(435);
}      

void draw() {
  if(djTestPassed) {
     background(bg);
  tracker.track();
  tracker.display();
  v1 = tracker.getPos();
  v2 = tracker.getLerpedPos();
  pX=mouseX;pY=mouseY;
  //pX=v2.x;pY=v2.y;
  //pX=map(v2.x, 0, 640, 0, width); pY=map(v2.y, 0, 480, 0, height);
  //pX=map(v1.x, 0, 640, 0, width); pY=map(v1.y, 0, 480, 0, height);
  if (pX>maxX) {
    maxX=pX;
  }
  if (pY>maxX) {
    maxY=pY;
  }
  int t = tracker.getThreshold();
  
  noStroke();
  lVolController.draw(songs[kS]);
  rVolController.draw(songs[aS]);
  float radius;
  for (Song ss : songs) {
    ss.draw();
  }
  //HOVER CLICKING CODE
  if ((kS!=99)&&(songs[kS].acapella==acapella)&&songClicked) {
    songs[kS].pause();
  } 
  if ((kS!=99)&&(songs[aS].acapella==acapella)&&songClicked) {
    songs[aS].pause();
  }
  int indexer=0;
  for (Song ss : songs) {
    if ((!ss.paused)&&(!ss.clicked())) {
      if (ss.acapella) {
        aS=indexer;
      } else {
        kS=indexer;
      }
    } 
    if (ss.clicked()) {
      songClicked=true;
    }
    indexer++;
  }
  int indexer2=0;
  for (Song ss : songs) {
    if ((ss.clicked())&&songClicked) {
      ss.click();
    }
  }
  indexer2=0;
  songClicked=false;
  //HOVER CLICKING CODE END
  if (CurRad<30) {
    CurRad=30;
  }
  if (pX<width/2) {
    if (lPaused) {
      radius = CurRad + 30 * sin( frameCount * 0.05f );
    } else {
      radius=CurRad;
    }

    fill(155, 18, 13);
  } else {
    if (rPaused) {
      radius = CurRad + 30 * sin( frameCount * 0.05f );
    } else {
      radius=CurRad;
    }
    fill(16, 46, 117);
  }
  if((randomSongMode)||(randomHover)) {fill(200);} else {fill(255);}
  rect(width/2-100,height/1.35,200,35);
  textFont(sFont);
  textAlign(CENTER);
  fill(0);
  text("RANDOM",width/2-100,height/1.35+8,200,35);
   textFont(sFont);
  imageMode(CENTER);
  image(hand,pX,pY,50,50);
  randomSong();
  randomSongMode();
} else {
  background(djBg);
  tracker.track();
  tracker.display();
  v1 = tracker.getPos();
  v2 = tracker.getLerpedPos();
  pX=mouseX;pY=mouseY;
  //pX=v2.x;pY=v2.y;
  //pX=map(v2.x, 0, 640, 0, width); pY=map(v2.y, 0, 480, 0, height);
  //pX=map(v1.x, 0, 640, 0, width); pY=map(v1.y, 0, 480, 0, height);
  if (pX>maxX) {
    maxX=pX;
  }
  if (pY>maxX) {
    maxY=pY;
  }
  int t = tracker.getThreshold();
  tracker.setThreshold(630);
  textSize(55);
  fill(0);
  textAlign(CENTER);
  if(!djTrainingReached) {
  text("Place your hand on the circles for DJ Training",width/18,100,width,100);
  }
  djTraining();
  imageMode(CENTER);
  if(djTestPassed) {
  image(hand,pX,pY,50,50);
  } else {
  image(bHand,pX,pY,50,50);
  }
}
}
void keyPressed() {
  int t = tracker.getThreshold();
  if (key == CODED) {
    if (keyCode == UP) {
      t+=5;
      tracker.setThreshold(t);
    } else if (keyCode == DOWN) {
      t-=5;
      tracker.setThreshold(t);
      
      
    }
    fill(255);
      text(t,width/2,height-100);
    println(t);
  }//END KINECT
  if (key == 'x'||key == 'X') {
  debugMode=!debugMode;
  }
    if (key == 'R'||key == 'r') {
  randomSongMode=!randomSongMode;
  }
      if (key == 'd'||key == 'D') {
        djTestMarks=0;
        djCircleClicked[0]=false;
        djCircleClicked[1]=false;
        djCircleClicked[2]=false;
        djTrainingReached=false;
        djTrainingFinishedCounter=0;
        djTestPassed=!djTestPassed;
        debugMode=true;
  }
}
boolean acapella;
boolean songClicked=false;

Song class:

class Song {
  Minim minimThing;
  AudioPlayer song;
  boolean paused=true;
  boolean acapella;
  float x, y;
  String title;
  int index;
  boolean clicked=false;
  int clickCounter=0;
  float aX=width/2;
  float aY=20;
  float kX=width/4;
  float kY=20;
  float curX=x;
  float curY=y;
  float prevX;
  float prevY;
  float defaultX;
  float defaultY;
  float Lx, Ly;
  boolean hover=false;
  Song(String fileName, boolean songType, String titleVal, float xPos, float yPos, int indexer) {
    minimThing = new Minim(this);
    song = minim.loadFile(fileName, 1024);
    acapella=songType;
    x=xPos;
    y=yPos;
    title=titleVal;
    index=indexer;
    prevX=x;
    prevY=y;
    defaultX=x;
    defaultY=y;
  }  
  void draw() { //boolean songType=true means acapella else kareoke

    textSize(24);
    textAlign(CENTER);
    
    if (paused) {
      clicked();
      if(!hover) {
      fill(255); } else {
      fill(100+clickCounter);
      }
      textSize(24);
      textFont(sFont);
      Lx = lerp(prevX, x, 0.05);
      Ly = lerp(prevY, y, 0.05);
      text(title, Lx, Ly, 300, 50);

      curX=x;
      curY=y;
    } else if (acapella) {
      //textFont(bFont);
      fill(0);
      Lx = lerp(prevX, aX, 0.05);
      Ly = lerp(prevY, aY, 0.05);
      fill(255);
      rect(aX, aY, 300, 50);
      fill(0);
      if(!hover) {fill(0);} else {fill(0,0,200);}
      rect(aX+10,aY+10,10,30);
      rect(aX+23,aY+10,10,30);
      fill(0);
      textAlign(CENTER);
      text(title, Lx+15, Ly+13, 300, 50);
      curX=aX;
      curY=aY;
    } else {
      //textFont(bFont);
      fill(0);
      
      Lx = lerp(prevX, kX, 0.05);
      Ly = lerp(prevY, kY, 0.05);
      fill(255);
      rect(kX, kY, 300, 50);
      if(!hover) {fill(0);} else {fill(0,0,200);}
      rect(kX+10,kY+10,10,30);
      rect(kX+23,kY+10,10,30);
      fill(0);
      textAlign(CENTER);
      text(title, Lx+40, Ly+13, 200, 50);
      curX=kX;
      curY=kY;
    }
    prevX=curX;
    prevY=curY;
  }
  void play() {
    //print(title);
    
    if (acapella) {
      if (clicked()) {
        songs[aS].pause();
      }
      aS=index;
    } else {
      if (clicked()) {
        songs[kS].pause();
      }
      kS=index;
    }
    song.loop();
    paused=false;
  }
  void pause() {
    song.pause();
    paused=true;
  }
  void togglePlay() {
    if (paused) {
      play();
    } else {
      pause();
    }
  }
  void setGain(float vol) {
    song.setGain(vol);
  }
  boolean clicked() {
    if ((pX>curX)&&(pX<curX+200)&&(pY>curY)&&(pY<curY+50)) {
      hover=true;
      
      if ((clickCounter==70)) {

        CurRad=30;

        boolean clicked=true;

        return true;
      } else {

        clickCounter++;

        CurRad=clickCounter;

        boolean clicked=false;
      }
    } else {
      hover=false;
      CurRad=30;

      clickCounter=0;
      clicked=false;
      return false;
    }
    clicked=false;
    return false;
  }
  void click() {

    togglePlay();
    //background(255);
  }
}

void randomSong() {
  //rect(width/2-100,height/1.35,200,35);
  if((pX>(width/2-100))&&(pX<(width/2+100))&&(pY>(height/1.35))&&(pY<((height/1.35))+35)) {
    randomHover=true;
    if((rSongCounter==25)||(((rSongCounter%125)==0)&&(rSongCounter!=0))) {
int first= int(random(0,songs.length));
int second=int(random(0,songs.length));
while((songs[first].acapella==songs[second].acapella)) {
first= int(random(0,songs.length));
second=int(random(0,songs.length));
}
songs[aS].pause();
songs[kS].pause();
songs[first].play();
songs[second].play();
    }
rSongCounter++;
  } else {
    randomHover=false;
  rSongCounter=0;
  
  }
}

void randomSongMode() {
  if(!randomSongMode) {r2SongCounter=0;} else {
    if((((r2SongCounter%400)==0)&&(r2SongCounter!=0))) {
int first= int(random(0,songs.length));
int second=int(random(0,songs.length));
while((songs[first].acapella==songs[second].acapella)) {
first= int(random(0,songs.length));
second=int(random(0,songs.length));
}
songs[aS].pause();
songs[kS].pause();
songs[first].play();
songs[second].play();
    }
r2SongCounter++;
  }
}

Upgraded Kinect Class from Dan Schiffman:

// Daniel Shiffman
// Tracking the average location beyond a given depth threshold
// Thanks to Dan O'Sullivan

// https://github.com/shiffman/OpenKinect-for-Processing
// http://shiffman.net/p5/kinect/

class KinectTracker {

  // Depth threshold
  int threshold = 435;

  // Raw location
  PVector loc;

  // Interpolated location
  PVector lerpedLoc;

  // Depth data
  int[] depth;
  
  // What we'll show the user
  PImage display;
   
  KinectTracker() {
    // This is an awkard use of a global variable here
    // But doing it this way for simplicity
    kinect.initDepth();
    kinect.initVideo();
    kinect.enableMirror(true);
    
    // Make a blank image
    display = createImage(kinect.width, kinect.height, RGB);
    // Set up the vectors
    loc = new PVector(0, 0);
    lerpedLoc = new PVector(0, 0);
  }

  void track() {
    // Get the raw depth as array of integers
    depth = kinect.getRawDepth();
    float totalArea=0;
    float handArea=0;
    // Being overly cautious here
    if (depth == null) return;

    float sumX = 0;
    float sumY = 0;
    float count = 0;

    for (int x = 0; x < kinect.width; x++) {
      for (int y = 0; y < kinect.height; y++) {
        totalArea++;
        int offset =  x + y*kinect.width;
        // Grabbing the raw depth
        int rawDepth = depth[offset];

        // Testing against threshold
        if (rawDepth < threshold) {
          handArea++;
          sumX += x;
          sumY += y;
          count++;
          
        } else {
        loc=new PVector(kinect.width/2,kinect.height/1.2);
        }
      }
    }
    // As long as we found something
    //println(sumY);
    //println((handArea/totalArea));
   if ((count != 0)&&((handArea/totalArea)<0.065)) { //if ((count != 0)&&(sumX<8000000)&&(sumX>150000)&&(sumY<6000000)&&(sumY>1000)) {
      loc = new PVector(sumX/count, sumY/count);
    }

    // Interpolating the location, doing it arbitrarily for now
    lerpedLoc.x = PApplet.lerp(lerpedLoc.x, loc.x, 0.3f);
    lerpedLoc.y = PApplet.lerp(lerpedLoc.y, loc.y, 0.3f);
  }

  PVector getLerpedPos() {
    return lerpedLoc;
  }

  PVector getPos() {
    return loc;
  }

  void display() {
    PImage img = kinect.getDepthImage();
    PImage webcam = kinect.getVideoImage();
    // Being overly cautious here
    if (depth == null || img == null) return;

    // Going to rewrite the depth image to show which pixels are in threshold
    // A lot of this is redundant, but this is just for demonstration purposes
    display.loadPixels();
    for (int x = 0; x < kinect.width; x++) {
      for (int y = 0; y < kinect.height; y++) {

        int offset = x + y * kinect.width;
        // Raw depth
        int rawDepth = depth[offset];
        int pix = x + y * display.width;
        if (rawDepth < threshold) {
          // A red color instead
          display.pixels[pix] = color(255, 10, 10);
        } else {
          //display.pixels[pix] = color(255);
          display.pixels[pix] = img.pixels[offset];
        }
      }
    }
    display.updatePixels();

    // Draw the image
    if(debugMode) {
    image(display,width/2, height*0.75,width/4,height/4); //display/webcam
    }
  }

  int getThreshold() {
    return threshold;
  }

  void setThreshold(int t) {
    threshold =  t;
  }
}

Volume Controller Class

class VolumeController {
  int x,y;
  int lVolCounter=0;
  int w;
  int h;
  int padding=50;
  float volume=0;
  PImage audioImage;
  float defaultVolume=0;
  VolumeController(int xPosition, int yPosition, int dWidth,int dHeight) {
    x=xPosition;
    y=yPosition;
    w=dWidth;
    h=dHeight;
    audioImage=loadImage("audio.png");
    audioImage.resize(25,25);
  }
  
void draw(Song curSong) {
      float displayVol=map(volume,-30,0,0,h);
      float pos=(y+h)-displayVol;
  if(((pX>x-padding)&&(pX<x+w+padding)&&(pY>y-padding)&&(pY<y+h+padding))) {
   // if(((pX>x-padding)&&(pX<x+w+padding)&&(pY>pos-20)&&(pY<pos+20))) {
  if(lVolCounter>15) {
    volume = map((y+h)-pY, 0, h, -30, 0);
    displayController(curSong);
    curSong.setGain(volume);
    } else {
    displayController(curSong);
    }
    lVolCounter++;
  } else {
    lVolCounter=0;
    displayController(curSong);
  }
}

void displayController(Song curSong) {
  image(audioImage,x+w/2,y-25);
  stroke(0);
  noFill();
  rect(x,y,w,h);
  noStroke();
        for(int yp=0;yp<h;yp++){
          if(curSong.paused) {
          if(yp%15==0) {
        fill(random(180,200));
      }
          }
          
      else if(yp%2==0) {
        fill(random(230,255));
      } else  if(yp%10==0) {
        fill(random(200,220));
      } else if(yp%15==0) {
        fill(random(180,200));
      }
     rect(x+5,y+yp,w-5,1);
     }
     
         fill(0);
    rect(x+w/2,y,5,h);
    fill(0,0,128);
    noStroke();
    //rect(5, height*0.65-volDis2, 30, volDis2);
    float displayVol=map(volume,-30,0,0,h);
float pos=(y+h)-displayVol;
if(pos>(y+h)-20) {pos=y+h-20;} else if(pos<(y)){pos=y;}
    rect(x,pos, w, 20);
}
}

DJ Training function:

void djTraining() {
  boolean playSound=false;
for(int x=0;x<(djHandPosX.length);x++) {
  //ellipse(djHandPosX[x],djHandPosY[x],40,40);
  if((pX>djHandPosX[x]-25)&&(pX<djHandPosX[x]+25)&&(pY>djHandPosY[x]-25)&&(pY<djHandPosY[x]+25)) {
  fill(255,0,0);
  //djHandPosX[x]=9000;
  //djHandPosY[x]=9000;
  djCounter[x]++;
  if(djCounter[x]==40) {playSound=true;}
  if(djCounter[x]==55) {
  djCircleClicked[x]=true;
  djTestMarks++;
  
  } 
  } else {
  fill(50);
  djCounter[x]=0
;  }

  if(!djCircleClicked[x]) {
    noStroke();
  ellipse(djHandPosX[x],djHandPosY[x],50+djCounter[x],50+djCounter[x]);
  } else {
      if (!splatterSound.isPlaying()) { splatterSound.rewind();
 }

  image(splatter,djHandPosX[x],djHandPosY[x],300,300);
  }
       if(playSound) {
   splatterSound.play();
   }} 
if(djTestMarks==djHandPosX.length) {
djTrainingFinishedCounter++;
djTrainingReached=true;
debugMode=false;
fill(0);
text("Get your DJ Belts on.",width/2,height/2);
if(djTrainingFinishedCounter>120) {
djTestPassed=true;
}
}
}

 

 

 

 

 

Fly Me to the Moon! (and the sun, planets, and a stray spacestation)

Brain Baby

I love cosmology and astrophysics. The idea that there is something out there that is more than life on Earth astounds me. Given the opportunity to base my project on any theme, I bounced on the opportunity to explore space.

I wanted to create something that is simple to navigate and would make people fall in love with cosmology as well. I thought the easiest thing for me to do so was to create an adventure type thing where people could fly a rocketship around to visit different “landmarks” in outer space. I initially wanted to have an Arduino x Processing project where essentially I would have a rocketship that is actually a magnet or something that would close the circuit and each planet/location in the solar system would be the switch, where when the rocket lands on it, the switch would close. When this happens, depending on which object the rocket lands on, a video/animation would play.

During our class discussing, Aaron brought up the idea of having a camera that tracks the location of the rocketship. He brought up something called TUIO, which of course I had no idea what it was but was immediately drawn to it because who doesn’t love cool technology right? I, therefore, decided I wanted to learn how this tracking thing worked and so abandoned my previous idea to do this new cool thing.

Resurging Regret

A few days later, I regretted my decision. When Aaron installed the projector and camera to the ceiling, I knew that I couldn’t back out anymore. (Abdullah took a video of my expression during the exact moment I realised how hard my life was going to be for the next two weeks, it was actually really funny but for my own appearances sake, I shall not post the video here.) Thankfully, Aaron worked with me for many hours to help me figure out how projection mapping worked, and how the PS3 Eye worked, how to calibrate the camera, and essentially explained to me every bit of the code that would allow me to proceed with the actual body of my project (i.e. the easy part). It took me a while of experimenting and looking at the code that Aaron had just explained to me for me to digest what exactly I needed to do to proceed with my project.

After teaching myself some code functions that I needed to successfully use OpenCV (sending my sketch to the projection), the rest of the project was easier. Knowing exactly what I wanted to do, the next part of my project was the easiest part. Utilising the coding knowledge that I already know, I completed the first “draft” of my project which I shared in my last blog post here.

(Please read that one before reading the next part of this reflection.)

Stressful Situations

Implementing the suggestions were hard. The part about the flashing Aaron fixed in a heartbeat but man, that first suggestion about being able to explore the planets really angered me. I couldn’t figure out why putting an if statement in my already existing if statement didn’t work, and why the while function basically made my whole sketch crash. NOTHING WAS MAKING SENSE! It was until I spoke to Aaron and he told me what was wrong and wrote the first part of the code for me and everything made sense again and nothing was wrong with the world. (Really though I still don’t understand how Aaron’s brain works and functions so fast and was able to solve my problem so quickly like honestly I was stuck on that part for at least a day, no joke. Ask someone who was in the lab then.) Implementing the switch mode/cases function that Aaron taught me, I was able to improve my project by allowing people to “explore” the planets/sun as much as they wanted and then going to the main page by moving the rocket to the exit at the right top corner. I also couldn’t figure out the playing of rocketship moving music when the rocketship was moving, but Aaron also helped me with this coding part and taught me the logic behind the code.

I also managed to print a rocketship as well, however, I didn’t use it during my first user-testing experience.

In the following class, I then got more feedback on how to improve. 1) Have another exit button so that it would be easier for people to navigate, 2) Have some text on top of the animations so that people could actually learn about the solar system like I originally wanted people to do. Kristopher jokingly said to get my facts from the first paragraph of each planets’ respective Wikipedia page; however, little did he know that it was exactly what I did. Thanks for the idea Kristopher!

I decided that instead of generating text directly onto the sketch, it would be easier to generate text boxes elsewhere and put them onto the animations as images. It took me a while to adjust the positions of the text boxes and the amount of text within each box so that the words would be legible but nothing was specifically hard about this process, it just took a lot of time.

The next step – which I thought was going to be the easiest step – actually took me a long time as well. I wanted to make the rocketship as mobile as possible and not situating it on top of the breadboard with the battery pack sitting just below the rocket. I wanted it to look professional and cool and ready to blast off any second. The original idea was to use coin batteries. But after multiple hours of soldering and experimenting with it, I realised that it didn’t have enough power to power the IR LED for long. Especially because the camera was even further away in the exhibition room, it was impossible to even track the light. Therefore, I had to resort back to using 4 double A batteries.

I don’t know how I even came up with the idea but I realised that the power strip part of the breadboard could be separated from the rest of the breadboard and it fit perfectly in the body of my rocket. I realised that I just needed to attach the LED on one end while having the battery pack plugged in the other end and that was all that was needed. After almost giving up on making my rocket look somewhat pretty, this idea worked! To not expose the batteries, I decided to cover it with a layer of aluminium foil and stick fire like images on all four sides of the battery so that it would like the rocket is blasting off.

With the completion of the rocket, I WAS DONE WITH MY PROJECT! HOORAY!

I forgot to mention that during the times I got stuck on my main project, I was coding this other makeshift computer generated solar system that has a sun in the middle with planets orbiting it. But wanting to make it unorthodox,  I had planets orbit the planets. The generated solar system was what would be seen when people land on the space station. An example of a generated solar system is seen here:

Showcase Shenanigans!

On the day of the showcase, I was geared and ready to go. I was excited that other people would be able to rekindle with their elementary school astronomy days and/or learn about our solar system.

The first 20 minutes of the showcase went by fast and really well. Everything was working, people were having fun. HOWEVER, 5 mins later while a person who looked pretty important was testing out my project, MY PROJECT STOPPED WORKING. A chill literally ran through me. My computer informed me that it had run out of RAM and before I could force quit my sketch and rerun it again, it literally froze. I had to resort to restarting my laptop, which was fine except for the fact that it takes around a billion years for my sketch to load. Approximately 7 minutes later, everything was back up and running again. Phew.

Anticipating that the same situation again, I made a mental note that in 20 mins, I would restart the sketch again so that my computer wouldn’t crash. I did so and everything was fine, except that 5 mins later after the sketch restarted again, SOMEONE PICKED UP MY ROCKET AND SHOOK THE WHOLE THING. The wires, unfortunately, came loose and so in front of everyone, I had to disect my rocket and reveal the anatomy of the poor thing. Man was I stressed. Fixing took a max 10 mins, but everything worked fine again. Everything was fine again until some little kid literally took my rocket apart and pulled the wires out but ya know, everything was cool. I tried to fix it but the clock stroke 8:30 and I abandoned my mission to fix my broken rocketship.

A compilation of people playing/learning with my project is here:

Despite my computer system crashing and the breaking of my rocketship, I think that the showcase went well. I got some super amazing feedback and some really interesting constructive criticism. Reflecting now, I learned a whole lot since 14 weeks ago when I literally knew NOTHING about coding. Though the past 14 weeks have been stressful, looking back, everything was really really worthwhile.

HERE’S A REALLY BIG THANK YOU TO THOSE WHO HAVE HELPED ME ALONG THE WAY AND BEEN PATIENT WITH ME, ESPECIALLY AARON, AND WHO HAVE TOLERATED ME THROUGHOUT THE PAST 14 WEEKS BECAUSE I PROBABLY HAVE ANNOYED EVERYONE WITH MY WEEKLY LOUD “WHY ISN’T THIS WORKING?!??!”‘S IN THE LAB. On top of learning some really cool and useful coding skills, I have met some really amazing people and have found a new interest that I hope I can pursue in the future. Surviving Intro to IM is probably the greatest highlight of this school year. 🙂

Here is the code for my final project:

Mainpage –

// import json file from mappingtackedspace 
import processing.sound.*;
import gab.opencv.*;
import org.opencv.imgproc.Imgproc;
import org.opencv.core.MatOfPoint2f;
import org.opencv.core.Point;
import org.opencv.core.Size;
import org.opencv.core.Mat;
import org.opencv.core.CvType;
import processing.video.*;
import com.thomasdiewald.ps3eye.PS3EyeP5;
import java.awt.Rectangle;
import codeanticode.syphon.*;
int posX, posY;


SoundFile rocketship;
PS3EyeP5 ps3eye;
Capture video;
OpenCV opencv;
JSONObject json;
PImage warped;
PImage irImage;
int videoWidth = 640;
int videoHeight = 480;
Mat transform;
ArrayList<PVector> vectors = new ArrayList<PVector>();
int blobSizeThreshold = 1;
SyphonServer server;

PGraphics canvas;

Movie sun, mercury, venus, earth, moon, mars, jupiter, saturn, uranus, neptune;
Movie generation1, generation2, generation3;

//String[] generation = {"generation1.mov", "generation2.mov", "generation3.mov"};

PFont title;
//PFont landing;

//String message1 = "Our Solar System";

PImage bg;
PImage ex, ex2;
PImage sun1, sun2, sun3, mercury1, mercury2, venus1, venus2, venus3, earth1, earth2, earth3, moon1, moon2, mars1, mars2, jupiter1, jupiter2, saturn1, saturn2, saturn3, uranus1, uranus2, neptune1, neptune2;

//int i=3;


void setup() {

  size(640, 480, P3D);
  //size(1440, 900, P3D);
  canvas = createGraphics(640, 480, P3D);

  bg = loadImage ("solarsystembackground2.jpg");
  ex = loadImage ("x.png");
  ex2 = loadImage("x2.png");
  sun1 = loadImage("sun1.png");
  sun2 = loadImage("sun2.png");
  sun3 = loadImage("sun3.png");
  mercury1 = loadImage("mercury1.png");
  mercury2 = loadImage("mercury2.png");
  venus1 = loadImage("venus1.png");
  venus2 = loadImage("venus2.png");
  venus3 = loadImage("venus3.png");
  earth1 = loadImage("earth1.png");
  earth2 = loadImage("earth2.png");
  earth3 = loadImage("earth3.png");
  moon1 = loadImage("moon1.png");
  moon2 = loadImage("moon2.png");
  mars1 = loadImage("mars1.png");
  mars2 = loadImage("mars2.png");
  jupiter1 = loadImage("jupiter1.png");
  jupiter2 = loadImage("jupiter2.png");
  saturn1 = loadImage("saturn1.png");
  saturn2 = loadImage("saturn2.png");
  saturn3 = loadImage("saturn3.png");
  uranus1 = loadImage("uranus1.png");
  uranus2 = loadImage("uranus2.png");
  neptune1 = loadImage("neptune1.png");
  neptune2 = loadImage("neptune2.png");


  posX=posY=0;
  warped = createImage(videoWidth, videoHeight, ARGB); 
  irImage = createImage(videoWidth, videoHeight, ARGB); 
  loadData();

  ps3eye = PS3EyeP5.getDevice(this);
  ps3eye.start();
  opencv = new OpenCV(this, irImage);


  //landing = createFont("Nasalization", 25);
  //textFont(landing, 25);

  //generations = new Movie[generations.length];
  //for (int i = 0; i < generation.length; i++) {
  //generation[i] = new Movie(this, generations[i]);
  //}

  rocketship = new SoundFile(this, "rocketship_mono.wav");
  rocketship.loop();
  rocketship.amp(0.0);

  sun = new Movie(this, "sun2.mp4");
  sun.loop();

  mercury = new Movie(this, "mercury2.mp4");
  mercury.loop();

  venus = new Movie(this, "venus2.mp4");
  venus.loop();

  earth = new Movie(this, "earth2.mp4");
  earth.loop();

  moon = new Movie(this, "moon2.mp4");
  moon.loop();

  mars = new Movie(this, "mars2.mp4");
  mars.loop();

  jupiter = new Movie(this, "jupiter2.mp4");
  jupiter.loop();

  saturn = new Movie(this, "saturn2.mp4");
  saturn.loop();

  uranus = new Movie(this, "uranus2.mp4");
  uranus.loop();

  neptune = new Movie(this, "neptune2.mp4");
  neptune.loop();

  generation1 = new Movie(this, "generation3.mov");
  generation1.loop();


  server = new SyphonServer(this, "Final Start");
}

void movieEvent (Movie m) {
  m.read();
}

void draw() {
  //background(0);
  //println(mouseX, mouseY);
  canvas.beginDraw();
  canvas.background (bg);
  title = createFont("Nasalization", 20);
  canvas.textFont(title, 20);
  canvas.textAlign(CENTER);
  canvas.text("Our Solar System", 480, 30);

  cv();
  fill(255, 0, 0);
  ellipse(posX, posY, 30, 30);

  canvas.endDraw();
  image(canvas, 0, 0);
  //image(warped,0,0);
  //server.sendImage(canvas);
  server.sendScreen();
}

Computer vision page –

int curX, curY = 0;
boolean isPlaying = false;
boolean available = true;
int mode = 0;
float volume = 0;
Movie gen;


void cv() {
  if (ps3eye.isAvailable()) {
    irImage = ps3eye.getFrame();
    opencv.loadImage(irImage);
    opencv.threshold(30);
    //changes the video feed to our mapped coords

    opencv.setGray(warpPerspective(vectors, videoWidth, videoHeight));
  }

  //image(irImage,0,0);
  //image(opencv.getOutput(), 0, 0);
  //image(warped,0,0);
  noFill();
  //stroke(255, 0, 0);
  //strokeWeight(3);
  noStroke();

  int prevX = curX;
  int prevY = curY;

  curX = posX;
  curY = posY;
  //print("These are my values : ");
  //println(curX, curY);
  //print("These were my values : ");
  //println(prevX, prevY);
  //println(mouseX, mouseY);


  for (Contour contour : opencv.findContours()) {

    Rectangle r = contour.getBoundingBox();

    //if (r.width > blobSizeThreshold || r.height > blobSizeThreshold) {
    //contour.draw();

    //stroke(255, 0, 0);
    //noFill();
    //strokeWeight(2);
    //rect(r.x, r.y, r.width, r.height);
    posX=(int)r.getCenterX();
    posY=(int)r.getCenterY();
    //ellipse((float)r.getCenterX(), (float)r.getCenterY(), 5, 5);


    if (((posX >0) && (posX<107)) && ((posY>0) && (posY<159)) && available) {
      mode = 1;
      available = false;
    }
    if (((posX > 128)  && (posX<153)) && ((posY >79) && (posY<108)) && available) {
      mode = 2;
      available = false;
    }
    if (((posX > 156) && (posX < 188)) && ((posY > 85) && (posY < 126)) && available) {
      mode = 3;
      available = false;
    }
    if (((posX > 193) && (posX<229)) && ((posY>99) && (posY<146)) && available) {
      mode = 4;
      available = false;
    }
    if (((posX > 230) && (posX<242)) && ((posY>80) && (posY<95)) && available) {
      mode = 5;
      available = false;
    }
    if (((posX > 234) && (posX<272)) && ((posY>124) && (posY<165)) && available) {
      mode = 6;
      available = false;
    }
    if (((posX > 274) && (posX<382)) && ((posY>121) && (posY<249)) && available) {
      mode = 7;
      available = false;
    }
    if (((posX > 327) && (posX<518)) && ((posY>223) && (posY<314)) && available) {
      mode = 8;
      available = false;
    }
    if (((posX > 467) && (posX<535)) && ((posY>306) && (posY<379)) && available) {
      mode = 9;
      available = false;
    }
    if (((posX > 558) && (posX<612)) && ((posY>359) && (posY<425)) && available) {
      mode = 10;
      available = false;
    }
    if (((posX > 72) && (posX<119)) && ((posY>399) && (posY<450)) && available) {
      //int i = int(random(generations.length));
      mode = 11;
      available = false;
    }
    if (((posX > 590) && (posX <width) && (posY < 30)&& (posY > 0)) || ((posX < 40) && (posX > 0) && (posY > 450) && (posY < height))) { 
      mode = 0;
      available = true;
    }
  }
  switch (mode) {
  case 0:
    canvas.background(bg);
    canvas.image(bg, 0, 0, width, height);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Our Solar System", 480, 30);
    break;
  case 1:
    canvas.image(sun, 0, 0, width, height);
    canvas.image(ex, 610, 13, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(sun1, 450, 60, 165, 120);
    canvas.image(sun2, 30, 250, 160, 120);
    canvas.image(sun3, 440, 350, 180, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Sun", width/2, 30);
    break;
  case 2:
    canvas.image(mercury, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(mercury1, 440, 60, 180, 120);
    canvas.image(mercury2, 20, 240, 160, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Mercury", width/2, 30);
    break;
  case 3:
    canvas.image(venus, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(venus1, 445, 60, 160, 120);
    canvas.image(venus2, 15, 280, 160, 120);
    canvas.image(venus3, 445, 350, 180, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Venus", width/2, 30);
    break;
  case 4:
    canvas.image (earth, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(earth1, 485, 60, 150, 120);
    canvas.image(earth2, 13, 260, 180, 120);
    canvas.image(earth3, 470, 350, 172, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Earth", width/2, 30);
    break;
  case 5:
    canvas.image(moon, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(moon1, 480, 55, 157, 120);
    canvas.image(moon2, 10, 260, 180, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Moon", width/2, 30);
    break;
  case 6:
    canvas.image(mars, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(mars1, 440, 60, 188, 120);
    canvas.image(mars2, 45, 260, 180, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Mars", width/2, 30);
    break;
  case 7:
    canvas.image(jupiter, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(jupiter1, 465, 60, 170, 120);
    canvas.image(jupiter2, 25, 260, 180, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Jupiter", width/2, 30);
    break;
  case 8:
    canvas.image(saturn, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(saturn1, 425, 50, 210, 130);
    canvas.image(saturn3, 55, 310, 180, 110);
    canvas.image(saturn2, 405, 345, 225, 100);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Saturn", width/2, 30);
    break;
  case 9:
    canvas.image(uranus, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(uranus1, 455, 60, 180, 100);
    canvas.image(uranus2, 5, 320, 200, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Uranus", width/2, 30);
    break;
  case 10:
    canvas.image(neptune, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.image(neptune1, 465, 50, 170, 140);
    canvas.image(neptune2, 40, 350, 200, 120);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Neptune", width/2, 30);
    break;
  case 11:
    canvas.image(generation1, 0, 0, width, height);
    canvas.image(ex, 610, 15, 20, 20);
    canvas.image(ex2, 20, 447, 20, 20);
    canvas.textFont(title, 20);
    canvas.textAlign(CENTER);
    canvas.text("Generated Solar System", width/2, 30);
    break;
  }
  if ((curX != prevX) && (curY != prevY) && (!isPlaying)) {
    volume+=.1;
  } else volume-=.1;
  volume = constrain(volume, 0., 1.);
  println(volume);
  rocketship.amp(volume);
}

IR tracking page –

Mat getPerspectiveTransformation(ArrayList<PVector> inputPoints, int w, int h) {
  Point[] canonicalPoints = new Point[4];
  canonicalPoints[0] = new Point(w, 0);
  canonicalPoints[1] = new Point(0, 0);
  canonicalPoints[2] = new Point(0, h);
  canonicalPoints[3] = new Point(w, h);

  MatOfPoint2f canonicalMarker = new MatOfPoint2f();
  canonicalMarker.fromArray(canonicalPoints);

  Point[] points = new Point[4];
  for (int i = 0; i < 4; i++) {
    points[i] = new Point(inputPoints.get(i).x, inputPoints.get(i).y);
  }
  MatOfPoint2f marker = new MatOfPoint2f(points);
  return Imgproc.getPerspectiveTransform(marker, canonicalMarker);
}

Mat warpPerspective(ArrayList<PVector> inputPoints, int w, int h) {
  Mat transform = getPerspectiveTransformation(inputPoints, w, h);
  Mat unWarpedMarker = new Mat(w, h, CvType.CV_8UC1);    
  Imgproc.warpPerspective(opencv.getGray(), unWarpedMarker, transform, new Size(w, h));
  return unWarpedMarker;
}

void loadData() {
  json = loadJSONObject("coords.json");
  JSONArray coordsData = json.getJSONArray("coords");
  for (int i = 0; i < coordsData.size(); i++) {
    // Get each object in the array
    JSONObject coord = coordsData.getJSONObject(i); 
    // Get a position object
    JSONObject position = coord.getJSONObject("position");
    // Get x,y from position
    int x = position.getInt("x");
    int y = position.getInt("y");
    vectors.add(new PVector(x, y, 0.0));
  }
}

 

And here is the code to make the computer generated solar system:

Mainpage –

import peasy.*;
Planet sun;

PeasyCam cam;

PImage bg;
//PImage starfield;

PImage sunTexture;
PImage[] textures = new PImage[9];

void setup() {
  //size(600, 600, P3D);
  //size (1280, 720, P3D); //starfield 2
  size (1280, 720, P3D); //starfield 1
  //fullScreen(P3D);
  
  bg = loadImage("starfield1.jpg");
  //starfield = loadImage("starfield1.jpg");

  
  sunTexture = loadImage("sun.jpg");
  textures[0] = loadImage("earth.jpg");
  textures[1] = loadImage("jupiter.jpg");
  textures[2] = loadImage("saturn.jpg");
  textures[3] = loadImage("mercury.jpg");
  textures[4] = loadImage("venus.jpg");
  textures[5] = loadImage("mars.jpg");
  textures[6] = loadImage("uranus.jpg");
  textures[7] = loadImage("neptune.jpg");
  textures[8] = loadImage("moon.jpg");
  
  cam = new PeasyCam(this, 500);
  sun = new Planet(50, 0, 0, sunTexture);
  sun.spawnMoons(5, 1);
}

void draw() {
  background(bg);
  //background(0);
  
  
  ambientLight(255,255,255);
  pointLight(255, 255, 255, 0, 0, 0);
  sun.show();
  sun.orbit();
  //image(starfield, 0, 0, width, height);


}

void keyPressed(){
  if(key==CODED&&keyCode==DOWN){
    setup();
  }
}

Planets class page –

class Planet {
  float radius;
  float distance;
  Planet[] planets;
  float angle;
  float orbitspeed;
  PVector v;

  PShape globe;

  Planet(float r, float d, float o, PImage img) {

    v = PVector.random3D();

    radius = r;
    distance = d;
    v.mult(distance);
    angle = random(TWO_PI);
    orbitspeed = o;

    noStroke();
    noFill();
    globe = createShape(SPHERE, radius);
    globe.setTexture(img);
  }

  void orbit() {
    angle = angle + orbitspeed;
    if (planets != null) {
      for (int i = 0; i < planets.length; i++) {
        planets[i].orbit();
      }
    }
  }

  void spawnMoons(int total, int level) {
    planets = new Planet[total];
    for (int i = 0; i < planets.length; i++) {
      float r = radius/(level*2.5);
      float d = random((radius*2), (radius+(r*1.5))*2.75);
      float o = random(-0.05, 0.05);
      int index = int(random(0,textures.length));
      planets[i] = new Planet(r, d, o, textures[index]);
      if (level < 2) {
        int num = int(random(0,3));
        planets[i].spawnMoons(num, level+1);
      }
    }
  }

  void show() {
    pushMatrix();
    noStroke();
    PVector v2 = new PVector(1, 0, 1);
    PVector p = v.cross(v2);
    rotate(angle, p.x, p.y, p.z);
    stroke(255);
    //line(0, 0, 0, v.x, v.y, v.z);
    //line(0, 0, 0, p.x, p.y, p.z);
    translate(v.x, v.y, v.z);
    noStroke();
    fill(255);
    shape(globe);
    //sphere(radius);
    //ellipse(0, 0, radius*2, radius*2);
    if (planets != null) {
      for (int i = 0; i < planets.length; i++) {
        planets[i].show();
      }
    }
    popMatrix();
  }
}

Here’s to the 567 lines of code that worked to create my final project!

To improve my project:

  • Move the spacestation further away from the “x” at the bottom left corner so that people don’t accidentally go over it.
  • Rather than have video animations, have images that would just loop to look like animations, perhaps this would take up less RAM and wouldn’t lead to my computer crashing.
  • Glue the rocketship to the battery so people won’t be able to pull it apart and somehow stick the wires so that it wouldn’t loosen and pop out.

The Magical Cloud

For our final project, our team (Vitoria and Aya), decided to explore the idea of an artificial cloud that worked as a sort of weather gadget. By searching for the weather, you would get a visual representation of what the weather was.

Initial sketch for idea:

From the very beginning, it was clear that obtaining in real time weather updates that would stimulate several functions and actions would be very difficult and perhaps too time consuming to try and figure out. We decided early on that we would choose certain cities to program their weather into the code manually. This also led to a design choice for the digital aspect of the piece. We decided to create a map on screen where you could press on a city to trigger the response from the physical cloud.

We first started by designing and printing rain and snow shapes out of acrylic using a laser cutter. We tested the motion of the rain by attaching them to servos and coding their movements. Originally the shapes were attached to long sticks that were attached to the servos. However, we did not like how the movements were jerky instead of smooth. Also, the long sticks interfered with a lot of the design decisions and made the building process of the cloud more difficult. We then decided to create a yo-yo based system that would be attached to the servo and the shape and roll the shape up and down.

After managing the servos, we moved on to looking at the lights we wanted to use. We started with RGB LEDs, however, we realized that it was pointless to try and program a large number of LEDs. Instead, we decided to use neopixel strips based off of Aaron’s advice to us. This was easier in terms of coding, power, and design.

Then came the building process, which was being played with the whole time. We used a clothing rack as the foundational support. We then added shelves to it to support the Arduino board as well as the servos. The neopixel strips were draped over the steel rods. We used lanterns as the base for the cloud and used white glue to stick cotton on to them. They worked well because they already had the required structure so that the design wouldn’t interfere with the servos and lights.

After building the cloud and working the physical parts, we began to work on the digital aspect. This went relatively smoothly using processing. This definetly took the least time.

Hooking up processing to Arduino created a few challenges. Trying to run everything at once created a lot of lagging. However, we added an external power source and that helped.

Lastly, we added on-screen feedback and audio (because of the feedback we received in class).

Disclaimer: The coding process involved a lot of help from people around the lab, Aaron, and the internet.

Processing Code: 

import java.awt.Rectangle;
import ddf.minim.*;
import processing.serial.*;
Serial myPort; // Create object from Serial class

Minim minim;
AudioPlayer rain;
AudioPlayer lightrain;
AudioPlayer heavyrain;
AudioPlayer storm;
AudioPlayer snow;
AudioPlayer birds;

PImage img;
PFont font;
boolean newYorkPressed = false;
boolean shanghaiPressed = false;
boolean abuDhabiPressed = false;
boolean tokyoPressed = false;
boolean parisPressed = false;
boolean florencePressed = false;
boolean madridPressed = false;
boolean baPressed = false;
boolean havanaPressed = false;
boolean icePressed = false;
boolean capeTownPressed = false;
boolean rainSound = false;
boolean birdSound = false;
boolean heavyRain = false;
boolean lightRain = false;
boolean stormSound = false;
boolean snowySound = false;

Rectangle newyork;
Rectangle shanghai;
Rectangle abudhabi;
Rectangle tokyo;
Rectangle paris;
Rectangle florence;
Rectangle madrid;
Rectangle ba;
Rectangle havana;
Rectangle ice;
Rectangle capetown;

void setup() {
String portName = Serial.list()[3]; //change the 0 to a 1 or 2 etc. to match your port
myPort = new Serial(this, portName, 9600);
fullScreen();
img = loadImage(“hey.jpeg”);
font = loadFont(“Futura-CondensedExtraBold-48.vlw”);
newyork= new Rectangle(371, 313, 160, 50);
shanghai = new Rectangle(1090, 370, 160, 40);
abudhabi = new Rectangle(890, 412, 175, 50);
tokyo =new Rectangle(1232, 344, 110, 40);
paris = new Rectangle(695, 255, 110, 40);
florence = new Rectangle(725, 320, 130, 50);
madrid = new Rectangle(540, 308, 110, 45);
ba = new Rectangle(420, 723, 215, 50);
havana = new Rectangle(360, 420, 160, 40);
ice = new Rectangle(593, 153, 150, 50);
capetown = new Rectangle(755, 730, 290, 50);
minim = new Minim(this);
birds = minim.loadFile(“birds.mp3”);
rain = minim.loadFile(“rain.mp3”);
lightrain = minim.loadFile(“lightrain.mp3”);
heavyrain = minim.loadFile(“rain2.mp3”);
storm =minim.loadFile(“storm.mp3”);
snow = minim.loadFile(“snow.mp3”);
}

void draw() {
//image(img, 0, 0, width, height);
img.resize(width, height);
background(img);
textFont(font, 32);
fill(#50F2FF);
text(“New York”, 370, 340); //437,330
text(“Shanghai”, 1100, 400); //1167,391
text(“Abu Dhabi”, 895, 440); // 968,429
text(“Tokyo”, 1240, 375); //1280,362
text(“Paris”, 698, 286); //735,276
text(“Florence”, 732, 345); // 791,335
text(“Madrid”, 550, 335); // 603,325
text(“Buenos Aires”, 427, 750); //523,742
text(“Havana”, 365, 446); // 420, 438
text(“Reykjavik”, 600, 190); // 666,181
text(“Cape Town”, 765, 760); // 840,751

textFont(font, 45);
fill(0);

if (newYorkPressed==true && birdSound == true) {
text(“New York is currently: Sunny”, 450, 850);
pause();
birds.play();
}
if (shanghaiPressed == true && rainSound ==true) {
text(“Shanghai is currently: Rainy “, 450, 850);
pause();
rain.play();
}

if ( abuDhabiPressed == true && birdSound == true) {
text(“Abu Dhabi is currently: Sunny “, 450, 850);
pause();
birds.play();
}
if (tokyoPressed==true && heavyRain == true)
{
text(“Tokyo is experiencing: Heavy Rain Fall “, 450, 850);
pause();
heavyrain.play();
}
if (parisPressed == true&& birdSound == true) {

text(“Paris is currently: Sunny”, 450, 850);
pause();
birds.play();
}

if ( florencePressed == true && lightRain == true) {
text(“Florence is experiencing: Sunshine and Rain “, 450, 850);
pause();
lightrain.play();
}

if (madridPressed==true && birdSound == true)
{
text(“Madrid is currently: Sunny “, 450, 850);
pause();
birds.play();
}
if (baPressed == true && stormSound == true) {

text(“Buenos Aires is experiencing: Storms “, 450, 850);
pause();
storm.play();
}

if (havanaPressed == true && lightRain == true) {
text(“Havana is experiencing: Sunshine and Rain “, 450, 850);
pause();
lightrain.play();
}
if (icePressed==true && snowySound == true)
{
text(“Reykjavik is currently: Snowing “, 450, 850);
pause();
snow.play();
}
if (capeTownPressed == true && birdSound == true) {
text(“Cape Town is currently: Sunny “, 450, 850);
pause();
birds.play();
}
}

void mousePressed() {
textFont(font, 32);
fill(#50F2FF);
if (newyork.contains(mouseX, mouseY)) {
myPort.write(3);
newYorkPressed = true;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;

rainSound = false;
birdSound = true;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = false;
}

if (shanghai.contains(mouseX, mouseY)) {
myPort.write(0);
newYorkPressed = false;
shanghaiPressed = true;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;

rainSound = true;
birdSound = false;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = false;
}

if (abudhabi.contains(mouseX, mouseY)) {
myPort.write(3);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = true;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;

rainSound = false;
birdSound = true;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = false;
}

if (tokyo.contains(mouseX, mouseY)) {
myPort.write(1);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = true;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;

rainSound = false;
birdSound = false;
heavyRain = true;
lightRain = false;
stormSound = false;
snowySound = false;
}

if (paris.contains(mouseX, mouseY)) {
myPort.write(3);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = true;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;

rainSound = false;
birdSound = true;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = false;
}
if (florence.contains(mouseX, mouseY)) {
myPort.write(5);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = true;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;
rainSound = false;
birdSound = false;
heavyRain = false;
lightRain = true;
stormSound = false;
snowySound = false;
}
if (madrid.contains(mouseX, mouseY)) {
myPort.write(3);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = true;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;

rainSound = false;
birdSound = true;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = false;
}
if (ba.contains(mouseX, mouseY)) {
myPort.write(0);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = true;
havanaPressed = false;
icePressed = false;
capeTownPressed = false;
rainSound = false;
birdSound = false;
heavyRain = false;
lightRain = false;
stormSound = true;
snowySound = false;
}
if (havana.contains(mouseX, mouseY)) {
myPort.write(5);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = true;
icePressed = false;
capeTownPressed = false;
rainSound = false;
birdSound = false;
heavyRain = false;
lightRain = true;
stormSound = false;
snowySound = false;
}
if (ice.contains(mouseX, mouseY)) {
myPort.write(2);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = true;
capeTownPressed = false;
rainSound = false;
birdSound = false;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = true;
;
}
if (capetown.contains(mouseX, mouseY)) {
myPort.write(3);
newYorkPressed = false;
shanghaiPressed = false;
abuDhabiPressed = false;
tokyoPressed = false;
parisPressed = false;
florencePressed = false;
madridPressed = false;
baPressed = false;
havanaPressed = false;
icePressed = false;
capeTownPressed = true;
rainSound = false;
birdSound = true;
heavyRain = false;
lightRain = false;
stormSound = false;
snowySound = false;
}
}

void pause() {
rain.pause();
lightrain.pause();
heavyrain.pause();
storm.pause();
snow.pause();
birds.pause();
}

Arduino Code: 

int mode = 3;

#include <Servo.h>

Servo myservo;
Servo myservo2;

int pos = 0;

#include <Adafruit_NeoPixel.h>
#ifdef __AVR__
#include <avr/power.h>
#endif
#define PIN 11
#define NUMPIXELS 70

Adafruit_NeoPixel pixels = Adafruit_NeoPixel(NUMPIXELS, PIN, NEO_GRB + NEO_KHZ800);

void setup() {
Serial.begin(9600);
myservo.attach(9);
myservo2.attach(3);
pixels.begin();

}

void flash(){
for(int i=0;i<NUMPIXELS;i++){
pixels.setPixelColor(i, pixels.Color(2,140,242));

pixels.show();

delay(100);
}
for (pos = 0; pos <= 180; pos += 1) {
// in steps of 1 degree
myservo.write(pos);
delay(20);
}
for (pos = 180; pos >= 0; pos -= 1) {
myservo.write(pos);
delay(20);
}

// for(int i=0;i<NUMPIXELS;i++){
// pixels.setPixelColor(i, pixels.Color(2,140,242));
//
// pixels.show();
// }
//
// for (pos = 0; pos <= 180; pos += 1) {
// // in steps of 1 degree
// myservo.write(pos);
// delay(15);
// }
// for (pos = 180; pos >= 0; pos -= 1) {
// myservo.write(pos);
// delay(15);
// }
}

void heavyrain(){

for (pos = 0; pos <= 180; pos += 1) {
// in steps of 1 degree
myservo.write(pos);
delay(15);
}
for (pos = 180; pos >= 0; pos -= 1) {
myservo.write(pos);
delay(15);
}

}
// light flicker in
void light(){

for(int i=0;i<NUMPIXELS;i++){
pixels.setPixelColor(i, pixels.Color(255,255,255));

pixels.show();

delay(500);

}
}
// motor spin at slow speed
void rain(){
for(int i=0;i<NUMPIXELS;i++){
pixels.setPixelColor(i, pixels.Color(2,140,242));

pixels.show();

delay(100);
}
for (pos = 0; pos <= 180; pos += 1) {
// in steps of 1 degree
myservo.write(pos);
delay(20);
}
for (pos = 180; pos >= 0; pos -= 1) {
myservo.write(pos);
delay(20);
}
}
// motors spin at medium speed
void snow(){
for(int i=0;i<NUMPIXELS;i++){
pixels.setPixelColor(i, pixels.Color(172,217,250));

pixels.show();

delay(100);
for (pos = 0; pos <= 180; pos += 1) {
// in steps of 1 degree
myservo2.write(pos);
delay(15);
}
for (pos = 180; pos >= 0; pos -= 1) {
myservo2.write(pos);
delay(15);
}
}
}
void sunrain(){

for(int i=0;i<NUMPIXELS;i++){
pixels.setPixelColor(i, pixels.Color(152,211,255));

pixels.show();

delay(100);

for (pos = 0; pos <= 180; pos += 1) {
// in steps of 1 degree
myservo.write(pos);
delay(20);
}
for (pos = 180; pos >= 0; pos -= 1) {
myservo.write(pos);
delay(20);
}
}
}
void stopLights() {
for (int i = 0; i < NUMPIXELS; i++) {
pixels.setPixelColor(i, pixels.Color(0, 0, 0));
}
pixels.show();
}

void sun(){

for(int i=0;i<NUMPIXELS;i++){
pixels.setPixelColor(i, pixels.Color(255,255,255));

pixels.show();

delay(100);

}
}

void loop() {
// put your main code here, to run repeatedly:

if (Serial.available())
{ // If data is available to read,
mode = Serial.read(); // read it and store it in val
}

switch (mode) {
case 0:
stopLights();
myservo.detach();
myservo2.detach();
myservo.attach(9);
rain();
break;
case 1:
stopLights();
myservo.detach();
myservo2.detach();
myservo.attach(9);
heavyrain();
break;
case 2:
stopLights();
myservo.detach();
myservo2.detach();
myservo2.attach(3);
snow();
break;
case 3:
myservo.detach();
myservo2.detach();
sun();
break;
case 4:
myservo.detach();
myservo2.detach();
flash();
break;
case 5:
myservo.detach();
myservo2.detach();
myservo.attach(9);
sunrain();
}

}

Final Product:

 

The entire process was a lot fo trial and error, there were a lot of issues that remained unsolved, however we received really great feedback at the showcase and we are still happy to end the course with this project.

Final Project: Intro to IM Show (Adham & Neyva)

So, here goes a shot at the description for Adham and I’s last project!

Firstly, the project did end up looking like the original idea. There was a controller which activated physical things on a board to move, which in turn caused the main character to move. The visuals even turned out cooler than I thought, as originally I wanted to paint the board, but Adham thought that projection mapping objects onto the wooden pieces on the board would look cooler. Which it did.

The project consisted of a 46in x 36in wooden board with 2 stepper motors, one servo, and 2 dc motors mounted onto it. Along with various wooden platforms for the character to walk on. In addition to this, there was a physical controller with 5 buttons which controlled each of these motors, and activated the animations for the board. The end goal is to get from the starting point, to the door in the bottom left-hand screen, and this is only possible if the user presses the buttons in the right sequence. Pressing the buttons will always cause the physical component on the board to move, but it only triggers the animation when they are pressed with correlation to where the heroine is on the board.

Here are some videos of people using our project:

I am quite proud of the code, as the animation took around 450 lines to write, making it the most code I have ever written for a project. Here is the code:

//compatible with madmapper
import codeanticode.syphon.*; 
import processing.serial.*;
Serial port;
SyphonServer server;


//right walk images
PImage[] walkRight = new PImage[10];
PImage img0;
PImage img1;
PImage img2;
PImage img3;
PImage img4;
PImage img5;
PImage img6;
PImage img7;
PImage img8;
PImage img9;

//left walk images
PImage[] walkLeft = new PImage[10];
PImage img_0;
PImage img_1;
PImage img_2;
PImage img_3;
PImage img_4;
PImage img_5;
PImage img_6;
PImage img_7;
PImage img_8;
PImage img_9;

//fire animation
PImage[] fireAni = new PImage[32];
PImage f0;
PImage f1;
PImage f2;
PImage f3;
PImage f4;
PImage f5;
PImage f6;
PImage f7;
PImage f8;
PImage f9;
PImage f10;
PImage f11;
PImage f12;
PImage f13;
PImage f14;
PImage f15;
PImage f16;
PImage f17;
PImage f18;
PImage f19;
PImage f20;
PImage f21;
PImage f22;
PImage f23;
PImage f24;
PImage f25;
PImage f26;
PImage f27;
PImage f28;
PImage f29;
PImage f30;
PImage f31;

//background designs
PImage plat;
PImage wall;
PImage ground;
PImage door;
PImage win;

//triggers
boolean standing = true;
boolean triggerWaterMill = false;
boolean triggerTrapDoor = false;
boolean triggerLift = false;
boolean triggerSpike1 = false;
boolean triggerSpike2 = false;

//count numbers
int animIndex = 0;
int animIndexFire = 0;
int animIndexFire0 = 0;

//coordinates of figure
int locX=80;
int locY=60;


int[] permissions = {0, 0, 0, 0, 0};
int timer = 120;

void setup() {
  size (920, 720, P3D);
  frameRate(10);

  server = new SyphonServer(this, "Processing Syphon");
  printArray(Serial.list());
  String portName = Serial.list()[5];
  port = new Serial(this, portName, 9600);




  //load right walk images
  walkRight[0] = loadImage("figure0.png");
  walkRight[1] = loadImage("figure1.png");
  walkRight[2] = loadImage("figure2.png");
  walkRight[3] = loadImage("figure3.png");
  walkRight[4] = loadImage("figure4.png");
  walkRight[5] = loadImage("figure5.png");
  walkRight[6] = loadImage("figure6.png");
  walkRight[7] = loadImage("figure7.png");
  walkRight[8] = loadImage("figure8.png");
  walkRight[9] = loadImage("figure9.png");


  //load left walk images
  walkLeft[0] = loadImage("figure_0.png");
  walkLeft[1] = loadImage("figure_1.png");
  walkLeft[2] = loadImage("figure_2.png");
  walkLeft[3] = loadImage("figure_3.png");
  walkLeft[4] = loadImage("figure_4.png");
  walkLeft[5] = loadImage("figure_5.png");
  walkLeft[6] = loadImage("figure_6.png");
  walkLeft[7] = loadImage("figure_7.png");
  walkLeft[8] = loadImage("figure_8.png");
  walkLeft[9] = loadImage("figure_9.png");

  //load fire images
  fireAni[0] = loadImage("fire0.png");
  fireAni[1] = loadImage("fire1.png");
  fireAni[2] = loadImage("fire2.png");
  fireAni[3] = loadImage("fire3.png");
  fireAni[4] = loadImage("fire4.png");
  fireAni[5] = loadImage("fire5.png");
  fireAni[6] = loadImage("fire6.png");
  fireAni[7] = loadImage("fire7.png");
  fireAni[8] = loadImage("fire8.png");
  fireAni[9] = loadImage("fire9.png");
  fireAni[10] = loadImage("fire10.png");
  fireAni[11] = loadImage("fire11.png");
  fireAni[12] = loadImage("fire12.png");
  fireAni[13] = loadImage("fire13.png");
  fireAni[14] = loadImage("fire14.png");
  fireAni[15] = loadImage("fire15.png");
  fireAni[16] = loadImage("fire16.png");
  fireAni[17] = loadImage("fire17.png");
  fireAni[18] = loadImage("fire18.png");
  fireAni[19] = loadImage("fire19.png");
  fireAni[20] = loadImage("fire20.png");
  fireAni[21] = loadImage("fire21.png");
  fireAni[22] = loadImage("fire22.png");
  fireAni[23] = loadImage("fire23.png");
  fireAni[24] = loadImage("fire24.png");
  fireAni[25] = loadImage("fire25.png");
  fireAni[26] = loadImage("fire26.png");
  fireAni[27] = loadImage("fire27.png");
  fireAni[28] = loadImage("fire28.png");
  fireAni[29] = loadImage("fire29.png");
  fireAni[30] = loadImage("fire30.png");
  fireAni[31] = loadImage("fire31.png");

  //load background stuff
  plat = loadImage("platform.png");
  wall = loadImage("block.jpg");
  ground = loadImage("wall2.jpg");
  door =loadImage("door.png");
  win =loadImage("win.png");
}

void draw() {
  //resets the background to white each frame


  background(255);
  if (millis()%1000 == 0){
    timer--;
  }
  text(str(timer),10,10);
  
  if (standing && !triggerWaterMill) {
    //loads the new image from the array
    PImage frameStand = walkRight[0];
    //draws this new image from the center at coordinate x,y
    imageMode(CENTER);
    image(frameStand, 80, 60, 40, 80);
  }

  //platform 1
  imageMode(CENTER);
  image(plat, 78, 115, 61, 28);
  //platform 2
  image(plat, 345, 175, 70, 28);
  //platform 3
  image(plat, 500, 209, 190, 28);
  //wall 1
  image(wall, 647, 112, 11, 182);
  //platform 4
  image(plat, 608, 414, 212, 28);
  //platform5
  image(plat, 763, 317, 70, 28);
  //platform6
  image(plat, 763, 510, 70, 28);
  //platform7
  image(plat, 551, 551, 63, 28);
  //platform 8
  image(plat, 438, 414, 178, 28);
  // //platform9
  image(plat, 79, 608, 157, 28);

  //door
  image(door, 71, 540, 90, 110);

  pushMatrix();
  translate(282, 575);
  //rotate(cos(angle))
  rotate(-PI/14);
  image(plat, 0, 0, 216, 30);
  popMatrix();

  //loads the new image from the array drawing FIRE AAYYYYY
  PImage frame0 = fireAni[animIndexFire];
  PImage frame1 = fireAni[animIndexFire0];
  //draws this new image from the center at coordinate x,y FIRE!!!!
  imageMode(CENTER);
  //ground 1
  image(ground, 75, 675, 146, 90);
  //fire 1
  image(frame1, 174, 655, 98, 200);
  //ground 2
  image(ground, 287, 675, 166, 140);
  //fire 2
  image(frame0, 394, 655, 100, 200);
  //fire 3
  image(frame1, 447, 655, 100, 200);
  //fire4
  image(frame0, 495, 655, 100, 200);
  //fire5
  image(frame1, 610, 655, 100, 200);
  //fire 6
  image(frame0, 660, 655, 100, 200);
  //fire 7
  image(frame1, 714, 655, 100, 200);
  //fire 8
  image(frame0, 820, 655, 100, 200);
  //fire 9
  image(frame1, 868, 655, 100, 200);
  //ground 3
  image(ground, 555, 675, 60, 200);
  //ground 4
  image(ground, 765, 705, 60, 200);
  image(ground, 765, 630, 60, 200);
  //ground 5
  image(ground, 900, 705, 60, 200);
  image(ground, 900, 650, 60, 200);




  //loops the animation 
  animIndexFire = (animIndexFire + 1) % fireAni.length;
  animIndexFire0 = (animIndexFire0 + 2) % fireAni.length;


  noFill();
  //draw the ladder
  strokeWeight(6);
  line(730, 100, 792, 100);
  line(730, 130, 792, 130);
  line(730, 160, 792, 160);
  line(730, 190, 792, 190);
  line(730, 220, 792, 220);

  //trampoline
  fill(255, 0, 0);
  strokeWeight(3);
  ellipse(605, 350, 50, 20);
  line(600, 360, 625, 370);
  line(625, 370, 590, 375);
  line(590, 375, 620, 385);
  strokeWeight(6);
  line(580, 385, 635, 385);



  if (permissions[0]>0 && !triggerWaterMill) {
    triggerWaterMill = true;
    standing = false;
  }
  if (permissions[1]>0) {
    if (triggerWaterMill&&!triggerTrapDoor) {
      triggerTrapDoor = true;
      standing = false;
    }
  }
  if (permissions[4]>0) {
    if (triggerTrapDoor&&!triggerLift) {
      triggerLift = true;
      standing = false;
    }
  }
  if (permissions[3]>0){
    if (triggerLift&&!triggerSpike1 ) {
        triggerSpike1 = true;
        standing = false;
      }
  }
  if (permissions[2]>0){
    if (triggerSpike1&&!triggerSpike2) {
        triggerSpike2 = true;
        standing = false;
      }
  }

  //if watermill
  //----------------------------FIRSTTRIGGER_WATERMILL---------------------------------
  if (triggerWaterMill&&!triggerTrapDoor) {
    if (locX <= 118 && locY== 60) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
    } else if (locX <= 135) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY -=3;
    } else if (locX <= 170) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY +=5;
    } else if (locX <= 260) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY ++;
    } else if (locX <= 280) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=2;
      locY +=5;
    } else if (locX <= 320) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY -=3;
    } else if (locX < 342) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
    } else {
      standing = true;
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      fuck = 1;
      //animIndex = (animIndex % (walkRight.length-1))+1;
    }
  }
  //if trapdoor
  //----------------------------SECONDTRIGGER_TRAPDOOR---------------------------------
  if (triggerTrapDoor&&!triggerLift) {
    if (locX <= 360) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
    } else if (locX <= 385) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY -=3;
    } else if (locX <= 414) {
      //PImage frame = walkRight[animIndex];
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=3;
      locY +=5;
    } 
    //double image for some reason
    else if (locX <= 605 && locY==152) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
    } else if (locY < 300 && locX<=620) {
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locY +=5;
    } else if (locY <= 315 && locX<=690) {
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY -=5;
    } else if (locY < 320 && locX<=725) {
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
      locY +=5;
    } else if (locX<=760 && locY<317) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=2;
    } else if (locX<=780 && locY>60) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locY -=5;
    } else {
      standing =true;
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      fuck = 2;

      //animIndex = (animIndex % (walkRight.length-1))+1;
    }
  }

  //if lift
  //----------------------------THIRDTRIGGER_LIFT---------------------------------
  if (triggerLift&&!triggerSpike1) {
    if (locX <= 860 && locY <=57) {
      PImage frame = walkRight[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX +=5;
    } else if (locX<=900 && locY<=416) {
      PImage frame = walkRight[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      locY +=10;
    } else if (locY>=416 && locX>=843) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX -=5;
      println(locX);
      println(locY);
    } else if (locY>=417 && locX>=802) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=4;
      locY+=4;
    } else if (locY>=461 && locX>=754) {
      PImage frame = walkLeft[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=4;
    } else {
      standing =true;
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
    }
  }
  //___________________________fourth Trigger_________________
  if (triggerSpike1 && !triggerSpike2) {
    if (locY>=386 && locX>=720) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=4;
      locY +=4;
    } else if (locY>=471 && locX>=582) {
      PImage frame = walkLeft[animIndex];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=4;
      locY -=4;
    } else if (locY>=457 && locX>=630) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=4;
      //println("here0");
      //println(locX);
      //println(locY);
    } else if (locY>=450 && locX>=610 ) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=4;
      locY+=2;
      //println("now0");
      //println(locX);
      //println(locY);
    } else if (locY<=490 && locX>=540) {
      PImage frame = walkLeft[0];
      image(frame, locX, locY, 40, 80);
      println("blah");
      println(locX);
      println(locY);
      locX-=4;
      locY+=2;
    } else {
      standing =true;
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //println("bl");
      //println(locX);
      //println(locY);
    }
  }

  //___________________________fourth Trigger_________________
  if (triggerSpike2 && triggerSpike1) {
    if (locY>=430 && locX>=480) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=5;
      locY -=5;
      //println("here1");
      //println(locX);
      //println(locY);
    }
    if (locY>=420 && locX>=475) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locY -=2;
      locX --;
      //println("here2");
      //println(locX);
      //println(locY);
    } else if (locY>=390 && locX>=378) {
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      //animIndex = (animIndex % (walkRight.length-1))+1;
      locX-=5;
      locY +=5;
      //println("b");
      //println(locX);
      //println(locY);
    } else if (locY>=390 && locX>=290)
    {
      standing =true;
      PImage frame = walkLeft[animIndex];
      imageMode(CENTER);
      animIndex = (animIndex % (walkRight.length-1))+1;
      image(frame, locX, locY, 40, 80);
      locX-=5;
      locY--;
      //println(locX);
      //println(locY);
    } else if (locY>=400 && locX>=220)
    {
      standing =true;
      PImage frame = walkLeft[animIndex];
      animIndex = (animIndex % (walkRight.length-1))+1;
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      locX-=5;
      locY+=2;
      println("now");
      println(locX);
      println(locY);
    } else if (locY>=400 && locX>=100)
    {
      standing =true;
      PImage frame = walkLeft[animIndex];
      animIndex = (animIndex % (walkRight.length-1))+1;
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      locX-=5;
      println("here");
      println(locX);
      println(locY);
    } else
    {
      standing =true;
      PImage frame = walkLeft[0];
      imageMode(CENTER);
      image(frame, locX, locY, 40, 80);
      image(win, 250, 410, 300, 300);
    }
  }
  server.sendScreen();
}

void keyPressed() {
  //watermill
  if (standing) {
    if (key == 'w' || key == 'W') {
      if (!triggerWaterMill) {
        triggerWaterMill = true;
        standing = false;
      }
    }
    //trapdoor
    if (key == 't' || key == 'T') {
      if (triggerWaterMill&&!triggerTrapDoor) {
        triggerTrapDoor = true;
        standing = false;
      }
    }
    //lift
    if (key == 'l' || key == 'L') {
      if (triggerTrapDoor&&!triggerLift) {
        triggerLift = true;
        standing = false;
      }
    }
    //spike1
    if (key == 's' || key == 'S') {
      if (triggerLift&&!triggerSpike1 ) {
        triggerSpike1 = true;
        standing = false;
      }
    }
    //spike2
    if (key == 'z' || key == 'Z') {
      if (triggerSpike1&&!triggerSpike2) {
        triggerSpike2 = true;
        standing = false;
      }
    }
  }
}

void serialEvent(Serial port) {
  String incoming = port.readStringUntil('\n');
  if (incoming!=null) {
    incoming = trim(incoming);
    if (incoming.length()>1) {
      String[] lis = split(incoming, ',');
      permissions[0] = int(lis[0]);
      permissions[1] = int(lis[1]);
      permissions[2] = int(lis[2]);
      permissions[3] = int(lis[3]);
      permissions[4] = int(lis[4]);
      println(permissions);
    }
    port.write('0');
  }
}

 

Now for a description of some of our difficulties on this proj, as there were some complications.

  1. Adham originally planned trying to make our controller wirelessly work with the board instead of using long wires. This was very time consuming, and in the end the idea was scrapped because the wireless communication was simply not working the best.
  2. This was the first time I did this type of animation on processing. Most times, I simply use photoshop for animations by making gifs, but it turns out that processing and gifs do not play nicely. Therefore, it was time to use processing for the animation.
  3. There were a TON of random bugs. Adham had trouble causing certain motors to move on the board, even after we attached wires to the proj. (In the end, one of the stepper motors just would not move.) And our first stepper motor would move, but super slowly (Not to mention that our laser cut wheel for it would occasionally fly of when the stepper spun…oops.)  Then there was also the fact that the slide potentiometer that was originally in the controller was on a logarithmic scale, making it hard to map. Another issue was the way I coded the animation, sometimes my character would slightly glitch as coordinate commands would partially overlap.  And although there were more, I can’t remember them all.
  4. After lots of help and patience from Aaron, I was able to projection map the animation onto the board. This was actually easier after I found and used the image of the girl above, instead of a stick figure drawn in processing which was originally used for testing. This was due to the fact that the image was simply drawn from the CENTER coordinate from it, while the stick figure needed to move various lines in relation to each other. In the end though, I have a better idea of how to projection map with madmapper, and I look forward to working with it more in the future.

Overall, I would say the project was a success. Although I do think we should have toned down the project a bit, I am happy with the results.

Journey of The Ohm Room

Starting Point

This project initially stemmed from my motivation to do something regarding mental health on this campus. I wanted people to become more aware of how they were feeling instead of them dismissing it, as people often do when trying to cater to their busy schedules. But I also wanted to make them realise that the feeling will pass, and it is not a permanent stage in their life — regardless of how challenging feelings can make people feel like distress is a permanent aspect of their lives.

This idea manifested itself in multiple ways. Initially, I imagined a user typing in a maximum of four feelings. A picture of a body would then be displayed on the user’s screen. The body would be divided into four parts: head, chest, gut, and legs. The user would then match the feeling to the appropriate body part, specifically where the emotion is felt. Next, the user would enter a room where their body parts would be constructed using the feelings they entered. After 30 seconds, the feelings would disperse, to add emphasis to the concept that the feelings will pass.

Coding

The challenging aspect of this idea was to code the body divided into four parts and to mix and match the feelings with the user’s body part. Hence, I decided to go with something simpler. With Aaron’s assistance, I narrowed down the idea to simply have the user enter a maximum of four feelings. Next, the user would enter a meditation space where they could see their body reflecting their emotions. After 30 seconds, the feelings would disperse and dissolve into the background. There would also be meditative music playing in the background.

The most challenging aspect of creating this project was the coding, it was definitely outside of my comfort zone. Especially the section where I wanted to manipulate the video letters to only focus on the individual, and then have the letters disperse. I tried to research and do as much as I could before I asked others for help. One resource that really helped me a lot (apart from Aaron) was Daniel Shiffman’s videos. When I was stuck on how to disperse the letters, his tutorials on the particle system was extremely useful. Before this class, I had never spent 5 hours stuck on a small piece of code, it was a really frustrating experience at times. But the joy I felt when fixing such issues was tremendous. I think this project is literally the definition of an emotional roller-coaster. I have never felt such an intense frustration and intense joy simultaneously. Again, as much as I tried to work on the coding myself, I realised I was quite dependent on other people’s knowledge. I didn’t like this at first, because I felt that I needed to work harder or take more ownership. But I came to accept that a certain extent of dependence is not a bad thing (a shout out to Unix lab, my cool coding friends, and of course, AARON!!). 

User Testing

After the user testing in class, Aaron and my peers helped me realise that instead of setting strict rules for the user (ex. only input a maximum of four feelings), it was better to keep the instructions more open to interpretation so users had more freedom with the project. The instructions were also quite unclear, which was evident from user testers that kept asking me questions to clarify certain things. Hence, I decided to make the instructions more specific, and also add some background information to what the project is so users had a general understanding of why they were interacting with the work. I was sceptical of adding context because I didn’t want to influence the user’s preconceptions of my work, but some sort of general context was also necessary for a better understanding and interaction.

Working with Space

I assumed that setting up space would be the easiest part of the journey — I was wrong. It wasn’t the hardest aspect, but it was definitely difficult and very different from my expectations. I thought the user would have a larger space to meditate, but space was cut off by the curtains.  But I also liked the curtains, as that created a clear distinction between the meditation space and the user input space. I also realised that a large space for the user wasn’t necessary. In fact, the small space made the environment cosier, and thus more comfortable. Instead of using a blue screen to manipulate the background, Aaron mentioned that why not just leave the video as it is (which was that the word surrounded the person’s body, negative space in the person’s body). I was a little taken aback at first but I realised that it did not take away from my concept, and also limited the work I had to do, so I agreed. It actually turned out to be a great idea. The project also looked nicer when there were more letters dispersed because I didn’t manipulate the background.

I had to make quite a few changes to the lighting. At first, I experimented with the lamp being on top of the panel. However, there wasn’t enough light reaching the camera and it cut off a majority of the user’s reflection. Aaron suggested that I put the lamp in front of the user, but the projection of letters was more focused on the light source. Aaron then turned the theatre light on and that worked perfectly. The only issue was that I had to leave the curtain slightly open, so the meditation space may have appeared less private, but there was enough space covered by the curtains to still distinguish the meditation space from the user input space.

IM Show! 

Two things that I really wanted to do before the show got started was to have a timer set up that would automatically start dispersing the letters (instead of me having to press a key to disperse the letters) and to also have a key that would automatically reset the program when I press it. Due to the lack of time, I wasn’t able to do it. However, it didn’t cause as much harm as I thought it would.

User interaction was a really awesome experience. I didn’t realise how much of an impact user interaction would be on my perception of my project. A two hour interaction honestly had more of an impact on how I perceived my project rather than me working two weeks on constructing it. The responses I received were very positive and optimistic. I was really surprised when people told me it was their favourite project or was genuinely helpful. The responses definitely helped me value my project more, and I also met many interesting people that gave me great feedback.

Craig offered a great idea of having the letters disperse once the user settles down and stays still, which portrays a more organic movement of the emotions. Ume recommended having some timers or more instructions in place so I wouldn’t have to spend so much time transitioning people to places. Some users wanted the experience to be longer, which I also think would have been better. Due to the nature of the show, however, I had to cut down the time.

It was also interesting to see how people interacted with the project. I saw two people’s head peeking out of the curtains as they were laying down on the floor completely. I never expected someone interacting with my project with that specific body posture — and I wonder what I could have done with my project to be more inclusive of that body posture.

Next time, I would also want to ensure and emphasise that people’s privacy would be respected somewhere in the instructions, as I felt that sometimes people may have been worried to input their genuine emotions because there wasn’t really an emphasis on the respect of privacy. Also, I would work on ensuring a smoother transition from the user input station to the meditation room, where I wouldn’t have to talk much to transition the user. Lastly, I would also want to design the room in such a way where the user knows to go to the user input station before entering the meditation room, as I saw that confusion of “where do I go first?” with many of the users.

Overall, this has been a really fulfilling project, which I think it largely because of my motive. I have always wanted to create something that is an amalgamation of the arts and sciences, and I did that while also having a subject that I am quite passionate about, which is mental health. I didn’t expect this project to be as fulfilling and meaningful as it turned out to be. I also definitely did not realise how powerful and amazing it was to have user interaction and actually converse with the users after. This has genuinely been an awesome experience.

 

 

 

 

 

Final Project

At the moment, my project is a clear case of “it worked 5 minutes ago”, so this is just the first is a series of updates. However, it was working at that time quite well, but that time was less than 10 minutes, which only gave enough time for me to test it. User testing will be coming tomorrow. For its current state though, it works!

The remaining bug is most likely power based, and therefore connection based, so I won’t solve it until I solder the connections on the back.

I have also tested the flippers support the weight of the ball, and that the ball correctly controls the advancement of control of the flippers along the path.

 

For my own interaction testing, the best results were generated with full bod moments at a significant distance, this should be easily fine tuned when the game is in its final location.

Final Project – User Testing

The responses from the users were mainly

Ooh that’s cool!

and

Oh wait, what did I do?

due to nice bubble columns that came up on the screen when the keys are pressed and the nice bubble that didn’t stop coming when the keys weren’t pressed.

The first user testing was done before things were working quite the way I wanted it to so I made some changes before the second user testing regardless of the feedback from the first user testing, The changes I made before the second user testing were:

  1. To make the display for the notes distinct from one another,
  • Spaced out the x position of each bubble columns.
  • Assigned each notes to a colour.
  1. To make the display more clearly visible
  • Put a white piece of cloth over the piano.

After the second user testing, I took the cloth down and just put it over the piano to see which set up looked better, then settled with not hanging the cloth on the wall as the user testing showed that the user(s) liked that better. Before the second user testing, I wanted to make the colours of each columns to be different and stay different but had failed to do so. When I brought it up with the user after the user testing, he said he likes how the colours of all the columns change to the colour of the new pressed key but I am unsure as to how much the user feedback should count towards the project.

Hand Music Mashup – User Testing

For context, here’s my 75% complete project:

and here’s my 90% complete project:

So as I was developing my project, I was constantly trying to user test my own project before I user test it on others as I would know best how to break my own project which resulted in many enhancements in the UI, design and how it functions overall:

    • I realized early on that there would be no ‘buttons’ to be clicked, rather places that could be hovered on for some time and some specific task will take place (If I refer to buttons in my post I mean places you can hover click)
    • Volume control of either side initially was controlled by simply hovering anywhere vertically in the two halves of the screen which has proven to be clunky. I now limited volume control to be 1/8th of the screen on the left corner for the karaoke and 1/8th of the screen on the right corner for the acapella.
    • I needed to make the cursor easier for the user to follow. I found out how to leave a trail of an object in processing from http://www.science.smith.edu/dftwiki/index.php/Creating_a_trail_of_moving_object_in_Processing as I couldn’t figure out how to do it in the draw function because of the background that kept resetting. I learned that for this to work there shouldn’t be a background reset, rather a rectangle that’s the size and width of the screen with an opacity.
  • Initially the volume control was simply assumed to be understood by scroll hovering on either side but I added a visualizer for the volume as I felt like it wasn’t clear to the user of how exactly the volume would increase and reduce and the current volume the sound is at. 
  • Integrating the kinect into the project was troublesome as it was very buggy. Initially, I put made the cursor move with the kinectTracker class that Dan Schiffman made but the cursor would move all around when I went away from the kinect, so I changed the kinectTracker class to return the PVector to the center of the screen if it doesn’t find any rawDepth that is lower than the threshold, that way there .
    • I made some optimizations to the kinect tracking class as it didn’t work well with my project. One thing I did is set up two integer variables called handArea and totalArea that were incrementally increased in the for loop that detects the depth using the threshold. The handArea would only increase when the raw depth is less than the threshold and the totalArea int would increase with the for loop always. I then made the cursor be able to move only if handArea/totalArea was less than 0.065 (tested to get that value) because the hand would only be less than 6.5% of the total kinect visiblity.
    • I also made the cursor move back to a specific point away from all buttons when there is no depth less than the threshold. By doing this random clicking of buttons would be reduced.
    • I made the hover time for every function about 0.5-2 seconds in order to reduce unintentional clicking.

User Testing 1