Final Project Atoka Jo

Assignment 19: Final Project

My project was about interacting with the Minion that was displayed on the TV screen using the webcam. Since I like Minions and I am interested in improving education for children, I wanted to create something that involved Minions, and would be appealing to children (whether it was in an educational context or in a recreational context). I used computer vision to accomplish this final project. Since I attended the workshop on AI, machine learning, and computer vision, and I felt that there was so much potential in this field, I wanted to use this technology in my final project. The computer vision was done through a software called “Wekinator,” which acted as the medium between the processing input and the processing output. I decided to use the design principle where the explanation of the final project, the video captured by the camera, and the Minion image, would be displayed all in one big TV screen. Initially, I was not going to have the video captured by the camera to be shown on the screen. However, during the process of building my final project, I found out that I was sometimes unsure of whether the object that I was showing to the camera was actually captured by the camera or not, and so, in the end, I decided to include the video as well.

Default: Unhappy Minion

When a banana is shown: Happy Minion 1

When a snack box is shown: Happy Minion 2

When the camera is covered (dark): Surprise! Pirate Minion

Materials

  1. 2 objects (banana and snack box)
  2. TV screen
  3. Webcam
  4. Computer

For this project, I wanted to do something that used minimal materials and involved computer vision. This is because I had already made a game using Arduino and created a box of three buttons where I used my soldering and hardware skills that I learnt in class. Although this was not for any project, since I created this, I wanted to do something completely different for my final project.

Box with buttons

Other photos showing the process of making this

In order to complete the final project, I had to learn how to use the Wekinator software, and sending inputs and outputs using Processing. Some of the useful websites I used were

  1. Wekinator Home
  2. GMU: Tutorials/Performance Platform/Recognizing Gestures with Wekinator

Input Codes (to capture the webcam video):

Output Codes (to display the different Minions and its sounds):

Wekinator Setup:

The three most difficult parts of my project was understanding how the pixels were read in processing, learning how the Open Sound Control (OSC) message worked, and dealing with the sound files. Although we touched on some basic video manipulation in class, it was difficult to understand how the codes for sending the pixels to the Wekinator were managed through processing. Moreover, Open Sound Control was something that we had not learnt in class, and so I had to read a lot of websites and look at tutorials to learn how the OSC worked. Since I wanted to include different Minion sounds for each Minion image, I downloaded video clips of Minions from YouTube, cut the parts that were necessary, and used Audacity to convert them into a readable format. In the beginning, my codes were not reading the sound and even though I tried different kinds of ways to reformat the sound and change my codes, nothing worked. It was only until I asked Professor Pierre for help, and he told me how I had to use Audacity to make the sound file in a mono format to make it recognizable by processing.

Now that I have completed the final project and have seen the reactions of the people who played with my “Make the Minion Happy!” I would use Minion videos instead of Minion images if I could do it again differently. Although using the images came to my mind at first, perhaps, it may have been more amusing if the Minions were in short video clips and the users were able to watch the Minion react/move/dance in different ways to different objects.

Video 1 of the Final Project

Video 2 of the Final Project

Video 3 of the Final Project

 

Homework for Dec 4 (Mon)

Assignment 18: Final Project Description and Prototype

Description:

I will be making an interactive project called “Make the Minion Happy.” There will be Minions on the TV screen and the person can interact with the Minions in different ways by showing different objects, such as the banana and the Purple Minion. If I decide to use the infrared LEDs, on the banana (which Minions like), there will be one IR LED attached to it, and when the sensor detects one IR LED, the Minions on the screen will show a happy face. On the Purple Minion (which Minions hate), there will be two infrared LEDs attached to it, and when the sensor detects two IR LEDs, the Minions on the screen will get scared. If I use Wekinator to detect the banana and the Purple Minion, I will have to use the technique of computer vision and directly analyze the different objects.

List of Materials if I use the IR LEDs:

  • 1 TV screen
  • 1 Camera
  • 3 infrared LEDs
  • Real Banana
  • Evil Purple Minion

List of Materials if I use Wekinator:

  • 1 TV screen
  • 1 Camera
  • White Background
  • Real Banana
  • Evil Purple Minion

Sketch (apologies for the horrible drawing):

List of Topics I need to learn:

  • How to use the Wekinator
  • Processing input to Wekinator
  • Processing output from Wekinator
  • Advanced Computer Vision

Prototype:

For my prototype, I decided to practice using Wekinator. I attended the Wekinator workshop that was held by Professor Aaron Sherwood last Sunday. However, since I had to leave in the middle of the workshop, I was not quite familiar with the sequence of using Wekinator and Processing. With a help of my friends who also attended the workshop, I was able to recreate the sound and colors project.

In this sound and colors project, the Processing code detects the images from my webcam and sends the data as an input to Wekinator. After training the Wekinator, the Wekinator sends a signal and the output of different sounds and colors are displayed through Processing.

I used the codes from here.

Prototype Video:

To record my screen, I used QuickTime Player and followed the instructions here.

To create a new screen recording with QuickTime Player:

  1. Launch QuickTime Player and choose File, New Screen Recording (the keyboard shortcut is Control, Command, N).
  2. Press the Record button.
  3. A screen with additional instructions will appear. …
  4. When you’re finished, click on Stop Recording on the Menu Bar.

What I think are going to be the 3 most difficult parts of my project:

  1. Mastering the use of Wekinator (a software that I am new to)
  2. Input and Output in Processing
  3. The codes using Processing to detect the objects (banana and Purple Minion)

Homework due Nov 29 (Wed)

Assignment 17: Final Project Plan

I would like to make an interactive project called “Make the Minion Happy.” There will be a screen projected on the wall with Minions, and an infrared sensor. On the banana (which minions like), there will be one infrared LED attached to it, and when the sensor detects one IR LED, the Minions on the screen will show a happy face. On the purple minion (which minions hate), there will be two infrared LEDs attached to it, and when the sensor detects two IR LEDs, the Minions on the screen will be scared. 

Materials

  • 3 infrared LEDs
  • 1 infrared sensor/camera
  • TV Screen

Perhaps, since I attended the Sunday Wekinator session which was taught by Professor Aaron, I might use the Wekinator to detect human movement, and one side could have Bananas and other side could have Purple Minions, which can change the expression of the Minion.

Homework due Nov 27 (Mon)

Assignment 16: Image Manipulation

For this assignment, what I originally wanted to do was to use computer vision to manipulate the image on the computer (we haven’t learnt computer vision yet so this was too optimistic). Since I like Minions, I used the Minion image that I grabbed from the internet.

For the first step, I decided to make the cursor on the mouse make the flashlight on the Minion image, which is initially dark. For this, I followed Daniel Schiffman’s Processing Tutorial 10.5: Image processing with Pixels.

Code:

Output:

For the next step, I wanted my webcam to track the brightest point/the flashlight using computer vision. I watched Daniel Schiffman’s Processing 11.1: Capture and Live Video – Processing Tutorial and 11.2: Using the Movie Object for Displaying Recorded Videos – Processing Tutorial. Since I am new to computer vision, I struggled a lot with making this work. In the end, I was able to find a code that dealt with Brightness Threshholding by Golan Levin.

Code:

I tried combining these two codes so that when I turned on the flashlight on my phone, my webcam would detect the light from the flashlight, which would light up the Minion image. However, this code did not work, and since this integration between image manipulation and computer vision is much more complex than I originally intended, I’ll try to achieve this after we learn about computer vision.

Failed Code:

I found this tutorial on “Tracking the Brightest Point” later on, but perhaps, this can help me achieve my final outcome.

Homework due Nov 22 (Wed)

Assignment 15: About Computing

HW: Write on this blog two paragraphs about what computing means to you at this point. Is it adding something to your life? Is it helping you become a better person? What are you getting out of it, and what do others get from it? Use the word “computing” as broadly or narrowly as you wish.

According to the BusinessDictionary.com, computing means “The process of utilizing computer technology to complete a task. Computing may involve computer hardware and/or software, but must involve some form of a computer system. Most individuals use some form of computing every day whether they realize it or not. Swiping a debit card, sending an email, or using a cell phone can all be considered forms of computing.” 

I cannot deny the fact that computing has made our lives convenient. The majority of the current world is digitized, even now as I type these words onto the blog post because we, humans, used to write everything with pen and paper, which has evolved into typewriters, and now, the majority (or all) of the essays I write for classes are typed on my computer. However, I do not know if computing has made me become a better person. First of all, how do we define a “better” person? A person who is friendly? A person who has a social impact? Or a person who types codes and creates his own website? As for me, I would define a better person as someone who can bring good to the society, whether that is in a form of taking leadership positions or creating innovative products to increase the efficiency of one’s performance. Although I am still at the baby stage of learning the mechanisms behind computing, computing, indeed, has opened so many doors of opportunities.  

As much as I feel that I am getting benefits out of computing, there are some drawbacks that are inherent in the nature of the effects of computing. My distractions come from the constant messages from friends on Messenger, posts on Facebook, and the live stories on Snapchat. When I was a child and when I did not possess an iPhone or a computer, I would directly interact with real people more, instead of checking what was happening or talking to people through a screen. Even on this campus, I see students and faculty walking while looking at something on their phone. When people are on their phones, I feel reluctant in bringing up a conversation with them because I would assume that they are occupied. So, by nature, I think that it decreases the number of abrupt conversations that arises when, for example, one meets another on the elevator and begins talking, which could potentially develop into a friendship because of computing.

Homework due Nov 6 (Mon)

Assignment 14: Get To The Minion Game (Arduino to Processing)

I decided to create a game (more like a fortune-telling-of-the-day type of game) where the user counts how many times he or she pressed the button in order to get the Minion onto the screen instead of the Getting There Turtle. Since the Minion appears on the screen in random order, to get the Minion to appear on screen is a matter of luck. Using that logic, I created the rules for the game so that the lower count it takes for the user to get the Minion up on screen, the better fortune he or she has for that day.

This game has many parts for improvement, such as adding framerate(), a counter to count how many times the user pressed the button, and a final display of the counter when the user gets the Minion up on the screen. I will keep on working on this game by adding additional functions to make it more aesthetically pleasing and user friendly.

Arduino Code:

Processing Code:

Photo:

Video:

Homework due Nov 1 (Wed)

Assignment 13: Processing to Arduino Communication

1. Servo Motor Control

Arduino Code:

Processing Code:

Video:

2. ON and OFF LED

Arduino Code:

Processing Code:

Video:

Homework due Oct 30 (Mon)

Assignment 12: Computer Graphics and Art in Processing

I created two computer art pieces in Processing. Although my ideas came from reading the Computer Graphics and Art, I also explored the different functions in Processing on my own.

1. Rotating Circle

I decided to create a computer graphic that contains a rotating circle with lines coming out from it.

2. Black and White Image

I used the image that my friend took when we went to Lebanon during fall break. Since old computers displayed images in black and white, I took that idea to produce this by using Processing.

I used Danief Shiffman’s Learning Processing website to build the base code for both of the above computer graphics and art. I found this website very useful because it provided explanations and had examples from the basics of Processing.

Homework due Oct 16 (Mon)

Assignment 11: Self Portrait and 5 Rectangles

Self Portrait

I drew a self-portrait using Processing. I first tried to draw a self-portrait with simple lines and shapes, but since that resulted in a more character-like figure rather than a self portrait, I found some codes online which I used it as my base. By adding circles, triangles, and lines onto the base code, I was able to integrate the skills that we learnt in class. Since I like going to the beach, I decided to have the beach scenery as the background. I had first intended to fill in the figure, but the code did not fully fill in the self-portrait but instead left some parts transparent. Therefore, I decided to leave the transparency of the figure of the self-portrait just for the aesthetics.  

Code:

Product:

 

5 Rectangles

Another homework for today was to draw 5 rectangles using a loop. I read through the processing loop reference page and typed in some extra codes to adjust the rectangles and add color. Very similar to Arduino, it created multiple rectangles by adding the detail codes inside the one for loop.  

Code:

Product:

Homework due Oct 11 (Wed)

Assignment 10: Motion Like A Person

I decided to use the DC motor to replicate the movement of a human eye. Originally, the concept that I came up with was to replicate the movement when people roll their eyes when they are annoyed, frustrated, etc. I used the laser cutter for the first time to cut the white part of the eye using white acrylic. I stuck it onto the motor with some scotch tape to see how it would be. Since the motor makes the rotation very fast, I realized that it rotated too fast for a human eye movement. Yet, I still decided to draw the black pupil onto the acrylic. Interestingly, since the rotation speed is so quick, the rapid rotation made it look like a static human eye.

Materials:

  • 1 red board
  • 1 bread board
  • 1 transistor (30N06L)
  • 1 DC motor
  • 1 acrylic eye
  • 4 wires
  • Some scotch tape

Code:

Schematic:

Picture:

Video: