Final Project Report

For my final project, I tried visualizing space in a new way by applying image manipulation to photos of planets. In order to accomplish this, I used brightness tracking to enable users’ interaction with the images on the screen, and also the pixel array in Processing.

I used only code for my project. Initially, I planned to mount an LED on a box to replicate a flashlight. However, in the end, I was running out of time, so I resorted to using the flashlight on my phone. After seeing people interacting with my project, I wished I could have replicated a flashlight in some way, not necessarily any complicated model, but only something that can enable people to play with light in a different way, for example, Tayla’s box. Because I was experimenting how to visualize space in a different way, having something that emits light but does not look like a flashlight would have provided a more complete experience.

As my project is about image manipulation, rather than learning new code, I was mostly tinkering with the Pixel Array and Objects. Besides Daniel Shiffman’s Image Manipulation series, I also consulted his Purple Rain Challenge to create the rain effect in my project.

For my project, I use Processing to track brightness and manipulate images. The program tracks the brightest pixel and uses it control the pixels on the screen. I manipulate photos of 3 planets:

  1. Earth: When the user moves the flashlight from left to right, Earth will gradually explode, in other words, the z-coordinate of each pixel increases. And as Earth is exploding, the user can see themselves on screen because the background takes in data from the camera.
  2. Jupiter: As I found the colors of Jupiter make a very nice color scheme, I decided to let the user draw with those colors. As the user moves the flashlight from left to right, the “brush” will follow their motion horizontally. However, then the user moves the flashlight up and down, he is changing the z-coordinate of the brush, so that the brush stroke appears as if it was coming farther away or closer towards him.
  3. Venus: I make a particle rain using the colors of Venus. Each circle/particle takes the color of one pixel in the photo of Venus and moves from top to bottom of the screen and back again. With 10,000 circles moving at the same time, it looks as if the particles were raining onto Venus and painting Venus at the same time. When the user moves the mouse from left to right, he is also controlling the speed of the rain. Unfortunately, I could not use brightness tracking in this part of my program because my computer cannot handle the 10,000-object array and the pixel array at the same time. I did try adapting this part of my program so that it would also use brightness tracking. However, I could not retain the rain effect so I decided to resort to the mouse instead.

My code consists of 5 parts which I split into 5 tabs in Processing so that my code would look cleaner.

Main code:

Earth:

Jupiter

Track brightest pixel

Venus

Note: the screen dimension needs to be 1024×640 for the program to run.

The three images that I used:

3 most difficult parts of my project: 

I struggled the most with the pixel array. The program we examined when we first learned about the pixel array runs through all of the pixels in the image. However, as I experimented with different kinds of image manipulation, I figured out that it was not always necessary to run through all of the pixels in the image, and doing this only slows the program down. Simple as it is, it took me a great deal of time to figure out because only when I experimented with different image manipulations to see which method works in each case.

Moreover, as I use multiple arrays in my project, the pixel array from the camera, the pixel array from the images, and some arrays for objects, I found debugging very difficult. Sometimes my code did not work only because the array was too large for it to process. However, there was no error message for this kind of error, so I often fixed random things in my code to make it work. Yet in the end, the code went more and more wrong. I had to start from scratch multiple times to identify which part was the problem.

Last but not least is the process of coming up with ideas for image manipulations and turning it into reality. At first, I had ideas for all 8 planets. However, after trying to put into code each of those ideas, I realized that they were not feasible for me in terms of time and also in terms of complexity. It took a great deal of time trying to put an idea into code because sometimes the idea was good but too complicated, so I also had to make modifications to simplify my ideas so that the program would run reliably. Some of my ideas involved moving pixels around, and although I had a very clear vision of my idea, I did not know how to translate it into calculations so that I could change the x and y positions of the pixels as I wanted.

After finishing my final project, I greatly regret that I did not create a system of physical interaction for my project. As we have been building a lot throughout the semester, this time I wanted to try focusing on coding more and thus to improve my coding skills. However, coding took up so much of my time that I did not have the chance to build anything. Therefore, my project does not achieve the interactive level that I had hoped for my final project. If I had the opportunity to redo my project, I would start from building something first, then begin incorporating technology into it. In that way, the focus of my project would be, instead of on the complexity of something they can only see, on how users can interact physically with the object, which is also my favorite gain from this course.

Homework due Monday December 4: Final Project description

 1. A sketch of  my concept

2. Describe, in at least a couple of paragraphs, your project

  • For my final project, I will make use of computer vision to create a controller for my game, which is related to space. In this game, the player has to use a flashlight to detect and kill evil aliens or save trapped astronauts.
  • Instead of choosing a particular round or mode of the game on the screen, the player will piece together LEGO bricks of some certain colors to choose the game mode. For example, if a player piece together bricks of white, red and orange, they will go to the game mode called “Mars.” I do this by telling the program to detect certain colors and remember colors that are present in the video, then I use the if statement to direct the program to a particular game mode.
  • Then, the player will use the LEGO block he or she has just made as a controller. The LEGO block is actually fixed upon a handheld device that has 2 pushbuttons, one to save astronauts and one to kill aliens. As the player moves the handheld device around, the device acts as a flashlight on screen, allowing them to see who to kill and who to save. I will attach an LED on this handheld device so that the camera can track the brightest pixel and reflect it on the screen.

3. A list of parts I will need 

  • I will need a screen and camera(s).
  • For the camera, I am not sure I need one or two. The camera that detects colors in the LEGO block should be stationary, similar to the one usually attached to the TV, so that the video will not be affected by colors not from the LEGO block. However, the camera that tracks the LED should be on top of the screen so that the user can look at the screen and move the controller at the same time.

4. A detailed list of topics I need to learn: 

  • 1. How to solder so that I can attach a prototype shield on an Arduino
  • 2. How to shorten my code so that my program will run faster: I think for this part I will need to finish my code and then show it to you, so that you will be able to identify which parts can be replaced with a function or a kind of inexplicit calculation, for example, state = 1 – state.

 5. A block diagram of my circuit

 6. The 3 most difficult parts of my project

  • 1. The code for the game:

For the game, I have to work with pixels which are notoriously slow to process. Therefore, my main obstacle in creating the game is to simplify the code as much as possible so that the visuals on the screen will respond instantly to the player’s movements.

  • 2. Precision in color tracking

At first, I intended to offer the player 7 colors to choose from, but as I tested the code, it is difficult for the program to infallibly distinguish between orange and red, or yellowish brown and brown. Thus, I am trying to adjust the threshold at which the distance between the current color and the track color is compared to, so that the color detection can be more precise. However, in the end, I might have to use fewer and easily distinguishable colors.

  • 3. Creating a user-friendly design for the controller

Initially, I intended to create a box for the controller. Yet it might be uncomfortable for the player has to hold and move a rectangular cube around. Therefore, I am thinking about a controller that is shaped like a rectangular panel bent on its left and right side so that the player has something to grip.

Homework due Wednesday November 29: Final project ideas

For my final project, I will use Lego bricks to create a remote control for my space-themed game. The game will adjust according to the colors of the bricks that users use to create their remote control. I will use computer vision to detect the colors on the remote control (I have not tested whether I can detect many colors or not, but I will do this as soon as possible). In order to make the remote control able to control the game, I will attach an LED to it, so that the computer can detect the brightest pixel. 

I will probably need:

  • A camera
  • A screen
  • An LED and maybe some buttons

After today’s class I also want to make my game a bit more physical. However, I am still on the fence about it because it might make my game too complicated to understand. 

Homework due Monday November 27: Manipulating an image

For this assignment, I tried playing with 3D to emulate Daniel Shiffman’s Explode example. Here is the link to that example: http://learningprocessing.com/examples/chp15/example-15-15-explode3D

In my project, I use 3D to magnify a portion of an image so that we can see all the little details in that portion. For example, the image that I use has food cut into cubes and we need to look closer to figure what the food is.

For this assignment, at first, I struggled a lot with the ArrayOutOfBoundException error, which caused my computer to crash quite a few times. I tried changing the size of the canvas and luckily it worked perfectly fine, so I guess that the error is probably due to some locations in the pixel array are actually bigger than the image itself. Thus the computer cannot locate the pixel to magnify.

Here is my code: 

Homework due Wednesday November 22: What computing means to me at this point

I used to think that computing is something technical and daunting to learn. Now, computing is still quite daunting to me, but empowering at the same time because now I can use it as a tool to create something, and to be creative. Since I became more acquainted with computing, I realize that when you put two disparate aspects or disciplines together and make them work in tandem, you get something absolutely fascinating. For example, art and computing make computer art, or the intangible code and the tangible Arduino change the way I see LEDs forever. This is how computing has greatly changed my thought process. I have started to delve into the intersection between various disciplines, which has changed the way I approach problems and fueled my interest in problems I am dealing with.

I still remember how excited I was when I succeeded in making the LED respond to the amount of light coming into the photoresistor. The delight I felt at that moment, I believe, applies to anyone who first has the opportunity to tinker with computing. Yet unfortunately, many people believe that computing is dry and difficult, like I used to believe, so they do not want to play with it. However, making computing seem entertaining is what interactive media can do, and this is also what we have learned throughout the semester, so I really want to bring the delight of learning about computing into my projects so that others can also experience the fun of playing with it. We all want to make others happy, and I hope I will be good at making people happy in this way.

Homework due November 20: LCD Circuit

For this assignment, we were asked to build the LCD circuit and print the numbers sent by Processing onto the LCD. At first, after I assembled my circuit, I uploaded the Hello World example but all I received is 16 white blocks without any word displayed. Because the code in the example cannot be wrong so I thought my circuit must have been wrong. I checked the tutorial in the Sparkfun Kit Guide and realized that I had built my circuit backwards. Luckily, when I rebuilt the circuit, it worked perfectly.

When I used the Serial Display example to print the numbers sent from Processing, I discovered that lcd.write() prints characters, not numbers, while lcd.print() prints numbers. Also, initially, I did not declare a variable for what I want to print, for example, byte numbers; and then use lcd.print(numbers) to print it. Instead, I used lcd.print(Serial.read()) and the numbers changed very slowly. Thus after that, I changed my code to resemble the code in the Dimmer example, which declares a variable for the numbers sent from Processing, and the LCD reflected the change instantly. 

Arduino code: 

Processing code: 

 

Homework due Monday November 6: Deleting Balls using Light Sensors

The game I created uses light sensors to delete moving balls on the screens. The balls are in 6 colors: red, yellow, blue, orange, purple, and green. On my Arduino, I use 3 sensors color-coded red, yellow, and blue.

If the red sensor is covered, red balls will be deleted. The same applies to the yellow and blue sensors. When the red and yellow sensors are covered at the same time, orange balls will be deleted (red + blue = purple; blue + yellow = green). 

Because I want the game to keep adding balls after some of them have been deleted, I tried to find a function that can add objects to an array, and there is actually one that does this: append(). I used this tutorial by Daniel Shiffman to learn how to use append(). On the website Learning Processing by Daniel Shiffman, there are also a lot of simple but very useful examples. It is a great resource to learn Processing. 

The difficulty that I encounter with this project is that in order to delete the balls, I change their color to white. Thus, they only disappear, not really deleted from the program. After a while, there will be too many balls on the sketch and the program will run very slowly. I have not found a way to remove certain objects from an array yet, but I thought of restarting the program after a period of time to free up memory. Thanks to Michael, I have been able to restart my game by resetting the array. However, the reset works for the about first 10 times, but after that, the reset does not make the balls disappear anymore. I have not figured out this problem yet, but I think it has something to do with the program taking up too much of my computer’s memory. 

Schematics 

Arduino code

Processing code

 

Homework due Wednesday November 1: Control servo and led with mouse using Arduino and Processing

  1. Control a servo motor connected to Arduino using your mouse in Processing

This task is quite similar to the Dimmer example, so I only need to change some code in the Dimmer example, such as the width of the canvas. Also, I draw left and right arrows to indicate the direction of the servo in Processing.

Arduino code:  

Processing code

Result

2. Control an LED connected to your Arduino using two rectangles in Processing

For this task, I use mouseClicked() to send the 1 or 0 to turn the led on or off. If the position of the mouse is inside the on button, ‘1’ is sent to the serial port. If the position of the mouse is inside the off button, ‘0’ is sent to the serial port. 

Arduino code 

 

Processing code: 

 

Result

Homework due Monday October 30: Recreate a computer art piece in Processing

For this assignment, my inspiration comes from a project in Computer Arts (issue August 2017). Here is the link to the project. In this work, they use lines to connect the points in the grid so that they can draw something. 

In my work, I draw 4 squares, then points inside of them, then random lines that connect 2 points in each square. Here is the result. 

Throughout the process of creating art piece,  I found myself struggling with how to create a parameter, or a set of values that random() can choose from. Usually we use random() to pick out values from a continuous set of values. For example: random(0,255) pick out any value from 0 to 254. However, in my case the points in are not right next to each other, thus in order to draw random lines between them, I have to create a set of values of the x and y coordinates of points in a particular square, then use that set to draw the lines. To do this, I use Arrays to store x and y values from points in each square. 

I also discovered that for example, random(0,255) picks out values from 0 up to, but not including, 255. I did not know this before, so when I pick random numbers out of my array, which has 6 values, numbered from 0 to 5, I wrote my code as numbers_xposPoint[ (int) random (1,5)] and created the error Array Index Out Of Bounds Exception , which made Processing stop responding and my computer crash.

CODE

In addition to this most basic version, I also created some variations of it:

Variation 1: 

The program does not stop drawing at 25 lines each square, but continues to draw. Each time you press ‘c’, the new lines change to another color

 

Variation 2: 

Instead of 4 squares, there are 400 squares. This time I have to use calculations to create x and y positions of the squares

 

 

Assignment due Monday October 16: Self-portrait & loop

For this assignment, we are asked to draw a self-portrait using Processing and to use for() to draw 5 different rectangles in 5 different places.

  1. Using for() to draw 5 rectangles

I find it hard for me to come up with good calculations for the position of the x and y coordinates, width, and height, in order to draw the rectangles separately from each other. The numbers I came up with in my code are just trial and error.

2. Self-portrait

In my self-portrait, I apply the for() loop into creating a color fading effect for my hair. I also use a circle to run across the canvas to paint the background, as it runs, the circle changes its color. 

I also learn to use arc to draw my eyes, here is the link to the arc() reference page . 

What took me the most time to draw is the shapes that uses translate() and rotate(), as it is hard to bring the shape to the desired position. After a lot of trial and error, I have figured out that it is best to change the coordinates in translate() to move the shape around, not the coordinates in rect().