How we can use Python to automatically generate Pinterest style color palette images from iconic scenes of our favourite movies
If you go to Pinterest, and type in “movie color palettes”, you can find many great examples of color palettes from iconic scenes of various well known movies. I thought it would be a cool idea to create a program, that can can automatically generate such color palettes.
In this tutorial I will walk you through the steps to create such a program. The program is simply a single Python based script, that launches an instance of the VLC player and it has been supplemented with several buttons and other GUI elements so we can create our color palettes. The color palettes themselves are created using a color clustering method with an algorithm called K-means. Let's dive in!
The full project code is available on my github page.
Note: You can also check out the video version of this tutorial on my channel:
For this project we will need three essential packages:
We will use pip to install our first package, Python VLC. Open up your Terminal and run the following command:
1$ pip install python-vlc
To get PyQt, we need to install it using Homebrew. Note: We specifically need to install version 4 _for this tutorial, so we will specify that when writing our install command:
1$ brew install pyqt@4
Lastly we need to install the OpenCV framework. We can do this easily through pip, but this package also relies heavily on packages —
numpy, so we will install those first:
1$ pip install matplotlib2$ pip install numpy3$ pip install scikit-learn4$ pip install opencv-python
Note: At the time of writing this tutorial, I am installing the 2nd version of OpenCV, so in Python it will be imported as cv2 module. In the future however the version might get updated.
Before we get to the actual coding part, we need to verify that all the packages are actually installed, so after all the installations are done it is always a good idea to open up Python via the command line and try to import each package one by one. If no errors show up on the command line interface after each import, it is safe to say that everything is working, and we can exit Python and close the Terminal window. Here’s how the process looks like. First open Python by simply executing the following command
Now let’s import each module sequentially:
1import vlc2import PyQt43import cv2
If the interpreter allows us to go to the next line, everything is working great and we can exit the Python interface by executing the following command
exit() This will close Python on the command line. We can also close Terminal itself at this point.
The Python VLC repository has some great ready to use samples on how to incorporate the VLC Player instance in various Python project setups. For this project we will copy the sample code, which showcases a ready to use Python VLC Player combined with a PyQt GUI interface. This will be the backbone of our color palette program. Let’s head to qtvlc.py and copy the file contents to use in our own project.
We can then make a new file in our root project directory and call it: vlc_player.py
We can now check out, how the file actually works. Open up the Terminal window, use the cd command to get to the folder of our project and run the following command:
$ python vlc_player.py Once we execute the command we should see the following window being launched:
So far so good, we now have a working instance of a VLC player. We now need to create our main file, where we will do all of our custom coding. Either create a new file and name it main.py or use the Terminal command to do this:
$ touch main.py
Before we begin working on main.py, we should go to the vlc_player.py file and comment out the last part of the file, which acts as a the program launcher. Since we will not use this file as a stand alone script and will instead use it as a helper class file, we should move this part to our main file. On vlc_player.py either comment out this part or delete it completely:
1'''if __name__ == "__main__":2 app = QtGui.QApplication(sys.argv)3 player = Player()4 player.show()5 player.resize(640, 480)6 if sys.argv[1:]:7 player.OpenFile(sys.argv)8 sys.exit(app.exec_())'''
In our main.py file, let’s import all the necessary packages:
1from vlc_player import Player2import sys3import os4from PyQt4 import QtGui, QtCore
Then let’s set up our global variables, which we will use later in our project:
1# Global variables2framesTaken = 03framesTakenList = 4framesSpecified = 05clusters = 56margin = 57borderSize = 408offset = 2
Now the fun part begins. We will create a subclass of our existing class found on the vlc_player.py file, that will inherit all the original attributes of this class. We will then append our own custom functions on top of it to make our custom program. Let’s start by defining our inherited class:
1# We define our custom VLC Player class2class Custom_VLC_Player(Player):3 def __init__(self):4 # We inherit all the attributes of the original class5 super(Custom_VLC_Player, self).__init__()
We will come back to this class later, but the next thing we have to do is to set up our main program launcher code, that we previously deleted from the vlc_player.py class. We will leave everything the same as in the original except for the window dimensions. We will use a new size, just because this will create a better looking UI layout for our particular project.
1# Initialize the QtGUI Application instance2app = QtGui.QApplication(sys.argv)34# Initialize our custom VLC Player instance5vlc = Custom_VLC_Player()6vlc.show()7# Let's change the size of the window. We will make it 660px by 530px8vlc.resize(660, 530)910if sys.argv[1:]:11 player.OpenFile(sys.argv)12sys.exit(app.exec_())
Now let’s run
python main.py We should see the same window as before. Nothing has changed yet — we have just created our custom class, which does the same thing as the original vlc_player class. So far so good. Let’s start adding our custom code.
Let’s go back to our init function for the Custom_VLC_Player class and let’s add the following code, which will add out new custom UI elements to the player window. Modify the init function to match the following code:
1class Custom_VLC_Player(Player):2 def __init__(self):3 # We inherit all the attributes of the original class4 super(Custom_VLC_Player, self).__init__()56 # We want the width and height to be fixed, so the layout dimensions7 # don't change in a weird way8 self.videoframe.setFixedWidth(640)9 self.videoframe.setFixedHeight(360)1011 # We create a new layout, where we will place our custom buttons12 self.snapbox = QtGui.QHBoxLayout()1314 # We create a button, that will execute our snapshot taking function15 self.snapbutton = QtGui.QPushButton("Take Snapshot")1617 # We will set it disabled for the start of the program18 self.snapbutton.setEnabled(0)1920 # Let's add the button to our layout21 self.snapbox.addWidget(self.snapbutton)2223 # We will connect a snapshot taking function to the button later.24 # Let's leave this command commented out for now25 '''self.connect(self.snapbutton, QtCore.SIGNAL("clicked()"),26 self.take_snapshot)'''2728 # We place a label for specifing the frame count to use29 self.l1 = QtGui.QLabel("Number of frames:")3031 # We add it to the layout32 self.snapbox.addWidget(self.l1)3334 # We create a spin box, where we can choose35 # how many frames we want to take36 self.sp = QtGui.QSpinBox()3738 # We will limit the number to 10 frames in this program39 self.sp.setMaximum(10)40 self.sp.setMinimum(0)4142 # We add it to the layout43 self.snapbox.addWidget(self.sp)44 # Connect a value change function to the spinbox.45 # Let's leave this command commented out for now46 '''self.sp.valueChanged.connect(self.valuechange)'''4748 # We add an empty space that streches to the right side49 # that way the next added element will be aligned to the right50 self.snapbox.addStretch(1)5152 # This label will hold information,53 # how many frames have been taken so far54 self.l2 = QtGui.QLabel("Frames taken: "+str(framesTaken))5556 # This is needed for the layout not to break57 self.l2.setFixedHeight(24)58 self.snapbox.addWidget(self.l2)5960 # While the process hasn't started yet, we can hide the label61 self.l2.setVisible(0)6263 # We add it to the layout64 self.vboxlayout.addLayout(self.snapbox)6566 # This is a layout which will consist of 10 labels, each label67 # will hold a thumbnail of the frame taken68 self.imageareaWidget = QtGui.QWidget(self)69 self.imageareaWidget.setFixedHeight(80)7071 # This is a wrapper around our image label widget72 self.imagearea = QtGui.QHBoxLayout(self.imageareaWidget)7374 # Let's create an array of 10 label objects and add them75 # to the widget we just created76 self.imageBoxes = 77 for i in range(0, 10):78 self.imageBoxes.append(QtGui.QLabel(str(i)))79 self.imageBoxes[len(self.imageBoxes)-1]80 self.imagearea.addWidget(self.imageBoxes[len(self.imageBoxes)-1])8182 # We will add this area to our layout, but initially we will set it83 # invisible, while the snapshot capture process hasn't started yet84 self.vboxlayout.addWidget(self.imageareaWidget)85 self.imageareaWidget.setVisible(0)
If we run the code again, we will see an updated layout to our main window:
Now we will define a valuechange function, which will just update the state of our UI elements throughout the frame capture process. Right after the init function, let’s define a new function:
1def valuechange(self):2 # We access our global variables within the function3 global framesTaken4 global framesSpecified56 # We set the framesSpecified value to7 # whatever is specified on the spinbox8 framesSpecified = self.sp.value()910 # We modify our label to give us info, how many11 # frames have been captured so far12 self.l2.setText(13 "Frames taken: "+str(framesTaken)+" from "+str(framesSpecified))1415 # Once the snapshot taking process has begun,16 # we need to disable the spinbox17 self.sp.setEnabled(0) if (framesTaken > 0) else self.sp.setEnabled(1)1819 # We enable our snapshot trigger button, if frames have been specified20 # and the process is ongoing.21 # If the process ends, we disable the button again22 if (self.sp.value() > 023 and framesTaken < 1024 and framesTaken < framesSpecified):25 self.snapbutton.setEnabled(1)26 else:27 self.snapbutton.setEnabled(0)2829 if (self.sp.value() > 0):30 self.l2.setVisible(1)31 else:32 self.l2.setVisible(0)
At this point, we can go back to the init function and uncomment the following line:
This will trigger our valuechange function every time, when a user change the spinbox value.
Now that we have a functional VLC player instance in our program, we can access its core functions, and the most important function we need to take advantage of in this project is the video_take_snapshot function. It lets us take a snapshot of the frame, that is currently visible in the player, and save it to a directory of our choice. So we will first snap the frame, save it to our directory and then use it for our image processing step. Let’s go ahead and write the function for our “Take snapshot” button:
1def take_snapshot(self):2 # Import the global variables we'll be using3 global framesTaken, clusters, borderSize, offset4 # This will be needed to check if the player was playing at the5 # time of the button press6 wasPlaying = None7 # We need to get the width and height of the video file8 videoSize = self.mediaplayer.video_get_size()9 # This is the VLC function, that let's us10 # take a snap shot of the video frame and save it in the directory11 self.mediaplayer.video_take_snapshot(12 0, "./img_"+str(framesTaken)+".png",13 videoSize,14 videoSize)1516 # While we do the image processing, let's pause the video17 if self.mediaplayer.is_playing():18 self.PlayPause()19 wasPlaying = True
We also need to uncomment the code in our init function, that connects this function to our button:
Now if we run the program and start playing the video, we can see that a new png image is saved in the project root directory every time we press the button. This is exactly how it should work up to this point. Next, let’s check out, how we can generate the color palettes for our extracted frames.
The actual name for this method is called color clustering and if you were to google this term, you could certainly find good tutorials on how to do this process. I found this very awesome tutorial by Adrian Rosebrock on color clustering which we’ll use as the basis for our own program: https://www.pyimagesearch.com/2014/05/26/opencv-python-k-means-color-clustering/
On a side note: You should definitely check out Adrian’s blog https://www.pyimagesearch.com if you want to learn some cool computer vision and deep learning tutorials. He is a master at this and he has some excellent free tutorials on his site on other cool subjects like face recognition and object detection.
So in Adrian’s tutorial there are two key functions, that we need to borrow and tweak a bit — centroid histogram and plot_colors.
Our version of plot_colors will be a bit different. We’ll use the original code of centroid_histogram though. Go ahead and paste this function inside our main.py file as a stand-alone function. (outside the VLC player class)
1# Courtesy of https://www.pyimagesearch.com/2014/05/26/opencv-python-k-means-color-clustering/2def centroid_histogram(clt):3 # grab the number of different clusters and create a histogram4 # based on the number of pixels assigned to each cluster5 numLabels = np.arange(0, len(np.unique(clt.labels_)) + 1)6 (hist, _) = np.histogram(clt.labels_, bins=numLabels)7 # normalize the histogram, such that it sums to one8 hist = hist.astype("float")9 hist /= hist.sum()10 # return the histogram11 return hist
And now go ahead and paste in our plot_colors function beneath:
1# Courtesy of https://www.pyimagesearch.com/2014/05/26/opencv-python-k-means-color-clustering/2def plot_colors(hist, centroids):3 # initialize the bar chart representing the relative frequency4 # of each of the colors5 bar = np.zeros((50, 300, 3), dtype="uint8")6 startX = 078 # Sort the centroids to form a gradient color look9 centroids = sorted(centroids, key=lambda x: sum(x))1011 # loop over the percentage of each cluster and the color of12 # each cluster13 for (percent, color) in zip(hist, centroids[offset:]):14 # plot the relative percentage of each cluster15 # endX = startX + (percent * 300)1617 # Instead of plotting the relative percentage,18 # we will make a n=clusters number of color rectangles19 # we will also seperate them by a margin20 new_length = 300 - margin * (clusters - 1)21 endX = startX + new_length/clusters22 cv2.rectangle(bar, (int(startX), 0), (int(endX), 50),23 color.astype("uint8").tolist(), -1)24 cv2.rectangle(bar, (int(endX), 0), (int(endX + margin), 50),25 (255, 255, 255), -1)26 startX = endX + margin2728 # return the bar chart29 return bar
Our version will be a bit different, because instead of plotting the colors with a width relative to a its percentage of how much this color is present in the frame, we want to display all n colors in a bar evenly, because that’s how these color palettes usually look like, plus I find it to be more visually appealing.
In the code above we first find the centroids that represent the colors with the most frequency, then we sort these centroids in an ascending matter to form sort of a gradient look, from the darkest color to the brightest. Again, this is just to make it look more visually appealing. We also add a margin defined in our global variables, just to add some spacing between the color blocks for a better overall look.
Notice, that here we use the global offset parameter we defined in the beginning of this tutorial. After tweaking around with the execution of this program, I noticed that the first one or first two colors of the color palette were always almost completely black, because videos often have a high frequency of black tones. This is why I decided to not include these first black tones in the cluster. This is why we specify the parameter — offset, which tells us how many colors to exclude starting from the beginning. This lets us produce a more colourful and thus more interesting color palette.
We have our color generator function ready, we now just have to put it in to use in our snapshot taking function. Let’s go back to the take_snapshot function and append the following code to it:
1# Let's fetch the video frame we just captured2 imagePath = os.getcwd() + "/img_"+str(framesTaken)+".png"3 # Transform the image to an OpenCV readable image4 image = cv2.imread(imagePath)56 # Let's make a copy of this image7 # to use for the color palette generation8 image_copy = image_resize(cv2.cvtColor(9 image, cv2.COLOR_BGR2RGB), width=100)1011 # Since the K-means algorithm we're about to do,12 # is very labour intensive, we will do it on a smaller image copy13 # This will not affect the quality of the algorithm14 pixelImage = image_copy.reshape(15 (image_copy.shape * image_copy.shape, 3))1617 # We use the sklearn K-Means algorithm to find the color histogram18 # from our small size image copy19 clt = KMeans(n_clusters=clusters+offset)20 clt.fit(pixelImage)2122 # build a histogram of clusters and then create a figure23 # representing the number of pixels labeled to each color24 hist = centroid_histogram(clt)2526 # Let's plot the retrieved colors. See the plot_colors function27 # for more details28 bar = plot_colors(hist, clt.cluster_centers_)2930 # Resize the color bar to be even width with the video frame31 barImage = image_resize(32 cv2.cvtColor(bar, cv2.COLOR_RGB2BGR),33 width=int(videoSize))3435 # This is just a whitespace to put between the image and the color bar36 im = np.zeros((borderSize/2, int(videoSize), 3), np.uint8)37 cv2.rectangle(im, (0, 0), (int(videoSize), borderSize/2),38 (255, 255, 255), -1)3940 # Now we combine the video frame and the color bar into one image41 newImg = np.concatenate([image, im, barImage], axis=0)42 cv2.imwrite(imagePath, newImg)
We start by retrieving the image we captured and saved in our root directory. Next, we transform this image into an OpenCV readable format and make a copy of this image to use for our color palette function. Since the K-means algorithm is very labour intensive, we run it on a downscaled version of this image. It does not affect any quality in our case, because even if we scale the image down, we can still have a pretty good understanding of which colors are the dominant ones.
We run the two color palette functions we defined earlier, and we end up with a new image, which is our color palette bar. Our final task is to append this bar image to the original one, adding a white margin between the two just for a better look, and the color palette is ready!
To provide a better overview of how the taken snapshots look like with their accompanied color palette, and to track the progress of how many snapshots have been taken so far, we can add a list of label elements, which hold a thumbnail of the image we just created. To do this in PyQt4, we need to create an pixmap element, that can hold images inside of it:
1# To show a thumbnail version of this image, we need to store it2 # in a pixmap and then add it to a label widget3 pixmap = QtGui.QPixmap()4 pixmap.load(imagePath)5 pixmap = pixmap.scaledToWidth(50)6 self.imageBoxes[framesTaken].setPixmap(pixmap)7 framesTaken = framesTaken + 189 # Now that we have at least one image captured and processed,10 # we can go ahead and show the bottom layout with our thumbnail labels11 self.imageareaWidget.setVisible(1)12 self.valuechange()1314 # Let's now add the full size image to a list of all captured images15 framesTakenList.append(imagePath)
We previously created a grid of ten image boxes as labels. We can now insert the pixmap inside these labels. Lastly we want to keep the final color palette image in memory, while we continue the snapshot capture process, so we can stitch the all together in the end. This is why we need to add the image to framesTakenList — an array that holds all our images taken so far.
Let’s continue appending code to the take_snapshot function. Next step is to check if we have gotten all our required snapshots. If that is the case, we are ready to stitch them together to form the last image:
1# Once the list is full with all the images we need, we can start to2 # combine it into the final output image3 if (framesTaken == framesSpecified):4 resultImgs = 5 for i in framesTakenList:6 # here we just add a whitespace rectangle as a top margin7 resultImgs.append(cv2.imread(i))8 im = np.zeros((9 borderSize*2,10 cv2.imread(i).shape,11 3), np.uint8)12 cv2.rectangle(13 im,14 (0, 0),15 (cv2.imread(i).shape, borderSize * 2),16 (255, 255, 255), -1)17 # here we just add a whitespace rectangle as a bottom margin18 resultImgs.append(im)1920 # Now that we have the list of all the needed images,21 # we concatinate them vertically and form the final image22 final = np.concatenate(resultImgs, axis=0)2324 # The final image still needs an outside border, so the last task25 # is to create this border line26 finalShape = final.shape27 w = finalShape28 h = finalShape29 base_size = h+borderSize, w+borderSize, 330 base = np.zeros(base_size, dtype=np.uint8)31 # We combine our main image with the border line32 cv2.rectangle(33 base,34 (0, 0),35 (w + borderSize, h + borderSize),36 (255, 255, 255), borderSize)37 base[38 (borderSize/2):h+(borderSize/2),39 (borderSize/2):w+(borderSize/2)40 ] = final4142 # The final output image is ready.43 # We now export it to the root directory44 cv2.imwrite("result_image.png", base)45 sys.exit(app.exec_())4647 # If the image capture process continues, we resume playing the video48 self.PlayPause() if wasPlaying else None
The code above lets us combine all the images and finish the task by saving the final input into our root directory. We then automatically exit the program, not to confuse the user. We can also tweak the global parameter borderSize to specify how thick we want the border to be around our image and the color palette. This is just for personal preference.
Success! We have now created our own color palette generator program. The code is available on my github page and is completely open source, so you can play around with it, tweak it or improve it to you own preferences.