Skip to content
Andris Gauracs
GithubLinkedIn

Generating color palettes from movies with Python

Python, Software, Apps8 min read

Color palettes

How we can use Python to automatically generate Pinterest style color palette images from iconic scenes of our favourite movies

If you go to Pinterest, and type in “movie color palettes”, you can find many great examples of color palettes from iconic scenes of various well known movies. I thought it would be a cool idea to create a program, that can can automatically generate such color palettes.

In this tutorial I will walk you through the steps to create such a program. The program is simply a single Python based script, that launches an instance of the VLC player and it has been supplemented with several buttons and other GUI elements so we can create our color palettes. The color palettes themselves are created using a color clustering method with an algorithm called K-means. Let's dive in!

The full project code is available on my github page.

Note: You can also check out the video version of this tutorial on my channel:

1. Installing the prerequisites

For this project we will need three essential packages:

  1. Python VLC — a Python binding for the VLC Player framework. We will use this to create an instance of a VLC player in our program.
  2. PyQt4 — A GUI package, that provides us with great tools, which we can use to create actual functioning programs with windows, buttons and other graphical UI elements.
  3. OpenCV — A very powerful image manipulation framework, which we will use to generate the final output images with the color palettes.

We will use pip to install our first package, Python VLC. Open up your Terminal and run the following command:

1$ pip install python-vlc

To get PyQt, we need to install it using Homebrew. Note: We specifically need to install version 4 _for this tutorial, so we will specify that when writing our install command:

1$ brew install pyqt@4

Lastly we need to install the OpenCV framework. We can do this easily through pip, but this package also relies heavily on packages — matplotlib and numpy, so we will install those first:

1$ pip install matplotlib
2$ pip install numpy
3$ pip install scikit-learn
4$ pip install opencv-python

Note: At the time of writing this tutorial, I am installing the 2nd version of OpenCV, so in Python it will be imported as cv2 module. In the future however the version might get updated.

Before we get to the actual coding part, we need to verify that all the packages are actually installed, so after all the installations are done it is always a good idea to open up Python via the command line and try to import each package one by one. If no errors show up on the command line interface after each import, it is safe to say that everything is working, and we can exit Python and close the Terminal window. Here’s how the process looks like. First open Python by simply executing the following command $ python

Now let’s import each module sequentially:

1import vlc
2import PyQt4
3import cv2

If the interpreter allows us to go to the next line, everything is working great and we can exit the Python interface by executing the following command exit() This will close Python on the command line. We can also close Terminal itself at this point.

2. Getting the VLC Player instance for our project

The Python VLC repository has some great ready to use samples on how to incorporate the VLC Player instance in various Python project setups. For this project we will copy the sample code, which showcases a ready to use Python VLC Player combined with a PyQt GUI interface. This will be the backbone of our color palette program. Let’s head to qtvlc.py and copy the file contents to use in our own project.

We can then make a new file in our root project directory and call it: vlc_player.py

We can now check out, how the file actually works. Open up the Terminal window, use the cd command to get to the folder of our project and run the following command: $ python vlc_player.py Once we execute the command we should see the following window being launched:

UI Interface

So far so good, we now have a working instance of a VLC player. We now need to create our main file, where we will do all of our custom coding. Either create a new file and name it main.py or use the Terminal command to do this: $ touch main.py

3. Modifying the UI

Before we begin working on main.py, we should go to the vlc_player.py file and comment out the last part of the file, which acts as a the program launcher. Since we will not use this file as a stand alone script and will instead use it as a helper class file, we should move this part to our main file. On vlc_player.py either comment out this part or delete it completely:

1'''if __name__ == "__main__":
2 app = QtGui.QApplication(sys.argv)
3 player = Player()
4 player.show()
5 player.resize(640, 480)
6 if sys.argv[1:]:
7 player.OpenFile(sys.argv[1])
8 sys.exit(app.exec_())'''

In our main.py file, let’s import all the necessary packages:

1from vlc_player import Player
2import sys
3import os
4from PyQt4 import QtGui, QtCore

Then let’s set up our global variables, which we will use later in our project:

1# Global variables
2framesTaken = 0
3framesTakenList = []
4framesSpecified = 0
5clusters = 5
6margin = 5
7borderSize = 40
8offset = 2

Now the fun part begins. We will create a subclass of our existing class found on the vlc_player.py file, that will inherit all the original attributes of this class. We will then append our own custom functions on top of it to make our custom program. Let’s start by defining our inherited class:

1# We define our custom VLC Player class
2class Custom_VLC_Player(Player):
3 def __init__(self):
4 # We inherit all the attributes of the original class
5 super(Custom_VLC_Player, self).__init__()

We will come back to this class later, but the next thing we have to do is to set up our main program launcher code, that we previously deleted from the vlc_player.py class. We will leave everything the same as in the original except for the window dimensions. We will use a new size, just because this will create a better looking UI layout for our particular project.

1# Initialize the QtGUI Application instance
2app = QtGui.QApplication(sys.argv)
3
4# Initialize our custom VLC Player instance
5vlc = Custom_VLC_Player()
6vlc.show()
7# Let's change the size of the window. We will make it 660px by 530px
8vlc.resize(660, 530)
9
10if sys.argv[1:]:
11 player.OpenFile(sys.argv[1])
12sys.exit(app.exec_())

Now let’s run python main.py We should see the same window as before. Nothing has changed yet — we have just created our custom class, which does the same thing as the original vlc_player class. So far so good. Let’s start adding our custom code.

Let’s go back to our init function for the Custom_VLC_Player class and let’s add the following code, which will add out new custom UI elements to the player window. Modify the init function to match the following code:

1class Custom_VLC_Player(Player):
2 def __init__(self):
3 # We inherit all the attributes of the original class
4 super(Custom_VLC_Player, self).__init__()
5
6 # We want the width and height to be fixed, so the layout dimensions
7 # don't change in a weird way
8 self.videoframe.setFixedWidth(640)
9 self.videoframe.setFixedHeight(360)
10
11 # We create a new layout, where we will place our custom buttons
12 self.snapbox = QtGui.QHBoxLayout()
13
14 # We create a button, that will execute our snapshot taking function
15 self.snapbutton = QtGui.QPushButton("Take Snapshot")
16
17 # We will set it disabled for the start of the program
18 self.snapbutton.setEnabled(0)
19
20 # Let's add the button to our layout
21 self.snapbox.addWidget(self.snapbutton)
22
23 # We will connect a snapshot taking function to the button later.
24 # Let's leave this command commented out for now
25 '''self.connect(self.snapbutton, QtCore.SIGNAL("clicked()"),
26 self.take_snapshot)'''
27
28 # We place a label for specifing the frame count to use
29 self.l1 = QtGui.QLabel("Number of frames:")
30
31 # We add it to the layout
32 self.snapbox.addWidget(self.l1)
33
34 # We create a spin box, where we can choose
35 # how many frames we want to take
36 self.sp = QtGui.QSpinBox()
37
38 # We will limit the number to 10 frames in this program
39 self.sp.setMaximum(10)
40 self.sp.setMinimum(0)
41
42 # We add it to the layout
43 self.snapbox.addWidget(self.sp)
44 # Connect a value change function to the spinbox.
45 # Let's leave this command commented out for now
46 '''self.sp.valueChanged.connect(self.valuechange)'''
47
48 # We add an empty space that streches to the right side
49 # that way the next added element will be aligned to the right
50 self.snapbox.addStretch(1)
51
52 # This label will hold information,
53 # how many frames have been taken so far
54 self.l2 = QtGui.QLabel("Frames taken: "+str(framesTaken))
55
56 # This is needed for the layout not to break
57 self.l2.setFixedHeight(24)
58 self.snapbox.addWidget(self.l2)
59
60 # While the process hasn't started yet, we can hide the label
61 self.l2.setVisible(0)
62
63 # We add it to the layout
64 self.vboxlayout.addLayout(self.snapbox)
65
66 # This is a layout which will consist of 10 labels, each label
67 # will hold a thumbnail of the frame taken
68 self.imageareaWidget = QtGui.QWidget(self)
69 self.imageareaWidget.setFixedHeight(80)
70
71 # This is a wrapper around our image label widget
72 self.imagearea = QtGui.QHBoxLayout(self.imageareaWidget)
73
74 # Let's create an array of 10 label objects and add them
75 # to the widget we just created
76 self.imageBoxes = []
77 for i in range(0, 10):
78 self.imageBoxes.append(QtGui.QLabel(str(i)))
79 self.imageBoxes[len(self.imageBoxes)-1]
80 self.imagearea.addWidget(self.imageBoxes[len(self.imageBoxes)-1])
81
82 # We will add this area to our layout, but initially we will set it
83 # invisible, while the snapshot capture process hasn't started yet
84 self.vboxlayout.addWidget(self.imageareaWidget)
85 self.imageareaWidget.setVisible(0)

If we run the code again, we will see an updated layout to our main window:

UI Interface

4. Defining the functions

Now we will define a valuechange function, which will just update the state of our UI elements throughout the frame capture process. Right after the init function, let’s define a new function:

1def valuechange(self):
2 # We access our global variables within the function
3 global framesTaken
4 global framesSpecified
5
6 # We set the framesSpecified value to
7 # whatever is specified on the spinbox
8 framesSpecified = self.sp.value()
9
10 # We modify our label to give us info, how many
11 # frames have been captured so far
12 self.l2.setText(
13 "Frames taken: "+str(framesTaken)+" from "+str(framesSpecified))
14
15 # Once the snapshot taking process has begun,
16 # we need to disable the spinbox
17 self.sp.setEnabled(0) if (framesTaken > 0) else self.sp.setEnabled(1)
18
19 # We enable our snapshot trigger button, if frames have been specified
20 # and the process is ongoing.
21 # If the process ends, we disable the button again
22 if (self.sp.value() > 0
23 and framesTaken < 10
24 and framesTaken < framesSpecified):
25 self.snapbutton.setEnabled(1)
26 else:
27 self.snapbutton.setEnabled(0)
28
29 if (self.sp.value() > 0):
30 self.l2.setVisible(1)
31 else:
32 self.l2.setVisible(0)

At this point, we can go back to the init function and uncomment the following line:

1self.sp.valueChanged.connect(self.valuechange)

This will trigger our valuechange function every time, when a user change the spinbox value.

Now that we have a functional VLC player instance in our program, we can access its core functions, and the most important function we need to take advantage of in this project is the video_take_snapshot function. It lets us take a snapshot of the frame, that is currently visible in the player, and save it to a directory of our choice. So we will first snap the frame, save it to our directory and then use it for our image processing step. Let’s go ahead and write the function for our “Take snapshot” button:

1def take_snapshot(self):
2 # Import the global variables we'll be using
3 global framesTaken, clusters, borderSize, offset
4 # This will be needed to check if the player was playing at the
5 # time of the button press
6 wasPlaying = None
7 # We need to get the width and height of the video file
8 videoSize = self.mediaplayer.video_get_size()
9 # This is the VLC function, that let's us
10 # take a snap shot of the video frame and save it in the directory
11 self.mediaplayer.video_take_snapshot(
12 0, "./img_"+str(framesTaken)+".png",
13 videoSize[0],
14 videoSize[1])
15
16 # While we do the image processing, let's pause the video
17 if self.mediaplayer.is_playing():
18 self.PlayPause()
19 wasPlaying = True

We also need to uncomment the code in our init function, that connects this function to our button:

1self.connect(self.snapbutton, QtCore.SIGNAL("clicked()"),self.take_snapshot)

Now if we run the program and start playing the video, we can see that a new png image is saved in the project root directory every time we press the button. This is exactly how it should work up to this point. Next, let’s check out, how we can generate the color palettes for our extracted frames.

5. The color palette generation function

The actual name for this method is called color clustering and if you were to google this term, you could certainly find good tutorials on how to do this process. I found this very awesome tutorial by Adrian Rosebrock on color clustering which we’ll use as the basis for our own program: https://www.pyimagesearch.com/2014/05/26/opencv-python-k-means-color-clustering/

On a side note: You should definitely check out Adrian’s blog https://www.pyimagesearch.com if you want to learn some cool computer vision and deep learning tutorials. He is a master at this and he has some excellent free tutorials on his site on other cool subjects like face recognition and object detection.

So in Adrian’s tutorial there are two key functions, that we need to borrow and tweak a bit — centroid histogram and plot_colors.

Our version of plot_colors will be a bit different. We’ll use the original code of centroid_histogram though. Go ahead and paste this function inside our main.py file as a stand-alone function. (outside the VLC player class)

1# Courtesy of https://www.pyimagesearch.com/2014/05/26/opencv-python-k-means-color-clustering/
2def centroid_histogram(clt):
3 # grab the number of different clusters and create a histogram
4 # based on the number of pixels assigned to each cluster
5 numLabels = np.arange(0, len(np.unique(clt.labels_)) + 1)
6 (hist, _) = np.histogram(clt.labels_, bins=numLabels)
7 # normalize the histogram, such that it sums to one
8 hist = hist.astype("float")
9 hist /= hist.sum()
10 # return the histogram
11 return hist

And now go ahead and paste in our plot_colors function beneath:

1# Courtesy of https://www.pyimagesearch.com/2014/05/26/opencv-python-k-means-color-clustering/
2def plot_colors(hist, centroids):
3 # initialize the bar chart representing the relative frequency
4 # of each of the colors
5 bar = np.zeros((50, 300, 3), dtype="uint8")
6 startX = 0
7
8 # Sort the centroids to form a gradient color look
9 centroids = sorted(centroids, key=lambda x: sum(x))
10
11 # loop over the percentage of each cluster and the color of
12 # each cluster
13 for (percent, color) in zip(hist, centroids[offset:]):
14 # plot the relative percentage of each cluster
15 # endX = startX + (percent * 300)
16
17 # Instead of plotting the relative percentage,
18 # we will make a n=clusters number of color rectangles
19 # we will also seperate them by a margin
20 new_length = 300 - margin * (clusters - 1)
21 endX = startX + new_length/clusters
22 cv2.rectangle(bar, (int(startX), 0), (int(endX), 50),
23 color.astype("uint8").tolist(), -1)
24 cv2.rectangle(bar, (int(endX), 0), (int(endX + margin), 50),
25 (255, 255, 255), -1)
26 startX = endX + margin
27
28 # return the bar chart
29 return bar

Our version will be a bit different, because instead of plotting the colors with a width relative to a its percentage of how much this color is present in the frame, we want to display all n colors in a bar evenly, because that’s how these color palettes usually look like, plus I find it to be more visually appealing.

In the code above we first find the centroids that represent the colors with the most frequency, then we sort these centroids in an ascending matter to form sort of a gradient look, from the darkest color to the brightest. Again, this is just to make it look more visually appealing. We also add a margin defined in our global variables, just to add some spacing between the color blocks for a better overall look.

Notice, that here we use the global offset parameter we defined in the beginning of this tutorial. After tweaking around with the execution of this program, I noticed that the first one or first two colors of the color palette were always almost completely black, because videos often have a high frequency of black tones. This is why I decided to not include these first black tones in the cluster. This is why we specify the parameter — offset, which tells us how many colors to exclude starting from the beginning. This lets us produce a more colourful and thus more interesting color palette.

6. Putting the color generation function to use

We have our color generator function ready, we now just have to put it in to use in our snapshot taking function. Let’s go back to the take_snapshot function and append the following code to it:

1# Let's fetch the video frame we just captured
2 imagePath = os.getcwd() + "/img_"+str(framesTaken)+".png"
3 # Transform the image to an OpenCV readable image
4 image = cv2.imread(imagePath)
5
6 # Let's make a copy of this image
7 # to use for the color palette generation
8 image_copy = image_resize(cv2.cvtColor(
9 image, cv2.COLOR_BGR2RGB), width=100)
10
11 # Since the K-means algorithm we're about to do,
12 # is very labour intensive, we will do it on a smaller image copy
13 # This will not affect the quality of the algorithm
14 pixelImage = image_copy.reshape(
15 (image_copy.shape[0] * image_copy.shape[1], 3))
16
17 # We use the sklearn K-Means algorithm to find the color histogram
18 # from our small size image copy
19 clt = KMeans(n_clusters=clusters+offset)
20 clt.fit(pixelImage)
21
22 # build a histogram of clusters and then create a figure
23 # representing the number of pixels labeled to each color
24 hist = centroid_histogram(clt)
25
26 # Let's plot the retrieved colors. See the plot_colors function
27 # for more details
28 bar = plot_colors(hist, clt.cluster_centers_)
29
30 # Resize the color bar to be even width with the video frame
31 barImage = image_resize(
32 cv2.cvtColor(bar, cv2.COLOR_RGB2BGR),
33 width=int(videoSize[0]))
34
35 # This is just a whitespace to put between the image and the color bar
36 im = np.zeros((borderSize/2, int(videoSize[0]), 3), np.uint8)
37 cv2.rectangle(im, (0, 0), (int(videoSize[0]), borderSize/2),
38 (255, 255, 255), -1)
39
40 # Now we combine the video frame and the color bar into one image
41 newImg = np.concatenate([image, im, barImage], axis=0)
42 cv2.imwrite(imagePath, newImg)

We start by retrieving the image we captured and saved in our root directory. Next, we transform this image into an OpenCV readable format and make a copy of this image to use for our color palette function. Since the K-means algorithm is very labour intensive, we run it on a downscaled version of this image. It does not affect any quality in our case, because even if we scale the image down, we can still have a pretty good understanding of which colors are the dominant ones.

We run the two color palette functions we defined earlier, and we end up with a new image, which is our color palette bar. Our final task is to append this bar image to the original one, adding a white margin between the two just for a better look, and the color palette is ready!

7. Supplementing the UI with thumbnails

To provide a better overview of how the taken snapshots look like with their accompanied color palette, and to track the progress of how many snapshots have been taken so far, we can add a list of label elements, which hold a thumbnail of the image we just created. To do this in PyQt4, we need to create an pixmap element, that can hold images inside of it:

1# To show a thumbnail version of this image, we need to store it
2 # in a pixmap and then add it to a label widget
3 pixmap = QtGui.QPixmap()
4 pixmap.load(imagePath)
5 pixmap = pixmap.scaledToWidth(50)
6 self.imageBoxes[framesTaken].setPixmap(pixmap)
7 framesTaken = framesTaken + 1
8
9 # Now that we have at least one image captured and processed,
10 # we can go ahead and show the bottom layout with our thumbnail labels
11 self.imageareaWidget.setVisible(1)
12 self.valuechange()
13
14 # Let's now add the full size image to a list of all captured images
15 framesTakenList.append(imagePath)

We previously created a grid of ten image boxes as labels. We can now insert the pixmap inside these labels. Lastly we want to keep the final color palette image in memory, while we continue the snapshot capture process, so we can stitch the all together in the end. This is why we need to add the image to framesTakenList — an array that holds all our images taken so far.

8. Stitching the final image together

Let’s continue appending code to the take_snapshot function. Next step is to check if we have gotten all our required snapshots. If that is the case, we are ready to stitch them together to form the last image:

1# Once the list is full with all the images we need, we can start to
2 # combine it into the final output image
3 if (framesTaken == framesSpecified):
4 resultImgs = []
5 for i in framesTakenList:
6 # here we just add a whitespace rectangle as a top margin
7 resultImgs.append(cv2.imread(i))
8 im = np.zeros((
9 borderSize*2,
10 cv2.imread(i).shape[1],
11 3), np.uint8)
12 cv2.rectangle(
13 im,
14 (0, 0),
15 (cv2.imread(i).shape[1], borderSize * 2),
16 (255, 255, 255), -1)
17 # here we just add a whitespace rectangle as a bottom margin
18 resultImgs.append(im)
19
20 # Now that we have the list of all the needed images,
21 # we concatinate them vertically and form the final image
22 final = np.concatenate(resultImgs, axis=0)
23
24 # The final image still needs an outside border, so the last task
25 # is to create this border line
26 finalShape = final.shape
27 w = finalShape[1]
28 h = finalShape[0]
29 base_size = h+borderSize, w+borderSize, 3
30 base = np.zeros(base_size, dtype=np.uint8)
31 # We combine our main image with the border line
32 cv2.rectangle(
33 base,
34 (0, 0),
35 (w + borderSize, h + borderSize),
36 (255, 255, 255), borderSize)
37 base[
38 (borderSize/2):h+(borderSize/2),
39 (borderSize/2):w+(borderSize/2)
40 ] = final
41
42 # The final output image is ready.
43 # We now export it to the root directory
44 cv2.imwrite("result_image.png", base)
45 sys.exit(app.exec_())
46
47 # If the image capture process continues, we resume playing the video
48 self.PlayPause() if wasPlaying else None

The code above lets us combine all the images and finish the task by saving the final input into our root directory. We then automatically exit the program, not to confuse the user. We can also tweak the global parameter borderSize to specify how thick we want the border to be around our image and the color palette. This is just for personal preference.

9. The full example

Success! We have now created our own color palette generator program. The code is available on my github page and is completely open source, so you can play around with it, tweak it or improve it to you own preferences.


© 2020 by Andris Gauracs. All rights reserved.