Ill then show you how to generate your own memes using the Google Colab,here.
After a brief discussion of results and next steps, you could see some sample memes in the appendix.
Oh, and Ill show a newly generated meme at the head of each section.
System components
The mainsystem components are shown in the diagram below.
The user starts by entering a search query to find a background image, like apple pie.
The system then checks for matching images inWikimedia Commonsand theOpenImages dataset.
Both datasets have corresponding text descriptions of the images.
I use theCLIP encoders from OpenAIto first perform a semantic search on the text descriptions.
A semantic search looks for matching concepts, not just a word search.
I then perform a semantic search on the images.
The user checks out the top 10 images that match the query and selects their favorite.
Either theGPT-3 model from OpenAIor theGPT-Neo model from EleutherAIis used to generate 10 possible captions.
The user selects the best caption to create the new meme, which can be downloaded.
What are memes, again?
The termoriginatedin Richard Dawkins book,The Selfish Gene.
It’s free, every week, in your inbox.
Dylan Wenzlaucreated an automatic meme generatorusing a deep convolutional web connection.
you’ve got the option to read about his systemhere, and run it onlinehere.
I was looking for a way to inject a set of new images into the memesphere.
I use OpenAIs CLIP to perform a semantic search.
For more information about how CLIP works, check out my article,here.
Wikimedia Commons
The Wikimedia Commons has over 73 million JPEG files.
Most of them are released with permissive rights, like the Creative Commons Attribution license.
I use the shutil.copyfileobj() function in Python to download the image files.
A dataset of image descriptionsis available for download.
I ran each of the descriptions through OpenAIs CLIP system and cached the embeddings for quick access.
I then download the top 20 matching images using theOpenImages download API.
I present the top 10 images to the user to choose their favorite.
Generating captions
I use two different implementations of GPT to generate the captions.
And theres the open-source GPT-Neo model from EleutherAI.
The model is a lot smaller, but its free to use.
GPT-3 Da Vinci
OpenAIs GPT-3 Da Vinci is currently the largest AI model for Natural Language Processing.
I am using their latest zero-shot style of prompting with their new Da Vinci Instruct model.
Here is the prompt that creates a caption for the apple pie picture.
Create a funny caption for a new meme about apple pie.
I pass the prompt into the call to OpenAI along with some additional parameters.
Heres the Python code.
Thetemperatureandtop_pparameters are similar in that they indicate the amount of variety in the response.
Thefrequency_penaltyandpresence_penaltyare also similar in that they control how often there are new deviations and new topics in the response.
The author generated the following text in part with GPT-3, OpenAIs large-scale language-generation model.
Running the code 10 times will yield the following results, at a total cost of $0.03.
Note that I formatted the text to be in uppercase.
Occasionally the system will produce text that may be inappropriate or offensive.
My code will show a warning for any of the generated captions that are flagged as sensitive or unsafe.
GPT-Neo
GPT-Neo is a transformer model created primarily by developers known as sdtblck and leogao2 on GitHub.
Theprojectis an implementation of GPT-2 and GPT-3-style models using the mesh-tensorflow library.
So far, their system is the size of OpenAIs GPT-3 Ada, their smallest model.
But GPT-Neo is available for free.
I used theHuggingface Transformersinterface to access GPT-Neo from my Python code.
Heres the prompt I wrote using Disaster Girl and Grumpy Cat memes with example captions.
Create a funny caption for a meme.
Theme: grumpy catImage description: A face of a cat who looks unhappyCaption: I dont like Mondays.
Heres the code to generate a caption.
1: I LOVE APPLE PIE2: I CANT.
IM NOT ALLOWED3: I LOVE THE SIMPLICITY OF AN APPLE PIE4: APPLE PIE.
THE ONLY THING BETTER THAN THIS IS A HOT BATH5: IM A PIE.
These are not as good as the GPT-3 captions.
Most of them are quite simple and not very funny.
Number 10 is just plain absurd.
But number 4 seems to be OK.
Lets use this as our caption.
The final step is to compose the meme by writing the caption into the background image.
Typesetting memes
Adding the captions to memes is fairly straightforward.
Most memes are composed using the Impact fontdesigned by Geoffrey Leein 1965.
For AI-Memer, I used some codeby Emmanuel Pirefor positioning and rendering the caption into the background image.
Here are our two memes.
The caption on the left was generated by GPT-3 and the one on the right was generated by GPT-Neo.
Although many of the generated captions are straightforward, occasionally they can be very clever and funny.
Next steps
Although the results are pretty good, there is definitely room for improvement.
For example, the choices for background images seem somewhat limited, especially for pop culture.
This may be due to the fact that I am restricting the search to use only freely licensed photos.
The developers behind GPT-Neo at EleutherAI are continuing to build and train bigger language models.
Their next model is calledGPT-NeoX.
Source code and acknowledgments
All source code for this project is available onGitHub.
it’s possible for you to experiment with the code using thisGoogle Colab.
I released the source code under theCC BY-SA license.
I would like to thank Jennifer Lim and Oliver Strimpel for their help with this project.
This article by Robert A. Gonsalves was originally published byTowards Data Science.
Robert is an artist, inventor, and engineer in the Boston area.
Dont forget to check out more of his generated memes in theappendix here.
