Using GPT-3 and Google Cloud Vision to Quantify the Visual Density of Languages

I was wondering which language expresses more information per square pixel, and I think the answer is somewhat counterintuitive: it's Mandarin, despite its apparent visual splendor. In other words, even though each Chinese letter is more visually complicated than any English letter, it also carries more information because you need fewer Chinese letters to express the same content.

Here is the chart that shows this. In short, this chart shows what happens if you downsample an image with the same content step by step, and calculate how much information is still left in the image. The peak for each language is each language's maximum information density per square pixel. Chinese wins, English is the worst.

The Algorithm

1. Pick a text. In this case, I picked a 7th grade reading comprehension text - a little story of a girl baking bread with her father of around 1,000 words.
2. Pick a target language.
3. Translate the text into the target language.
4. Print the text in the language onto an image. Force it onto a 2,000 pixel-wide image, and wrap the text.
5. Run image recognition on the image to read the text.
6. Translate the recognized text back into English.
7. Feed the text into the OpenAI Completions API (running on GPT-3). Run through 10 questions that need one-word answers, and have OpenAI answer the questions. This gives us a score from 0 to 1 that expresses how many answers it got right.
8. Feed the text again into the OpenAI Completions API. Now ask GPT-3 straight up to compare the recognized, re-translated text to the original text. This gives us another score from 0 to 1 that expresses how similar GPT-3 thinks the two texts are (through its mysterious inner workings).
9. Calculate the text embedding of the recognized, re-translated text using the OpenAI Embeddings API. This gives us another score from 0 to 1 that expresses how similar the two texts' embeddings are, using cosine similarity, in embeddings space.
10. Now downsample the original image by 10% - i.e., just shrink the image. Don't change anything else. That means the original content is now expressed on fewer square pixels.
11. Repeat steps 5 to 10, until the text quality on the image gets so low that the algorithms start completely failing.

The Details

Some details on each of the steps above.

• We need a font that can express any language you want, without giving any particular language an "advantage" (by adding or subtracting visual "flair" relative to another language). The Google Noto Sans font family can do that.
• We need to pick a downsampling filter for when we shrink the image. I wanted it to be a simple one, so I used the Box filter in the Python Pillow library. Other filters might retain visual language information for longer (in smaller images). It would be weird if that changed the relative results between languages.
• Notice that there are no languages in this comparison set that write from right to left. That's because I couldn't figure out how to install the libraqm library with Pillow and I eventually just gave up. You need that library to change text direction. (It took me a while to figure out why the OCR was so bad at reading Hebrew. Then I realized it was just rendered the wrong way around.)
• I find my usual observation to be that when using a large language model for anything, the LLM is often better the less you constrain it. I.e., it knows better how it works than you know how it works. I played a bunch with asking the "right questions" of the text through GPT-3, and eventually I just told GPT-3 to compare two texts straight up ("how similar in meaning are these"). The two results were incredibly similar.
• The embeddings comparison isn't that useful, on the other hand: let's say we compare a massively downsampled text image with the original text. The text image quality is so low that GPT-3 only answers 2 out of the original 10 questions correctly. Also, GPT-3 says "I think this is semantically similar at 4 out of 10". In that case, the embedding cosine similarity is still 0.77 or so. Just too high. Maybe I forgot to take the square root or something. Or it's a deep insight about how much of the regularity of language, and thus the similarity of texts, is just in its rhythm, and that doesn't get nuked when you down-sample.

Here is another way to visualize all this. This shows the "information value" (multiplying the two semantic scores coming out of the GPT-3 tests on question asking and similarity, ignoring the embeddings) vs. the square pixels of each downsampled image, by language. Same data as in the chart above. This shows better that there are indeed some structural weaknesses in the various machine learning APIs that are in the analysis pipeline here - for example, GPT-3's comparison of the re-translated Japanese texts (this all happens in English, remember!) is never that high, so the information value starts at a lower level. Still, what happens at the lower square pixel numbers is what matters, and there it's a consistent picture. Though it's possible that Japanese is more visually dense than it's given credit for here. But whatever happens, English remains the worst!