Compare & Benchmark AI Models
How to use this page?
Enter a prompt, add models, and click "Run" to generate images from all selected AI models. Images models will generate a PNG while text and LLM models generate SVGs.
You can add more AI models to compare and benchmark by selecting them from the dropdown and clicking "Add Model". You must be logged in to use the AI. Most AI models require tokens to use.
Image Tokens
Use image tokens to create images with AI. Each token is good for one image.
Why are people using SVGs to test LLM abilities?
SVGs (Scalable Vector Graphics) are a text-based vector image format. Because SVGs are written in code, they are a perfect test for Large Language Models (LLMs): you can prompt an LLM to generate SVG code, and then immediately see the result as an image. This tests not only the model's language understanding, but also its ability to reason about spatial relationships, follow instructions, and compose complex scenes.
How does SVG testing work?
- You give the LLM a prompt, such as "a pelican riding a bicycle".
- The LLM outputs SVG code that, when rendered, should visually match the prompt.
- This tests the model's compositional, spatial, and creative reasoning abilities.
Simon Willison's Blog
Simon Willison also regularly uses prompts like "Generate an SVG of a pelican riding a bicycle" to benchmark and compare LLMs' SVG generation abilities. See his blog for many more examples and analysis.
I love using prompts like "Generate an SVG of a pelican riding a bicycle" to test LLMs. It's a great way to see how well they can reason about spatial relationships and follow instructions. April 12, 2024
Karpathy's Example
AI researcher Andrej Karpathy frequently uses SVG prompts to test LLMs. For example, in this tweet, he asks for "a pelican riding a bicycle":
pelican riding a bicycle
— Andrej Karpathy (@karpathy) April 17, 2024
Why does this matter?
- SVG generation is a challenging, "hard" test for LLMs: it requires both understanding and precise, structured output.
- It's a fun, visual way to compare models and spot differences in their reasoning and creativity.
WHY USE DRAW GPT?
No monthly cost or subscriptions. You do not need to subscribe to ChatGPT Plus or any other services to use Draw GPT. It is a one-time cost allowing you to purchase individual image tokens without any subscriptions or monthly fees. This allows you to get access to the latest models like DALL-E 3, Flux, Stable Diffusion, GPT-4, o1, and more without ChatGPT Plus.
IS THIS BETTER THAN CHATGPT?
This uses the same or better AI to create images as ChatGPT. By default this uses either OpenAI DALL-E 3 or Google Gemini ImageGen 3.0. At some point in the future this may be changed out for better or more specific AI models but the goal is to always use the best AI for the job.
bicycle
balloon party
beautiful beach
cute puppy dog
new business logo
house