On Wednesday, a collaborative whiteboard app maker called “tldraw” made waves online via establishment A prototype of a feature called “Make Real” that lets users draw software images and bring them to life using AI. The feature uses OpenAI’s GPT-4V API to translate vector graphics into action Tailwind CSS and JavaScript web code that can make user interfaces or even create simple implementations of games like Break out.
“I think I need to go lie down,” posted designer Kevin Cannon in the beginning following the X virus that revealed the creation of function sliders that turns the sound on the screen, the sound is visual for it changing colorsand game works of tic-tac-toe. Soon, others followed with the introduction of a clone Break outcreating a the call clock works that sign, on the snake gamemaking a Pong rewardmeaning a visual state chartand a lot more.
Users can Experiment with a live demo of Make It Real online. However, its operation requires the provision of an API key from OpenAI, which is a security risk. If others access your API key, they can use it to collect huge amounts of money in your name (OpenAI the costs by the amount of data going in and out of your API). Those who are technically inclined can run the code locally, but will still require access to the OpenAI API.
Tldraw, developed by Steve Ruiz in London, is an open source collaborative whiteboard tool. It offers a basic unlimited canvas for drawing, text, and media without requiring a login. Launched in 2021, the project get $2.7 million in seed funding and is supported by GitHub sponsors. When the GPT-4V API was recently launched, Ruiz integrated a prototype called “cause-a-ui“created by Sawyer Hood to bring AI-powered functionality into tldraw.
GPT-4V is a version of OpenAI’s large language model that can interpret visual images and use them as vectors. Like AI expert Simon Willison explain on X, make it a real task by “generating the base64 PNG code of the graphics components, then transferring that to GPT-4 Vision” with the program and instructions to convert the image into a file using Tailwind. In fact, here’s the full program specification that tells GPT-4V how to take the inputs and turn them into working code:
const systemPrompt=”You are an expert web designer who specializes in style css.
A user will give you a small wireframe of an application.
You will return an HTML file that uses HTML, tailwind css, and JavaScript to create a high fidelity website.
Include any additional CSS and JavaScript in the html file.
If you have any images, upload them from Unsplash or use solid color corners.
The user will give you notes in blue or red text, arrows, or drawings.
The user may include images of other websites as physical references. Picking styles as best as you can, matching fonts/colors/backgrounds.
They can also give you the html of the previous design they want you to review from.
Make any changes they ask of you.
In the wireframe, the pre-designed html will appear as a white rectangle.
Use creative license to make the material more meaty.
Use JavaScript modules and unpkg to import any necessary dependencies.’
As more people experiment with GPT-4V and combine it with other protocols, we will likely see more novel applications of OpenAI vision technology emerging in the coming weeks. Also on Wednesday, a developer used the GPT-4V API to create a live, real-time video feed by AI-generated voice David Attenborough, which we’ve covered separately.
Right now, it feels like we’ve been given a preview of a possible future state of software development—or visual design, at least—where creating a functional design is as simple as creating a visual mockup and having an AI. The model does the rest. According to developer Michael Dubakov wrote when showing his own creation Make It Real, “OK, @tldraw is officially crazy. It’s really interesting where we’ve ended up in 5 years… I can’t keep up with the pace of innovation anymore.”