It can also draw and combine various objects and provide different views, including cuts and interiors of objects. Unlike previous text-to-image programs, it still infers details that are not mentioned in the description, but that would be necessary for a realistic image. For example, with the description “a painting of a fox sitting in a field during the winter”, the agent was able to determine that a shadow was needed.
“Unlike a 3D rendering engine, whose inputs must be specified unambiguously and in full detail, DALL · E is often able to ‘fill in the blanks’ when the caption implies that the image must contain a certain detail which is not explicitly stated, ”according to the OpenAI team.
OpenAI also exploits a feature called “zero shot reasoning”. This allows an agent to generate a response from a description and tip without any additional training and has been used for translation and other tasks. This time, the researchers applied it to the visual domain to perform the translation from image to image and from text to image. In one example, he was able to generate the image of a cat from a sketch, with the cue “exactly the same cat at the top of the sketch at the bottom”.
The system has many other talents, such as understanding how phones and other objects change over time, grasping geographical facts and landmarks, and creating images in photographic, illustration and even clip-art styles.
For now, DALL-E is quite limited. Sometimes, it delivers what you expect from the description and other times you just get some strange or bad images. As with other AI systems, even researchers themselves do not understand exactly how it produces certain images due to the black box nature of the system.
Still, if further developed, DALL-E has a vast potential to disrupt fields such as photography and illustration, with all the good and bad things that this entails. “In the future, we plan to analyze how models like DALL · E relate to social issues, such as economic impact on certain work processes and professions, the potential for partiality in the model’s results and the long-term ethical challenges implied by this technology”, the team wrote. To play DALL-E yourself, check out the OpenAI blog.