The gap between graphic designers andcoders just got smaller. Until now, if you wanted to build a graphic user interface,whether for app design or custom websites, you would have to enlist a developer to turn yourideas into code. But designers could soon take this power into their own hands.
Copenhagen-based startupUlzard Technologieshas developed an app, Pix2Code, which uses a trained neural network to transform screenshots into linesof code. All it needs is an image of a design for a graphic user interface (GUI) and, once recognised, it beginsto automatically churn out corresponding code. Even more impressively, it creates code that’s compatible with Android, iOS and web-based technologies – a rare functionality across platforms.The Pix2Code prototype only needs a single input image and has demonstrated an accuracy of 77 per cent.Tony Beltramelli, the founder of Ulzard, states that he believes Pix2Code has the potential to “end the need for manually-programmed GUIs”.
In order to program the neural network, Beltramelli says that the team had to overcome three main problems. The first being computer vision – that a computer will not automatically understand the given scene and cannot naturally identify the objects present, their positions, and characteristics (i.e. buttons and labels). Secondly, they had to overcome a language problem – the issue of teaching the network to understand text, so it could create accurate samples. Finally, the deep learning programme had to be trained to understand the link between code, text and the corresponding images, so it could connect all three while generating new code.