Sketch2Code

Summary

Sketch2Code is a Microsoft AI Lab project that uses AI and Computer Vision to convert hand-written drawings to working HTML prototypes, streamlining the creative design process.

Abstract

The Sketch2Code project by Microsoft AI Lab aims to expedite the creative design process by using AI and Computer Vision to convert hand-written drawings on a whiteboard to working HTML prototypes. By training a model using Microsoft Cognitive Services and Azure Cognitive Services, the system can detect HTML objects, extract handwritten text, and generate HTML snippets of different design elements, allowing designers to collaborate and see the resulting code to test their new web page ideas immediately.

Bullet Points

• •Sketch2Code is a Microsoft AI Lab project that uses AI and Computer Vision to convert hand-written drawings to working HTML prototypes.
• •The system uses Microsoft Cognitive Services and Azure Cognitive Services to train a model to recognize hand-drawn web design elements and extract handwritten text.
• •By combining the design element and the extracted content, the system generates HTML snippets of the different elements in the design and infers the layout of the design from the position of the identified elements to generate the final HTML code.
• •Sketch2Code allows designers to collaborate and see the resulting code to test their new web page ideas immediately, streamlining the creative design process.
• •The system is built on Azure Cloud Platform, using Custom Vision and Computer Vision APIs for object detection, OCR including handwriting recognition, and identifying the layout.