Project information
- Project Title: Code Generation
- Skills: AI/ML, Computer Vision
- Colloborations: Benjamin Wu, Joshua Davis, Davit Soselia, Zlatko-Salko
- Project URL: [GITHUB]
Project Description
This project explored the application of Actor-Critic fine-tuning to train a model that generated front-end code capable of reproducing an input image. The research built upon existing literature on RL-based code generation from text specifications and extended the idea to image or multimodal contexts. The proposed approach outperformed transformer and CNN-based baselines, and showed robustness over varying sample complexity.
The methodology employed in this research on generating HTML code from images using Actor-Critic fine-tuning involved defining the problem of generating HTML code from images, developing a model that could accurately reproduce the structure and visual elements of an input image by generating corresponding HTML code, and evaluating the model's performance using the modified CodeBLEU metric to attempt to better approximate the visual similarity of the output that would result from running the generated code.
The proposed approach involved using a pre-trained image encoder to extract features from the input image, which were then fed into an actor-critic model that generated HTML code. The model was fine-tuned using a combination of supervised and reinforcement learning, with the CodeBLEU metric used to evaluate the quality of the generated code. The results showed that the proposed approach outperformed transformer and CNN-based baselines, and was robust over varying sample complexity.
The results obtained in this study provided valuable insights into the potential of Actor-Critic fine-tuning for improving the generation of HTML code from input images. The proposed approach advanced automated code generation towards potential applications in web development and design automation, enabling efficient conversion of images into functional HTML code. The study also highlighted the importance of using appropriate evaluation metrics for assessing the quality of generated code, and the need for further research to explore the generalization capabilities of the proposed approach.