Using Large Multimodal Models (LMM) to digitalize scanned HVAC Schematics into Metadata Schemas for Buildings

The digitization of HVAC schematic drawings is a time-consuming and error-prone process. Current methods, including machine learning and deep learning, have limitations to implement comprehensive solutions in this field. This research proposes a framework using Large Multimodal Models (LMM), specifically the GPT-4o model, to automate the conversion of HVAC schematics diagrams into a machine-readable metadata schema that adheres to the Brick Schema. The approach demonstrates the feasibility of generating serialized Brick models with high reliability in symbol recognition and classification. However, the results also reveal the limitations of GPT-4o in understanding the relationship between symbols, which led to a low reliability in recognizing and classifying connections. The findings highlight the adaptability of GPT-4o to recognize new data once provided within a prompt, and the relevance of introducing legends and Brick classes to significantly improve the performance of the model, while revealing the need for further research for improving the performance of the model to capture the relationships between symbols in HVAC schematics diagrams.