Runtime loading of IFC assets in the Luminous project 

Pilot3 in the Luminous project is about integrating LLMs with BIM. 

One of the main features of the tool is to load and show an IFC files at runtime and expose their BIM metadata. 

“Runtime” is a key factor here, if you are not familiar with Unreal Engine (the Rendering/Game Engine used for Pilot3), its standard behaviour is to include all of the project assets in the final build (through a process called “cooking”). This is obviously a huge limit given that we want to build a software that is able to load any IFC files from the user. 

Lucky enough 2 open source solutions exist covering most of our requirements: 

https://ifcopenshell.org/ (open source modules for parsing IFC files and generating 3d meshes from them) 

https://github.com/rdeioris/glTFRuntime (an Unreal Engine open source plugin for loading various kind of assets at runtime, mainly glTF, but other formats, included audio ones, are available) 

From IFC to glTF 

The first challenge is to extract geometry informations from the IFC files and assemble them in a file format loadable by glTFRuntime. 

One of the tools provided by the ifcopenshell project, IfcConvert (https://docs.ifcopenshell.org/ifcconvert.html) allowed us to convert an IFC file to a glTF one. 

–use-element-guids 

The resulting glTF files keeps all the original IFC guids (unique identifications of each part of the BIM project) mapped to every single 3d mesh generated by the export. 

This mapping, allows us (for example) to retrieve the BIM metadata by clicking over an object (or by an LLM query, like “the object in front of the door”)  

In addition to this, bounds of each object (the lines in the screenshot below are just part of the spheres defining the bounds) are automatically computed, allowing spatial queries and basic collissions management (we use this for placing objects aligned with walls or ceilings) 

Python integration with ifcopenshell 

We now have a pretty solid infrastructure for generating a visual representation of a BIM model (included its spatial informations). The main goal of the Luminous project is LLM integrations, so most of the interactions between the user and the 3d environment (we call it the “Sandbox”), happen via voice or text. 

To make things comfortable for data scientist and programmers, we decided to expose a Python API allowing various interactions with the Sandbox. 

This Python integrations is managed via a simple JSON-based protocol via TCP/IP: a set of wrappers exposing a pythonic set of functions is exposed to the final users. 

The advantage of this approach is that we can split the sandbox environment (that requires a GPU) from the control system (the one interacting with the LLM system). In a second phase we are going to run a stripped-down version of the Sandbox natively on XR Headsets, so decoupling LLM logic from the rendering side is a very important step (the control system can even run on an embedded device like a RaspberryPi) 

Thanks to this design decision, we have been able to directly use the ifcopenshell python module in our control system, delegating to the sandbox only the visual and spatial management (well, actually we use it even for accessing the microphone for sending waveforms to the control system) . 

An example scenario is a user wearing a VR headset and moving around the environment using just the voice (with simple commands, like “move forward”). 

Thanks to the “spatial awareness” of the Sandbox, the user can ask about the data of the object in front of him/her: under the hood the sandbox will retrieve the guid of the object in front the virtual camera managed by the VR headset and will return it to the control system that via ifcopenshell will be able to retrieve the object’s metadata. 

Bonus Section: Enriching the scene 

A pleasant consequence of being able to load glTF assets at runtime from basically every source, allowed us to expose the feature of “enriching” the base BIM scene with additional props. The position and yaw rotation in space is stored in separated json file that can be automatically retrieved (and versioned) by the system