LEGO models can be described in the LDraw format. The file format was developed by James Jessiman in 1995. Since then this has become a defacto standard for creating LEGO models on computers.
This program rhildraw is a simple importer of LDraw files. A model can be
described by one or more files. We'll get to the details of these files as we go
on our journey to implementing the importer.
By the time you get to the end you should have had a chance to read my thoughts on the implementation and see the code.
This literate document is still rough around the edges and needs some more Time Love and Care to massage it for better readability. I hope however that we all can enjoy reading this already.
The repository for this literate program is at https://github.com/jesterKing/rhildraw
If you don't feel like reading documentation much you can skip ahead and just go straight to the generated script.
Here a couple of links to the specifications on LDraw.org
TBD
The following diagram shows the main structure of the script
flowchart TD
subgraph init
A[[Initialization]]
B[Read file]
end
subgraph geom [Read Module]
direction LR
C[[Handle commands]]
D[[Add part geometry]]
E[[Create block definition]]
end
A --> B
B --> geom
C --> D
D --> C
D --> E
We will be using a couple of dictionaries to keep track of files, materials and so on. These will make finding data easier as the command to import another file uses the file name as the key either without or with a parent folder prepended.
Dictionaries for the follow data are used:
vfiles: dictionary to file representations. Each file will be represented
twice. Once with just the file name and once with parent folder prepended
parent/file.datidefs: dictionary holding all block definitions. The block definiton name is
the key. Block definition names are part file names without the extension. The
block definitions are the values. The corresponding RhinoCommon class is
InstanceDefinition.materials: dictionary containing LEGO colors as RenderMaterials. The color
code is the key. Colors are materialized as RenderMaterial only when
actually needed.
vfiles: Mapping[str, LDrawFile]= dict()
idefs : Mapping[str, InstanceDefinition]= dict()
materials : Mapping[str, LDrawMaterial]= dict()
vertidx = 0
zoom_extents = False
def prepare_parts_dictionary():
global lib_path
library_path : Path = Path(lib_path)
library_path_net : DirectoryInfo = DirectoryInfo(lib_path)
all_parts_net = library_path_net.EnumerateFiles("*", SearchOption.AllDirectories)
for p in all_parts_net:
fn = Path(p.FullName)
ldrawfile = LDrawFile(fn)
vfiles[ldrawfile.name] = ldrawfile
vfiles[ldrawfile.pname] = ldrawfile
def get_ldraw_file(part_name : str) -> LDrawFile:
global vfiles
part_name = part_name.replace('/', '\\')
if part_name in vfiles:
return vfiles[part_name]
raise Exception(f"Part file not found: {part_name}")
def add_virtual_file(model : Path, filename : str, data : List[str]):
virtual_file_path = model.parent / 'virtual' / filename
virtual_file = LDrawFile(virtual_file_path, data)
vfiles[virtual_file.name] = virtual_file
vfiles[virtual_file.pname] = virtual_file
def prepare_idefs_dictionary():
for idef in sc.doc.InstanceDefinitions:
idefs[idef.Name] = idef
def update_idefs_dictionary(part_name):
idef_part_name = clean_name(part_name)
idef = sc.doc.InstanceDefinitions.Find(idef_part_name)
if idef:
idefs[idef_part_name] = idef
return idef
return None
def get_part_idef(prt):
p = Path(prt)
pname = clean_name(p.name)
if pname in idefs:
return idefs[pname]
return None
Colors are defined just like parts and assemblies. Line type 0 is used through the Colour Definition Language Extension (specification).
For this script we'll be using the colors defined in LDConfig.ldr from the
LDraw complete library, although there are alternatives available. Technically
the file with standard color definitions contains the following line in its
header:
0 !LDRAW_ORG Configuration UPDATE YYYY-MM-DD
But we'll forgo this check and just assume the file we specify is the correct file.
Preparing the colors means preparing color data for materials so that is what we'll be doing.
We need just one simple method to set up the materials dictionary that will
contain all the defined colors, and that is the <<method to load colors>>.
<<method to load colors>>
To load the colors we just need to get the LDrawFile instance for
LDConfig.ldr and pull out the commands. Then we loop over all the commands and
handle those that start with the required 0 !COLOUR string. Again we assume
that a command is well-formed. For each color command we
<<read color properties from command>>, create an instance of LDrawMaterial
and add it to the materials dictionary.
def load_colors():
<<load color commands>>
COLOR_CMD = "0 !COLOUR "
TO_REMOVE = "0 !"
for cmd in cmds:
if cmd.startswith(COLOR_CMD):
<<read color properties from command>>
<<create material and save to dictionary>>
print("Colors read")
To <<load color commands>> we simple get the LDrawFile instance for
LDConfig.ldr and use get_commands() on that.
colorldr = get_ldraw_file("LDConfig.ldr")
cmds = colorldr.get_commands()
Then for each line we clean up the part we're not using (0 !). Most keys have
values, but for finishes there is a little snag. Most of the finishes, CHROME,
PEARLESCENT, RUBBER, MATTE_METALLIC and METAL when specified make this
an odd list. To even that out we will add a dummy value to the list, so we can
simple iterate over the list in pairs and generate a dictionary out of it.
Eventually we'll be able to properly handle finishes. Perhaps MATERIAL SPECKLE
and MATERIAL GLITTER with the use of some good procedural texturing.
properties = dict()
cmd = cmd[len(TO_REMOVE):]
cmd_split = cmd.split()
if len(cmd_split) % 2 == 1:
cmd_split.append('dummy')
keyvalue_count = len(cmd_split) // 2
for i in range(0, keyvalue_count*2, 2):
properties[cmd_split[i]] = cmd_split[i+1]
With the properties now converted to a dictionary we can instantiate an
LDrawMaterial and add it to the main materials dictionary with the CODE
value as key. This key we can later use during handling of other commands that
specify a key code to find the correct entry.
ldraw_material = LDrawMaterial(properties)
materials[properties["CODE"]] = ldraw_material
class LDrawMaterial:
def __init__(self, props):
self.properties = props
self.name = props["COLOUR"]
self.render_material = None
def _get_color4f(self, colstr):
colstr = colstr[1:]
r = int(colstr[0:2], 16) / 255.0
g = int(colstr[2:4], 16) / 255.0
b = int(colstr[4:6], 16) / 255.0
return Color4f(r, g, b, 1.0)
def _alpha(self, alphastr):
alpha = 1.0 - (float(alphastr) / 255.0)
return alpha
def get_render_material(self):
if self.render_material == None:
raise Exception(f"Material non-existant: {self.name}")
return self.render_material
def create_render_material(self):
for rm in sc.doc.RenderMaterials:
if rm.Name == self.name:
self.render_material = rm
return
pbr_rm = RenderContentType.NewContentFromTypeId(pbr_guid)
_basecolor = self._get_color4f(self.properties["VALUE"])
_roughness = 0.2
_metallic = 0.0
_opacity = 1.0
if "ALPHA" in self.properties:
_opacity = self._alpha(self.properties["ALPHA"])
_roughness = 0.03
if "METAL" in self.properties or "CHROME" in self.properties:
_metallic = 1.0
_roughness = 0.03
if "MATTE_METALLIC" in self.properties:
_metallic = 1.0
_roughness = 0.3
pbr_rm.SetParameter(PbrNames.BaseColor, _basecolor)
pbr_rm.SetParameter(PbrNames.Opacity, _opacity)
pbr_rm.SetParameter(PbrNames.Metallic, _metallic)
pbr_rm.SetParameter(PbrNames.Roughness, _roughness)
pbr_rm.Name = self.name
self.render_material = pbr_rm
sc.doc.RenderMaterials.Add(pbr_rm)
The content GUID used to create Physically Based Material instances is
short-handed in pbr_guid.
pbr_guid = ContentUuids.PhysicallyBasedMaterialType
The importer uses a simple virtual file system. It is really not more than a
mapping of file names to LDrawFile instances. For physical files on the actual
file system they will be initialized with a full path to a file. But there are
also virtual files that we insert into the system so we can make the access to
either file type simple and look the same.
All LDraw files consist of lines where each line is of a certain line type. Each line type denotes a command. All LDraw files have these line types, so it is easy to parse them.
LDraw files have the extenions .mpd, .ldr and .dat. These extensions are
typically used for files with specific usage although from just the contents it
doesn't really matter.
However, the following are well-established conventions. The .dat file is used
to define geometry of parts and subparts. The .ldr files often contain
commands to load parts, they express assemblies or models. The .mpd is a form
of multi-part document, where in one file several files are combined.
In our script we represent LDraw files with the class LDrawFile. The class
itself is very simple. It has only a method for
<<initialization of LDrawFile>> and a method to
<<get commands from LDrawFile>>.
class LDrawFile:
<<initialization of LDrawFile>>
<<get commands from LDrawFile>>
<<method to determine if LDrawFile contains poly commands>>
Instantiation of an LDrawFile instance takes a required parameter path. From
this Path instance the file name, suffix and filename with parent are
extracted.
If the contents of the file are known during instancing the data can be passed in as a list of strings.
def __init__(self, path : Path, data : List[str] = []):
self.commands = data
self.path = path
self.name = path.name
self.suffix = path.suffix
self.pname = f"{path.parent.name}\\{self.name}"
For most files the contents will not be known in advance. When retrieving the contents the file gets lazily loaded and the contents cached in case this particular file is used again in a model.
Files are to be read in as UTF-8 files. At some point it would probably be good to perhaps support falling back to encodings of old files.
Reading of the file also strips already out empty lines.
def get_commands(self):
if len(self.commands)==0:
with self.path.open(encoding="utf-8") as f:
cmds = [l.strip() for l in f.readlines()]
cmds = [c for c in cmds if len(c) > 0]
self.commands = cmds
return self.commands
def contains_poly_commands(self):
cmds = self.get_commands()
for cmd in cmds:
if len(cmd) == 0: continue
if cmd[0] in ("2", "3", "4", "5"):
return True
return False
The heart of the LDraw specification revolves around line types. The specification essentially is a collection of lines whereby the first character of the line tells us what kind of operation has to happen.
For now we'll keep the importer simple and go where the path is easiest. We'll implement mainly line types 3, 4 and 1, with a smidge of 0 through the color definition language. Eventually we may want to include also 2 and 5 lines, but for just importing complete models for set dressing and rendering purposes this should get us going.
Line types 3 and 4 method to add a polygons, triangles and quads respectively.
Handling line types 3 and 4 means adding either a triangle or a quad. The form
of line types 3 and 4 are similar. A line type three starts with the number 3
followed by a color code and three triplets of values that are either integers
or floating point. For both line types we ignore the color code as we'll be
using the color codes defined on line type 1 commands. But we'll get into that
later.
The line type four has a similar format, starting with number 4, the color
code and then four triplets of values.
From the start number we parse how many triplets - vertices we'll be handling. This is the first character of the command string.
vertices = int(cmd[0])
elements = vertices * stride
to = start + vertices * stride
Each triplet is a coordinate in the polygon. The order in which the triplets are presented is also the vertex order.
The format also has a facility to tell the winding for polygons, but we ignore that here. We'll be using simple CW winding.
Each line has all their elements separated by white space. A simple split will suffice. We'll take only all parts after the color code. Parse all elements as floats, since they form vertices. If any element fails to parse as a float we'll skip adding this as a polygon.
d = cmd.split()[start:to]
try:
d = [float(f) for f in d]
except Exception:
return
With all numbers parsed we can add them as vertices to the mesh, increasing the
vertidx count for each item.
for i in range(0, elements, stride):
V = apply_transforms(d[i:i+stride], xforms)
m.Vertices.Add(*V)
vertidx = vertidx + 1
With all vertices added and vertidx adjusted we can now add the face
definition for either the triangle or the quad.
if vertices == 4:
m.Faces.AddFace(vertidx - 4, vertidx - 3, vertidx - 2, vertidx - 1)
elif vertices == 3:
m.Faces.AddFace(vertidx - 3, vertidx - 2, vertidx - 1)
To bring it all together we can say that to add a polygon from a command string
we first <<parse vertex count>> and <<parse vertices>>, then
<<add vertices to mesh>> followed by the final step <<add face to mesh>>.
def add_poly(m : Mesh, cmd : str, xforms : list):
global vertidx
stride = 3
start = 2
<<parse vertex count>>
<<parse vertices>>
<<add vertices to mesh>>
<<add face to mesh>>
flowchart
A[Model]
B[Assembly]
C[Geometry]
A --> B
B --> B
B --> C
C --> C
def load_model(model : LDrawFile):
lines = model.get_commands()
FILE_START = '0 FILE '
first_file = ''
if model.suffix.lower() == '.mpd':
cur_file = ''
file_data = []
for l in lines:
if l.startswith(FILE_START):
if cur_file != '':
add_virtual_file(model.path, cur_file, file_data)
if cur_file == '':
first_file = l[len(FILE_START):]
cur_file = l[len(FILE_START):]
file_data = [l]
else:
file_data.append(l)
add_virtual_file(model.path, cur_file, file_data)
start_part = get_ldraw_file(first_file)
load_assembly(start_part, [rhino_orient])
def load_assembly(part : LDrawFile, xforms : list):
cmds = []
cmds = part.get_commands()
cmds = [l for l in cmds if len(l)>0]
for cmd in cmds:
if cmd.startswith('1'):
d = cmd.split()
color_code = d[1]
materials[color_code].create_render_material()
rm = materials[color_code].render_material
xform = LDrawXform(cmd)
prt = ' '.join(d[14:])
_xforms = xforms[:] + [xform]
obattr = ObjectAttributes()
obattr.Name = clean_name(prt)
obattr.Visible = True
obattr.MaterialSource = ObjectMaterialSource.MaterialFromObject
obattr.RenderMaterial = rm
if prt.lower().endswith(".ldr"):
ldr_file = get_ldraw_file(prt)
if ldr_file.contains_poly_commands():
add_geometry(prt)
idef = update_idefs_dictionary(prt)
if idef != None:
xform = collate_transforms(_xforms)
sc.doc.Objects.AddInstanceObject(idef.Index, xform, obattr)
else:
print(f"Couldn't add part {prt}")
else:
load_assembly(ldr_file, _xforms)
else:
idef = get_part_idef(prt)
xform = collate_transforms(_xforms)
if idef != None:
sc.doc.Objects.AddInstanceObject(idef.Index, xform, obattr)
else:
add_geometry(prt)
idef = update_idefs_dictionary(prt)
if idef != None:
sc.doc.Objects.AddInstanceObject(idef.Index, xform, obattr)
else:
print(f"Failed to add part {prt}")
refresh(zoom_extents)
Actual geometry data is recorded in the .dat files. As all files they can
reference other files, the assumption is made that only .dat files are
referenced from .dat files. Also the assumption is that no .dat file has
been generated that references itself. That would currently lead to an infinite
loop, so something to improve on in the future.
For .dat files we'll look only at line type 1, 3 and 4. When seeing a line
type 1 we'll find the LDrawFile specified by the command and recurse into
load_geometry_from_file to load that part.
Line types 3 and 4 will be passed on to the <<method to add a polygon>>.
All these methods will be adding to the same Mesh instance.
The entry-point for adding geometry is really the add_geometry method, which
is given only the part name. If an existing instance definition with the cleaned
name is found that will be used with an early exit out of the method.
def load_geomety_from_file(part : LDrawFile, m : Mesh, xforms : list):
cmds = part.get_commands()
for cmd in cmds:
if cmd.startswith('1'):
d = cmd.split()
xform = LDrawXform(cmd)
prt = ' '.join(d[14:])
_xforms = [xform] + xforms[:]
try:
part_file = get_ldraw_file(prt)
except Exception:
print(f"\tERR: Failed getting part {prt}, skipping")
continue
load_geomety_from_file(part_file, m, _xforms)
elif cmd.startswith('3') or cmd.startswith('4'):
add_poly(m, cmd, xforms)
def add_geometry(part_name : str):
global vertidx
vertidx = 0
name = clean_name(part_name)
existing_idef = sc.doc.InstanceDefinitions.Find(name)
if existing_idef:
print(f"\tSkipping {part_name}, instance already created")
return
tmesh = Mesh()
mesh = Mesh()
obattr = ObjectAttributes()
obattr.Name = name
obattr.Visible = True
obattr.MaterialSource = ObjectMaterialSource.MaterialFromParent
ldraw_file = get_ldraw_file(part_name)
load_geomety_from_file(ldraw_file, tmesh, [id_xform])
tmesh.Normals.ComputeNormals()
tmesh.Compact()
<<weld mesh data>>
if mesh.Vertices.Count > 0 and mesh.Faces.Count > 0:
sc.doc.InstanceDefinitions.Add(obattr.Name, "", Point3f.Origin, mesh, obattr)
Next to actually reading in the geometry data and constructing a mesh out of it
the add_geometry commanda also does <<weld mesh data>>. The geometry defined
in .dat files is typically fairly coarse and does not include information
about normals. Doing a double step of first welding disjoin mesh parts, then
welding again with all the pieces put back together again gives relatively good
looking pieces in most cases.
meshes = tmesh.SplitDisjointPieces()
for submesh in meshes:
submesh.Weld(rhmath.ToRadians(60))
submesh.UnifyNormals()
mesh.Append(submesh)
mesh.Weld(rhmath.ToRadians(60))
mesh.Compact()
Be mindful though that this won't always work with every part geometry. For
instance minifig heads won't necessarily look always great. It all depends on
how the geometry was encoded in the .dat file.
The LDrawXform class is a simple helper class to handle conversion from the
LDraw transformation encoding in line types to Rhino.Geometry.Transform and
use those to apply the transforms expressed in a model and related files.
The main point is that the X, Y, Z coordinates of a command go in the right-most column of a Rhino transform.
class LDrawXform:
def __init__(self, data : str):
data = data.strip()
if len(data) > 0:
try:
d = [float(f) for f in data.split()[2:14]]
self.x = d[0]
self.y = d[1]
self.z = d[2]
self.a = d[3]
self.b = d[4]
self.c = d[5]
self.d = d[6]
self.e = d[7]
self.f = d[8]
self.g = d[9]
self.h = d[10]
self.i = d[11]
xform : Transform = Transform.Identity
xform.M00 = self.a
xform.M01 = self.b
xform.M02 = self.c
xform.M03 = self.x
xform.M10 = self.d
xform.M11 = self.e
xform.M12 = self.f
xform.M13 = self.y
xform.M20 = self.g
xform.M21 = self.h
xform.M22 = self.i
xform.M23 = self.z
self.xform = xform
except Exception as e:
self.xform = Transform.Identity
else:
self.xform = Transform.Identity
def set_xform(self, xform):
self.xform = xform
def transform_point(self, u : float, v : float, w : float):
p = Point3f(u, v, w)
p.Transform(self.xform)
return [p.X, p.Y, p.Z]
def get_xform(self):
return self.xform
rhino_orient = LDrawXform("")
rhino_orient.set_xform(
Transform.Rotation(
rhmath.ToRadians(-90.0),
Vector3f.XAxis,
Point3f.Origin
)
)
id_xform = LDrawXform("")
def apply_transforms(v, xforms):
for xform in xforms:
v = xform.transform_point(*v)
return v
def collate_transforms(xforms):
xform = Transform.Identity
for _xform in xforms:
xform = xform * _xform.get_xform()
return xform
LDraw files are created with the Y axis up, so we have a rhino_orient
transform that will be given to the top <<method to load a complete model>> so
that the end result will be rotated properly to the Rhino Z-axis up world.
def refresh(zoom_extents = False):
sc.doc.Views.Redraw()
if zoom_extents:
Rhino.RhinoApp.RunScript("ZEA", False)
Rhino.RhinoApp.Wait()
def clean_name(part_name):
part_name = part_name.removesuffix(".dat")
part_name = part_name.removesuffix(".DAT")
part_name = part_name.removesuffix(".ldr")
part_name = part_name.removesuffix(".LDR")
return part_name
import scriptcontext as sc
import math
from typing import Mapping, List
import Rhino
from Rhino.Geometry import Transform, Mesh, Vector3f, Point3f
from Rhino.Display import Color4f
from Rhino.DocObjects import ObjectAttributes, ObjectMaterialSource
from Rhino.DocObjects import InstanceDefinition
from Rhino.Render import ChildSlotNames, ContentUuids, RenderContentType
PbrNames = ChildSlotNames.PhysicallyBased
rhmath = Rhino.RhinoMath
from System.IO import DirectoryInfo, Directory, File, FileInfo
from System.IO import EnumerationOptions, SearchOption
from pathlib import Path
<<LDrawFile class>>
<<LDraw material class>>
<<LDrawXform class>>
<<initialize global variables>>
<<misc helper methods>>
<<method to prepare virtual files>>
<<method to add a virtual file>>
<<method to prepare instance definitions>>
<<method to update instance definitions>>
<<method to get InstanceDefinition instance>>
<<method to get LDrawFile instance>>
<<method to add a polygon>>
<<methods to load geometry>>
<<method to load an assembly>>
<<method to load a complete model>>
<<initialize color definitions>>
lib_path = "e:/dev/brickdat/ldraw"
prepare_parts_dictionary()
prepare_idefs_dictionary()
load_colors()
zoom_extents = True
fl : Path = vfiles["10030-1.mpd"]
load_model(fl)
refresh(zoom_extents)
sc.doc.Views.EnableRedraw(True, True, True)
Rhino.RhinoApp.RunScript("ZEA", False)
print("Done")