Accuracy, and how it was measured
Structura grades itself against ground truth on every release and publishes the result with its method, its thresholds and its gaps. The numbers on this page are read from the harness output committed alongside this site — the same file the product’s own evaluation screen renders. Read the caveat first: it is the most important thing here.
Where the ground truth comes from
Parametric, synthetic, seeded.
A generator in the repository emits each fixture from parameters: DXF plans per storey and elevation sheets with real layer conventions, blocks for doors and windows, and dimension text. A seed fixes every choice, so the same command reproduces the same drawings byte for byte.
Built from the parameters, never through the pipeline.
Each fixture also emits a parameter-level element list and a ground-truth IFC authored directly from those parameters. The pipeline never touches it. A conversion is therefore compared against what the building was defined to be, not against another run of the same code.
Matched pairs, then per-class scores.
Generated elements are matched to ground-truth elements by the geometric rules below. Matches give precision, recall and F1 per class; the matched pairs give dimensional error. Thresholds live in one versioned file and nowhere else, so a passing run cannot be produced by moving the bar in code.
What counts as a match
| Class | Rule | Tolerance |
|---|---|---|
| Walls | length-weighted centreline overlap against the ground-truth wall | overlap at least 0.60, within 300.0 mm |
| Doors and windows | centroid distance, and the same host wall | at most 300.0 mm |
| Slabs | footprint intersection-over-union | at least 0.70 |
| Roofs | footprint intersection-over-union | at least 0.70 |
| Rooms | boundary intersection-over-union | at least 0.70 |
| Columns and stairs | centroid distance | at most 300.0 mm |
| Geometry cross-check | the authored IFC solid is re-measured against the model it came from | within 500.0 mm |
Plans and elevations route
- IfcWall 43 in truthPrecision 1.00Recall 1.00
matched 43 · missing 0 · unexpected 0
- IfcDoor 40 in truthPrecision 1.00Recall 1.00
matched 40 · missing 0 · unexpected 0
- IfcWindow 70 in truthPrecision 1.00Recall 1.00
matched 70 · missing 0 · unexpected 0
- IfcColumn 22 in truthPrecision 1.00Recall 1.00
matched 22 · missing 0 · unexpected 0
- IfcSlab 9 in truthPrecision 1.00Recall 1.00
matched 9 · missing 0 · unexpected 0
- IfcStair 3 in truthPrecision 1.00Recall 1.00
matched 3 · missing 0 · unexpected 0
- IfcRoof 0 in truthNot graded
A pitched roof is not recoverable from plans, so no plans-route fixture carries one and eval/metrics.yaml publishes no threshold for it. Nothing was measured, so nothing is claimed.
matched 0 · missing 0 · unexpected 0
- IfcSpace 25 in truthPrecision 1.00Recall 1.00
matched 25 · missing 0 · unexpected 0
Dimensions, storeys and rooms
aster-house · birch-studio · cedar-court
dorm-house is excluded from 2D-route grading (eval/metrics.yaml 2d_exclude): its ground truth contains geometry the 2D route cannot recover. Supply a mesh-route model to grade it.
3D mesh export route
- IfcWall 47 in truthPrecision 1.00Recall 1.00
matched 47 · missing 0 · unexpected 0
- IfcDoor 41 in truthPrecision 1.00Recall 1.00
matched 41 · missing 0 · unexpected 0
- IfcWindow 75 in truthPrecision 1.00Recall 1.00
matched 75 · missing 0 · unexpected 0
- IfcColumn 22 in truthNot graded
The mesh route does not reconstruct columns. Every mesh conversion records that as an assumption; the ground truth still counts them, so they show as missing rather than as a score.
matched 0 · missing 22 · unexpected 0
- IfcSlab 10 in truthPrecision 1.00Recall 1.00
matched 10 · missing 0 · unexpected 0
- IfcStair 3 in truthNot graded
The mesh route does not reconstruct stairs. Same treatment as columns: recorded as an assumption, counted as missing, never scored.
matched 0 · missing 3 · unexpected 0
- IfcRoof 1 in truthPrecision 1.00Recall 1.00
matched 1 · missing 0 · unexpected 0
- IfcSpace 0 in truthNot graded
Rooms are a plans-route feature. A mesh-route building gets no IfcSpace at all and records the deferral as a building-scoped assumption.
matched 0 · missing 0 · unexpected 0
Dimensions, storeys and rooms
aster-house · birch-studio · cedar-court · dorm-house
No fixture is excused from grading on this route.
No real-drawing census published yet
The honest version of this section is a census: a set of real drawings from real projects, converted without tuning, with per-file outcomes — converted, converted with corrections, refused, and why. Structura has not published one. Until it does, this space stays empty rather than being filled with the synthetic numbers above wearing a different hat.
What that means for a reader deciding whether to trust the file: the scores above prove the pipeline does what it says on drawings that follow the conventions it was built against. They say nothing about a drawing set that does not. The limits page states exactly which conventions those are, and a trial is how you find out on your own sheets.
- Drawings Structura did not generate, from more than one drafting office.
- A ground truth for each one that was not produced by the pipeline being graded.
- Every file in the set reported, including the ones that were refused.
- Permission to publish the outcome without naming the drawing or its owner.