Augmented Machine Vision



Over the last few years, the proliferation of artificial intelligence applications in architectural design has led to conversations around authorship and agency in collaborative human-machine design processes. In 2016, Mario Carpo argued that we had arrived at a second digital turn in architecture - one defined by data surplus rather than data scarcity.1 We are now able, as designers, to work in this data rich environment by using machine vision in architectural design processes. Through unsupervised learning, neural networks can learn latent features of large datasets and generate novel outputs for a designer to evaluate. This is a fundamental shift in how we use digital tools not only for production but also design. By working in this way, this thesis leverages three dimensional latent space unsupervised learning to train a neural network on point cloud models and generate novel architectural form.

By training neural networks on purely architectural data sets they can begin to learn latent spatial and organizational patterns in everything from floorplans, construction details, building regulations in a specific region, and even 3D architectural solutions. The opportunities that machine vision brings to design can ultimately change the existing workflows and representation within architecture. These opportunities include the introduction of unsupervised learning

Traditional 3D architectural representation techniques epitomize the 3D world into two dimensions. This culture has culminated in BIM where coordination, synthesis and standardization take precedence to form an efficient process, for the production of architecture. New design languages have the potential to arise through the homogeneous segmentations of data within unsupervised learning.

Our work builds on 2D and 3D Generative Adversarial Networks as outlined by Goodfellow et al2 and Shu et al3 respectively. GANs such as 2D Neural Style for 2D image generation tasks have since been widely studied across many disciplines. GANs for 3D point clouds generation have rarely been studied in the computer vision field, yet alone the field of architecture. Three-dimensional geometric data offer an excellent domain for studying generative modeling.

Through the use of Tree-GAN, we generate novel point cloud outcomes from a modelled dataset of workspaces and offices using unsupervised learning. Point clouds are a homogeneous, expressive and compact representation of surface based geometry, with the ability to represent geometric operations while taking up little space. This design technique, whose mainstay is Tree-GAN, which has the ability to learn architectural features based on a three dimensional point cloud dataset. Tree-GAN can understand and generate semantic parts of objects without any prior knowledge by sampling a learnt latent space. It contains two networks, a discriminator and a generator. The generator, which is a tree structured graph convolutional network (Tree-GCN), performs graph convolutions based on tree structures. The Tree-GCN utilizes ancestor information from a tree and employs multiple supports to produce and represent 3D point clouds as outputs. The discriminator differentiates between real and generated point clouds to force the generator to produce more realistic points. The number of models used for training the GAN is crucial. For our thesis, a dataset of 960 workspaces was modeled to train TreeGAN.

This thesis seeks to understand the perception of machines through unsupervised learning and generative modeling. Several 3D generated language difficulties arise when interpreting the training results (Point Cloud-to-voxel; Point Cloud-to-mesh; and Point Cloud-to-Point Cloud). Through the process of curating a database of workplaces, training a neural network and generating new spatial outcomes, this thesis seeks to question disciplinary assumptions about agency and authorship in the design process as well as reflect on the ethics of designing in a data rich environment and the kinds of cultural values that reflect.


1 Carpo, Mario. The Second Digital Turn: Design Beyond Intelligence. Writing Architecture. Cambridge, Massachusetts: The MIT Press, 2017.
2 Goodfellow, Ian J., Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, 2014.
3 Shu, Dong Wook, Sung Woo Park, and Junseok Kwon. “3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions.” CoRR abs/1905.06292 (2019). http://arxiv.org/abs/1905.06292.
Faculty Advisor:
Matias del Campo