# data[].attributes.headline
Article "Art Made With Artificial Intelligence Wins at  State Fair" published 2022 by the Smithsonian Magazine written by Margaret Osborne. 

# data[].attributes.subject
Article "Art Made With Artificial Intelligence Wins at  State Fair" published 2022 by the Smithsonian Magazine written by Margaret Osborne. 

# data[].attributes.body
This article (published 2022 Titled "Art Made With Artificial Intelligence Wins at State Fair") talks about how the winner of the Colorado state fair art contest, Jason Allen, won $300 with A.I. art. This is a great debate about whether Jason Allen Deserved to win 1st place vs. artists who did NOT use A.I.

# data[].attributes.attachment_link.readable_text
Jason Allen, a video game designer in Pueblo, Colorado, spent roughly 80 hours working on his entry to the Colorado State Fair 's digital arts competition. Judges awarded him first place, which came with a $300 prize.

But when Allen posted about his win on social media late last month, his artwork went viral-for all the wrong reasons.

Allen's victory took a turn when he revealed online that he'd created his prize-winning art using Midjourney, an artificial intelligence program that can turn text descriptions into images. He says he also made that clear to state fair officials when he dropped off his submission, called Théâtre D'opéra Spatial. But over the last week or so, his blue ribbon has sparked an impassioned debate about what constitutes art.

We're watching the death of artistry unfold right before our eyes - if creative jobs aren't safe from machines, then even high-skilled jobs are in danger of becoming obsolete
What will we have then?
- OmniMorpho (@OmniMorpho) August 31, 2022

Allen, for his part, says he intended to make a statement with his artwork-and, considering the lively online discourse around it, he feels like he accomplished that goal, he tells the 's Anna Lynn Winfrey. He doesn't appear to have broken any official state fair rules, either.

Colorado's 150-year-old state fair is held each summer in Pueblo, a town roughly 115 miles south of Denver. Per the Chieftain, the fair's submission guidelines do not directly mention A.I.-generated art, but they define digital arts as "artistic practice that uses digital technology as part of the creative or presentation process."

The competition's two judges tell the Chieftain they were unaware that Allen had used A.I. to create his piece. But even if they had known, they still would've given him first place. They said they awarded the top prize based on the story Théâtre D'opéra Spatial tells, as well as the spirit it invokes.

"Even as the controversy is coming out, it's still invoking that, it's still causing an uproar," Cal Duran, one of the judges, tells the Chieftain. "That in itself is kind of remarkable."

Allen created Théâtre D'opéra Spatial by entering various words and phrases into Midjourney, which then produced more than 900 renderings for him to choose from. He selected his three favorites, then continued adjusting them in Photoshop until he was satisfied. He boosted their resolution using a tool called Gigapixel and printed the works on canvas.

Allen entered all three pieces into the competition, paying an $11 submission fee for each one. He listed them for sale for $750 a piece, a price he came up with by considering quotes from other artists, he tells the Chieftain.

The winner, Théâtre D'opéra Spatial, "depicts a strange scene that looks like it could be from a space opera, and it looks like a masterfully done painting," Matthew Gault writes for . "Classical figures in a Baroque hall stare through a circular viewport into a sun-drenched and radiant landscape."

Allen said he believes the criticism of his work stems from fear. Artists are concerned that technology will one day become so sophisticated that they'll be out of jobs.

"To developers and technically minded people, [A.I. is] this cool thing, but to illustrators, it's very upsetting because it feels like you've eliminated the need to hire the illustrator," cartoonist Matt Bors, founder of the , tells the 's Charlie Warzel. "The bottom came out of illustration a while ago," Bors adds, "but A.I. art does seem like a thing that will devalue art in the long run."

As Cade Metz wrote for the New York Times earlier this year, A.I. art tools may also have other unintended consequences, especially when bad actors get their hands on them. These technologies have the potential to spread disinformation and create deep fakes, an umbrella term for deceptive photos and videos that are digitally altered.

The controversy around Allen's artwork may prompt the Colorado State Fair to change its rules or possibly even create a standalone A.I. category. But in the meantime, as state fair spokesperson Olga Robak tells the Chieftain, it's sparking a "broader conversation about how do we decide what art is, and how do we judge it appropriately?"

# data[].attributes.attachment_link.title
Art Made With Artificial Intelligence Wins at State Fair

# data[].attributes.attachment_link.description
Artist Jason Allen placed first in a Colorado contest, generating debate about A.I.'s role in art

# data[].attributes.headline
Article "AI Art and Its Impact on Artists" Published in 2023 by AIES written by : Harry H. Jiang, Lauren Brown, Jessica Cheng, Abhishek Gupta, Teresa Zohar, Timnit Gebru.

# data[].attributes.subject
Article "AI Art and Its Impact on Artists" Published in 2023 by AIES written by : Harry H. Jiang, Lauren Brown, Jessica Cheng, Abhishek Gupta, Teresa Zohar, Timnit Gebru.

# data[].attributes.body
This article Made in 2023, written by Jiang et al, Explains how much The AI industry makes money ($48 billion) just from A.I. alone. These A.I. based platforms include midjourney, DALL-E, and stable Diffusion.

# data[].attributes.headline
Article "AI Art in Architecture" published in 2023 by AI in Civil Engineering by Joern Ploennigs and Markus Berger.

# data[].attributes.subject
Article "AI Art in Architecture" published in 2023 by AI in Civil Engineering by Joern Ploennigs and Markus Berger.

# data[].attributes.body
explores how A.I. could be used as a tool for work like architecture and how it can be used for sketches and possibly interior design

# data[].attributes.attachment_link.readable_text
Abstract

Recent diffusion-based AI art platforms can create impressive images from simple text descriptions. This makes them powerful tools for concept design in any discipline that requires creativity in visual design tasks. This is also true for early stages of architectural design with multiple stages of ideation, sketching and modelling. In this paper, we investigate how applicable diffusion-based models already are to these tasks. We research the applicability of the platforms Midjourney, DALL\(\cdot\)E 2 and Stable Diffusion to a series of common use cases in architectural design to determine which are already solvable or might soon be. Our novel contributions are: (i) a comparison of the capabilities of public AI art platforms; (ii) a specification of the requirements for AI art platforms in supporting common use cases in civil engineering and architecture; (iii) an analysis of 85 million Midjourney queries with Natural Language Processing (NLP) methods to extract common usage patterns. From this we derived (iv) a workflow for creating images for interior designs and (v) a workflow for creating views for exterior design that combines the strengths of the individual platforms.

Similar content being viewed by others

Discover the latest articles, books and news in related subjects, suggested using machine learning.

1 Introduction

1.1 Motivation

Recent versions of AI art generation tools are reaching levels of output quality that allow them to support architects and designers in parts of their daily work. This gained them the attention of several architects across the globe, reflected also in a special edition of the AEC magazine.1

Beyond this public discussion, we want to take a deeper look into the current capabilities of this technology and analyse qualitatively as well as quantitatively what benefits it offers to architects now and in the future. In this paper we therefore review the technology behind the three leading commercial AI art platforms and evaluate what use cases they currently support in architecture. We investigate how users are already using these tools by analysing more than 85 million public queries from one of these platforms with Natural Language Processing (NLP) workflows. Finally, we present a collection of case studies in which we apply the practical experience we collected in working with these systems.

The novel contributions of this paper are thus:

a comparison of the technology of three leading AI art platforms;

an analysis of how well current AI art tools can handle different use cases;

a mapping for what specific design tasks each platform supports;

an NLP analysis of how these platforms are used for architecture today;

a collection of practical workflows for specific architectural design tasks.

1.2 State of the art in generative methods

The potential benefits of AI art platforms for creative work is hard to overstate. These AI art platforms are all using generative machine learning models, specifically text-to-image generative models. Despite their specialization on generating images, many are based on the natural language model GPT-3. GPT-3 is trained to generate text that completes a textual input query Brown et al. ( 2020). More precisely, it predicts the next possible combinations of words to an input text. The specific Image GPT-3 models used by AI art platforms are trained to instead predict the next cluster of pixels, called patch, in an image for a given input text.

The most recent generation of generative models combines natural language and so-called diffusion models. The idea for diffusion models was first proposed in Sohl-Dickstein et al. ( 2015), in which (structured) image information is slowly destroyed through a forward diffusion process that introduces noise into the image data and then generated anew through a reversed diffusion process. This reverse process generates completely new image data, as the original information was fully destroyed by noise. This approach was constantly improved over the years with a strong focus on optimizing the underlying neural network architectures Ho et al. ( 2020), resulting in several variants, like OpenAI's GLIDE model Nichol et al. ( 2022). It consists of an encoder that creates a text encoding based on the user prompt, a model implementing the diffusion based on this text encoding, as well as an upsampler that upscales and denoises the result. Current diffusion models often implement the process of text encoding and the association of those text encodings with image parts with the CLIP (Contrastive Language Image Pre-training) architecture presented by Radford et al. ( 2021) and used by DALL\(\cdot\)E 1.

One of the currently most advanced incarnation of the technology is DALL\(\cdot\)E 2 (from here on out simply referred to as DALL\(\cdot\)E), based on the unClip method developed by Ramesh et al. ( 2022). It uses one image encoder for both text and images into a diffusion-based joint representation space (the prior). The image generation is done by a similarly trained decoder, which translates the prior's encoding back into an image. Another main platform in the field is the open-source model Stable Diffusion, which is also based on the CLIP text encoder. The third contender, Midjourney, does not published their models, but it is assumed that it is using a similar structure.

This kind of diffusion model architecture can solve various image generation tasks. Is a completely new image generated from a user-written text prompt then a text-to-image model (txt2img) is used. Is an existing image modified based on a text prompt then Image-to-image models (img2img) are used. They either change the style or arrangement of the image based on the text prompt. If a certain part of the original image was deleted, the model can replace it with entirely new content based on the prompt. This approach is called inpainting by Lugmayr et al. ( 2022). A similar approach is outpainting, also called uncropping by Saharia et al. ( 2022), which adds additional content outside the image. If a user is requesting changes to the original image without manually deleting or masking out parts, then this is called image editing. Image editing is not yet available in commercial AI art platforms, but there is recent work in single-image editing through text prompt, for example by Kawar et al. ( 2022). Is another diffusion step applied to add more details to the image in a higher resolution then it is up-scaled, a method also called super-resolution by Saharia et al. ( 2022). Often platforms offer multiple or all these methods, with configurable weights between the individual images and words.

Assessing the quality of all these models and architectures systematically is difficult. Attempts at quantitative evaluation are being made such as Borji ( 2022). However, in such cases it is difficult to evaluate subjective metrics like style, i. e., whether an oil painting style looks better than a photorealistic one.

As for research into use cases from architecture, Seneviratne et al. ( 2022) describe a systematic grammar for using DALL\(\cdot\)E for the purpose of generating images in the context of urban planning. They open-sourced 11.000 images generated with Stable Diffusion and 1.000 created with DALL\(\cdot\)E by that grammar. They found that, though, many realistic images could be generated, the model has weaknesses in creating real-world scenes with a high level of detail. But, these models advance quickly and DALL\(\cdot\)E 2 outperforms DALL\(\cdot\)E 1 significantly.

Recent progress was also made in generating video (Ho et al., 2022), 3D models via point clouds (Luo and Hu, 2021; Zhou et al., 2021; Zeng et al., 2022), and even 3D animation data (Tevet et al., 2022). While this is beyond the scope of this paper, especially the generation of 3D models will be revolutionary for architecture. The 3D workflows would be similar to the image-based workflows we present.

2 Model Architectures and Interfaces

Although the concept of generative art has been a research area for years, it only entered public perception with the advent of publicly available diffusion model platforms like DALL\(\cdot\)E, Midjourney or Stable Diffusion, which combine txt2img, img2img, inpainting and upscaling into easy-to-use workflows.

These models are not only competing on the technical level, but also in terms of user experience. Midjourney2 directly interacts with its community by sharing queries across public (or private) channels in the Discord messaging app. They do not provide a dedicated user interface, but simply return the generated images as a chat response to the query. Direct interactions are possible with attached links that usually result in new queries. In contrast, DALL\(\cdot\)E 23 is only accessible by individuals through a dedicated web-based user interface with authoring and editing tools. It provides a simply query interface without additional query parameters. Stable Diffusion on the other hand is released as open source, which fosters a plethora of community-created tools that are usually used more than the official web-interface4.

The internal model architecture as well as the interface paradigm influences how these models can be utilized. In Fig. 1 we illustrate some of the similarities and differences between DALL\(\cdot\)E 2, Stable Diffusion (v2.1), and Midjourney (v4). The core workflow of foundational models in grey are similar across technologies as stated before. Differences lie in the workflows that they provide, which often results from their different interface approaches.

Model architecture and image generation process in different models. Grey elements show the AI workflow, coloured elements the user interaction

It is apparent in Fig. 1 that while the core workflow in grey is similar, the ways to refine their results do vastly differ. Only Midjourney offers successive upscaling of resolution and does so with multiple different sizes. Thus, Midjourney's workflow focuses on generating and comparing different image variants and then upsampling the best results in multiple ways, with limited possibilities to remix the original text prompt throughout.

In contrast, both DALL\(\cdot\)E and Stable Diffusion allow direct editing of uploaded images or previous results. They do not provide traditional image editing tools like drawing, filling, layering, or stamping. Instead, all image editing must be done by img2img-based operations like erasing sections (inpainting) or extending the canvas (outpainting). All networks have ways to create images of different sizes and aspect ratios, either by specifying the size in the query or by altering it later through outpainting.

Notably, image generation only takes a few tens of seconds in all models, making it fast enough to use in creative sessions alone or with clients. All three models also allow importing external images in some capacity. Therefore, they can easily be combined into composite workflows, as shown in Sect. 5.2.

One aspect not included in Fig. 1 is the training data. There is little information on the training data used for DALL\(\cdot\)E and Midjourney. However, Stable Diffusion was trained on the LAION-5B dataset (Rombach et al., 2022), which is based on image and text data scraped from the web. Similar internet datasets are very likely the source for Midjourney and DALL\(\cdot\)E. However, it is evident that either biased by the training data or the training process, these models have developed very different image styles. DALL\(\cdot\)E and Stable Diffusion are good in generating both drawn images as well as photorealistic outputs. Midjourney tends towards a more artistic style, especially with earlier model versions. But, with carefully chosen keywords most styles can now be targeted by all models.

3 Architectural use cases

Given the discussed differences of the platforms, they vary in the architectural use cases that they support. To analyse this, we collected a series of use cases where architects and planners normally create or edit images. A direct comparison between the platforms and the use cases is often complicated, because the style and quality of the results heavily depends on the input prompts. We identified that the more differentiating factor is, whether a platform supports a specific technical feature that is required to realize the use case.

Therefore, we evaluated for each use cases what features they require and how qualitatively well they are supported by each platform. In addition to the image operations explained in the previous chapter, we also consider support for architectural semantics, i. e., structured knowledge that goes beyond common image training datasets. Table 1 shows the results for the use cases that we discuss below:

Ideation: Developing ideas by randomly generating images for inspirations. This is what txt2img models are made for and it works splendidly. Additional image prompts can add style and object references.

Sketches: Drawing architectural sketches with specific target style and items. This works well for common examples in the training data, but, less so with specific requests.

Collages: Combining and filling existing images with life by adding people and objects. This can be done through inpainting for individual items, but not through generic requests like: "Add many people".

Image combination: Taking multiple existing image elements (for example multiple buildings), arranging them on a canvas and then creating a coherent composite image.

Build variants: Taking an existing sketch or picture and generating versions in which certain elements are altered (like adding a garage). This works well through inpainting.

Style variants: Taking an existing image and transforming its style (e. g., a sketch to photorealistic art deco) without changing content. This works well with certain models.

Construction plans: Creating detailed layout plans to establish spatial relations. This rarely works as the models do not understand the semantics of line styles, areas, etc.

Exterior design: Finding style and feeling for a building and the surrounding area/landscape. This works well for common scenarios.

Interior design: Finding a style or feeling for an interior space. This also works well in many scenarios.

Creating txtures: Creating tiled patterns to serve as surface materials for 2D or 3D models. This is currently a unique feature of Midjourney.

Table 1 Top part-comparison of platforms with their supported features; Lower part-Mapping of architectural use cases to features

Even though Stable Diffusion seems to support fewer features than DALL\(\cdot\)E 2, it's main advantage can not be overstated: It is possible to run it locally and train it on own data to introduce e. g. new architectural concepts. Together with the high quality of Stable Diffusions outputs, this makes it the most potent of the three models to be specialized on architectural drawings.

4 Analysis of architectural queries

We also explored how people use these AI art platforms in practice. We analysed about 85 million user queries that we collected over one year since Jan. 30\(^\text {th}\), 2022 from Midjourney. It is the only platform for which many user queries are publicly visible. Midjourney uses the Discord messaging app as its main interface, which allowed us to monitor the public channels for queries that we consider to be of an architectural nature. We selected queries containing either the word "architect", "interior" or "exterior" design or one of 38 architectural keywords like "building", "facade", or "construction" (listed in Fig. 2 (b) and5). We identified these keywords by selecting only those from architectural glossaries6, 7 that co-occurred in at least 10 % of all cases with "architect", "interior" or "exterior" in the queries. We also added to the list of keywords the names of 941 famous architects from Wikipedia8 as we noted that several queries refer to their style by naming them. By applying these filters, we identified 5.7 million queries (6.7 %) with potential architectural intent including 2.2 million queries (2.6 %) explicitly containing "architect", "interior" or "exterior" design.

In the next steps we filter out stopwords (e. g. "a", "and", "the") and build a Word2Vec model (Mikolov et al., 2013) from these queries to get a model of the occurrence and co-occurrence of terms in these queries. For understanding the results, it is important to know that most Midjourney users do not formulate full sentences, but a prompt is rather a collection of terms that refer to the content, style, or render quality of the targeted image.

Results from analysing Midjourney queries

Figure 2 visualizes the main results of our analysis. Figure 2(a) shows the most frequent terms used in the filtered queries. The colour blue represents the frequency across all 85 million queries, red is the frequency within the filtered 5.7 million queries and green within 2.2 million queries explicitly containing "architect", "interior" or "exterior" design. Note that red, green, and blue are mutably inclusive and overlap. The top 10 of words has a similar frequency across all three classes. Many of those refer to Midjourney style commands like "detailed", "realistic", or "cinematic". However, some terms like "black", "creative" or "full" have high overall frequency, but, low frequency in architectural context. Other terms like "architecture", "interior", "house" do only occur exclusively within our filtered results as they are part of our keyword list.

Figure 2(b) shows our extended keyword list and their respective frequency. As we filter on these keywords, their total frequency is identical with the filtered one and we do not display it. It is of note that "architecture" and "interior" keywords are the most and third frequent keyword. It is notable that "interior" is six times more popular than "exterior" design as keyword, but, this simply may be that users refer to it implicitely trough "architecture".

Figure 2(c) shows the frequency of famous architects that we extracted from 430.333 queries that referred to at least one of them. Zaha Hadid is by far the most frequently queried architect, given her well recognizable parametric style that is popular in the community of people experimenting with AI tools. Michelangelo and William Morris are second and third. The low red bar shows that they are usually not used in architectural context but for their art contributions. Adrian Smith is also often used in other context probably referring to the musician. The architects Frank Lloyd Wright, Tadao Ando, Frank Gehry, Antoni Gaudi, Lebbeus Woods, and Peter Zumthor complete the top 10 and are often used in explicit architectural context given the strong red bar.

Figure 2(d) shows the links between keywords and the most likely connected term. We analysed this by predicting with the Word2Vec for each keyword on the left the most probably co-located word on the right, weighted by probability. Interesting combinations here are links between interior-design, floor-plan-drawing, architecture-visualization, cathedral-gothic, or swimming-pool. From this it is possible to build an of auto-complete function for architectural queries.

Figure 2(e) shows the mean length of queries depending on whether they got upscaled, remastered or left in draft mode. A draft mode image is of low image size, so users will normally upscale or remaster them if they like one of the variants. It is notable that for the medium upscale options as well as the remastered version the mean query length increases above 35 terms per query in comparison to 30 terms for draft mode queries. We also manually classified the most frequent 150 terms into the categories: style, content, quality. It is notable that for the upscale and refined queries, the percentage of style terms increases significantly from 6.6 % to 8.3 %.

Figure 2(f) shows the mean number of iterations needed to develop a query. We classify a query as iteration if the same user is rerunning the same or extended query within 30 min. 54 % of all unique queries (single, 852k) are run only once. The other half of the queries are improved in multiple iterations. Queries that remain in draft mode require about 3.7 steps. 7.8 % of the queries are good enough to be upscaled require about 6.75 steps in total. They are upscaled after 4.1 draft steps into different variants (light, medium/beta, max). 5.3 % queries that are remastered take about 5.1 iterations. They have only 1.4 draft mode queries, but 1.2 remastering steps, and 2.3 final upscale steps.

This analysis illustrates that users do not come-up with perfect queries from scratch. We can derive multiple insights:

6.7% (5.7 million) of all queries are related to architecture;

"architecture" is the most popular keyword, and "interior" is six times more popular than "exterior" as a keyword;

only 13.1 % of all unique queries upscaled or remastered;

these queries are usually refined in more than 4 iterations in mean;

these queries usually contain more keywords specifying style and quality.

5 Case studies

In this section, we present some refined workflows for architects to utilize the strengths of all three AI art platforms. These workflows are based on the learning of our analysis and by a large number of experiments to identify the workflows leading to the best results. It should be noted that Midjourney v3 is in use here, which raises the importance of the remastering step. Remastering can usually be skipped as of model version 4. As we have identified in the analysis, users rarely run only a single query and gain a perfect result-instead they iterate many times. This requires a good understanding of effective workflows to avoid dead ends and come to adequate results quickly.

5.1 Interior design-comparing the models

First, we will look at how the models perform purely on their own. The example will be an interior-design scenario. We start with a simple query without any special command of a platform. On all three we try to generate a high-quality rendering of a room using the prompt "cozy living room, wood paneling, television, large sofa, natural light, lived in, realistic, full view". We developed this prompt by watching common prompting patterns in the Midjourney query data and testing out different iterations across models until we arrived at that final version. Each comma is considered as topic separator by the AI art platforms. Thus, we are asking for an image of a (i) cozy living room that (ii) has wood paneling; (iii) contains a TV; (iv) a large sofa, etc. With this we ensure that the resulting image should contain similar elements across platforms.

Minimal workflow for Midjourney ( a- d), DALL\(\cdot\)E 2 ( e- h), and Stable Diffusion ( i- l) for the given query

Midjourney starts out with several results that are stylized or of strange perspective, visible in Fig. 3(a, b). The first upscale of the chosen interior design in (c) greatly improves material quality and overall detail, but the central sofa remains as an incoherent form in the centre of the image. Anytime the normal image output does not attain a sufficient level of cohesion and realism, we can invoke the "remaster" step, shown upscaled in (d). However, even this last result contains smaller perspective errors, which are difficult to fix without intervention through manual image editing.

The DALL\(\cdot\)E 2 first results in Fig. 3(e) shows that it struggles to correctly response to the "realistic" term in the query. Once one of the two realistic variants is picked, the next variant generation step in (f) creates more useful results. To remove perspective or coherence errors we mask out certain areas of the image in (g) to generate inpaint variants. The final variant is shown in (h). Of note is that even the inpainted regions react correctly to the previously established lighting of the scene. This result is of good quality, but cannot be upscaled any further within the web interface.

Stable Diffusion starts out in Fig. 3(i) with much stronger results than its two contenders, generating images that incorporate the "realistic" and "lived in" aspect of the query very well. These rooms look like inhabited and not like artificial renderings. However, all images seem like close-ups of a proper interior view. Which is why we do not just use inpainting in (j) to fix errors, but also add additional canvas space for outpainting. This results in some quite incoherent variants for the outpainted areas. After some additional iterations the variant in (l) was selected as the best.

Overall, Stable Diffusion performs best in this scenario. All of its first variants were realistic and showed no real deficiencies in the later steps. Only during the final outpainting stage it was necessary to manually smooth the transitions. Midjourney generated a final result of similar quality, however offered a weaker beginning selection, with not all images containing all prompted elements (e. g. missing TV) and frequent perspective errors and other visual faults.

5.2 Exterior design-combining the models

The best results can be archived by combining the strengths of all three models and knowing their specific command keywords. For example, one of the first hints that DALL\(\cdot\)E shows new users is the information that the keyword "digital art" can significantly improve many prompts, which is deeply related to the data it was trained on.

In the case of Midjourney, refinement starts as a process of including and removing certain phrases within the prompt to get as close to a desired style as we can. These phrases can be very convoluted. It often helps to include the kinds of modelling software that would create the desired kind of image (like "octane render" or "cinema3D") or even quality signifiers like "top 10 on artstation" into the prompt. A much more directed way to influence queries is the use of word weights, image references and parameters, which add additional parameterization to the prompt system.

In contrast, DALL\(\cdot\)E and Stable Diffusion provide more control over changes with their in- and outpainting tools. Once we understand these tools and strengths of the platforms, we can combine them into a more flexible workflow. In the following ideation workflow, we will start with a Midjourney prompt to create a desired scene. While Stable Diffusion tends to generate better looking first results, Midjourney is a strong contender once the remaster step is done, and with its workflow excels at free-form ideation, which makes it the most appropriate starting point for an exterior design. From the Midjourney remaster stage, we will then refine any errors or undesired results through in-/outpainting in DALL\(\cdot\)E and Stable Diffusion to attain the targeted result. An overview of the workflow is provided in Fig. 4 and we will explain it along the results in Fig. 5. The workflow highlights how the different image editing steps can be combined to get the best results independent of the AI tool used. The specific models most appropriate for each step may change with newer versions. The logic behind the workflow is targeting the best image quality, by: (1) generating the image; (2) upscaling it; (3) extending the canvas; (4) finally editing details with inpainting.

The proposed combined workflow over Midjourney, DALL\(\cdot\)E and (optionally) Stable Diffusion

Figure 5(a- c) shows the beginning stages of an idea as generated in Midjourney. The query that led to this particular result was "single-family home with garden, full exterior view, modern architecture, photo, sunlight". Multiple keyword arrangements and weights on different terms were tried before this result was selected, remastered and then upscaled. It is also possible to start with one or more reference images, though that technique was omitted here.

Refinement and variant generation in Midjourney ( a- c), DALL\(\cdot\)E 2 ( d- f), and Stable Diffusion for a walkway ( g) and a second story ( h, i)

The result was then uploaded to the DALL\(\cdot\)E, where it was outpainted in Fig. 5(d) to create a wider viewing angle and subsequently inpainted in Fig. 5(e, f) to replace unwanted details like the cables hanging in the air or the differently colored windows.

After unsuccessful attempts to add a paved path from the sideway to the entrance through inpainting in DALL\(\cdot\)E, we transferred image (f) to Stable Diffusion. We erased the walkway and replaced it with inpainting using a modified query without a garden reference (as the walkway would often be overgrown by plants) and adding "paved" early in the prompt (as earlier keywords are weighted higher). This resulted in the prompt "single-family home, paved between sidewalk and door, full exterior view, modern architecture, photo, sunlight" with the result in image (g).

The speed of the process allows to develop this design together with a client. He may also suggest major changes like adding a second story or some other roof element. This can easily be accomplished via inpainting by erasing the roof and part of the sky and a slightly changed prompt specifying the style of the new image. Figure 5(h, i) shows the result for the Stable Diffusion query "single-family home, two stories, clear blue sky, curved roof, full exterior view, modern architecture, photo, sunlight" after the roof and central area of the sky have been erased and the canvas has been extended upwards to give more room for roof elements. Note that some latent effects like the tree branches reaching into the image are hard to avoid.

5.3 Common limitations

Working with current AI models is an trial-and-error process. They rarely present perfect results on first try, but expect the user to pick the best variants and refine their prompts multiple times. This may not be straight forward, but giving a group of architecture students the task to draw a design with similarly rough specifications would also result in many variants and would be way more time consuming.

Nonetheless, many variants that users explore fail entirely. This is evidenced by the high number of single step queries without upscaling in our Midjourney analysis in Fig. 2(f). Some of the common failure cases an architect would encounter while using these models are shown in Fig. 6. Case (a) shows the result for a floor plan query. It does well in imitating the style of bold lines for walls and thin lines for objects, but is completely nonsensical on closer look. This is due to the fact that AI art tools replicate the style, but have no semantic understanding of the meaning of the lines in a floor plan. Case (b) shows the result of a query with multiple specific technical terms, which are also somewhat ambiguous in themselves. The specific architectural element "bow window" was turned into a window with a bow over it. The "clock gable" was represented by a different gable element with a clock below it. Case (c) shows a query for a landscape design with five buildings, which invokes the common problem that these AI tools are just bad at counting and complex spatial arrangements beyond foreground and background.

6 Conclusion

In this paper we looked at AI art generation tools in their applicability to architecture and civil engineering. We compared three of the currently available AI art platforms and identified use cases that can be tackled now or are soon to be unlocked. To understand how users are already using these tools we analyzed millions of queries providing some insights on how users iterate. Finally, we presented two workflows, for interior and exterior design, with the latter combining the strengths of the different platforms.

The various use cases shown in this paper illustrate the strong potential for AI tools in architecture. The AI platforms still struggle with more complex prompts, usually due to missing semantic understanding of the image content. A floor plan for example is not just a collection of lines. These lines carry semantic and contextual information, like that they form walls enclosing a room with a door to get in. As we already have Building Information Models (BIM) that provide this semantic information it is just a matter of time that new diffusion models will arrive that are trained specifically on these data sets, and it is likely that they will be able to fulfil all requirements from Table 1.

Nonetheless, the high number of 5.7 million queries with architectural context that we identified show that the tools are already adopted. In the coming months and years these platforms will further improve. We can already see workflows across tools that converge toward Fig. 4. In the end, single platforms will deliver the full workflows for use cases like ideation, collages, build and style variants that will drastically improve productivity and creativity. It is thus likely that these tools will first be adopted for brain storming sessions with clients and for competitions. We observed that the designs created by current AI tools tend toward organic forms, openwork facades and complex arrangements that break out of the common minimalistic modern design. With the ongoing research on automated evaluation of structure dynamics and in the field of additive and robotic construction technologies, more and more of these designs are becoming structurally and financially possible. This may form a perfect storm situation leading to a new generation of architecture styles based on AI-generated designs.

Data availibility

The Midjourney datasets generated and analysed during the current study are not publicly available for data privacy reasons. Code for retrieving the data can be made available from the corresponding author on reasonable request.

References

Borji, A. (2022). Generated faces in the wild: Quantitative comparison of stable diffusion, midjourney and DALL-E 2. arXiv preprint http://arxiv.org/abs/2210.00586.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., & Agarwal, S. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-901.

Google Scholar

Ho, J., Jain, A., & Abbeel, P. (2020). Denoising diffusion probabilistic models. Nips, 33, 6840-6851.

Google Scholar

Ho, J., Salimans, T., Gritsenko, A.A., Chan, W., Norouzi, M., & Fleet, D.J. (2022). Video diffusion models. ICLR workshop on deep generative models for highly structured data.

Kawar, B., Zada, S., Lang, O., Tov O, Chang, H., Dekel, T., Mosseri, I., & Irani, M. (2022). Imagic: Text-based real image editing with diffusion models. arXiv preprint http://arxiv.org/abs/2210.09276.

Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., & Van Gool, L. (2022). Repaint: Inpainting using denoising diffusion probabilistic models. CVPR (pp. 11461-11471).

Luo, S., & Hu, W. (2021). Diffusion probabilistic models for 3D point cloud generation. CVPR (pp. 2837-2845).

Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., & Dean, J. (2013). Distributed representations of words and phrases and their compositionality. Nips (Vol. 26).

Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., & Chen, M. (2022). Glide: Towards photorealistic image generation and editing with text-guided diffusion models. ICML.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G. (2021). Learning transferable visual models from natural language supervision. ICML (pp. 8748-8763).

Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022). Hierarchical text-conditional image generation with CLIP latents. arXiv preprint http://arxiv.org/abs/2204.06125.

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022, June). High-resolution image synthesis with latent diffusion models. CVPR (p. 10684-10695).

Saharia, C., Chan, W., Chang, H., Lee, C., Ho, J., Salimans, T., Fleet, D., & Norouzi, M. (2022). Palette: Image-to-image diffusion models. ACM SIGGRAPH (pp. 1-10).

Saharia, C., Ho, J., Chan, W., Salimans, T., Fleet, D.J., & Norouzi, M. (2022). Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence.

Seneviratne, S., Senanayake, D., Rasnayaka, S., Vidanaarachchi, R., & Thompson, J. (2022). DALLE-URBAN: Capturing the urban design expertise of large text to image transformers. International Conference on Digital Image Computing: Techniques and Applications.

Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., & Ganguli, S. (2015). Deep unsupervised learning using nonequilibrium thermodynamics. ICML (pp. 2256-2265).

Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-Or, D., & Bermano, A.H. (2022). Human motion diffusion model. arXiv preprint http://arxiv.org/abs/2209.14916.

Zeng, X., Vahdat, A., Williams, F., Gojcic, Z., Litany, O., Fidler, S., & Kreis, K. (2022). LION: Latent point diffusion models for 3D shape generation.

Zhou, L., Du, Y., & Wu, J. (2021). 3D shape generation and completion through point-voxel diffusion. CVPR (pp. 5826-5835).

Download references

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

AI for Sustainable Construction, University of Rostock, Rostock, Germany

Joern Ploennigs & Markus Berger

Authors

1. Joern Ploennigs

2. Markus Berger

Contributions

JP: Ideation and manuscript, Midjourney Study, Writing Sects. 1, 3, 4. MB: Use Case Experiments, Writing Sects. 2, 3, 5. Both authors reviewed and edited all sections as well as read and approved the final manuscript.

Corresponding author

Correspondence to Joern Ploennigs.

Ethics declarations

Competing interests

The authors have no financial or proprietary interests in any material discussed in this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Reprints and permissions

About this article

Cite this article

Ploennigs, J., Berger, M. AI art in architecture. AI Civ. Eng. 2, 8 (2023). https://doi.org/10.1007/s43503-023-00018-y
Download citation

Received:

Revised:

Accepted:

Published:

DOI: https://doi.org/10.1007/s43503-023-00018-y

Share this article

Anyone you share the following link with will be able to read this content:

Get shareable link

Provided by the Springer Nature SharedIt content-sharing initiative

Keywords

# data[].attributes.attachment_link.title
AI art in architecture - AI in Civil Engineering

# data[].attributes.attachment_link.description
Recent diffusion-based AI art platforms can create impressive images from simple text descriptions. This makes them powerful tools for concept design in any discipline that requires creativity in visual design tasks. This is also true for early stages of architectural design with multiple stages of ideation, sketching and modelling.

# data[].attributes.headline
Article  "AI's Impact on Artists" Published in 2023 and written by Joseph Wakelee-Lynch published by LMU magazine. 

# data[].attributes.subject
Article  "AI's Impact on Artists" Published in 2023 and written by Joseph Wakelee-Lynch published by LMU magazine. 

# data[].attributes.body
explores how A.I. has been stealing artists work through data and how it is effecting artist "Karla Ortiz". This shows how A.I. can mimic artists styles and steal their content and revenue.

# data[].attributes.attachment_link.readable_text
In the past, a masterwork of art might have been painstakingly copied for weeks to be sold to an unwitting art lover or collector. Today, with AI technology, the work of just about any great artist in human history can be mimicked in moments to produce an image on spec by a user - and openly sold.

AI generators rely on databases of already existing art, the internet being a massive trove of readily available resources. Images are "scraped" from the internet and used as data. Generators are "trained," or taught, to recognize art. Based on human prompts and drawing from thousands or millions of images, They can produce a new image that meets particular specifications.

Many images used as the basis of replication are in the public domain, but artists charge that copyrighted images are used as well, raising questions about the uncompensated use of the intellectual property of others. Artists who make their living by their work may go uncompensated for the use of their creations, or even find themselves priced out of some markets by less expensive AI-generated images.

"If millions of artists' works are being unknowingly and/or unwantedly used to enable (to train) AI systems," says Selwa Sweidan, clinical assistant professor of multimedia arts in the LMU College of Communication and Fine Arts, "so far it seems to happen wholly without the artists' consent and without their benefit."

Sweidan points out that AI platforms enjoy an asymmetry of power in the form of economic resources that individuals and few entities can make use of and a high barrier to entry that limits competitors. But redress is possible, even it appears to be a game of catch-up. "Within a capitalist system like ours, legislation and oversight are ways to attempt to reign in the harms that happen to the less advantaged and the marginalized."

"If we only think about using artificial intelligence to create machines that will think for us instead of helping us how to reason the world, that is a problem," Garcia Moreno explains.

Another form of redress is to use technology to fight technology.

Ben Zhao, professor of Computer Science at the University of Chicago, has worked with his colleague Heather Zhen to develop a program, called GLAZE, that will protect artists' work from unauthorized mimicry through the use of platforms such as Stable Diffusion, Midjourney and DALL-E.

In a keynote address on art and AI at a March 2023 conference in Chicago, Zhao pointed out that artists may not only face a loss of income, they may find their artistic identity - their brand - diluted or, as some say, polluted. A knock-off of a Picasso may satisfy every wish of a user, and, no doubt, will be cheaper. But because users of AI platforms can prompt systems to generate specific images in the style of a particular artist, any well-known and highly respected painter, illustrator or photographer working today faces the same prospect.

Zhao said some artists may stop promoting their art on their personal website, to prevent their work from being scraped without permission. Some, he added, are experiencing signs of stress and depression, and young artists are considering whether to leave their field. "AI art models can't yet replace human artists via 'training,'" he said, "but [AI] is already disrupting art industries."

Zhao pointed to the example of Karla Ortiz, a San Francisco-based illustrator and fine artist. She has worked on films including has seen her style of "Avengers: Infinity War," "Black Panther," and "Rogue One: A Star Wars Story" and "Doctor Strange." She has seen her work mimicked with AI technology to create works that resemble her well-known, highly valued style. In order to protect the work of Ortiz and other artists, Zhao has created GLAZE, a tool that allows an artist to post their work online in a way that when scraped generates an altered rendition of the work, disrupting AI-based imitations. A painting originally rendered in an impressionist style, for example, may appear in an AI-generated form in an entirely different style - abstract, for example - and thus useless to those hoping to mimic the work of artists.

GLAZE, then, is a tool in the battle against copying. Yet, copying has long existed in the art world, and often in a useful way.

Damon Willick is a professor of art history in the LMU College of Communication and Fine Arts. In the past, he says, artists in training copied the work of masters in order to learn the craft and develop their own aesthetic. More recently, artists like Andy Warhol or Marcel Duchamp co-opted existing imagery to create new images that critique works of the past.

"AI isn't doing either of those things," he says. "It's doing its own copying. It doesn't strengthen the image that it copies, neither does it critique that image."

The question of the relationship of reproduced art to the original is a central concern of José Garcia Moreno, professor of animation in the LMU School of Film and Television and director of the Academy of Catholic Thought and Imagination.

Although he is concerned about theft with the tools of artificial intelligence, Garcia Moreno also sees creative possibilities, especially in terms of expanding human understanding of the world in which we live.

"If we only think about using artificial intelligence to create machines that will think for us instead of helping us how to reason the world, that is a problem," Garcia Moreno explains. "But if we use machines to help us to reason the world because we're creating a language and interacting with a language, then it's only expanding our consciousness."

Garcia Moreno believes artificial intelligence will create an even deeper desire for audiences to go beyond reproduced art and seek out the real thing. "The physicality of art will be very much desired, because it's the real thing," he says. "We will go back to understanding what is real and what has been reproduced." The reproduced image, in other words, will highlight the beauty of the original.

"AI art models can't yet replace human artists via 'training,'" Zhao said, "but [AI] is already disrupting art industries."

Everything will be in the public domain eventually, Garcia Moreno says. "When things enter the public domain, everyone can use it. The old Mickey Mouse from the 1928 Steamboat Willie film is going to appear in many of other forms because now it is not restricted. Everything is fair game at some point. What is fair game now for artificial intelligence? Of course, we want it to be conscientious of the value of original work."

Garcia Moreno is working with new media by using 2D digital animation that uses a mobile device app to overlay animated images found in the St. John's Bible. That bible is a project of Donald Jackson, artistic director and principal illuminator of the St. John's Bible, a handwritten and illuminated bible.

He hopes that his app will draw young people to an appreciation of Jackson's work. Garcia Moreno sees his project as adding to pre-existing art, which, of course, stands within the long tradition of creating contemporary art speaking to speak to earlier art. His work, he says, "has a purpose, a goal, it's done with a lot of respect. It's just an exploration of what has been done, to create a new audience." Nonetheless, Garcia Moreno believes that ethical frame is essential.

Garcia-Moreno grants that misuse of capabilities will occur, but the future lies in not stopping the evolution of artificial intelligence but, rather, in ensuring that the creative possibilities of artificial intelligence are not channeled in a harmful way.

Sweidan, while deeply concerned about issues such as who benefits from new technology, is reluctant to judge a new technology as either good or bad. Photography was once considered by some, she points out, to be a threat to non-photographic art. "I don't think any contemporary artists would consider that position for a moment at all in 2023. ... For me it's more about thinking about how a system is made and how do we allow it to be used."

Nonetheless, Willick believes a strong dose of skepticism is called for. He considers AI-generated art "flat and very cold," precisely because it lacks the crucial human creative element.

"I don't think there is any stopping this technology," Willick says. "What I hope happens is that there emerges an avant-garde that pushes back and that creates an alternative that's more human, expressive and emotional, so that people move away from the allure of the easiness of AI-generated art."
Joseph Wakelee-Lynch is editor of LMU Magazine.

# data[].attributes.attachment_link.description
LMU Magazine is the university magazine of Loyola Marymount University, a Jesuit, Catholic university in Los Angeles, California.

# data[].attributes.headline
Website article Published 2023 published by RTF (Rethinking the Future) "The AI Effect on Art Styles"

# data[].attributes.subject
Website article Published 2023 published by RTF (Rethinking the Future) "The AI Effect on Art Styles"

# data[].attributes.body
explores how A.I. can be a tool but this article also does a deep dive into how A.I. might make art viewers feel. Does it make people feel emotion at all?

# data[].attributes.headline
Article website "Art in an Age if Artificial Intelligence" by Anjan Chatterjee that was published in 2022 by Frontiers in Psychology.  

# data[].attributes.subject
Article website "Art in an Age if Artificial Intelligence" by Anjan Chatterjee that was published in 2022 by Frontiers in Psychology.  

# data[].attributes.attachment_link.title
Art in an age of artificial intelligence

# data[].attributes.attachment_link.description
Artificial intelligence (AI) will affect almost every aspect of our lives and replace many of our jobs. On one view, machines are well suited to take over automated tasks and humans would remain important to creative endeavors. In this essay, I…

# data[].attributes.headline
Article "Colonizing art" published 2023 by Open minded Magazine written by Payal Dhar  

# data[].attributes.subject
Article "Colonizing art" published 2023 by Open minded Magazine written by Payal Dhar  

# data[].attributes.attachment_link.authors[].name
Payal Dhar is a freelance science and technology journalist living in India.

# data[].attributes.attachment_link.description
The euphoria surrounding generative AI often ignores ethics and rights, especially when it comes to artists whose work may be lifted for free and altered…

# data[].attributes.headline
Website article "Why AI Art is Winning Over Young Collectors Published 2025 by Artsy Written by Maxwell Rabb

# data[].attributes.subject
Website article "Why AI Art is Winning Over Young Collectors Published 2025 by Artsy Written by Maxwell Rabb

# data[].attributes.attachment_link.title
Why AI Art Is Winning over Young Collectors

# data[].attributes.attachment_link.description
After Christie’s controversial AI auction, Artsy asked experts why AI art is captivating Gen-Z and millennial audiences.

# data[].attributes.headline
Scholarly Article "Ways of Seeing, and Selling, AI Art" 2025, arts 14, by Imke van Heerden - Department of Digital Humanities King’s College London Strand.

# data[].attributes.subject
Scholarly Article "Ways of Seeing, and Selling, AI Art" 2025, arts 14, by Imke van Heerden - Department of Digital Humanities King’s College London Strand.

# data[].attributes.attachment_link.description
In early 2025, Augmented Intelligence – Christie’s first AI art auction – drew criticism for showcasing a controversial genre. Amid wider legal uncertainty, artists voiced concerns over data mining practices, notably w…

# data[].attributes.attachment_link.title
Why AI Art Is Winning over Young Collectors

# data[].attributes.attachment_link.description
After Christie’s controversial AI auction, Artsy asked experts why AI art is captivating Gen-Z and millennial audiences.

# data[].attributes.attachment_link.readable_text
It's not often that the work of authenticating art makes mainstream news, but that's exactly what happened last year when a team of researchers in the UK determined that an anonymous, centuries-old painting known as the de Brécy Tondo was likely made by the Renaissance giant Raphael. It was a bold claim-one with potentially huge financial implications-but what really caught people's attention was the technology the researchers used to get there: AI.

The group, led by two science professors-Christopher Brooke from the University of Nottingham, and Hassan Ugail from the University of Bradford-developed a facial recognition model to compare the Madonna depicted in the de Brécy Tondo with other portraits of the same figure. After finding a 97 percent match with Raphael's Sistine Madonna altarpiece, the researchers concluded, in Ugail's words, that "identical models were used for both paintings and they are undoubtedly by the same artist."

But the novelty of this discovery did not last long. Some eight months after Brooke and Ugail's announcement, a Swiss AI company called Art Recognition used its own model to determine, with 85 percent certainty, that the de Brécy Tondo was not made by the Renaissance master. In an op-ed, Art Recognition's founder, Carina Popovici, defended her company's findings, pointing to the many art historians on her staff and the sophistication of her model, which was trained on images of real and forged Raphael paintings. Notably, she stopped short of discrediting Brooke and Ugail. "The most straightforward explanation for the strong discrepancy between the two results is that the models are essentially addressing different questions," Popovici wrote.

Dubbed the " battle of the AIs" by the Guardian, this brief controversy did little to convince skeptics of AI's ability to interpret paintings. Instead, it served as a microcosm of the larger debates playing out in the institutional art world as this technology comes closer and closer to infiltrating its hallowed halls. AI is already curating museum shows and biennials. Could it change how we approach, study, or regard art of the past, too?

When it comes to AI and art history, it's important to remember that, for many, this is not just a technological question, but an urgent ideological one too. In 2013, the scholar and theorist Johanna Drucker published a paper called " Is There a 'Digital' Art History? " The piece touches on wide-ranging and complex topics, from data mining to the evolution of critical methodologies. But ultimately, Drucker's answer to the question posed by the title of her paper is surprisingly simple: No. Engineered to aggregate and extrapolate, contemporary computational technology had made art history infinitely more accessible and navigable, Drucker noted. But it had not, in her words, altered the field's "fundamental approaches, tenets of belief, or methods."

Within the digital humanities-an increasingly prominent field of academic study that incorporates advanced computational techniques into the study of nonscientific disciplines like art and literature-Drucker's paper proved to be divisive. So did a related essay published two years later, in which art historian and critic Claire Bishop argued that the limits of a so-called Digital Art History reflect a broader socioeconomic problem: neoliberalism's relentless push to metricize and optimize.

"Digital art history, as the belated tail end of the digital humanities, signals a change in the character of knowledge and learning," writes Bishop in her essay, " Against Digital Art History." More and more, she argues, "research and knowledge are understood in terms of data and its exteriorization in computational analyses. This raises the question of whether there is a basic incompatibility between the humanities and computational metrics. Is it possible to enhance the theoretical interpretations characteristic of the humanities with positivist, empirical methods-or are they incommensurable?"

Bishop and Drucker's papers are roughly a decade old now. In the booming world of AI development, that's a significant amount of time, and much about the technology has changed in the interim. What hasn't changed, though, is the broader context that informed these two important thinkers' positions, which arrived in the middle of a worrying-and still ongoing-trend of lawmakers, grant-givers, and educational institutions opting to fund the fields of STEM at the expense of the humanities. On some simple, subtextual level, Bishop and Drucker were arguing for the importance of critical connoisseurship in a culture war that's starting to tip the opposite way.

MIT Press

Are art historians becoming outmoded? Some experts do not seem worried. "I don't think there's any AI or machine learning technique that could ever replace an art historian," Amanda Wasielewski, a digital humanities professor at Uppsala University in Sweden. Hers is a reassuringly pragmatic take on this issue, partly due to the privilege of hindsight-though she does lean toward the thinking of Bishop and Drucker on some critical points too.

Wasielewski is not against the integration of these tools into her field, nor is she necessarily convinced that their effects will be so radical. "There are already applications for machine learning and AI in everyday research uses for art historians, archivists, museums, and galleries," she says, citing the tool's foregone role in archival databases and collection management software. "These are practical applications that don't replace humans. They just make our work slightly more streamlined."

Ultimately, Wasielewski is less concerned that AI will introduce new ways of thinking than she is with the prospect that it may bring back old ones too. Last year, the scholar published Computational Formalism, a book that charts the ways in which machine learning has reintroduced the kind of strict, "close reading" methodologies of art study that have not been fashionable among critics and historians for many years.

This approach, which emphasizes an artwork's physical properties (composition, color, scale) over the external contexts of its making (the artist's identity and intentions, say), was the dominant theoretical mode for much of the modernist era of the early 20th century. But by the 1960s and '70s, as new critical approaches-feminism, postcolonialism, structuralism-evolved in response to a cultural landscape shaped by political violence and burgeoning social movements, the formalists' close-eyed approach seemed out of step. In a 1974 postmortem on the movement, literary scholar Gerald Graff offered a conclusion eerily similar to the one Bishop would come to write about Digital Art History four decades later. Formalism, Graff wrote, was just "one more symptom of the university's capitulation to the capitalist-military industrial-technological complex."

Now, Wasielewski fears that machine learning will bring this dogma back from the dead. With their hefty data diets and swift algorithmic operations, these computer vision systems are fine-tuned for formal analysis and pattern recognition-hence their early integration into the sectors of art authentication and archive management. But the more we turn to these systems for study, Wasielewski suggests, the more we ignore the many "critical frameworks" and "different methodological paradigms" that were developed before them. "When you think that somehow you're going to draw out objective things from a formalist methodology," she continued, "you don't do any extra methodological work."

Portrait of Don Diego Messía Felipe de Guzmán, Marqués de Leganés was attributed to Anthony van Dyck by two experts following AI analysis.

The formalist thinking encouraged by computers does not necessarily equate to close reading. In the early 2000s, literary historian Franco Moretti proposed a concept called "distant reading"-now referred to as "distant viewing" in art-whereby vast amounts of formal data from a field are crunched to reveal broad patterns and trends across time, place, and style. A recent project by two digital humanities professors-Leonardo Impett from the University of Cambridge, and Fabian Offert from the UC Santa Barbara-illustrates the upshot of this approach.

Last year, the researchers sought to reexamine the ideas in Drucker's " Is There a 'Digital' Art History?" through Moretti's methods and newtransformer-based vision models, which can "learn" the relationship between different pieces of data.These systems "can tell you much more about a painting than we could ever hope to have achieved with more traditional computational methods," Offert said. He pointed to Diego Velazquez's 1656 painting Las Meninas -a masterpiece on the shortlist of the most studied artworks in history-which he and Impett analyzed through their own model, finding surprising compositional similarities to a pair of 20th-century photographs by Robert Doisneau and Joel Meyerowitz.

The blunt name of Impett and Offert's paper reveals tips their intentions: "There Is a Digital Art History." But the titlecomes with an asterisk too.

"With this generation of models, we're getting closer to doing actual art historical work with machine learning, with the caveat that we buy into whatever it is that these models have been trained on," Offert explained. In other words, these new machine learning models are only as good as what is fed into them, and what is fed into them is only what art historians, with all their human biases, have chosen to digitize. "We can have the benefits of these new models, but at the same time," Offert continued, we have to "always critique [them] and figure out the limitations of their...weird machinic visual culture."

"They're are not magical machines. They come from somewhere," Wasielewski said of these models. "We need to question not just how these tools are applied, but where they have their origins, what kind of data they were trained on, what biases might be contained within [them]."

The brief row with Brooke and Ugail was not the only time that Popovici has had her company's efforts questioned. Last fall, roughly a month after Art Recognition shared its findings on the de Brécy Tondo, a German art history professor named Nils Büttner published an essay that targeted the company's work to build a broader argument about the limitations of AI trained on digital images.

Popovici called Büttner's essay "very aggressive," and said the tone was symptomatic of the anxiety many old-school experts feel when confronted withAI. "They feel that the technology is going to push them out of the way and take their jobs." But Popovici has come to view incidents like this as an opportunity for dialogue, not mudslinging."We've really put a lot of effort into talking to them because it's not true," she said. "In order to train the AI, you need images, and the images come from catalogue raisonnés, [which] are made by experts" like Büttner. Their knowledge, Popovici added, "is absolutely paramount"

Earlier this year, Popovici teamed up with Büttner on a research paper examining an old painting attributed to the 17th-century artist Anthony van Dyck. They went about the task before them in separate ways. Büttner, a by-the-book historian, picked over the painting in person, checked his observations against past research, then determined that the work was not made by the Baroque master, but one of his workshop pupils. Popovici, an AI expert with commercial interests, assembled a collection of images related to the artist, fed them through her model, then concluded that there is a 79 chance that the painting was not made by the artist.

Two different methods, two similar conclusions: a traditionalist and technologist reaching across the ideological aisle in the name of normal, productive dialogue. It was a simple gesture, but also a meaningful one. As Wasielewski reminds, it's people who design these systems, not the other way around, and the dialogue about how we use them has really only just started. When it comes to the development of AI, she said, "We in the field of art and in the field of art history need to be involved in these conversations."

Read more about "Artificial Intelligence and the Art World" here.

# data[].attributes.attachment_link.description
AI's ability to analyze paintings is already clear, but some experts say by-the-book art historians are still needed.