# data[].attributes.attachment_link.description
We’re currently checking your connection. This shouldn’t take long.If you keep getting this page, check if there’s a problem with Canva.

# data[].attributes.body
Ableton Knowledge Base (n.d.) Avoiding Disk Overload, Ableton website, accessed 19 October 2024. https://www.lib.rmit.edu.au/easy-cite/?styleguide=styleguide-6#stn-5

# data[].attributes.body
It’s with a great sense of accomplishment (relief) that all components appear to be technically functioning. While some compromises had to be made I sincerely believe it portrays the prompt ‘surreal’ in an engaging medium. While initially sceptical of how haptic feedback can apply meaning, the addition of this component increases the sense of immersion which can expand the affective qualities of the audio and visual components. I found when bass is sent to the back sub it can be very effective as gaining or maintaining attention while depticing the mass/scale of the organism. Subsequently I found that sending diegetic audio gestures to the hands, feet and ribs worked well and altering your perception of the relationship being formed. By utilising body parts with higher sensitivity I experienced a greater sense of empathy by perceiving the physical movements of the organism displayed. There was also some novelty to haptic feedback in this context, which I suspect can increase audience enjoyment and engagement.

While I am content with the sound design, there are some aspects that need improvement. Time was a major limitation, however many iterations and learnings were required to get the visual components and haptic feedback to its current state. I am satisfied with the time spent as I learned how to create engaging content in a manner that feels authentic to my vision, while utilising current generative tools. This has opened many possibilities to enhance my own practice, and I feel more capable as a multidisciplinary designer. I enjoyed treating artificial intelligence as an extremely technical “artist” with poor listening skills. It opened up more possibilities by utilising aesthetics of chance that lead to interesting outcomes I did not suggest, but happily accepted into my work.

# data[].attributes.body
I experienced a new issue with Ableton this week because of the large number of files requiring sends to the haptic feedback vest. It seems that even with my new Macbook the CPU can easily be overloaded and results in visual and audio stuttering. Initially I attempted to increase the buffer size with only temporary improvement. From this article I applied the troubleshooting technique of applying tracks to computer RAM opposed to CPU processes. Once again this did not resolve the issue and caused a major crash, which in hindsight makes perfect sense when you apply over 100 tracks to RAM while playing an Ultra HD video simultaneously. I did not want to reduce the video quality or the audio sample rate as this would introduce new issues and only mask some of the stuttering at the cost of overall quality reduction. I was able to resolve this issue by bouncing a total of 6 tracks in Logic Pro, organised by the type of sound component and where it is designed to be placed on the feedback vest. This has now resolved the issue and all components are functioning with the bonus of a decreased buffer size to improve playback latency.

# data[].attributes.body
This week has been spent completing the final components for assignment 3 submission. This involves finishing the sound design and score, editing the video and upscaling to ultra high definition and applying haptic feedback in Ableton. The feedback received this week has been mostly positive now that I’ve removed the strobing video effect, however as I feared this has exposed some of the weaker aspects of my current sound design. The less fluid animation has become more jarring and noticeable, from Darrin’s feedback this may be resolved by leaning into the visual flaw as an affordance for sound. I believe that the combination of organic and heavily processed electronic sounds may work as it remains contextually appropriate and can be perceived as diegetic. I suspect the electronic sounds can be used to build a synchretic relationship with some of the visual flaws, while breaking this relationship may lead to interesting and perhaps more complicated audio-visual relationships if applied diligently. While I would love to test this idea further I unfortunately have not allowed myself ample time to experiment and will need to utilise it in practice, in a timeboxed manner to ensure the assignment is submitted by the looming deadline.

# data[].attributes.body
Duarte C et al. (2021) The soundscape of the Anthropocene ocean, Science website, accessed 12 October 2024. https://www.science.org/doi/10.1126/science.aba4658

Bunish C (2002) The Art of Sound Design, Post Magazine website, accessed 12 October 2024. https://www.postmagazine.com/Publications/Post-Magazine/2002/November-1-2002/The-Art-of-Sound-Design.aspx

Ant Pruitt (2020) ‘How To PROPERLY Upscale HD Video to 4K In Premiere Pro’ [video], Ant Pruitt, Youtube website, accessed 12 October 2024. https://youtu.be/TMztp7h8290?si=T8rKxhsiyBaXg_iG

# data[].attributes.body
It’s with some disappointment that I won’t be completing the audio reactive project I initially planned, however this provides the opportunity to really focus and execute Speculative Evolution to the best I can. At this point the focus is to simply get **it done. With initial sound design and some haptics implemented the remaining time will be used to complete these aspects and render a 4K version of the video.

# data[].attributes.body
Based on testing my visual footage at the Capitol I think it’s important to try and render the video footage to the highest quality in order to enhance the perception of reality. The limitation with my current workflow is Luma can only generate HD quality video which is designed for smaller screens, not large scale projection. Premiere Pro has added AI upscaling which can simply enlarge HD quality footage to 4k, however some additional processing is required to improve the fidelity. Through the use of colour grading and increasing sharpness the footage appears less grainy with added clarity. The last step involves exporting the video at a higher bitrate to ensure smooth motion.

# data[].attributes.attachment_link.title
How To PROPERLY Upscale HD Video to 4K In Premiere Pro

# data[].attributes.body
With the visual components close to completion I have shifted focus to sound design and composition. Initially I found it quite challenging to articulate the movements of the organisms but found some inspiration watching nature documentaries. I noticed simple water streams, bubbles and whooshes with reduced high frequencies were commonplace. While not particularly creative or exciting it seemed to work well at contextually bringing the animals to life and creating the perception of realism. I incorporated this learning into my latest iteration of Speculative Evolution.

# data[].attributes.body
I read two articles this week to provide further inspiration for the sound design of my project. The objective was to continue reviewing technical sound design to see which devices I could utilise in my work and to develop a better understanding of other potential sound objects may be included.

From the grim and aptly titled ‘The soundscape of the Anthropocene ocean’ I discovered noise pollution can interfere with communication and induce a stress response. While this may not come as a surprise, it presents an interesting idea that relates to my initial readings on cortisol effects on fragmentation of dreams. To influence my work perhaps noise pollution may be a mechanism to drive specific movement of the fictional organisms and used to drive further implication in the project. It’s also made me consider the space in which the objects exist. I’ve viewed the space previously as a scientific microcosm like an aquarium or petri dish. Perhap they could be in a mud puddle created by radioactive waste.
While there wasn’t much new information in ‘The Art of Sound’ article there was an interesting technique mentioned. Sound designer/composer Miyagi applied distortion and filtering to musical beats in order to create the desired soundscape. I’ve been finding it difficult to pair music with the video and perhaps there’s a middle ground to be explored where music can be perceived diegetic with appropriate manipulation even though contextually fanciful. The article also reminded me of the BBC sound archive where I may source realistic animal sounds and apply editing techniques to create diegetic sounds for the fictional organisms.

# data[].attributes.attachment_link.description
“Absolutely,” affirms Tim Gedemer, sound designer with Santa Monica’s Spank! Music and Sound Design (www.spankmusic.com). “The process of creating sound design to picture is inherently subjective and therefore needs to be linked to art.”

# data[].attributes.body
I must admit I was quite taken back by the positive comments on my Speculative Evolution project and feel that the visuals are more engaging than I may have previously considered. The video fidelity was better than expected when played on the projector in the Black Box and I may not need to use visual effects like VHS/camcorder grain or strobing to mask. From the feedback I received it appears I should aim for the opposite and try to increase the fidelity further, perhaps by upscaling or increasing sharpness. When played on the large 4K projector at the Capitol there was some pixelation at a close distance which is to be expected when the video resolution is 1024x1024. There is an interesting affective quality when you stand right in front of the projector screen and look up at the organisms. I can’t quite accurately describe it at this point, but perhaps there is a sublime quality created by the vastness and the act of looking upwards as if you’re admiring a large animal up close at the zoo or aquarium. The haptic feedback also seemed to increase immersion from my personal testing and provided another mechanism to maintain attention and possibly build empathy or shift the audience's point of perception. I believe the optimal outcome would involve the audience forgetting they’re looking at a screen but rather sharing a space and actively observing and empathising.

# data[].attributes.body
Duxbury A, Byrne R (2024) The transition stage, Britannica website, accessed 5 October 2024. https://www.britannica.com/science/seawater/The-transition-stage

Northumbria University Newcastle (2024) EXPERT COMMENT: The animal sounds in most nature documentaries are made by humans – here’s how they do it and why it matters, Northumbria University Newcastle website, accessed 5 October 2024. https://newsroom.northumbria.ac.uk/pressreleases/expert-comment-the-animal-sounds-in-most-nature-documentaries-are-made-by-humans-heres-how-they-do-it-and-why-it-matters-3297094

Insider (2020) ‘How Animal Sounds Are Made For Movies And TV | Movies Insider’ [video], Insider, Youtube website, accessed 5 October 2024. https://youtu.be/l2WCrXc7J-I?si=4dcKpH1492FpYyF1

Insider (2021) ‘How Sounds Are Faked For Nature Documentaries | Movies Insider’ [video], Insider, Youtube website, accessed 5 October 2024. https://youtu.be/AcmhWs7HM1c?si=Uov9TG0m77bQN5n9

# data[].attributes.body
This week I generated video assets for Speculative Evolution using the high quality generated images from last week. I initially created around 130 4-10 second videos which were categorised and edited down to just over 5 minutes. Overall I am extremely happy with the edit, the narrative structure feels engaging and generates interest over time by strategically placing my 4 favourite sequences at the beginning, end and in the middle after a subdued sequence. I experimented with haptic feedback in Ableton and created a very simple sound composition. I’m not particularly impressed with the composition, I felt limited by Ableton’s workflow and felt I couldn’t find a creative mindset while figuring out nuances in the software. While I identified this issue last week, I was hoping with some practice I could complete everything in Ableton but it appears unlikely.

With the time required for the remaining sound and haptic elements I’m beginning to think that my Immersive Music project may have to be dropped in order to focus on Speculative Evolution. While I’ve thoroughly enjoyed creating audio reactive content in TouchDesigner it does feel novel and perhaps decorative in its current form. I also feel that it doesn’t quite hold true to the prompt surreal and lends towards mesmerising, hypnotising and maybe even uncanny. I will need to discuss this with Darrin and Nik to determine an outcome.

# data[].attributes.body
Seeking inspiration for the diegetic sound effects I investigated techniques used by foley artists for nature documentaries. Richard Hinton describes foley as a “contact sport” often utilising physical properties to bring sound to life. He utilises recording techniques like proximity effect to draw attention to intricate movement, while materials like wet leather, metal coils and magnetic tape are used to recreate animal movement. For ocean dwelling animals a large tub of water is used and splashes are created by hand at different depths to depict traits like animal mass and quantity. Notably he mentions stylized shots often invite creative freedom due to unrealistic properties such as a timelapse. Mark Constazo has a similar technical approach but emphasises the “personality” he attempts to evoke. For the dragonfly sound design at the beginning of Men in Black he utilises a small electronic hand fan which he disrupts with his finger to give an organic quality to the fluttering wings. He also mentions the importance of capturing the surface of the interaction to enhance the perception of realism.

# data[].attributes.attachment_link.title
How Sounds Are Faked For Nature Documentaries | Movies Insider

# data[].attributes.body
I continued to experiment with my Immersive Music project by pushing the audio reactive orb to its limit by utilising additional audio data and modifying the fundamental shapes the orb is based on. Overall it lead to some interesting effects but also reduced the visual fidelity. I enjoy that it feels less predictable but don’t believe the output quality is high enough to be projected on a large screen.

# data[].attributes.body
While we’ve all held our head underwater at some stage and heard the pressurised sound of water, how can these acoustic properties be examined and applied to create the perspective of an ocean organism? The effects of ocean water on sound can be determined by several factors. Seawater can increase the speed of travelling sound and alter tonal properties through refraction and attenuation. Deeper water can further attenuate high frequencies even though sound speed increases with pressure. Sound can travel at an increased distance with pressure, however high frequency attenuation occurs due to temperature changes. This is why sonar technology utilises low frequency sound to travel over greater distances. Deep water refraction can result in a natural phenomenon known as SOFAR (Sound Fixing and Ranging) which allows sound waves to travel over long distances with minimal loss. This is useful for communication and tracking marine life through acoustic tags. To apply this knowledge to my project I’ll continue to attenuate high frequencies with diegetic ocean noise. Creatively this present the opportunity to amplify the perception of radar noise to add an additional sound element.

# data[].attributes.attachment_link.title
Seawater - Acoustics, Salinity, Temperature | Britannica

# data[].attributes.attachment_link.description
Seawater - Acoustics, Salinity, Temperature: Water is an excellent conductor of sound, considerably better than air. The attenuation of sound by absorption and conversion to other energy forms is a function of sound frequency and the properties of water. The attenuation coefficient, x, in Beer’s law, as applied to sound, where Iz and I0 are now sound intensity values, is dependent on the viscosity of water and inversely proportional to the frequency of the sound and the density of the water. High-pitched sounds are absorbed and converted to heat faster than low-pitched sounds. Sound velocity in water is determined by the square root of elasticity

# data[].attributes.body
From the feedback received this week I feel confident that the work is on track despite some concern around my audio reactive work perhaps landing outside of the prompt surreal. An interesting idea was brought up around taking a documentary style approach to the sound design for Speculative Evolution. This has influenced my research as I look to find devices to evoke a sense of perceived realism while also using composition to drive the affective qualities and personalities of the designed organisms.

# data[].attributes.body
Berkeley University of California (2020) Macroevolution through evograms, Berkeley University of California website, accessed 28 September 2024. https://evolution.berkeley.edu/what-are-evograms/the-evolution-of-whales/

Carolina Friends School (n.d.) Adaptions In Marine Organisms: The Deep Sea, Carolina Friends School, accessed 28 September 2024. https://www.cfsnc.org/whats-new/post-landing-page/~board/research/post/adaptations-in-marine-organisms-the-deep-sea

Science Learning Hub (2019) Marine organisms and adaptations, Science Learning Hub website, accessed 28 September 2024. https://www.sciencelearn.org.nz/resources/142-marine-organisms-and-adaptations

Ethan Davis (2024) ‘The NEW Ableton 12 Beginner Guide (in 22 Minutes)’ [video] Ethan Davis, Youtube website, accessed 28 September 2024. https://youtu.be/WMg4Y4q6FQ0?si=DIFb7rg2vI-hIXCh

# data[].attributes.body
The week was spent generating final assets for my Speculative Evolution project and initial composition in Ableton. While the image generation and lineage has been successful, designing haptic feedback has been quite challenging. I’ve noticed when too many elements are sent to the haptic feedback vest it results in muddiness which detracts from the experience by preventing the articulation of sonic microstructures. This may be an exercise of restraint by only sending elements that are relevant to narrative structure or enhancing the perception of a surreal experience. Another challenge involves setting up return channels for audio effects. As Max for Live is using audio sends to the vest it’s preventing additional returns. This does make sense as I expect the sends are directly outputting to hardware and would require internal remapping in Ableton. This has shifted my approach to composition and I believe it's best to revert to my regular process by using Logic Pro then exporting the stems to mix in Ableton.

# data[].attributes.body
I will need to use Ableton to implement haptic feedback through the music:notimpossible vest. The vest utilises Max for Live which runs natively in Ableton Suite. I watched this tutorial to improve my workflow and get a better understanding of keyboard shortcuts and software specific terminology. Simple tips like turning off analysis and warp clips will prevent audio files from automatically time stretching to the set BPM. I was able to add mixing faders into my arrangement view which allows for mixing while editing in the same window. Fade ins/outs can be applied directly to clips by selecting the top or bottom corner. While the “H” and “W” buttons quickly adjust the height and width of tracks in the arrangement view to become uniform.

# data[].attributes.attachment_link.title
The NEW Ableton 12 Beginner Guide (in 22 Minutes)

# data[].attributes.body
I have begun generating final image assets for my Speculative Evolution project using Stable Diffusion and utilising its upscale function to ensure high quality resolution. From over 600 generated images I have created what I consider a lineage to group creatures of compatible attributes. Notably all creatures and not linked by the same prompt. Attributes like colour, material, shape and biology and driving selection choices. Some are also selected via creative intuition gathered by experimentation. The attributes may not necessarily match and chance is applied, however this may lead to interesting transformations once video is generated in Luma. I also generated some water-like visuals to be played on supplementary screens if multichannel video is applied. However these felt lacklustre and decorative at best.

# data[].attributes.body
To further my research from last week I want to examine the physical attributes of evolution found in ocean inhabiting organisms. There are notable differences in characteristics based on the depth where the organisms are located. Deep sea organisms often have gelatinous bodies with very little skeletal structure to withstand high pressure which can be considered 500 times that on land. A common adaptation involves bioluminescence or enlarged eyes to allow for vision in darkness. While strange digestive adaptations like expanding mouths and stomachs can aid in consumption due to prey scarcity.

Ocean mammals like whales originally evolved from land dwelling organisms. Through this process limbs become vestigial and the body becomes streamlined. Nasal cavities shift further back on the head and a spiracle (blow hole) develops. Slow moving species are unable to protect themselves by fleeing and have developed characteristics for protection. These often include physical spines, chemical deterrents or the ability to regenerate. While a common behaviour involves forming colonies comprised of many organisms to camouflage as plants.

# data[].attributes.body
This week we presented our work in progress on Monday and Thursday’s class. It was interesting to see how others have iterated on their initial work and the interpretation of the prompt. In particular I’ve noticed others that have selected surreal have utilised similar colour palettes and video format, but with very different content. Some works seem to utilise more conventional themes such as eyes and whales which align with surrealism, notably influenced by the likes of Salvador Dali’s work. The projects I’ve found the most engaging seem to use the prompt uncanny and the Capitol’s lighting.

# data[].attributes.body
Tyler White (2021) ‘VHS Retro Camcorder Effect Premiere Pro Tutorial - Unbelievably Easy’ [video] Tyler White, Youtube Website accessed 21 September 2024. https://youtu.be/CPu8iVrZTh0?si=TyjTUeaOlJtoLNfn

Cooper G (2000) The Cell: A Molecular Approach. 2nd edition, National Library of Medicine website, accessed 21 September 2024. https://www.ncbi.nlm.nih.gov/books/NBK9841/

# data[].attributes.body
This week I completed the work in progress presentation for assignment 3. It provided the opportunity to reflect on failures, successes and the natural iteration of the project from its initial conception. I have changed my view on haptic feedback, initially I thought it had no meaningful qualities however the ‘Haptic Empathy’ paper has led me to believe that there are some fundamental human emotions that can be conveyed through patterns of vibration. I believe the latest video iterations for Speculative Evolution have been successful and feel confident to push forward with the final versions.

While the foundation for my Immersive Video project is strong, it has fallen behind because of the time required to iterate on the generative components of my other project. This has given me some time to think about the outcomes I’d like to achieve and I believe that conveying the emotional attributes of music with haptic feedback is a crucial property for the success of this project.

# data[].attributes.body
To achieve the transmission-like video effect for my Speculative Evolution project I found a video that shows the process for creating a retro camcorder effect. The overall effect consists of colour grading to alter hue, channel blur for low quality aesthetic and the combination of mask and noise to add grain to the footage.
I also continued to iterate on my video generation process by experimenting with different Luma prompts utilising the glossary and Claude AI. While the prompts will never ensure meaningful accuracy I’m noticing that applying certain key phrases to different images can result in a consistent visual aesthetic for video sequences.

# data[].attributes.attachment_link.title
VHS Retro Camcorder Effect Premiere Pro Tutorial - Unbelievably Easy

# data[].attributes.body
I have continued to iterate on my video generation process using Stable Diffusion to generate images and LumaLabs to animate. For this piece I wanted to test the perceived evolution process by using elements that may not necessarily morph cohesively according to previous iterations. What I found using this process was that some of the elements lead to strange and fascinating evolutions like 0:14 where a cell morphs into a centipede-like creature. While others like 0:19 went too far and detracted from the experience. This seems to align with the ‘Aesthetics of chance’ paper and allows for unforeseen compositions.

# data[].attributes.body
I wanted to develop a further understanding on the physical attributes of cellular evolution to inform my Speculative Evolution project. While my intention is not to directly apply these findings to my work, it would be interesting to understand how my creations align with the expectations set by the current understanding of evolutionary biology.

There are two fundamental classes of cells. Prokaryotic cells are defined by their simple structure, lacking a nucleus and organelles. While eukaryotic cells are complex and can divide functions by containing a nucleus and organelles. To put simply, prokaryotic cells can be considered bacteria while eukaryotic cells can be found in animals, plants and fungi. Physical differences can be found in terms of size with eukaryotic cells presenting as larger due to multicellular design and increased structures. The cytoskeleton encases eukaryotic cells which determines the shape based on its origin, notably this can influence plant cells to appear rectangular due to the fixed cell wall.

# data[].attributes.attachment_link.readable_text
Cells are divided into two main classes, initially defined by whether they contain a nucleus. Prokaryotic cells (bacteria) lack a nuclear envelope; eukaryotic cells have a nucleus in which the genetic material is separated from the cytoplasm. Prokaryotic cells are generally smaller and simpler than eukaryotic cells; in addition to the absence of a nucleus, their genomes are less complex and they do not contain cytoplasmic organelles or a cytoskeleton ( Table 1.1). In spite of these differences, the same basic molecular mechanisms govern the lives of both prokaryotes and eukaryotes, indicating that all present-day cells are descended from a single primordial ancestor. How did this first cell develop? And how did the complexity and diversity exhibited by present-day cells evolve?

The First Cell

It appears that life first emerged at least 3.8 billion years ago, approximately 750 million years after Earth was formed ( Figure 1.1). How life originated and how the first cell came into being are matters of speculation, since these events cannot be reproduced in the laboratory. Nonetheless, several types of experiments provide important evidence bearing on some steps of the process.

It was first suggested in the 1920s that simple organic molecules could form and spontaneously polymerize into macromolecules under the conditions thought to exist in primitive Earth's atmosphere. At the time life arose, the atmosphere of Earth is thought to have contained little or no free oxygen, instead consisting principally of CO 2 and N 2 in addition to smaller amounts of gases such as H 2, H 2S, and CO. Such an atmosphere provides reducing conditions in which organic molecules, given a source of energy such as sunlight or electrical discharge, can form spontaneously. The spontaneous formation of organic molecules was first demonstrated experimentally in the 1950s, when Stanley Miller (then a graduate student) showed that the discharge of electric sparks into a mixture of H 2, CH 4, and NH 3, in the presence of water, led to the formation of a variety of organic molecules, including several amino acids ( Figure 1.2). Although Miller's experiments did not precisely reproduce the conditions of primitive Earth, they clearly demonstrated the plausibility of the spontaneous synthesis of organic molecules, providing the basic materials from which the first living organisms arose.

The next step in evolution was the formation of macromolecules. The monomeric building blocks of macromolecules have been demonstrated to polymerize spontaneously under plausible prebiotic conditions. Heating dry mixtures of amino acids, for example, results in their polymerization to form polypeptides. But the critical characteristic of the macromolecule from which life evolved must have been the ability to replicate itself. Only a macromolecule capable of directing the synthesis of new copies of itself would have been capable of reproduction and further evolution.

Of the two major classes of informational macromolecules in present-day cells (nucleic acids and proteins), only the nucleic acids are capable of directing their own self-replication. Nucleic acids can serve as templates for their own synthesis as a result of specific base pairing between complementary nucleotides ( Figure 1.3). A critical step in understanding molecular evolution was thus reached in the early 1980s, when it was discovered in the laboratories of Sid Altman and Tom Cech that RNA is capable of catalyzing a number of chemical reactions, including the polymerization of nucleotides. RNA is thus uniquely able both to serve as a template for and to catalyze its own replication. Consequently, RNA is generally believed to have been the initial genetic system, and an early stage of chemical evolution is thought to have been based on self-replicating RNA molecules-a period of evolution known as the RNA world. Ordered interactions between RNA and amino acids then evolved into the present-day genetic code, and DNA eventually replaced RNA as the genetic material.

The first cell is presumed to have arisen by the enclosure of self-replicating RNA in a membrane composed of phospholipids ( Figure 1.4). As discussed in detail in the next chapter, phospholipids are the basic components of all present-day biological membranes, including the plasma membranes of both prokaryotic and eukaryotic cells. The key characteristic of the phospholipids that form membranes is that they are amphipathic molecules, meaning that one portion of the molecule is soluble in water and another portion is not. Phospholipids have long, water-insoluble ( hydrophobic) hydrocarbon chains joined to water-soluble ( hydrophilic) head groups that contain phosphate. When placed in water, phospholipids spontaneously aggregate into a bilayer with their phosphate-containing head groups on the outside in contact with water and their hydrocarbon tails in the interior in contact with each other. Such a phospholipid bilayer forms a stable barrier between two aqueous compartments-for example, separating the interior of the cell from its external environment.

The enclosure of self-replicating RNA and associated molecules in a phospholipid membrane would thus have maintained them as a unit, capable of self-reproduction and further evolution. RNA-directed protein synthesis may already have evolved by this time, in which case the first cell would have consisted of self-replicating RNA and its encoded proteins.

The Evolution of Metabolism

Because cells originated in a sea of organic molecules, they were able to obtain food and energy directly from their environment. But such a situation is self-limiting, so cells needed to evolve their own mechanisms for generating energy and synthesizing the molecules necessary for their replication. The generation and controlled utilization of metabolic energy is central to all cell activities, and the principal pathways of energy metabolism (discussed in detail in Chapter 2) are highly conserved in present-day cells. All cells use adenosine 5′-triphosphate ( ATP) as their source of metabolic energy to drive the synthesis of cell constituents and carry out other energy-requiring activities, such as movement (e.g., muscle contraction). The mechanisms used by cells for the generation of ATP are thought to have evolved in three stages, corresponding to the evolution of glycolysis, photosynthesis, and oxidative metabolism ( Figure 1.5). The development of these metabolic pathways changed Earth's atmosphere, thereby altering the course of further evolution.

In the initially anaerobic atmosphere of Earth, the first energy-generating reactions presumably involved the breakdown of organic molecules in the absence of oxygen. These reactions are likely to have been a form of present-day glycolysis-the anaerobic breakdown of glucose to lactic acid, with the net energy gain of two molecules of ATP. In addition to using ATP as their source of intracellular chemical energy, all present-day cells carry out glycolysis, consistent with the notion that these reactions arose very early in evolution.

Glycolysis provided a mechanism by which the energy in preformed organic molecules (e.g., glucose) could be converted to ATP, which could then be used as a source of energy to drive other metabolic reactions. The development of photosynthesis is generally thought to have been the next major evolutionary step, which allowed the cell to harness energy from sunlight and provided independence from the utilization of preformed organic molecules. The first photosynthetic bacteria, which evolved more than 3 billion years ago, probably utilized H 2S to convert CO 2 to organic molecules-a pathway of photosynthesis still used by some bacteria. The use of H 2O as a donor of electrons and hydrogen for the conversion of CO 2 to organic compounds evolved later and had the important consequence of changing Earth's atmosphere. The use of H 2O in photosynthetic reactions produces the by-product free O 2; this mechanism is thought to have been responsible for making O 2 abundant in Earth's atmosphere.

The release of O 2 as a consequence of photosynthesis changed the environment in which cells evolved and is commonly thought to have led to the development of oxidative metabolism. Alternatively, oxidative metabolism may have evolved before photosynthesis, with the increase in atmospheric O 2 then providing a strong selective advantage for organisms capable of using O 2 in energy-producing reactions. In either case, O 2 is a highly reactive molecule, and oxidative metabolism, utilizing this reactivity, has provided a mechanism for generating energy from organic molecules that is much more efficient than anaerobic glycolysis. For example, the complete oxidative breakdown of glucose to CO 2 and H 2 O yields energy equivalent to that of 36 to 38 molecules of ATP, in contrast to the 2 ATP molecules formed by anaerobic glycolysis. With few exceptions, present-day cells use oxidative reactions as their principal source of energy.

Present-Day Prokaryotes

Present-day prokaryotes, which include all the various types of bacteria, are divided into two groups-the archaebacteria and the eubacteria-which diverged early in evolution. Some archaebacteria live in extreme environments, which are unusual today but may have been prevalent in primitive Earth. For example, thermoacidophiles live in hot sulfur springs with temperatures as high as 80°C and pH values as low as 2. The eubacteria include the common forms of present-day bacteria-a large group of organisms that live in a wide range of environments, including soil, water, and other organisms (e.g., human pathogens).

Most bacterial cells are spherical, rod-shaped, or spiral, with diameters of 1 to 10 μm. Their DNA contents range from about 0.6 million to 5 million base pairs, an amount sufficient to encode about 5000 different proteins. The largest and most complex prokaryotes are the cyanobacteria, bacteria in which photosynthesis evolved.

The structure of a typical prokaryotic cell is illustrated by Escherichia coli ( E. coli), a common inhabitant of the human intestinal tract ( Figure 1.6). The cell is rod-shaped, about 1 μm in diameter and about 2 μm long. Like most other prokaryotes, E. coli is surrounded by a rigid cell wall composed of polysaccharides and peptides. Within the cell wall is the plasma membrane, which is a bilayer of phospholipids and associated proteins. Whereas the cell wall is porous and readily penetrated by a variety of molecules, the plasma membrane provides the functional separation between the inside of the cell and its external environment. The DNA of E. coli is a single circular molecule in the nucleoid, which, in contrast to the nucleus of eukaryotes, is not surrounded by a membrane separating it from the cytoplasm. The cytoplasm contains approximately 30,000 ribosomes (the sites of protein synthesis), which account for its granular appearance.

Eukaryotic Cells

Like prokaryotic cells, all eukaryotic cells are surrounded by plasma membranes and contain ribosomes. However, eukaryotic cells are much more complex and contain a nucleus, a variety of cytoplasmic organelles, and a cytoskeleton ( Figure 1.7). The largest and most prominent organelle of eukaryotic cells is the nucleus, with a diameter of approximately 5 μm. The nucleus contains the genetic information of the cell, which in eukaryotes is organized as linear rather than circular DNA molecules. The nucleus is the site of DNA replication and of RNA synthesis; the translation of RNA into proteins takes place on ribosomes in the cytoplasm.

In addition to a nucleus, eukaryotic cells contain a variety of membrane-enclosed organelles within their cytoplasm. These organelles provide compartments in which different metabolic activities are localized. Eukaryotic cells are generally much larger than prokaryotic cells, frequently having a cell volume at least a thousandfold greater. The compartmentalization provided by cytoplasmic organelles is what allows eukaryotic cells to function efficiently. Two of these organelles, mitochondria and chloroplasts, play critical roles in energy metabolism. Mitochondria, which are found in almost all eukaryotic cells, are the sites of oxidative metabolism and are thus responsible for generating most of the ATP derived from the breakdown of organic molecules. Chloroplasts are the sites of photosynthesis and are found only in the cells of plants and green algae. Lysosomes and peroxisomes also provide specialized metabolic compartments for the digestion of macromolecules and for various oxidative reactions, respectively. In addition, most plant cells contain large vacuoles that perform a variety of functions, including the digestion of macromolecules and the storage of both waste products and nutrients.

Because of the size and complexity of eukaryotic cells, the transport of proteins to their correct destinations within the cell is a formidable task. Two cytoplasmic organelles, the endoplasmic reticulum and the Golgi apparatus, are specifically devoted to the sorting and transport of proteins destined for secretion, incorporation into the plasma membrane, and incorporation into lysosomes. The endoplasmic reticulum is an extensive network of intracellular membranes, extending from the nuclear membrane throughout the cytoplasm. It functions not only in the processing and transport of proteins, but also in the synthesis of lipids. From the endoplasmic reticulum, proteins are transported within small membrane vesicles to the Golgi apparatus, where they are further processed and sorted for transport to their final destinations. In addition to this role in protein transport, the Golgi apparatus serves as a site of lipid synthesis and (in plant cells) as the site of synthesis of some of the polysaccharides that compose the cell wall.

Eukaryotic cells have another level of internal organization: the cytoskeleton, a network of protein filaments extending throughout the cytoplasm. The cytoskeleton provides the structural framework of the cell, determining cell shape and the general organization of the cytoplasm. In addition, the cytoskeleton is responsible for the movements of entire cells (e.g., the contraction of muscle cells) and for the intracellular transport and positioning of organelles and other structures, including the movements of chromosomes during cell division.

The eukaryotes developed at least 2.7 billion years ago, following some 1 to 1.5 billion years of prokaryotic evolution. Studies of their DNA sequences indicate that the archaebacteria and eubacteria are as different from each other as either is from present-day eukaryotes. Therefore, a very early event in evolution appears to have been the divergence of three lines of descent from a common ancestor, giving rise to present-day archaebacteria, eubacteria, and eukaryotes. Interestingly, many archaebacterial genes are more similar to those of eukaryotes than to those of eubacteria, indicating that the archaebacteria and eukaryotes share a common line of evolutionary descent and are more closely related to each other than either is to the eubacteria ( Figure 1.8).

A critical step in the evolution of eukaryotic cells was the acquisition of membrane-enclosed subcellular organelles, allowing the development of the complexity characteristic of these cells. The organelles are thought to have been acquired as a result of the association of prokaryotic cells with the ancestor of eukaryotes.

The hypothesis that eukaryotic cells evolved from a symbiotic association of prokaryotes- endosymbiosis-is particularly well supported by studies of mitochondria and chloroplasts, which are thought to have evolved from bacteria living in large cells. Both mitochondria and chloroplasts are similar to bacteria in size, and like bacteria, they reproduce by dividing in two. Most important, both mitochondria and chloroplasts contain their own DNA, which encodes some of their components. The mitochondrial and chloroplast DNAs are replicated each time the organelle divides, and the genes they encode are transcribed within the organelle and translated on organelle ribosomes. Mitochondria and chloroplasts thus contain their own genetic systems, which are distinct from the nuclear genome of the cell. Furthermore, the ribosomes and ribosomal RNAs of these organelles are more closely related to those of bacteria than to those encoded by the nuclear genomes of eukaryotes.

An endosymbiotic origin for these organelles is now generally accepted, with mitochondria thought to have evolved from aerobic bacteria and chloroplasts from photosynthetic bacteria, such as the cyanobacteria. The acquisition of aerobic bacteria would have provided an anaerobic cell with the ability to carry out oxidative metabolism. The acquisition of photosynthetic bacteria would have provided the nutritional independence afforded by the ability to perform photosynthesis. Thus, these endosymbiotic associations were highly advantageous to their partners and were selected for in the course of evolution. Through time, most of the genes originally present in these bacteria apparently became incorporated into the nuclear genome of the cell, so only a few components of mitochondria and chloroplasts are still encoded by the organelle genomes.

The Development of Multicellular Organisms

Many eukaryotes are unicellular organisms that, like bacteria, consist of only single cells capable of self-replication. The simplest eukaryotes are the yeasts. Yeasts are more complex than bacteria, but much smaller and simpler than the cells of animals or plants. For example, the commonly studied yeast Saccharomyces cerevisiae is about 6 μm in diameter and contains 12 million base pairs of DNA ( Figure 1.9). Other unicellular eukaryotes, however, are far more complex cells, some containing as much DNA as human cells have ( Table 1.2). They include organisms specialized to perform a variety of tasks, including photosynthesis, movement, and the capture and ingestion of other organisms as food. Amoeba proteus, for example, is a large, complex cell. Its volume is more than 100,000 times that of E. coli, and its length can exceed 1 mm when the cell is fully extended ( Figure 1.10). Amoebas are highly mobile organisms that use cytoplasmic extensions, called pseudopodia, to move and to engulf other organisms, including bacteria and yeasts, as food. Other unicellular eukaryotes (the green algae) contain chloroplasts and are able to carry out photosynthesis.

Multicellular organisms evolved from unicellular eukaryotes at least 1.7 billion years ago. Some unicellular eukaryotes form multicellular aggregates that appear to represent an evolutionary transition from single cells to multicellular organisms. For instance, the cells of many algae (e.g., the green alga Volvox) associate with each other to form multicellular colonies ( Figure 1.11), which are thought to have been the evolutionary precursors of present-day plants. Increasing cell specialization then led to the transition from colonial aggregates to truly multicellular organisms. Continuing cell specialization and division of labor among the cells of an organism have led to the complexity and diversity observed in the many types of cells that make up present-day plants and animals, including human beings.

Plants are composed of fewer cell types than are animals, but each different kind of plant cell is specialized to perform specific tasks required by the organism as a whole ( Figure 1.12). The cells of plants are organized into three main tissue systems: ground tissue, dermal tissue, and vascular tissue. The ground tissue contains parenchyma cells, which carry out most of the metabolic reactions of the plant, including photosynthesis. Ground tissue also contains two specialized cell types (collenchyma cells and sclerenchyma cells) that are characterized by thick cell walls and provide structural support to the plant. Dermal tissue covers the surface of the plant and is composed of epidermal cells, which form a protective coat and allow the absorption of nutrients. Finally, several types of elongated cells form the vascular system (the xylem and phloem), which is responsible for the transport of water and nutrients throughout the plant.

The cells found in animals are considerably more diverse than those of plants. The human body, for example, is composed of more than 200 different kinds of cells, which are generally considered to be components of five main types of tissues: epithelial tissue, connective tissue, blood, nervous tissue, and muscle ( Figure 1.13). Epithelial cells form sheets that cover the surface of the body and line the internal organs. There are many different types of epithelial cells, each specialized for a specific function, including protection (the skin), absorption (e.g., the cells lining the small intestine), and secretion (e.g., cells of the salivary gland). Connective tissues include bone, cartilage, and adipose tissue, each of which is formed by different types of cells (osteoblasts, chondrocytes, and adipocytes, respectively). The loose connective tissue that underlies epithelial layers and fills the spaces between organs and tissues in the body is formed by another cell type, the fibroblast. Blood contains several different types of cells, which function in oxygen transport (red blood cells, or erythrocytes), inflammatory reactions ( granulocytes, monocytes, and macrophages), and the immune response ( lymphocytes). Nervous tissue is composed of nerve cells, or neurons, which are highly specialized to transmit signals throughout the body. Various types of sensory cells, such as cells of the eye and ear, are further specialized to receive external signals from the environment. Finally, several different types of muscle cells are responsible for the production of force and movement.

The evolution of animals clearly involved the development of considerable diversity and specialization at the cellular level. Understanding the mechanisms that control the growth and differentiation of such a complex array of specialized cells, starting from a single fertilized egg, is one of the major challenges facing contemporary cell and molecular biology.

# data[].attributes.attachment_link.description
Cells are divided into two main classes, initially defined by whether they contain a nucleus. Prokaryotic cells (bacteria) lack a nuclear envelope; eukaryotic cells have a nucleus in which the genetic material is separated from the cytoplasm.

# data[].attributes.body
I received feedback on Monday's class which confirmed the organisms and audio reactive orb are functional and seem to be landing the prompt of surreal. The AI dancing video however isn’t quite achieving this and appears to resemble some current memes, which breaks the sense of anything surreal and feels novel at best. I also think it would be best to separate the organism and audio reactive elements into two projects which feel more contextually and aesthetically appropriate. Further feedback suggests I may not need multichannel sound to achieve the desired affect, however this may add an additional layer of immersion if time allows for this additional element.

On Thursday’s class we visited RMIT Gallery to see The Hideous Replica exhibition. This exhibition featured experimental pieces that could be considered uncanny and based on the framework of “monstrous replication”. The piece I found the most interesting involved a mechanical silhouette designed to imitate a distorted hand and flora. While witnessing the piece I felt as if I was camping in the set of a horror film, the proportions of the hand and staggering motion were uncomfortable and erie. This illusion was broken when looking behind the veil to see an array of mechanical cogs driving the creepy facade.

# data[].attributes.attachment_link.description
We’re currently checking your connection. This shouldn’t take long.If you keep getting this page, check if there’s a problem with Canva.

# data[].attributes.body
Infinite Library Podcast (2022) ‘Why Nam June Paik's "TV Buddha" Is So Cool’ [video], Infinite Library Podcast, Youtube website, accessed 15 September 2024. https://youtu.be/F3NWNqH2Ayw?si=nwCUpII-GgQWaaSh

Jensen J (2011) Aesthetics of chance, pissinginthewind website, accessed 15 September 2024. https://www.pissinginthewind.no/Bilder/TXT/Aestetics%20of%20chance.pdf

# data[].attributes.body
This week I have locked in the video creation method and had success with this iteration of image and video generation. It appears I won’t need to use Stable Diffusion as an API within TouchDesigner which saves time and leads to simpler programming. I continue to add music to my surreal playlist and plan to start sound design and composition next week. I’m considering adding spoken word to the sound design and test against different visual content. I am also feeling confident that the experience should adhere to the prompt of surreal from initial user testing completed.

# data[].attributes.body
I decided to test haptic feedback using my subpac. While the experience is different to the music:notimpossible it should help me decide what types of haptic feedback in context to sound and vision are most affective. I noticed in the context of music that slower songs with pronounced bass worked well in terms of depth and definition. In terms of affect sparse bass in slow and sad music seemed to increase the emotional quality of the music which lines up with the findings from the ‘Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback’ paper.

What I found interesting was the relationship the subpac formed with video and games. I was playing the space exploration game ‘No Man’s Sky’ in which you have a jetpack which allows for faster movement and flight. When I triggered the jetpack (located on the characters back in the form of a backpack) I felt a rich sense of immersion being able to feel the subpack rumble in unison with my character. While watching video I noticed that people with deep voices would trigger the subpac which lead to a strange sensation of hearing and feeling the content of narration/speaker.

I also had the opportunity to test my current audio reactive content with the subpac, headphones and a screen. I had 5 subjects test this and all subjects mentioned they felt immersed in the experience and expressed joy and wonder from the novel experience.

# data[].attributes.body
Using the generation methods established in past weeks I created two videos. The first involves animating fictional organisms and seeing how long it takes before the program distorts the initial image. I wanted to establish the optimal amount of generations using the goldilocks principle.

The other was inspired by disjointed AI videos found on social media where typically a recognisable character engages in a fictional series of events usually in a humorous and uncanny manner. I found a collection on Adobe stock images using a single model on the same background gradient. I wanted the person to appear like they are dancing and this generation lead to a really interesting sequence that felt based in reality before distorted by the program - https://youtu.be/rZ_Tgl-JxxY

# data[].attributes.body
I was interested in exploring the origins of video art to get a better understanding of its original and applications. During this exploration I discovered the ‘Fluxus’ movement and Nam June Paik’s piece “TV Buddah”. While this movement feels underwhelming in modern context this piece was successful in achieving a deep hedonic tone with an internal locus. The piece involves a Buddah statue “watching” itself on a CRT TV via a live close circuit video feed. As the Buddah views itself you begin to wonder if it has achieved enlightenment in context to what was a modern world.

Another article I came across was the article ‘Aesthetics of Chance’ which felt appropriate given a large portion of my video content relates to probability. It portrays chance as a creative tool which reduces authorial control to all artwork to evolve independently from the artists initial intention. Indeterminacy and audience engagement lead to new directions and allows room for interpretation based on the subjects own experience. It can also inspire new aesthetics through fluid exploration and accidents that occur along the way.

# data[].attributes.attachment_link.title
Why Nam June Paik's "TV Buddha" Is So Cool

# data[].attributes.body
On Monday’s class we visited the RMIT Gallery Art collection store and were shown pieces that can help inspire work for our prompt. There was a notable piece in which the artist had carefully painted layered circles using watercolour paint. After completing each circle layer the artist then placed a single drop of watercolour paint and allowed it to drip. This contrast portrayed a sense of tension where the perfect circles met a sporadic and almost violent droplet which disrupted the perfect circles and resembled a droplet of blood or perhaps a raindrop depending on your interpretation.

I received feedback from Mark on the current state of my video content for Assignment 3. He enjoyed my current prototypes and mentioned the figure ground relationship was successful in this context. To further this relationship I am attempting to contrast the fictional and abstract content with content based in reality. I noticed while watching the anti-war protests on the news that seeing a space I’m quite familiar with filled with violence and chaos that this felt surreal and opened up a new interpretation of the prompt.

# data[].attributes.body
Schubert E, North A and Hargreaves D (2016) Aesthetic Experience Explained by the Affect-Space Framework, Empirical Musicology Review website, accessed 30 August 2024. https://emusicology.org/index.php/EMR/article/view/5115/4604

Easy Start AI (2024) ‘I Analyzed 500+ LumaLabs AI Generations: Here's How to Prompt’ [video], Easy Start AI, Youtube website, accessed 29 August 2024. https://youtu.be/OiNkRJMX_5U?si=Timpc65DOAH7pgL2

# data[].attributes.body
This week was spent testing AI generated video. The results were mixed but there was some success that offers potential for later iterations. Initially the challenge was attempting to create meaningful content without having to pay for subscriptions. I decided to sign up for Luma Dream and experience the difference between free and paid tiers. The paid tier is notably faster and allows for rapid iterations of material, however similar issues arose around the accuracy of prompts and poor quality generations. To test whether the issue is prompt or program related I created an AI prompt generator in Claude specifically using Luma Dream’s supplied glossaries and example prompts. While this improved the quality of the generations, the issue of accuracy persists. This has led me to the conclusion that to utilise AI content I should not attempt to rigidly control the generated content but rather let the program hallucinate in order to create interesting and engaging content.

# data[].attributes.body
Creating prompts for AI programs can vary and require different vocabulary based on the dataset and glossary it has been trained on. In order to test the capabilities of programs and create rapid iteration cycles I trained AI program “Claude” to generate prompts. I trained “Claude” using the method found on this Youtube tutorial with “Luma Labs” glossary of terms. Overall while helpful I must admit that there was still great variability when creating images or turning images to video. This confirmed that prompt engineering is useful, however limited by the hallucinations caused by taxonomy of training data and the interpretation of the word structure.

# data[].attributes.attachment_link.title
I Analyzed 500+ LumaLabs AI Generations: Here's How to Prompt

# data[].attributes.body
For creative practice I wanted to test “Luma Labs” ability to animate still images. The aim is to create something functional and restrained, then push the program and see where it leads. For the first experiment I created some images of fictional organism using stable diffusion then asked Luma Labs to add movement that can be interpreted as realistic. This was quite successful and happy to conclude that this method achieved its desired effect.

However the aptly titled Atrocity 1 lead to a very strange outcome the further I let the program iterate on its initial video. While the outcome isn’t ideal it has lead to curiosity around how I can potentially use this method to achieve an outcome that could be perceived as surreal.

https://youtu.be/s8WR9EmqwsM

# data[].attributes.body
Emotion valence references to the nature or property of the portrayed emotion. It depicts the internal experience of an individual that can be categorised by terms like happiness, sadness or anger. While affect valence relates more to the evaluation of an experience and whether it indicates a positive or negative experience. A piece of music may evoke a sense of happiness which encourages the person to continue listening. This can be seen as positive affect valence.

Deep hedonic tone can be described as an experience with significant and complex emotional engagement. This experience can be interpreted as profound and evokes an internal locus which feels meaningful or conveys a deeper, powerful message. While shallow hedonic tone is more superficial and doesn’t quite create meaning beyond aesthetics. It relates more to an external locus which appears to focus more on the object than deep emotional engagement.

# data[].attributes.attachment_link.title
Aesthetic Experience Explained by the Affect-Space Framework | Empirical Musicology Review

# data[].attributes.attachment_link.description
THIS paper presents a semantic framework that organizes critical aspects of aesthetic experience for psychological and philosophical research. Recent years have seen a boom in attempts to draw together theories and empirical data on aesthetic experience (Jacobsen, 2006; Leder & Nadal, 2014) . Yet, as some accounts point out, there exist numerous gaps and confusions (e.g., Augustin, Wagemans, & Carbon, 2012; Konečni, 2012; Locher, Overbeeke, & Wensveen, 2010). One of the most critical is the definition of the aesthetic experience itself. An aesthetic experience may be defined through the philosophical concept of disinterested contemplation (Levinson, 1992). It may involve a combination of cognitive and emotional processes (Locher, et al., 2010). It may be defined in terms of the qualities of the object (music, artwork, etc.) or in terms of the affective sensations of the individual as a result of contemplating the object (Beardsley, 1969). The act of creative production (such as a musician playing at a concert, an artist painting in a studio, or a composer at work) and its relationship to the perception and contemplation of art works are involved (Tinio, 2013). The brain activity that corresponds to aesthetic experience is another way of defining the process, although the neuroscientific understanding of aesthetic experience is in its infancy (Brattico, Bogert, & Jacobsen, 2013; Brattico & Pearce, 2013; Calvo-Merino, Jola, Glaser, & Haggard, 2008; Cela-Conde, Agnati, Huston, Mora, & Nadal, 2011; Chatterjee, 2004; Chatterjee & Vartanian, 2014; Jacobs, Renken, & Cornelissen, 2012; Thakral, Moo, & Slotnick, 2012; Trost, Ethofer, Zentner, & Vuilleumier, 2012; Vartanian & Skov, 2014; Zatorre & Salimpoor, 2013). Other authors have argued that it can refer to pure pleasure, highly profound feelings, or the formation of meaning, interpretation and understanding (Armstrong & Detweiler-Bedell, 2008; Bullot & Reber, 2013; Leder, Belke, Oeberst, & Augustin, 2004; Leder & Nadal, 2014). None of these perspectives on aesthetic experience are mutually exclusive (Pouivet, 2000).

# data[].attributes.body
Unfortunately I was unable to attend classes this week due to illness. I decided to spend time reflecting on the video content to support the multisensory experience. The goal is to lock in the video creation processes and focus on connecting TouchDesigner to Ableton and Stable Diffusion. While testing video content I must remain unbiased and honestly evaluate whether it creates a deep hedonic tone with an internal locus.

# data[].attributes.body
Luma (2024) Dream Machine (1.5 version) [video generator], accessed 24 August 2024. https://lumalabs.ai/dream-machine

NickName Game (2016) ‘Deadpool - Angel Of The Morning (Opening Scene - 1080p)’ [video], NoName Game, Youtube website accessed 23 August 2024. https://youtu.be/p1ze539XdLc?si=uNLl8c5FbomAXph8

Runway AI (2024) Runway (Gen-3 Alpha version) [video generator], accessed 24 August 2024. https://runwayml.com/

SöDerströM U, Larsson W, Lundqvist M, Norberg O, Andersson M and Mejtoft T (2022) Haptic feedback in first person shooter video games, Association for Computing Machinery website accessed 25 August 2024. https://dl.acm.org/doi/fullHtml/10.1145/3552327.3552333

Stability AI (2024) Stable Video Diffusion (1.1 version) [video generator], accessed 24 August 2024. https://www.stablevideo.com/welcome

# data[].attributes.body
Still seem to be experimenting with ways to generate video content but feeling more confident in creating enough material using the generative methods tested this week. I have also continued to think about accompanying music for experience and was reminded of the opening sequence to the film Deadpool. The use of “Angel of the morning” by Juice Newton accompanying violent slow motion footage was an excellent use of cognitive dissonance for comedic purpose and leads me to wanting to experiment with a similar idea.

# data[].attributes.attachment_link.title
Deadpool - Angel Of The Morning (Opening Scene - 1080p)

# data[].attributes.body
I wanted to explore other means of generating video content and experimented with AI programs “Luma”, “Runway” and “Stable Video”. All programs are designed to generate video from prompts, images or prerecorded video. While there were varying degrees of success there were some interesting outcomes particularly from “Luma”. A key difference between the programs is that “Luma” allows the user to select an end point image which creates a visual warping effect seen in the video above. It also allows the user to extend the sequence to create longer form video through the use of text prompts. This is extremely advantageous compared to other programs like “Stable Video” or “Runway” which require 5-10 second segments to be exported manually with fine text prompt modifications and edited together in video software. While all programs were limited by pay subscriptions I feel confident that by running “Stable Diffusion” natively for no cost and utilising a lower tier subscription to “Luma” I should be able to create enough video content for the multisensory experience.

# data[].attributes.body
I have updated my audio reactive prototype based on class feedback and having a better understanding of the updated engine used to generate the visuals. The softer colour palette now resembles an oil sleek and the motion is smoother with less moving points. This acts as a great counterpoint for the brighter colours that occur as music becomes more energetic. It also feels more intricate in the way it behaves to dynamic music like Piggy by Nine Inch Nails.

# data[].attributes.body
I wanted to investigate the use of haptic feedback in videogames to see if there are any learnings that can be applied to the music:notimpossible vest. In this study the participants were required to wear a haptic vest while playing first person shooter games. While the sentiment was mostly positive among participants there were two areas of interest to examine; increased awareness and enhanced realism. Some participants mentioned the vest could be distracting at times in regards to increased awareness and feel there is a goldilocks principal that can be applied to this notion. The perception of enhanced realism was high amongst participants with 83% expressing it added more immersion. This outcome fills me with confidence as immersion is the outcome I wish for with the music:notimpossible vest.

# data[].attributes.attachment_link.readable_text
Haptic feedback has been around for a long time and one of the earliest implementations was for large airplanes [ 12]. The use of different haptics in video games is common for gaming consoles today but the technology is still developing [ 2]. The overall goal with haptic feedback in game development is to make the player feel more connected to the game and make the player so immersed that he/she not only plays the game but also feels the game. This can be achieved through a controller by targeting joints, muscles, and skin receptors in the hands. When taking hold of a coffee cup, as an example, it will affect as many as 18 joints, 79 muscles, and 19.000 skin receptors [ 13]. Imagine how many joints and muscles that are going to be targeted when holding a controller with haptic feedback, or how many receptors that can be affected on more parts of the human body, e.g. the upper body.

With the popularity of the video game industry increasing and now surpassing movies in total revenue [ 21], a new trend where game developers are re-releasing old software for brand new game consoles and personal computers (PC) has emerged. This practice is often referred to as "remastering" the game, with the new release commonly providing enhanced graphical fidelity and renewed music and audio in comparison to the original release, but often also implementing haptic feedback in the new version. A modern game controller has the functionality to express haptic feedback, also called "rumble". These controllers produce rumble by one or several builtin haptic motors which can shake the controller at different frequencies.

In this paper we will investigate the ability to immerse the player in the game world in two different ways. Immersion, or the feeling of presence in the world, is described by Ijsselsteijn and Riva as a combination of real-time multisensory data and cognitive functions that changes dynamically as the user interacts with the world, no matter if that environment is physical or mediated in some manner [ 7]. Both experiments are carried out in first-person shooter (FPS) games; the first experiment uses haptic feedback only in the controllers and the second experiment a haptic vest that the participant is wearing. FPS games are a game type that are played in first person, where the player assumes the personality and body of the character in the game. This means that when playing, the player will rely upon what they can see on the screen, hear in the headphones and feel through haptic feedback. Is it possible to increase the immersion in the gameplay by targeting joints, muscles, and skin receptors throughout the chest, the back, and the hands?

The theory section gathers relevant explanations and definitions for this study. It introduces the different haptic technologies, the application of haptics, mental models and how these models are used to interpret our environment and define the state of immersion.

Haptic motors were first introduced for video game controllers with the Nintendo 64 in 1997 following the release of the Rumble Pak add-on [ 3]. Since then, the use of haptics in modern controllers has been following the same design philosophy, with two haptic motors located close to each palm of the player's hands. A haptic feedback vest was first presented in 1994 [ 11] but it has never been included as a standard part of video game equipment; standard haptics for games comes through the controller.

A mental model is a cognitive construction that represents situations and entities as well as the interrelationships between these two in a real or imaginary world [ 4]. It can be described as a merger between the knowledge a person has about the world with memories of past experiences, but with the term situations being used in a very broad sense [ 4]. They are formed through our previous experience in personal, social and mediated situations and can also adapt over time when faced with new information [ 15]. Mental models have in the past been used to understand a variety of phenomena, such as reasoning, problem-solving and spatial cognition [ 4]. When playing a video game the player will constantly be interacting with the game world and will in the process have to make split-second decisions about which actions to make. This will require the player to construct, activate and apply their own mental models [ 16] while simultaneously being influenced by the games own model, called mechanics. These mechanics are what defines the rules and physical properties of the game world, as well as the abilities of the avatar which can be used to solve the challenge that the game poses. These two models will have to be compared and evaluated for the player to interpret the game effectively. This process of comparing personal schemes to game information is called model matching [ 16]. Research shows that if a mental model of a real experience more closely resembles the model of the game, the player will not only get a greater sensation of realism but immersion as well [ 16]. As the gap between the game and the real-world based schemas narrows, the experience will feel increasingly real and more immersive.

In real life as well as mediated interactive or non-interactive media, the feeling of "being there" is the experience that immersion induces through multisensory stimuli from our environment [ 7]. The term can also be described in different scenarios as "spatial presence" (e.g. virtual reality), "telepresence" (e.g. video conference) with the feeling of presence in the environment being the common denominator [ 7]. Immersion can be recognized by a set of characteristics such as a displacement of self-perception, perceived transparency of the medium and a lack of awareness of time and space in the simulated or recreated reality [ 7]. These characteristics can be manipulated by several factors, such as the amount of sensory data being experienced, the control the user has over sensory mechanisms and the extent to which the user can change the environment [ 7]. In short, factors that create a rich mental model of the game world and factors that creates consistency between all aspects of the environment [ 14]. In video games, studies show that newer games that feature more vivid and realistic representations of graphical fidelity contribute to a higher sense of presence and feeling of being involved in the game world in comparison to older games [ 9, 10]. This matches the factor of an increase in sensory data being presented to the user at any given time. The perspective can also affect immersion, where a first-person perspective has been recorded to increase presence compared to a third-person perspective [ 6] as a result of having greater control and higher transparency to the player's mental models.

The definition of awareness is "the quality or state of being aware : knowledge and understanding that something is happening or exists". This is a vital part when playing FPS games. Audio is an important feedback for the player, by hearing the surrounding the player can distinguish what and where things are. However, this might sometimes be difficult to achieve only through sound and this is where the haptic vest could play a part. It could have the potential to improve the players awareness by giving more sensory information based on the players surroundings. The definition of Realism is "the theory or practice of fidelity in art and literature to nature or to real life and to accurate representation without idealization". Realism is maybe not something that is important for the players ability to play the game but it could be a vital part of the players ability to feel immersed in the game.

Ethics is something that plays a big part in the development of all new things, especially when it comes to electronics. Ethics within haptic feedback can be somewhat different and according to Morelli "we risk losing control not just of our data or privacy, but of our bodies themselves" [ 17] when it comes to haptics. This might be an extreme but ethics must be considered when designing interaction technology that physically can affect the user and this becomes even more true when the used should wear an entire vest, or be immersed in virtual reality [ 19].

The experiments were carried out separately, where we first will introduce the test with haptic feedback in the controllers and then we will describe the experiment with the haptic vest.

To investigate the effect that haptic feedback has on immersion a case study was conducted. The selection of participants, the experimental task, as well as test variables and the following questionnaire are presented below.

To investigate if haptic feedback improves the immersion in a first person shooter (FPS) game, an A/B test was conducted. The participants, materials used, and procedure are described in this section.

Move around in the map

Throw a grenade in your vicinity

Shoot with you primary and secondary weapon

Call in an airstrike, or be close to anyone else that calls in one

Could you feel which direction you got shot from?

Did it feel different when playing with the haptic feedback?

Would you say that you perceived you surroundings better?

Was there any difference in how you perceived explosions?

Could the vest be a distraction?

The result for the two experiments are described in the sections below.

An independent two-sided t-test was used to verify whether haptic feedback turned on would increase the immersion, attention and suspension of disbelief on a question-by-question basis from the MEC-SPQ questionnaire (Table 1). For all questions, the t-tests were conducted with a significance level p = 0.05. Regarding the scale SL, no statistical significance could be found between each test group, with each questions' t-value failing to exceed the critical value. The mean values for questions SL-1 through SL-3 do however indicate a slight increase in a sense of self-location when haptic feedback is turned on, see table I. The highest t-values were also recorded for this scale, with values ranging from 0,68 to 0,90 which still is significantly off from exceeding the critical value. All of the answers in both groups were also highly sensitive to differences in the answers, as indicated by the highest average variances and standard deviations that were recorded. The scale of PA also fails to reach statistical significance, but yields slightly higher scores for questions PA-3 and PA-4 for haptic feedback turned on in comparison to haptic feedback turned off, see table I. PA-4 with haptics turned on even recorded 32% higher mean values compared to haptics turned off, the largest difference in mean across all tested questions. Variance and standard deviation values also recorded high levels, especially when haptic feedback was turned on. Finally, the last two sub-scales AA and SD also fails to reach statistical significance across all tested points. As stated with previous sub-scales, the critical values are far from reaching any solid conclusions and high variance and standard deviations inhibit any sort of decisive patterns to be found. AA did however score the highest mean values in both categories indicating that the participants focus on the game had very little impact on the results for the other scales. Question SD-1 asks about participant focus on inconsistencies in the game also shows a 30% higher mean value with haptics on, which is the second-largest gap between the test groups across all questions.

The feedback and opinions from the user tests showed a positive attitude to the vest and its haptic feedback in general. However, some subjects had some negative experiences with the vest.

Awareness: one out of six users felt which direction they were shot from when taking fire. Five of the six did not feel the direction but they perceived the action of getting shot at with more awareness. One of the subjects even said, "You know what is happening nearby, everything that is happening nearby". When it came to explosions or larger sounds like grenades, explosions, and airstrikes the feedback was unified as six out of six felt it more. All six subjects also said that explosions felt more "directional".

Realism: The vest had a big impact on the realism of the game. Five out of six subjects felt that the vest brought more realism to the game. Some even said that "it felt more like they assumed control of the in-game character". The one that had a different opinion felt more or less the same. The difference was that the subject felt the vest a bit to much when sitting down. However, the one subject that raised some concerns regarding the vest also felt that the vest brought more realism to the game in one way or another.

"Felt more realistic, liked it sharply. Felt better. You felt more like the person in the game. One more sense is added to the gameplay."

"Definitely more realistic. Larger explosions felt most realistic. You felt like you were shooting when shooting with the vest on."

There were also some dislikes from the subjects as to how the vest worked (even though it added some realism). The concern/dislike was that the vibrations emitted from the vest all felt the same according to the subject, i.e. the participant did not experience any difference in the vibrations pattern regardless if the vibration came from e.g. a gunshot or an explosion. When you can see what happens on the screen it is easy to relate the vibrations to the correct event. But when you don't see the event you will most likely require some training to be able to discriminate between different vibrational patterns. The emitted patterns are different, some user simply didn't experience a difference. Another comment was that the vest was a bit clumsy and noticeable, in other words, the subject didn't appreciate wearing the vest itself.

The survey (described in section 3.2.3) shows that the majority of the test subjects like the vest and thought that the haptic feedback enhanced the gaming in some way.

Awareness: Answers from the survey show that most of the test subjects thought that the vest provided some extra awareness to the game-play. The negative feedback in this part was mainly the materials of the vest and how it fit the person wearing it. 50% Think that the vest could be a distraction when answering the question in the survey.

Realism: 83% Of the answers in the survey showed that the vest, according to the subjects, added more reality to the game and were positive to the vest, with answers like:

"I liked that the vest made me feel immersed in the battle."

"It was a fun experience, you really feel like you are part of the game." However, there were still some dislikes in the area of realism as well. One example of that is the following answer from the survey "It was a bit too uncomfortable while sitting down. I think it could be effective, but not so much in this area in my opinion. The vibrations from different things were often really similar which made it hard to tell what it was supposed to represent."

This was, however only one one isolated answer, as the five other subjects had more positive thing to say.

Slightly more participants said that they experienced shooting of their own more in the haptic vest compared to being shot by someone else, but almost everyone answered both questions with 3 or higher. The "free text" questions also supported the numbers as four out of six subjects stated that the haptic feedback positively impacted the realism in some way.

A linguistic analysis was made from the results of both the survey and the user tests (the think aloud sessions). This was done to strengthen the results from both the survey and user tests. The result from the analysis is shown in table 2.

The experiment with haptic feedback in the controller indicate some interesting results. The resulting metrics for the MEC-SPQ sub-scale Attention Allocation (AA) shows that Halo 2 was able to greatly affect the player' ability to focus and concentrate on the game. Across both groups, the mean values for each question are towards the higher end of the spectrum, which would suggest that the presence or absence of controller haptics is not a factor in deciding players attention allocation while gaming. The first sub-scale closely tied to immersion, Self Location (SL), did not score as highly as AA, with results hovering just slightly above 3 on the 5-point grading scale. This means that the participants only partly felt as if they were physically present within the game environment. Adding the sensation of touching and feeling the game through haptic feedback, did however show that participants were slightly more able to mentally model themselves in the shoes of the player character, as shown in question SL-3, but only at a modest 9% increase and still not significant in the t-test. One could argue that this result implicitly shows that aspects that affect human senses, such as graphics and audio, might be more important than touch in triggering immersion of self-location. Possible Actions (PA) is the second sub-scale focused on immersion and saw similar results as SL overall with the mean values around 3 points out of 5 with some higher values recorded with haptic feedback turned off. One could suggest that this can be the case because of the particular choice of a game rather than a clear factual statement for immersion in a first-person shooter in general. Halo 2, even in the remastered Anniversary edition is quite simplistic in its gameplay, physics and game mechanics which could contribute to the lacking sense of being able to control and affect the environment. The level chosen for the user test, Outskirts, could also affect the game-play as it forces the player to defend a closed-off plaza before opening up the rest of the level. Only two participants reached that point in the level where it opens up beyond the plaza, which could be one of the reasons why participants felt like they had very limited actions at their disposal.

In the vest experiment there where both positive and negative results from the user test and the survey. This may be due to several different factors but we were not evaluating the vest itself in this paper; the vest is merely the means of distribution. However, just by looking at the result, one can see that the "delivery-system" i.e. the vest affects the results, and will therefore be taken into consideration in the discussion.

Awareness is one of the measurements that will determine how immersive the vest and its haptic feedback can make FPS games feel. Awareness is essentially the ability to know your surroundings and what is happening around you.When looking at the results from both the survey and the user tests we can see that the players ability to not only hear the surroundings but also feel it makes them more aware of their surroundings. However, there was also some negative feedback regarding the awareness. These was the materials of the vest and how it felt when wearing it. At first, this was not something that we considered to be an issue since the vest merely is the chosen way of delivery, and another way of delivery might not have the same problem [ 16]. However, after analyzing the results we concluded that the vest material and general fit on the person wearing them plays a big part in how the haptics are perceived. This might be the reason for the negative feedback regarding the awareness in the study, even if a majority of the participants thought that the vest enhanced the awareness of the game.

Realism can be a hard term to measure in a first-person shooter game. However, in the experiment with the haptic vest, realism was measured by the players' ability to feel; feel whether they were shooting or being shot at through the haptics of the vest. Other in-game events such as vehicles passing by, explosions, etc. where also accounted for in the results. Just by looking at the results, we can see a trend of positive feedback regarding the vest and its haptic feedback from both interviews and the survey. This result correlates to previous studies done where haptic feedback made the game more realistic as the players felt things more realistic. In the study [ 8] they tested haptic feedback along the back of a custom-made chair. Other research [ 5] shows that haptic feedback is used in learning. To simulate how a task should feel when doing it for real has been done with haptic feedback. This really shows how useful haptic feedback can be in the regards to realism. This also strengthens the results that the haptic feedback from the vest improves how realistic the game is perceived.

The negative feedback, regarding realism was more surrounding the vest itself, than the performance of the haptic feedback produced by the vest. This might be because the vest isn't optimized for each individual but instead being a "one size fits all" equipment.

One other thing that have to be taken into consideration is ethics. How realistic should a FPS game be? Is it ethical for this kind of games to feel like the real world? We also have to consider the players body. Since haptic feedback have brought up a new kind of security issue "we risk losing control not just of our data or privacy, but of our bodies themselves" [ 1]. The ethics includes the players' physical state as well which can be seen as a limitation in regards to realism.

The majority of the test subjects thoughts were more positive, than negative, regarding the vest and its haptic feedback. The linguistic analysis also strengthens those results by illuminating commonly used words during the user tests and surveys. We can clearly see that the positive words used were all related to awareness and realism. The negative words that occurred were closely related to the vests fabrics/materials and the general feeling of wearing the vest and the number of negative comments and negative feedback was significantly lower than the positive. The vest indeed improve how immersive the game feels. By improving awareness and realism, the haptic feedback contributes to more spatial presence, e.g more immersion.

The studies contains some limiting factors that may have had an impact on the final results.

One limitation is the number of participants in the two experiments. More participants would potentially help to minimize the high variance and standard deviation statistics that are common throughout the studies.

The questionnaire was another, intentionally, limited factor. Instead of the entire MEC-SPQ suite of questions, we chose to only include half of the measuring sub-scales and to use the 4-question version instead of the full 8-question version.

Perhaps the difficulty setting for the first experiment was another contributing factor as to why so few participants reached the point in the level beyond the plaza. The normal difficulty setting was used, which is the default setting when starting a new game. However, many participants said that they died repeatedly during their playtime and all participants died at least once. Dying in the game is an event that could easily pull you out of the experienced immersion and may have been a contributing factor to the inconclusive results as well.

The vest that was used was not tailored-made to fit the participants. A more suitable vest could be used, meaning that different vest sizes should be used, or all participants should have a similar body shape, that fits the vest.

Another limiting factor can be the limited playing time and limited experience with the haptic vest. More experience with the vest and it's vibrational patterns could possibly reduce some negative comments about the vest and it's usage.

The main objective was to find out whether haptic feedback would increase the sense of immersion that a player feels when playing. No statistical difference could be found in the results but several factors indicate that haptic feedback actually will improve the immersion. Haptic feedback that is targeting the upper body, i.e back and chest have a positive effect and makes the experience of FPS games more immersive. The delivery system, i.e., the vest received some negative feedback and described as clumsy, stiff, and uncomfortable by a few of the test participants. However, the majority did not raise these concerns so our conclusion is that the vest works and that haptic feedback improves how immersive FPS games feels. This is accomplished by adding more in-game awareness and realism to the player which enhances the players spatial presence.

# data[].attributes.attachment_link.title
Haptic feedback in first person shooter video games

# data[].attributes.attachment_link.description
Haptic feedback has been around for a long time and one of the earliest implementations was for large airplanes [ 12]. The use of different haptics in video games is common for gaming consoles today but the technology is still developing [ 2].

# data[].attributes.body
After receiving feedback for assignment 3 I decided to spend time experimenting with the colour palette of my first audio reactive prototype in TouchDesigner. Previously I was using vibrant colour palettes that would change based audio parameters. While happy with the effect, it seemed to lose impact over time and I believe this problem could be solved with restraining the colours by washing out the hue with white. I tested 3 variations of colour and asked Nick and other students which they preferred and the results were nearly unanimous with the most muted colours being the most popular.

At the capitol we got to see the work of other students and discuss progress. It was interesting to see how others have interpreted their prompt through the use of light, colour and sound choices. Notably there were two that I found interesting for different reasons. There was a lighting sequence that quickly changed from red to blue which led to a very hypnotic effect when paired with appropriately repetitive music. Though simple it achieved the desired state quite quickly and I felt immersed in the piece. The other was based on the prompt of uncanny and used eerie field recordings of office equipment which was engaging sonically.

# data[].attributes.body
Elekktronaut (2020) ‘Controlling Video With Audio – TouchDesigner Tutorial 14’ [video], Elekktronaut, Youtube website accessed 15 August 2024. https://youtu.be/oviwpILXo5A?si=wQ0sj-Z_wG6l6N-K

Elekktronaut (2022) ‘ Perform Mode / Projector Setup – TouchDesigner Tips, Tricks & FAQs 15’ [video], Elekktronaut, Youtube website accessed 15 August 2024. https://youtu.be/UBmTkOCovus?si=VMiVIdNHHzSHN2HB

Zheng D, Hynds D, Kunze K and Minamizawa K (2021) ‘Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback’ ResearchGate website, accessed 17 August 2024. https://www.researchgate.net/publication/351426474_Haptic_Empathy_Conveying_Emotional_Meaning_through_Vibrotactile_Feedback

# data[].attributes.body
This week I continued to test different methods of generating content in TouchDesigner using audio to control shape/noise and tempo parameters. I also tested whether there is potential to performance the experience in a live context.

I also started putting together a music playlist to compliment the potential visual components. Notably I want to ensure the music has strong emotional qualities to help drive the narrative and provide opportunities to further iterate into a multichannel audio context.

In terms of narrative structure I have been considering whether to create an AI generated song to include in the experience as potential comic relief, however this still needs to be investigated further and tested.

# data[].attributes.body
I wanted to see how viable it would be to complete a live performance in context to the multisensory experience. Through experimentation in TouchDesigner I was able to use an “audio device in” node which allowed my audio interface inputs to be used as data to manipulate visual content. I was able to use my Modal Cobalt8X synthesiser in real time to produce audio reactive material with minimal latency issues. For larger scale performances with higher resolution or more complex data chains a “lag” node can be used to minimise latency issues at the cost of being slightly out of time with playback. For textural pieces there shouldn't be any issues, but may prove problematic with rhythm based pieces in a live context. Additionally there may be benefits in testing an overall latency compensation to see the impact on overall latency. The next step will involve testing a MIDI device like a Novation Launchpad to see if this can be used to control video content within TouchDesigner to use in conjunction with live audio.

# data[].attributes.body
I created two mock ups this week to test different ways to utilise audio frequency data and video speed to create content.

AR Experiment 2 is an iteration of last week’s experiment. I have changed the visual generation to appear smoother and more reactive to new streams of audio data. I used different songs to test how this updated visualiser behaves with different genres and was impressed with how dynamic and different the behaviours were.

Video Link - https://youtu.be/WKjNWSpK-Vc?si=4TxVvRU4oAOTf6qD

In Speed Test 1 I used a video tempo based approach in which the kick drum of the music will effect the playback speed of the video. It does function but requires more work to smooth the visuals and prevent grainy playback. I think method may come in useful for video art components of the experience.

Video Link - https://youtu.be/GDE2l0voPnk?si=XGpW6mjmgID1EM06

# data[].attributes.body
Haptic Empathy: Conveying Emotional Meaning through Vibrotactile Feedback

While the study was limited to 7 participants (3 male 4 female) between 24-30 years old there were some interesting findings. The study used vibrotactile feedback to convey four emotions; joy, anger, sadness and relaxation.

In the study these four emotions can be defined through the following vibrotactile patterns:

Joy: “rhythmic patterns are mostly consistent and prominent. The rhythms tend to be more quantized than others.”

Anger: “rhythmic patterns are very prominent and constant. These patterns are more driving and abstract than joy.”

Sadness: “rhythmic content is sparse and tends to come in short patterns/gestures, with large sections of inactivity.”

Relaxation: “rhythmic content/density is quite sparse. The amount of rhythmic content is nearly non-existent.”

The study reported that the accuracy of conveying these four emotions was comparable to recognition of facial emotions. Joy and anger were the easiest to identify, joy had an accuracy rate of 71.43% while anger received 65.30%. While lower in accuracy relaxation scored 51.02% and sadness 46.94%.

Based on the results joy and anger and quite easily expressed and understood through the medium of vibrotactile feedback while relaxation and sadness are more difficult. I am curious if this potentially relates to the level of arousal/activation defined by the circumplex model of emotions.

# data[].attributes.body
After completing the Assignment 2 presentation I was presented with feedback that challenged my concept of haptic feedback. Previously I thought that haptic feedback can be treated like lighting in the sense that it is meaningless but can direct audience attention and imply importance in context to visual hierarchy. This may not be correct and will need to deepen my research into how haptic feedback can in fact convey emotional meaning. Through this research I may be able to add another level of narrative structure which highlights this affective quality without relying on multisensory relationships.

I was also reminded about the concept of the Umwelt which involves the unique sensory perception of an organism. This may aid in the narrative I’m attempting to convey but depicting the life cycle of a single celled organism. By allowing the audience to empathise with the organism I can increase the level of emotion and create a more meaningful story.

# data[].attributes.body
Acrylicode (2023) ‘Audio-reactive psychedelic visuals | TouchDesigner Beginner Tutorial’ [video], Acrylicode, Youtube website accessed 6 August 2024. https://youtu.be/Mt2hwb5cngA?si=8pwpCJhMjxXXvT32

Lafta A, Ayad A and Abeer J (2017) ‘A Systematic Review of Deep Dream’ Iraqi Journal of Computers, Communications, Control & Systems Engineering website, accessed 8 August 2024. https://www.iasj.net/iasj/download/aa72b36a05d3b64b

Reflekkt (2024) ‘Audio Reactive Instancing - TouchDesigner Tutorial 008’ [video], Reflekkt, Youtube website accessed 7 August 2024. https://youtu.be/SSSSbq2YZcA?si=iGYIiRjHPyWYY4g6

# data[].attributes.body
My aim was to map out the technical components of the multisensory experience, categorise the functions of haptic feedback, experiment with visual aesthetics and begin thinking about narrative structure. On miro I have created a specification layout and continue to investigate the best method of executing the experience. Currently I believe that using TouchDesigner with Stable Diffusion and Ableton running as APIs allows the most flexibility for generating and displaying content.

Miro Board Link - https://miro.com/app/board/uXjVKvai2lM=/?share_link_id=567595574476

# data[].attributes.body
The visual and audio reactive (AR) engine in the creative practice video was from a tutorial by youtuber "reflekkt". However other methods were trialed including TouchDesigner's inbuilt engine.

Another tutorial I recreated was by "Acrylicode" which displays a geometric AR pattern. The engine was a far simpler design, but the logic of the sequence was more effective. The "reflekkt" tutorial made the AR engine overly complicated with only a single stream of meaningful audio data. I was able to correct this by adjusting earlier nodes in the signal chain.

While enjoying the rudimentary aesthetic from "reflekkt"s tutorial the AR engine was terrible. It had too many nodes with meaningless values assigned. In the end it was only using bass and time to generate animation. Even when it's modified to use the correct data it felt less efficient than other methods.

# data[].attributes.attachment_link.title
Audio-reactive psychedelic visuals | TouchDesigner Beginner Tutorial

# data[].attributes.body
I created a mock up of audio reactive visuals in Touch Designer. I was trialling a new method of audio reactivity by using different data sources such as time to drive visuals compared to the frequency band method used in the past. Overall did work, but seemed less responsive compared to the prior method. I will need to experiment further to determine if this is due to the method or input parameters. The score was created in Logic Pro and was influenced by late 90’s trip hop and ambient music.

# data[].attributes.body
A Systematic Review of Deep Dream

Lafta R. Al-Khazraji, Ayad R. Abbas, Abeer S. Jamil

Deep Dream is a program created by Google to enhance and manipulate images input by a user. What I found interesting about this article relates to the surreal aesthetic created by the augmentation of the original image. This can be attributed to the processing component which uses a convolutional neural network. The aesthetic is completely unintentional and a byproduct of the neural network used.

# data[].attributes.body
Feedback from the assignment 1 presentation has confirmed that I will need to continue working in two streams for the visual content. These streams can be determined as audio reactive content and narrative structure. The audio reacted content can be used to convey a sense of synthesia when combined synchretically with audio. Narrative content can be created with generative engines like Stable Diffusion and 3D modelling engines like Unity or Unreal Engine.

After another demo of the music: not impossible vest I’ve drawn the conclusion haptic feedback has no actual meaning. This is similar to light where it can be used to imply significance or further the emotional affect of the narrative. Because of this finding I spent some time categorising the different types of haptic feedback in order to determine possible combination with sound and vision. They can be categorised as follows;

Oscillate - vibration increases and decreases in intensity representing a sine waveform

Pulse - vibration plays in quick rhythmic bursts
Shock - a single burst of intense vibration with varying duration
Ramp - vibration gradually increases or decreases in a linear manner

Rhythm - vibration mimics the rhythm of music or sound from an audio source

# data[].attributes.attachment_link.description
We’re currently checking your connection. This shouldn’t take long.If you keep getting this page, check if there’s a problem with Canva.

# data[].attributes.body
AI Covers (2024) ‘Boulevard of Broken Dreams AI Cover ft Frank Sinatra’ [video], AI Covers, Youtube website, accessed 3 August 2024. https://youtu.be/wd5EvdD1A64?si=R5cbKO2euOClYjZD

Gardland, A (2018) Annihilation [motion picture], Paramount Pictures, United States.

MeowCentre (2023) ‘Careless Whisper - Frank Sinatra AI Cover’ [video], MeowCentre, Youtube website, accessed 3 August 2024. https://youtu.be/_Y543NEiR5w?si=7K7r-AsvzHTCBYIC

Payne J and Nadel L (2004) ‘Sleep, dreams, and memory consolidation: The role of the stress hormone cortisol’, National Library of Medicine website, accessed 4 August 2024.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC534695/

Villeneuve, D (2016) Arrival [motion picture], Paramount Pictures, United States.

# data[].attributes.body
For assignment 3 I have been thinking of potential narrative for the video component of the multisensory experience. I believe that portraying the lifecycle of a fictional organism could be an interesting idea that lends to a surreal experience. Through researching assignment 1 I noticed that death and memory are themes found throughout multiple pieces. The lifecycle of a fictional organism could employ a similar narrative device if proven to be affective in this context.

I have also started to look at surreal examples of nature for inspiration. I wanted to examine a few examples of plants that appear surreal aesthetically. I must admit my favourite example was the romanesco broccoli with its spiralling, almost kaleidoscopic appearance resembling a Tool album cover. Second, the Crassula Buddha's Temple with its geometric scales. These structures may act as inspiration for flora that accompanies the fictional organism.

# data[].attributes.body
For the visual components I am planning to use Unity or Unreal Engine. You can create 3D models and animation in both platforms, however Unreal Engine appears to be leading in animation quality. In the past I have been limited by my computer's CPU and GPU but recently ordered a new mac which will overcome this hurdle. I can combine the videos produced by Unreal Engine with generative visual overlays to create interesting effects and interactions.

# data[].attributes.body
I wanted to create a mock up video displaying variations of fictional organisms that act as potential inspiration for a narrative concept. I also wanted to experiment with sound by recording my Modal Cobalt 8X hardware synth and manipulating the sound with time stretching and audio effects. The visual components were generated by AI program stable diffusion and animated using stable video.

Miro board link - https://miro.com/app/board/uXjVKvai2lM=/?share_link_id=175559525128

# data[].attributes.body
Assignment 1 allowed the opportunity to reflect on works that lend to the prompt of “surreal” and find common themes in the content. An interesting theme found in the films Arrival and Annihilation involved the portrayal of fragmented memory producing a dream-like quality. I wanted to understand the biological mechanism that creates this illusory and sometimes concerning byproduct of sleep. It appears that the fragmentation and episodic memory related to sleep is caused by cortisol levels comparable to a mild-moderate stressor response. When the brain attempts to process the fragmented dreams it creates a narrative which can influence the content of the dream and create strange and distorted reinterpretations.

# data[].attributes.body
After discussing the models of emotion and affect in class I wanted to examine some instances of AI generated music that fit outside of typical uncanny or humorous qualities. I have noticed that voice modelling is becoming quite advanced, in particular a model of Frank Sinatra. The AI cover of “Careless Whisper” in the style of Sinatra captures not only the characteristics of voice but also the style and genre instrumentation. What I found quite impressive is that if I have no prior knowledge of music release timelines and was subject to a blind test I don’t believe I would be able to identify this as generated content. Though the voice may lack some emotion and characteristics of older style recordings it doesn’t sound synthetic.

I decided to listen to another AI Sinatra cover, this time of Boulevard of Broken Dreams and had a similar experience in terms of voice quality and instrumentation. I think there are few that may explain this, notably Sinatra has a vast discography that can improve the accuracy of machine learning. While Sinatra has identifying vocal characteristics his voice is quite typical of a person of his age and demographic which means machine learning content can be supplemented with additional content from those with similar voice characteristics.

# data[].attributes.attachment_link.title
Careless Whisper - Frank Sinatra AI Cover

# data[].attributes.body
Anadol, R (2019) Machine Hallucination - NYC, Refik Anadol website, accessed 27 July 2024.
https://refikanadol.com/works/machine-hallucination-nyc/

Ellis, R and Simons, R (2005) The impacts of music and physiological indices of emotion while viewing films, RMIT canvas website, accessed 27 July 2024.
https://rmit.instructure.com/courses/134417/modules/items/5966246

Online Etymology Dictionary (2024) Surreal, etymonline website, accessed 25 July 2024.
https://www.etymonline.com/word/surreal#etymonline_v_30608

Online Etymology Dictionary (2024) Uncanny, etymonline website, accessed 25 July 2024.

https://www.etymonline.com/word/uncanny#etymonline_v_4467

Online Etymology Dictionary (2024) Visceral, etymonline website, accessed 25 July 2024.
https://www.etymonline.com/word/visceral#etymonline_v_7829

# data[].attributes.body
The week was spent examining Assignment 3 prompts and mediums required to create an affective multi sensory experience. I have narrowed down three prompts which feel appropriate to the emotional state I would like to evoke. These prompts are “uncanny”, “visceral” and “surreal”.

While enjoying the potential of all three, I felt that “uncanny” was limiting as defined by feeling strange in a context that is mildly frightening. While “visceral” relies on a deep emotional state with an etymology that relates to internal or inner parts of the body. “Surreal” similarly to “uncanny” relates to the strange but is not limited by a mildly frightening context, and evokes a more dream-like state. This provides the opportunity to create content that juxtaposes lighter and darker matter, while also creating deep emotional states which are not limited by context.

The mediums I would like to combine are multichannel sounds, projected video, haptic feedback and lighting. The combination of these mediums aims to create an experience that is accessible to all users regardless of audience needs. Someone with limited vision may engage with the experience alongside someone with limited hearing. This provides a unique medium with high levels of subjectivity. With high subjectivity the question then relates to appropriate narrative and whether literal story telling or an experimental sequence of images is more emotionally engaging. I’m reminded of a phrase in advertising/marketing: if you try to create for everyone, it’s for no one.

# data[].attributes.body
The music:notimpossible vest can allow for meaningful relationships with sound, vision and lighting. Typically the vest is used in context to music accessibility and provides a 1:1 relationship with the input sound source. This means that musical instruments tracks can be assigned to haptics at different locations on the body where audio frequency and rhythm is perceived as tactile feedback. The problem is that this relationship doesn’t capture the emotional quality of the music and can be compared to film subtitles. It will provide accessibility to the text but does not capture the affective human performance.

Through the use of MIDI information opposed to audio there is the opportunity for nuanced feedback by utilising virtual instruments and automation. This haptic feedback can build relationships with the affordances of sound, vision and lighting. In moments of low activated sound the haptic feedback can mimic the pulsation of lights or the movement of video content. I believe this will provide more meaningful engagement by changing the point of focus and maintaining interest over time. Touch Designer appears to be the most useful software to achieve as programs like Ableton or Stable diffusion can run natively as APIs (application programming interface) and function together in real time to prevent clock synchronisation issues.

# data[].attributes.body
I created a mood board in miro to begin collecting thoughts and inspiration for Assignment 3. I used the board to examine different prompts and mock up technical components. It was also an opportunity to reflect on artists/designer aesthetics and experiment with colour theory.

Miro board link - https://miro.com/app/board/uXjVKvai2lM=/?share_link_id=175559525128

# data[].attributes.body
Refik Anadol - Machine Hallucination

Refik Anadol’s work “Machine Hallucination” is a multisensory experience comprising 18 channel 4K video and 32 channel sound. The content aims to portray urban memories reimagined by artificial intelligence. This evokes thought on the relationship between human and machine outside of the typical context of a dystopian future.

# data[].attributes.body
Lighting can be used to manipulate the attention of the audience through the sense of vision. It can create the perception of meaning by interacting with objects that render the focus of the audience and invite subjective interpretation. Through the use of visual hierarchy lighting can be used to progress a narrative structure.

Colour is extremely subjective and often has strong cultural roots in regards to interpretation. The temperature of colour can be an affective mechanism to create meaning or display character archetypes. Vivid colours are best used sparingly and are most impactful when placed with an opposing colour in the field of view.

During class time was spent dissecting the differences between the prompts “visceral”, “brutal” and “overwhelming”. In relation to the human body “visceral” can be interpreted as internal while “overwhelming” external. From an etymology perspective “visceral” is related to viscera which describes internal organs of the body. While “overwhelming” etymology relates to turning something upside down, knock over or submerge. “Brutal” however resembles inhuman or animalistic traits whether superficial or as action.