It provides the ability to turn a quick sketch into a skybox,

generating complete 3d environments in order to visualise scenes and lighting quickly.

Developments in camera rigs aim to match the human visual system:

• Camera separation (IA) = Human eye separation (IO)

• Camera image magnification = human magnification

• Distance to imaged subject = distance to viewed subject

4d exhibition:

https://www.gaudiexperiencia.com/en/4d-and-exhibition/

How it works:

https://stereoscopy.blog/2023/06/21/the-art-of-stereoscopy-andrew-brooks/

Underwater Stereoscopic Technology used in Avatar 2:

https://ymcinema.com/2022/10/24/the-underwater-cinematography-behind-avatar-2-3d-beam-splitter-and-nikonos-lenses/

In future Gen-2 aims to generate video from text prompts.

Can generate a 3d object to replace and accurately track a subject in a video. It can also be used in order to generate a background.

Once the AI has produced the video, it gives 3d data for the model just created and this allows for fine tuning in a different 3d software e.g. Mya and Blender

Can be useful for creatives requiring a quick mock-up to be able to quickly visualise a scene or 3d space.

Mixed Reality (MR) - A blend of AR and VR where you use a headset or glasses to be shown AR content within your environment.

Virtual Reality (VR) - This can be computer generated or filmed, but is accessed through a VR headset or immersive environment that separates the viewer from the space they are in.

There was also a talk given by industry professional, Jack Østergaard-Churchill. He detailed his lengthy experience with 360 video, XR technologies and AI which was useful in discerning different AI generative softwares and their uses.

Her presentation gave a useful insight into emerging technologies which are working closely with AI.

We explored its use as a tool to create 3D environments. This can be used in conjunction with AR and AI to create backgrounds for movie sets and immersive experiences for exhibitions and education.

“Virtual Production combines virtual and augmented reality with CGI and game-engine technologies to enable production crews to see their scenes unfold as they are composed and captured on set.” - Moving Picture Company (MPC), from The Virtual Production Field Guide presented by Epic Games

VP brings post-production talent into pre-production processes, opening up new avenues for post production skills. Post-production knowledge holds a lot of value for optimizing new workflows since it is useful to know how certain effects will look once edited.

Often these technologies can be paired with AI and AR to further enhance the user experience. This opens up many new possibilities to re-use archive in new ways and tell stories in places that no longer exist.

Many of these immersive technologies have been used in public spaces in conjunction with billboards, in order to promote tourism or advertise products.

3D Cat promoting tourism in Shibuya, Japan:

https://www.timeout.com/tokyo/things-to-do/tokyo-now-has-three-3d-billboards-with-cute-realistic-animation

AR screen spaces near Tottenham Court Road:

https://www.outernet.com

R&D scanner speed = 1.6FPS @ 16K

Previous/other scanners = 0.4FPS @ 4K

• Writers rooms to format text

• Concept stages, enabling quick mock ups of lighting/costume/set

Production:

• Generating actors in a virtual space, utilising AR and large screens to create an immersive 3d environment

Post-production:

• Colourisation of archive footage

• Restoration of footage that has been damaged or is missing content

• Upscaling and format renewal

• Audio editing (sync, ADR)

Pros:

• If utilized correctly we can harness this technology to work in conjunction with industry standard software, streamlining processes that require less human involvement, its output can then be evaluated with human context and QC’d.

• AI can also be used to mitigate ‘budget vs schedule vs expectations’ issues by providing predictions for the length of time required for post production processes.

Cons:

• Consent is required for using AI to replicate actors image and likeness

• Regurgitating and adapting content without original IP owner’s consent can lead to data protection/copyright issues

• If used to replace too much of the creative process, jobs are put at risk and the value of creative work is diminished

  
  

New forms of workflow required for 360 video:

• AI involvement in order to make the final output more seamless and interactive.

• Data management is essential to collect this much data from the different lenses and make sure it is in the correct order

• Editor must perform an initial "rough stitch" to piece together footage from the lenses

• Final work includes smaller changes like removing the rig to hold the cameras/mounts/drone

Technology used:

Meta camera is one of the leading companies in the 360 video industry, capable of producing 12k footage and popularly used for sporting events and large scale VR productions.

https://meta-camera.io

Mistika VR is a well known program used to edit and stitch footage together to create a coherent 360 video.

https://www.sgo.es/mistika-vr/

Las Vegas Sphere immersive experience:

https://pacificdomes.com/msg-sphere-las-vegas/

Art gallery using 360 Video for virtual tours:

https://theninebritishart.co.uk/exhibitions/jeremy-gardiner-south-by-southwest-2020/

A company that generates virtual property tours from 360 video by using AI:

https://help.ricoh360.com/hc/en-us/articles/360061734594-AI-Video-Maker#:~:text=AI%20technology%20automates%20the%20entire%20process&text=The%20AI%20is%20able%20to,and%20automatically%20generates%20a%20video.

"The AI is able to identify points of interest that customers like from millions of 360˚ property images and automatically generates a video." - Ricoh 360 Tours

Video describing current 360 post production workflow:

https://www.screenskills.com/starting-your-career/first-day-in-post/

AI can also be used to map audio to appear as if it is generated from certain parts of a space e.g. creating a immersive surround-sound experience from a simple stereo recording.

Study into generating spacial audio using neural networks:

http://www.svcl.ucsd.edu/~morgado/spatialaudiogen/spatialaudiogen.pdf

How 360 audio works:

https://endabates.wordpress.com/2016/03/29/360-audio-in-practice/

Types of microphones for 360 audio:

https://www.vrtonung.de/en/360-microphone-overview/

How AI is revolutionising spacial audio:

https://film-sound.berlin/blog/2023/06/09/adr-in-film-with-ai-revolutionizing-audio-post-production/

• Stereoscopic vision limited to 60°cone

• Narrower 30° cone defines zone of focused detail

• Wider 120° cone defines edges

Common AR fields of view:

• Multiplex cinema FOV is about 54°

• IMAX cinema average FOV is 70°

• Samsung Gear VR horizontal FOV is 96°

• Oculus Rift VR headset is 100°

I have chosen to focus on AI and its impact on the industry as many of the topics we have covered in the last weeks have had relevance to the emergence of AI and how its rapid advancement will have a large impact on the creative landscape.

We also had a talk by James Uren, the technical director at Mo-Sys. His company focuses in cutting-edge virtual production.

Virtual Production Uses:

• Creates sets and environments quickly

• Reduces the cost of traditional sets and props

• Enables filmmakers to make instant adjustments

• Allows for a more natural and controlled lighting setup

How does it work?

The main process behind virtual production is the integration of different technologies.

These include: CGI, motion capture, facial recognition, laser scanning, robotics, AR and AI. Most commonly, this is achieved through programs such as Unreal Engine and ILM’s Stagecraft.

Definition of VP:

https://www.studiobinder.com/blog/what-is-virtual-production-definition/#:~:text=Virtual%20production%20is%20a%20method,effects%20on%20a%20virtual%20set.

https://www.autodesk.co.uk/products/flame/overview?term=1-YEAR&tab=subscription

Nuke:

https://www.foundry.com/products/nuke-family/nuke

1899 was released in 2022 on Netflix and follows a group of European emigrants travelling from Southampton, UK on a steamship named Kerberos to start new lives in New York City, US. The series soon revealed itself to have more sci-fi elements embedded in the narrative.

This production enabled advances in:

• Remote working

• Carbon footprint

• Atmospherics

• Crowd scenes

• Weather

• Use of VR in location scouting

• Photogrammetry

Considerations:

• Lighting

• Atmospherics (rain/smoke/dust)

• Happy accidents

• Actor and Crew experience

• Actors not being present

• The rise of experiential cinema (Gravity, Revenant, Fall)

LED screens have become more and more advanced and able to be constructed in large wall mainly used in festivals and gigs.

LED screen technology such as pixel pitch becomes more detailed and affordable mean that having 'Final Pixel' on screen will allow productions to have much more versatility in the set up of their shots.

The Silent Killer of Virtual Production:

https://www.youtube.com/watch?v=M-vtf2TpopY&t=211s

The movements used to be very simple and limited to dollying and jib or crane work but with cutting edge advancements in robotics mean that there is much more versatility in movement.

This website shows the advancements robotics have made on camera work:

https://medium.com/@adelstein/how-robotics-is-revolutionizing-film-production-eeb5e79cccd5#:~:text=By%20integrating%20robotics%20with%20Virtual,chances%20of%20post-production%20surprises.

Computer controlled camera movement allows adjustments for:

• Multiple passes

• Crowd

• Scale

• Atmospheres

• Speed

This ability to potentially duplicate and therefore undermine the value of artistic expression has left many concerns for the future of original online art.

I applied some of the concepts mentioned in these references to highlight the importance of ethical considerations when using AI for profit and creative work:

How AI is affecting copyright issues:

https://thenewstack.io/the-power-and-ethical-dilemma-of-ai-image-generation-models/

An article from Harvard detailing the potential issues brought by AI's rapid advancement "Great promise but potential for peril":

https://news.harvard.edu/gazette/story/2020/10/ethical-concerns-mount-as-ai-takes-bigger-decision-making-role/

https://www.theverge.com/2024/4/8/24124070/spotify-ai-playlist-music-beta-launch-premium-uk-australia.

Netflix machine learning:

https://research.netflix.com/research-area/machine-learning

After trying various different prompts I settled on "AI in creative industry" which yielded the most accurate result for my topic.