# data[].attributes.body
This is a good subject and area to pursue for the summative and I think it is something that you can easily develop.  I think it raises many questions about human interaction and the use of foley. Essentially this has always been a synthetic way to record sound and is made somewhat obsolete by the ease in which you can download and manipulate sound effects from the likes of freesound.org and the BBC archive. In light of this how does this aid our creativity and how effective will it become.? Will this be developed into some form of Strong AI whereby the algorithm can produce a sound effect that is better suited than the human ear? I these are questions you could address and perhaps take further.

# data[].attributes.body
Could AI remove the need for human involvement in sound design?

# data[].attributes.body
Foley sound for films is usually done by someone called a foley artist which means someone or a group of people usually in a studio using different objects to make sounds for existing pieces of video to creare the soundscape for a film, tv show or even game. This has led to many iconic sounds throughout the years in films. Now foley sound can also be generated by AI ‘’deep synthetic sound network’’ is a process where the AI analysis and will recognise the action that is happening in the video and then produce a sound effect that will be appropriate for it.

# data[].attributes.attachment_link.readable_text
Turing Test

If you'll permit us to spoil a little bit of movie magic, many of the sound effects you hear in film and TV are actually recreated and edited in later by Foley artists. Now, researchers are attempting to create sound effect-generating artificial intelligence to see if they can do their jobs well enough to fool the general population.

In a recent study, a small cohort of participants fell for the trick: Most they believed that the AI-generated noises were real, IEEE Spectrum reports. Sometimes, they even chose the AI version over a video's original audio.

Sleight Of Hand

In the study, which was published in June in the paper IEEE Transactions on Multimedia, 41 of the 53 participants were fooled by the AI-generated sounds.

That sounds damning for the Foley artists who recreate sound effects in the studio, but for now, the algorithm's still far from perfect.

Sleight of Ear

Where the AI currently falters - and the challenge going forward - is figuring out how to match the audio up to the actual video it supposedly represents. In order to fool an audience, the sound effects need to not only sound real, but also, synch up with a video's action.

That's why the embedded videos - spoilers, they're all AI-generated - of rainfall or a crackling fire are convincing but this one of a running horse, where the clipping and clopping clearly don't line up with the steps the horse takes, is a clear fake. Needless to say: The jobs of foley artists and sound editors are secure - for now. But it's hard not to imagine a not-at-all distant reality where you might believe this horse is making these horse noises:

READ MORE: New AI Dupes Humans into Believing Synthesized Sound Effects Are Real [ IEEE Spectrum]
More on AI: Elon Musk: If You Don't think AI Could Outsmart You, You're an Idiot

# data[].attributes.attachment_link.title
AI-generated sound effects are now fooling human ears

# data[].attributes.attachment_link.description
If you'll permit us to spoil a little bit of movie magic, many of the sound effects you hear in film and TV are actually recreated and edited in later by Foley artists. Now, researchers are attempting to create sound effect-generating artificial intelligence to see if they can do their jobs well enough to fool the general population.

# data[].attributes.body
Could AI evolve to becoming fully sentient in games?

# data[].attributes.attachment_link.readable_text
Films are generally immersive experiences, made with the aim to impress their viewers with their engaging plotlines and dazzling special effects. While some sounds may be recorded at the time of filming, movies also rely on convincing sound effects - often made during post-production by someone known as a Foley artist - to fill in those all-important background noises like footsteps, rustling leaves or falling raindrops to create a sense of reality in a film. Not surprisingly, creating and integrating such sound effects is a time-consuming and costly part of any film budget.

Now, new work from a University of Texas at San Antonio research team shows that the Foley process can be automated - using artificial intelligence that can analyze motion in a given video, and then generate its own matching artificial sound effects.

A 'Deep Sound Synthesis Network'

Dubbed AutoFoley, the team's system uses deep learning AI to create what they call a "deep sound synthesis network," which can analyze, categorize and recognize what kind of action is happening in a video frame, and then produce the appropriate sound effect to enhance video that may or may not already have some sound.

"Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as inter-frame relations in fast-moving video clips," explained the researchers in their paper, which was recently published in IEEE Transactions on Multimedia.

To achieve this, the AutoFoley system first identifies the actions in a video clip, then selects a suitable sound from a customized database that matches the action. AutoFoley then attempts to ensure that the sound matches the timing of the movements in each video frame. The first part of the system analyzes the association of movement and timing in video-frame images by extracting features like color, using a multiscale recurrent neural network (RNN) combined with a convolutional neural network (CNN). However, for faster-moving actions in video clips where there may be missing visual information between consecutive frames, an interpolation technique using CNNs and a temporal relational network (TRN) is utilized so that the system can preemptively "fill in" any missing gaps and link them smoothly, so that it can still accurately time the actions along with the predicted sound.

Next, AutoFoley synthesizes a sound to correspond with the action identified from the video in the previous steps. To aid in its training, the team curated their own database of common sound effects, categorized in different "sound classes" that included things like rainfall, crackling fire, galloping horses, breaking objects, and typing.

"Our interest is to enable our Foley generation network to be trained with the exact natural sound produced in a particular movie scene," said the researchers. "To do so, we need to train the system explicitly with the specific categories of audio-visual scenes that are closely related to manually generated Foley tracks for silent movie clips."

Some of the sounds in the database were created by the team, while others were culled from online videos. All told, the researchers' Automatic Foley Dataset (AFD) contains sounds from a total of 1000 videos from 12 different classes, with each video duration averaging about five seconds each. As seen and heard below, the resulting AI-synthesized audio as applied to sample video clips does sound pretty realistic.

To test how convincing the results were, the research team presented the finalized videos with the AI-generated sound effects to 57 volunteers. Surprisingly, 73% of participants believed that the synthesized AutoFoley sounds were actually the original soundtracks - a significant improvement over comparable methods that also generate sound from visual inputs.

To improve their model, the researchers now plan to expand their training dataset so to include a wider variety of realistic-sounding audio clips, in addition to further optimizing time synchronization. The team is aiming to also boost the system's computational efficiency so that it will be capable of processing and generating sound effects in real-time. With AI now able to generate rather convincing pieces of music, literature, informational texts, and even faked videos of politicians or famous works of art that are almost indistinguishable from the real thing, it was only a matter of time before machines fooled humans with their artificially created sounds as well.

Read more in the team's paper.

Images: Eduardo Santos Gonzaga via Pixabay; University of Texas at San Antonio

# data[].attributes.attachment_link.title
These AI-Synthesized Sound Effects Are Realistic Enough to Fool Humans

# data[].attributes.attachment_link.description
Dubbed AutoFoley, this system uses deep learning AI to create what they call a "deep sound synthesis network," which can analyze, categorize and recognize what kind of action is happening in a video frame, and then produce the appropriate sound effect to enhance video that may or may not already have some sound.

# data[].attributes.attachment_link.readable_text
One mistake and it's all over."

Video games carry an aura of experimentation that the consequences of life cannot provide. If you fail a mission, you have the choice to restart and try again like nothing ever happened. If you want to complete an objective as efficiently as possible, you can replay that scenario over and over again. If you have the urge to pillage an entire village and murder all the residents in a fit of God-like rage but not endure the moral repercussions of your actions? Just quick load your save file of Skyrim and we won't talk about it. The choice to erase a played scenario and re-do it in the fashion a player wants to is one of the more alluring aspects of video games, and one of the reasons players can feel comfortable doing terrible things without second guessing the actions: it's just a game, right? No one gets hurt in the long run.

Visual novels are a connoisseur of this allure of choice. Most novels will provide multiple endings, expected to be explored, reloaded, and experienced. These characters face a multitude of emotions: love, hate, jealousy, rage, disappointment, relief; all to be subsequently wiped for the player to play through the story once again. These range of emotions the characters inquire are deleted after each load, wiped clean to encounter once again for the very first time. But what if these characters knew of your actions? Knew how many times you reneged on promises made to see the other side of the coin? When the characters realized how selfish your actions really were?

YOU and ME and HER revolves around the high school lives of Susuki Shinichi, the main protagonist, his childhood friend, Sone Miyuki, and their mysterious classmate, Mukou Aoi. Aoi is often found on the school rooftop clutching a clam-shell phone trying to phone God at her self-proclaimed "center of the universe". This stance has caused her to unceremoniously be the loner of the class. Shinichi sees her ways on the roof as he frequents there for class clean-up duties, and offers to extend a hand in friendship to a girl who looks in need of help, but is unsure how to ask.

Miyuki, on the other hand, boasts herself as the queen bee of the classroom, a perfect individual who can befriend all with her gentle demeanor, but keeps all who covet her past arm's distance. She glides into each facet of life with determination and grace, a life built to be above others, to be the best she can be. Miyuki is seen as an unattainable goal to Shinichi, one that is nauseatingly pounded into submission throughout the first half of the game, and Shinichi actively keeps his distance from her, avoiding contact outside of school in order to make sure he stays romantically unavailable, lest someone of her caliber is seen with a loser as self-stated by himself.

The first half of Y&M&H carries on as your run-of-the-mill visual novel standard, though the inclusion of full Japanese audio is a welcoming touch. A near-insufferable protagonist downplaying obvious advances from a tsundere secondary, and your weirdly kinky friend, enter Akebono Yuutarou, who plays hit-or-miss comic relief as filler in between plot lines. The writing feels very one-tone and the only real stand out is Aoi, who during your first route feels oddly "4 th wall" about her knowledge in regards to visual novels and the tropes around them. While Y&M&H does provide multiple endings to experience, the way the choices are presented feel catered to reach a specific ending for one of the two main female characters.

When you start digging into the other character's route, the game almost feels like it's pushing you away from it, to go back to what has been played and accomplished before, before you go against the enjoyable feeling of being wanted and seeing two souls finally intertwine. Talks of "breaking routes" and "only being the friend in this game" fill the route, subtly second guessing player actions, little breaks in the 4 th wall with reminders to save your progress: actions that during any other novel may seem suspect.

When Y&M&H initially boots up, the game states that it contains mature themes including violence and language, and if you're familiar with nitroplus' Saya no Uta, you know what is capable. After a very tame first act, the end of the second main route grips the proverbial E-brake and hits a loud, abrupt 180, putting the characters, scenes, and the overall state of the game on a wild turn down to the unknown.

While I won't dive too far into what lies after because it really needs to be experienced firsthand, the amount of detail and craziness is an absolute godsend to the mediocrity of what laid before it. The writing is razor sharp, and the atmosphere is substantially tense. I commend nitroplus' added additions to the genre's normally static exterior, but the tasks at hand for the protagonist can involve a lot of trial and error and are not presented very clearly, which will have you fumbling around to figure out your next move. One of the coolest pieces of Y&M&H will take you by surprise, since where your experiences with a visual novel are held specific to you, Y&M&H's choices and details of the story itself are also specific to you. Pay attention to the lives you play with.

Soon you reach the end of their journey: one filled with love, tears, jealousy, and many, many more emotions. Despite the world once seeming so one-dimensional, Y&M&H tosses and turns it in ways I haven't seen in recent years. Never has a title made me question the morality in which players treat the characters whom set the stage since Spec Ops: The Line. Y&M&H pleads with you to stand firm with your choices, to understand the promises that were made throughout your playthrough, and to hold true the emotions felt.

For every coin, there are two sides: one must be called, and the choice must be kept.

So, which will you choose?
Reviewed on Steam.

# data[].attributes.attachment_link.title
YOU and ME and HER: A Love Story - Review

# data[].attributes.attachment_link.description
Video games carry an aura of experimentation that the consequences of life cannot provide. If you fail a mission, you have the choice to restart and try again like nothing ever happened. If you want to complete an objective as efficiently as possible, you can replay that scenario over and over again.

# data[].attributes.attachment_link.readable_text
During the fall season, I spend a lot of time searching for fresh scares. I'm immune to haunted houses and horror movies, because neither one force me out of my comfort zone. I know that movies will progress without my input; even if I'm shrinking back a bit in my seat, it'll keep moving, and I can live out the full experience. As for haunted houses, with the exception of a few "intensified" experiences, ghouls can't touch me, and the scares are predictable. I can round a corner and, well, gosh! It's ... a dude in a costume and usually some pretty cool makeup.

I often feel the same sense of ease when playing horror video games. Gore doesn't irk me, and monsters are so yesterday. I've already seen my fair share of Resident Evil 7Let's Plays without feeling much in my gut. Like I do with haunted houses and horror films, I always know that none of the scares are real.

But a romantic visual novel, of all things, petrified me recently - just by completely subverting my expectations. It exploited my confidence in the visual novel genre and turned that into a twisted game of its own. It taught my nightmares new tricks.

(Some light spoilers follow below; here's your spoiler warning.)

Although it looks like a dating sim, Doki Doki Literature Club is a free-to-play, psychological horror game, produced by indie studio Team Salvato. Since its release, the game has earned a reputation as an innovative scare. It's a slow burn that begins with you and group of cute girls who must prove that their literature club is worth becoming an official school organization. Of course, your protagonist is hoping to forge new bonds with some of them along the way. The game encourages you to pick a girl to write a poem for, and depending on your choices, you may draw closer to the club's charming, sweet members.

Eventually, true to the game's advertised content warnings (which you should take seriously, by the way), Doki Doki Literature Club leads you down a dark path, leading to the shocking and emotional death of one character. But in the same moment that it rips your heart open, the game instantly takes a much more unusual twist.

After this initial playthrough ends and the main menu restarts, the dead character's image is pixelated and warped, as if her death affected the game client itself. The save files become inaccessible, forcing the player into a new game. If and when the player moves ahead to begin this new file, the game seems to react at any hint of this former character, and the client loudly glitches and morphs until it seems satisfied with its outcome.

Soon into the next run-through, the client repeatedly takes control of itself, speeding through text and tacking on unusual images to create grotesque jump scares. But the reality of a sentient game client becomes nightmare fuel on its own.

To make matters worse, in the moment of the character's death, keen eyes will notice that the game directs the player outside of the client by naming a specific game file, alluding to an outside force changing the game's universe. It becomes a terrifying mystery, weaving in and out between the in-game plot and the game files' cryptic implications: What happened to Doki Doki Literature Club?

As I crawled into this "second run," I wasn't just horrified; I was mentally trapped in the game's world and its antics. But I still wanted to dive back in, and I spent time with myself to understand what I had to overcome in order to continue the game. In the process, I realized how Doki Doki Literature Club utilizes an underrated aspect of the horror experience: control, or the lack thereof.

On a basic level, the fear of fear (the anticipation, in a word) is what makes horror as a genre so difficult for many people. In interactive media, you're especially aware of how you're prompting the horrors by progressing through the work. Worse, a game can mask the world around you in highly efficient ways, bringing down your guard before a good scare. You can start crawling into a space before getting dragged out, or you can open a door that seemed safe before and encounter a new monster. The point of many games becomes, then, that fear often makes you lose control, and just as often, loss of control makes you lose the game.

The "action-adventure horror house" is an effective and proven genre, with tactics such as jump-scares, claustrophobic encounters and high-adrenaline chases that work despite their tired use. Developers have learned how to capture the traditional horror atmosphere and drive tension in such an efficient way that even the aforementioned classic methods are amplified in these settings.

Doki Doki Literature Club uses many of these same familiar concepts, but not in the way you'd expect, and that's mostly thanks to its breakdown of the fourth wall. Known as "metafiction," this genre of work is explicitly designed to restructure the dynamic between the work and the person interacting with it. In this case, it almost entirely removes control from the player.

It's worth noting that metafiction already has huge potential in horror games, because of their inherent nature of directly interacting with the player. This isn't supposed to happen in a video game, right? The window isn't supposed to exit, that file wasn't there before, is that character addressing me? It creeps into and mocks our imagination and ideas of what that game is supposed to be, and that quickly gets under players' skins. Much like horror, a primary theme is control.

As a work of horrific metafiction, Doki Doki Literature Club quite literally overwrites that control, taking the tropes of metafiction, horror and visual novel genres and amping them up a few notches. It builds moments based on what is assumed to be your understanding of the genre's tropes and the fundamentals of interactive media. There are ominous text files appearing out of nowhere; a "bonus poem" unlocked by the player in this new run-through can either be a sweet "I love you" or a dark regret of our former character. "Scenes" I expected to be cute are overlaid with static and blacked-out eyes, and characters sometimes zoom in with uncomfortable closeness. The game knew that I knew this wasn't normal, and it used that as a tool to make me uneasy.

More importantly, it kills a character, then explicitly prevents you from doing anything about it. I expected to return to your save file to rescue her or maybe date another girl, but I was not allowed to. She was gone. When the game restarts, it constantly mocked my decisions and regrets. I wasn't given any idea of how fix the universe, short of a hard reset on the game altogether - which would only bring me back to square one.

In sum, this is what's horrifying about Doki Doki Literature Club: You, as a player and a character within the game's world, are helpless.

As I dug into the game's newly-warped world, extra poems and the suddenly-appearing game files, my grasp of the game slipped further. The quest to discovering the game's secrets becomes a paradox, because as much as I wanted to know more, the process untangles the universe further and further. And, when that realization sunk in, I became less sure what the goal was. All I could do was move forward and find some light at the end of the tunnel. (Well, hopefully.)

Team Salvato has unearthed a new way of terrifying us by ripping apart our expectations of how interactive media should work. Doki Doki Literature Club is a game that played me, and somehow, I'm content to let it take control.

# data[].attributes.attachment_link.title
Doki Doki Literature Club is an uncontrollably horrific visual novel

# data[].attributes.attachment_link.description
During the fall season, I spend a lot of time searching for fresh scares. I'm immune to haunted houses and horror movies, because neither one force me out of my comfort zone. I know that movies will progress without my input; even if I'm shrinking back a bit in my seat, it'll keep moving, and I can live out the full experience.

# data[].attributes.attachment_link.readable_text
Are you a human?

We apologize for the confusion, but we can't quite tell if you're a person or a script.

Please don't take this personally. Bots and scripts can be very much lifelike these days!

To help us better protect your account security, please check the CAPTCHA box below.

detecting...

If you're interested in accessing Newegg API service, please submit a request.

We would love to hear your opinion. Let us know your feedback.

# data[].attributes.attachment_link.description
We apologize for the confusion, but we can't quite tell if you're a person or a script. Please don't take this personally. Bots and scripts can be very much lifelike these days! To help us better protect your account security, please check the CAPTCHA box below. detecting...

# data[].attributes.attachment_link.readable_text
copyright by spectrum.ieee.org

Imagine you are watching a scary movie: the heroine creeps through a dark basement, on high alert. Suspenseful music plays in the background, while some unseen, sinister creature creeps in the shadows... and then-BANG! It knocks over an object.

Such scenes would hardly be as captivating and scary without the intense, but perfectly timed sound effects, like the loud bang that sent our main character wheeling around in fear. Usually these sound effects are recorded by Foley artists in the studio, who produce the sounds using oodles of objects at their disposal. Recording the sound of glass breaking may involve actually breaking glass repeatedly, for example, until the sound closely matches the video clip.

In a more recent plot twist, researchers have created an automated program that analyzes the movement in video frames and creates its own artificial sound effects to match the scene. In a survey, the majority of people polled indicated that they believed the fake sound effects were real. The model, AutoFoley, is described in a study published June 25 in IEEE Transactions on Multimedia .

"Adding sound effects in post-production using the art of Foley has been an intricate part of movie and television soundtracks since the 1930s," explains Jeff Prevost, a professor at the University of Texas at San Antonio who co-created AutoFoley. "Movies would seem hollow and distant without the controlled layer of a realistic Foley soundtrack. However, the process of Foley sound synthesis therefore adds significant time and cost to the creation of a motion picture."

Intrigued by the thought of an automated Foley system, Prevost and his PhD student, Sanchita Ghose, set about creating a multi-layered machine learning program. They created two different models that could be used in the first step, which involves identifying the actions in a video and determining the appropriate sound.

The first machine learning model extracts image features (e.g., color and motion) from the frames of fast-moving action clips to determine an appropriate sound effect.

The second model analyzes the temporal relationship of an object in separate frames. By using relational reasoning to compare different frames across time, the second model can anticipate what action is taking place in the video.

In a final step, sound is synthesized to match the activity or motion predicted by one of the models. Prevost and Ghose used AutoFoley to create sound for 1,000 short movie clips capturing a number of common actions, like falling raining, a galloping horse, and a ticking clock. [...]

Read more: spectrum.ieee.org

# data[].attributes.attachment_link.title
AI-Generated Sound Effects Brought to a Brand New Level

# data[].attributes.attachment_link.description
copyright by spectrum.ieee.org Imagine you are watching a scary movie: the heroine creeps through a dark basement, on high alert. Suspenseful music plays in the background, while some unseen, sinister creature creeps in the shadows... and then-BANG! It knocks over an object.