# data[].attributes.headline
Association between gait features assessed by artificial intelligent system and cognitive function decline in patients with silent cerebrovascular disease: study protocol of a multicenter prospective cohort study (ACCURATE-2)

# data[].attributes.subject
Association between gait features assessed by artificial intelligent system and cognitive function decline in patients with silent cerebrovascular disease: study protocol of a multicenter prospective cohort study (ACCURATE-2)

# data[].attributes.attachment_link.readable_text
Study design

The ACCURATE-2 is a multicenter prospective cohort study that follows the changes in scores of neuropsychological test in patients with SCD to compare the differences between normal gait group and abnormal gait group, and analyzes the correlation between gait spatiotemporal parameters and the changes of neuropsychological test scores.

Setting

The trial will be conducted at the neurology departments of general hospitals. Fourteen secondary or tertiary hospitals located in the Shanghai and Guizhou areas will be included in this study. All staff members of the trial trained.

Participants

In this study, patients with SCD aged 60-85 years will be consecutively recruited and agree to participate for at least 12 months. Neurologists will identify eligible patients by browsing outpatient cranial MRI images. Patients will be informed about the research contents, and those willing to participate will be given a written informed consent form to sign. After recruitment, suitable patients will be selected according to the inclusion and exclusion criteria.

Inclusion criteria

(2)
Diagnosed with SCD according to the 2016 statement issued by the American Heart Association and the American Stroke Association [ 17]:

1. (A)
Absence of clinically recognized acute stroke symptoms.

2. (B)
Cranial MRI taken in the past 1 year having shown one of the following (according to STRIVE [ 18]):

1. (a)
Lacunes of presumed vascular origin: defined as a subcortical, round, or ovoid fluid-filled cavity with a diameter of 3-15 mm showing a low central signal and irregular marginal high signal on T2 fluid-attenuated inversion recovery (T2-FLAIR). The central signal is similar to that of cerebrospinal fluid, and the distribution is consistent with the blood supply area of the perforating artery. Fazekas scores should be ≥2 points.

2. (b)
WMH of presumed vascular origin: defined as hyperintensity on T2-FLAIR in the white matter area (periventricular or subcortical), the signal of which differs from cerebrospinal fluid. Fazekas scores should be ≥2 points.

3. (c)
Microbleed: defined as a small round area devoid of signal with associated blooming on susceptibility weighted imaging or T2-weighted imaging generally 2-10 mm in diameter. Microbleed lesion number should be ≥5.

1. (3)
Conscious and able to finish all neuropsychological test.

2. (4)
Ability to stand and walk independently without waking aids, and complete the intelligent gait assessment.

3. (5)
Sign the informed consent form.

Exclusion criteria

1. (1)
Definitively diagnosed as demyelination disease, leukodystrophy, space-occupying lesions, or autoimmune encephalitis, etc.

2. (2)
Definitively diagnosed as Parkinson's disease, normal pressure hydrocephalus, inner ear disease, subacute combined degeneration, peripheral neuropathy, osteoarthritis, or lumbar disease, etc.

3. (3)
Definitively diagnosed as Alzheimer's disease, frontotemporal dementia, or Lewy body dementia, etc.

4. (4)
Previous history of severe neurological diseases such as cerebral trauma, epilepsy, alcoholic encephalopathy, or myeleterosis, etc.

5. (5)
Severe cardiovascular complications and inability to tolerate the assessment.

6. (6)
Severe visual impairment, severe hearing impairment, aphasia, severe cognitive dysfunction, severe gait, balance disturbance and inability to complete the cognition scale and intelligent gait assessments.

7. (7)
Refusal to participate in the study.

8. (8)
Other anomalies not specified that the researchers considered inappropriate for inclusion in this study.

Assessments

Clinical data such as demographic characteristics, history of present illness, comorbidities, physical examination findings of the nervous system, previous laboratory examination results and imaging findings, and details of falls events in the past year will be collected. The conversations between doctors and patients during data collection will be recorded by voice recorders, and the audio files will only be used for quality control.

Intelligent gait analysis (SAIL system)

The vision-based artificial intelligent gait analyzer we proposed (named SAIL system) includes a data acquisition client and a data analysis server. The data acquisition client consists of a laptop and an RGB-depth camera. The computer application we designed can record the entire TUG test by calling the RGB-depth camera, and upload the video files subsequently to the data analysis server via the network. The data analysis server will detect gait spatiotemporal parameters and walking pattern classification for each patient through the built-in intelligent algorithm and transmits the analysis report back to the data acquisition client through the network.

The TUG videos will be recorded in a bright, clean, and tidy indoor square area (at least 3.8 m × 3.8 m) without any obstruction. A armless chair with standard seat height will be placed on one side, and a visible colored line on the floor will mark a distance of 3 m from the seat. The RGB-depth camera on a 1.2-m-high tripod will be placed 3.8 m away in a vertical line from the midpoint of the footpath. The patients will be required to wear well-fitting trousers and a pair of everyday flat shoes, take off the long coat that reaches the hips, put down the items in their hands, and remove their backpacks before the TUG test. Each participant will be asked to sit on the chair first, stand upon hearing the doctor's instructions, walk straight to the 3-m mark, turn around, return to the chair, and sit down finally. All participants will complete the TUG test twice at their usual comfortable walking pace, and at least a five-minute break will be given between the tests. Participants can practice 1-2 times to familiarize with the walking test requirements.

On the data analysis server, the pose estimation algorithm will be used to identify the human key body surface points in each frame of the TUG video (such as head, shoulder, elbow, wrist, hip, knee, and ankle), and process the data by filtering and double-threshold signal-detection methods to calculate the six gait spatiotemporal parameters, including step length, gait speed, step frequency, step width, standing up time, and turning back time. Step length is defined as the average distance from the landing of one foot to the subsequent landing of the other foot, and a shorter step length will indicate worse gait performance. Gait speed is defined as the average velocity of walking during the straight parts of the TUG test, and a slower gait speed will indicate worse gait performance. Step frequency is defined as the average steps taken per second, and a slower step frequency will indicate worse gait performance. Step width is defined as the depth distance between two heels, and a wider step width will indicate a worse gait balance. Standing up time is defined as the time interval from the beginning of the video to the timepoint when the height of two shoulders reach the highest, which includes reaction times and stand-up action times, and a longer time will indicate worse gait performance. Turning back time is defined as the time between the start and end of the shoulder twisting, and a longer time will indicate worse gait performance. The native Bayes classifier that has learned the expert gold standard before will be applied to classify gait performance as normal or abnormal.

Neuropsychological evaluation

Participants will be evaluated on the following neuropsychological tests by trained doctors:

1. (1)
Mini-Mental State Examination (MMSE), Chinese version translated by Mingyuan Zhang: evaluates time and place orientation, attention and calculation, recall, language, and copying. Scores range from 0 to 30 points, with a lower score indicating worse cognitive function.

2. (2)
Montreal Cognitive Assessment (MoCA), Beijing version: evaluates visual-spatial ability, executive function, naming, memory, attention, language, abstraction, and orientation. Scores range from 0 to 30 points, with a lower score indicating worse cognitive function.

3. (3)
The Stroop Color word test (CWT), Huashan version: evaluates semantic activation, dominant response inhibition, attention, working memory, and information processing speed. The time spent in each card and the number of errors will be recorded. A longer time cost or more errors indicate worse cognitive function.

4. (4)
The digit span test (DST), a subtest of the Chinese version of the Wechsler Adult Intelligence Scale-Revised: evaluates immediate memory and attention. A shorter string of repeated numbers indicates worse cognitive function.

Grouping and follow-up plan design

During the baseline period, all participants will be evaluated by intelligent gait analyzer and neuropsychological tests. According to the intelligent gait report, each patient will be classified as normal gait group or abnormal gait group. All participants will be naturally observed for at least 1 year, and all their medical behaviors will be recorded. At six months after enrollment, all patients will be interviewed by telephone to collect falls events, cerebrovascular events, and the changes in medical behaviors during the follow-up period. The participants will be invited to undergo intelligent gait assessment and neuropsychological evaluation in the hospital at 12 months after enrollment, and falls events, cerebrovascular events, the changes in medical behaviors during the follow-up period will be collected. The details are presented in Fig. 1 and Table 1. All data will be encrypted and stored on the local server in Zhongshan hospital.

Flow diagram of the ACCURATE-2 study

Table 1 Overview of the study assessments

Outcomes

The primary outcome of this study will be the changes in the MMSE score in patients with SCD after 1-year follow-up.

The secondary outcomes of this study will be the changes in intelligent gait parameters (step length, gait speed, step frequency, step width, standing up time, and turning back time) and the changes in scores on other neuropsychological tests (MoCA, CWT, and DST) after 1-year follow-up, the prevalence of abnormal gait and cognition decline in patients with SCD, and the incidence of cerebrovascular events and falls in patients with SCD.

Sample size

Referring to previous studies of cognitive decline among patients with SCD [ 19, 20, 21, 22, 23], we assume that the MMSE score will decrease by 0.5 ± 0.5 points in the normal gait group and 0.7 ± 1.2 points in the abnormal gait group within 1 year; the ratio of abnormal gait to normal gait among outpatients with SCD is 1:2. Test efficiency 1-beta is set at 90%, while test level alpha is set at 0.05. An estimated 20% shedding rate, sample size of 1050 patients with a normal gait and 550 patients with an abnormal gait will be required. Therefore, a total of 1600 patients will be included.

Statistical analysis

All measurement data will be tested using the Shapiro-Wilk normality test. Normally distributed continuous measurement data will be expressed as mean and standard deviation, while non-normally distributed measurement data will be expressed as median and interquartile range. The t-test, Wilcoxon signed-ranks test, repeated measures t-test, or linear generalized model will be used to compare measurement data. The enumeration data will be expressed as a composition ratio or rate. The chi-square test or Fisher exact method will be used to compare enumeration data. Pearson linear correlation analysis or Spearman correlation analysis will be performed for correlation analysis. Survival will be estimated using a Kaplan-Meier survival curve. The propensity score matching method will be used to minimize the influence of other confounders. Subgroup analyses will include region and hospital levels. All cases with missing values will be deleted. A significant difference will be identified P  < 0.05.

# data[].attributes.attachment_link.title
Association between gait features assessed by artificial intelligent system and cognitive function decline in patients with silent cerebrovascular disease: study protocol of a multicenter prospective cohort study (ACCURATE-2) - BMC Neurology

# data[].attributes.attachment_link.description
Background Gait disturbances may appear prior to cognitive dysfunction in the early stage of silent cerebrovascular disease (SCD). Subtle changes in gait characteristics may provide an early warning of later cognitive decline. Our team has proposed a vision-based artificial intelligent gait analyzer for the rapid detection of spatiotemporal parameters and walking pattern based on videos of the Timed Up and Go (TUG) test.

# data[].attributes.headline
A systematic review of gait analysis methods based on inertial sensors and adaptive algorithms

# data[].attributes.subject
A systematic review of gait analysis methods based on inertial sensors and adaptive algorithms

# data[].attributes.attachment_link.readable_text
Abstract

The conventional methods to assess human gait are either expensive or complex to be applied regularly in clinical practice. To reduce the cost and simplify the evaluation, inertial sensors and adaptive algorithms have been utilized, respectively. This paper aims to summarize studies that applied adaptive also called artificial intelligence (AI) algorithms to gait analysis based on inertial sensor data, verifying if they can support the 1.84). Therefore, AI algorithms seem to be able to support gait analysis based on inertial sensor data. Further research, however, is necessary to enhance and standardize the application clinical evaluation. Articles were identified through searches of the main databases, which were encompassed from 1968 to October 2016. We have identified 22 studies that met the inclusion criteria. The included papers were analyzed due to their data acquisition and processing methods with specific questionnaires. Concerning the data acquisition, the mean score is 6.1 ± 1.62, what implies that 13 of 22 papers failed to report relevant outcomes. The quality assessment of AI algorithms presents an above-average rating (8.2 ± in patients, since most of the studies used distinct methods to evaluate healthy subjects.

Introduction

Human gait corresponds to the physiological way of locomotion, which can be affected by several disorders [1]. Thus, gait analysis plays an important role in clinical practice, it provides information about subject's functional level and can be used for health monitoring to verify the efficiency of rehabilitation and to objectively evaluate surgeries' success [2]. The gold standard technologies for gait analysis are optoelectronic systems [3] that offer high accuracy in measuring kinematic features [4]. Unfortunately, such systems can only be used in large laboratories, as it is expensive and space consuming [5].

Regarding gait analysis, there are different nomenclatures for the classification of the events subdividing the gait cycle in distinct phases. Thus, we will start introducing the nomenclature used in this paper that is suggested by Perry and Burnfield [6]. As displayed in Fig. 1, one gait cycle can be divided into two phases, stance (ST) and swing (SW), which can be subdivided into five and three phases, respectively. Corresponding to the beginning of the ST phase, the initial contact (IC) describes the moment when one part of the foot, in physiologic gait the heel, touches the ground. The loading response (LR) starts with the IC and ends with the contralateral toe-off. Subsequently, the mid stance (MS) lasts until the heel-off of the reference leg. The MS is followed by the terminal stance (TS), which ends with the contralateral IC. The ST phase terminates with the pre-swing (PS), which ends with the toe-off of the reference leg. Considering the SW phase, the initial swing follows the PS and terminates when the ankle joint of the reference leg crosses the supporting leg. This phase is followed by the mid swing, which ends with the tibia of the reference leg being vertical to the ground. The gait cycle is concluded by the terminal swing, which ends with the initial contact of the reference leg.

There are several devices and methods to detect the aforementioned events [7]. Due to its practicality, wearable devices become more popular for providing people the possibility to measure bodily features constantly. Therefore, the request for such technology, which measures gait characteristics either for activity recognition or for gait event classification, has risen lately. The most cost-effective method uses either whole inertial measurement units (IMUs) consisting of gyroscopes, accelerometers, and magnetometers or parts of it to determine kinematic data [8]. The attempt of miniaturizing and mobilizing sensor technology [9] showed encouraging results [8]. These systems are used to measure different kinds of kinematic gait data for diverse applications: functional electrical stimulation (FES) [9], [10], [11], gait initiation and termination [12], abnormal gait detection [13], [14], [15], [16], [17], [18] and real-time applications [19], [20], [21], [22], [23].

In addition to the simultaneous acquisition of information from different joints involved in the movement pattern, the system must be able to deal with the complexity of processing such data in short time. In this context, artificial intelligence (AI), also called adaptive, algorithms have been applied to evaluate gait data, which was acquired either based on optoelectronic systems or inertial sensors [2], [24]. Such algorithms are able to adapt their decision-making process based on the input data, despite its variability. Most importantly, adaptive algorithms can simplify the walking evaluation.

Our overarching goal in this article is to summarize those studies that applied adaptive algorithms to gait analysis based on inertial sensors data, to verify whether these algorithms are able to support the evaluation of different kinematic gait parameters. Therefore, the accuracy in determining gait phases, spatiotemporal features and joint angles of former published studies is stated and compared. Furthermore, by systematically reviewing the literature, we intend to proof the following hypotheses: (i) IMU systems are able to acquire gait kinematic parameters such as walking phases, spatiotemporal features and joint angles; (ii) adaptive algorithms can accurately classify gait events; (iii) the classification performed by the AI algorithms is relevant to the clinical practice.

Section snippets

Study identification and selection

A systematic literature search was conducted to find related works to the research hypotheses posed here. The review process was divided into four phases as shown in Fig. 2. As this review is related to engineering and medicine, an automated search in the main databases available, namely, Web of Science, ScienceDirect, IEEE, PubMed/MEDLINE, SCOPUS, CINAHL and Cochrane Library was undertaken to identify relevant publications. The search terms used were defined as (gait OR walking OR locomotion)

Results

This section was divided into three parts. The first subsection deals with the results gathered from the CASP and the second subsection provides information about the analysis and rating of the adaptive algorithms used. The third one summarizes the gait parameters evaluated in the different studies and provides information about the sensor placement, sensor type, and measurement accuracy. The literature search yielded 22 journal papers that met the inclusion criteria (Fig. 2).

Discussion

The purpose of this systematic review is to verify whether adaptive algorithms can support gait analysis based on inertial sensors data. Quality assessment is originally used as the basis for weighting quantitative data in a meta-analysis, which is an important data synthesis strategy [26]. However, the retrieved data in this review was produced by diversified experimental designs, which leads to distinct data. A meta-analysis, therefore, would be inappropriate. For this reason, we did not use

Conclusions

After screening the literature, we realized that some of the included studies failed to report expected information in diagnostic test studies and did not cater for some indispensable methodological procedures. Even though, we found that inertial measurement units could acquire the kinematic features of the human gait, considering the high-rated papers. Especially, gait events such as initial and end contact can be recognized reliably.

Nonetheless, there is a lack of standardization to perform

Conflict of interest

We hereby certify that there is no conflict of interest, and that this paper content has not been published or submitted elsewhere.

Acknowledgements

This review was supported by the Brazilian National Council for Scientific and Technological Development (CNPQ 207445/2014-1) through a doctoral scholarship granted to R.C.

View full text

# data[].attributes.attachment_link.title
A systematic review of gait analysis methods based on inertial sensors and adaptive algorithms

# data[].attributes.attachment_link.description
The conventional methods to assess human gait are either expensive or complex to be applied regularly in clinical practice. To reduce the cost and simplify the evaluation, inertial sensors and adaptive algorithms have been utilized, respectively.

# data[].attributes.headline
Holandês paraplégico volta a andar após tratamento com implantes e inteligência artificial

# data[].attributes.subject
Holandês paraplégico volta a andar após tratamento com implantes e inteligência artificial

# data[].attributes.attachment_link.title
Inteligência artificial é usada para identificar alterações na marcha e diagnosticar doença de Parkinson

# data[].attributes.attachment_link.description
Grupo da Unesp testou técnicas de aprendizado de máquina para avaliar parâmetros como comprimento, velocidade e largura dos passos. Além de tornar o diagnóstico clínico mais preciso, método permite determinar o estágio de evolução da doença

# data[].attributes.headline
A Survey of Human Gait-Based Artificial Intelligence Applications

# data[].attributes.subject
A Survey of Human Gait-Based Artificial Intelligence Applications

# data[].attributes.attachment_link.readable_text
Human Performance and Robotics Laboratory, Department of Mechanical and Aerospace Engineering, California State University Long Beach, Long Beach, CA, United States

Department of Electrical Engineering, California State University Long Beach, Long Beach, CA, United States

Department of Biomedical Engineering, California State University Long Beach, Long Beach, CA, United States

We performed an electronic database search of published works from 2012 to mid-2021 that focus on human gait studies and apply machine learning techniques. We identified six key applications of machine learning using gait data: 1) Gait analysis where analyzing techniques and certain biomechanical analysis factors are improved by utilizing artificial intelligence algorithms, 2) Health and Wellness, with applications in gait monitoring for abnormal gait detection, recognition of human activities, fall detection and sports performance, 3) Human Pose Tracking using one-person or multi-person tracking and localization systems such as OpenPose, Simultaneous Localization and Mapping (SLAM), etc., 4) Gait-based biometrics with applications in person identification, authentication, and re-identification as well as gender and age recognition 5) "Smart gait" applications ranging from smart socks, shoes, and other wearables to smart homes and smart retail stores that incorporate continuous monitoring and control systems and 6) Animation that reconstructs human motion utilizing gait data, simulation and machine learning techniques. Our goal is to provide a single broad-based survey of the applications of machine learning technology in gait analysis and identify future areas of potential study and growth. We discuss the machine learning techniques that have been used with a focus on the tasks they perform, the problems they attempt to solve, and the trade-offs they navigate.

1 Introduction

Smart Gait (SG) is a term for any integrated human gait data analysis system utilizing Artificial Intelligence (AI). It is a growing research field capitalizing on the advancements in modern sensing technologies, automation, cloud computing, data analytics, parallel processing, and Internet of Things (IoT).

Some of the most prominent tasks SG performs are gait phase detection, gait event prediction, human activity recognition, fall detection, recognition of a person's age and gender, abnormal gait detection such as fatigued state, stroke and neurological disease (ND), Parkinson's Disease (PD), estimation of joint angles and moments, the walking person's intent recognition and trajectory prediction, human pose estimation, localization and mapping, person identification, re-identification and authentication, step counting, assessment of physical skill and mobility, balance assessment, fall risk assessment, gait modeling, and simulation. Often SG performs a combination of two or more tasks simultaneously.

SG systems can analyze single or multiple gaits simultaneously. The multi-gait SG performs tasks such as crowd and occupancy sensing, crowd behavior prediction, multi-gait recognition as in identifying a person walking with one or more others, generating multi-gait for animation and virtual environments, and detecting abnormal gait in crowds or indoors for security applications. SG is thus a smart tool in the toolbox of experts in many fields such as health and wellness, security, user privacy, forensics, enhanced user experience, animation, energy, wearables and related fields like insurance, longevity, geriatrics, workplace safety, and productivity. SG can also easily be integrated with other smart systems that utilize heart rate, audio, haptic, speech, etc., for an even wider reach across many applications and industries. As such, SG is a component of many smart devices, smart homes, stores, cities, and energy grids.

This work reviews most of the research in the field of smart gait in recent years. It provides a broad-based survey of the current state of the field and identifies future areas of potential study and growth. First, we performed an electronic database search on six well-known electronic libraries: IEEE Xplore, Science Direct, PubMed Central, Google Scholar, ACM, and Web of Science. We searched for "artificial intelligence" AND "human" AND "gait" in the full text (when available), title, keywords, and abstract. Due to the sheer volume of the work and based on the assumption that most of the more recent works build and improve upon previous work, we limited our search to 2012 to mid-year 2021. We started our work in early 2020 by reviewing papers published in 2012-2019, and updated our review with papers from 2020 and the first half of 2021 as we continued our work through the end of July 2021. All references were downloaded into EndNote. Figure 1 offers an overview of the number of references located by this original search and reflects the fast growth of SG research. Google Scholar, even though at first glance it shows the number of references in the 10,000 range, it allows downloads only up to 1,000 references. However, during our work, we came across other relevant papers that we included in this review. For instance, to identify all the work in studying fatigue by SG, we performed another search to include the word "fatigue" in our search.

Our search resulted in over 6,000 references. First, all papers were reviewed by title, and those not relevant to this study were excluded. These included papers with 1) no "gait" - such as research on upper limb movement 2) no "human" - such as research on animal and robotic gait, and 3) no AI - such as human gait studies that do not incorporate machine learning methods. After the exclusions by title, our database was reduced to about 3500 papers, which were then reviewed by title and abstract. The final selection criteria included at least one paper for each application, even smaller niche examples such as depression detection by gait ( Fang et al., 2019). In areas where many papers existed for the same application, such as neurological disease gait detection, the most impactful papers were selected by looking at the overall influence of the paper by the number of citations on Google Scholar and the journal's impact factor.

Additionally, 393 review papers were originally excluded from this study. None of the works in our database search presents a comprehensive review of the applications of AI in gait studies. These works fall in one of the three categories as below:

1) studies in related fields but not exactly within the inclusion criteria of this work, For instance, Najafi and Mishra (2021) compiled a narrative review on digital health technologies for diabetic foot syndrome. While ML algorithms are probably embedded in some of the technologies they discuss, the study does not mention or discuss artificial intelligence.

2) studies that partially overlap with our studies, such as the literature review on PD diagnosis by Mei et al. (2021), that in addition to PD diagnosis by gait, also discuss other modalities such as voice, handwriting, magnetic resonance imaging (MRI), etc.

3) studies that are in the scope of this review but only cover a specific topic such as human motion trajectory prediction ( Rudenko et al., 2020), wearable sensing technologies for sports biomechanics ( Taborri et al., 2020), self-powered sensors and systems ( Wu et al., 2020), person re-Identification ( Wang et al., 2016), ( Nambiar et al., 2019), ( Karanam et al., 2019), machine learning in soft robotics ( Kim et al., 2021), ambient assisted living technologies (mostly AI-enabled and gait-related) ( Cicirelli et al., 2021), human action recognition ( Gurbuz and Amin, 2019), biomechanics ( Halilaj et al., 2018), gait recognition ( Kusakunniran, 2020), ( Singh et al., 2018), ( Wan C. et al., 2018), gait event detection and gait phase recognition ( Prasanth et al., 2021), clinical gait diagnostics of knee osteoarthritis ( Parisi et al., 2020), knee pathology assessment ( Abid et al., 2019), data preprocessing in gait classification ( Burdack et al., 2019), age estimation ( Aderinola et al., 2021), and banchamrk datasets ( Nunes et al., 2019). A survey paper by Alzubaidi et al. (2021) provides an overview of deep learning, with helpful definitions and a discussion of strengths, limitations, and future trends of various deep learning techniques. Similarly, Abiodun et al. (2019) review Artificial Neural Network (ANN) applications in pattern recognition. Our paper's goal is to review applications of SG technology, in a wider, all-encompassing overview. The above-mentioned papers were not used in our study; the reader is encouraged to read them if they wish to go deeper into any of the discussed topics.

We identified six main applications of SG: 1) Gait analysis where analyzing techniques are improved through AI algorithms, 2) Health and Wellness, with applications in gait monitoring for abnormal gait detection, recognition of human activities, fall detection, and sports performance, 3) Human Pose Tracking covering one-person or multi-person tracking and localization systems, 4) Gait-based biometrics with applications in person identification, authentication, and re-identification as well as gender and race recognition 5) Smart Gait devices and environments ranging from smart socks, shoes, and other wearables, to smart homes and smart retail stores that incorporate continuous monitoring and control systems and 6) Animation that reconstructs human motion through gait modeling, simulation, and machine learning techniques. The categories are sometimes not easy to separate and overlap. For instance, gait phase detection algorithms could belong to Health and Wellness, SG Devices, or Gait Recognition categories.

The paper is organized in the following sections: Section 2) Smart Gait Vocabulary, sections 3-8 discuss the applications of AI in 3) Gait Analysis 4) Health and wellness 5) Tracking Human Pose 6) Gait Based Biometrics 7) Smart Gait devices and 8) Animation. In sections 9 and 10, we conclude with 9) Discussion and Future Trends and (10) Conclusions.

2 The Smart Gait Vocabulary

Here we introduce certain key definitions of essential terms commonly used in SG studies (See Table 1).

3 Gait Analysis

Machine learning techniques are successfully utilized to improve many aspects of gait analysis. In this collection of papers AI 1) helps with data aggregation and pre-processing, 2) works along with another AI to improve its performance, interpretability, or accuracy, and 3) classifies gait phases and predicts gait events.

3.1 Data Aggregation and Pre-processing Using ML Techniques

PCA is a dimension reduction technique that transforms the original feature space into a set of linearly uncorrelated variables called principal components (PCs). The first few PCs alone are usually enough to account for most of the variance in the data. For instance, the first 3-6 PCs alone accounted for 84-99% of the overall variance ( Dolatabadi et al., 2017). Other variants of PCA have been suggested in the SG literature, such as kernel-based PCA ( Semwal et al., 2015), multi-linear PCA that achieves sparse and discriminative tensor to vector projection ( Zhang et al., 2015), or a combination of PCA, linear discriminant analysis (LDA) and other feature reduction techniques ( Wu C. et al., 2019). ( Phinyomark et al., 2015) attempt to evaluate the importance of intermediate to higher-order PCs in running biomechanics, finding that low order PCs (that account for up to 90% of the cumulative variance in the data) can be successfully used for age and gender recognition. Still, the more subtle running behavior patterns such as between-group variations in improvements after a 6-weeks rehabilitation program of runners with patellofemoral pain (PFP) can be captured by intermediate and high order PC's (that explain 90-99% and 99-100% of the variance of the data correspondingly). PCA, some variation of PCA, or a combination of PCA and other linear dimension reduction techniques were used in most papers we reviewed. t-Distributed Stochastic Neighbor Embedding (t-SNE) was used by Costilla-Reyes et al. (2021) to achieve a 2D representation and visualization of different age clusters of the trained data. Besides feature reduction, AI is employed for data augmentation ( Bhattacharya et al., 2020) or data engineering ( Johnson et al., 2021).

Other considerations in carefully selecting the input features are avoiding overfitting ( Zhang et al., 2020a), improving interpretability, especially in medical applications ( Dindorf et al., 2020b), ( Horst et al., 2019), reducing the energy expenditure of wearable sensors ( Lan et al., 2020), ( Russell et al., 2021), improving patient comfort ( Di Nardo et al., 2020), fairness to people with disabilities ( Trewin et al., 2019) and user experience ( Kim et al., 2020). To achieve the highest possible accuracy, usually, more complex sensing technology is required. The goal is to apply feature reduction and selection to avoid redundant features such that only the significant features are extracted from the minimum sensing hardware ( Khademi et al., 2019). Information Gain (IG) and not PCA is used to improve interpretability by preserving and ranking the original features ( Onodera et al., 2017).

3.2 AI2AI

Similarly, as a first phase to creating an activity recognition algorithm for construction workers ( Kim and Cho, 2020), the authors first implemented five ML algorithms 32 times each, once for every possible combination of the number of Inertial Measurement Unit (IMU) sensors and their location in the body, and compared their performance by a cross-validation (CV) technique. This systematic approach of evaluating how many sensors are sufficient for activity recognition and where they should be placed in the construction workers' body revealed that using two motion sensors located at a certain distance will achieve motion recognition performance like using all 17 motion sensors located throughout the entire body. Using the first phase results, the authors then proceed to build a long-short term memory (LSTM) algorithm for activity recognition that achieves 94.7% accuracy with just two sensors placed in the hip and neck.

3.3 Gait Phase Classification and Gait Event Prediction

Gait analysis evaluates a person's walking pattern, which is seen as a sequence of gait cycles, where each gait cycle follows the movement of a single limb from heel-strike to heel-strike again ( Gage et al., 1995). The two main gait phases are the stance phase and the swing phase. Depending on the reason for gait analysis, detecting just these two phases can be enough. That simplification permits less complex and cheaper gait analysis, which is desirable, especially in wearable systems ( Di Nardo et al., 2020). A more common four-phase cycle includes initial contact, mid-stance, pre-swing, and swing ( Jiang et al., 2018). The importance of AI in these studies is in facilitating real-time gait analysis, appreciated in many control devices like orthotics and prosthetics, rehabilitation monitoring, and fall detection systems for aging-in-place applications. (See Table 2).

4 Health and Wellness

Clinical gait analysis, though by itself not reliable for a definitive diagnosis of neurological disease or other impairment, often suggests a pathology if it detects a pattern different from a typical walking behavior considered the normal gait. Normal gait is a controlled, coordinated, and repetitive series of limb movements that advance the body in the desired direction with minimum energy expenditure ( Gage et al., 1995). There are four reasons for performing a clinical gait analysis: diagnosis, assessment, monitoring, and prediction ( Baker et al., 2016). In these applications, the AI can continuously monitor and learn data, look for patterns, classify human activities and detect the anomaly. If connected to a display, it is an excellent monitoring and assessment tool. AI also predicts a future gait event, in which case it can either alert a human operator such as a clinician, caretaker, or facility supervisor or, if integrated with a control device, activate an automatic response to prevent falls or injury. For instance, WeedGait passively monitors the gait of a person and then alerts them if they are at risk of Driving Under the Influence of Marijuana (DUIM) ( Li et al., 2019), while an in-home rehabilitation system provides qualitative and quantitative feedback to post-stroke survivors ( Lee et al., 2019).

We identified four major applications of ML gait analysis in health and wellness: 1) detecting abnormal gait due to a person's condition or disease, 2) sports management, 3) fall detection, and 4) activity recognition.

4.1 Abnormal Gait

AI is well suited at learning patterns and detecting an anomaly in the data based on a pre-defined abnormal event (supervised learning) or a clustering algorithm (unsupervised learning), or a combination of the two. A very wide range of human diseases and conditions can affect the way a person walks such as Parkinson's ( Flagg et al., 2021), ( Wahid et al., 2015), Huntington's ( Acosta-Escalante et al., 2018), ALS ( Aich et al., 2018), idiopathic normal-pressure hydrocephalus (iNPH) ( Ishikawa et al., 2019), ASD ( Hasan et al., 2018) neuromuscular disease ( Gotlin et al., 2018), pediatric hereditary spastic paraplegia (HSP) ( Pulido-Valdeolivas et al., 2018), aging ( Strath et al., 2015), ( Costilla-Reyes et al., 2021), dementia ( Kenney et al., 2018), ( Arifoglu and Bouchachia, 2017), fatigue ( Zhang J. et al., 2014), depression ( Fang et al., 2019), anxiety ( Zhao et al., 2019), emotional state ( Bhattacharya et al., 2020), dual task, or walking while performing a cognitive task ( Costilla-Reyes et al., 2021), knee osteoarthritis, ( Kotti et al., 2017), stroke (PSH gait), ( Cui et al., 2018), ( Clark et al., 2015), diabetes ( Sutkowska et al., 2019), COVID-19 ( Maghded et al., 2020), inflammation ( Lasselin et al., 2020), ( Renner et al., 2021), level of physical activity ( Renner et al., 2021), kidney disease ( Yadollahpour et al., 2018), vertigo ( Cao et al., 2021), sleep quality ( Liu X. et al., 2019), Trendelenburg ( Michalopoulos et al., 2016), arthritis ( Karg et al., 2015), ( Struss et al., 2018), idiopathic toe walking ( Kim et al., 2019), drunkenness ( Arnold et al., 2015) and influence of marijuana ( Li et al., 2019). Thus, monitoring human gait can provide key insight into a person's health. Gait data can serve as a biomarker by itself or in association with other biomarkers, demographic data, other measured or calculated body and health parameters ( Andersson et al., 2014), and patient-generated health data (PGHD) ( Jim et al., 2020).

The main role of ML techniques for this cluster of studies is in classifying 1) healthy control from pathology or 2) healthy control from multiple classes of functional gait disorders. Some frequently used words in these papers, such as unobtrusive, plug-and-play system, automatic, affordable, integrated into the home or clinic, adaptable, etc., shed light on the perceived values of these applied systems and where most of the research is focused.

The benefits of utilizing machine learning in abnormal gait detection for health applications are as follows:

1) Deep learning models can automatically extract the optimal features directly from raw spatiotemporal gait data without the need for data preprocessing or engineering. ( Costilla-Reyes et al., 2021).

2) A combination of ML techniques can deploy simultaneously to perform more than one task automatically. For instance, ML can simultaneously detect pathological gait and automatically identify which body part has been affected the most by the disease ( Dolatabadi et al., 2017).

3) ML can process a lot of data fast, including active online learning from streams of live and repository data ( Flagg et al., 2021).

4) ML community can benefit from the collaborative effort. It fosters competition against established benchmarks and open research promoted by initiatives such as the Parkinson's Disease Digital Biomarker (PDDB) DREAM Challenge ( Zhang et al., 2020a)

From a practical viewpoint, using the analogy of a human operator driving a car, some common issues with these studies are:

1) Can't drive faster than sight: the problem of data labeling.

There is a need to minimize the distance between the experimental setup and the real-life application of a system. For instance, the use of a home monitoring system for abnormal gait was studied ( Guo et al., 2019), but the subjects used in the study were healthy subjects imitating abnormal gaits such as in-toeing, out-toeing, drop foot, supination, and pronation. To study the gait of dementia patients, an artificially generated dataset was proposed ( Arifoglu and Bouchachia, 2017) where a dataset of normal ADL data was injected with instances of skipped or repeated activities during the day and sleep disruptions at night to mimic the abnormal activities that people with dementia (PwD) would manifest. A biped that imitates human motion was used to train the data for the recognition of anterior cruciate ligament (ACL) injury of the knee ( Zeng et al., 2020). Often, studies are performed on normal gait subjects and cannot be extended to a pathological gait. Trewin et al. (2019) discuss considerations of fairness for people with disabilities and outline a few guideposts on problem scoping, data sourcing and pre-processing, AI model training, and deployment. They advocate inclusive, participatory, and value-sensitive design.

Similarly, there is also the problem of a small sample size. Given the clinical nature of these studies and the impaired gait participants they require, the barriers to experimentally collecting sufficient data are understandable. For instance, 19 healthy participants had to consent to intravenous injection of lipopolysaccharide to induce inflammation in a fatigued gait study and were paid 3500SEK each ( Lasselin et al., 2020). In some scenarios, the number of input features is larger than the sample size ( Zhang et al., 2015). In others, the sample size is too small to be representative enough to support any assertions fully and reduces the paper to the level of an exploratory effort ( Baghdadi et al., 2021). In others, researchers generate synthetic data ( Arifoglu and Bouchachia, 2017) that reflect features similar to the disease for an abnormal gait detection task; apply deep AL technique to reduce the number of required labels and consequently the time cost of manual labeling in a gait phase detection task. Finally, there is a developing trend to take measurements out of the lab ( Russell et al., 2021) and into the subjects' natural environments and implement deep learning techniques to do the labeling ( Costilla-Reyes et al., 2021), ( Cronin et al., 2019).

Many of these techniques in aggregating, pre-processing, and learning the data, often represent work-around strategies to cost, user privacy, and clinical constraints. They both simplify the systems and introduce some errors in them simultaneously.

2) Can't see in the dark: The Black Box problem.

3) Can't leave the parking lot: the research to commercialization gap and the need for government approval.

Medical devices that utilize data and ML techniques will need Food and Drug Administration (FDA) approval and general buy-in from medical professionals and their patients. Doctors, clinicians, therapists, carers, et al. will need to be willing and capable of embracing the newness of the technology. To date, the authors are not aware of any FDA-approved medical devices that utilize AI and gait data. Neurodegenerative diseases are not symptomatic until years after their onset, but clinical usefulness needs to be demonstrated for the approval of a medical device. The cost of developing, installing, and maintaining such systems also becomes a barrier to their commercialization and practical usefulness ( König et al., 2015).

The most recent advancements seem to address some of these concerns, but those are only in the beginning phases, and the long-term implications on user safety and privacy, as well as their actual performance, remain to be proved.

4.1.1 Fatigued Gait

Fatigue is defined as "a lower level of strength, physical capacity, and performance" ( Lu et al., 2017). Detecting the onset of fatigue and creating systems that manage the associated risks is an important part of production quality and human factors engineering in the workplace.

Measuring and analyzing gait for fatigue monitoring makes sense because 1) walking is a task that is possible to track via unintrusive technology suitable to workplace settings, such as video, wearable sensors, radar, and force plates on the floor or any combination of these. 2) Walking is a significant part of occupational tasks for workers in manufacturing, mining, construction, nursing, warehouse and distribution centers ( Baghdadi et al., 2021). 3) in the context of advanced manufacturing (known as Industry 4.0) for example (but other modern occupational settings as well), in which a worker's daily tasks involve interacting with automation, computing, and sensing technologies, the most relevant features extracted from readily available gait data can be pre-processed and analyzed through ML techniques, often real-time, which makes it possible to detect the onset of fatigue with accuracy and speed, at a relatively low cost. 4) Lastly, fatigue in the walking behavior is correlated to fatigue in other physical tasks, and while safety systems in the workplace should be custom-tailored to the relevant tasks, detecting fatigue in a worker's gait could serve as an excellent general fatigue monitoring technique applicable to most industrial settings.

There are two main goals in fatigued gait studies: 1) intra-person recognition or continuous recognition of the person walking in different fatigue states 2) inter-person recognition or the recognition of fatigue in an individual. The first one answers the question: Can we still identify the person by their gait in a fatigued state? The second one answers the question: Can we recognize the onset of fatigue to avoid overtraining and injury in sports or improve worker safety in the workplace? Researchers conclude that a person's gait pattern maintains its individuality even while manifesting situation-dependency, as is the case with fatigue ( Janssen et al., 2011).

Overall, fatigue studies utilizing gait data and AI point to the importance of SG in managing fatigue in the workplace, sports performance management, rehabilitation exercises, reducing fall risk in the elderly, and finally, as part of an integrated system for overall health management, both at the individual level and in public health applications, as is the case with Covid-19 related studies. The fact that most studies we reviewed collected gait data from just one sensor or just the smartphone ( Karvekar, 2019) shows that effort is already invested in making these systems unintrusive, cost-effective, and adaptable (see Table 3).

4.1.2 Neurological Disease

Gait-based detection and classification algorithms for disease diagnosis and monitoring are one of the major applications we saw, and Parkinsonian gait, with its many classifiable features, such as freezing of gait (FoG), shuffling steps, slow gait, gait asymmetry, etc., is the most prevalent disease in the studies (See Table 4). Researchers propose an automated, accurate, and sensor-free gait detection deep learning algorithm that depends on video recordings from pervasive devices such as smartphones, web cameras, and surveillance cameras as cheaper and more accessible alternatives to Vicon camera systems ( Ajay et al., 2018). Procházka et al. (2015) move away from expensive, complex camera systems and recommend using MS Kinect image and depth sensors for synchronized data acquisition and spatial modeling of a moving person. The recommended Bayesian Classification (BC) algorithm distinguishes PD gait from healthy gait based on decision boundaries of three features: gait speed, stride length, and age, with an achieved accuracy of 94.1%. Wahid et al. (2015) contribute to the gait-based PD detection by suggesting that the spatial-temporal gait data be normalized first using multiple regression to account for the patient's age, height, body mass, gender, and walking speed. ( Wan S. et al., 2018) employ a deep MLP to analyze both movement and speech data captured through a smartphone and estimate the severity of PD. Ye et al. (2018) go a step further and propose a Neural Network (NN) combined with Fuzzy Logic (FL) approach that recognizes the gait of patients with neurodegenerative disease (ND) from normal gait. Since the motor function impairment in various NDs such as ALS, HD, and PD is caused by different factors, the particle swarm optimization (PSO) algorithm was used along with an adaptive neuro-fuzzy inference system (ANFIS) to classify the non-linear gait dynamics. Bilgin (2017) also attempts to distinguish ALS from other ND diseases and healthy patients using Discrete Wavelet Transform (DWT), LDA, and Naïve Bayesian Classifier (NBC).

One of the most encouraging recent developments in ND gait research using AI is the DREAM PDDB Challenge launched by Sage Bionetworks that promotes an open and competitive research infrastructure with large-scale data for developing digital signatures of PD. Similar challenges in computer vision, such as the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) ( Russakovsky et al., 2015), have shown to be conducive to tremendous growth with classification accuracy improving year after year against the previously established benchmark. The DREAM PDDB challenge utilizes the self-reported and sensor data collected through the mPower app ( Bot et al., 2016) from 15,000 PD and healthy control (HC) subjects. The best performing team as of July 2021 reported Area Under the Receiver-Operating Characteristic Curve (AUROC) of 0.86 with a deep CNN algorithm that employs three spatial and temporal data augmentation techniques to deal with overfitting.

Another initiative, still in the early phases but very promising, is the Early Detection of Neurodegenerative Diseases (EDoN) project. Launched in February 2020, it is a global research initiative that has secured generous funding, prominent cross-disciplinary expertise, and UK government support ( Wakefield, 2020). It is already in the first phase of collecting smartwatch data (gait data along with heart rate, sleep, navigation data, etc.) from volunteers in the Greater Boston area through a partnership with Boston University Alzheimer's Disease Research Centre (BUADRC). The data will be used to generate a digital "fingerprint" for dementia which, in the application phase, will detect dementia 10-15 years earlier than the current clinical methods.

4.2 Sports

The general trend we see in sports and fitness applications of ML-based gait analysis systems is to create low-cost, adaptable, fast, easy, and scalable systems that reach the right balance of accuracy and comfort for the given application (See Table 5). In sports management and sports injury prevention, gait data is lower limb movements signals acquired from various sensing technologies: video, IMUs, force plates, etc. SG monitors walking and running activities but also water sports, basketball, and football.

SG systems in sports applications are diverse, performing data engineering ( Johnson et al., 2021) and data labeling ( Cronin et al., 2019), evaluating the role of the shoe structure on running biomechanics ( Onodera et al., 2017), monitoring fatigue to prevent injury ( Russell et al., 2021), ( Zhang J. et al., 2014), counting steps ( Kang et al., 2018), assessing physical autonomy and functional ability ( Khan et al., 2015), articulating real-time control of an electrical muscle stimulation (EMS) device for sports training ( Hassan et al., 2017), predicting and preventing injury ( Taborri et al., 2021), achieving multi-player tracking, identification, and re-identification ( Zhang R. et al., 2020), classifying and counting different sports activities ( Sundholm et al., 2014), and recognizing and analyzing sports behavior ( Guo and Wang, 2021).

4.3 Fall Detection and Human Activity Recognition

Gait data for these studies were collected from force sensors and three-axis accelerometers concealed under intelligent tiles ( Daher et al., 2017), built-in smartphone IMU sensors ( Hakim et al., 2017), wearable sEMG sensors ( Xi et al., 2017), wearable motion sensors ( Özdemir and Barshan, 2014), or from an integrated data collection system such as inertial sensor and Bluetooth nodes data captured on a smartphone ( Santoyo-Ramón et al., 2018). Sensor data from both a smartwatch and a smartphone is shown to work better than either one individually ( Weiss et al., 2019). The smart home prides itself in capturing data real-time and device-free, utilizing wi-fi enabled IoT platforms ( Yang et al., 2018). Finally, radar data from continuous-wave radar systems can be used for activity recognition and fall detection providing an unobtrusive solution for data collection and posing no privacy concerns ( Wu Q. et al., 2015), ( Seyfioğlu et al., 2018). CapSense sensing technology uses KEH capacitor voltage traces to recognize among five different activities with an accuracy of over 90% ( Lan et al., 2017). In subsequent work, the accuracy was improved by using two capacitors, one in the sole of a shoe and one in front ( Lan et al., 2020). The overall system energy usage was also reduced by 75% since the conventional machine learning algorithms such as NB or KNN used accumulated voltage data as input, reducing the computational load of the system. Future work will make these sensors fully self-powered and battery-free and address activity recognition accuracy and user experience issues.

In summary, SG systems have performed with increasing accuracy, minimizing false alarms, thus providing accurate, low-cost, automatic activity recognition and fall detection systems (see Table 6).

5 Tracking Human Pose

6 Gait Based Biometrics

6.1 Person Identification

Most person identification SG systems deal with the case when gait data is extracted from video. These systems are model-based ( Hu et al., 2012), ( An et al., 2020), or appearance-based ( Guan et al., 2012; Zhang W. et al., 2019), also called model-free. Gait Energy Image (GEI) is an average of all the human body silhouette images in one gait cycle ( Han and Bhanu, 2005). It is widely used in appearance-based person identification algorithms because it allows for a simple representation of gait data, removing noise while preserving relevant gait information. It strikes a good balance between reducing computational cost and maintaining a good gait recognition rate. On the other hand, GEI is sensitive to appearance variations such as camera view angles and whether a subject is wearing a coat or carrying a bag, and it loses temporal information. There have been efforts to replace GEI with a representation that preserves more of the temporal gait information such as Chrono Gait Image (CGI) ( Wang et al., 2011) or to fuse GEI features with temporal features ( Liu et al., 2018). Chao et al. (2019) proposed GaitSet, which uses unordered sets of equally sized gait silhouettes as the input to a CNN architecture with set pooling, providing an effective way to preserve spatial and temporal information without the sequential constraints. argued that not all body parts contain discriminative information for gait recognition tasks, and they proposed GaitPart, a part-based and micro-motion model that preserves only the relevant part-dependent spatial features and the local short-range temporal information. An et al. (2020) acknowledges the challenges of appearance-based models and proposes a pose-based model approach for gait recognition. The gait poses are extracted from video using deep learning techniques AlphaPose ( Fang et al., 2017) and OpenPose ( Cao et al., 2017). Tracking the 3-D pose of walking pedestrians in video surveillance systems in cases where multiple people move together and cast a shadow or cause occlusion has also been attempted ( Rogez et al., 2014).

Besides video, gait data can come from depth sensors, force plates, radar, Wi-Fi-enabled IoT devices, and IMU sensors. Radar-Id, a radar-based human identification algorithm ( Cao et al., 2018), employs a CNN with architecture similar to that of AlexNet ( Krizhevsky et al., 2012) to learn the necessary features from raw micro-Doppler spectrograms directly without a need to explicitly design the features. This algorithm has excellent anti-noise performance, and it can identify one person amid up to 20 other people. Similarly, Vandersmissen et al. (2018) use radar device data for indoor person identification and intruder detection. Costilla-Reyes et al. (2019) propose a footstep recognition system that can differentiate between the legitimate users (clients) and the impostor users of a biometric system from sensors on the floor. Biometric recognition by gait makes it possible to identify intruders since gait is difficult to fake. Deep CNN architectures that utilize footstep representations extracted from GRFs serve as automatic continuous person identification and verification systems with applications in security and anomaly detection at airports, workplace environments, and smart homes. Zhong and Deng (2014) propose a gait representation using accelerometer and gyroscope data invariant to sensor orientation. Haque et al. (2016) present a method for human identification when given only depth images. Zou et al. (2018) propose Auto-ID, a human identification system that collects CSI measurements data from WiFi-enabled IoT devices referred to as shapelet "signatures" of human identification. Yang et al. (2018) also use the CSI curve of the human body for occupancy sensing and activity recognition. This system is good for anomaly detection, such as identifying intruders in a smart home automatically. A person walking in a smart space equipped with RFID devices affects the radio frequency (RF) signals. The effect can be captured by Received Signal Strength Indicator (RSSI) and the phase of the RF signal and then used for person identification. A system that employs TSNet, a tag selection deep reinforcement learning algorithm, PCA for feature reduction, and an attention-based LSTM algorithm performs RFID-based gait recognition that can easily be integrated into a smart home or smart office environment ( Luo et al., 2020).

Person identification studies have benefitted from the developments in computer vision, established benchmarks, and public datasets. The code proposed in the studies is often made public, encouraging continued and collaborative research ( Chao et al., 2019). Recent studies employ deep learning approaches that extract gait representations directly from raw gait data and learn discriminative features for human identification (see Table 8)

6.2 Person Re-Identification

6.3 Person Authentication

Proper authentication includes user authentication: whether the user has authorized access, and person identification: who the current user is ( Liang et al., 2020). Sensor-based user authentication uses biological features categorized in physical, physiological, and behavioral ( Hernández-Álvarez et al., 2021). Gait provides behavioral biometric-based authentication, which by comparison with knowledge-based (passwords, personal identification number (PIN)) and physiological biometric-based (face recognition, fingerprint) is unobtrusive, continuous, less prone to attacks, and easily tracked through wearable devices, videos, and smartphones in the context of IoT environments. Gait-based authentication is studied either by itself ( Qin et al., 2019) or as part of an authentication system that uses multiple modalities ( Hintze et al., 2019) ( Acien et al., 2019) (See Table 10). Authors ( Acien et al., 2019) showed that the fusion with behavioral data improves the authentication system results.

6.4 Gender Recognition

7 Smart Gait Devices and Environments

This section reviews systems that incorporate ML, IoT, and advanced sensing and textile technologies for automatic, real-time gait data processing to perform an intelligent task with health, sports, entertainment, and security applications.

7.1 Smart Gait Devices

The domain of smart gait devices and environments is exciting, brave, creative, extensive, and ever-growing (See Table 12). SG devices include wearable shoes ( Zou et al., 2020), socks ( Zhang et al., 2020f), kneepads and anklets ( Totaro et al., 2017), insoles ( Low et al., 2020), as well as devices attached to the body, such as smartphones ( Poniszewska-Maranda et al., 2019), smartwatches ( San-Segundo et al., 2018), ( Sigcha et al., 2021), etc., implantable medical devices such as ActiGait ( Sturma et al., 2019), wearable robotics ( Shi et al., 2019) such as prosthetics ( Gao et al., 2020) orthotics ( Zhang et al., 2020e), ( Choo et al., 2021), assistive devices such as smart walkers ( Jimenez et al., 2018), and environmental devices such as smart tiles ( Daher et al., 2017). SG devices use gait data to facilitate health monitoring, including passive mental health assessment ( Rabbi et al., 2011) and transfer data to control devices for health, sports, security, and entertainment applications. For instance, UbiHeld (Ubiquitous Healthcare for Elderly) incorporates gait and other data from a smartphone as well as additional data from an inexpensive Kinect camera to keep a status of the overall health, location, and activities of the elderly at home ( Ghose et al., 2013).

The requirements for soft smart wearable garments can be conflicting: stretchable, entirely conformable to the body, designed ergonomically and esthetically pleasing, small size and weight, flexible, washable, robust, unobtrusive, reliable, and durable ( Yang and Yin, 2021). Person identification is an integral part of smart gait devices due to the need for customized user experience, which introduces the need to preserve users' privacy. Additionally, continuous, seamless user authentication is vital to prevent malicious attacks on users' medical records without the burden of frequently entering a personal identification number (PIN) ( Xu et al., 2021). Finally, other requirements include water-proof capability, mechanical durability, and connectivity with other smart devices and environments depending on the specific use ( Zou et al., 2020). To accommodate these requirements, the main focus of the research is on sensing technology. Sensors for wearables are IMUs, capacitive sensors ( Lan et al., 2020), KEH ( Xu et al., 2017; Xu et al., 2018), ( Ma et al., 2018), solar cells ( Sandhu et al., 2021), resistive sensors, stretchable conductive micro fluids ( Low et al., 2020), and TENGs ( Zhang et al., 2020g).

There is generally a trade-off between the complexity of sensing technology in smart gait devices and the accuracy of the intelligent task they perform ( Khademi et al., 2019). Depending on the task and granted sufficient accuracy, one can stop the chase for performance and focus on patient comfort ( Di Nardo et al., 2020). A multi-object optimization (MOO) technique is implemented ( Khademi et al., 2019) to navigate this trade-off: it selects an optimal feature subset that maximizes accuracy while minimizing sensing hardware. Fusing different bio-signals in hBCI systems for gait applications improves classification accuracy and the number of control commands. Still, it introduces the problem of channel configuration, information transfer rate, and temporal synchronization between the modalities ( Khan et al., 2021).

SG devices will be an integral part of smart cities, smart buildings, smart homes, smart transportation, smart factories, energy grids, and e-Healthcare and are poised for tremendous growth in the future as IoT in the 5G framework will facilitate better and faster inter-connectivity between the devices ( Chettri and Bera, 2019), end-to-end deep learning algorithms will provide real-time intelligence, and sensing technology will be embedded more comfortably into our everyday objects and clothing.

7.2 Smart Homes

Of interest in this discussion is the project HABITAT (Home Assistance Based on the Internet of Things for the Autonomy of Everybody) that had only four key AI applications at the time this paper was written: an indoor localization system, smart armchair, smart belt, and a wall panel and mobile devices as the user interface ( Borelli et al., 2019). The smart home embeds smart objects into objects of everyday life. The belt, which assesses body movement, is of interest to this study since it captures postural transitions and gait biometric data. The AI modules follow the Event Calculus (EC) modelling approach, which makes it easy to define the properties of the system. The system is expandable, more smart devices can be incorporated into it, and is an example of what is possible when activities and objects of our daily lives become "smart." In summary, AI algorithms take care of the "smart" part of the home. Instead of being a passive recipient of care, the patient becomes an active agent of an intelligent health ecosystem ( Najafi and Mishra, 2021).

8 Animation and Virtual Environments

For this review, we consider studies requiring and using gait data and implementing an ML algorithm to create animated characters for movies, video games, and virtual reality environments. These studies include research in gait modeling, motion reconstruction, and character control. (See Table 14)

In Virtual Reality (VR) applications, the virtual character is controlled real-time by the user's behavior. The task of the AI algorithm is to correctly recognize the user's gait phases, with no delay, for a good real-time visual representation of the avatar's motion. In video games, deep reinforcement learning is the algorithm of choice where the policy controls the action at each step. The policy is defined carefully to positively reward the desired action at each step, such as maintaining balance, tracking pose and orientation, alignment, mimicking the reference motion, surviving perturbations, and negatively rewarding actions such as falls. The avatars must be real-looking, preserving both the motion extracted from the skeleton features and the shape ( Loper et al., 2014), ( Huang et al., 2015).

Responsiveness to user demand, quality of visual representation, robustness to different walking styles and terrains, system runtime performance, adaptation to disturbances, maintaining balance, and retargeting to different morphologies are some of the key goals of these systems. Usually, motion data is extracted from expensive high-quality multi-camera motion capture systems. Authors have studied the possibility of retrieving high-quality representations from low-cost video clips recorded with pervasive monocular videos such as YouTube clips ( Peng et al., 2018b) and reconstructing human pose from inertial measurements, in the case when direct-line-of-sight camera recording is not available due to occlusion for instance ( Huang et al., 2018).

9 Discussion and Future Trends

In this study, we reviewed different applications of the smart gait with a focus on the various tasks artificial intelligence algorithms perform across many industries and disciplines, such as health and wellness, security, forensics, and energy management. We further identify four emerging trends in the SG research: 1) population-wide scale health data will be available and empower end-to-end automatic, ubiquitous, and continuous deep learning approaches for big data-driven intelligent systems, 2) Fast-growing innovations in other technologies such as cloud computing, smart textiles, blockchain, and 5G will offer new opportunities and pose new challenges for SG systems and demand fast congruent growth, 3) SG systems will need to address concerns about user privacy, safety, comfort, and experience captured by the paradigm, "human-in-the-loop" ( Sedighi Maman et al., 2020); these will be enforced by human rights advocates and regulatory bodies, and 4) the need for AI in health applications to benefit from the fast, low-cost, and accuracy of "black box" intelligent systems while still being transparent and understandable, as captured by the paradigm of XAI. SG is a valuable tool in kinetic and kinematic analysis, disease monitoring, diagnosis, and rehabilitation, sports performance, fall risk assessment, detection and prevention, gait-based biometrics for person identification, re-identification and continuous automatic authentication, age and gender recognition, physical skill and mobility assessment, fitness tracking, gait modeling and simulation, crowd monitoring and anomaly detection, human pose estimation, indoor tracking, and localization. SG is often integrated with other smart systems that utilize biomedical signals such as electrocardiograms, body parameters such as temperature, blood pressure, respiration rate, energy expenditure, and heart rate, and environmental signals such as room temperature and humidity. SG is often part of a smart IoT framework embedded with sensors, monitoring devices, and AI-enabled actuators that are all connected and in continuous communication. SG is everywhere, in our smart devices, smart homes, classrooms, cars, stores, cities, and energy grids. Smart Gait research will continue to grow fast in the future and will benefit from advancements in other technologies such as sensors, blockchain, IoT, textiles, 5G, cloud computing, and big data. These new technologies will also pose new demands and offer new opportunities for smart gait research. SG research will continue to address user privacy issues, security of health data, patient comfort, worker safety in the workplace, enhanced user experience, fatigue monitoring and injury prevention in sports.

We also identify these three needs for the SG research community: 1) inter-disciplinarity 2) the need for SG to become an organized field of study with specific definitions and good practices in place, and 3) the need for open competition and collaboration. Looking into the future, digital technologies such as smartwatches, headbands, and smartphones will be exploited to collect population-wide scale data to facilitate health monitoring and early detection of various diseases. One aspect of such systems is the multimodality of the data ( Frey et al., 2019), ( Abeysekara et al., 2020). Their integrated approach will demand cross-disciplinary research and collaborations. The gait research teams will need to include experts across many disciplines. We see the collaborative culture becoming more prevalent in the future both, within the SG community and between the SG community and the larger research community across many fields. With the recent development of programming languages and the open research community around them on Github, Kaggle, and other online and app-based forums, it is possible and important that the Smart Gait research community is open and the datasets, AI code, and suggested strategies for improvement are available for future collaborative work. For instance, some authors have made the data and the ML toolbox public and available for other researchers ( Baghdadi et al., 2021) and ( Horst et al., 2019). Wherever such open and collaborative efforts have flourished in the past, the results have been outstanding. Computer vision studies have seen tremendous results by reducing the cost of entry to new research and encouraging collaboration and competition towards a goal. The authors of this paper are excited to see similar growth in the SG research community.

10 Conclusion

The utilization of AI in gait analysis is a growing field. In this paper, we refer to it as the Smart Gait. It is compellingly multi-disciplinary, drawing from cutting-edge research in multiple mathematical and engineering fields, and it continues to welcome new approaches and new data-capturing methods. With the constant advances in wearable sensors technology, cloud computing, and new advances in ML, the progress is formidable. Coupled with the growing realization of real-world applications such as non-invasive person identification, person re-identification, intruder detection, medical diagnosis and treatment, advanced fall detection, etc., will keep the demand for new gait detection and analysis methods high and keep AI at the forefront of the research. The field should only grow and expand in scope over the next 10-15 years. From smart home devices to smart grids and smart cities, AI is here to stay, and it will become very pervasive. AI-driven gait-based systems will take the shape of a chair you sit on, shoes you wear, a mat you exercise on. The smart door of your home will open as it recognizes your walk via a continuous SG authentication system, and the phone will lock itself in the hands of the thief as the abnormal gait is recognized. A crowd density algorithm will warn you of high Covid-19 risk when you enter a store.

With the growing demand and promising results, accessibility remains a limitation. ML is a tool, and as such, it should be in the hands of those who need it; early detection of PD in the hands of the clinician, monitoring of treatment in the hands of the caregiver, occupancy sensing in the hands of a family who cares about energy management in their home, indoor tracking and localization in the hands of the police in your town police station, kinematic analysis in the hands of a competing athlete. For ML systems to be more accessible, the technology will need to become easy to understand and implement. It must become less expensive and more scalable. XAI is a trend that will continue. The experts in related fields and the general population will continue to become more AI savvy. In a not-too-distant future, knowing how to use an open-source Python library or write your line or two of code in R will be as common a task as writing an email today. With that will come privacy concerns, ethical concerns, and a need to adjust our laws and regulations as a society.

Author Contributions

EH conducted the systematic review of the papers. I-HK helped with the review of the papers and the writing of the manuscript. ED planned the systematic review, helped with the review of the papers, and contributed to the writing.

Funding

The work has been supported by the California State University Small Faculty Grant.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Abeysekara, L. L., Tran, H., Pathirana, P. N., Horne, M., Power, L., and Szmulewicz, D. (2020). "Multi-domain Data Capture and Cloud Buffered Multimodal Evaluation Platform for Clinical Assessment of Cerebellar Ataxia," in 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 5640-5643.

Rabbi, M., Ali, S., Choudhury, T., and Berke, E. (2011). Passive and In-Situ Assessment of Mental and Physical Well-Being Using Mobile Sensors.
Google Scholar

# data[].attributes.attachment_link.title
A Survey of Human Gait-Based Artificial Intelligence Applications

# data[].attributes.attachment_link.description
We performed an electronic database search of published works from 2012 to mid-2021 that focus on human gait studies and apply machine learning techniques.

# data[].attributes.headline
The application of artificial intelligence andcustom algorithms with inertial wearabledevices for gait analysis and detection ofgait-altering pathologies in adults: A scopingreview of literature

# data[].attributes.subject
The application of artificial intelligence andcustom algorithms with inertial wearabledevices for gait analysis and detection ofgait-altering pathologies in adults: A scopingreview of literature

# data[].attributes.headline
Explainable Artificial Intelligence and Wearable Sensor-Based Gait Analysis to Identify Patients with Osteopenia and Sarcopenia in Daily Life

# data[].attributes.subject
Explainable Artificial Intelligence and Wearable Sensor-Based Gait Analysis to Identify Patients with Osteopenia and Sarcopenia in Daily Life

# data[].attributes.attachment_link.readable_text
Department of Computer Software, University of Science and Technology, Daejeon 34113, Korea

Intelligent Convergence Research Laboratory, Electronics and Telecommunications Research Institute, Daejeon 34129, Korea

Author to whom correspondence should be addressed.

1. Introduction

Osteopenia and sarcopenia can cause various senile disorders and are key factors related to the quality of life in old age [ 1, 2, 3]. Gait is a suitable indicator of musculo-skeletal diseases [ 4]. With the miniaturization of sensors and the development of intelligent monitoring technology, interest in wearable-sensor-based daily health management solutions is increasing [ 4, 5]. Therefore, portable tools and methods that can analyze osteopenia and sarcopenia risks in our daily lives, rather than requiring a specialized hospital setting, can be considered.

Musculoskeletal disorders are increasingly being recognized as conditions that are associated with significant morbidity, mortality, and healthcare costs [ 1, 2]. Osteopenia is a cause of fracture and increases the risk of complications, in addition to pain caused by fractures. Osteoporotic fractures generate costs that reach USD 25 billion, and sarcopenia generates costs of approximately USD 18 billion [ 2, 6]. Patients with sarcopenia have a slow gait, reduced muscular endurance, face difficulty in daily living, and frequently need help from others. Osteoporosis, falls, and fractures can occur easily, whereas the blood and hormonal buffering action of the muscle are moderated, reducing the basal metabolic rate, making chronic diseases unmanageable, and increasing the likelihood of aggravating diabetes and cardiovascular disease [ 3].

Osteoporosis is defined by the World Health Organization (WHO) as a medical condition in which the bone mineral density (BMD) is less than −2.5 standard deviation (SD) below the mean level for young adults, and for osteopenia it is between −2.5 and −1.0 [ 7]. Sarcopenia is defined by the European Working Group on Sarcopenia in Older People (EWGSOP) as the presence of low muscle mass, reduced muscle strength, and physical performance [ 8]. BMD and muscle mass are diagnosed via dual-energy X-ray absorptiometry (DEXA) [ 1], although cannot be measured without expert assistance. Therefore, a system that can easily manage musculoskeletal diseases in daily life is required.

Human gait involves interactions between the musculoskeletal system and the nervous system. Thus, gait analysis is effective in identifying neuromusculoskeletal disorders such as Parkinson's diseases (PD) [ 9, 10], fall risk [ 11, 12], total hip arthroplasty (THA) [ 5, 13], and sarcopenia [ 4]. Traditionally, cameras and force plates have been used as clinical gait assessment tools; however, these tools are used only in large institutions such as university hospitals and are difficult to apply in daily life or complex environments because of their high cost and large space requirements [ 14]. Given the recent miniaturization and increased accuracy of sensor technology, inertial measurement units (IMU) are increasingly being used for gait analysis [ 4].

Gait analysis methods include statistical comparisons of gait parameters obtained from control and target groups and a method of analysis of the classification results of the groups using machine learning. In the analysis of gait for osteopenia, osteoporosis, sarcopenia, and osteosarcopenia conducted by Intriago [ 2], the slowest walking speed was observed in osteosarcopenia: 0.9 m/s in osteopenia-osteoporosis, 0.893 in sarcopenia, and 0.7 in osteosarcopenia. Choi [ 15] investigated the correlation between kinetic gait parameters and femoral BMD of the femoral neck, trochanter, shaft, and total proximal femur. The highest correlation (r = 0.153, p = 0.014) was observed between the walking speed and femoral neck BMD among the older female participants. ElDeeb [ 16] aimed to investigate the gait characteristics of postmenopausal women with low BMD (n = 17) and to determine the predictive parameters of BMD. When the normal BMD group and women with low BMD were compared, the ankle joint showed less push-off (p = 0.000), which seemed to be used to obtain gait stability. Sung [ 17] divided 77 older participants (n = 48 female + 29 male) into normal BMD and low BMD groups using DEXA. The spatial-temporal gait parameters (speed, stride length, and support times) of both groups were subsequently investigated. The support times included those of the initial double support, single support, and terminal double support in the stance phase. The support time was confirmed to have a high ratio of the main foot (the foot mainly used), the stride length was found to be longer on the main foot side than on the other side, and the stride length was positively associated with the single support time on the dominant limb.

Although there are many studies on gait speed for osteopenia and sarcopenia, only a few studies have analyzed gait parameters such as PD, fall risk, and THA. Recently, explainable artificial intelligence (XAI) has received considerable attention as a method to analyze the importance and contribution of parameters. XAI presents predictive results for machine learning in a human-understandable form [ 18]. It is primarily used to enhance the reliability of machine learning results. Low machine learning accuracy results in the misinterpretation of XAI. The XAI technique detects feature importance and explains the influence of features on model decisions [ 19]. Therefore, for the management of osteopenia and sarcopenia in daily life, this study proposes an algorithm for detecting gait parameters and identifying patients based on inertia signals and interpreting the results using XAI.

2. Related Studies

This section describes research related to gait parameter detection techniques, wearable-sensor-based patient identification, and XAI. This information should help with the understanding of the proposed wearable-based gait analysis method for the management of osteopenia and sarcopenia in daily life. This study proposes an algorithm for detecting gait parameters based on inertial signals, identifying patients, and interpreting the results using XAI for gait analysis.

2.1. Gait Parameter

IMU-based gait analysis is used to identify PD, fall risk, THA, and sarcopenia. Gait parameters for analysis based on IMU include spatial-temporal parameters (e.g., step length, stance phase, swing phase, single support, double support, step time, cadence, and speed), kinematic parameters (the rotational angles of the sagittal, coronal, and transverse plane of the pelvis, hip, knee, and ankle), and descriptive statistical parameters (such as the maximum, mean, and standard deviation) of inertial signals for each gait phase. The results of gait analysis using spatial-temporal parameters can be compared with the results of other gait analysis tools such as cameras and force plates. However, the disadvantage is that the motion information acquired by the inertial sensor is reduced, resulting in a low classification result [ 4].

Gait events and phases are detected to extract the gait parameters. Taborri [ 20] classified gait into two to eight phases. Whittle [ 21] classified gait into seven phases, and this is the most widely used classification method. One stride is from a heel strike to the next heel strike. Gait is broadly classified into a stance phase from heel strike (HS) to toe off (TO) and a swing phase from TO to the next HS. The stance phase is classified into the loading response, mid stance, terminal stance, and pre swing phases, and the swing phase is classified into the initial swing, mid swing, and terminal swing phases. The spatial-temporal parameters can be obtained by extracting the HS, TO, opposite HS, opposite TO, and walking distance. HS and TO are detected by the time and frequency signal processing of the inertial signal. Kim [ 22] obtained HS and TO with high accuracy within 0.03 s through time-frequency analysis.

The inertial-sensor-based distance measurement algorithm is widely used as the basis for the distance measurement algorithm in indoor navigation research, and it is difficult to accurately measure the distance using only the inertial sensor [ 23, 24]. Kinematic parameters are obtained by attaching inertial sensors to locations such as the pelvis, hip, knee, and ankle, but they are difficult to use in daily life due to the number of sensors required. Descriptive statistics and frequency analysis have been employed to analyze the signal obtained from the inertial sensor.

2.2. Identifying Patients Based on Inertial Signals

IMU-based gait analysis is used to identify PD, fall risk, THA, and sarcopenia. With gait-parameter-based disease identification, Caramia [ 9] classified PD using linear discriminant analysis (LDA), naïve Bayes (NB), k-nearest neighbor (k-NN), support vector machine (SVM), SVM radial basis function (RBF), decision tree (DT), and the majority of votes. The performance of the machine learning technique-SVM with a nonlinear kernel-was the best. Eskofier [ 10] analyzed the gait of PD using descriptive statistical parameters such as the energy maximum, minimum, mean, variance, skewness, and kurtosis of the signal measured by the inertial sensor and the fast Fourier transform, a frequency analysis method. Howcroft [ 11] predicted the risk of falls using accelerometer data and used temporal (cadence and stride time) and descriptive statistics (maximum, mean, and SD of acceleration). NB, SVM, and neural network (NN) were used as classification methods, and the best single-sensor model was NN. An advantage of deep learning is that it can detect features within the algorithm from a raw signal, although Tunca [ 12] achieved higher accuracy in long short-term memory (LSTM) when certain parameters (e.g., speed, stride length, cycle time, stance time, swing time, clearance, stance ratio, and cadence) were used as the inputs, compared with raw signals.

Teufl [ 5] classified THA patients using stride length, stride time, cadence, speed, hip, and pelvis range of motion (ROM) as features of the SVM, and obtained an accuracy of 97%. Dindorf [ 13] used local interpretable model-agnostic explanations (LIME) to understand the features for identifying THA, and found that the sagittal movement of the hip, knee, and pelvis, as well as transversal movement of the ankle, were particularly important for this specific classification task. Kim [ 4] obtained feature importance using the Shapley additive explanations (SHAP) approach for spatial-temporal parameters and descriptive statistical parameters detected from signals measured by inertial sensors on both feet of ten sarcopenia and control participants. Twenty descriptive statistical parameters of high importance were used as inputs to classification models such as SVM, RF, and multi -layer perceptron (MLP); the highest accuracy (95%) was achieved using the SVM model, as shown in Table 1.

2.3. Explainable Artificial Intelligence

XAI is a method that allows humans to understand the basis of decisions made by artificial intelligence models [ 25]. It is primarily used to enhance the reliability of machine learning results. Low machine learning accuracy results in a misinterpretation of XAI. The XAI technique detects feature importance and explains the influence of features on model decisions [ 19]. LIME and SHAP are often used to explain existing handcraft feature-based classification algorithms, and layer-wise relevance propagation (LRP) and class activation mapping (CAM) are used as algorithms to interpret deep learning.

LIME is effective for tabular data, text, and images. However, it is difficult to set the kernel width in tabular data, and different results are obtained during repeated execution because the sampling process is performed randomly. SHAP was proposed to consider the dependency between features. When the dependence between features is high, the SHAP feature importance is judged to be better than the permutation importance [ 19]. SHAP is based on Shaply values from game theory. The main advantages of the SHAP method are local explanations and consistency in the global model structure. SHAP is used in many machine learning models as a model-agnostic method.

LRP outputs a heatmap to the input image by tracing back the results of the deep learning model. Unlike LIME and SHAP, which interpret the model using the sensitivity analysis technique, LRP [ 25] is a mixture of relevance propagation and decomposition. Relevance propagation is a method for calculating the relevance of the contribution of the hidden layer to the output after the decomposition process. CAM visualizes model decisions by computing a weighted linear summation of the last convolutional feature map. CAM is limited to model architectures where the model must consist of one fully connected (FC) layer with global average pooling (GAP) [ 26]. Grad-CAM describes models without constraints on the model architecture. Gradient-based CAM methods share the problem of shattered gradients, causing noise saliency maps in the intermediate layer. An LRP-based Relevance-CAM has been proposed to solve the gradient problem [ 27].

3. Methods

To analyze the gait of the osteopenia and sarcopenia groups, the patients were identified using machine learning, and the machine learning model was interpreted using XAI. The inertial sensor signals and spatial-temporal and descriptive statistical parameters detected in the proposed algorithm were used as machine learning inputs. By analyzing the model that obtained high-accuracy identification results, the inertial signal and gait parameters of the osteopenia and sarcopenia groups were analyzed. The flowchart of patient identification and gait analysis for osteopenia and sarcopenia is shown in Figure 1.

3.1. Patient Data Collection

Gait signals of 42 women over 65 years of age were obtained to analyze the gait characteristics for osteopenia and sarcopenia. Among the 42 subjects, there were 21 patients with osteopenia and 21 patients without osteopenia. The BMD obtained by measuring DEXA was compared with that of a healthy young person: when the T-score was −1 SD or higher, the data were assigned to the control group; when it was lower than −1, the data were assigned to the osteopenia group. Additionally, 10 sarcopenia and 10 non-sarcopenia patients were selected among the 42 subjects. Sarcopenia was diagnosed using the skeletal muscle mass index (SMI, appendicular skeletal muscle mass in kg/height in m ²) that was less than 5.4 kg/m ² (as obtained through DEXA), whereas the grasp strength was less than 18 kg. The group without sarcopenia included participants with SMI of 5.5 or more and a grasp strength of 19 kg or more. Relevant statistics, including age, height, weight, foot size, Mini-Mental State Examination (MMSE) [ 28], the Mores Fall Scale (MFS) [ 29], SARC-F questionnaire [ 30], Berg Balance Scale (BBS) [ 31] and Timed Up and Go (TUG) scores [ 32], grasp power, T-score for DEXA, and SMI, are shown in Table 2.

The limitations of this study were that the physiological and psychological variables of the participants could not be controlled, the age range of the participants could not be expanded, the study was conducted on women only, and the treadmill gait experiment with fall risk factors for older adults was excluded, and only the preferred speed through walking on flat ground was measured.

All participants wore the same sneaker model and walked the 27 m corridor four times in a straight line. The gait data were acquired from the right and left insoles using IMU, as shown in Figure 2. The IMU settings included an acceleration sensitivity of 8G, a gyro sensitivity of 1000°/s, and a sampling frequency of 100 Hz [ 4].

Additionally, 20 participants measured 9 m gait simultaneously with the clinical standard system and the proposed inertial system to verify the proposed device and algorithm. The clinical system consisted of ten cameras (Vicon, Oxford Metrics, Oxford, UK) and four force plates (Advanced Mechanical Technology, MA, USA). Data analysis was performed using the Vicon Polygon 3.5.2. Ethics approval was obtained from the Chungnam National University Hospital Institutional Review Board before conducting this study (File No: CNUH 2019-06-042).

3.2. Gait Signals and Parameters

Gait is a motion in which both feet alternately repeat the stance and swing phases, and the event points of gait that separate the stance and swing phases are called HS and TO. HS is at the start of the stance phase, and TO is at the start of the swing phase. The gait data obtained from the IMU sensor were 6-axis signals that included the xyz-axis acceleration and angular velocity signal. When the measured sensor data were separated based on HS and normalized to 100 samples, they exhibited periodic characteristics, as shown in Figure 3. The characteristics of the gait signal differ from person to person, and for IMU gait analysis, the spatial-temporal parameter was detected from the gait signal, and the gait signals were expressed as descriptive statistical parameters and analyzed. Additionally, patients were classified using raw data as inputs for deep learning without detecting the parameters; then, the gait signals were analyzed by interpreting the deep learning results.

The spatial-temporal parameters were extracted from the inertial signals using the proposed algorithm [ 4]. Twenty-four spatial-temporal parameters were detected: stance phase time right, stance phase time left, swing phase time right, swing phase time left, stance phase percent right, stance phase percent left, double support first phase time right, double support first phase time left, double support second phase time right, double support second phase time left, single support phase time right, single support phase time left, double support first phase percent right, double support first phase percent left, double support second phase percent right, double support second phase percent left, single support phase percent right, single support phase percent left, stride length right, stride length left, stance phase time SI, swing phase time SI, stance phase percent SI, and cadence. The definitions are summarized in Table 3.

After detecting HS and TO, the opposite HS, opposite TO, cadence, stance phase (time), swing phase (time), single support phase (time), and double support phase (time) could be obtained by arithmetic calculations. Secondary parameters, such as balance of difference between the right and left foot, were also collected through comparative analysis of both feet. Stride was detected through a distance estimation algorithm based on zero-velocity detection (zero-velocity update) using an extended Kalman filter [ 23, 24].

To obtain descriptive statistical parameters, the six-axis gait signal was classified into seven phases, as proposed by Whittle. The detection of HS, TO, heel rise (HR), feet adjacent (FA), and tibia vertical (TV) is required to classify seven phases; it was detected using the method proposed in a previous study [ 4]. Ten descriptive statistical parameters were obtained from signals classified into seven phases, and the descriptive statistical parameters were max., min., SD, AbSum, root-mean-square (RMS), kurtosis, skewness, MMgr, DMM, and Mdif. A total of 840 descriptive statistical parameters (both feet (2) × sensor signal (6) × gait phase (7) × (10 parameters)) were detected.

3.3. Patient Identification

To identify patients using the inertial gait signal and the proposed gait parameters, osteopenia and sarcopenia groups were classified through various models such as RF, XGBoost (Extreme Gradient Boosting), SVM, and deep learning models.

RF is a decision tree ensemble classifier that combines multiple single classifiers to obtain the result of each classification model either through majority vote or weighted average [ 33]. RF lowers the risk of overfitting by using some data and features from the training data. XGBoost is a decision tree ensemble model and improves the performance of the gradient boosting machine in terms of speed. Boosting models increase accuracy by iteratively updating the parameters of the previous classifier to reduce the slope of the loss function, thereby generating a robust classifier [ 33]. SVM is a binary classifier that aims to determine the optimal separation hyperplane that maximizes the margin between two classes. Kernel functions are used to map data to a higher-dimensional space; thus, an SVM can compute nonlinear decision boundaries [ 4].

The representative deep-learning-based models were convolutional neural network (CNN) and LSTM. A CNN is composed of one or more convolutional, pooling, FC, and dense layers. CNNs exhibit high performance in detecting and classifying features in images. Unlike LSTM, which only has forward hidden layers, BiLSTM has both forward and backward hidden layers. Therefore, it learns both before and after information and demonstrates high performance in time-series data. As a CNN backbone, ResNet exhibits excellent classification accuracy [ 34]. ResNet uses skip connections (or short connections) to pass the input from the previous layer to the next layer. This skip connection solves the gradient loss/burst problem, enabling deep neural networks. ResNet uses 18, 34, 50, 101, and 152 layers depending on the depth, and there are structural differences in approximately 50 layers. In particular, ResNet is a popular architecture despite the existence of other models that have improved performance in various fields. Moreover, it is a representative CNN architecture for which many supporting materials are available [ 34, 35].

Transfer learning is applied as a solution to address the difficulty of training a model based on small datasets. In transfer learning, data similar to the target data are learned in advance and a specific layer is frozen, such that only the layer which is not frozen when learning the target data is learned [ 36]. Specific data characteristics can be overfitted, because patient identification is a binary classification. Therefore, person identification is pre-trained because high-resolution features can be detected by comparing and analyzing the gaits of multiple people.

3.4. Gait Analysis

The gait signals and parameters were analyzed using statistical methods and XAI techniques that interpret machine learning results. The independent t-test was used as a statistical method to compare the spatial-temporal parameters and descriptive statistical parameters. To improve the reliability of the machine-learning-based analysis method, a higher accuracy should first be obtained. Therefore, the osteopenia and sarcopenia groups were classified through various models, such as RF, XGBoost, SVM, and deep learning models. The CNN and LSTM models were used as the deep learning models.

Spatial-temporal parameters and 100 descriptive statistical parameters with low p-values of the t-test were used as inputs for RF, XGBoost, and SVM. The following RF parameters were used: number of trees = 50, max_depth = 30, and number of features = square root of the gait parameters. The XGBoost parameters were booster = gbtree, objective = binary:logistic, eta = 0.018, max_depth = 15, gamma = 0.009, subsample = 0.98, and colsample_bytree = 0.86. SVM explored the linear and RBF kernels, and the parameters were gamma = 1.0 and C = 5.0.

The 12 axes of acceleration and angular velocity signals obtained from both feet were applied to the deep learning models. We proposed a low-layer-based CNN and BiLSTM model and applied ResNet50. As the input of the deep learning model, a stride based on HS was detected and normalized to 100 samples using spline interpolation because the signal was collected at 100 Hz [ 4]. ResNet50 is reduced in size by pooling as the layers progress. The ResNet50 backbone cannot be used with an input of shape 12, and removing pooling lowers the accuracy. Therefore, the input shape (36,100) was generated by amplifying the signal of 12 axes threefold because the input size of ResNet50 must be 12 or more, and the kernel size was 3. The layers of each model are shown in Table 4. The parameters of the deep learning model were as follows: learning rate = 0.0005, training epoch = 100, batch size = 16, loss = CrossEntropyLoss, optimizer = Adam, and activation function = Rectified Linear Unit.

XGBoost can calculate the built-in importance (Gini importance) and permutation importance using the learned model. Permutation importance measures the increase or decrease in prediction error compared with the original data when the feature data are transformed [ 19]. Permutation importance does not consider the correlation between features; therefore, SHAP was proposed as a method to consider the dependency between features. In particular, the SHAP feature importance is considered to be better than the permutation importance because gait parameters have a high dependence on the features. The Gini, permutation, and SHAP importance of the spatial-temporal and descriptive statistical parameters were calculated to obtain important parameters of osteopenia and sarcopenia, and the results of deep learning were analyzed using LRP, Grad-CAM, and Relevance-CAM.

4. Results

4.1. Patient Identification

The identification results of 21 osteopenia and 21 non-osteopenia subjects showed the highest accuracy in SVM when 24 spatial-temporal parameters were used as inputs, but showed an accuracy of less than 65%. The descriptive statistics parameter obtained the highest accuracy of 68.45% in XGBoost by using 100 parameters with a low p-value as an input, as a result of an independent t-test. For training and testing, 21 cross-validations were performed on 21 subjects, and the average was obtained. Using an inertial sensor as an input for deep learning, ResNet showed the highest accuracy among CNN, BiLSTM, and ResNet. The results of applying transfer learning to the ResNet model showed lower accuracy than when no transfer learning was applied. However, when performing transfer learning, it was shown that the accuracy increased when features were extracted, including the test subject. This implies that ResNet was pre-trained for human identification using the data of 42 patients, and the patient identification was cross-validated for 21 patients. The osteopenia group obtained the highest recognition result in the transfer learning ResNet.

The identification results of 10 sarcopenia and 10 non-sarcopenia cases were over 70% accurate in terms of the spatial-temporal parameters, and the accuracy in case of sarcopenia was better than that in osteopenia. When the descriptive statistics parameter was used as the RF input, the highest accuracy was obtained, and the deep learning method of the inertial sensor input did not yield satisfactory identification results. Therefore, analysis of the results of XAI based on parameters is more reliable than the analysis of results based on deep learning. The patient identification results of machine learning are presented in Table 5 and Table 6.

4.2. Importance of Descriptive Statistical Parameter

The order of Gini, permutation, and SHAP importance was obtained for the descriptive statistical parameters of osteopenia and sarcopenia. When using highly important parameters such as RF, XGBoost, and SVM inputs, SHAP obtained the highest identification rate; however, when using the inner 20 important parameters as inputs, more identification results than the 100 descriptive statistical parameters were obtained. Table 7 and Table 8 showed the classification results as the number of parameters increased, and the average accuracy was obtained by performing 21 cross-validations for osteopenia and 10 cross-validations for sarcopenia. According to the result of each cross-validation, SHAP-based feature importance has different values. For example, osteopenia was trained with 40 datasets (20 osteopenia datasets and 20 non-osteopenia datasets) and tested with two datasets (1 osteopenia dataset and 1 non-osteopenia dataset) during 21 cross-validations. As a result of the training, the Shapley values were obtained based on the training data of 40 people, and the Shapley values were obtained in different orders. Table 9 shows the average results for 20 high-order Shapley values generated during 21 cross-validations. In osteopenia, the Shapley value is relatively high in the upper parameter and less than 0.1 from the 10th parameter. In sarcopenia, the difference in the Shapley value between the parameters is small. The parameter numbers of descriptive statistical parameters are shown in Table 10. The results of learning RF, XGBoost, and SVM with 20 parameters with high importance in Table 9 are shown in Table 11. Osteopenia obtained an accuracy of 88.69% in XGBoost using the top 4 parameters as inputs, and sarcopenia obtained an accuracy of 93.75% in RF using the top 18 parameters as inputs.

4.3. Gait Analysis

In the spatial-temporal parameters of the osteopenia group, the stance phase percentage decreased, double support percentage (time) decreased, and single support percentage increased. The sarcopenia group showed an increase in the value of the SI parameter compared with the non-sarcopenia group, implying that the difference between both feet was large. Except for the SI parameter, the p-value did not have a statistical significance of less than 0.001. Table 12 shows the mean and Shapley values of the spatial-temporal parameters; * indicates that the p-value is less than 0.025, and ** indicates that the p-value is less than 0.001.

From the result of the SHAP plot of osteopenia, as the value of the single support phase percent left (parameter 18) increased, the risk of osteopenia increased, as indicated by the positive SHAP value. As the value decreased, the risk also decreased, with the SHAP value being negative. When the single support phase percent left value increased, the risk increased linearly, and the osteopenia risk was low, at 39 or lower, and the risk increased at 42 or higher. A low double support first phase percent (parameter 13 and 14) increased the risk of osteopenia, with a decreased risk above 9 and an increased risk below 9. A low value of double support first phase time left (parameter 8) increased the risk, a high value decreased the risk of osteopenia, and a double support first phase time left lower than 0.075 led to an increase in risk. A low value of the stance phase percent right (parameter 5) increased the risk.

As a result of the SHAP plot of sarcopenia, the risk of sarcopenia increased when the double support first phase time left (parameter 8) had a very low value (less than 0.07). Stance phase percent left (parameter 6) increased the risk above 60 and decreased below 60, but did not show linearity. The risk increased when the value of the stance phase time SI (parameter 21) increased, and the risk was high at 0.35 or higher, although it was low at less than 0.35. SHAP plots of the spatial-temporal parameters of osteopenia and sarcopenia are shown in Figure 4.

The parameter with the highest SHAP value within the descriptive statistical parameters of osteopenia is the skewness of the x-axis of the accelerometer in the initial swing phase (parameter number 247). Initial swing refers to the FA after TO. When the skewness is negative, the probability density function has a long tail on the left side, and the data, including the median, are more distributed on the right side. When the skewness is positive, there is a long tail on the right side of the probability density function, indicating that the data are more distributed on the left side. Skewness has a positive value when the mean is smaller than the median, negative when the mean is larger, and has a larger value as the difference between the median and the mean becomes larger. When there is a negative value, the right part of Figure 5 a has a long tail, but the average decreases and the skewness value becomes small or negative.

The inertial signals and SHAP dependence plots of the descriptive statistical parameters 247 and 114 of osteopenia cases are shown in Figure 5. The blue signal represents osteopenia, and the red signal represents non-osteopenia. A skewness of 0.5 or higher shows a low risk of osteopenia, whereas a negative value shows an increased risk. The absolute sum of gyro z values in the mid stance (parameter number 114) decreased in osteopenia. This implies that there is no rotation of the z-axis in the mid stance. When the absolute sum of values was 2.6 or higher, the risk decreased, and when the absolute sum of values was 1.89 or lower, the risk increased, as shown in Table 13.

The inertial signals and SHAP dependence plots of the descriptive statistical parameters 430 and 524 of sarcopenia are shown in Figure 6. Here, the blue signal represents sarcopenia, whereas the red signal represents non-sarcopenia. The maximum difference between two successive values of accelerometer x in the loading response (parameter number 430) was lower in the sarcopenia group than in the non-sarcopenia group. When the maximum value was less than 2.74, the risk of sarcopenia increased; however, when the maximum value was 3.79 or more, the risk of sarcopenia decreased. In the sarcopenia group, the change in the acceleration was smooth. The absolute sum of gyro y values in the mid stance (parameter number 524) increased in the sarcopenia group. As the absolute sum value increased, the risk of sarcopenia increased.

The output for layer2, when the deep-learning-based XAI technique, LRP, Grad-CAM, and Relevance-CAM were applied to ResNet50, is shown in Figure 7. In ResNet50, the CAM technique shows low resolution in small-sized images because the feature map is reduced in layer2. It is difficult to interpret the CAM results for ResNet50 with input sizes of 100 horizontal and 36 vertical. Therefore, it is desirable to interpret ResNet results as LRP. Figure 8 shows the analysis results of LRP for the ResNet of osteopenia and sarcopenia. The LRP attention map of the osteopenia group had high values in 64~67 samples of the acceleration x-axis of the right foot. Its position is the section where the acceleration value rises after TO, and it is the same as the position in Figure 5a, the parameter-based SHAP result. Osteopenia pays attention to changes in acceleration after TO in SHAP and LRP. The LRP attention map of sarcopenia group has a high value at 99~100 positions of the right acceleration x. The position is the section where HS occurs, and it is the same section as Figure 6 a. The result of paying attention to the various sections of the acceleration left is similar to having a high identification result when used as various parameter inputs in SHAP. The sarcopenia group pays attention to the HS section and the various sections of the signal.

5. Discussion

The objective of this study was to propose and evaluate a method that can utilize the gait parameters obtained from a wearable device with an inertial sensor in the health management of patients suffering from sarcopenia and osteopenia in daily life.

In the proposed method, the patient was identified using gait phase description-based descriptive statistical parameters as the handcrafted feature-based machine learning input and the original signal of the inertial sensor as the input for the deep learning algorithm. For gait analysis, the identification results were analyzed using XAI tools, such as SHAP and LRP. To verify the proposed gait analysis method, the results of functional tests and questionnaires obtained at the hospital for participants, the results using the existing gait parameters, and the results of the proposed method are discussed.

To identify osteopenia and sarcopenia, a decrease in walking speed and poor body balance has been reported in previous studies. It has been reported that patients with sarcopenia have a slower walking speed than those with osteopenia. The result of the 3 m TUG was 11.71 s in the sarcopenia group and 10.96 s in the osteopenia group, indicating that the walking speed was slower in the patients with sarcopenia. Except for TUG, statistical significance was not obtained for the MMSE, MFS, SARC-F questionnaire, or BBS.

In gait analysis using inertial sensors, spatial-temporal parameters have traditionally been used as tools to conveniently identify diseases such as Faller, PD, and THA in everyday life. In this study, to identify patients with osteopenia and sarcopenia, 24 spatial-temporal parameters used for conventional disease identification were detected, and descriptive statistical parameters were detected to analyze the inertial sensor signals according to the gait phase. Statistical significance was obtained for the stance phase, double support phase, and single support phase percent in osteopenia, and SI in the stance and swing phase in sarcopenia.

With gait analysis using XAI, SHAP demonstrates the importance of parameters and the positive/negative contribution of parameters to the classification results of machine learning. To apply SHAP to machine learning classifiers, it is necessary to obtain a high machine learning accuracy. Good classification results were obtained for osteopenia in XGBoost and sarcopenia in RF. It has been reported that XGBoost has the advantage of being the most accurate among tree-based classifiers, and RF has a strong advantage in terms of overfitting. Comparing various machine learning results, osteopenia showed an accuracy of lower than 70% and sarcopenia showed overfitting. Overfitting in sarcopenia was inferred from the results of the deep learning model.

From the SHAP results of spatial-temporal parameters, single support phase percent left, double support first phase percent left, double support first phase percent right, double support first phase time left, and stance phase percent right were highly important for the osteopenia group. In the sarcopenia group, double support first phase time left, stance phase percent left, stance phase time SI, double support first phase percent left, and stance phase percent right were found to be of high importance. The important parameters obtained similar results to the statistical analysis; in osteopenia, the phase had a high contribution, whereas in sarcopenia, SI had a high contribution. Double support first phase time left and stance phase percent right showed a high contribution in both groups. The double support first phase time decreased in the osteopenia group compared with that in the sarcopenia group, and the stance phase percentage decreased. An increase in the double support first phase time and an increase in the stance phase percentage indicate a decrease in the walking speed. The time-related parameters were lower in the osteopenia group than in the non-osteopenia group; therefore, it is difficult to identify the osteopenia group based on walking speed. Sarcopenia significantly contributed to the reductions in walking speed and balance parameters, as in the previous study results [ 1, 2, 3].

The accuracy of identification of osteopenia patients with spatial-temporal parameters, which is the existing gait analysis parameter, was lower than 70%; thus, it was difficult to analyze the results in SHAP. The inertial sensor had a high temporal resolution; therefore, it was possible to obtain differences between groups by segmenting and analyzing the gait. As a result of SHAP for 840 descriptive statistical parameters, a high contribution from the skewness of the x-axis of the accelerometer in the initial swing phase (247) and the absolute sum of values of gyro z in the mid stance (114) were observed for osteopenia. For sarcopenia, the maximum difference between two successive values of accelerometer x in loading response (430) and the absolute sum of values of gyro y in the mid stance (524) showed a high contribution.

The descriptive statistical parameter 247 for osteopenia was smaller than that for sarcopenia, and the low skewness of osteopenia is due to the rapid occurrence of the maximum value of the acceleration x-axis after TO and a large negative value. This result is related to the increase in the swing phase time. The descriptive statistical parameter 430 represents the change in acceleration, and its value of osteopenia is larger than that of the sarcopenia group, implying that the gait speed of the sarcopenia group is slow because the acceleration x is the walking direction. Parameters 114 and 524 are the absolute sum of the gyro values in the mid stance. The absolute sum decreases in osteopenia indicating less foot movement in the mid stance, as shown in Table 14.

The interpretation of results for LRP-based deep learning was similar to the results of descriptive statistical parameter analysis based on SHAP. However, the reliability was low due to the change in the attention map of the inertial sensor according to the learning results of deep learning as a result of repeated experiments and low identification accuracy. If high-accuracy identification results are obtained, it is expected that the inertial signal characteristics of osteopenia and sarcopenia can be obtained using deep learning.

Functional tests and questionnaires conducted in the hospital were not statistically significant, except for TUG in the sarcopenia group. Spatial-temporal parameters, which have previously been used as gait parameters, showed statistical significance in the sarcopenia group and the osteopenia group, but showed a low identification accuracy of 63% in the osteopenia group. The proposed descriptive statistical parameters obtained an accuracy of 76% or more, and the descriptive statistical parameters attributed similar meanings to the results of the spatial-temporal parameters, had high statistical significance, and can be used as a new clinical tool because the difference in parameter values between the osteopenia and sarcopenia groups is remarkable. Descriptive statistical parameters can be used as useful tools for patient identification and risk detection.

6. Conclusions

The inertial-sensor-based gait signal was acquired and analyzed for patients with osteopenia and sarcopenia. Spatial-temporal parameters used in conventional clinical evaluation and diagnosis are effective tools for understanding gait. However, they have poor temporal resolution and do not include the function of kinematic signals during the gait cycle. Therefore, the inertial sensor data can obtain descriptive statistical parameters for each gait phase.

For analyzing the patients and control groups, parameters can be statistically analyzed or analyzed through machine-learning-based XAI. To apply XAI, high-accuracy machine learning is required; thus, useful parameters obtained from parameter analysis are used to increase the accuracy of machine learning. Therefore, parameter interpretation is important for patient identification and risk estimation. As a machine learning algorithm, XGBoost for osteopenia and RF for sarcopenia showed high performance, whereas for deep learning, ResNet50, which transfer-learned a human identification model, achieved high accuracy. For the analysis of gait parameters, SHAP was applied to the machine learning model to detect the importance and contribution of the parameters. Unlike Gini and permutation importance, SHAP has advantages of lowering the importance of a parameter when there are similar characteristics between the high-importance parameters. When deep learning identifies patient, the attention map of the inertial sensor signal was analyzed using LRP.

Analyzing the signal of the inertial sensor through XAI, we can diagnose and manage osteopenia and sarcopenia in daily life using a smart insole rather than an expensive clinical tool because the inertial sensor signal contains abundant information on gait. Although the number of participants in this study was extremely small to enable fully understanding the gait characteristics of osteopenia and sarcopenia, the proposed method is effective in analyzing osteopenia and sarcopenia. Therefore, in future studies, additional clinical evaluations will be performed to obtain and analyze many patients and segment data according to sex, age, and dominant leg.

Author Contributions

Conceptualization, J.-K.K., M.-N.B., K.L., J.-C.K. and S.G.H.; Data curation, J.-K.K. and M.-N.B.; Formal analysis, J.-K.K.; Funding acquisition, K.L. and J.-C.K.; Investigation, J.-K.K.; Methodology, J.-K.K. and S.G.H.; Project administration, M.-N.B., K.L. and J.-C.K.; Software, J.-K.K.; Supervision, K.L. and J.-C.K.; Validation, J.-K.K. and S.G.H.; Visualization, J.-K.K.; Writing-original draft, J.-K.K.; Writing-review and editing, J.-K.K., M.-N.B. and S.G.H. All authors have read and agreed to the published version of the manuscript.

Funding

This study was supported by a National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2021M3I2A1077405) and by a grant (22DRMS-B146826-05) from the Development of Customized Contents Provision Technology for Realistic Disaster Management Based on Spatial Information Program funded by the Ministry of the Interior and Safety of the Korean government.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki, and approved by the Institutional Review Board of Chungnam National University Hospital Institutional (File No: CNUH 2019-06-042).

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

The data are not publicly available due to company security policy and personal protection of subjects. Data are available from the authors upon reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

References

1. Intriago, M.; Maldonado, G.; Guerrero, R.; Messina, O.D.; Rios, C. Bone Mass Loss and Sarcopenia in Ecuadorian Patients. J. Aging Res. 2020, 2020, 1072675. [Google Scholar] [CrossRef] [PubMed][Green Version]

2. Eskofier, B.M.; Lee, S.I.; Daneault, J.F.; Golabchi, F.N.; Ferreira-Carvalho, G.; Vergara-Diaz, G.; Sapienza, S.; Costante, G.; Klucken, J.; Kautz, T.; et al. Recent machine learning advancements in sensor-based mobility analysis: Deep learning for Parkinson's disease assessment. In Proceedings of the 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), Orlando, FL, USA, 16-20 August 2016; pp. 655-658. [Google Scholar]

3. Whittle, M.W. Gait Analysis: An Introduction; Butterworth-Heinemann: Oxford, UK, 2014. [Google Scholar]

4. Nilsson, J.-O.; Gupta, A.K.; Handel, P. Foot-mounted inertial navigation made easy. In Proceedings of the 2014 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Busan, Korea, 27-30 October 2014; pp. 24-29. [Google Scholar]

5. Skog, I.; Nilsson, J.-O.; Handel, P. Pedestrian tracking using an IMU array. In Proceedings of the 2014 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT), Bangalore, India, 6-7 January 2014; pp. 1-4. [Google Scholar]

6. Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.-R.; Samek, W. On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation. PLoS ONE 2015, 10, e0130140. [Google Scholar] [CrossRef] [PubMed][Green Version]

7. Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; Torralba, A. Learning deep features for discriminative localization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 27-30 June 2016; pp. 2921-2929. [Google Scholar]

8. Lee, J.R.; Kim, S.; Park, I.; Eo, T.; Hwang, D. Relevance-CAM: Your Model Already Knows Where to Look. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, 20-25 June 2021; pp. 14939-14948. [Google Scholar]

9. Kang, Y.; Na, D.L.; Hahn, S. A validity study on the Korean Mini-Mental State Examination (K-MMSE) in dementia patients. J. Korean Neurol. Assoc. 1997, 15, 300-308. [Google Scholar]

10. Podsiadlo, D.; Richardson, S. The timed "Up & Go": A test of basic functional mobility for frail elderly persons. J. Am. Geriatr. Soc. 1991, 39, 142-148. [Google Scholar]

11. Duan, H.; Zhao, Y.; Chen, K.; Shao, D.; Lin, D.; Dai, B. Revisiting Skeleton-based Action Recognition. arXiv 2021, arXiv:2104.13586. [Google Scholar]

Publisher's Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

# data[].attributes.attachment_link.title
Explainable Artificial Intelligence and Wearable Sensor-Based Gait Analysis to Identify Patients with Osteopenia and Sarcopenia in Daily Life

# data[].attributes.attachment_link.description
Osteopenia and sarcopenia can cause various senile diseases and are key factors related to the quality of life in old age. There is need for portable tools and methods that can analyze osteopenia and sarcopenia risks during daily life, rather than requiring a specialized hospital setting.

# data[].attributes.headline
Concurrent validity of artificial intelligence-based markerless motion capture for over-ground gait analysis: A study of spatiotemporal parameters

# data[].attributes.subject
Concurrent validity of artificial intelligence-based markerless motion capture for over-ground gait analysis: A study of spatiotemporal parameters

# data[].attributes.attachment_link.readable_text
Abstract

is used in research and clinical environments; yet several limitations exist in current methodologies. Markerless systems, utilizing high-speed video and artificial intelligence, eliminate most limitations encountered in marker-, depth-, or inertial sensor-based systems; however, further development is needed to improve their utility and accessibility in practice. Spatiotemporal parameters from 22 Gait analysis young adults were estimated during over-ground gait. Nine parameters were calculated using events determined from force plate information combined with foot segment tracking and from motion of the foot relative to the sacrum using marker-based and markerless tracking. Two-way mixed effects, single measurement, absolute agreement and relative consistency interclass correlation coefficients, Bland-Altman bias and limits of agreement, and Lin's concordance correlations were used to examine the validity of parameters from markerless tracking compared to parameters calculated from gait event methods using force plates and marker-based tracking. Gait speed, stride length, step length, cycle time, and step time from the markerless system all showed strong agreement with the force plate method. Other markerless-determined parameters were not as accurate. Differences in stride width are attributable to inconsistencies in foot segment definitions between models; while differences in stance time, swing time, and double limb support time were influenced by gait event methods. Mean differences in gait parameters were smaller than meaningful clinical differences in Parkinson's disease patients and within ranges of reference values for elderly subjects. Further studies are needed to determine the validity across other patient groups, but results support the continued development of markerless systems for over-ground gait analysis.

Introduction

Gait analysis has multiple applications including assessment of disease state for patients with neurological disorders (Alharthi et al., 2020, Bringas et al., 2019, Lu et al., 2020) and monitoring of rehabilitation progress (Buckley et al., 2020, Farah et al., 2019). Optoelectronic motion capture (mocap) systems typically used to perform gait analysis remain an unrealistic option in clinical and outdoor settings given their high cost and the infrastructure requirements for successful operation. Additionally, optoelectronic mocap systems rely on the use reflective markers, which may interfere with subjects' natural movements and produce data errors due to soft tissue artifacts (Moltedo and Sakka, 2014).

Several alternatives to marker-based mocap systems have been proposed in the literature, including depth sensors (Eltoukhy et al., 2017a, Eltoukhy et al., 2017b), RGB-depth sensors (Antico et al., 2021, Balta et al., 2020, Pantzar-Castilla et al., 2018, Ripic et al., 2022), and wearable sensors such as inertial measurement units (IMUs) (Park and Yoon, 2021, Kobsar et al., 2020, Vienne et al., 2017); however, these solutions have significant limitations. For instance, depth sensors tend to have inherent limitations, such as limited range of view, that diminish accuracy (Kumarasiri et al., 2018, Guffanti et al., 2020, Yeung et al., 2021). Additionally, IMUs suffer from integration drift (Razavian et al., 2019), which accumulates errors in the kinematic variables used unless treated with drift correction algorithms as in Takeda et al., 2014.

There is a clear need for new, accessible technologies to facilitate in-clinic gait assessment. Artificial intelligence (AI) has been used in multiple supporting fields, such as pattern recognition (Souza at al., 2018), and image processing (Leightley, et al., 2017), classification (Mu et al., 2010), and prediction (Xiong, et al. 2019). Recently, deep learning approaches have become popular due as they can handle high-dimensional, temporal, and complex data; (Khera and Kumar, 2020) while providing accurate results, particularly for gait analysis (Horst et al., 2019, Zou et al., 2020, Hannink et al., 2017). Deep learning is based on the concept of artificial neural networks, which learn data representations automatically. Convolutional neural networks (CNN) are commonly applied to analyze visual imagery using a convolution technique to perform image classification and object recognition tasks (Sudharshan and Raj, 2018, Hayat et al., 2018, Li et al., 2014). Markerless pose estimation architectures use backbone CNN networks to extract features of an image and decoder networks to determine two-dimensional (2D) keypoint body part locations in the image space (reviewed in Mathis et al., 2020). These are applied to sequential video frames collected from a calibrated multi-view system of synchronous cameras' views around the mocap space. Such non-invasive, AI-based systems, which rely on video data from multiple camera views, eliminate the use of reflective markers and can potentially provide clinicians with quantitative gait assessments in diverse clinical environments by eliminating several time and methodological limitations such as the need for specialized clothing, lengthy subject preparation, specialized lighting, time intensive marker labeling which are hallmarks of current marker-based approaches. Nonetheless, before AI-based markerless mocap systems can be utilized for gait analysis in clinical settings, their validity and reliability should be evaluated.

Therefore, this study was designed to establish the validity of the KinaTrax markerless mocap system for estimating over-ground gait spatiotemporal parameter using only kinematic trajectories. We compared the spatiotemporal parameters obtained from force plate (FP)- and kinematically-determined gait events from markerless and marker-based tracking. Spatiotemporal parameters using gait events from both tracking methods were compared to the gold-standard FP method. Additionally, spatiotemporal parameters obtained from both markerless and marker-based tracking were compared to determine the effect of the kinematic method. We hypothesized a nearly perfect concordance and excellent agreement and consistency for all spatiotemporal parameters among the three systems. Additionally, mean differences were expected to be minimal and attributable to gait event determination, rather than the tracking source used to calculate parameters.

Section snippets

Participants

A convenience sample of twenty-two (M/F: 10/12; Age: 22.72 ± 3.49 years; Height: 1.69 ± 0.11 m; Mass: 69.90 ± 11.97 kg) healthy adults was enrolled in this validation study. Subjects were recreationally-active and free from any neuromuscular or musculoskeletal injuries that could affect their gait.

Experimental setup and procedure

The University's Institutional Review Board approved the protocol and all participants provided informed written consent. The subject's anthropometric measurements, height and mass, were measured

Results

Spatiotemporal parameters (means ± SD) for the FP, marker-based and markerless analyses are presented in Table 1. ICCs between-methods are presented in Table 2 and Bland-Altman analyses are presented in Table 3.

Discussion

The aim of this study was to establish the validity of the KinaTrax markerless mocap system for spatiotemporal parameter estimation during overground gait in young healthy adults. Additionally, we sought to examine differences in spatiotemporal variables among the markerless, marker-based, and gold-standard FP methods for determining gait events. Results showed excellent agreement and consistency and moderate to substantial for gait speed, stride length, and step length when comparing the

Conclusions

The aim of the present study was to establish the validity of the KinaTrax markerless mocap system to quantify spatiotemporal parameters during over-ground gait in young, healthy adults. Results suggest that gait speed and selected spatial and temporal variables are valid measures in these subjects. Additionally, biases fall within meaningful clinical differences in PD patients and within standard deviations for reference values in adults 65 years and older. Therefore, the markerless method

CRediT authorship contribution statement

Zachary Ripic: Writing - original draft, Methodology, Data curation, Conceptualization. Joseph F. Signorile: Writing - review & editing, Data curation. Christopher Kuenze: Writing - review & editing, Data curation. Moataz Eltoukhy: Writing - original draft, Validation, Supervision, Methodology, Data curation, Conceptualization.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

A two-dimensional clinical gait analysis protocol based on markerless recordings from a single RGB-Depth camera
Paper Presented at the 2020 IEEE International Symposium on Medical Measurements and Applications (MeMeA)

Beauchet, O., Allali, G., Sekhon, H., Verghese, J., Guilain, S., Steinmetz, J.-P., . . . Helbostad, J.L., 2017....

Bringas, S., Salomon, S., Duque, R., Monta ́ na, J. L., & Lage, C., 2019. A convolutional neural network-based method...

Buckley, C., Mico-Amigo, M. E., Dunne-Willows, M., Godfrey, A., Hickey, A., Lord, S., Rochester, L., Del Din, S.,...

V. Cicchetti
Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instructions in psychology

Dubois, A., Bresciani, J.P., 2018. Validation of an ambient system for the measurement of gait parameters. J....

Hass, C.J., Bishop, M., Moscovich, M., Stegemöller, E.L., Skinner, J., Malaty, I.A.,...& Okun, M.S., 2014. Defining the...

View full text

# data[].attributes.attachment_link.title
Concurrent validity of artificial intelligence-based markerless motion capture for over-ground gait analysis: A study of spatiotemporal parameters

# data[].attributes.attachment_link.description
is used in research and clinical environments; yet several limitations exist in current methodologies. Markerless systems, utilizing high-speed video and artificial intelligence, eliminate most limitations encountered in marker-, depth-, or inertial sensor-based systems; however, further development is needed to improve their utility and accessibility in practice.

# data[].attributes.headline
Proof of Concept in Artificial-Intelligence-Based Wearable Gait Monitoring for Parkinson’s Disease Management Optimization

# data[].attributes.subject
Proof of Concept in Artificial-Intelligence-Based Wearable Gait Monitoring for Parkinson’s Disease Management Optimization