Embed Size (px)
Citation preview
Gait analysis using computer vision for the early detection of elderly syndromes.
A formal proposal
Mario Nieto Hidalgo
Departamento de Tecnologıa Informatica y Computacion
Escuela Politecnica Superior
Gait analysis using computer vision for
the early detection of elderly syndromes.
A formal proposal.
Mario Nieto Hidalgo
Tesis presentada para aspirar al grado de
DOCTOR O DOCTORA POR LA UNIVERSIDAD DE ALICANTE
DOCTORADO EN TECNOLOGIAS DE LA SOCIEDAD DE LA
INFORMACION
Dirigida por
Dr. Juan Manuel Garcıa Chamizo
December 18, 2016
Esta tesis se ha desarrollado como parte del proyecto FRASE (TIN2013-47152-C3-2-R)
financiado por el Ministerio de Economıa y Competitividad.
“Science is a cooperative enterprise, spanning the generations. It’s the passing of a torch
from teacher, to student, to teacher. A community of minds reaching back to antiquity
and forward to the stars”
Neil deGrasse Tyson
Resumen en castellano
Introduccion
El grado de doctor es un hito que normalmente se consigue haciendo una tesis a partir
de una conjetura. Esta tecnica proporciona la evidencia de que el autor debe obtener el
grado de doctor porque el proceso de hacer una tesis doctoral es una prueba empırica
de las habilidades del autor para crear conocimiento.
Este trabajo trata de llevar al autor a un estado de consciencia y actitud que mantiene
un compromiso entre una contribucion cientıfica directa, cerca del desarrollo, y una
contribucion abstracta, cerca del ambito de los conceptos y metodos.
Por esa razon, esta propuesta aborda muchos aspectos diferentes, desde los puramente
epistemologicos orientados a clarificar las claves subyacentes bajo los criterios y procedi-
mientos, hasta los mas puramente instrumentales para obtener una instancia particular
de solucion a un problema dado. Por tanto, el objetivo final de obtener el grado de doctor
se concretiza en dos objetivos subsidiarios. Por una parte, avanzar en la justificacion de
los metodos de resolucion de problemas. Esto esta orientado a reducir el empirismo y ar-
bitrariedad, que comunmente rigen los procesos de toma de decisiones, e incrementar el
rigor procedimental, el cual deberıa mejorar la calidad de las soluciones proporcionadas.
Por otra parte, producir un avance inmediato en el estado del conocimiento, particu-
larmente, en los problemas de aumentar la calidad de vida de las personas mayores a
traves de sistemas de vision por computador.
Esta investigacion ha sido desarrollada como parte del proyecto FRASE (TIN2013-
47152-C3-2-R) financiado por el Ministerio de Economıa y Competitividad y, por esa
razon, el problema que trata es la caracterizacion de la marcha de personas mayores a
traves de tecnicas de vision artificial con el fin de obtener un diagnostico precoz de los
sındromes de fragilidad y senilidad.
Ademas de proponer una solucion, esta tesis proporciona respuesta a la relevancia de
las decisiones de diseno que llevan a la solucin, lo cual ha llevado a investigar la causa
que justifique cada decision de diseno y por tanto, abordar los temas de la relevancia
de resolver en el ambito de los modelos, ası como abordar aspectos metodologicos para
avanzar de los generales a los particulares.
La formalizacion del paradigma orientado a modelo se consigue estableciendo las clases
de decisiones de diseno involucradas en cualquier proceso de resolucion de problemas.
Establecemos un orden de las clases proporcionando un mecanismo para determinar
que decisiones de diseno deben realizarse primero. Tambien proponemos una forma de
seleccionar el modelo en el cual enmarcar la solucion y un metodo para resolver el sub-
problema especıfico para proponer objetivos y requerimientos aplicando la tecnica divide
et vinces hasta que se obtengan subproblemas suficientemente simples. Esto proporciona
una especificacion estructural que contiene los modulos que componen la solucion y un
grafo organizacional que establece las relaciones entre dichos modulos.
Aplicando la metodologıa propuesta previamente, hemos disenado y desarrollado un sis-
tema basado en vision artificial para el analisis de la marcha orientado a la deteccion
de los sındromes de fragilidad y senilidad. La metodologıa proporciona la especificacion
estructural, es decir, los diferentes modulos que componen la solucion. Este trabajo pro-
pone una nueva tecnica basada en vision artificial que es usada para medir parametros
de la marcha y caracterizar el andar humano utilizando unicamente camaras de video.
Los esfuerzos de desarrollo se centran en algoritmos de clasificacion e inferencia especia-
lizados en el modelado de la marcha humana. Esta propuesta permitira disenar sistemas
inteligentes de bajo coste. El uso de los sensores de imagen instalados en las camaras
digitales es tambien deseable desde el punto de vista del consumo energetico ya que se
basan en la luz visible, es decir, la condicion de obtener informacion de la escena sin
anadir energıa es un requerimiento esencial. Desde el punto de vista de la escalabilidad,
al no haber energıa anadida, todos los riesgos de saturacion e interferencias debido a la
adicion de mas sensores quedan descartados.
La marcha humana puede ser visualizada por camaras digitales pero la marcha engloba
una complejidad no observable facilmente, de hecho, esta plagada de dificultades pro-
venientes de la variabilidad de poses y acciones que caracterizan la marcha. Una forma
de contrarrestar esto es identificar las principales caracterısticas mediante el proceso de
segmentacion, seguimiento y analisis de la pose utilizando algoritmos de bajo nivel y
evaluando esas caracterısticas con algoritmos de clasificacion de alto nivel.
Como se ha mencionado anteriormente, nuestro objetivo es proporcionar un marco con-
ceptual para realizar sistematicamente operaciones de diseno y ayudar a tomar decisiones
para resolver problemas y crear inventos. Este marco podrıa proporcionar criterios para
desarrollar procedimientos sistematicos para tomar decisiones formalmente justificadas.
Como consecuencia, la base objetiva de las soluciones a los problemas se incrementara y
la justificacion arbitraria que proviene de la inspiracion del ingeniero sera reducida. Por
tanto, el procedimiento gana rigor y el resultado sera mas fiable.
La inspiracion es la contribucion a la solucion de un problema relacionada con la expe-
riencia, preferencia, casualidad, moda, costumbre, escuela o influencia externa, y otras
causas subjetivas. Esa inspiracion puede ser de naturaleza endogena, que es lo que lla-
mamos intuicion y produce resultados artısticos (resultados basados en la conviccion); e
incluso puede ser de naturaleza exogena, que es lo que llamamos revelacion (resultados
basados en las creencias).
Como inspiracion, conviccion y confianza son garantıas debiles, su rol en el proceso de
disenar inventos y metodos es aceptado como ultimo recurso, cuando no hay deducibi-
lidad teorica e induccion empırica para garantizar objetividad. Por tanto, el proposito
general de la metodologıa de diseno es maximizar la base objetiva para tomar decisiones
y, consecuentemente, reducir la subjetividad.
En el futuro, el objetivo es desarrollar entornos computacionales de desarrollo para
proporcionar asistencia a los ingenieros en la actividad de toma de decisiones para crear
objetos, equipamientos, sistemas o metodos, es decir, para proponer soluciones a los
problemas.
Del estado del conocimiento concluimos que hay un interes creciente en el desarrollo
de herramientas para la ayuda en la toma de decisiones. Este interes es mas evidente
actualmente debido al incremento en la talla y complejidad de los problemas, y por
tanto, de los sistemas para resolverlos. Relacionado con el incremento en la talla de los
problemas, encontramos argumentos a favor de la metodologıa de diseno top-down frente
a bottom-up. De modo que nuevos enfoques fueron desarrollados como la ingenierbottom-
upia orientada a modelo.
La disponibilidad creciente de entornos computarizados para el apoyo al diseno ha sido
ayudado por los avances en enfoques metodologicos. Estos enfoques se han incorporado
en el diseno ingenieril en el cual variables arquitecturales de escalabilidad, sistematiza-
cion y reusabilidad son incluso mas relevantes.
Aunque aspectos de la actividad de diseno son tratados rigurosamente en la bibliografıa,
en general, la justificacion de las propuestas es esencialmente empırica, probablemen-
te basada en la experiencia de los autores. Esto da a tales contribuciones la garantıa
proporcionada por la autoridad academica de dichos autores, la cual merece todo el
reconocimiento personal pero que pertenece al dominio de la conviccion o confianza.
En cualquier caso, es deseable disponer de un marco formal que garantice el rigor de
la metodologıa de diseno y apoye su progreso de una manera consistente. Por tanto,
es necesario avanzar en los aspectos conceptuales de relacion entre los problemas y los
modelos para ası establecer un marco teorico para dar soporte al desarrollo de entornos
de ayuda a la especificacion y resolucion.
A medida que la gente se va haciendo mayor, sus habilidades disminuyen. Este proceso se
intensifica progresivamente y, en cierto punto, esas limitaciones afectan dramaticamente
a las condiciones de vida, suponiendo en algunos casos un impedimento al desarrollo
apropiado de las actividades de la vida diaria. La poblacion global esta envejeciendo,
es decir, el numero de personas mayores esta incrementando a la vez que la esperanza
de vida. Es por ello que tratar con los sındromes de fragilidad y demencia son de gran
interes para la poblacion. Un diagnostico precoz podrıa ser muy beneficioso ya que de
esa forma se podrıan retrasar los sıntomas o al menos ralentizar el deterioro.
En las actividades humanas, el sistema nervioso y componentes fisiologicos intervienen
en una compleja relacion, por ejemplo, una tarea simple como andar contiene multiples
procesos complejos como son el mantenimiento del equilibrio o el movimiento armonico
de las diferentes partes del cuerpo y su sincronizacion. Durante un ciclo de la marcha
estan involucrados procesos tanto cognitivos como fısicos. Es por ello que un deterioro
cognitivo o fısico afecta a la marcha. De la revision del estado del conocimiento conclui-
mos que existe una correlacion entre algunos sındromes de enfermedades que afectan a
las personas mayores y la marcha. Por tanto, el analisis de la marcha proporciona una
herramienta util para el diagnostico precoz de los sındromes de fragilidad y senilidad.
El paradigma de Ambientes Inteligentes tiene como objetivo el desarrollo de entornos
inteligentes para ayudar a las personas en sus actividades de la vida diaria. Un entorno
en el que sistemas inteligentes proporcionan ahorro energetico, control de iluminacion y
temperatura, seguridad, etc. Dentro del paradigma de Ambientes Inteligentes, la Vida
Asistida por el Entorno tiene como objetivo el desarrollo de sistemas que puedan ayu-
dar a las personas dependientes a reducir su dependencia en las actividades de la vida
diaria proporcionando servicios de monitorizacion para detectar emergencias y avisar a
familiares o cuidadores si por ejemplo la persona sufre una caıda.
El analisis de la marcha humana es el estudio sistematico del caminar humano a traves del
analisis de observadores experimentados con la ayuda de instrumentacion para medir los
movimientos y mecanicas del cuerpo y la actividad de los musculos. Durante un ciclo de la
marcha, diferentes partes del cuerpo trabajan en conjunto para lograr el desplazamiento
del cuerpo entero. Los pies proporcionan el apoyo en el suelo, las piernas proporcionan el
avance alternativo de cada pie para lograr el desplazamiento deseado, la cadera establece
el centro de gravedad mantenido gracias al movimiento armonico del tronco y los brazos.
El analisis de la marcha estudia el movimiento de todas estas partes del cuerpo. Para
obtener parametros del movimiento de diferentes partes del cuerpo, existen dos metodos:
usando dispositivos inerciales anclados a distintas partes del cuerpo con el fin de grabar
los movimientos desde un punto de vista interno o usar tecnicas de vision artificial para
observar los movimientos desde un punto de vista externo. De estos dos enfoques, el
enfoque inercial proporciona normalmente mejores resultados ya que los dispositivos se
instalan en las partes especıficas del cuerpo que se quieren analizar, sin embargo, esta es
una forma intrusiva de obtener la informacion. Esto podrıa alterar la marcha debido a
que el sujeto debe andar con dispositivos inusuales. El enfoque con vision artificial, por
otra parte, puede realizarse mediante la implantacion de marcadores en las diferentes
partes del cuerpo para realizar el seguimiento, en este caso el resultado es similar al
enfoque inercial con las mismas ventajas e inconvenientes. Otra forma de analizar la
marcha con tecnicas de vision artificial es mediante complejos analisis de la silueta y
segmentacion para extraer las diferentes partes del cuerpo de una forma no intrusiva.
Este metodo proporciona menos precision pero tiene la ventaja de que el sujeto no
necesita llevar puesto ningun dispositivo ni marcador y, por tanto, no altera la marcha
de ninguna forma.
El analisis de la marcha mediante vision artificial es un campo bastante comun. Se
han propuesto diferentes tecnicas durante la ultima decada. Sin embargo, pocos han
abordado el problema de la marcha anomala o patologica. La mayorıa de casos tienen
como objetivo el analisis de patrones de la marcha con fines biometricos.
La mayorıa de las contribuciones relevantes en el campo tienden a dividir las tecnicas de
vision artificial para el analisis de la marcha en dos amplios enfoques: tecnicas basadas en
modelo y tecnicas basadas en la forma o silueta. En las tecnicas basadas en modelo, los
investigadores tratan de encajar un modelo matematico en la estructura y movimiento
observada de la marcha. Algunos usan un modelo del esqueleto; otros segmentan la
silueta mediante formas elıpticas o volumetricas para ir midiendo caracterısticas espacio
temporales (ancho de zancada, pasos por segundo y similares). En las tecnicas basadas
en la forma, por otra parte, los investigadores se centran en la forma del sujeto y su
cambio con el tiempo.
De la revision del estado del conocimiento podemos concluir que existe una gran can-
tidad de enfoques para determinar los parametros de la marcha mediante tecnicas de
vision artificial, pero la mayorıa tratan el tema de identificacion de sujetos asumiendo
patrones de marcha normal. Sin embargo, estas tecnicas podrıan no cumplir con las ex-
pectativas al tratar con patrones anomalos de la marcha. Nuestra contribucion se centra
en proporcionar un enfoque basado en vision artificial capaz de extraer parametros de
la marcha siendo efectivo tanto para marcha normal como anomala usando camaras de
video exclusivamente. Una tarea comun en la mayorıa de enfoques es la extraccion del
fondo para obtener la silueta del sujeto. Esta es una tarea crıtica ya que el procesamiento
subsiguiente depende de ello. Tras esta fase, el procesamiento varıa de un enfoque a otro.
La mayorıa de enfoques se centran en las piernas ya que esa es la parte de la silueta
donde se produce la mayor parte del movimiento y presenta la mayor variabilidad en el
tiempo.
Resultados y conclusiones
El objetivo de formalizacion nos ha llevado a profundizar en los aspectos conceptuales
y procedimentales del conocimiento y de su creacion, con la consecuencia de aportar
sendas definiciones para problema y modelo, ası como hallar una justificacion formal,
basada en la teorıa de conjuntos, que confiere coherencia causal al metodo experimental.
Esta tesis propone una clasificacion de las decisiones que hacen los ingenieros para
disenar soluciones para los problemas de ingenierıa a los que se enfrentan. Hemos dividido
el problema inicial en tres subproblemas:
Subproblema modelo. Es la parte del problema que pertenece al ambito de uno de
sus modelos y es compartida con todos los problemas de ese modelo.
Subproblema especıfico. Es la parte del problema que pertenece exclusivamente a
el y no comparte con el resto de problemas que pertenecen al modelo.
Subproblema contextual. Es la parte del problema relacionada con las circumstan-
cias en las cuales el problema es resuelto, es decir, el proyecto de planificacion.
La division del problema en tres subproblemas nos permite establecer una base formal
para proponer una taxonomıa de la I+D+i:
Investigacion basica: si lo que se resuelve es el subproblema modelo. Da como
resultado conocimiento teorico.
Investigacion aplicada: si lo que se resuelve es el subproblema especıfico. Resolver
una sola instancia de un problema podrıa considerarse innovacion mientras que re-
solver un conjunto de problemas podrıa considerarse desarrollo. Da como resultado
conocimiento particular.
Innovacion procedimental: si lo que se resuelve es el subproblema contextual.
Hemos obtenido cinco clases de factores de diseno:
Una clase de decisiones dirigida a una familia de problemas, que son decisiones
abstractas sobre el modelo. Hemos relacionado esta abstraccion a la pregunta que es
el problema, que nos ha permitido conocer como es el problema.
Tres clases de decisiones relacionadas con la resolucion del problema especıfico
a traves de la concrecion del modelo. Estas decisiones son sobre la arquitectura,
la estructura y la tecnologıa, respectivamente. Hemos relacionado esta concrecion
con la secuencia de preguntas para que es el problema, como es la solucion y con
que esta hecha la solucion, respectivamente. Todas estos factores de diseno que
hemos llamado factores esenciales tienen nombres especiales: factores explıcitos que
son aquellos relacionados con la pregunta para que, que son factores arquitecturales;
factores implıcitos que son aquellos relacionados con la pregunta como, que son
factores estructurales; y factores intrınsecos que son aquellos relacionados con la
pregunta con que, que son factores tecnologicos.
Una clase de decisiones de diseno dirigida a la implementacion de un prototipo.
Esas son las decisiones para resolver el subproblema contextual. Hemos relacionado
las circunstancias de implementacion con todas las otras preguntas restantes: quien,
cuanto, cuando, donde, etc. A estos factores de diseno los hemos llamado factores
contextuales de diseno.
Hemos ordenado las cinco clases de factores de diseno con la secuencia:
modelo > arquitectura > estructura > tecnologıa > contexto
Por tanto, hemos establecido una base formal y hemos obtenido un metodo para tomar
decisiones de diseno de una forma coherentemente causal.
La hipotesis representa una version del problem cercana al ambito del modelo y de
la arquitectura. Por tanto, para que el diseno sea causal y formalmente correcto, es
necesario empezar tomando decisiones relacionadas con el modelo que proporciona el
marco de la solucion. Despues continuar tomando decisiones de forma secuencial en el
orden que hemos establecido previamente. Esto proporciona:
Una tecnica para definir un modelo para un problema dado y un conjunto de otros
problemas relacionados.
Una tecnica para obtener una especificacion funcional de un problema dado basado
en un modelo y una conjuncion de objetivos arquitecturales. Esta especificacion
ignora cualquier restriccion que provenga del dominio de la solucion, por tanto
tiene en cuenta tan solo restricciones que hayan sido establecidas en la hipotesis
inicial, evitando ası cualquier poda prematura en el arbol de posible soluciones.
Una tecnica para proponer una especificacion estructural de una solucion consis-
tente en un arbol de modulos y un grafo organizacional en el que se reflejan las
relaciones entre los distintos modulos que componen el arbol estructural.
Una discusion acerca de la forma de tomar decisiones de diseno relacionadas con
los factores contextuales.
El aspecto mas relevante de las tecnicas descritas esta relacionado con la naturaleza
observacional del conocimiento envuelto, al contrario de la forma convencional de tomar
decisiones de diseno basado en fuentes inspiracionales no formales como la experiencia,
las preferencias, etc. Por tanto, una definicion procedimental de la creacion de conoci-
miento en terminos de la percepcion humana ha sido propuesta.
Dentro de cada clase de factores de diseno, criterios de precedencia para determinar que
decisiones de diseno deben realizarse primero y cuales son las que deben ser subordina-
das, son dependientes de los objetivos de optimizacion y, por tanto, de cada caso. Por
ejemplo, las decisiones de diseno relacionadas con la estructura (eleccion de uno u otro
requerimiento, arbol estructural mas profundo con menos niveles, etc.) son susceptibles
de optimizacion, la cual depende de cada solucion. Esto puede proporcionar la semilla
para ayudar a los ingenieros a tomar decisiones de diseno mediante sistemas compu-
tarizados o incluso sistemas de inteligencia artificial, el cual es el trabajo futuro que
planeamos desarrollar.
Ademas de encontrar formalmente la justificacion causal del metodo cientıfico, hemos
podido encajar en ese marco formal los enfoques divide et vinces, model driven y top-
down de resolucion de problemas ingenieriles. Al tiempo que hemos encontrado que
la tecnica top-down de diseno es coincidente con el metodo cientıfico de resolucion de
problemas, el metodo bottom-up es coherente con la implementacion de prototipos, lo
cual justifica la restriccion de su utilizacion al diseno de instancias para las que ya se
conoce solucion.
Este procedimiento ha sido validado con un caso de uso real. El caso de uso elegido ha
sido el del diseno y desarrollo de un sistema basado en vision artificial para detectar los
sındromes de fragilidad y senilidad mediante el analisis de la marcha. Empezamos pro-
porcionando una definicion de la marcha para nuestro problema estableciendo ejemplos
y contra ejemplos para delimitar el modelo. La definicion resultante es la siguiente: ”La
marcha es el desplazamiento antropomorfico erguido, alternando pasos con dos pies, sin
apoyo adicional, manteniendo al menos un punto de apoyo en todo momento, en una
superficie horizontal o ligeramente inclinada”Despues procedemos a proporcionar una
especificacion funcional del problema estableciendo sus objetivos y los requerimientos
que cubren todos ellos. Recursivamente, hemos definido los siguientes niveles de espe-
cificacion proporcionando un arbol estructural. Finalmente, hemos construido el grafo
organizacional anadiendo relaciones entre los nodos hoja del arbol estructural.
El sistema resultante, capaz de extraer de forma automatica las variables cinematicas
que caracterizan la marcha humana, funciona de la siguiente forma: primero tomamos
como entrada una secuencia de imagenes de color y varios clasificadores determinan si
la salida puede ser interpretada como marcha normal o anomala. Para alcanzar nuestros
objetivos, disenamos una disposicion para capturar imagenes de sujetos andando en un
entorno controlado. Colocamos una camara de forma perpendicular a la direccion de la
marcha del sujeto y otra camara en frente. Despues realizamos una serie de experimentos
a partir de los datasets grabados para validar nuestra propuesta.
El sistema propuesto esta formado por cinco modulos tal y como se extrae de la espe-
cificacion estructural. Estos modulos son extraccion del fondo, localizacion de los pies,
extraccion de caracterısticas, extraccion del esqueleto y clasificacion. El primer modulo
toma como entrada la imagen obtenida por la camara sin procesar y realiza una extrac-
cion del fondo para obtener la silueta del sujeto, posteriormente lleva a cabo los calculos
necesarios para obtener el rectangulo que la engloba. La informacion de la silueta y el
rectangulo que la engloba pasa al modulo de localizacion de los pies para, mediante
segmentacion, obtener la posicion de ambos pies. En este modulo se obtiene la puntera
y talon de cada pie para las secuencias de imagenes laterales y la puntera para las se-
cuencias de vista frontal. La informacion de la posicion de los pies de este modulo y la
informacion de la silueta del modulo anterior es la entrada del modulo de extraccion del
esqueleto que, mediante segmentacion usando un modelo antropometrico y el calculo
de centros de gravedad obtiene los puntos de cabeza, cuello, cadera y rodillas. Estos
puntos junto con los puntos correspondientes a los pies del modulo anterior, forman una
aproximacion del esqueleto del sujeto. El algoritmo encargado de obtener la represen-
tacion del esqueleto del sujeto recibe el nombre de FASE. El modulo de extraccion de
caracterısticas usa como entrada la serie temporal de la localizacion de los pies para,
mediante anlisis de gradiente, obtener el instante de apoyo del talon y despegue de la
puntera para las secuencias de imagenes laterales. En el caso de la vista frontal, este
modulo utiliza la serie temporal de la diferencia de la componente Y de ambos pies y
hace uso de los maximos y mınimos locales de dicha serie temporal para acotar tempo-
ralmente los instantes de apoyo de talon y despegue de la puntera. Este modulo junto con
el modulo de localizacion de los pies forman el llamado algoritmo HSTOD o algoritmo
de extraccion de apoyo del talon y despegue de la puntera. Finalmente, el modulo de cla-
sificacion usa las diferentes variables obtenidas por los modulos anteriores para realizar
una clasificacion de la marcha. Estas variables son: serie temporal de ancho de zancada,
serie temporal del angulo de las piernas, vector caracterıstico de tres dimensiones con
tiempo de ciclo, velocidad y longitud de zancada y una imagen caracterıstica de cada
ciclo de la marcha obtenida mediante la computacion de la silueta media de todos los
fotogramas correspondientes a dicho ciclo.
Durante la realizacion de diferentes pruebas de precision y rendimiento detectamos que
la fase de extraccion del fondo es la mas costosa computacionalmente. Esto se debe a que
en esta fase, el sistema trabaja con la imagen completa. Ademas el buen funcionamiento
de este modulo es crıtico ya que de el dependen el resto de modulos. Por tanto, hemos
propuesto una optimizacion de este modulo. La optimizacion propuesta llamada IMPRe-
Ti permite procesar imagenes iterativamente de menor a mayor resolucion de modo que
si el tiempo disponible se agota se pueda disponer de, al menos, una respuesta subopti-
ma que permita al resto de modulos continuar con el procesamiento. Esta optimizacion
permite al sistema funcionar con restricciones temporales.
En el primer experimento realizado, el objetivo fue determinar la precision del algoritmo
HSTOD de deteccion de apoyo del talon y despegue de la puntera tanto para vista
lateral como frontal; en el segundo experimento, el objetivo fue determinar la precision
del algoritmo FASE de extraccion de articulaciones; en el tercer experimento, probamos
la precision de los clasificadores y las caracterısticas propuestas; finalmente, en el cuarto
experimento, determinamos el correcto funcionamiento del modulo IMPReTi de tiempo
real.
En el primer experimento, comprobar el funcionamiento del algoritmo HSTOD, utiliza-
mos un marcado manual de los instantes de apoyo del talon y despegue de la puntera
de todas las secuencias grabadas. Despues calculamos la diferencia en fotogramas en-
tre la respuesta del algoritmo y el marcado manual. La precision maxima que se puede
alcanzar esta acotada por la cantidad de fotogramas por segundo del video por tanto
consideramos que el margen de error para el marcado manual es de un fotograma. De
la misma forma, consideramos como valido un error de un fotograma en la respuesta
del algoritmo. Combinando ambos margenes de error damos como bueno cualquier re-
sultado que sea inferior o igual a 2 fotogramas de diferencia entre el marcado manual
y la respuesta del algoritmo. Los resultados del experimento muestran que, para vista
lateral, de los dos filtros usados, el primero proporciona menos precision que el segundo
pero menos casos no detectados para analisis de la marcha normal. Sin embargo, dado
que nuestro objetivo es caracterizar la marcha anomala, es por tanto el segundo filtro
el que mejor sirve a nuestros propositos. En vista frontal, el experimento muestra que
el enfoque basado en los cruces por cero no proporciona buenos resultados para marcha
anomala, problema que es solucionado con el enfoque basado en los maximos y mınimos
locales.
En el segundo experimento de la serie, comprobar el funcionamiento del algoritmo FA-
SE, utilizamos tambien un marcado manual de las posiciones de los puntos de cabeza,
cuello, cadera, rodillas y pies los cuales son comparados con los resultados obtenidos
por el algoritmo. Los resultados del experimento muestran que el algoritmo extrae con
mayor precision las articulaciones de la cabeza, el cuello y la cadera mientras que las
articulaciones de pies y rodillas tienen menos precision debido a la imposibilidad de
distinguir cada pierna durante la fase de cambio del pie de apoyo.
El tercer experimento de la serie, que fue llevado a cabo para validar los clasificadores,
muestra que, para la tarea de clasificar entre marcha normal y anomala, el clasificador de
las series temporales del angulo formado por las piernas usando K-Nearest Neighbours
y Dynamic Time Warping ofrece los mejores resultados de clasificacion. El vector de
caracterısticas tridimensional de tiempo de ciclo, longitud de zancada y velocidad ofre-
ce tambien un buen grado de precision. El experimento con marcha normal, diplejica,
hemiplejica, neuropatica y parkinson muestra que diplejica y parkinson son confundidas
entre ellas frecuentemente. El mismo problema esta presente con hemiplejica y neu-
ropatica, pero en este caso, un clasificador jerarquico nos permite mejorar los resultados
hasta un 74 % del 71 % obtenido por el no jerarquico.
En el cuarto experimento se repitio el primer experimento utilizando sucesivas reduccio-
nes de resolucion a la imagen original. Se comprobaron un total de cuatro resoluciones
con un factor de reduccion de 2 de modo que las resoluciones resultantes fueron 1080,
540, 270 y 135 pıxeles de altura respectivamente. Los resultados mostraron que el siste-
ma es robusto ante imagenes de entrada de menor tamano, aunque en 135 la precision
disminuye haciendo muy impreciso el algoritmo con resoluciones menores a esa. Por tan-
to, la optimizacion IMPReTi permite que el sistema pueda funcionar hasta a 100 FPS
(con el hardware usado en los tests) con una pequena reduccion en la precision.
Finalmente, indicar que el algoritmo HSTOD trabaja con un retardo de unos 3-4 fo-
togramas debido a la aplicacion de los filtros, y que el algoritmo FASE puede trabajar
fotograma a fotograma, eso junto con la optimizacion IMPReTi, permite al sistema ob-
tener medidas a la frecuencia del video siempre y cuando el tiempo de computacion sea
menor que el tiempo de fotograma.
Limitaciones y trabajo futuro
Aunque hemos proporcionado una metodologıa para reducir la subjetividad de las so-
luciones a los problemas, el estado actual de desarrollo requiere a un ingeniero humano
para tomar decisiones. La metodologıa proporciona herramientas al ingeniero para to-
mar decisiones limitando la arbitrariedad, sin embargo, como el ingeniero sigue siendo
humano, la subjetividad de cada individuo sigue presente aun. Todavıa queda trabajo
que hacer para automatizar el proceso.
Con respecto al sistema de vision artificial, la solucion actual presenta varias limitacio-
nes. La principal limitacion es el entorno, las imagenes de vista lateral requieren una
habitacion de unas dimensiones considerables para ser capaz de obtener suficiente canti-
dad de pasos. La vista frontal requiere menos espacio pero proporciona menos precision.
La secuencia de entrada tiene que contener un fondo estatico y la camara debe ser estati-
ca tambien, es decir, el movimiento del sujeto es el unico movimiento permitido en la
grabacion para poder realizar la extraccion del fondo, especialmente en la zona de los
pies y piernas.
El estudio ha sido realizado con un dataset relativamente pequeno y los sujetos de
pruebas eran personas sanas fingiendo distintos tipos de marcha patologica. Aunque el
estudio realizado es suficiente para comprobar la correccion de los algoritmos propuestos,
un estudio mayor con pacientes reales debe realizarse para validar la propuesta en un
escenario real.
Finalmente, debido al corto espacio de tiempo en el que se ha realizado esta tesis (cuatro
anos), el test peridico de la marcha para comprobar la posibilidad de la deteccion precoz
de los sındromes de fragilidad y senilidad no ha sido posible.
El trabajo futuro se centrara en abordar las limitaciones previamente mencionadas.
La posibilidad de usar una cinta de correr para grabar secuencias de la marcha sera
comprobado. Esto podrıa reducir potencialmente el tamano del entorno necesario para
grabar las muestras.
Nuestro trabajo futuro tambien se centrara en mejorar el algoritmo FASE para que sea
capaz de abordar el problema de localizar cada pierna durante la fase de cambio de pie
de apoyo y tambien localizar los brazos.
La posibilidad de procesamiento en tiempo real abre nuevas posibilidades como propor-
cionar correcciones a la marcha del sujeto en tiempo real o asistencia al equilibrio. Con
respecto a la asistencia al equilibrio, estamos contemplando la posibilidad de desarrollar
un sistema basado en giroscopios para reducir los riesgos durante situaciones de perdida
del equilibrio. Basamos esta asuncion en la hipotesis de que un grupo de giroscopos apro-
piadamente orientados podrıa proporcionar un vector de fuerza en cualquier direccion
capaz de compensar una perdida de equilibrio o, al menos, reducir la velocidad de la
caıda.
Agradecimientos
Mi agradecimiento empieza por Juan Manuel Garcıa Chamizo, director de esta tesis
doctoral, por el apoyo y la orientacion en todo lo relacionado con el desarrollo de esta
investigacion, por su paciencia para revisar los textos, y por la motivacion que me ha
inculcado, sobre todo ante los eventos mas adversos. Mi gratitud a Juanma, ademas
de por los conocimientos tecnicos, se extiende a lo que he aprendido de la actitud que
corresponde al cientıfico a la hora de investigar.
El apoyo recibido por parte de los demas miembros del equipo del proyecto FRASE
de la Universidad de Alicante, Jero, Javi, Rafa, Florez, Alexandros y Padilla, abarca la
otra gran parcela de mi deuda con el sistema investigador. Ellos, junto con el equipo
de mi otra Universidad, la de Castilla - La Mancha, Pepe, Ramon, Jesus e Ivan. Mi
agradecimiento por cuanto ha mejorado esta tesis tras pasar el filtro de las sesiones de
trabajo con ellos.
A mis companeros de laboratorio les agradezco su generosidad para enriquecer mi trabajo
con sus diferentes puntos de vista. De ellos, es emblematico el especial empuje y carino
que siempre he encontrado en la ayuda que me ha ofrecido Marcelo.
Al Departamento de Tecnologıa Informatica y Computacion de la Universidad de Ali-
cante lo tengo como mi hogar cientıfico, tanto por los profesores como por los tecnicos
y el equipo de administracion.
La esencia de las aptitudes de humildad y tenacidad, indispensables en un investigador,
ası como la pasion por conocer algo nuevo y aprender se las agradezco emotivamente a
los familiares que desde siempre me las han inculcado, Manoli, Esteban, Ana y Toni.
Gracias a todos por vuestro generoso apoyo el cual me otorga la confianza para iniciar
este nuevo viaje como doctor.
xvii
Contents
Resumen en castellano V
Agradecimientos XVII
List of Figures XXIII
List of Tables XXVII
Abbreviations XXIX
Symbols XXXI
1. Introduction 1
1.1. Starting conjecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2. Design decision making and design support systems . . . . . . . . . . . . 3
1.2.1. Top-down and bottom-up . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2. Model-driven engineering . . . . . . . . . . . . . . . . . . . . . . . 4
1.2.3. Towards automatic requirement analysis and formal specification . 5
1.2.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.3. Evaluation of frailty and senility syndromes through gait analysis . . . . . 7
1.4. Computer vision based gait analysis . . . . . . . . . . . . . . . . . . . . . 8
1.4.1. Shape-based techniques . . . . . . . . . . . . . . . . . . . . . . . . 10
1.4.2. Model-based techniques . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4.3. Abnormal gait recognition . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.4. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2. Formalization of the model driven paradigm 15
2.1. The activity of design and problem solving . . . . . . . . . . . . . . . . . 15
2.2. Procedural conception about knowledge . . . . . . . . . . . . . . . . . . . 18
2.3. Concepts about problems . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4. Classifying the design factors . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.4.1. Empiric evidence to classify the design factors . . . . . . . . . . . . 25
2.4.2. The relation scope of design decision . . . . . . . . . . . . . . . . . 28
2.5. Ordering the classes of design factors . . . . . . . . . . . . . . . . . . . . . 32
2.6. Model driven paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
xix
Contents xx
2.7. Design decisions about models . . . . . . . . . . . . . . . . . . . . . . . . 39
3. Formalization of the top-down method 43
3.1. Classifying the essential factors . . . . . . . . . . . . . . . . . . . . . . . . 43
3.1.1. Empiric evidence to classify . . . . . . . . . . . . . . . . . . . . . . 43
3.1.2. The relation role of an essential factor . . . . . . . . . . . . . . . . 45
3.2. Ordering the classes of essential factors . . . . . . . . . . . . . . . . . . . . 48
3.3. Scientific method and top-down design . . . . . . . . . . . . . . . . . . . . 49
3.3.1. Study case: Vegetable starter plate . . . . . . . . . . . . . . . . . . 51
3.3.2. Study case: Gait analysis using computer vision . . . . . . . . . . 54
3.4. Formal specification of a problem . . . . . . . . . . . . . . . . . . . . . . . 58
3.4.1. Study case: Vegetable starter plate . . . . . . . . . . . . . . . . . . 60
3.4.2. Study case: Gait analysis using computer vision . . . . . . . . . . 64
3.5. Gait classifier using vision features . . . . . . . . . . . . . . . . . . . . . . 66
3.5.1. Extract vision features . . . . . . . . . . . . . . . . . . . . . . . . . 67
3.5.2. Gait classification . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
3.6. Structural specification of a solution . . . . . . . . . . . . . . . . . . . . . 70
3.6.1. Structural tree . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.6.2. Organizational graph . . . . . . . . . . . . . . . . . . . . . . . . . . 73
3.6.3. Structural specification of computer vision for gait analysis . . . . 75
3.6.4. Organizational graph of computer vision for gait analysis . . . . . 76
3.7. Design decisions about contextual factors . . . . . . . . . . . . . . . . . . 77
3.8. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
4. Gait-A 83
4.1. Low-cost RGB devices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
4.2. Data set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
4.3. Extraction of vision features - Side view . . . . . . . . . . . . . . . . . . . 86
4.3.1. Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.3.2. Feet location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.3.3. Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
4.3.4. Skeleton extraction algorithm . . . . . . . . . . . . . . . . . . . . . 92
4.4. Extraction of vision features - Frontal view . . . . . . . . . . . . . . . . . 96
4.4.1. Feet location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
4.4.2. Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
4.4.3. Skeleton extraction algorithm . . . . . . . . . . . . . . . . . . . . . 103
4.5. Code optimizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.5.1. Iterative Multiresolution Processing in Real Time . . . . . . . . . . 104
4.5.1.1. IMPReTi first approach . . . . . . . . . . . . . . . . . . . 105
4.5.1.2. IMPReTi second approach . . . . . . . . . . . . . . . . . 106
4.5.2. IMPReTi results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.6. Gait classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.6.1. K-Nearest Neighbours for single value variables classification . . . 110
4.6.2. K-Nearest Neighbours and Dynamic Time Warping for time seriesclassification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
4.6.3. Principal Component Analysis and SVM for Gait Energy Imageclassification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
Contents xxi
4.6.4. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.7. Periodical gait test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
5. Conclusion 119
5.1. Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5.2. Future work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
A. Implementation of Gait-A 125
A.1. PC implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
A.2. Android implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
Bibliography 129
List of Figures
2.1. Questions and their corresponding answers to elaborate a salad. . . . . . . 16
2.2. General empiric scheme of causal design process: starts in a given state-ment belonging to the problem domain and finishes in a valid instance inthe solution domain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3. Classes of design factors and interpretation of their relation with the in-vention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.4. Causally coherent sequence of making design decisions and its correspond-ing process to implement a prototype. . . . . . . . . . . . . . . . . . . . . 34
2.5. Model-driven paradigm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.1. Classes of essential factors: intrinsic factors of the salad, its constituents;implicit factors, the organization of the salad; and explicit factors, theutility of the salad, that is, the salad itself. . . . . . . . . . . . . . . . . . 45
3.2. Classes of essential factors: intrinsic factors of Gait-A, its modules; im-plicit factors, the organization of Gait-A; and explicit factors, the utilityof Gait-A. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
3.3. Classes of essential factors and their meaning in the design decisions. . . . 47
3.4. Causally coherent sequence of making design decisions and its correspond-ing process to implement a prototype. This is the Top-Down design ofinventions sequence, as well as the scientific method to create knowledge. 50
3.5. Steps to follow with the causally coherent procedure to create an invention. 51
3.6. There is an external interface that connects functional and structuralspecifications, and an internal interface that connects decisions and pro-totyping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
3.7. Structural tree of vegetable starter plate. . . . . . . . . . . . . . . . . . . 72
3.8. Structural tree of an organoleptic solution to the vegetable starter problem. 72
3.9. Structural tree of a dietetic solution to the vegetable starter plate. . . . . 73
3.10. Structural tree of a slightly different vegetable starter plate. . . . . . . . . 73
3.11. Global organizational graph. . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.12. Detailed organizational graph. . . . . . . . . . . . . . . . . . . . . . . . . . 75
3.13. Structural tree of GAIT-A. . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.14. Organizational graph of GAIT-A. . . . . . . . . . . . . . . . . . . . . . . . 76
3.15. Causally coherent steps of our method for making design decisions. . . . . 80
3.16. Analogies and differences between our causally formal design proposaland the empirical top-down and bottom-up methods. . . . . . . . . . . . . 81
4.1. Recording set up. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
4.2. Flow diagram of the proposed system for side view. . . . . . . . . . . . . . 87
4.3. Output of the get external toe and heel function. . . . . . . . . . . . . . . 88
xxiii
List of Figures xxiv
4.4. Output of the get internal toe function. . . . . . . . . . . . . . . . . . . . 88
4.5. Output of the get internal heel function. . . . . . . . . . . . . . . . . . . . 88
4.6. MFF heel gradient without any filter. . . . . . . . . . . . . . . . . . . . . 90
4.7. MFF heel gradient applying filter 1 removing isolated values. . . . . . . . 90
4.8. MFF gradient applying filter 2 with Gaussian filter. . . . . . . . . . . . . 90
4.9. Graphic showing the amount of correct, wrong and undetected cases forHS and TO in side view using filter 1. . . . . . . . . . . . . . . . . . . . . 91
4.10. Graphic showing the amount of correct, wrong and undetected cases forHS and TO in side view using filter 2. . . . . . . . . . . . . . . . . . . . . 92
4.11. Head and torso COGs obtained. . . . . . . . . . . . . . . . . . . . . . . . 93
4.12. Thig and foot COGs obtained and foot location output. . . . . . . . . . . 93
4.13. Fast Approximate Skeleton Extraction output. . . . . . . . . . . . . . . . 93
4.14. Graphic comparison between 2D manual marking and FASE output. . . . 95
4.15. Normal gait leg angle comparison. . . . . . . . . . . . . . . . . . . . . . . 96
4.16. Abnormal gait leg angle comparison. . . . . . . . . . . . . . . . . . . . . . 96
4.17. Flow diagram of the proposed system for frontal view. . . . . . . . . . . . 97
4.18. Silhouette obtained after background subtraction and toes obtained bymarking the minimum Y of each halves of the silhouette. . . . . . . . . . . 98
4.19. Graph showing the difference of component Y of each foot. A red verticalline marks the instant at which a HS is produced and a green vertical linemarks TO. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.20. Graphic showing the amount of correct, wrong and undetected cases forHS and TO in frontal dataset using zero crosses approach. . . . . . . . . . 100
4.21. Difference of component Y of each foot with abnormal foot dragging. . . . 101
4.22. Difference of component Y of each foot after several Gaussian filters. . . . 101
4.23. Graphic showing the amount of correct, wrong and undetected cases forHS and TO in frontal dataset using local maxima and minima approach. . 102
4.24. Mipmaps generated. Overall image size increased by 1/2. . . . . . . . . . 106
4.25. RGB optimized mipmap. Overall image size increased by 1/3. . . . . . . . 106
4.26. Silhouettes computed in each resolution. From left to right: 1080, 540,270, 135 pixels height. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.27. RMSE result of the heel strike and toe off detection algorithm for eachresolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
4.28. Amount of undetected cases of heel strike and toe off for each resolution . 109
4.29. Processing time in milliseconds and FPS required for silhouette extractionphase for each resolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
4.30. Graph showing the idle time for each frame with different time constraints.A negative value means that the time constraint is exceeded. . . . . . . . 110
4.31. This figure shows the different synthetic samples obtained from the orig-inal (left). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
4.32. Classification with different number of principal components . . . . . . . . 115
4.33. Normal gait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.34. Diplegic gait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.35. Hemiplegic gait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.36. Neuropathic gait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.37. Parkinsonian gait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.38. Graphic showing the classification rates of the different classifiers used. . 117
List of Figures xxv
A.1. Gait-A PC implementation GUI. . . . . . . . . . . . . . . . . . . . . . . . 126
A.2. Gait-A Android implementation GUI. . . . . . . . . . . . . . . . . . . . . 127
List of Tables
2.1. Questionnaires to find the activities that have to be done in order tocreate inventions or solve the problems related as examples. . . . . . . . . 16
2.2. Traditional way of preparing a salad using an empirical approach. . . . . . 18
2.3. Classification of knowledge. . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.4. The answers to the questionnaire to make a keyboard can concern to amodel of the invention keyboard (M), to essential factors of the keyboard(E), or to circumstantial factors of the activity of design the keyboard (C). 26
2.5. Classes of answers to the questionnaire to make a salad. Model (M:food), essential factors (E: nutritive, tasty, lettuce, cucumber, olive oil,bed of vegetables, dressing, etc.), and circumstantial factors (C: planning,expert, cost, etc.). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.6. Answers to the questions to make the access control at a room. . . . . . . 27
2.7. The answers to the questionnaire to make Gait-A can concern to its model(M), to essential factors of (E), or to circumstantial factors of the activityof design Gait-A (C). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.1. Empirical classification of answers to questions about the essential factorsto make a keyboard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.2. Empirical classification of answers to questions about essential factors toelaborate a salad. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
3.3. The projection of each positive example of salad fall into the regionSALAD but any counterexample projection is not entirely inside the re-gion SALAD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.4. Examples and counterexamples of gait . . . . . . . . . . . . . . . . . . . . 58
3.5. Examples and counterexamples of gait and their projection into the model. 59
3.6. Human gait variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.7. Objectives of the vegetable starter plate. . . . . . . . . . . . . . . . . . . . 61
3.8. Objectives that the vegetable starter plate must satisfy. . . . . . . . . . . . 62
3.9. Formal specification examples. . . . . . . . . . . . . . . . . . . . . . . . . 63
3.10. Traceability matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
3.11. Gait classifier using vision features traceability matrix . . . . . . . . . . . 67
3.12. Extract vision features traceability matrix . . . . . . . . . . . . . . . . . . 69
3.13. Gait classification traceability matrix . . . . . . . . . . . . . . . . . . . . . 70
3.14. Requirements to cover the objectives of the vegetable starter plate problemas a salad. The objectives are defined in Table 3.8 . . . . . . . . . . . . . 71
4.1. Comparison between smartphone camera and Kinect sensors. . . . . . . . 84
xxvii
List of Tables xxviii
4.2. Results of the HSTOD algorithm showing the amount of correct detec-tions (less than 2 frames of difference between algorithm and manualmarking), undetected cases, wrong detection (more than 2 frames of dif-ference) and the root mean square error of both correct and wrong cases.F1: first filter in which isolated values are removed, F2: second filter inwhich the foot is used instead of only heel or toe. . . . . . . . . . . . . . . 93
4.3. Comparison between FASE and manual marking as percentage of sub-ject’s height. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
4.4. Results of the HSTOD algorithm for the first approach. . . . . . . . . . . 100
4.5. Results of the HSTOD algorithm for frontal view. . . . . . . . . . . . . . . 101
4.6. 10-fold and leave-one-out cross-validation results of each classifier. C1:KNN with DTW using stride width testing each cycle, C2: KNN withDTW using legs angle testing each cycle, C3: KNN with DTW usingstride width testing each subject, C4: KNN with DTW using legs angletesting each subject, C5, C6 and C7: Bayes, KNN and SVM respectivelyusing cycle time, stride length and speed . . . . . . . . . . . . . . . . . . . 114
4.7. 10-fold and leave-one-out cross-validation results of each classifier. C1:KNN with DTW using stride width testing each cycle, C2: KNN withDTW using legs angle testing each cycle, C3: KNN with DTW usingstride width testing each subject, C4: KNN with DTW using legs angletesting each subject. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
4.8. Confusion matrix of legs-angle with DTW and KNN . . . . . . . . . . . . 115
4.9. Confusion matrix of PCA with GEI and SVM . . . . . . . . . . . . . . . . 116
4.10. Confusion matrix of PCA with GEI and SVM after applying reclassifica-tion of hemiplegic-neuropathic with legs angle time series and KNN. . . . 117
4.11. Confusion matrix of PCA with GEI and SVM after applying reclassifica-tion of hemiplegic-neuropathic with legs angle time series and KNN andremoving diplegic gait. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
Abbreviations
AAL Ambient Assisted Living
ADL Activities of Daily Living
AmI Ambient Intelligence
BS Background Subtraction
CBF Coming Behind Foot
CBK Coming Behind Knee
COG Center Of Gravity
DTW Dynamic Time Warping
EFD Elliptic Fourier Descriptor
FASE Fast Approximate Skeleton Extraction
FPS Frames Per Second
GEI Gait Energy Image
GM Granular Model
HMM Hidden Markov Model
HS Heel Strike
HSTOD Heel Strike and Toe Off Detection
IMPReTi Iterative Multiresolution Processing in Real Time
KNN K-Nearest Neighbours
MDE Model-Drive Engineering
MFF Moving Forward Foot
MFK Moving Forward Knee
MGM Multi-Granular Modelling
MoG Mixture of Gaussians
PEI Pose Energy Image
PSA Procrustes Shape Analysis
xxix
Abbreviations xxx
RGB Red Green Blue
RMSE Root Mean Square Error
SVM Support Vector Machine
TO Toe Off
UML Unified Modelling Language
VDM Vienna Development Method
Symbols
c(x) constraint
d domain
d(x, y) euclidean distance
e essential factor
f filtering function
m model
p problem
q design decision
x variable
A analysis
B background
C camera
D distance
E observational effect — environment
F invention — field of view
G gait
H observer — human
I identity
K knowledge
L light
M mental process
N nil
P human perception
R resolution
S sensing process
xxxi
Symbols xxxii
U universe
W whatever
α hypothesis
β speed m/s
δ derivation of a domain or constraint from another
θ cycle time s
λ stride length m
σ scope of a design factor
ω solution
Θ organizational graph
Π set of problems
Υ structural tree
Φ concretion
Ψ abstraction
P power of a design factor
R role of a design factor
Chapter 1
Introduction
A Ph.D. degree is a milestone usually obtained by doing a thesis starting from a conjec-
ture. This technique provides the evidence that the author must obtain the degree of
doctor because the process of doing a doctoral thesis empirically supports the author’s
ability to create knowledge.
This work tries to bring the author to an state of conscience and attitude that maintains
a compromise between a direct scientific contribution, close to development, and an
abstract contribution, near the scope of concepts and methods.
For that reason, this proposal addresses many different aspects, from pure epistemologi-
cal ones aimed at clarifying the underlying key under the criteria and procedures, to the
purely instrumental ones to obtain a particular instance of solution to a given problem.
Thus, the global end of obtaining a doctor degree is concreted in two subsidiary objec-
tives. On one hand, to advance in the problem resolution methods justification. This is
aimed at reducing the empiricism and arbitrariness, that often rule the decision making
process, and incrementing the procedural rigour, which should increase the quality of so-
lutions. On the other hand, to produce an immediate advance in the state of knowledge,
particularly, in the problems of increasing the quality of life of elderly people through
computer vision techniques.
1.1. Starting conjecture
This research was developed as part of the FRASE MINECO project (TIN2013-47152-
C3-2-R) funded by the Ministry of Economy and Competitiveness of Spain and, for that
reason, the problem that addresses is to characterize the gait of elderly people through
computer vision techniques in order to early diagnose the frailty and senility syndromes.
1
Introduction 2
In addition to propose a solution, this thesis provides answers to the relevance of the
decisions leading to the solution, which has led to having to investigate the cause that
justifies every design decision and thus, to address issues on the relevance of solving
in the scope of models, as well as addressing methodological aspects to move from the
general to the particular ones.
The formalization of the model-driven paradigm is achieved by establishing the classes
of design decisions involved in any problem solving process. We also establish an order
of the classes providing a way to determine which design decisions has to be made first.
We also propose a way of selecting the model in which the solution will be framed and
a method to solve the specific subproblem by proposing objectives and requirements
applying divide et vinces technique until sufficiently simple subproblems are obtained.
This provides a structural specification that contains the modules that compose the
solution and an organizational graph that establishes the relations among those modules.
Applying the previously proposed methodology, we design and develop a computer vision
based system for gait analysis aimed at detecting the syndromes of frailty and senility.
The methodology provided the structural specification, that is, the different modules
that compose the solution. This work proposes a new technique based in computer vi-
sion that is used to measure gait parameters and to characterise human walking using
only video cameras. The development effort focuses on classification and inference algo-
rithms specialised in the modelling of human walking. This proposal will allow designing
intelligent and low-cost systems. The use of image sensors installed in digital cameras
is also desirable from an energy consumption point of view because they rely on visible
light only, i.e., the condition of obtaining information from the scene without adding
energy is an essential requirement. From a scalability point of view, since there is no
energy added, all the risk of saturation and interferences due to adding more sensors are
discarded.
Human gait can certainly be visualised by digital cameras but gait study entails a
complexity not readily observable; in fact, it is riddled with difficulties arising from the
variability of poses and actions walking involves. One way current research has devised
to counteract this variability effect is to identify the main features of human gait through
the process of segmentation, tracking and pose analysis using low-level algorithms and
then evaluating those features with high-level classification algorithms.
In the following sections, a review of the state of the art is presented. As stated before,
this thesis has different parts and, thus, the state of the art is divided in three parts:
design decision making and design support systems, evaluation of frailty and senility
syndromes and computer vision techniques for gait analysis.
Introduction 3
1.2. Design decision making and design support systems
We aim at proposing a conceptual framework to systematically perform design opera-
tions and to help on making decisions to solve the problems and create inventions. This
framework could provide criteria to develop systematic procedures to make formally jus-
tified decisions. As a consequence, the objective basis for the solutions to the problems
will be increased and the arbitrary justification that comes from the inspiration of the
engineer will be reduced. Therefore, the procedure gains rigour and the result will be
more reliable.
Note that inspiration is the contribution to the solution of a problem related to the
experience, preference, chance, fashion, custom, school or external influence, and other
subjective causes. That inspiration may be of endogenous nature, which is what we call
intuition, and produces artistic results (backed by the conviction); and even it may be
of exogenous nature, which is what we call revelation (results supported by beliefs).
As inspiration, conviction and confidence are weak guarantees, their role in the process
of designing inventions and methods is accepted as a last resource, when there is not
theoretical deductibility and empirical induction to guarantee objectivity. Therefore,
the general purpose of a design methodology is to maximize the objective basis to make
design decisions and, consequently, reduce the subjectivity.
In the more distant horizon, the aim is to develop computational environments to pro-
vide assistance to engineers in the activity of making decisions to create the objects,
equipment, systems or methods, i.e., to propose solutions to the problems.
1.2.1. Top-down and bottom-up
Top-down is a recursive methodology for solving problems. It starts with a view of the
whole problem that break it in multiple sub-problems related to the main problem and
repeat the procedure for each sub-problem until there are sufficiently simple or known.
Bottom-up, on the other hand, starts with different sub-problems that are joined to form
successively more complex problems.
In the field of designing electronic systems with mixed signals, Kundert [1, 2] discarded
the bottom-up approach because it does not have enough expressive power to produce
the circuit design solutions with the complexity and celerity demanded by the market
and manufacturing capacity. He considered that a formal top-down method that could
support a methodical and systematic design process is necessary. Therefore, he proposed
a systematic sequence for making decisions and verification through simulation.
Introduction 4
Some studies also attempted to provide criteria to determine, between the top-down and
the bottom-up approaches, which one best fits a particular problem within a system
framework. Crespi et al. [3] presented a case of study based on a robotics problem
to compare the results obtained with both approaches. They address issues about:
analytical and design challenges, the mental processes used by the designer to obtain
a solution, the type of assumptions that are crucial towards the performance of the
solution, and about the possibility to obtain the same solution from both approaches.
Mantyla [4] supported the top-down design of mechanical systems and proposed that the
structuring of information about the solution in several layers, consistent with each stage
of the design process, as well as maintaining the components geometric relationships
using constraint satisfaction mechanisms, should be part of the CAD systems. Chen
et al. [5] developed a multi-level assembly model which supports top-down design as
suggested by Mantyla. The proposed model has the ability to capture important data
and knowledge in design supporting different stages of the top-down assembly design.
This model can support different CAD systems through adaptation and extension.
However, the main drawback of these approaches are that the evaluation process is
purely empirical, which serve to evaluate the top-down and bottom-up approaches for
a specific problem, but lacks a formal basis that supports the extrapolation of these
results for the general case.
1.2.2. Model-driven engineering
Since the early years of computing, the efforts of the engineers was to develop environ-
ment that could help them design ignoring the complexity of the underlying hardware.
And thus, the high level programming language were popularized as they represent a
higher layer of abstraction that removes the necessity of knowing the complex machine
code and also provides a way to write code ignoring the hardware architecture. As the
complexity of systems grew larger, an evolution of these programming language was
oriented to the Computer-Aided Software Engineering (CASE) which were focused in
providing graphical programming representations for the engineers to design like state
machines, flow diagrams, etc. Those were the precursors of the Unified Modelling Lan-
guage (UML) which provided a standard for graphical programming diagrams for object
oriented programming languages. [6]
The paradigm of model-driven engineering solves aspects of large scale composition and
integration. This approach considers that there are aspects of problem solving that
belong to experts in the problem domain. The paradigm separates design decisions
related to the nature of the problem, from the ones related to the solution, through
Introduction 5
the development of the solution using a higher level of abstraction than programming
languages [7].
In 2001, the Object Management Group (OMG) proposed their Model Driven Arqui-
tecture (MDA). MDA emphasizes the role of models in developing software systems and
support that models should be accurate enough to be able to perform an automated
transformation to code [8, 9]. But even if most of the MDE practitioner acknowledge
its benefits, MDE has not been widely adopted and is used mostly for specific parts
of the system and rarely for a whole system [8]. Recent studies suggest that there are
scalability issues with MDE, large scale systems are difficult to specify and maintain
using current MDE approaches [10]. That might be one of the reasons MDE has not
been widely adopted and is used for specific parts of a system.
The Multi-Granular Modelling (MGM) deals with aspects of propagation of character-
istics between the modelling phases: requirement analysis, conceptual design, detailed
design and prototyping [11, 12]. They defined the MGM with three layers: user interface
layer which provides a way to interact with each Granular Model (GM); GM decompo-
sition and envelopment layer which contains the hierarchy of GMs; and data support
layer which provides data communication and storage for model development. Each GM
is considered as a subproblem and is defined by its domain, a list of meta-parameters
and a series of functions that describe the relationship of the meta-parameters.
1.2.3. Towards automatic requirement analysis and formal specifica-
tion
Automatizing the generation of formal specifications from the problem requirements
has been the subject of considerable investigation effort for fifty years. Thus, there are
proposals to use two-level grammars as formal requirement specification languages to
represent programs as high-level descriptions of recursive functions. The first level of the
grammar is used to specify the domain and the second level is used for rules of evaluation.
Its main drawback is that it can only be used as a programming language for recursive
functions [13, 14]. Subsequent refinements incorporate object-oriented approach [15, 16].
Other authors proposed approaches like Vienna Development Method (VDM) [17] and
VDM ++ [18].
Brunetti and Golob [19] proposed a design approach based on the characteristics of the
problem to take into account the semantics contained in the early stages of design in
order to use such information in the final stages of prototyping. They found empirically
the dual tree and graph structure later proposed by Qi [20], and using this concept to
develop the design aid environment SINFONIA [19]. Qi proposed a double structure
Introduction 6
based on a tree and a graph to model the assembly and a notation to represent the
composed structure. The relationship level of the assembly model is represented by a
tree, and the connection between modules of the assembly model tree is represented by
a graph [20]. Harald shows the principles and underlying mechanisms of the system
architecture and data structures for generating solutions. He transforms concepts and
characteristics in consistent formal specifications but he finds necessary to develop a
theoretical basis to provide support to the conceptual features analysis of the problems
[21].
Literature also contains proposals for software requirement analysis in natural language.
Likewise, there are many of them to automatically transform certain types of formal
specification to obtain source code, validating it from the specification. Among the
formal specification languages designed to facilitate the specification of information sys-
tems, the Z notation [22] was developed by the Oxford University. Z, which become
ISO standard in 2002, is used to formally describe software systems. It is based on the
set theory, the lambda calculus, and the first-order logic. It decomposes specifications
into subproblems called schemas that are used to describe the static and the dynamic
aspects of the system. One of its main advantages is that there are many tools. Object-Z
[23] is an extension of Z notation for object-based design solutions. Alloy specification
language [24], based on Z, allows to define structural properties, its syntax is compati-
ble with graphics object-oriented models and its formal expressions, based on sets, can
express complex constraints. The purpose of this language is to be as simple as possible,
so it ignores certain operators which reduce its power. However, later versions solved
some of these problems.
There are proposals to analyze natural language requirements to obtain entity-relation-
ship models that use a natural language parser to interpret the requirements and obtain
a representation of the entity-relationship model. Gomez et al. developed a design
support system that asks questions to the user [25].
UMGAR uses natural language processing to obtain the UML model [26]. The work of
Ilieva is oriented to obtain a formal representation [27]. The translation in the opposite
direction, i.e., to obtain a natural language specification from UML, has been addressed
in the GeNLangUML system, although the natural language obtained is very similar to
UML diagram itself [28].
Bruel and France [29] developed a system, FuZe, to translate UML models to formal
specification in Z notation. It uses the Fusion method [30], technique of object-oriented
development that emphasizes ease of use and intelligibility, but it lacks accuracy. This
is one of the main shortcomings of this method, hence the motivation for FuZe to obtain
an accurate formal specification.
Introduction 7
1.2.4. Conclusion
We conclude from the related works that there is an increasing interest in the develop-
ment of tools for aiding in the design decision making. This interest is more evident
nowadays due to the increase in size and complexity of problems, and hence, the systems
to solve them. Related to the increased size of the problems, arguments in favour of
top-down design methodology versus bottom-up is found. Thus, new approaches were
developed like the model-driven engineering.
The growing availability of computerized environments to support design has been aided
by advances in methodological approaches. These approaches have been incorporated
into the engineering design in which architectural variables of scalability, systematization
and re-usability are even more relevant.
Although some aspects of designing are treated rigorously in the literature, in general,
justification of proposals is essentially empiric, probably based on the experience of
the authors. This gives such contributions the guarantee provided by the academic
authority of these authors, which deserves all the personal recognition but belonging to
the domain of conviction or confidence. Nevertheless, it is desirable to have a formal
framework that ensures the design methodology rigour and supports its progress in a
consistent manner. Therefore, it is necessary to advance in the conceptual aspects of
relationships between problems and models in order to establish a theoretical framework
to support the development of environments to aid in the specification and resolution.
1.3. Evaluation of frailty and senility syndromes through
gait analysis
As people grow older their abilities are diminished. This process is intensified progres-
sively and, at certain point, those limitations dramatically affect the living conditions,
supposing in some cases an impediment to properly develop daily living activities. Global
population is ageing, i.e., the number of elderly people is increasing as well as the life
expectancy. Therefore, dealing with frailty and senility are of great relevance to the pop-
ulation. An early diagnose could be very beneficial because it could delay the symptoms
or at least slowdown the decay.
Several branches of medicine like orthopaedics, rheumatism, neurology and rehabilitation
have shed light on the relationship between human walking and disease symptoms such
as frailty and dementia [31, 32]. One of the most widespread techniques used to gain
insight into the area is gait analysis, the quantification of measurable gait parameters
Introduction 8
together with its interpretation [33]. This process, based on subjective measurements,
is basically performed under clinical conditions by specialists.
Most of the relevant studies in the literature note that the main component involved
in frailty syndrome evaluation is physical activity; however, they observe that some
biological markers are also present [34]. There are five highly accepted criteria proposed
by Fried [35] that indicate if a person is considered frail. These criteria are: involuntary
weight lose, general exhaustion, weakness, slow walking and low level of physical activity.
Three of them are related to physical activity, that is why gait is a clear indicator for
frailty detection. Gait is also related to cognitive deterioration [36], Hausdorff et al.
[37] consider gait to be a high cognitive task. In this sense, Mulder et al. [38] refer
to a gait cycle in which attention, planning, memory and other cognitive processes are
involved. Waite at al. [39] conclude their study by stating that gait slowdown can safely
be assumed to be a predictor factor of dementia. Studies in the same line can be found
in [40–43]. Therefore, the study of gait variability over time could potentially predict
the frailty and senility syndromes [44, 45]. Recent studies also suggest that frailty is a
predictor factor for Alzheimer and dementia [46, 47].
In human activities, the nervous system and physiological components intervene in a
complex relationship, e.g., a simple task like walking includes multiple complex processes
from equilibrium maintenance to harmonic movement of different body parts and their
synchronization. During a gait cycle there are both cognitive and physical processes
involved. Therefore, both cognitive and physical deterioration affect gait. From this
review, we conclude that there exists a correlation between some disease syndromes
that affect elderly people and gait. Therefore, gait analysis provides a useful tool for the
early diagnose of frailty and senility syndromes.
1.4. Computer vision based gait analysis
The paradigm of Ambient Intelligence (AmI) aims at developing intelligent environment
to help people in their Activities of Daily Living (ADL). An environment in which
smart systems provide energy saving, illumination and temperature control, security,
etc. Within the AmI paradigm, the Ambient Assisted Living (AAL) aims at developing
systems that could help dependant people to reduce their dependency in their ADL by
providing monitoring services to detect emergencies and warn family members or care
givers, e.g., if the person falls.
Advances in new technologies have led to the proliferation of applications that study gait
analysis like human identification [48, 49], video surveillance [50] or sport [51]. These
Introduction 9
new technologies provide, on one hand, techniques to measure parameters of human
gait more accurately and, on the other hand, intelligent analysis systems. Non-wearable
(e.g. video cameras, floor sensors, etc.) [50, 52] or wearable devices (e.g. accelerometers,
gyroscopes, etc.) [53] can measure and characterise human gait parameters. Intelligent
analysis techniques provide new strategies to interpret and understand human walking.
Human gait analysis is the systematic study of human walking through the observation
of experienced observers with the help of instrumentation for measuring body move-
ments, body mechanics and muscles activity [54]. During a gait cycle, different body
parts work together to achieve the displacement of the entire body. The feet provide the
ground support, legs provide the alternate advance of each foot to achieve the desired
displacement, the hip establish the center of gravity maintained thanks to the harmonic
movement of the arms and trunk. Gait analysis studies the movement of all these body
parts. To obtain parameters of the movement of different body parts, there are two
approaches: using inertial devices attached to body parts to record the movements from
an internal point of view or using computer vision techniques to observe the movements
from an external point of view. From these two approaches, the inertial approach usu-
ally provides better results because the devices are installed in the specific body part
analysed, however, this is an intrusive way of obtaining data. This could alter the gait
because the subject is wearing unusual devices. The computer vision approach, on the
other hand, could be performed planting markers in different body parts to be tracked,
in this case the result is similar to the inertial approach with the same advantages
and inconveniences. Another way of analysing gait with computer vision techniques is
through complex silhouette analysis and segmentation to extract different body parts in
a non-intrusive way. This method provides less accuracy but it has the advantage that
the subject does not need to wear any device nor marker and, thus, does not alter the
gait in any way.
Gait analysis through computer vision methods is a really common field. Different
techniques has been proposed during the last decade. However, few has addressed the
abnormal gait problem. The majority of cases are related to human biometrics by
analysing gait patterns.
Most of the relevant contributions in the field tend to divide computer vision techniques
for gait analysis into two broad approaches: model-based techniques and shape-based
techniques. In model-based techniques, researchers try to fit a mathematical model into
the observed gait structure and movement. Some use a skeleton model; others segment
the silhouette into elliptic or volumetric shapes to subsequently measure spatio-temporal
features (stride length, steps per second and the like). In shape-based techniques, on the
other hand, researchers inquire into the shape of the subject and its change over time.
Introduction 10
1.4.1. Shape-based techniques
With regard to shape-based techniques, Man and Banhu [55] proposed a gait represen-
tation feature called GEI, it is defined as the average of all the silhouettes in a gait cycle.
They use this representation to identify different persons by applying PCA to reduce
the dimensionality of the input and then comparing the n principal components. An-
other gait representation is proposed by Roy et al. [49]. They launch a high-resolution
representation of motion they come to name Pose Energy Image (PEI). They divide a
gait cycle in K key poses using K-means clustering algorithm. A PEI is then defined
as the average image of all the silhouettes of a subject in a gait cycle that belongs to a
particular key pose. The computation performance of PEI is slow. To resolve this, Pose
Kinematic is proposed. It is defined as the percentage of time spent in each of the N key
pose states. Applying the PEI approach, Roy et al. [56] propose the use of gait phase at
the instant of entering and leaving from one camera to another for tracking a person in a
multicamera system. They use a sequence of the silhouettes of a person’s gait as input.
From this sequence, they define some key poses using the K-means clustering algorithm.
Each key pose is a PEI. Once key poses are obtained, each silhouette of the sequence is
projected in eigenspace to calculate the feature vector. Following, a similarity score of
the input and each key pose is calculated.
A cascade fusion scheme is presented by Zheng et al. [52]. They represent links between
gait images and its correspondent foot accumulative pressure image. These images are
accumulative records of the reaction force of the ground against the pressure produced
by a foot during a gait cycle. Although this is a high precision technique, the main
disadvantage of this approach is that it requires the person to walk on barefoot because
accumulative pressure images vary depending on the footwear. It also requires expensive
sensors on the floor to obtain such images.
The work of Collins et al. [57] present a viewpoint dependent technique based on tem-
plate matching of body silhouettes that analyses cyclic gait patterns to extract key
frames from a sequence. The extracted frames are classified using KNN with normalized
correlation as a distance function. Classification rates greater than 90% show that the
method has great potential for gait recognition assuming a fixed viewpoint.
The shadow projected by a person is used by Iwashita et al.[50] to improve gait recogni-
tion using aerial cameras in which the person is shown from above. They use spherical
harmonics to analyse gait features showing better results than other methods using
Fourier, Gait Energy Image or Active Energy Image. They also propose the combined
use of shadow and person silhouette to improve recognition using oblique camera where
there is a good view of both. The main drawback of this approach is that it is dependant
Introduction 11
of the light to produce shadows. Then they launch a robust identification method to
changes in appearance [48] by using subdivisions of GEI and matching each division
individually and then fusing the results.
The work of Zeng et al. [58] analyses a set of different gait features for person identi-
fication. They extract the silhouette and then obtain the contour using Canny’s edge
detector. Then they extract 4 features: evolution of the width/height proportion, evolu-
tion of maximum and minimum width of the bounding box, evolution of the silhouette
area and evolution of the centre of gravity. These 4 features follow a cyclic pattern that
match the gait cycle and are used to identify a person through deterministic learning.
The system is previously trained to learn the patterns of the 4 features for each person
that are intended to be identified.
1.4.2. Model-based techniques
In shape-based techniques deep knowledge of gait is not essential. Perhaps this is one
reason for these techniques to produce notably accurate person identification. However,
since we attempt to perform a full characterisation of gait, a mathematical model, that
is, model-based techniques, will be more suitable. We start by examining Leu et al.’s
approach [59]. They initiate the process by extracting the silhouette by background
subtraction and then, they apply an anatomical model to segment head, torso, thighs
and feet. Tracing the mean point of each segment and applying linear regression, they
obtain the skeleton. Another method for extracting skeleton joints is proposed by Yoo
and Nixon [60]. They also use an anatomical model to segment the silhouette, then they
obtain the mean points of each segment and apply linear regression to obtain a line that
represent the bones. During double support gait phase they apply motion tracking to
estimate the location of the occluded points.
The use of Kinect sensor to measure gait parameters during a treadmill walking with
frontal view is examined by Xu et al. [61]. Heel strike and toe off events obtained with
a Kinect sensor are compared with those obtained with a motion tracking system. The
results show that heel strike has less error than toe off because heel strike is produced
closer to Kinect sensor. The study shows that Kinect sensor may be used as an alter-
native device to measure some gait parameters for treadmill walking. However, the use
of a treadmill could alter the gait of the subject. Also Kinect emits infra-red light to
the environment to obtain depth measures, this produces some scalability issues because
using more than one device require a synchronization to avoid interferences.
Choudhury et al. proposed a method called STM-SPP [62]. This method is composed
by three modules: extraction and post-process of the silhouette, subject classification
Introduction 12
using Procrustes Shape Analysis (PSA) and Elliptic Fourier Descriptor (EFD), and com-
bination of both results. Silhouette extraction module is carried out using background
subtraction and morphologic operations to remove noise. PSA module analyses a group
of shapes using matching of settings. A setting is defined as a set of geometrical location
points of a figure. Those points are obtained from an anatomical analysis in which dif-
ferent key body points (head, torso, knee...) are considered as fraction of the subject’s
height. They calculate stride length using: the width of the bounding box of the zone
defined as feet. EFD allows to characterize the contour of the subject in key points of a
gait phase.
Cunado et al. [63] present a feature-based method for gait recognition that is robust
against noise and occlusion. It uses a gait signature based on phase-weighted Fourier
magnitude of the hip rotation. Classification performed using KNN showed a rate of
80% when using only magnitude information and 100% when using phase-weighted mag-
nitude. Although good results, the experiments were performed in a small database of
subjects. The method is improved by Yam et al. [64] applying it to not only gait but
running as well showing better recognition in the latter.
Even though we consider that a model-based technique best adapts to our present needs,
we bank on shape-based techniques for improving our gait characterisation. We think
it will be fruitful to further evaluate a combined method as the one illustrated in the
three-phase gait recognition approach Chodhury et al. [65] deal with. They analyse
the spatio-temporal shape and movement of a subject’s silhouette to identify him. In
the first phase, they use the phase-weighted magnitude spectra of the Fourier descriptor
of the silhouette contour in 10 phases of gait period to analyse the spatio-temporal
changes in shape. In the second phase, they perform an analysis of the body shape and
movement by fitting ellipses in the segments of the contour for the 10 phases of gait
using a histogram matching with the Bhattacharyya distance as a dissimilarity score. In
the last phase, they use Dynamic Time Warping to analyse the angular rotation pattern
of the front knee taking into account the arms swing. Finally, the scores obtained in
each phase are fused using a score weight system to obtain a robust identification against
occlusion or unavailable or distorted frames.
1.4.3. Abnormal gait recognition
A lot of proposals for human identification using gait analysis exist, however, there are
few proposals for normal and abnormal gait classification.
Introduction 13
Meyer et al. [66] proposed a method to classify gait between walking, hopping, running
and limping, they track the head and trunk position and train some Hidden Markov
Models (HMM). They obtained 62.2% accuracy rate between the four gait patterns.
Bauckhage et al. [67] proposed a homeomorphisms between 2D lattices and binary
shapes to obtain a vector space where the silhouette is encoded. They perform successive
bounding box splittings to obtain different lattices, then Support Vector Machine (SVM)
is used for classifying obtaining around 80% accuracy rate.
Wang [68] proposed a method based on optical flow to calculate a histogram of silhouette-
masked flows, then an eigenspace transformation is performed. They compare with a
normal gait template to calculate a deviation with which they obtain 90% accuracy rate.
An approach for parkinsonian gait recognition is presented by Khan et al. [69]. They
use the silhouette to obtain the bounding box and fit a human model to find head, torso
and legs segments. The skeleton is obtained by computing the mean points of each
body segment. Following, they obtain legs movement and the posture inclination while
the person is walking to get a particular score that it is compared with a normal gait
model in order to get a similarity score. This approach provides 100% accuracy to detect
parkinsonian gait through legs angle and posture inclination because there is a sufficient
difference between normal and parkinsonian gait. To distinguish other gait types of
pathological gait this approach could be insufficient but it provides a first approach.
Similarly, Krishnan et al. [70] use lean and ramp angles to detect abnormalities such us
postural instability and heel strike instability with satisfactory results.
1.4.4. Conclusion
We can conclude from this review that there are a myriad of approaches to figure out
gait parameters through vision-based techniques, but for the most part, they tackle sub-
ject identification assuming a normal gait pattern. However, these techniques might not
fulfil expectations when dealing with abnormal gait patterns. Our contribution aims at
providing a vision-based approach capable of extracting gait parameters that is effective
in both normal and abnormal gait using RGB images exclusively. A common processing
phase in most approaches is the background subtraction to obtain the subject’s silhou-
ette. This is a critical task because the subsequent processing depends on it. After this
phase, the processing varies from one approach to another: some divide the silhouette in
key parts, others try to fit mathematical models into it or analyse its change over time.
Most of the approaches focus on legs because that is the part of the silhouette where
most of the movement is produced and present the greatest variability over time.
Chapter 2
Formalization of the model driven
paradigm
2.1. The activity of design and problem solving
Being the considerations of this work extremely open, the starting point must be as
general as possible, i.e. there must be no restrictions or they must be reduced to the
unavoidable ones. For example, it should be noted that the activity done by a maı to
serve a salad, a chef to prepare a pizza, a student to find the area of a triangle or the
radius in a sphere, or the activity performed by an engineer to design a system, or to
develop a keyboard or even an access control, all those activities share the same essence:
to answer the intuitive question what to do to do something. That is, answering the to
do questions.
That general question can be decomposed into countless more specific questions, so any
compendium of the individual answer to each of them constitute an answer to the to
do questions. Some of these questions are: what (. . . is the thing to do), who (. . . has to
do it), with what (. . . must it be done), whom (. . . to do it), how much (. . . it must cost),
how (. . . to do it), when (. . . to do it), what for (. . . must it be done), where (. . . to do it),
etc. Table 2.1 shows questions to be answered to make decisions that provide solution
for any problem.
This concept resembles the 6 W (or 5 W and an H ) used in journalism to obtain the
maximum amount of information by answering the questions what, who, when, where,
why and how. Or the WWWWWHH principle of software engineering proposed by
Boehm [71] which organises the life-cycle plan in objectives (why is the system de-
veloped), milestones and schedules (what will be done by when), responsibilities (who
15
Formalization of the model driven paradigm 16
is responsible and where are they located), approach (how will the job be done) and
resources (how much resources are needed).
Each answer represents an action, an event or a variable value. Hereafter, we will refer to
these factors that belong to the creation of an invention or correspond to the solution of
a problem, as design factors. Design decision are just actions to establish design factors.
Table 2.1: Questionnaires to find the activities that have to be done in order to createinventions or solve the problems related as examples.
What to do to do something
Invention Activitydesign a system what, how, when, who, where, with what,...serve a salad whom, when, where, how, how much,...calculate a triangle who, what, with what,...produce a keyboard with what, how much,...
WITH WHAT
lettuce, cucumber,
olive oil,…
WHAT FOR
tasty, healthy,
nutritive, aromatic,…
WHAT
food, meal,…
WHERE
into the cook
etc.WHOM
two dinners
HOW MUCH
less than 10 €,
medium size,…
WHO
a chef
WHEN
just to be served
HOW
a bed of vegetables
with sprayed oil
Figure 2.1: Questions and their corresponding answers to elaborate a salad.
Figure 2.1 shows examples of the answers corresponding to each question related to how
to elaborate a salad.
The statement of a problem, i.e., its hypothesis, must belong to the domain of statements
to be clarified which is the scope of the problems. To avoid preconceptions that could
bring into the hypothesis some aspects pertaining to the solution, the hypothesis should
consist of conditions conceived under an approach that has to be external to the solution
of the problem. Therefore, the hypothesis will answer only to the two external issues
that are in the to do questions. These are: what and what for. Note that answers are
related to nature and utility that the solution will have, respectively.
Even though other possibilities could exist, consistency of design procedure can be en-
sured causally and, therefore, their starting point has to be the initial conjecture, that
is, the hypothesis.
Figure 2.2 shows the empiric general scheme of causal design process: the starting point
is a conjecture that falls within the scope of the problem. A sequence of actions leads,
Formalization of the model driven paradigm 17
firstly, to a correct and accurate definition of the problem, secondly, to a provisional
instance of a solution, and finally, to a valid instance of that solution, also called proto-
type.
Each step of the method is causally based on the previous step. In order to prevent an
early bound into the tree of possible solutions, the hypothesis should express the problem
from an outside perspective to any solution, ignoring the structure and composition of
any solution. That is, the hypothesis should be a view of the problem from the Universe,
but cannot be excluded that conditions related to components or to the structure could
be present.
formulate propose experiment
PROBLEM DOMAIN
(external focus)
SOLUTION DOMAIN
(internal focus)
DESIGN MAKE
idea prototype
Figure 2.2: General empiric scheme of causal design process: starts in a given state-ment belonging to the problem domain and finishes in a valid instance in the solution
domain.
The approach represented in figure 2.2 is considered intuitive by international scien-
tific community, that is, it is part of the axiomatic system of engineering praxis and,
therefore, is assumed that no further explanation is required. This assumption implies
acceptance of the process of creating inventions as an activity whose essence comes from
the inspiration of the engineer: knowledge, experience, skills, etc. In short, it is assumed,
that creation in engineering is based on the intuition of the engineer. As a consequence,
it is really difficult to find scientific works that formally ensure their suitability except,
perhaps, epistemological contributions or other coming from the scope of the history
and philosophy of science [72–79].
In general terms, the quality of solutions that an engineer obtains would be accepted
depending largely on the consideration of each individual. Thus, a creation made by
an engineer is subjective. In order to influx objectivity into that creation, usually, a
partial induction is done that consists of experimentally prove that the solution fit the
objectives that were established as conditions to characterise the problem.
Table 2.2 shows a known Spanish saying that provides the recipe to elaborate a salad,
whose foundation is the empirical evidence that these ingredients (with what), in these
proportions (how much), and that sequence (how) results in salads that are considered
tasty.
Formalization of the model driven paradigm 18
Table 2.2: Traditional way of preparing a salad using an empirical approach.
Empiric procedure to elaborate a salad
Salad must be salted, with few vinegar, and plenty of olive oil(La ensalada, salada, poco vinagre, y bien aceitada)
with what with salt, vinegar and olive oilhow much a lot of salt, a little quantity of vinegar, and a lot of olive oilhow first add salt, then add vinegar and, finally, add olive oil
What we propose below is a formally justified procedure to make decisions about the
design factors. This procedure is based on the set theory, specifically, binary relations
between these design factors. Briefly, what is at issue here is to characterise the design
factors that share certain properties, assuming all factors that belong to the same class
will be open to admit a similar approach. Next, ordering the kinds of factors will
provide criteria over the adequate sequence we must follow to obtain answers to the to
do questions.
Previously, to contextualise this approach, we will need to think over the concept of
knowledge, problem and the relationship between them.
2.2. Procedural conception about knowledge
As we are interested in a procedural understanding that could support the transfor-
mations required by the resolution process, we propose that knowledge creation is the
process of getting an answer from a question or, equivalently, making decisions to obtain
a solution from a problem.
In a empirical interpretation, we define knowledge creation in the following terms.
F being an invention, entity, event or phenomenon in the Universe.
F ∈ U (2.1)
An observational effect, E, of the invention F will depend on F . Assuming functional
notation,
E = E(F ) (2.2)
Human perception, P , of the observational effect is the activity of apprehend something
by means of the mind through the senses. Then, P provides information to the individual
about F through E. This information is knowledge, K, about the inventions, entities,
Formalization of the model driven paradigm 19
events or phenomena in the Universe.
K(F ) = P (E(F )) (2.3)
Perception can be decomposed in two sequential steps: a sensory process, S, that consists
of a transformation that filters the observable effect, followed by a mental process, M ,
that modulates by means of interpreting the incoming information to obtain the final
knowledge.
K(F ) = M(S(E)) (2.4)
That is, knowledge is created when an answer to an observational effect is obtained,
through perception, which is a double sequential filtering: sensory followed by inter-
pretative. Furthermore, knowledge creation can be expressed as a sequence: sensory
process, mental process.
Perception can be defined in the following domain:
P ∈ {W, I,N} (2.5)
Being:
I: identity, the filter does not transform the incoming information, so P = E
N : null, the filter produces no response, so P is not defined (@P )
W : whatever, so P = W (E) excluding any filters that produces no change or that
do not produce a response.
A classification of knowledge creation can be made, as shown in table 2.3.
Knowledge creation allows us to establish a classification. Therefore, according to the
fundamental theorem of equivalence relations, we can affirm that there exists an equiv-
alence relation between the process of knowledge creation that establishes three classes
of knowledge: formal, non-formal and null
Formal knowledge. Completely defined processes of perception, that consist of in-
coming sensory information followed by interpretation, can produce observational
knowledge. Its main characteristic is that knowledge can be reproduced consis-
tently from the same E.
Non formal knowledge. Incompletely defined processes of perception, that consist
of only interpretation, produce non observational (inspired) knowledge. Knowledge
Formalization of the model driven paradigm 20
Table 2.3: Classification of knowledge.
Classes of knowledge
PerceptionNature: meaning Class
Sensory Mental
W WObservational: sensing
and interpreting areinvolved
FormalI W
W II I
N W Inspired: interpretingin absence of sensing
Non-formalN I
W NNull: absence ofinterpretation
NullI NN N
about a given F through the same E lacks dependency, then, it can only be
reproduce through belief.
Null knowledge. No interpretation does not produce knowledge.
2.3. Concepts about problems
The concept of problem has received an extremely reluctant treatment in the scientific
literature. The few authors who addressed it, did it from an eminently epistemological
and philosophical approach, which serves to meet the problems but does little about
how they are made and how to solve them [78, 79]. Both, Agre and Landry agree on
considering that the problems are the mechanism to create knowledge, and handle the
concept of problem in a dynamic sense, related to the activity to solve it. This is what
we are calling in this work making design decisions.
Agre considers that the term problem is used to refer to what is to change, and the
word solution is used to denote an acceptable version of that change (in addition to
using it to refer to the process of changing or solving). He considers that the terms
question and answer refer to a similar context of inquiry and learning, and that both
couples, problem-solution and question-answer, maintain a sequential relationship in the
sense that the first leads to the second. Also, he notes that the relationship between the
two pairs of terms is so close that often are indistinctly used (problems are answered
and questions are solved). Agre proposes a generic definition of the concept of problem
that is based on the notions of consciousness, difficulty and conviction to be resolved.
Finally, he notes the absence of scientific work on the relationship that may exist between
Formalization of the model driven paradigm 21
problem solving and learning, recognizing that there appears to be a complex conceptual
connection between learning, problem and solution.
Landry notes that many definitions of the concept of problem have been established but
have not been empirically tested. He reviewed the three following approaches:
Objectivist. The problems are part of the external universe to the subject that
is aware of them. It considers that the problems have autonomous existence. This
approach gives the problems the ontological status of having real existence, and
intervening in a problem is to intervene on reality.
Subjectivist. It considers that the problems are the subject’s convictions. On-
tologically, the problems are conceptions.
Constructivist. One problem is the representation that a subject performs from
an object. From the point of view of ontology, the problems are knowledge pre-
conceptions. Epistemologically, the resolution of the problems are the processes of
knowledge creation. The solutions are knowledge that is created.
Preston, in relation to management information systems, notes that the problems have
inherent structure composed of finite variables and rules governing the relationships
between variables. He also suggests that the key issue is to identify the set of ordered
activities that make up the problem solving process [80].
We are interested in the procedural aspects of scientific research and technological design.
Thus, we propose an approach focused in that particular purpose. The statement of a
problem is a conjecture or hypothesis. To provide a solution is to obtain a conclusion or
thesis. To solve it is to procedurally reach that thesis through successive transformations
from the hypothesis. Following a coherent resolving procedure will provide consistency
to the obtained solution.
According to Preston’s proposal, in general, we consider that the term problem refers to
a declaration that is not sufficiently clear. For this concept to be as general as possible,
we propose the following definitions:
Definition 2.1. Problem. A problem is a statement to be clarified. Components of
the statement are terms referring to elements involved in the problem, and they are
called variables of the problem. The set of variables, di, is the domain of existence of
the problem pi. The meaning of the statement is a constraint between variables, ci.
Variables xj can be non quantifiable and constraints on them ci(xj) can be non func-
tional.
Formalization of the model driven paradigm 22
pi ≡ 〈di, ci〉 /di = {xj} , , ci = ci(xj) (2.6)
Instrumentally, the problems are observable effects. The fact of clarifying the statement,
inevitably includes subjectivity but leaves it limited to the context of the process of
enlightenment, that is, resolution of the problem.
Definition 2.2. Design decision. It is the action that derives a statement from
another. Derivation of statements establishes causal coherence between a problem and
a solution thereof. Design decisions work both on variables and on the constraint of the
problem. A sequence of design actions transforms the problem into a solution statement.
We denote the derivation of a set of variables from another, or the derivation of a
constraint from another, respectively, with the symbol “δ”.
δ : dn → dm
di → dj = δ(di)(2.7)
δ : cn → cm
ci → cj = δ(ci)(2.8)
Design decisions just consist of establishing design factors. Responses to the to do
questions, that is, the design factors must be derivable from the variables of the problem,
according to the constraint of the problem.
qt(pi) ≡ 〈dt, ct〉 /(δ(di) = dt) ∧ (δ(ci) = ct) (2.9)
Both, design decisions and design factors have the nature of problems, possibly elemen-
tary.
Definition 2.3. Solution. A solution to a problem is a statement derived from the
problem that clarifies it. The nature of a solution is the same as the problem, since the
condition of being solution arises from the externally assigned semantic value.
ωk(pi) ≡ 〈dk, ck〉 /dk = {xl} , , ck = ck(xl)/(δ(di) = dk) ∧ (δ(ci) = ck) (2.10)
Definition 2.4. Model. Term model is referred to a set of problems Π: model of a
set of problems. A model is a problem whose statement is part of the statement of each
problem in the set. Its variables are common to all problems and its constraint includes
Formalization of the model driven paradigm 23
the union (join) of the constraints of the problems. Therefore, a model is an abstraction
of each problem, and therefore a subproblem common to all problems in the set.
∀pi ∈ Π, ,Mn(Π) ≡ 〈dn, cn〉 /dn = {xr} , , cn = cn(xr)/(δ(⋂di) = dn) ∧ (δ(
⋃ci) = cn)
(2.11)
All problems that share a given model form a set of problems that owns the model itself
in its interpretation as a problem. As the model is an abstraction of each problem in
the set (Eq. 2.12), usually, the problem is considered an instance of the model that can
be obtained by concretion of it (Eq. 2.13).
Ψ: pn → p
Π→ mi = Ψ(Π)(2.12)
Φ: p→ p
mi → pi = Φ(mi)(2.13)
The act of concretion from a model to one of their problems is the specific subproblem of
that problem in relation of that model. This specific subproblem is the complementary
of the model into the problem (Eq. 2.14).
mi ≡ 〈di, ci〉, , pj ≡ 〈dj , cj〉, ,Φ(mi) = pj → dj ⊇ di ∧ cj ⊆ ci (2.14)
On the other hand, as a given problem can be part of numerous sets of problems, that
problem could be abstracted from numerous models.
As an example about models, consider the following cases of board game pieces:
Let X = {b, f, a, s, c} be a set of variables where b is base, f form, a attribute, s side
and c color.
p1 ≡ 〈d1 = {b, f, a, s, c}, c1 = {b ∈ [square, circle]∧f = figure∧a ∈ [movement, value]∧s = c}〉: The chess pieces are circular or square based, represent figures, have attributes
of movement and value, and side linked to its color.
p2 ≡ 〈d2 = {b, a, s, c}, c2 = {b = [circle] ∧ a = [value] ∧ s = c}〉: The checkers pieces are
circular, have attributes of value, and side linked to its color.
Formalization of the model driven paradigm 24
p3 ≡ 〈d3 = {b, s, c}, c3 = {b = [circle] ∧ s = c}〉: The parcheesi pieces are circular, and
sides linked to its color.
p4 ≡ 〈d4 = {b, s, c}, c4 = {b = any ∧ s = c}〉: The tic tac toe pieces can be of any form,
and side linked to its base color.
p5 ≡ 〈d5 = {b, a}, c5 = {b = [rectangle] ∧ a ∈ [value, relationship]}〉: The dominoes
pieces are rectangular, have attributes of value, and relationship with other pieces.
p6 ≡ 〈d6 = {b, a}, c6 = {b ∈ [circle, rectangle] ∧ a = [value]}〉: The roulette pieces are
circular or rectangular, and have attribute of value.
A model of pieces of board Π1 = {p1, p2, p3, p4}, is m1 ≡ 〈d = {b, s, c}, c = {b = any∧s =
c}〉: pieces of board have regular geometric base, and sides linked to its color.
A model of pieces Π2 = {p1, p2, p3, p4, p5, p6}, is m2 ≡ 〈d = {b}, c = {b = any}〉: pieces
have simple geometric base.
From this point of view, the cards of trivial are also pieces. The disks to accommodate
the cheeses are pieces of board. The cheeses are pieces.
The objectivist, subjectivist and constructivist approaches would correspond to a tax-
onomy based on possible ways of perception to create knowledge. Objectivism would
be the cases where interpretation were the identity, Eq. 2.15. Subjectivism would cor-
respond to sensing equal identity, Eq. 2.16. When both, perception and interpretation
were any functions, Eq. 2.17, we would have constructivism, thus, constructivism seems
to coincide with formal knowledge. Even more, if sensing and interpretation were iden-
tity functions, we could say that the resulting knowledge were the truth, which means
that perceived knowledge is equal to observational effect, that is the objective truth, Eq.
2.18.
O =
I(W )
I(I)(2.15) S =
W (I)
I(I)(2.16)
C =
W (I)
I(W )
W (W )
I(I)
(2.17) T ={I(I) (2.18)
Science is a formal knowledge body overall oriented to know and to know how entities
are. Technology is a formal knowledge body overall oriented to know how to do entities.
Formalization of the model driven paradigm 25
No sensing would lead to inspired knowledge, 2.19. Depending on the origin of in-
spiration it can be intuitive (endogenous) or revealed (exogenous), respectively. The
subjective truth will be no sensing and interpretation identity, Eq. 2.20. Art, religion,
hallucination, fantasy, illusion, aristotelianism, etc. produce inspired knowledge.
I =
N(W )
N(I)(2.19) TS =
{N(I) (2.20)
As special consideration, you can note that a considerable amount of the knowledge
engaged at a school is perceived by the students as revelation, or as very light par-
tial induction at most, which is the expression of the observational method nearest to
revelation: the Bohr’s atomic model, Darwin’s evolutionism, The Roman Empire, etc.
2.4. Classifying the design factors
Considering a problem as a statement that establishes a constraint among variables,
which must be clarified, definitions for the terms solution, design decision and model
have been proposed. These definitions provide a basis to classify the design factors and
to order the classes of factors involved in obtaining solutions.
2.4.1. Empiric evidence to classify the design factors
Design factors are contained into the answers to the questions underlying the generic to
do questions: the ingredients of the dressing, its quantities and the order to mix them.
A phenomenological analysis suggests the possibility of classifying them.
Taking as examples the problem how to make a keyboard, the problem how to elaborate
a salad, and the problem how to do the access control at a room, the answer to the to
do questions are shown in tables 2.4, 2.5, 2.6 and 2.7, respectively. The column Feature
in these tables contains a characteristic of the corresponding answer concerning the role
that it plays in the solution. The symbols of the column Class identify the possible
category of each answer.
In the example to design a keyboard, an answer could concern to a superset of the
invention keyboard conceived as an instance of problem (device). This answer will be
related to a keyboard model (M). Nevertheless, other answers could concern to a subset
of the elements that compound the keyboard conceived as an instance of solution (input,
key, click). We will call them essential factors of the invention that is being designed (E).
But the answer could too concern aspects belonging to the context where the activity
Formalization of the model driven paradigm 26
Table 2.4: The answers to the questionnaire to make a keyboard can concern to amodel of the invention keyboard (M), to essential factors of the keyboard (E), or to
circumstantial factors of the activity of design the keyboard (C).
Classes of answers to make a keyboard
Question Answer Feature Class
What is it A deviceThe answer is an abstractionof the problem
M
What is it forTo input characters andorders
The answer is contained inthe solution
EWith what is itdone
With keys, circuits,chassis, etc.
The answer is contained inthe solution
HowTransforming clicks inelectrical signals
The answer is contained inthe solution
When The next weekThe solution does not dependof the answer
CWho An electronic engineer
The solution does not dependof the answer
How much Up to 50 eThe solution does not dependof the answer
Where, from who,for who, etc.
. . .The solution does not dependof the answer
of design happens (planning, expert, cost). They are the circumstantial factors of the
activity of making a keyboard (C).
In the example of designing a salad, similarly, the answers could concern to the set food
as a model of salad (M), or concern to features, parts or ingredients (essential factors,
E), or even refer to the context of preparing the salad (C).
In the example of access control at a room, similarly, the answers may concern to an
abstraction of the access control seen as a problem (M), to features or modules of the
access control (E), or to the context of the designing process (C).
Finally, in the example of Gait-A, the answers may concern to the model vision based
system (M), to algorithms, features and modules (E), or to the context of researching
process (C).
Formalization of the model driven paradigm 27
Table 2.5: Classes of answers to the questionnaire to make a salad. Model (M: food),essential factors (E: nutritive, tasty, lettuce, cucumber, olive oil, bed of vegetables,
dressing, etc.), and circumstantial factors (C: planning, expert, cost, etc.).
Classes of answers to elaborate a salad
Question Answer Feature Class
What is it FoodThe answer is an abstractionof the problem
M
What is it forAromatic, tasty, nutritiveand healthy
The answer is contained inthe solution
EWith what is itdone
Lettuce, cucumber,arugula, oil, ...
The answer is contained inthe solution
HowA bed of marinatedvegetables
The answer is contained inthe solution
When At time to be servedThe solution does not dependof the answer
CWho A chef
The solution does not dependof the answer
How much Up to 10 eThe solution does not dependof the answer
Where, from who,for who, etc.
. . .The solution does not dependof the answer
Table 2.6: Answers to the questions to make the access control at a room.
Classes of answers to do a control access
Question Answer Feature Class
What is it A systemThe answer is an abstractionof the problem
M
What is it for SecurityThe answer is contained inthe solution
EWith what is itdone
Electronic technologyThe answer is contained inthe solution
How Access protocolsThe answer is contained inthe solution
When Before three monthsThe solution does not dependof the answer
CWho An electronic engineer
The solution does not dependof the answer
How much Up to 500 eThe solution does not dependof the answer
Where, from who,for who, etc.
. . .The solution does not dependof the answer
Formalization of the model driven paradigm 28
Table 2.7: The answers to the questionnaire to make Gait-A can concern to its model(M), to essential factors of (E), or to circumstantial factors of the activity of design
Gait-A (C).
Classes of answers to make Gait-A
Question Answer Feature Class
What is it A vision based systemThe answer is an abstractionof the problem
M
What is it forFor the early detectionof frailty and senilitysyndromes
The answer is contained inthe solution
E
With what is itdone
With low-cost,sustainable andnon-intrusive devices
The answer is contained inthe solution
HowAnalysing gait featuresof subjects
The answer is contained inthe solution
When In three yearsThe solution does not dependof the answer
CWho A research team
The solution does not dependof the answer
How much Up to 50.000 eThe solution does not dependof the answer
Where, from who,for who, etc.
. . .The solution does not dependof the answer
2.4.2. The relation scope of design decision
The existence of the partition that appears in the three previous examples {M,E,C},considering the fundamental theorem of equivalence relations, indicates that an equiv-
alence relation among the design factors exists. Moreover, we can propose a statement
that performs such equivalence relation.
Let be a problem:
pi ≡ 〈di, ci〉 (2.21)
Let be a model of this problem:
Mn(pi) ≡ 〈dn, cn〉 /(δ(ci) = cn) (2.22)
Let be a design factor (equivalently, the design decision that gives us that factor):
qt ≡ 〈dt, ct〉 (2.23)
Formalization of the model driven paradigm 29
We define the scope, σ, of a design factor in a problem, pi, inside a model, Mn, as the
set of problems concerned to the design decision. The scope can be:
σ ≡ {pi,Mn} (2.24)
σ ≡ {pi} (2.25)
σ ≡ ∅ (2.26)
Definition 2.5. The statement scope of the answer establishes an equivalence relation
in the set of design factors (Σ,∼).
qu(pi) ∼ qv(pi)⇒ σ(qu) = σ(qv) (2.27)
The scope of the answer to each to do question determines whether that answer concerns
the model and the problem (M), only the problem (E), or even the process to obtain
the solution (C), i.e., to build a prototype. Therefore, the scope of the answer indicates
the field of design decision that carries with it the answer: the model, the peculiarities
of the problem, or circumstances of the resolution (context).
Proof. The scope of the answer satisfies the following properties:
Reflexivity: ∀u, , qu ∼ qu
qu ∼ qu ⇒ σ(qu) = σ(qu) (2.28)
Symmetry: ∀u, v, , qu ∼ qv ⇒ qv ∼ qu
qu ∼ qv ⇒ σ(qu) = σ(qv)⇒ σ(qv) = σ(qu)⇒ qv ∼ qu (2.29)
Transitivity: ∀u, v, w, , (qu ∼ qv) ∧ (qv ∼ qw)⇒ qu ∼ qw
(qu ∼ qv) ∧ (qv ∼ qw)⇒ [σ(qu) = σ(qv)] ∧ [σ(qv) = σ(qw)]
⇒ σ(qu) = σ(qw)⇒ qu ∼ qw(2.30)
There are three classes of factors P (F ):
P (F ) = {M,E,C} (2.31)
Formalization of the model driven paradigm 30
Design factors obtained answering the question what is the problem concerns the
problem and the model, therefore, they express an abstraction of the problem,
giving us a generalization of the problem.
∀pi ∈ Π,Mn(Π), qt(pi)/(δ(ci) = ct) ∧ (δ(cn) = ct)⇒ qt(pi) ∈Mn(Π) (2.32)
Design factors obtained answering the questions what for, with what, how concerns
parts of the problem. Subsidiary design decisions of these questions clarify local
aspects. This class of factors comprises the subset of the design factors which
are constituent parts of the solution and, for that reason, we call them essential
factors: components, properties, structure, modules, etc.
∀pi ∈ Π,Mn(Π), qt(pi)/(δ(ci) = ct) ∧ (δ(cn) 6= ct)⇒ qt(pi) ∈ pi ∧ qt(pi) /∈Mn
(2.33)
Circumstantial factors are obtained by answering the remaining to do questions
when, who, how much, where, whom, .... The context solves issues related to the
process of getting the result: cost, design equipment, planning, place, etc.
∀pi ∈ Π, qt(pi)/(δ(ci) 6= ct)⇒ qt(pi) /∈ pi (2.34)
Figure 2.3 represents the scope that design decisions have regarding the scope that
clarify the problem or the aspects of the invention that help to create. The meaning
of each to do question helps to do intuitive the correspondence of design factors and
its associated decisions. The interpretation of the meaning of the three kinds of design
factors in relation to the invention comes to be evident.
It could be the case that we decide to consider that some genuinely circumstantial
factors belong to the model. For example, the factor salad to have maximum freshness
is a genuine response to the question when but can also be obtained in response to
the question what for if the restaurant would consider that the freshness of salad is a
fundamental issue.
From a formal point of view, the same design factor can be obtained as an answer to the
question what for and to the question when, therefore, the rigour of the classification
would be broken.
By convention, to maintain the rigour of the classification, we impose the prevalence
of the conditions established in the conjecture above the common or genuine nature of
design factors.
Formalization of the model driven paradigm 31
HOW MUCH
WHO
etc.
WHEN
WHERE
invention
WHAT
universe
WHAT FOR
HOW
WITH WHAT
WHOM
Figure 2.3: Classes of design factors and interpretation of their relation with theinvention.
This is a way to maintain causality within the design method that we are formalizing:
if a constraint is part of the hypothesis, it must generate consequences, and it has no
sense to re-obtain this condition as a result of a solving process.
Current knowledge creation approaches correspond to Kuhn’s proposals about research
programs [81]. The social benefit produced by knowledge creation justifies the invest-
ment of public resources and, being those resources public, a regulation is needed. This
led to the enactment of legislation and norms: UNE 16600, law 27/2014 (Spanish gov-
ernment), etc.
Those regulations propose definitions oriented to fiscally deal with R&D and, thus, there
are ambiguities that provoke some controversies. Taxonomies of R&D are also proposed
in those regulations. Some of these divide R&D in basic research, applied research and
development. This is more referred to a scientific and technical approach but there is also
ambiguity because it is not a disjoint classification but an arbitrary one. Other proposed
taxonomies are composed of applied research, technological development, precompetitive
research, etc. The disjoint nature of the proposed taxonomies produces a great amount
of misunderstanding and confusion regarding to which class is the appropriate for certain
project and thus which fiscal regulation applies.
Our proposed classification could provide a formal basis to indulge some rigour to pro-
pose a proper taxonomy of R&D:
Formalization of the model driven paradigm 32
Resolving the model subproblem produces abstract knowledge, i.e. the rules that
govern all the problems included in that model. This is related to what we know
as research or basic research because it creates theoretical knowledge and a theory
can be interpreted as a large model.
Resolving the specific subproblem produces particular knowledge by concretion of
a model. This is related to the notion of innovation if it resolves an instance of
the problem itself or development (or applied research) if it resolves a submodel or
a restricted family of instances of the problem.
Resolving the contextual subproblem produces procedural innovation.
The formal rigour of this classification could contribute to the social system management
of the R&D (I+D+i).
2.5. Ordering the classes of design factors
The classes of design factors that we have obtained through the classification criterion
based on the scope of the decisions can be ordered to get criteria to determine what
design decisions have to be made initially and which must be made later.
We define power, P, of a design factor in a problem, pi, under a model, Mn, as the
cardinal of its scope.
P = card(σ) (2.35)
The domain of power is:
P ≡ {0, 1, 2} (2.36)
Definition 2.6. The statement power of a design factor orders the classes of design
factors: (P (F ),≤)
qu ≤ qv ⇒ P(qu) ≤ P(qv) (2.37)
Proof. The power of a design factor satisfies:
Reflexivity: ∀u, , qu ≤ qu
qu ≤ qu ⇒ P(qu) ≤ P(qu)⇒ card(σ(qu)) ≤ card(σ(qu)) (2.38)
Transivity: ∀u, v, w, , (qu ≤ qv) ∧ (qv ≤ qw)⇒ qu ≤ qw
(qu ≤ qv) ∧ (qv ≤ qw)⇒ [P(qu) ≤ P(qv)] ∧ [P(qv) ≤ P(qw)]
⇒ P(qu) ≤ P(qw)⇒ qu ≤ qw(2.39)
Formalization of the model driven paradigm 33
Antisimmetry: ∀u, v, , (qu ≤ qv) ∧ (qv ≤ qu)⇒ qu = qv
(qu ≤ qv) ∧ (qv ≤ qu)⇒ [P(qu) ≤ P(qv)] ∧ [P(qv) ≤ P(qu)]
⇒ P(qu) = P(qv)⇒ qu = qv(2.40)
Totality: ∀u, v, , (qu ≤ qv) ∨ (qv ≤ qu)
(qu ≤ qv) ∨ (qv ≤ qu)⇒ [P(qu) ≤ P(qv)] ∨ [P(qv) ≤ P(qu)] = 1 (2.41)
Assimetry: ∃u, v, , qu ≤ qv ⇒ qv � qu ⇔ qv > qu
(P (F ), >)is a strict total order
C < E < M (2.42)
The conclusion is that there exists formal precedence that establishes what is the correct
sequence to make decisions regarding the design factors.
circumstantialfactors < essentialfactors < model (2.43)
Then, we have a causally coherent procedure to create an invention. It consists of three
type of activities:
A Decide the model which will serve as reference to specify the problem.
B Make design decision to obtain a concretion of the solution from an abstract model,
i.e., resolve the specific subproblem.
C Establish the context.
Make design decision will consist of the following sequence:
1. Solve the model subproblem choosing an existing one or creating a new one. The
involved paradigm is abstraction.
2. Solve the specific subproblem determining the essential factors. The involved
paradigm is concretion (design).
Formalization of the model driven paradigm 34
3. Solve the contextual subproblem establishing the circumstances to produce a pro-
totype. The involved paradigm is projection.
The starting point to make a prototype is the project obtained by mean of designing.
Therefore, to make the prototype in a causally coherent way, all we have to do is to
proceed in the reverse sequence:
1. Materialize the context (contextual subproblem).
2. Assembly components and modules (specific subproblem).
3. Confirm (model subproblem).
Note that the same sequence of producing, coherently, a prototype is the same sequence
that corresponds to the production of many instances, e.g., mass production.
Figure 2.4 represents the causally coherent sequence of making design decisions starting
from an hypothesis, as well as its corresponding process to reach a thesis or develop a
prototype of the invention that solves the problem.
context < essential factors < model
HOW MUCH
WHO
etc.
WHEN
WHERE
invention
WHAT
universe
WHAT FOR
HOW
WITH WHAT
WHOM
model > essential factors > context
hypothesis
thesis
DESIGN
PROTOTYPING
Figure 2.4: Causally coherent sequence of making design decisions and its correspond-ing process to implement a prototype.
Formalization of the model driven paradigm 35
2.6. Model driven paradigm
We have started from the statement of an hypothesis about a device, invention, system
or process, i.e., the problem. We consider the hypothesis expressed as a conjecture of
constraints on the elements contained in it, i.e., the problem statement. We seek to
obtain the invention, system or process, i.e., a solution by transforming the elements
and constraints contained in the problem statement. The choice of the transformations
performed on the problem statement to obtain a solution are design decisions of the
invention, the system or process, i.e., to solve the problem. We abstracted the activity
of making design decisions as the answer to the to do questions, that is, what to do
to do something or alternatively what to do to obtain the solution to a problem. As an
immediate consequence, we have proposed that the nature and effects of design decisions
are related with intuitive underlying to do questions: with what, how many, where, how,
etc., as design decisions are precisely the answers to those questions.
We have proposed the concept of scope of design decisions to indicate that the elements
and conditions of the problem are affected by each design decision. The scope classifies
decisions into three classes, the model of the problem, the constituents of the solution,
and the configuration of the solving activity.
On the other hand, the power of design factor strictly orders the classes of design factors.
That is, the number of elements and constraints of the problem that are involved in every
design decision, provide criteria to decide in what order design decisions must be made.
The hypothesis is the milestone that triggers the activity of design and, in essence, it
expresses very abstractly the solution, so it falls within the scope of the model. Therefore,
to be consistent with the power of the factors on which decisions are made, the formally
correct sequence to make design decisions is: first decide the model that will characterize
the solution, then make decisions about the essential factors of the solution and, finally,
decide on the context in which the creation of such solution will take place.
In conclusion, the scope of design decisions produce classes of factors that can be ordered
by its power, which constitutes a formal approach of model driven design.
The causal procedure of making design decisions that we have developed up to this point
concerns the global aspects of problem solving: model, constituents and context. The
main aspects are:
Design decisions related to the models essentially are directed to define the frame-
work where the solution will be obtained. So, reasoning elements will belong to the
domain of the problem but the results would belong to the domain of the solution.
Formalization of the model driven paradigm 36
Design decisions related to the essential factors are those that elucidate the con-
stituent ingredients of the technological solution, the interface between the solution
and the rest of the world, and the other aspects that constitutes the solution it-
self, that are structural and organizational. The instrumental purpose of these
decisions are substantial, related to composition and constitution of the solution.
Design decisions related to the context are procedural, and establish the ambient
where the solution will occurs.
The model oriented design we have formally justified to obtain a solution to a prob-
lem, tries to obtain a solution through a divide et vinces strategy which canonically
decomposes the initial problem into three subproblems:
Model subproblem of the problem, i.e., if food is a model of the invention salad
then the problem can be: a salad is food.
Specific subproblem comprising the constraints between the model and the initial
problem, i.e., a salad is an aromatic, tasty, nutritious, and healthy food.
Contextual subproblem, i.e., a cook elaborates salad at the moment to be served.
Who and when to do it questions have been answered.
The scope of the problem statement is, therefore, the domain of observable phenomena,
free of subjectivity that can be incorporated by perception and interpretation. The
design decisions are related to the filters of observation and interpretation. The scope
of the statement of the solution is, therefore, the domain of knowledge. Thus, a solution
to a problem is knowledge that is created from an observable effect, through perception
and interpretation.
Starting from an observable phenomena, proposed as a problem, the perception process
gives us the epistemological approach to divide the initial problem in the three separated
subproblems we proposed before:
– Model subproblem that is an abstract interpretation of the problem.
– Specific subproblem that is the same initial problem but focused in that model.
– Contextual subproblem that does not have constraints coming from the initial
problem. Its constraints are related to the model and the specific subproblems.
Formalization of the model driven paradigm 37
The model driven paradigm aims at solving a given problem by referring it to one of its
models. Consider the abstraction of the problem pi to one of its models mj : Ψ(pi) = mj ,
a series of transformation over the initial conjecture T (α) expresses the solution as a
constraint in which the model is included:
T (α) = {q1, q2, . . . , qn} = ω ≡ 〈dk, ck〉, ,mj ≡ 〈dj , cj〉, , dj ⊇ dk ∧ cj ⊆ ck (2.44)
From this approach, the first stage of design we propose is to break the problem into
the following three subproblems:
Model subproblem. It is the part of the problem concerning the variables
and constraints on them that this problem shares with other similar problems.
Resolution of the model subproblem consists of making design decisions at the
level of the models, i.e., food is a model of the invention salad. Having determined
a model, you can define a problem in terms of the model: a salad is food, gait is
anthropomorphic upright self-displacement.
Specific subproblem. It is the part of the problem concerning the variables and
constraints on them that this problem does not share with other similar problems.
Resolution of specific subproblem consists of making design decisions involving the
aspects of the problem that does not solve the model, that is, to solve a specific
subproblem implies activities of values and constraints concretion in the model.
For example: a salad is an aromatic, tasty, nutritive and healthy food based on
a cold macro mixture of raw or previously cooked vegetables and raw oil based
dressing, and salted spicy seasoning, and perhaps other ingredients, and with no
added water. Gait is an alternating stepping of two feet, with no additional fulcra,
keeping at least a point of support at every time, on a horizontal or slightly inclined
surface.
Contextual subproblem. It represents the circumstances around the resolution
of the problem. Therefore, it is not a part of the invention but it is related to the
activity of solving the problem. Contextual problem resolution is to establish the
values and constraints of the variables that, not being a part of the problem, are
involved in its resolution. For example: a cook elaborates salad at the moment
to be served. It has been established who and when will the salad be elaborated.
Or the Gait-A memory memory for the application of the MINECO 2013 call for
projects.
Figure 2.5 shows the three subproblems in the context of the model-driven paradigm.
Note that the figure is almost empty as it will be filled as we advance in our proposal.
Formalization of the model driven paradigm 38
CONTEXTUAL
SUBPROBLEM
CAUSALLY
FORMAL DESIGN
MODEL
SUBPROBLEM
SPECIFIC
SUBPROBLEM
INTERNAL
VIEW
predominantly
solution’s
domain
EXTERNAL
VIEW
predominantly
problem’s
domain
STRUCTURAL
SPECIFICATION
EXECUTION
PROJECT
FORMAL
SPECIFICATION
Figure 2.5: Model-driven paradigm
Decide a model for the problem restricts its solution into the domain of the model and
therefore it causes a prune into the tree of possible solutions. Expressing the problem
starting in one of its models provides a comprehensive definition of the problem and, as
such, its value is declarative for the problem.
Characterization of the specific problem, in turn, will express the complementary con-
straint between the problem and its model. Therefore, in the general case, it will involve
Formalization of the model driven paradigm 39
all the elements of the problem. The constraint of the specific subproblem could be
obtained by making exact and accurate transformations into the hypothesis.
So far we have presented, phenomenologically, the model driven paradigm as commonly
accepted by the scientific community. Following sections will provide a formal justifica-
tion of this paradigm. On one hand, we will provide a justification of the subproblems
and, on the other hand, we will provide the procedural sequence to follow.
2.7. Design decisions about models
Frequently, engineering problems seem to be solved by first establishing the context of
the solution, then deciding on the constituents and finishing by characterizing them as
members of a family of inventions. This is a correctly ordered technique in terms of
the power of design factors: the sequence of design decisions begins with the shorter
range ones and continues progressively for longer range ones. However, as the elements
and constraints of the hypothesis are not the ones affected by minor decisions, this
sequence of design decisions is inconsistent with the nature of the activity of design.
In fact, as the minor decisions operate on elements and constraints that are a function
of the elements and constraints of the hypothesis, this technique of design imposes the
additional hypothesis of characterizing the nature of the solution in advance. This is a
formal contradiction except, perhaps, for cases where this precondition is justified.
Taking as an example the serial production of a device using an assembly line, circum-
stantial factors of device manufacturing are represented in the architectural character-
istics of the assembly line (its robustness, being single or multiple flow, etc.), the device
essential factors are represented in the operation of the assembly line (implantation of
components, assembly of the modules of the device, and quality control to verify satisfac-
tion of the performance of the manufactured device), and the device model is abstracted
in the configuration of the assembly line.
The engineer constructing the assembly line can set three assembly flow channels, and
seven stages of the assembly line length. Conceptually, it is as much as deciding on how
much productivity and how to mount the device. Clearly, here, the precondition is that
the device had to be designed in advance.
Redefining the early decision on the nature of the solution as a constraint explicitly
or implicitly contained in the hypothesis, is an interpretation that does not alter the
outcome and, instead, is consistent with the formality of making decisions progressively
Formalization of the model driven paradigm 40
from minor to major. The tacit assumption for the manufacturing line is that it pro-
duces a device according to a given prototype which constitutes, at the same time, the
hypothesis or the specifications of the assembly line.
Particular problems such as the realization of devices that have previously been designed,
can be confusing because aspects that are usually circumstantial, explicitly or implicitly,
are contained in the hypothesis. We postulated earlier that by the mere fact of being
part of the hypothesis, aspects acquire the category of objective of the problem.
Therefore, the only causally coherent way to proceed in solving problems, making design
decisions, follows the ordered sequence to solve the three subproblems starting from the
model subproblem.
There may be many models of a given problem. As many as results from combining
design factors of the latter and to propose conditions that contain the problem.
In topological terms, we can refer to the space of representation of all design factors
and conceive each problem as the region of the space that satisfies the condition of the
problem. Under this approach, any subproblem of a given problem, including models of
the problem will be another region defined on a subspace of the problem space satisfying
the following constraint: the projection into that subspace of the region of the problem
will have to be contained inside that region.
A model of a set of problems may be understood as a subproblem of each problem, the
model being defined on a space that satisfies the constraint of being a subspace of the
space of each problem in the set, and in such a way that the constraint that characterizes
the model will have to contain the projection of the constraint of each problem, i.e., the
constraint that characterizes the model may arise from the union of the constraints of
the problems.
The paradigm of solving a given problem by means of a model driven strategy is to later
solve other problems that have the same model, by reusing the now obtained solution
for that model. This aim provides criteria to decide a proper model of the initial given
problem. The corresponding technique is described in the following paragraphs.
In the general case, the hypothesis represents the problem we intend to solve in terms,
perhaps, of a model and a compendium of objectives that particularize the problem
from the model, and even requirements to be satisfied by the solution. All this, using an
statement of the jargon of the problem domain, but such expression could be inaccurate,
vague, ambiguous, and even erroneous.
Due to that possibility of the hypothesis to be incorrect, we avoid calling problem state-
ment to the initial version of the hypothesis or conjecture. Therefore, we reserve the
Formalization of the model driven paradigm 41
term problem for a formally correct and accurate statement that would be obtained by
transforming the hypothesis, lexically, syntactically and even semantically. The accurate
version of the problem would consist of:
A statement to generalize, that expresses the model, e.g., a vegetable starter plate
is a mixture of vegetables; an access control to a room is a system of protocols to
entry and exit ; gait is the anthropomorphic upright self-displacement.
A statement for concretion, that expresses the condition of the problem as the
conjunction of other lighter conditions extracted from the hypothesis but being
consistent with the model, e.g., a vegetable starter plate is nutritive, tasty, healthy
and aromatic; an access control to a place consists of an entrance having an exter-
nal door and an internal one, which opening is mutually exclusive; gait is produced
in an alternating stepping of two feet, keeping at least a contact point at all times,
with no additional fulcra and in an horizontal or slightly inclined surface.
We call objectives of the problem to the correct and exact conditions that characterize
the problem in respect to the model.
The aim of deciding the model of the problem is thus double: to prune the solution tree
and, on the other hand, specify the framework of the hypothesis.
The technique to decide the model to be proposed for a problem is to answer the question
what is the problem. E.g. the question what is the vegetable starter plate, will obtain
models of vegetable starter plate as answers: cold starter, healthy meal, salad, etc. On
the other hand, the question what is the control access to a room will be answered
with models of it: identify people, system of protocols to entry and exit, etc. The same
way, the qustion what is gait will be answered with models: anthropomorphic upright
self-displacement, horizontal displacement, etc.
Then, there is an intuitive way to decide the model, based on experience, surely. Episte-
mologically, this assignment of a model to the problem falls into the inspirational type of
knowledge and, therefore, is non formal knowledge. This implies that the model has to
be previously known and, therefore, it has to exist. E.g., to decide a model of vegetable
starter plate:
1. Propose several models:
The chef approach: salad.
The diet point of view: healthy food.
Formalization of the model driven paradigm 42
The maitre approach: cold starter.
2. Select one of the precedent models: salad is a mixture of vegetables.
Note that this definition is focused more on elaborating vegetable starter plate than to
serve it or even its nutritional value. A model of vegetable starter plate based on the diet
point of view could be: vegetable starter plate is a hypo caloric mixture of vegetables
and perhaps other ingredients to facilitate the intestinal transit.
On the other hand, a systematically formal way to characterize a model for a problem,
i.e. answering the question what is the vegetable starter plate, would consists of the
following procedure:
1. Propose a set of positive examples consisting of other problems that will belong to
the same model. They will be considered favourable or positive examples.
2. Propose a set of counterexamples consisting of other problems that will not belong
to the model. They will be considered negative examples or counterexamples.
3. Propose a constraint that will establish a region in a subspace shared by all the
positive examples, in such a way that this region contains the projection of each
example. Projection of counterexamples will be completely or partly outside of
this region.
Chapter 3
Formalization of the top-down
method
3.1. Classifying the essential factors
Having a model of the problem, to know how is the solution, we must solve the specific
subproblem. Therefore, additional decisions have to be made in order to concretize
the variables of the problem that have been ignored by the model, and to constraint
the condition that defines the model: concretize the name of implied vegetables, their
proportions, their relations,. . . Later, to know how to elaborate the salad or how to
build the Gait-A prototype, the contextual subproblem have to be solved: when, who,
where,. . .
Then, it can be noted that:
1. The model subproblem let us know the nature of the solution
2. The specific subproblem let us know how is the composition of the solution.
3. The contextual subproblem let us know how to do the solution.
3.1.1. Empiric evidence to classify
The essential factors E are answers to questions what for, with what and how. Proceed-
ing in the same way we classify general design factors, a phenomenological analysis of
essential factors suggests the possibility of classifying them too. Taking as examples to
make a keyboard and to elaborate a salad, Tables 3.1 and 3.2, respectively, shown the
43
Formalization of the top-down method 44
Table 3.1: Empirical classification of answers to questions about the essential factorsto make a keyboard.
Essential questions to make a keyboard
Question Answer Role Class
What is itfor
For input of charactersand orders
The utility of thekeyboard
A
With whatis it made
With keys, circuits,chassis, etc.
The constituents T
HowTransforming pressurein electrical signals
Its functioning S
Table 3.2: Empirical classification of answers to questions about essential factors toelaborate a salad.
Essential questions to elaborate a salad
Question Answer Role Class
What is itfor
Aromatic, tasty,nutritive and healthy
The utility of thesalad
A
With whatis it made
Lettuce, cucumber,olive oil,...
The ingredients T
HowA bed of vegetable andsprayed oil
Its composition S
questions and their corresponding answers to find solutions. The field Role describes the
paper that factors obtained as answers of the questions play in the composition of the
invention. A partition is obtained in both cases. Therefore, based on the fundamental
theorem of equivalence relations, we can say that there is an equivalence relation on the
set of essential design factors. Even more, the partition consists of the following three
classes:
Class A factors, that represent the utility, are the essential factors that the in-
vention explicitly exhibits to the rest of the world. They are perceived from an
external viewpoint. These are the factors of architecture (functionality) of the
invention. We call them explicit factors or architectural factors. Their operational
characteristic is to be drawn from the hypothesis, and are obtained by answering
the question what for. Examples of explicit factors in the case of the salad are
aromatic, tasty, nutritive and healthy.
Class T factors are technological ingredients, i.e., representing the constitution of
the solution. We call them intrinsic factors or technological factors. Their oper-
ational characteristic is that they are elementary by definition, and are obtained
Formalization of the top-down method 45
by answering the question with what. Examples of intrinsic factors in the case of
the salad would be: lettuce, arugula and olive oil.
Class S factors are the constructive factors of the invention. That is, the structural
and organizational aspects of the solution. We call them implicit factors or struc-
tural factors, and are obtained by answering the question how. They represent the
composition of the invention. An example of implicit factors in the case of the
salad would be a core of cheeses on a bed of vegetables with raw olive oil based
dressing.
Therefore, the role of an essential factor is the impact that design decisions have in the
invention, i.e., in the activity of obtaining the solution. The role of the explicit factors
(architecture) corresponds to be deduced from the hypothesis. The role of the implicit
factors (structure) will be induced. The role of the intrinsic factors (technology) has to
be indirect, through the structure.
Figure 3.1 shows intrinsic (components), implicit (organization) and explicit (perfor-
mance) factors corresponding to the example of salad. Figure 3.2 show the same for the
example of Gait-A.
Essential factors to elaborate a “salad”
Intrinsic factors Implicit factors Explicit factors
Technology Structure Architecture
With what How What for
A bed of vegetables with sprayed oil
Figure 3.1: Classes of essential factors: intrinsic factors of the salad, its constituents;implicit factors, the organization of the salad; and explicit factors, the utility of the
salad, that is, the salad itself.
3.1.2. The relation role of an essential factor
We can propose a statement that performs the equivalence relation.
Formalization of the top-down method 46
Intrinsic factors Implicit factors Explicit factors
Technology Structure Architecture
With what How What for
Use of RGB
devices
Run gait test
periodically
Code
optimizationsBuild data set
Background
subtraction
Heel strike
and toe off
Feet location
and trackingSkeleton
extraction
K-Nearest
Neighbors
Dynamic
Time Warping
Principal
Component
Analysis
Vision based
system
Detection of
elderly
syndromes
Reduced
costSustainability
Gait analysis
based system
Non-intrusive
Low
computational
cost
Early
detection
Use of RGB
devices
Run gait test
periodically
Code
optimizations
Build data set
Background
subtraction
Heel strike
and toe off
Feet location
and tracking
Skeleton
extraction
K-Nearest
Neighbors
Dynamic
Time Warping
Principal
Component
Analysis
Record
Play
Get
Silhouette
Get FeetGet Time
series
Test
Multiresolution
Classify
Es ent al f c o s of Gai -
Figure 3.2: Classes of essential factors: intrinsic factors of Gait-A, its modules; im-plicit factors, the organization of Gait-A; and explicit factors, the utility of Gait-A.
Let be a problem:
pi ≡ 〈di, ci〉 (3.1)
Let be a design essential factor, i.e., the design decision that produces it:
et ≡ 〈dt, ct〉 (3.2)
We define the role, R, of a design factor in a problem, pi, as the appearance it has in
the invention.
Definition 3.1. The statement the role that an essential factor has in the created in-
vention establishes an equivalence relation in the set of essential factors.
ei ∼ ej ⇒ R(ei) ≡ R(ej) (3.3)
Proof. Let be the following set of essential factors:
E = {e1, e2, e3, . . . ei, . . . en} (3.4)
Reflexivity: ∀i, , ei ∼ ei
ei ∼ ei ⇒ R(ei) ≡ R(ei) (3.5)
Formalization of the top-down method 47
Simmetry: ∀i, j, , ei ∼ ej ⇒ ej ∼ ei
ei ∼ ej ⇒ R(ei) ≡ R(ej)
⇒ R(ej) ≡ R(ei)⇒ ej ∼ ei
(3.6)
Transitivity: ∀i, j, k, , (ei ∼ ej) ∧ (ej ∼ ek)⇒ ei ∼ ek
(ei ∼ ej) ∧ (ej ∼ ek)⇒ [R(ei) ≡ R(ej)] ∧ [R(ej) ≡ R(ek)]
⇒ R(ei) ≡ R(ek)⇒ ei ∼ ek(3.7)
The set of equivalence classes has the following three elements:
P (E) = {A, T, S} (3.8)
Figure 3.3 shows the classification of the essential factors that we already had but for-
mally obtained now, which incorporates rigour to the distinction among the factors of
architecture (explicit), the structure (implicit) and technology (intrinsic).
HOW MUCH
WHO
etc.
WHEN
WHERE
invention
WHAT
universeWHOM
WHAT FOR
utility
ARCHITECTURE
explicit factors
INTERFACE HOW
constitution
STRUCTURE
implicit factors
WITH WHAT
composition
TECHNOLOGY
intrinsic factors
Figure 3.3: Classes of essential factors and their meaning in the design decisions.
Formalization of the top-down method 48
In the same way, it can be interesting to reconsider the scope of a design factor, the role of
an essential factor that is genuinely technological or structural (e.g., CMOS technology)
could be considered an objective of the problem (a constraint that the solution to the
problem must satisfy). That is, the imposition of causality to the design method we
are formalizing can lead to intrinsic or implicit factors to be part of the objectives
that establishes the problem. In this case, the same factor (CMOS technology) can be
obtained by answering the question what for, which breaks the partition. To avoid it,
we impose the condition of prevalence of the specific of a problem (established in the
hypothesis) over the genuine of an essential factor to be explicit, implicit or intrinsic so
its nature is unique.
3.2. Ordering the classes of essential factors
We propose to establish an order among the three classes of essential factors and use
such order as a criterion to determine which design decisions about the essential factors
have to be carried out before and which ones have to be carried out after.
Definition 3.2. A given essential factor is a part of another essential factor, if the first
is an essential factor of the second (seen as things, systems or facts). Each essential
factor is a part of itself.
The sentence be a part of orders the classes of essential factors:
(P (E),≤) (3.9)
ei ⊂ ej ⇒ ei ≤ ej (3.10)
Proof.
Reflexivity: ∀i, , ei ≤ eiei ≤ ei ⇒ ei ⊆ ei (3.11)
Transitivity: ∀i, j, k, , ei ≤ ej ∧ ej ≤ ek ⇒ ei ≤ ek
(ei ≤ ej) ∧ (ej ≤ ek)⇒ (ei ⊆ ej) ∧ (ej ⊆ ek)
⇒ ei ⊆ ek ⇒ ei ≤ ek(3.12)
Antisymmetry: ∀i, j, , ei ≤ ej ∧ ej ≤ ei ⇒ ei = ej
(ei ≤ ej) ∧ (ej ≤ ei)⇒ (ei ⊆ ej) ∧ (ej ⊆ ei)⇒ ei = ej (3.13)
Formalization of the top-down method 49
Totality: ∀i, j, , ei ≤ ej ∨ ej ≤ ei
(ei ≤ ej) ∨ (ej ≤ ei)⇒ (ei ⊆ ej) ∨ (ej ⊆ ei) = 1 (3.14)
Asymmetry: ∃i, j, , ei ≤ ej ⇒ ej � ei ⇔ ej > ei
(P (E), >)is a strict total order (3.15)
T < S < A (3.16)
3.3. Scientific method and top-down design
The classes of design factors and, therefore, the design activities are ordered:
circumstantial factors < technology < structure < architecture < model (3.17)
Taking into account that:
The hypothesis constitutes the starting milestone of the activity of designing.
The hypothesis expresses the problem from an external point of view, it corre-
sponds to the scope of the model (abstract aspects) and the architecture (specific
aspects).
Then, the formally correct sequence to be followed to make design decisions is what
establishes, in the first place, the model of the problem that will characterize the solution.
Then make decisions about the architecture of the problem, followed by decisions about
the structure and organization of the solution. Later, make decisions about technology
and, finally, decisions about the context in which the creation of such solution occurs.
Then, the coherent sequence of design is:
model→ architecture→ structure→ technology → circumstantial factors (3.18)
Formalization of the top-down method 50
In terms of the to do questions, the coherent sequence of questions is:
what→ what for → how → withwhat→ others (3.19)
Figure 3.4 shows the sequence of design method formally justified by causal coherence.
Note that this sequence represents:
The Top-Down method of engineering design.
The scientific method to create knowledge.
confirm
HOW MUCH
WHO
etc.
WHEN
WHERE
invention
WHAT
universe
WHAT FOR
HOW
WITH WHAT
WHOM
hypothesis
thesis
model
functional spec.
structural spec.
technological spec.
projectcomponents
modules
experiment
context < technology < structure < architecture < model
model > architecture > structure > technology > contextDESIGN
PROTOTYPING
Figure 3.4: Causally coherent sequence of making design decisions and its corre-sponding process to implement a prototype. This is the Top-Down design of inventions
sequence, as well as the scientific method to create knowledge.
Finally, the causally coherent procedure to create an invention consists of the steps
shown in Figure 3.5.
Figure 3.6 summarizes the causal sequence to solve a problem, or to design inventions,
or to create knowledge, emphasizing the two global stages of proposing a solution by
making decisions, and verify correctness of the solution on a prototype. The project of
the prototype is the milestone that connects decisions and prototyping.
On the other hand, the initial conjecture happens in an external way and leads up to
the functional specification of the problem. Most of the decisions usually correspond
Formalization of the top-down method 51
Solve the model
subproblem by
abstracting a family of the
problem
Solve the architectural
subproblem determining
the explicit factors
Solve the technological
subproblem determining
the intrinsic factors
Solve the contextual
subproblem establishing
the circumstances to
produce a prototype
Confirm
Experiment
Assembly the constituents
Materialise the context
Make design decisions Make a prototype
{Architecture
Structure
Technology
Context
ModelAbstract
Concretion
Project
Figure 3.5: Steps to follow with the causally coherent procedure to create an inven-tion.
to an internal point of view of the problem. Finally, the process of validating the
solution against the objectives of the problem takes place from an external point of
view. Decisions mostly related to the problem belong to the external point of view and
decisions related to a solution belong to the internal point of view.
3.3.1. Study case: Vegetable starter plate
To characterize the vegetable starter plate, we can propose the following positive cases
to obtain a model:
Murcian salad is a macro mixture of tomato, onion, canned tuna, olive, boiled egg,
olive oil, salt, and vinegar.
Pipirrana is a macro mixture of tomato, onion, green pepper, cucumber, garlic,
bread, olive oil, and salt.
Caesar salad is a macro mixture of lettuce, toast bread, parmesan cheese, and
caesar sauce.
Formalization of the top-down method 52
when
how much
where, etc.
} ith h t h f h t h twhat h t f h ith h t {
execution projectDESIGNING PROTOTYPING
idea t t
Figure 3.6: There is an external interface that connects functional and structuralspecifications, and an internal interface that connects decisions and prototyping.
Waldorf salad is a macro mixture of lettuce, celery, apples, nuts, raisins, and
mayonnaise.
Tabbouleh is a macro mixture of semolina, onion, parsley, mint, tomato, lemon
juice, olive oil, and salt.
Olivier salad is a macro mixture of boiled potato, boiled carrot, boiled egg, canned
tuna, mayonnaise, olive, canned pepper, and salt.
Escalivada is a macro mixture of aubergine, red pepper, onion, tomato, garlic,
olive oil, and salt.
As counterexamples, the following:
Gazpacho is a cold micro mixture of tomato, cucumber, pepper, bread, garlic,
vinegar, water, olive oil, and salt.
Vichyssoise is a cold micro mixture of leek, potato, onion, butter, cream, water,
olive oil, and salt.
Stew is a cooked macro mixture of chickpea, chicken, blood sausage, carrot, bacon,
cabbage, garlic, potato, water, and salt.
Paella is a cooked macro mixture of chicken, rabbit, garlic, tomato, rice, pepper,
olive oil, saffron, water and salt.
Boiled vegetables is a cooked macro mixture of potato, green beans, carrot, onion,
olive oil and salt.
Formalization of the top-down method 53
Grilled vegetables is a macro mixture of courgette, aubergine, pepper, asparagus,
onion, mushroom, olive oil and salt.
As can be seen in Table 3.3, we have considered the following set of variables:
V = {mixture, vegetables, dressing, salt,
water, other, spicy seasoning, temperature}(3.20)
A model of the set of positive examples of salads that does not represent any counterex-
ample is:
Msalad =<mixture : MACRO > ∧ < vegetables : RAW
∨ PREV IOUSLY COOKED > ∧ < dressing : RAW OILBASED >
∧ < salt : Y ES > ∧ < water : NO > ∧ < other : PERHAPS >
∧ < spicy seasoning : Y ES > ∧ < temperature : COLD >
(3.21)
This functional expression can be said in natural language as follows: Salad is a cold
macro mixture of raw or previously cooked vegetables, raw oil based dressing, salted spicy
seasoning, and perhaps other ingredients with no added water.
Table 3.3: The projection of each positive example of salad fall into the region SALADbut any counterexample projection is not entirely inside the region SALAD.
Deciding a a model for vegetable starter plate
mixture vegetables dressing salt water other spicy temp.EXAMPLES
Murcian salad macro raw raw oil based yes no egg vinegar coldPipirrana macro raw raw oil based yes no bread mix cold
Caesar salad macro raw raw oil based yes no bread mix coldWaldorf salad macro raw raw oil based yes no fruit celery cold
Tabbouleh macro raw raw oil based yes no semoline mix coldOlivier salad macro previously boiled raw oil based yes no tuna vinegar coldEscalivada macro previously boiled raw oil based yes no oil garlic cold
SALAD macro raw previously boiled raw oil based yes no perhaps yes coldCOUNTEREXAMPLES
Gazpacho micro raw raw oil based yes yes bread mix coldVichyssoise micro previously boiled fried oil based yes yes cream almond cold
Stew macro just boiled animal fat yes yes meat mix hotPaella macro just boiled fried oil based yes no meat saffron hot
Boiled vegetables macro previously boiled raw oil based yes no nil nil coldGrilled vegetables macro just boiled raw oil based yes no nil nil hot
In the access control example, the following cases can be favourable:
Double door security access to bank offices.
Formalization of the top-down method 54
Cattle transfer dock.
Sluice gate to compensate marine level differences.
Point to point connection channel between a personal computer and a server.
As counterexamples:
The climate control in public buildings based in a twin door system since it leaves
entry and exit without restrictions.
An earthenware pitcher, though this will provide little or no information to define
the model.
A model for the problem of controlling access to a jewellery can be: method to concur-
rently access in “entry” mode or “exit” mode, and the two modes in a mutually exclusive
manner. This is the so-called critical section problem in operating systems of computers
[82].
At this point, being solved the subproblem model, we know the nature of the problem:
a cold macro mixture. . . . Note that the model plays the role of an abstract pattern of
the problem.
3.3.2. Study case: Gait analysis using computer vision
The problem of early detection of frailty and senility syndromes was used to exemplify
the proposed method with a real problem. Selection of a framework model through
the projection of examples and counterexamples to propose a definition of gait and to
propose variables is performed using the proposed methodology. This provides the first
prune in the tree of possible solutions allowing to delimit the state of the art analysis.
Computer vision to characterize environment and human behaviour is part of our re-
search area related to technological systems for AAL. In particular, we are working in
early detection of frailty and senility syndromes in elderly people through the analysis
of gait using computer vision.
The main advantages of visible light based systems are: innocuousness to people and
environmental sustainability. The former allows to extract information from the ob-
ject of study without affecting the analysed phenomenon or system. This remark is
more referred to be consequent with the postulates of Physics, in the meaning of what
Heisenberg established [83], than a practical context, because at a macroscopic level, the
Formalization of the top-down method 55
physic of capturing images does not affect gait. However, we consider appropriate this
innocuousness, especially to remark that most of the sensors are intrusive, i.e., they are
transducers that flood the environment and the subject of study with energy (infrared
light, acoustic emissions, microwaves, etc.). Other systems require the cooperation of
the subject of study or attach some devices to certain parts of the body. In those cases,
the subject of study might be affected by the direct effect of the physical phenomenology
or by the subjective incidence derived from the cooperation of the subject. The latter
means that the amount of energy that is thrown to the environment grows with the
power of the devices and with the amount of them that are operating in a specific region
of space and time. This accumulative effect can overflow the healthy limits. This is the
reason why it is necessary to watch over the using of such systems in order to control
the excessive accumulation in a certain place and time.
Let p be the problem of gait analysis performed by a specialist:
p ≡ 〈E,G,H,A〉 (3.22)
Where E represents the environment, G the subject’s gait, H the observer (usually a
human specialist) and A the analysis. The subject’s gait is located in the environment
E(G) and the specialist obtains the information of gait by observing it:
H(E(G)) (3.23)
Making a correspondence with what we proposed in the previous chapter about concepts
of knowledge, H(E(G)) = S(E(G)), that means, the human senses the gait in the
environment. This process, as it is performed by a human, it contains subjectivity,
therefore the observation produces a filtering f(x) of the real gait:
H(E(G)) = H(f(E(G))) ' G (3.24)
Then this filtered version of gait is analysed to render a diagnostic:
A(H(f(E(G))) = A(' G) (3.25)
The correspondence with the concepts of knowledge will be A(' G) = M(' G) where
M(' G) = M(S(f(E(G)))) and therefore M(S(f(E(G)))) = K(' G), which means
that the human obtains knowledge of a filtered version of gait through its perception of
it which, in time, is a double filtered sensitive and mental process. This mental analysis
is also performed by a human specialist and, thus, it is also subjective.
Formalization of the top-down method 56
Let p′ be the problem of computer vision gait analysis:
p′ ≡ 〈E,G,C,A〉 (3.26)
Where E represents the environment, G the subject’s gait, C the camera and A an
analysis. We intend to obtain an analysis of gait A(G), however, an independent analysis
of gait is not possible because the subject’s gait G is located in the environment E(G)
and all of it is recorded with the camera C(E(G)). What we get instead is the camera
recording analysed:
A(C(E(G))) (3.27)
We need then a function f(x) that allows to extract the subject’s gait from the camera
recording:
f(C(E(G)) ' G (3.28)
Similar to the problem of gait analysis by a human specialist, the obtained gait is a
filtered version of it, the difference in this case is that the filter is performed by a
computer. The computer is not altered by emotional state or due to experience and its
results can be reproduced, therefore, the objectivity is increased. The same applies to
the analysis A which will be performed by a computer as well.
The environment E ≡ 〈B,L〉 is composed of a background B and light conditions L, we
impose the constraint of B being static and L being luminous. The camera C ≡ 〈F,R,D〉has a field of view F , a resolution R and a distance D to the subject, the constraint here
will be that the values of F , R and D provide a sufficient amount of steps and sufficient
resolution of the subject. The subject’s gait G is more complex and, thus, it will be
more thoroughly defined by providing a definition of it. The analysis A and the function
f(C(E(G)) will be the proposed method for gait analysis that will be developed. Making
the correspondence with the concepts of knowledge, we will developed the functions of
perception P ≡ 〈M,S〉 in order to obtain knowledge of gait K(G).
The fact that there exists a correlation between some disease syndromes that affect
elderly people and gait analysis has already been evaluated in Chapter 1. However, our
aim transcends the analysis of this group of people to open up to any range of age and
activities in areas such as sports, rehabilitation, children syndromes and a whole variety
of cases. Throughout our review of the state of the art, we have observed that the
parameters of gait analysed are mainly related to the physiology of gait, which we find
causally coherent in that a phenomenon is described in terms of the parameters that
regulate it. At such effect, we have been able to find several definitions of gait:
Gait is defined by Roy et al. [56] as a periodic activity in which each cycle is composed
by two steps: right foot forward and left foot forward. This definition lacks some details
Formalization of the top-down method 57
and, therefore, it only has global declarative value (related to the most notorious part
of gait) but it is not valid to conceive operations and algorithms for the gait analysis
taking into account the detail we intend to perform. Nixon and Carter [84] define gait
as a kinematic problem, so it has static and dynamic aspects. Static components are
related to physical construction of a person from the different gait phases. The dynamic
components are related to the temporal changes of a person during a gait cycle. Gait is
also defined as a sequence of repetitive movements that varies over time and from one
subject to another by Perry et al. [31]. These definitions involve some variables of the
physical movement of a person and, perhaps, physiological parameters, and even from
psychology.
Our visual analysis of gait focuses on the study of the information extracted from sequen-
tial images taken from the subject’s movements. Although transduction is performed
from visible light, the analysis produces an indirect measure of kinematic magnitudes,
that is, movement without considering its causes. Computer vision enables us to sepa-
rate the results from the gait dynamic approach, those dealing with physiological and
emotional forces involved in movement, which may seriously compromise the required
objectivity of the study. Our study depends on the classification system to draw in-
ferences from the spatio-temporal parameters the computer vision system delivers to
categorise gait parameters, and tentatively pinpoint those of them that may possibly
lead to disease syndromes.
The absence of sufficiently objective and complete formal specification of gait definition
to provide coverage to computer vision analysis has led us to propose a new kinematic
specification of human gait using the method proposed. The process to obtain a formal
specification of gait requires to decide a reference model and to define the objectives
that gait should satisfy. To delimit the model, we have considered the cases shown in
Table 3.4, cases we consider human gait as well as other cases we do not consider as
human gait for our problem (note that those might be considered examples of human
gait in another problem). This will provide the definition of the parameter G presented
in equation 3.26.
A definition that includes all the examples and none of the counterexample shown in
Table 3.5 is as follows:
Definition 3.3. Gait is the anthropomorphic upright self-displacement, in an alternat-
ing stepping of two feet, with no additional fulcra, keeping at least a point of support
at every time, on a horizontal or slightly inclined surface.
Formalization of the top-down method 58
Table 3.4: Examples and counterexamples of gait
Examples Counterexamples
Gait of elderly peopleGait of an adult person
Gait of a childSport gait
Walking over a treadmillWalking under psychotropic influence
Upright translation of apesUpright translation of bears
A bipedal robot walkingWalking with an exoskeleton
Translation of other animalsWith more or less than two legs
SwimFlyRun
ClimbWalking with crutches
Up or down ramps or stairsStand on a mechanic stairs
Crawl
Manthropomorphic upright self displacement =
< feet movement : ALTERNATING STEPPING 2 FEET >
∧ < medium : HORIZONTAL SURFACE
∨ SLIGHTLY INCLINED SURFACE >
∧ < additional fulcra : NO > ∧ < minimum contact points : 1 >
(3.29)
Now that we have decided that the model is anthropomorphic upright self-displacement
we can consider the kinematic variables that belong to that model as well as others
that belong to specific subproblem of gait. Table 3.6 shows the variables taken into
consideration.
3.4. Formal specification of a problem
We have said that the hypothesis belongs to the domain of the problem. Nevertheless, its
potential incorrectness, inaccuracy and imprecision can include extra requirements that
has to be satisfied by the solution. This incorporates preconditions that can prematurely
prune the potential solution tree.
Once the framework of the hypothesis has been precisely established through a model
of the problem, to complete the transformation of that hypothesis in a correct, accurate
and precise expression, avoiding preconceptions about the solution, the new hypothesis
statement has to be within the scope of the problem. Then, transforming the hypothesis
must consist of the following sequence:
Formalization of the top-down method 59
Table 3.5: Examples and counterexamples of gait and their projection into the model.
Feetmovement
MediumAdditionalfulcra
Minimumcontactpoints
EXAMPLES
Gait ofelderly people
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
Gait of anadult person
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
Gait of achild
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
Sport gaitAlternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
Walking overa treadmill
Alternatingstepping 2 feet
Horizontal surface No 1
Walking underpsychotropicinfluence
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
Upright translationof apes and bears
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
A bipedal robotwalking
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
Walking with anexoeskeleton
Alternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 1
COUNTEREXAMPLES
Translation ofother animals
Alternatingstepping 2or 4 feet
Horizontal or slightlyinclined surface
No 1-2
With more orless than twolegs
Alternatingstepping 1or >2 feet
Horizontal or slightlyinclined surface
No 0 - >2
SwimAlternatingstepping 2feet and 2 arms
Liquid surface No - Yes 1
Fly No movement Gas No 0
RunAlternatingstepping 2 feet
Horizontal or slightlyinclined surface
No 0
ClimbAlternatingstepping 2 feetand 2 arms
Vertical surface No - Yes 0
Walking withcrutches
Alternatingstepping of feetand crutches
Horizontal or slightlyinclined surface
Yes 2
Up or downramps or stairs
Alternatingstepping 2 feet
>horizontal and <vertical surface
No 1
Stand on amechanic stairs
No movement >horizontal and <vertical surface No 2
CrawlAlternatingstepping 2 feetand 2 arms
Horizontal or inclined surface No 3
Formalization of the top-down method 60
Table 3.6: Human gait variables
Shape variables Movement variables
Silhouette shapeSilhouette positionDisplacement plane
CentroidHeight and other body variables
Arm angles and lengthsLeg variablesFoot variables
Step and stride lengthStep and stride time
CadenceSingle support timeDouble support time
SpeedAcceleration
Foot maximum height
1. Correct and refine statements, objectives and requirements, contained in the initial
conjecture.
2. Transform the requirements of the solution that may be contained in the initial
conjecture into objectives of the scope of the problem. That is, transform re-
quirements into expressions built from an external point of view to the interface
between the invention and the rest of the universe. This transformation frees the
statement of the problem of potential restrictions about the solution that could
have been incorporated arbitrarily and even unintentionally. On the other hand,
as may occur that we decided that certain requirement of the solution has to be
an objective of the problem (e.g.: a circuit to be CMOS), it corresponds to retain
such constraint but as an objective of the problem. Thus, the new expression of
the hypothesis will have formal consistency to be a statement coming from an
external approach to the problem and, therefore, free from preconstraints which
would correspond to an internal point of view to the solution.
3. Proposing additional objectives. The objectives extracted from the hypothesis
can be extended by adding other ones obtained from the study of the state of
knowledge about the problem.
4. Arbitrary objectives. Additional objectives could be incorporated. They may arise
from intuition, by revelation, awareness, experience, and so on.
The formal specification eventually contain the compendium of the objectives extracted
from the hypothesis, the objectives extracted through the analysis of the state of knowl-
edge, and the objectives arising from inspiration.
3.4.1. Study case: Vegetable starter plate
Having established a model, the second step to obtain the formal specification is the
phase of architecture. It expresses the functionality that the solution must satisfy and
Formalization of the top-down method 61
is obtained answering the question what for. The answer provides the objectives of the
problem from an external point of view.
For example, the following hypothesis: create vegetable starter plate, a complete meal
to consume when weather is hot, that will be nutritive, healthy, tasty and aromatic.
Obviously, salad can be a model of vegetable starter plate.
The hypothesis can be rewritten in terms of the model as follows: vegetable starter
plate is a nutritive, healthy, tasty and aromatic salad. The architectural objectives that
vegetable starter plate have to satisfy are expressed in the hypothesis: nutritive, healthy,
tasty and aromatic.
Nevertheless, additional objectives can be considered, e.g., freshness, that can be ob-
tained from the state of the art, and low cost, which is based in a cultural conviction.
Table 3.7 shows the objectives that the vegetable starter plate must satisfy.
Finally, table 3.8 shows the objectives for the example of vegetable starter plate, as a
salad, that are contained in the hypothesis and two additional ones, obtained from the
state of the art and intuitively (inspiringly) incorporated, respectively.
Table 3.7: Objectives of the vegetable starter plate.
OBJECTIVES
1 Nutrition HIGH2 Flavor GOOD3 Health GOOD4 Odor PLEASANT5 Mixture MACRO6 Vegetables RAW PREVIOUSLY COOKED7 Dressing RAW OIL BASED8 Salt YES9 Water NO10 Others -11 Spicy seasoning YES12 Temperature COLD13 Harvest RECENT14 Price LOW
Note that the last two objectives are genuinely circumstantial factors: when to elaborate
the salad, and how much it must cost. Nevertheless, they have to be considered as
explicit essential factors because they belong to the hypothesis.
As another example, the following design factor is genuinely circumstantial: being able to
transport pieces of furniture in the trunk. Imposing that as objective of the hypothesis,
produces solutions whose main characteristic is that the assembly of the furniture could
Formalization of the top-down method 62
Table 3.8: Objectives that the vegetable starter plate must satisfy.
Objectives of vegetable starter plate
num Objectiveobj 0 SALADobj 1 Nutritiveobj 2 Tastyobj 3 Healthyobj 4 Aromaticobj 5 Freshnessobj 6 Low cost
be performed by the buyer at his home. That it is the essence of IKEA’s successful
business model.
The model of the problem along with the objectives (drawn from the hypothesis, taken
from state of the art and built by inspiration) that the problem must satisfy, answer,
respectively, the questions what is the problem and what for is a solution. This is a
characterization of the problem seen from the viewpoint of its universe and, therefore,
from an external focus to the problem.
The relevance of this aspect is that the external approach is objectivist. Therefore, it
is independent of potential constraints that could come from preconceptions about the
solution.
The definition of the problem is clearly formal, essentially based on the nature of the
problem and its utility, regardless of the nature and characteristics of the solution that
finally would be adopted.
Formal specification can be supported by a simple syntax structure consisting of a tran-
sitive predicate to assign the identity of the model and a formal predicate to assign
objectives as a conjunction.
< problem >< to be verb > < article >< model >
< preposition >< objective >
(< conjunction >< objective >) ∗ .
(3.30)
The expression of equation 3.30 is a logic formal expression and, perhaps, also mathe-
matical. Nonetheless, it is a formal expression constructively obtained. That guarantees
that every problem can be formally specified. Examples of formal specification are shown
in table 3.9.
Formalization of the top-down method 63
Table 3.9: Formal specification examples.
Formal specification of problems
Problem Formal specification
keyboardis an input device with keys representingcharacters and keys representing functions.
control accessis a system of protocols to entry and exitwith an internal and external door which openingis mutually exclusive.
tomatois a fleshy fruit with tegument, acid endotheliumand flat seeds.
gait
is the anthropomorphic upright self-displacement,in an alternating stepping of two feet, with noadditional fulcra, keeping at least a point of supportat every time, on a horizontal or slightly inclinedsurface [85].
vegetable starterplate
is a nutritive, healthy, tasty, aromatic, fresh saladhaving low cost.
To recap, the process of obtaining a formal expression of the problem starts from the
hypothesis and consists of the following steps:
1. Correct the mistakes, inaccuracies and inconsistencies potentially contained in the
hypothesis.
2. Redefine potential constraints about solution that could have been included in the
hypothesis as objectives.
3. Decide a reference model for the problem.
4. Include additional objectives from the state of the art and from inspiration.
5. Formally express the problem as the composition of a predicate of the model and
other about the objectives.
The formal specification of the problem provides declarative information about it, that
is, allows to know the problem.
To provide a solution, the information about the problem has to be constitutive and
operational. That is, the specification must allow us to know how are the constructive
modules, and to know how to do the construction of the solution.
Formalization of the top-down method 64
3.4.2. Study case: Gait analysis using computer vision
The proposed definition assumes that the whole body participates in gait, which is
reasonable even if there are some body parts that have a greater effect than others and,
therefore, they allow to characterize the gait easily, as is the case of feet.
This definition has declarative character but it does not have operational value. We
need a functional definition for our system. To do so we first establish the objectives our
system must satisfy, i.e., the utility of our system seen from an external point of view.
Objectives can be found answering the question what for is our gait analysis system
Gait-A (A(f(C(E(G))):
Definition 3.4. Gait-A is a vision based system for the early detection of frailty and
senility syndromes, analysing gait features of subjects using low cost, sustainable and
non intrusive devices.
Note that low-cost, which is clearly a contextual factor, has been included in the hypoth-
esis. The reason behind is that, in the last years, there has been a progressive reduction
of the budget the Spanish government provides to R&D purposes. This has forced the
researchers to adapt to such low budget and, thus, the trend for low-cost systems arises.
The characteristic of a system to be low-cost should clearly be a contextual factor of
the proposed solution, however, the low available budget has forced us to promote this
contextual factor to a role of essential factor. This will produce an early prune of the tree
of possible solutions which will still provide a solution but not necessarily the optimal
one.
From the previous hypothesis, we extract the following objectives of our system:
1. Detection of frailty syndrome
2. Vision based system
3. Gait analysis based system
4. Low computational cost
5. Reduced cost
6. Sustainability
7. Non intrusive
8. Early detection
Formalization of the top-down method 65
An issue that arises from the state of the art that could compromise the objective 1 is
that we do not know for certain if we could detect frailty or dementia syndromes through
computer vision analysis. Therefore we add the following objective:
9. Contingency in case of impossibility of detecting frailty or dementia.
Also, our experience with computer vision algorithms is that the computational cost is
high so this could compromise the objectives 3 and 4. We add the following objective:
10. Reduce computational cost of vision algorithms
We now have the objectives of our problem, i.e., the utility that provides to the external
world. We now define it internally, to do so we define the requirements that the system
needs in order to satisfy the objectives. The requirements are obtained by answering
the question how to satisfy the objectives:
a Extract features from RGB devices. This covers the objectives Vision based system
and Gait analysis based system.
b Gait classifier. This covers the objectives Detection of frailty syndrome as it detects
the presence of abnormalities in gait, and Gait analysis based system.
c Use low cost devices. This covers the objective Reduced cost.
d Use of rgb devices. The use of rgb devices covers the objectives of Vision based
system, Reduced cost, Sustainability and Non intrusive.
e Run gait test periodically. By running the gait test periodically we cover the
objective of Early detection and of course the objective of Gait analysis based
system.
f Code optimizations. Code optimizations are needed to cover the objectives Low
computational cost and Reduce computational cost of vision algorithms.
g Build data set. The vision data set covers the objective of Vision based system
and is also intended as a contingency in case we cannot detect frailty or dementia
because it will provide a data base of abnormal gait sequence to the scientific
community allowing others to continue our work.
The traceability matrix of the requirements is shown in table 3.10. Requirement a can
be joined with requirement b because they are interrelated (Corollary 3.5). Therefore,
Formalization of the top-down method 66
Table 3.10: Traceability matrix
Requirements
a b c d e f g
Objectives
1 X OK2 X X X OK3 X X X OK4 X OK5 X X OK6 X OK7 X OK8 X OK9 X OK10 X OK
requirement b now it is called gait classifier using vision features covering now objectives
1 and 3. In the next level, the requirement b can be extended by proceeding in the same
way: establish objectives and the requirements that cover them. Requirement c can be
removed because requirement d already covers the objective 5 (Corollary 3.6).
Considering the previous formulation of our problem in Eq. 3.26, requirement a imple-
ments the function f(C(E(G)). Requirement c corresponds to the camera variable C.
Finally, requirement b corresponds to the analysis A.
Corollary 3.5. If an objective is partially covered by more than one requirement, then
it is possible that those requirement can be fused into a single one.
Corollary 3.6. If an objective is fully covered by more than one requirement, then
a requirement that only cover an already fully covered requirement can be removed or
joined.
3.5. Gait classifier using vision features
This is the module that classifies gait between normal (no signs of frailty nor dementia)
and abnormal (presents signs of frailty or dementia), using vision devices. This is a
subproblem of the initial problem and, thus, we proceed in the same way. The objectives
of this module are the following:
1. Classify gait between normal and frailty
2. Use vision devices
Therefore the requirements are the following:
Formalization of the top-down method 67
a Extract vision features from gait. This requirement covers the Use vision devices
objective.
b Use classification techniques. This covers the objective of Classify gait between
normal and frailty.
Table 3.11: Gait classifier using vision features traceability matrix
Requirements
a b
Objectives1 X OK2 X OK
The traceability matrix of this module is shown in table 3.11. This is a straightforward
node since it is the result of the joint of requirements a and b from the parent node, it is
just the separation of the previously fused requirements. The decision of fusing them in
the first place was aimed at providing a more balanced tree, otherwise the father node
would have a lot of children.
3.5.1. Extract vision features
This is the module that deals with the extraction of gait features from vision devices
to obtain gait variables. Therefore, the objectives for this module are the variables
established in Table 3.6:
1. Human segmentation
2. Silhouette shape
3. Silhouette position
4. Displacement plane
5. Centroid
6. Height and other body variables
7. Arm angles and lengths
8. Leg variables
9. Foot variables
10. Step and stride length
Formalization of the top-down method 68
11. Step and stride time
12. Cadence
13. Single support time
14. Double support time
15. Speed
16. Acceleration
17. Foot maximum height
We added the Human segmentation objective obtained from the state of the art as it is
a required phase in order to extract the silhouette of the subject. Then the requirements
are the following:
a Background subtraction. This requirement provides a silhouette of the subject so
it covers Human segmentation, Silhouette Shape, Silhouette position, Displacement
plane, Centroid, Height and other body variables and Speed.
b Feet location and tracking. This requirement deals with locating both feet and
covers the objectives Foot variables, Step and stride length, Acceleration and Foot
maximum height.
c Heel strike and toe off detection. This deals with detecting the events of heel strike
and toe off that cover the objectives Step and stride length, Step and stride time,
Cadence, Single support time, Double support time and Speed.
d Skeleton extraction. This extract some skeleton joints and cover the objectives
Height and other body variables, Arm angles and lengths and Leg variables.
Table 3.12 shows the traceability matrix for this module.
3.5.2. Gait classification
This module deals with the classification of gait features into normal or abnormal gait.
In Table 3.6 we established different types of variables. There are two dimensional (or
three dimensional) variables, time series and shape variables. These are the types of
variables our classifier should deal with, therefore the objectives are the following:
1. Classify positional features
Formalization of the top-down method 69
Table 3.12: Extract vision features traceability matrix
Requirements
a b c d
Objectives
1 X OK2 X OK3 X OK4 X OK5 X OK6 X X OK7 X OK8 X OK9 X OK10 X X OK11 X OK12 X OK13 X OK14 X OK15 X X OK16 X OK17 X OK
2. Classify time series features
3. Classify shape features
4. Reduce dimensionality
From the state of the art, we incorporate the Reduce dimensionality objective to deal
with the large dimensionality of the shape variables. Then the requirements that cover
those objectives are the following:
a K-Nearest Neighbors. This provides cover of the objectives of classifying features:
Classify positional features (Euclidean distance can be used to compare positional
features so no additional requirements needed) and partially Classify time series
features (a different distance function is required to compare time series of arbitrary
length) and Classify shape features (compare images is a high dimensionality task
so a reduction is needed).
b Dynamic Time Warping. This allow to compare time series of arbitrary length so
it partially cover the objective Classify time series features.
c Principal Component Analysis. This is used to cover the objective Reduce dimen-
sionality for silhouette shape classification so it also partially cover the objective
Classify shape features.
Formalization of the top-down method 70
Table 3.13: Gait classification traceability matrix
Requirements
a b c
Objectives
1 X OK2 X X OK3 X X OK4 X OK
3.6. Structural specification of a solution
The formal specification of a problem, as mentioned above, provides a definition of itself
that is objective, correct, precise and accurate, and that definition is an interpretation
of the problem from an external point of view, i.e., from the universe containing the
problem. Therefore, formal specification is free of any assumption related to the nature
of any possible solution under the chosen model. Note, however, that the chosen model
produces a prune in the tree of possible solutions.
To obtain a solution, an interpretation of the problem from the internal point of view
is required because you need to understand the nature of the elements of the solution
and the relationships between them. These elements, and these relations constitute the
structure, the organization and the components of the solutions, and each approach
about the problem from an internal point of view can produce a different solution of the
problem.
In this section we indicate a method to obtain the structural specification and organiza-
tion of a solution (ω) within the formal framework of graph theory, as a pair consisting
of a tree (Υ) that defines a hierarchical module structure of the solution and a directed
graph (Θ) organizing relations between modules as shown in Eq. 3.31.
ω ≡ 〈Υ,Θ〉 (3.31)
The tree Υ = {t1, t2, . . . , tn} is composed of modules ti. The graph Θ = {t1, t2, . . . , tm}is composed of modules where m >= n and Υ ⊂ Θ, i.e., the graph Θ is composed of the
modules of the tree Υ plus some other modules that serve as synchronization between
the tree modules.
Formalization of the top-down method 71
Table 3.14: Requirements to cover the objectives of the vegetable starter plate problemas a salad. The objectives are defined in Table 3.8
Requirements to cover the objectives of the vegetable starter plate
n. Objectives
Requirements obj0 obj1 obj2 obj3 obj4 obj5 obj6
aBase of good qualityraw vegetables just
preparedX X X X X
b Light cheeses core X X Xc Olive oil dressing X X X Xd Add pasta Xe Use small plate X
3.6.1. Structural tree
The path starts from the hypothesis that may contain requirements (informal internal
view) to obtain a formal specification based on a model and objectives (formal external
view), and again a structural specification that consists of requirements (formal internal
vision), is a two-way process to obtain what we had before but as an objective, complete,
in an accurate and correct way, and within the formal framework of graph theory.
The objectives are the constraints imposed on the problem (external focus), the require-
ments are the features that a solution must comply (internal focus).
The starting point to obtain a structural specification is the formal specification and,
therefore, what we have to do is to propose a set of requirements for a solution, in such
a way that the requirements provide a cover of the objectives.
Any conjunction of the requirements of a solution can be considered as other requirement
that constitutes the constraint of a subproblem of the original problem, and a solution
of that subproblem is a module of the structure of a solution of the initial problem.
Therefore, a cover of the objectives consisting of requirements provides a two levels tree:
level 0 contains a root node representing a solution, and level 1 contains a node for each
requirement of the cover, each node representing a module of the solution.
The structural specification phase consists of answering the question how is the veg-
etable starter plate to obtain the requirements of a solution. All we must do is propose
requirements to obtain a cover of the objectives. Table 3.14 shows a set of requirements
proposed to solve the vegetable starter plate example as a salad.
Figure 3.7 shows a selection of salad modules that covers the objectives.
Formalization of the top-down method 72
cheesesspirali
vegetable
starter plate
vegetables dressing
Figure 3.7: Structural tree of vegetable starter plate.
Recursively, proceed for each leaf node as if the subproblem that represents were a
new problem, that is, considering that the requirements of that node represent the
hypothesis of the subproblem to be solved. The termination condition will consist of
obtaining trivial nodes or known ones. The leaves of the structural tree are basic modules
that solve atomic subproblems, namely technological components of the solution. They
provide the answer to the question with what is made the solution.
Figure 3.8 shows a complete structural tree for the example of the vegetable starter
plate, assuming that the raw materials which appear as leaf nodes, i.e., vegetables,
pasta, seasonings and cheeses are purchased on the market. The conceptual meaning
of this process is that the statement of the problem have been clarified and, therefore,
a solution is given. In the case of a restaurant that has its own crops, each node of
the vegetables would be, in turn, the root of a subtree representing the agricultural
production process of the corresponding vegetable.
cheesesspirali
vegetable
starter plate
burgos manchego
vegetables
arugula mushroomspinachlettuce
dressing
olive oil salt paprika
Figure 3.8: Structural tree of an organoleptic solution to the vegetable starter prob-lem.
Other design criteria focused more on the nutritional equilibrium than the organoleptic
aim could lead to the structural tree shown in Figure 3.9.
Moreover, other ingredients could lead to the structural tree shown in Figure 3.10.
Formalization of the top-down method 73
plants
vegetable
starter plate
spirali
salt animals
vegetables
arugula mushroomspinachlettuce
burgos manchegodressing
olive oil paprika
Figure 3.9: Structural tree of a dietetic solution to the vegetable starter plate.
plants
vegetable
starter plate
spirali
minerals animals
vegetables
tomato avocadoonionlettuce
manchego
cheeseyork hamdressing
olive oil garlic
salt platter
Figure 3.10: Structural tree of a slightly different vegetable starter plate.
3.6.2. Organizational graph
The organization of the structure of a solution is a directed graph expressing the re-
lationships between the modules of that solution. Nodes represent the modules of the
structure and each edge represents a relationship between connecting nodes. The graph
Formalization of the top-down method 74
with larger details of the organization is the one containing the leaf nodes of the struc-
tural tree.
The algorithm we propose to construct the organizational graph is the following:
Form a graph with the nodes of the first level of the structural tree.
Add nodes to represent relations among them.
Draw edges between the nodes of the graph to represent their relationships.
Replace each node in the partial graph for the corresponding nodes belonging to
the first level of the subtree of that node in the structural tree.
Proceed recursively.
Figure 3.11 shows the organization at first level of the structure of the vegetable starter
plate. Note that additional nodes are modal.
There is no relationship between nodes of ingredients which represents that the vegeta-
bles, cheeses, pasta and condiments are provided independently of one another, and that
it is in preparing the salad when the relationship between them arises, i.e., the sequence
of incorporation of the ingredients.
mount plate
form a bed
serve
vegetables pasta
cheeses
dressing
Figure 3.11: Global organizational graph.
Figure 3.12, shows the larger detailed organizational graph that has been obtained re-
cursively replacing each node coming from the structure by the nodes of level 1 in its
rooted subtree inside the structural tree.
Formalization of the top-down method 75
mount plate
form a bed
serve
brew
arugula
lettuce spirali
mushroomspinach
manchego
burgospaprika
olive oil
salt
Figure 3.12: Detailed organizational graph.
3.6.3. Structural specification of computer vision for gait analysis
The previously defined traceability matrices ensure a cover of the established objectives.
The requirements that perform such cover form a structural tree. This is the structural
specification. The root node is the name of our system, GAIT-A. From it, the require-
ments that cover the initial objectives of the hypothesis are the children of this root
node. Subsequent expansions of the children nodes following the requirements estab-
lished in the previous sections produce the structural tree of figure 3.13. The decision to
join some requirements in the root node provides an additional level to the tree. In case
we would not decided to join those requirements, a different tree could show up instead.
Note that we could have decided to include all the objectives from the children nodes
in the root node, obtaining a tree with a root and a single level. This, although a valid
solution, might not provide a good structured solution in most cases.
The structural specification provides a way of composing the solution from different
modules. This is precisely the structure that the next chapter follows to describe the
solution proposed for Gait-A.
Formalization of the top-down method 76
Gait-A
Gait classifier
using vision
features
Use of RGB
devices
Run gait test
periodically
Code
optimizationsBuild data set
Extract vision
features
Gait
classification
Background
subtraction
Heel strike
and toe off
Feet location
and tracking
Skeleton
extraction
K-Nearest
Neighbors
Dynamic
Time Warping
Principal
Component
Analysis
Figure 3.13: Structural tree of GAIT-A.
3.6.4. Organizational graph of computer vision for gait analysis
Following the algorithm established in section 3.6.2 we build the organizational graph
adding some modal nodes as shown in figure 3.14. This graph shows the relationships
Use of RGB
devices
Run gait test
periodically
Code
optimizations
Build data set
Background
subtraction
Heel strike
and toe off
Feet location
and tracking
Skeleton
extraction
K-Nearest
Neighbors
Dynamic
Time Warping
Principal
Component
Analysis
Record
Play
Get
Silhouette
Get FeetGet Time
series
Test
Multiresolution
Classify
Figure 3.14: Organizational graph of GAIT-A.
Formalization of the top-down method 77
between the different modules of the system. The execution begins with the Run gait test
periodically which starts the gait test. This module can use the input image sequence
directly from the Use of RGB devices module or from the stored in Build data set
module. The algorithm then goes to the Background subtraction module which uses
the optimization established in Code optimizations module. Then the gait silhouette
is obtained and passed to Feet location and tracking module and Principal Component
Analysis. The former obtains the feet position and passes it to Skeleton extraction
module to obtain the skeleton joints and to Heel strike and toe off module to obtains
the time series that Dynamic Time Warping and K-Nearest Neighbours use to classify
between normal and abnormal gait comparing with the stored cases in Build data set.
The latter reduces the dimensionality of the silhouette image and passes it to K-Nearest
Neighbours to perform the classification.
3.7. Design decisions about contextual factors
Circumstantial factors characterize the context where the realization of the solution takes
place. This context is the solution to the contextual subproblem, that is, the answers to
all the other to do questions different from the model and the essential factors: when,
who, how much, where, etc.
The conclusion of the analysis of the state of the art about the context of the imple-
mentation of a solution to a problem is that there are as many procedures as authors
or even as problems, versions, updates, etc. Each engineer uses its own criteria to make
contextual design decisions, and does it following a particular sequence, according to his
own preferences and even uses a different one for each problem to be solved [86]. This is
in line with the absence of adequate methodological foundation that Preston mentions
[80].
Such absence of systematization, empirically confirms that the existence of precedence
among circumstantial factors has not conceptual meaning. In practice, this obvious fact,
validates any sequence of making design decision about the context.
For example, the answers to the question how much, is usually among the first to be
addressed, causing significant impact on the quality of the solution. We can also find
solutions driven by the answers to the question who is the author but with category
of contextual factor. These solutions often have the stylistic imprint of the author.
Precedence of responses to the question who is the author that go beyond, puts these
answers in the category of explicit essential factors (what for) and can produce solutions
in which stylistic aspects are above other architectural factors such as usability or the
Formalization of the top-down method 78
robustness of the solution (inhospitable houses, impassable bridges, endless cathedral
constructions, smartphones that bend or even explode. . . ).
It is conceptually clear that the justification for action sequences to make design decisions
on the contextual subproblem, is not dependent of the problem. Therefore, the sequences
are arbitrarily proposed, using forms, templates, preferences, etc. In summary, the
contextual subproblem deals with external aspects of the problem.
The formal consequence is that there are no binary relations of order causally inspired
by the problem, in general. The methodological consequence is that the context of
solving a problem is an inherently parallel activity. Therefore, each solution to the
contextual subproblem will be obtained taking into account the corresponding particular
concurrence constraints.
We consider the context of problem solving as another problem that can be solved
recursively under the model-driven approach on a top-down way through the algebraic
formalization proposed in this work to solve the specific subproblem.
At most, we can propose the following model about the context of a problem:
Definition 3.7. The general context of solving a problem is a problem whose domain
is the set of design factors contained in the answers of circumstantial questions related
to the main problem, excluding the explicitly ones stated in its hypothesis, and whose
constraint will be whatever.
Keeping in mind that the solution of the contextual subproblem is the execution project,
we can indicate some potential models for the contextual subproblem for the following
examples:
Keyboard: the contextual subproblem is the assembly instructions and electronic
circuit.
Vegetable starter plate: the solution of its contextual subproblem will be the recipe
of its elaboration and the protocol to serve it in a restaurant.
Gait analysis using computer vision: the contextual subproblem is the memory for
the application of the MINECO 2013 call for projects (TIN2013-47152-C3-2-R).
3.8. Conclusions
Between this chapter and the previous, we have proposed a formal method for the
resolution of problems with causal coherence. Its theoretical framework is based on
Formalization of the top-down method 79
binary relation between the elements of the set of design decisions that has to be made
in order to resolve the problems. This causally coherent method match with the scientific
method of knowledge creation and the top-down technique enclosed in the model-driven
approach followed in engineering. This method admits an intuitive interpretation based
on answering the to do questions in an orderly fashion.
Whereas the scientific method and top-down technique follow a causally coherent se-
quence of design decisions, the bottom-up lacks that causally coherence. This explains
that the bottom-up approach can only guarantee the result if it is complemented with
prior knowledge of the structure of the solution, i.e., to resolve instances of problems
that belongs to a family of problems through solutions already known. On the contrary,
for the scientific method and top down technique, the universality of their validity is
proved, even to resolve problems whose solution is unknown.
Figure 3.16 compares our causally justified proposal with the empirical top-down and
bottom-up methods.
Figure 3.15 shows the steps of the causally coherent method of making design decisions.
Decomposition of the initial problem in a sequence of the three canonical subproblems
(model, specific and contextual) has been represented, as well as the milestones related
to specifications: functional, structural, and execution project.
Formalization of the top-down method 80
CONTEXTUAL
SUBPROBLEM
architectural objetives
WHAT FOR
model
WHAT
structural features
HOW
hypothesis
technological components WITH WHAT
modal context WHEN, ETC.
prototype
abstraction
concretization
organization
composition
planning
prototyping
CAUSALLY
FORMAL DESIGN
MODEL
SUBPROBLEM
SPECIFIC
SUBPROBLEM
INTERNAL
VIEW
predominantly
solution’s
domain
EXTERNAL
VIEW
predominantly
problem’s
domain
STRUCTURAL
SPECIFICATION
EXECUTION
PROJECT
FORMAL
SPECIFICATION
Figure 3.15: Causally coherent steps of our method for making design decisions.
Formalization of the top-down method 81
architectural
objetives
model
structural
features
hypothesis
technological
components
modal context
prototype
abstraction
concretization
organization
composition
planning
prototyping
TO
P-D
OW
N D
ES
IGN
architectural
objetives
model
structural
features
hypothesis
technological
components
modal context
prototype
abstraction
concretization
co
mp
ositio
n
prototyping
BO
TT
OM
-UP
DE
SIG
N
architectural objetives
WHAT FOR
model
WHAT
structural features
HOW
hypothesis
technological components WITH WHAT
modal context WHEN, ETC.
prototype
abstraction
concretization
organization
composition
planning
prototyping
CA
US
AL
LY
FO
RM
AL
DE
SIG
N
pla
nn
ing
Figure 3.16: Analogies and differences between our causally formal design proposaland the empirical top-down and bottom-up methods.
Chapter 4
Gait-A
In the previous chapters, we have provided a causally coherent method to resolve prob-
lems using the model-driven approach. That method was applied to the gait analysis
using computer vision problem (Gait-A) obtaining its structural tree and its organiza-
tional graph. This chapter develops every module that make up the previously obtained
structural tree, especially the modules that conform the computer vision gait analysis
and the classification of normal and abnormal gait. Finally, experiments were performed
to validate the proposed modules.
4.1. Low-cost RGB devices
The decision to use RGB cameras is also oriented to environmental sustainability because
this type of cameras relies only in visible light. They only need a source of visible light
that could be sun light or artificial light. Other types of camera like Kinect sensor, in
addition to a visible light source, emits infra-red light that floods the environment to
obtain depth measures. In addition, the simultaneous use of several cameras in the same
location is possible without affecting other devices.
Aimed at providing a low cost solution, we propose the use of a smartphone camera
which can be considered as free since every smartphone in the market contains at least
one camera and almost everyone has a smartphone. This means that there is no need
to acquire an additional camera.
On one hand, A mid-high end smartphone is between 200 and 1000 with computational
power capable of processing the images captured by its camera. The resolutions of any
mid-high end smartphone is at least Full-HD (1920x1080 pixels), that resolution is more
than enough for our purposes. Regarding storage capacity, smartphones has internal
83
Gait-A 84
Table 4.1: Comparison between smartphone camera and Kinect sensors.
Light source CostComputationcapabilities
Resolution Storage Connectivity
Smartphone Visible50e-1000e
Yes FullHD-4K>1GB andcloud storage
USB-WiFi-Bluetooth-3/4G
KinectVisible andinfra-red
50e No VGA No USB 2.0
Kinect 2Visible andinfra-red
150e No FullHD No USB 3.0
storage that allows to store recordings to be processed later. Smartphones also have
Internet connectivity through Wi-Fi which allows to upload the recordings to the cloud.
On the other hand, Kinect is around 60 - 150. The old Kinect has VGA resolution
(640x480) and the new one has Full-HD resolution (1920x1080). Both provide depth
information as well as a skeleton representation of the subject. Although the range to
provide a skeleton is limited. This however has to be processed in an external computer.
The computational power required for the old Kinect are not high but the requirements
for the new Kinect are. The storage of recordings has to be performed in this additional
computer.
In summary, a smartphone provides more versatility than Kinect sensor. The advantage
of Kinect sensor over a smartphone is the fact that Kinect provides an accurate skeleton
representation and depth information, however, it has a limited range of operation,
requires a high computational power computer to record and process sequences. The
smartphone camera uses only visible light whether the Kinect requires an structured
infra-red light source that can produce interferences with other infra-red light dependant
devices even other Kinects which will require a synchronization to be able to work
together. Table 4.1 shows a comparison between smartphone and Kinect cameras.
4.2. Data set
One of the decisions made during the specification phase was to build a data set of
gait that included normal and abnormal cases. The reason to do so was to provide the
scientific community with a data set so anyone could continue our work and replicate
it. This will also provide a contingency mechanism in case we will not be able to detect
frailty and dementia syndromes.
The data set is composed of samples of persons walking over an 8 m hall. We set a
camera perpendicular to gait direction at 4 m distance to record a side view of the
person (around four steps). We set another camera at the end of the walking path to
Gait-A 85
record a frontal view of the person. Each sample then has a folder side and a folder
front with the side and frontal view respectively. Figure 4.1 shows the set up of two
cameras previously described.
8 m
4 m
Figure 4.1: Recording set up.
Each sample is stored as an ordered set of images, one for each frame of the sequence
with the frame number as name. In order to be able to test the accuracy and correctness
of the proposed methods we manually marked the data set indicating for each sample
the frames in which a heel strike (HS) and toe off (TO) is produced.
Following this scheme, we recorded three different datasets. The first one was recorded
to obtain a side view of the subject. We recorded a total of 15 samples of normal gait
an another 15 samples of abnormal gait. This dataset was used to test the feature
extraction module for side view. The second dataset was recorded to obtain a frontal
view of the subject. We recorded 24 samples of normal gait and 20 samples of abnormal
gait. This dataset was used to test the feature extraction module for frontal view. The
third dataset was recorded to obtain samples of different pathologies with side view. We
recorded 3 samples of each of 5 gait types using 5 subjects. The recorded gait types were
normal, diplegic, hemiplegic, neuropathic and parkinsonian. The aim of this dataset was
to test the discriminatory power of a classifier for the 5 classes of gait recorded. The
abnormal gait in all three datasets was feigned by healthy subject.
Gait-A 86
4.3. Extraction of vision features - Side view
In this section, the different methods and techniques used to extract vision features
are explained. Figure 4.2 shows the flow diagram of the proposed system for side view
which is divided into 5 modules described in the following subsections. The first module,
Preprocessing, is a common module in most of the vision based systems. This module
deals with removing the background to obtain the subject’s silhouette and its bounding
box. The second module is the Feet Location. In this module, the position of heel
and toe of each foot is determined. The third module, Feature Extraction, performs
a gradient analysis of the heel and toe positions over time to detect the events of HS
and TO. The fourth module, Skeleton, deals with obtaining some skeleton joints in the
silhouette. The fifth module, Classification, is the one that compare the obtained gait
features with a database to determine whether the subject presents abnormal gait or
not.
Our proposed algorithms work with sequences of images captured with an RGB camera.
They can obtain HS and TO events that are used to segment the gait phases and obtain
spatio temporal parameters, and they can also work with abnormal gait patterns. The
algorithms can extract an approximation of the skeleton of the silhouette providing
legs and lean angle. These variables are then processed by the classifiers to perform a
classification between normal and abnormal gait.
In [87] a method to obtain HS as well as a method to obtain joints of lower limbs is
provided. This method works with normal gait patterns, however, as it relies on a spatial
separation between HS, it might not provide good enough results with some abnormal
gait patterns where there are small steps. The lower limb joints extraction requires a
post process to work, our method provides less accuracy but it can work with only a
single frame making it suitable for providing feedback to the subject.
4.3.1. Preprocessing
In the pre-processing module, the input RGB image is segmented for foreground silhou-
ette extraction through background subtraction techniques. At this stage, we derive the
foreground silhouette applying the Mixture of Gaussians [88] background subtraction
algorithm and then, we remove the noise generated with opening and closure morpho-
logical operators. These operators enlarge or reduce the silhouette, making it possible
for any small hole to be either filled or removed so that the actual bounding box of the
silhouette can be computed. This phase is the most computationally expensive since it
has to deal with the entire image and it is also critical because the rest of the algorithm
Gait-A 87
HSTOD
Background
Subtractor
Bounding
Box
Get External
Toe and
Heel
Get Internal
Toe
Gradient
Analysis
Toe O
Heel Strike
Feet Loca on Feature Extrac on
Preprocessing
Classi ca on
Compare
Gait
Skeleton (FASE)
Find Joints
Legs Angle
Gait
Variables
Normal Abnormal
Get Internal
Heel
Figure 4.2: Flow diagram of the proposed system for side view.
depends on the extraction of the silhouette. An optimization could be to limit the back-
ground subtraction to an area slightly bigger than the previously computed silhouette
and thus reducing the amount of pixels analysed. Another optimization is proposed
later in subsection 4.5.
Once the silhouette is extracted we proceed to obtain its bounding box by computing
the x, y positions using Equations 4.1, then those points are converted to a rectangle
(x, y, width, height) using Equation 4.2.
minx = arg minx,y
(∀x,y ∈ silhouette : x)
maxx = arg maxx,y
(∀x,y ∈ silhouette : x)
miny = arg minx,y
(∀x,y ∈ silhouette : y)
maxy = arg maxx,y
(∀x,y ∈ silhouette : y)
(4.1)
boundingBox = (minx ,miny ,maxx −minx ,maxy −miny) (4.2)
4.3.2. Feet location
This module deals with locating the position of heel and toe of each foot and com-
pose, along with the feature extraction module, the Heel Strike and Toe Off Detection
Gait-A 88
Figure 4.3: Out-put of the get ex-ternal toe and heel
function.
Figure 4.4: Out-put of the get inter-
nal toe function.
Figure 4.5: Out-put of the get inter-nal heel function.
algorithm (HSTOD).
The feet location process starts with retrieving the silhouette from the previous phase.
Segmentation is then applied to split the upper and lower part of the silhouette up to
71% (according to a human model [89]). The relevant study area (the bottom part) is
then brought to focus, just for the feet and legs to be clearly visualised. Next, the foot
length is extracted from the bounding box to correspond to its minimum width (Eq.
4.6), attained when the feet are parallel to each other. The next step consists in finding
out the point of the toe of the moving forward foot (MFF) and that of the heel of the
coming behind foot (CBF). The MFF toe point is derived as the silhouette maximum x
(Eq. 4.3) and the CBF heel point as the minimum x (Eq. 4.4), always assuming that
gait direction displacement is from left to right, as shown in Fig. 4.3.
arg maxx,y
(∀x,y ε silhouette : x) (4.3)
arg minx,y
(∀x,y ε silhouette : x) (4.4)
max bbox(i)→ grad(x)i < 0 ∧ grad(x)i−1 ≥ 0 (4.5)
min bbox(i)→ grad(x)i > 0 ∧ grad(x)i−1 ≤ 0 (4.6)
Next we identify the maximum CBF x to obtain its toe and minimum MFF x to obtain
its heel. Each foot silhouette is cut into two sections as seen in Figures 4.4 and 4.5,
using the foot length as bounding box. Figures 4.4 and 4.5 also show the internal toe
and the internal heel respectively in the boxes.
These two cuts conform each foot’s bounding box from which the lower right pixel for
toe, Figure 4.4, and the lower left pixel for heel, Figure 4.5, are identified.
Gait-A 89
4.3.3. Feature extraction
The module in subsection 4.3.2 delivers a vector of size n for each foot, being n the
number of sequence frames. The location (x, y) of both heel and toe are given for each
position.
By means of gradient analysis on x, HS and TO events can be detected in side view. HS
can be detected on MFF when its heel gradient decreases from positive to zero, Eq. 4.7.
TO, on the other hand, can be detected on CBF when its toe gradient increases from
zero to positive, Eq. 4.8.
heel strike(i)→ grad(x)i ≤ 0 ∧ grad(x)i−1 > 0 (4.7)
toe off(i)→ grad(x)i > 0 ∧ grad(x)i−1 = 0 (4.8)
This approach produces some false positives. The HS event will mostly be produced
before the bounding box reaches its local maximum (Eq. 4.5) and the TO event will
be completed after reaching its local maximum. Our first approach to deal with these
false positives consisted in setting to zero any value lower than a threshold and selecting
the closest hits to the local maximum bounding box width, and removing all the others.
Moreover, some other types of false positives emerge from the first and last frames of
the sequence when the silhouette has still not been completely captured by the camera.
In this case, we spatially limited the working area to keep the events generated within
that area. The filtering process of false positives through bounding box local maximum
offers acceptable accuracy as we found in [85]. However, it is still not optimal because HS
events tend to converge at the bounding box maximum while they are actually produced
earlier.
We solved the problem we faced in our previous approach by applying two filters. The
first filter (Eq. 4.9) effectively removes isolated values as can be observed comparing
figures 4.6 and 4.7. Removal of isolated values implies interpolating values between the
nearest non-zero values if two or fewer isolate consecutive values equal zero, and setting
to zero if two consecutive or fewer values are greater than zero.
Gait-A 90
-10
0
10
20
30
40
50
60
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96
No Filter
No Filter
Figure 4.6: MFF heel gradient without any filter.
0
10
20
30
40
50
60
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96
Remove Noise (Filter 1)
Remove Noise
Figure 4.7: MFF heel gradient ap-plying filter 1 removing isolated val-
ues.
0
10
20
30
40
50
60
1 6 11 16 21 26 31 36 41 46 51 56 61 66 71 76 81 86 91 96
Foot Gauss (Filter 2)
Foot Gauss
Figure 4.8: MFF gradient applyingfilter 2 with Gaussian filter.
v′i =
if vi = 0
if vi−1 6= 0 ∧ vi+1 6= 0→ (vi−1 + vi+2)/2
if vi−1 6= 0 ∧ vi+1 = 0 ∧ vi+2 6= 0→ (vi−1 + vi+2)/3
if vi−2 6= 0 ∧ vi−1 = 0 ∧ vi+1 6= 0→ (vi−2 + vi+1)× 2/3
if vi 6= 0
if vi−1 = 0 ∧ vi+1 = 0→ 0
if vi−1 = 0 ∧ vi+1 6= 0 ∧ vi+2 = 0→ 0
if vi−2 = 0 ∧ vi−1 6= 0 ∧ vi+1 = 0→ 0
(4.9)
The second filter (shown in figure 4.8), in turn, undertakes the problem of improving re-
sults in situations in which incomplete silhouettes or occluded areas produce imprecision
in determining HS or TO events. This filter proves to be a valid tool for abnormal gait
detection when the events are not clearly defined. Instead of using only heel position to
obtain heel strike, we use the mean point of MFF heel and toe and analyse the gradient
at that point to obtain an HS event, and we repeat the process for the CBF heel and toe
to obtain a TO event. Adding a Gaussian filter to the gradient analysis and removing
isolated values (Eq. 4.9) complete the process. This filter is intended for improving
detection in abnormal gait where HS and TO are not very well defined.
Once HS and TO events are identified, derivation of step and stride length and time as
well as single and double support time variables can be readily determined.
Gait-A 91
0%
10%20%30%
40%
50%
60%70%
80%
90%100%
Filter 1
Undetected
Wrong
Correct
Figure 4.9: Graphic showing the amount of correct, wrong and undetected cases forHS and TO in side view using filter 1.
Completed the variable identification stage, experimentation was called forth: First, we
recorded the gait sequences as described in section 4.2; then, we manually processed each
sequence to delimit HS and TO events. This manual processing provided the ground
truth needed for testing the accuracy of the proposed algorithms. We allowed a margin
error of ±1 frame (because it is the minimum measure of time) and an error of ±1 frame
in the algorithm output, making a global error margin of ±2 frames. Next, we analysed
the frame difference between the ground truth and that of the proposed algorithm. If any
difference we obtained was less than or equal to the global margin error, we considered
the result acceptable. Finally, we computed the root mean square of the differences by
Eq. 4.10
RMSE =
√√√√ 1
n
n∑i=0
(mi − ai)2 (4.10)
Where n is the number of HS and TO events, mi is the frame of the event i in the
manual marking and ai is the frame of the event i in the algorithm output.
We tested not only our own dataset but also our HSTOD algorithm with the CASIA
dataset A [90] (only 90o camera samples). In Table 4.2 we illustrate the results from
applying the HSTOD algorithm and the two filtering methods described in section 4.3.3.
As observed, HS events are more accurately delimited than TO events. In addition, com-
paring the two filters, the second filter (Figure 4.10) offers better precision in detecting
normal and abnormal gait even though it yields a larger number of undetected cases in
normal gait than the first filter (Figure 4.9).
Gait-A 92
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Filter 2
Undetected
Wrong
Correct
Figure 4.10: Graphic showing the amount of correct, wrong and undetected cases forHS and TO in side view using filter 2.
4.3.4. Skeleton extraction algorithm
This work focuses on gait characterisation of the lower part of the body, below the hip,
but given that we pursue to perform a full body characterisation in a future research
work, we find it imperative to formulate a skeleton extraction algorithm.
We have found that the process of extracting the skeleton of a person exclusively out of
RGB data reaches a high degree of complexity. Besides, dealing with normal and abnor-
mal gait simultaneously adds another level of complexity to the task to be compounded
by the use of low-cost devices. Considering the nature of the difficulties, we have opted
for an approximate method that ensures sufficient accuracy without sacrificing execution
speed in low-performance computational devices.
Our approach resembles that of Leu et al. [59], but instead of projecting horizontal and
vertical silhouette pixels to get the neck joint and then apply an anatomical model to get
the hips, knees and ankles, we bring our Fast Approximate Skeleton Extraction (FASE)
algorithm into play. FASE allows us to divide the silhouette into fractions of its height
in consonance with a human model [89] to get the head, torso, thighs and feet. Figures
4.11 and 4.12 show the four fractions we obtained.
Once the parts are segmented, we divide the head and torso crosswise into halves and
compute the centre of gravity (COG) of each half as illustrated in figure 4.11. Consid-
ering the origin of coordinates to be the top left corner, where heady is the y position
of the head segment; headheight its height; upperHeadCOGx and lowerHeadCOGx are
the x position of the COG of the upper and lower half of the head segment respectively.
The same applies to the torso segment. What we do, in summary, is compute the COG
Gait-A 93
Table 4.2: Results of the HSTOD algorithm showing the amount of correct detections(less than 2 frames of difference between algorithm and manual marking), undetectedcases, wrong detection (more than 2 frames of difference) and the root mean squareerror of both correct and wrong cases. F1: first filter in which isolated values are
removed, F2: second filter in which the foot is used instead of only heel or toe.
Filter Correct Undetected Wrong RMSE
DAI Dataset Normal GaitHeel Strike
F1 83 (91.2%) 0 (0%) 8 (8.8%) 1.5 frames (50 ms)F2 85 (93.4%) 1 (1%) 5 (5.5%) 1.2 frames (40 ms)
Toe Off
F1 66 (75%) 1 (1.1%) 21 (23.9%) 2.18 frames (73 ms)F2 83 (94.3%) 2 (2.3%) 3 (3.4%) 1.24 frames (41 ms)
DAI Dataset Abnormal GaitHeel Strike
F1 111 (78.2%) 18 (12.7%) 13 (9.2%) 1.85 frames (62 ms)F2 131 (92.3%) 5 (3.5%) 6 (4.2%) 1.48 frames (49 ms)
Toe Off
F1 94 (69.6%) 8 (5.9%) 33 (24.4%) 2.87 frames (96 ms)F2 109 (80.7%) 5 (3.7%) 21 (15.6%) 2.09 frames (70 ms)
CASIA DatasetHeel Strike
F1 316 (95.8%) 8 (2.4%) 6 (1.8%) 0.98 frames (33 ms)F2 319 (96.4%) 9 (2.7%) 2 (0.9%) 0.89 frames (30 ms)
Toe Off
F1 315 (96.3%) 5 (1.5%) 7 (2.1%) 0.95 frames (32 ms)F2 305 (93.3%) 8 (2.5%) 14 (4.3%) 1.62 frames (54 ms)
TotalHeel Strike
F1 510 (90.6%) 26 (4.6%) 27 (4.8%) 1.32 frames (44 ms)F2 534 (94.9%) 15 (2.7%) 14 (2.5%) 1.12 frames (37 ms)
Toe Off
F1 475 (86.4%) 14 (2.6%) 61 (11.1%) 1.81 frames (60 ms)F2 497 (90.4%) 15 (2.7%) 38 (6.9%) 1.7 frames (57 ms)
Figure 4.11: Headand torso COGs ob-
tained.
Figure 4.12: Thigand foot COGs ob-tained and foot loca-
tion output.
Figure 4.13: FastApproximate Skele-ton Extraction out-
put.
Gait-A 94
of the upper and lower halves and then move them to top and bottom respectively to
obtain the following points:
upperHead = (upperHeadCOGx, heady)
lowerHead = (lowerHeadCOGx, heady + headheight)
upperTorso = (upperTorsoCOGx, torsoy)
lowerTorso = (lowerTorsoCOGx, torsoy + torsoheight)
We obtain the thigh points in a slightly different manner. We cut the shape horizontally
and find out the COGs as previously explained. Then, we cut the shape again, but
this time vertically to obtain two more COGs. In this way, we obtain four COGs as
shown in Figure 4.12. Let’s name the upper part of the thigh, upperThigh; the moving
forward knee, MFK ; the coming behind knee, CBK for our formulation purposes. Then
we obtain the following points:
upperThigh = (upperThighCOGx, thighy)
MFK = (rightThighCOGx, thighy + thighheight)
CBK = (leftThighCOGx, thighy + thighheight)
Then foot is obtained using the previously calculated heel and toe from the feet location
algorithm in section 4.3.2:
MFF = (MFFHeel +MFFToe)/2
CBF = (CBFHeel + CBFToe)/2
Once all the points are obtained, we still need to connect all the segments. We do so by
averaging the connections:
neck = (lowerHead+ upperTorso)/2
hip = (lowerTorso+ upperThigh)/2
In the final stage in side view, we accommodate the knee points to predict their position.
When walking, as the legs move forward, the knees move up and down to some degree.
Gait-A 95
100
200
300
400
500
600
700
800
900
0 500 1000 1500 2000
Y
X
Manual Head
FASE Head
Manual Neck
FASE Neck
Manual Hip
FASE Hip
Manual Forward Knee
FASE Forward Knee
Manual Forward Foot
FASE Forward Foot
Figure 4.14: Graphic comparison between 2D manual marking and FASE output.
To simulate knee movement, we start by tracing three circles; two along the length of
both feet and thighs, and the other around the hip. We make the hip circle radii equal to
thigh length, and the feet circle radii equal to the length of the tibia. We then intersect
the hip circle with each foot circle and save that intersection furthest to the right as the
new knee point for each leg. Figure 4.13 shows the final result.
As previously mentioned, this process constitutes an approximate skeleton, but it is able
to yield leg and torso angles with sufficient accuracy. In addition, it is fast, and just
one single frame affords valuable information. However, it falls short of detecting the
location of the occluded leg, let alone the arms. It can neither distinguish left from right
leg, but swapping the legs when the silhouette bounding box width is minimum achieves
the desired result. Although the method successfully applies to almost every type of
gait, we have already discovered that with dragging legs, it simply does not produce
satisfactory results in leg identification.
We evaluated the accuracy of the FASE algorithm by comparing the joints obtained
through it and a manual marking of the positions of head, neck, hip, knees and feet.
The Euclidean distance between manual marking and FASE output is analysed to com-
pute the average distance and root mean square, and then normalise the distance as
a percentage of the subject’s height. As Table 4.3 shows, the greatest accuracy cor-
responds to head, neck and hip scores while the poorest hits to the knees and feet,
precisely because of the leg occlusion problem. Figure 4.14 shows the accuracy of the
FASE output in one sample.
For the purpose of testing the relevance of such results, we have also tested the leg angles
(the angles formed between each foot and the hip). The results show a RMSE of 6.65o for
normal gait and 3.98o for abnormal gait cases. Figures 4.15 and 4.16 show the respective
comparison between manual marking and normal and abnormal gait samples. It is worth
mentioning at this stage that a lower RMSE in abnormal gait means that in those cases,
Gait-A 96
0
5
10
15
20
25
30
35
40
45
50
1 7
13
19
25
31
37
43
49
55
61
67
73
79
85
91
97
103
Angle
Frame
Manual
FASE
Figure 4.15: Normal gait leg anglecomparison.
0
5
10
15
20
25
30
35
40
1 9
17
25
33
41
49
57
65
73
81
89
97
105
113
121
129
Angle
Frame
Manual
FASE
Figure 4.16: Abnormal gait leg an-gle comparison.
the leg angles are smaller, which consistently correspond to smaller differences between
manual marking and FASE output.
Table 4.3: Comparison between FASE and manual marking as percentage of subject’sheight.
Joint % AVG % MAX % MIN % RMSE
Normal Gait
Head 1.75 5.42 0.15 2.05Neck 2.66 8.64 0.45 3.11Hip 2.49 7.77 0.38 2.90
MFK 3.99 20.91 0.25 4.98MFF 3.19 10.56 0.38 3.83CBK 3.50 9.78 0.34 4.02CBF 3.97 9.74 0.86 4.48
Abnormal Gait
Head 3.69 16.08 0.25 5.08Neck 3.79 13.86 0.36 4.89Hip 3.09 8.42 0.71 3.53
MFK 3.14 7.85 0.10 3.59MFF 2.31 5.97 0.23 2.61CBK 2.25 10.22 0.21 2.90CBF 1.91 6.26 0.14 2.35
4.4. Extraction of vision features - Frontal view
In the same way section 4.3 presented the methods and techniques for side view, this
section presents those of the frontal view. Figure 4.17 shows the flow diagram of the
frontal view system which is divided into 5 modules described in the following subsec-
tions. The first module, Preprocessing, is exactly the same as the side view, it performs
a background subtraction and obtains the bounding box of the silhouette. The sec-
ond module, Feet Location, obtains the toes positions and the difference between the
toes of each foot. The third module, Feature Extraction, obtains the zero crosses of
Gait-A 97
the toes difference time series and the maxima and minima to obtain HS and TO. The
fourth module, Skeleton, deals with obtaining some skeleton joints in the silhouette. The
fifth module, Classification, is the one that compare the obtained gait features with a
database to determine whether the subject presents abnormal gait or not.
HSTOD
Background
Subtrac on
Bounding
Box
Get ToesToe Y
Di erence
Toe O
Heel Strike
Feet Loca on Feature Extrac on
Preprocessing
Classi ca on
Compare
Gait
Skeleton (FASE)
Find Joints
Legs Angle
Gait
Variables
Normal Abnormal
Get Max
and Min
Figure 4.17: Flow diagram of the proposed system for frontal view.
4.4.1. Feet location
This module deals with locating the position of toes of each foot and compose, along
with the feature extraction module, the Heel Strike and Toe Off Detection algorithm
(HSTOD). Most of the vision based gait analysis proposals use sagittal view because it
provides more information to work with. However, there are some benefits of a frontal
gait analysis.
According to Whittle [54], there are more gait abnormalities that can be observed from
a sagittal view than from a frontal view. However, we decided to also include frontal
view for the following reasons:
There are some abnormalities that can only be observed from a frontal point of
view. Whittle [54] mention that circumduction, hip hiking, abnormal foot contact
and rotation among others are better observed from a frontal view.
Large amount of physical space required to record sagittal gait sequences, while
only a small hall or corridor is required for frontal gait sequences. Some work-
arounds are to use a treadmill although it could alter the gait patterns specially
Gait-A 98
with frailty people. Another workaround is to use a motorised camera that fol-
lows the subject although it is expensive and could complicate the background
subtraction as the background is moving as well.
Sagittal images show a clear view of the movement of the feet and enough information to
locate heel and toe of each foot so our previous method works with sufficient accuracy.
However, in frontal view it is not easy to determine where the heel and toe are located
in each foot. Therefore a different approach is required in this case.
In frontal view both toes are always visible but heels are occluded constantly so heels
cannot be properly located. Therefore we rely on toe information only.
To obtain toes we proceed by dividing the silhouette in four parts according to the human
model established in [89]. We focus only in the feet segment. Then we split vertically
that segment in half. We obtain the left and right foot toe by locating the pixel with
minimum y component in the left and right half respectively (Eq. 4.11) (Figure 4.18).
arg minx,y
(∀x,y ∈ silhouette : y) (4.11)
Figure 4.18: Silhouette obtained after background subtraction and toes obtained bymarking the minimum Y of each halves of the silhouette.
4.4.2. Feature extraction
The module in subsection 4.4.1 delivers a vector of size n for each foot, being n the
number of sequence frames. The location (x, y) of the toes of each foot are given for
each position.
Gait-A 99
Using frontal view we propose to use the difference of the y component of the toes
obtaining a curve in which zero crosses indicate the feet adjacent gait phase. HS and
TO of each foot are located between each zero cross as shown in Figure 4.19. We can
estimate HS and TO by assuming HS is produced before TO and HS is produced in the
first half of each region and TO in the second half. Therefore we can estimate HS and
TO following equations 4.12 and 4.13 respectively, where zci is the frame in which a
zero cross point occurs and zci−1 is the frame of the previous zero cross point.
HS = zci−1 +(zci − zci−1)× 3
4(4.12)
TO = zci −zci − zci−1
4(4.13)
-80
-60
-40
-20
0
20
40
60
80
Di Y
Di Y
Figure 4.19: Graph showing the difference of component Y of each foot. A red verticalline marks the instant at which a HS is produced and a green vertical line marks TO.
We performed some experiments using our own dataset of frontal gait. To record the
dataset we place a camera in front of a corridor of 8 m and ask the person to walk
towards it. There are a total of 24 samples of normal gait and 20 samples of abnormal.
The same process of manual marking done in side view was performed. Table 4.4 and
figure 4.20 show the amount of correct detections (less than 2 frames of difference between
algorithm and manual marking), undetected cases, wrong detection (more than 2 frames
of difference) and the root mean square error of both correct and wrong cases. The RMSE
of both HS and TO in normal gait are a bit bigger than the error margin of 2 frames,
but they are much bigger in the case of abnormal gait. Therefore, results are acceptable
for normal gait but not so for abnormal. A different approach that deals with abnormal
gait is required.
To solve the problem of the previous approach we proceed by applying several Gaussian
filters to remove noise (Figure 4.22 shows the curve of Figure 4.21 after applying several
Gaussian filters), then, instead of relying in zero crosses, we obtain the local maxima
and minima. These maxima and minima are located more or less at the center of each
Gait-A 100
Table 4.4: Results of the HSTOD algorithm for the first approach.
Correct Undetected Wrong RMSE
DAI Dataset Normal GaitHeel Strike
94 (83.2%) 16 (12.4%) 19 (16.8%) 2.02 frames (67 ms)Toe Off
75 (66.4%) 16 (12.4%) 38 (33.6%) 2.73 frames (90 ms)
DAI Dataset Abnormal GaitHeel Strike
96 (64.9%) 49 (24.9%) 52 (35.1%) 3.38 frames (111 ms)Toe Off
62 (41.9%) 48 (24.5%) 86 (58.1%) 4.1 frames (135 ms)
TotalHeel Strike
185 (72.5%) 65 (20.3%) 70 (27.5%) 2.88 frames (95 ms)Toe Off
132 (51.8%) 64 (20.1%) 123 (48.2%) 3.60 frames (119 ms)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Frontal 1
Undetected
Wrong
Correct
Figure 4.20: Graphic showing the amount of correct, wrong and undetected cases forHS and TO in frontal dataset using zero crosses approach.
pair of zero crosses. Although the curve does not cross zero in some cases, e.g., when
a foot is always behind the other or is dragged due to some injure or pain, this is not
necessary to produce a maximum or minimum.
HS are located before a maximum or minimum and TO after. We know that both events
are located in that region. Empirically adjusting them we obtain that the HS is located
at 1/4 of the distance between one maximum (or minimum) and the previous (Eq. 4.14)
and TO is located at 1/8 of the distance between one maximum (or minimum) and the
next one (Eq. 4.15).
Gait-A 101
-20
-10
0
10
20
30
40
50
60
70
80
Di Y
Di Y
Figure 4.21: Difference of compo-nent Y of each foot with abnormal
foot dragging.
-10
0
10
20
30
40
50
60
70
Di Y Gauss
Di Y Gauss
Figure 4.22: Difference of com-ponent Y of each foot after several
Gaussian filters.
Being M an ordered set of maxima and minima in ascending chronological order:
M = {m1,m2,m3 . . .mn}
HS of mi is obtained as:
HSi = mi −mi −mi−1
4(4.14)
and TO is obtained as:
TOi = mi +mi+1 −mi
8(4.15)
The same experiment was performed for this approach. Results are shown in table 4.5
and graphically in Figure 4.23. Results show a vast improvement reducing the number
of undetected cases.
Table 4.5: Results of the HSTOD algorithm for frontal view.
Correct Undetected Wrong RMSE
DAI Dataset Normal GaitHeel Strike
115 (89.1%) 0 (0%) 14 (10.9%) 1.89 frames (62 ms)Toe Off
115 (89.1%) 0 (0%) 14 (10.9%) 1.65 frames (54 ms)
DAI Dataset Abnormal GaitHeel Strike
102 (68.9%) 6 (3.9%) 46 (31.1%) 2.68 frames (88 ms)Toe Off
91 (61.5%) 6 (3.9%) 57 (38.5%) 3.03 frames (100 ms)
TotalHeel Strike
217 (78.3%) 6 (2.1%) 60 (21.7%) 2.34 frames (77 ms)Toe Off
206 (74.4%) 6 (2.1%) 71 (25.6%) 2.48 frames (82 ms)
As shown in there, the RMSE of both HS and TO in normal gait are smaller than
the error margin of 2 frames, but they are slightly bigger in the case of abnormal gait.
Gait-A 102
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Frontal 2
Undetected
Wrong
Correct
Figure 4.23: Graphic showing the amount of correct, wrong and undetected cases forHS and TO in frontal dataset using local maxima and minima approach.
Therefore, results are acceptable although abnormal gait could be improved. Most of
the error is produced in the first steps when the silhouette is smaller (the subject is the
farthest from the camera).
The results obtained with our sagittal view approach are similar for normal gait. We
obtained there 1.2 frames for HS and 1.24 for TO, those were slightly better than those
obtained with frontal approach (1.89 - 1.65) but very similar. However in the case of
abnormal gait we obtained 1.48 frames for HS and 2.09 for TO in sagittal view which
where way better than the obtained with the frontal view approach (2.68 - 3.03).
Although a different dataset and recording environment was used in Xu et al. [61] we
can do a comparison between our approach and theirs as they use frontal view as well.
We can only compare our results for normal gait as their approach uses only that. For
HS they obtained a standard deviation of 1.45 frames (1.3 for right foot and 1.6 for
left foot) and our result was 1.89 frames, a fairly similar result. However for TO they
obtained a standard deviation of 3.55 frames (3.4 for right foot and 3.7 for left foot)
whether we obtained 1.65 frames, a better result. Xu et al. stated in their paper that a
greater error in TO was due to the TO occurring at a far distance to the camera where
the depth error was bigger, in our case we do not have that problem. As we stated
before, Xu et al. used a different dataset and a different recording environment so the
conclusion of the comparison is that both approaches show similar results although our
approach uses only RGB image.
Gait-A 103
4.4.3. Skeleton extraction algorithm
The FASE modules is the same as in the side view, however, in frontal view there is no
need to distinguish each leg because they are never occluded. The adjustment of the
knees using circle is not needed here. Finally, the foot joint used corresponds to the toe
position obtained in 4.4.1.
4.5. Code optimizations
Computer vision demands a high computational power that makes it difficult to work
in low cost devices under time constraints, however, it is convenient to advance in this
matter so every geriatrics could make use of these kind of solutions. We propose a
real-time approach for the background subtraction phase. With real time capabilities,
not only the method could be used to analyse gait, but also to provide feedback to the
subject which is precisely the evolution of our project.
Our proposed method can work at real time only for specific computational power, i.e.,
if the device that perform the computation is not sufficiently faster then the system
cannot work under a specific time constraint. This approach is sufficient when it is
executed in specific hardware and do not require scalability. However it cannot be
executed in lower computational performance hardware due to its inability to reach
the deadline in time. A proper real time application should adapt to the available
computational power, providing a suboptimal response if the deadline is reached before
finishing the processing. A way to obtain a suboptimal response in our case is based on
the multiresolution pyramid.
The pyramid approach has been widely used in computer vision and computer graphics.
The simplest pyramid approach consists in iteratively computing copies of lower resolu-
tion of the original image. An example of this is the mipmap used in computer graphics
for texture filtering. A multiresolution texture is computed at loading time, then the
appropriate resolution is selected depending on the distance at which the texture has to
be rendered [91].
According to Lindeberg [92, 93], Pyramids, wavelets and multi-grid methods were the
precursors of the scale space theory developed in image processing, computer vision
and signal analysis. Space scale theory deals with multi-scale representation of images
(or signals). This states that when building a multi-scale representation image, each
subsequent downsampling should constitute a simplification of previous level and should
not add artefacts due to the smoothing method used. Therefore, the requirement for a
Gait-A 104
kernel is linearity and spatial shift invariance. A kernel that fulfil these premises is the
Gaussian kernel and its derivatives.
There are two main approaches for pyramid representation: Gaussian Pyramid and
Laplacian Pyramid. In the former, each level of the pyramid is first processed with a
gaussian smoothing and then downsampled to 1/2 size, the process is repeated for the
number of levels required. In the latter, in each level, a difference of the blurred image
of previous level is stored, except the smallest level in which the image is only blurred.
The objective is to have different scaled convolution versions of the same image in an
efficient way. The orientation invariant approach is called Steerable Pyramid [94]
There are many applications of these approaches. Some of these applications are image
compression, detail manipulation, computer graphics... [95, 96]
A method for hallucinating high resolution faces from low resolution ones using Steerable
Pyramid is proposed in [97]. Strengert et al. proposed the construction of pyramid based
filters using the built in bilineal filtering capabilities of Graphics Processing Unit (GPU)
[98]. An edge aware image processing method using Laplacian Pyramid is proposed by
Paris et al. [99]. Yadav et al. [100] present a method based on Gaussian pyramid to
classify microscopic images of hardwood species.
4.5.1. Iterative Multiresolution Processing in Real Time
In most vision based applications, background subtraction is the initial step followed
by noise reduction techniques like morphological operations. Those techniques often
reduce the resolution of the silhouette. This resolution downgrade is accepted because
the benefits of a de-noised silhouette are greater than those of a high resolution one.
Most of the applications does not even require high resolution silhouettes. Assuming
this, the first step in order to reduce the computation time of a background subtraction
algorithm would be reducing the input image resolution. However, even if the silhouette
resolution is not important, a higher resolution could be beneficial in some cases.
In our approach the silhouette extraction phase using Mixture of Gaussians [88] is the
most computationally expensive task. With Full HD resolution, 90% of the time is
consumed there. This task takes around 90 ms to complete using an Intel Core i7
2630QM CPU at 2.00GHz and 8GB RAM DDR3.
For our approach to work under time constraints, we need to redesign this critical phase
so it can provide a fast suboptimal response that can be iteratively improved depending
on the available time.
Gait-A 105
What we propose is an Iterative Multiresolution Processing in Real Time (IMPReTi)
approach that provides increasingly higher resolution silhouettes as long as there is time
remaining.
4.5.1.1. IMPReTi first approach
The first step is to create a multiresolution image from the original image, to do so
we proceed creating a mipmap of the input image. The mipmap is organised following
the order shown in Figure 4.24 using the Algorithm 1. This organisation ensures that
the first obtained silhouette will be the lower resolution one providing also some steps
of some of the next resolutions. However, this also increases the memory and time
needed by 50% (considering a linear algorithm) to complete the maximum resolution.
The usable space required is only 1/3 bigger than the original image since the sum
1/4 + 1/16 + 1/64 + · · · + 1/22n converges to 1/3. The mipmap approach [91] achieve
this 1/3 increase in space by splitting the RGB channels of the image as shown in figure
4.25. This however will require a modification in the background subtraction algorithm
to work with this specific layout.
Algorithm 1 MipMap organization
lastRect← Rect(0, 0, src.width, src.height)lastImage← srcdst(lastRect)← srcfor i < numMipMaps do
Resize(lastImage, 0.5)if imod2 = 0 then
lastRect.x← lastRect.x+ lastRect.width0if i = numMipMaps -1 then . The smallest resolution must be put on top so
it is processed firstlastRect.y ← 0
elselastRect.y ← lastRect.y + lastRect.height ∗ 0.25
end ifelse
lastRect.x← lastRect.x+ lastRect.width ∗ 0.25lastRect.y ← 0
end ifdst(lastRect)← lastImage
end for
Algorithm 2 shows the procedure to work in real time using the mipmap image.
This approach could provide good results with linear or less complexity algorithms. How-
ever, as the size of the image is increased a 50% (being a 19% empty) it will significantly
increase the processing time required to even compute the lower resolution.
Gait-A 106
Figure 4.24: Mipmaps generated. Overall image size increased by 1/2.
Figure 4.25: RGB optimized mipmap. Overall image size increased by 1/3.
Algorithm 2 Real-time iterative background subtraction first approach
for i < m doProcessLine(i)if timeexceeded then
breakend if
end forResult← GetMaxResolutionCompleted(i) . Depending on the number of linesprocessed obtain the maximum resolution completed
4.5.1.2. IMPReTi second approach
Another solution is to compute separately each resolution, starting with the lowest
and iteratively increasing it as long as there is time remaining. We proceed creating
different resolution images like in the previous method, however we do not combine
them. Then the real time algorithm will be as shown in Algorithm 3. The advantage
of this approach is that it only increases the total time and space in a 1/3 rather than
a 1/2 like the previous approach (assuming the initialization time of the background
subtraction algorithm is minimum and linear complexity) achieving the same spatial
increase as the RGB mipmap. As downside, in this approach, each resolution is processed
independently so a previously computed resolution do not provide anything to next
resolution computation.
In order to mitigate this issue, we propose the use of lower resolution computed to
remove noise. To do so we average all the computed resolutions and keep as silhouette
Gait-A 107
only the pixels with value greater than a threshold. This threshold can be empirically
adjusted to reduce more or less noise.
Algorithm 3 Real-time iterative background subtraction second approach
time← 0for i < numResolutions do
lastT ime← CalcT ime(Process(resolution[i]))time← time+ lastT imeif time+ PredictNextResolutionT ime(lastT ime) > frameTime then
breakend if
end for
Being independent each resolution, multiprocessing techniques could be added to process
each in parallel, in which case the algorithm will be as shown in Algorithm 4.
Algorithm 4 Real-time iterative background subtraction second approach in parallel
for i < numResolutions doStartThread(Process(resolution[i]))
end forWait(frameT ime)CancelUnfinishedThreads() . This function should keep in mind that at least oneresolution has to be completed.
Another alternative would be to use the times from the previous frames computed to
estimate the maximum resolution that can be computed in the required time and com-
pute only that. The first frames are processed exactly like shown in 3 or 4 and then,
when a sufficient amount of frames computed could provide an accurate estimation of
the required time for each resolution, use the estimator to only compute the maximum
resolution possible.
4.5.2. IMPReTi results
The first experiment performed is aimed at determining how the resolution of the sil-
houette affect the final output of the heel strike and toe off detection. We have tested
4 different resolutions (1920x1080, 960x540, 480x270, 240x135 pixels respectively). The
silhouette output of each resolution is shown in figure 4.26. We compare the output of
the second filter method described in section 4.3.
Figure 4.27 shows the RMSE of the heel strike and toe off detection compared to a
manual marking, as described in section 4.3.3, for each resolution. There is practically
no difference between 1080 and 540 resolutions in terms of accuracy.
Gait-A 108
Figure 4.26: Silhouettes computed in each resolution. From left to right: 1080, 540,270, 135 pixels height.
0
0,5
1
1,5
2
2,5
1080 540 270 135
RMSE
Heel Strike
Toe Off
Figure 4.27: RMSE result of the heel strike and toe off detection algorithm for eachresolution.
Figure 4.28 shows the amount of undetected cases for each resolution. In this case, 1080
shows the best results closely follow by 540.
Finally, Figure 4.29 shows the processing time required for each resolution as well as
the frames per second (FPS). The processing time increases linearly along with the
resolution size, this is nothing new since the complexity of the background subtraction
algorithm used is linear (number of pixels in the image). This chart shows the greatest
differences between each resolution. Therefore it is precisely time the parameter that
establishes which resolution to use.
The second experiment checks the real-time suitability of the algorithm 3. The IMPReTi
algorithm is executed with 4 different FPS constraints: 15, 30, 60 and 120. The time
estimator is computed as shown in equation 4.16. The time of the next resolution is
estimated to be 4 times bigger than the last since the next resolution have 4 times more
pixels than the previous one. Since the objective of our project is to develop a low-cost
Gait-A 109
0
2
4
6
8
10
14
1080 540 270 135
Undetected cases
Heel Strike
Toe Off
Figure 4.28: Amount of undetected cases of heel strike and toe off for each resolution
0
20
40
60
80
100
120
140
160
1080 540 270 135
Time
Processing Time
FPS
Figure 4.29: Processing time in milliseconds and FPS required for silhouette extrac-tion phase for each resolution.
system, we use a conventional operating system without real time capabilities where is
difficult to estimate the amount of time certain operation will take to complete. If we
add multicore capabilities it becomes even difficult. Therefore we decided to use the
theoretical time. We use the 4 resolutions of the previous experiment.
elapsedT ime+ lastT ime× 4 < deadline (4.16)
Figure 4.30 shows the adjustment to the deadline performed by the algorithm with each
time constraint. The graph represents the idle time for each frame processed, i.e., the
remaining time until the deadline is reached. As shown in the graph, 120 FPS is the
limit at which the algorithm can work under the tested hardware. The reason is that
the minimum time required for computing the smaller resolution is greater or equal to
the time constraint. Improvements can be made by increasing the number of levels of
the pyramid but smaller resolutions could compromise the accuracy of the results. The
graph also shows a great variability in processing time, that is due to the use of a general
purpose operating system without real time capabilities.
Gait-A 110
-20
-10
0
10
20
30
40
50Id
le T
ime
in
ms 15 FPS
30 FPS
60 FPS
120 FPS
Figure 4.30: Graph showing the idle time for each frame with different time con-straints. A negative value means that the time constraint is exceeded.
4.6. Gait classification
The variables obtained in the previous module are here processed to perform a classi-
fication between normal and abnormal gait. We have different types of variables so we
cannot use the same classifier for all of them. Depending on the type of variables we
propose different classifiers.
4.6.1. K-Nearest Neighbours for single value variables classification
These are variables that contains a single value for each sample, e.g., the speed of
movement. Although there is a single value, we can represent each sample with several
of these variables creating a feature vector. In this case we propose the following feature
vector:
Fv = {θ, λ, β} (4.17)
Where θ is the cycle time in seconds, λ is the stride length in meters and β is the speed
in meters per second.
Gait-A 111
To obtain these variables we use the HS obtained in the previous module. Let H be an
ordered set of HS:
H = {h1, h2, . . . , hn} (4.18)
Where each hi = 〈t, x, y〉 is a 3-tuple where t represents the frame in which it is produced
and x, y the two dimensional position of the heel in screen space in that instant. We
obtain cycle time as shown in equation 4.19 where frame rate is expressed as frames per
second (FPS). The stride length is obtained as shown in equation 4.20 where d(hx,yi , hx,yi+2)
is the euclidean distance between the position of the HS. Finally, we obtain the speed
as shown in equation 4.21.
θ =1
n− 2
n−2∑i=1
hti+2 − htiframe rate
(4.19)
λ =1
n− 2
n−2∑i=1
d(hx,yi , hx,yi+2)× pixel width (4.20)
β =d(hx,y1 , hx,yn )
htn − ht1× pixel width (4.21)
4.6.2. K-Nearest Neighbours and Dynamic Time Warping for time
series classification
Each sample has a different number of cycles and different length so we have to deal
with these issues first. The former issue is solved by splitting the sample in each cycle
using the HS obtained by the feature extraction module. The latter is solved by using
DTW as a measure of the similarity of each cycle time series. For this purpose, DTW
has great potential as it can provide a similarity measure of arbitrary length time series
which is exactly the case of gait.
We obtain the cycle i as the corresponding value of stride width or leg angle in each of
the frames between hti and hti+2 for each i ≤ n− 2. Let F be a set of feet locations for
each frame:
F = {f1, f2, . . . , fn} (4.22)
Where each fi = 〈cbh, cbt,mfh,mft〉 is a 4-tuple where cbh is the CBF heel, cbt the
CBF toe, mfh the MFF heel and mft the MFF toe, and all of them are two dimensional
positions. Then the stride width time series is obtained as shown in equation 4.23.
Stsj =
htj+2⋃
i=htj
d(f cbhi , fmfti ) (4.23)
Gait-A 112
Let S be a set of skeleton joints locations for each frame:
S = {s1, s2, . . . , sn} (4.24)
Where each si = 〈head, neck, hip, cbk, cbf,mfk,mff〉 a 7-tuple containing the two di-
mensional position of head, neck, hip, CBK, CBF, MFK and MFF respectively. Then
leg angle is obtained as shown in equation 4.25.
Ltsj =
htj+2⋃
i=htj
scbfi shipi smff
i (4.25)
4.6.3. Principal Component Analysis and SVM for Gait Energy Image
classification
Using the HS we segment the gait sequence and obtain a GEI [55] for each cycle. A
GEI is obtained by averaging all the silhouettes of the cycle aligned by the COG of its
bounding box. We compute a GEI for each cycle to obtain more samples to train and
thus requiring HS to segment the cycle. The processing to obtain HS can be avoided if
we average all the silhouettes of a sequence simplifying the acquisition of the input, but
we obtain one for each cycle to obtain a large amount of samples. We compute each
GEI as described in equation 4.26.
GEIi =1
hti+2 − hti
hti+2∑
k=hti
sk (4.26)
Being sk the silhouette of frame k and i = {1, 2, . . . , n− 2} the corresponding HS of the
set H.
The main issue of GEI is the amount of dimensions it has to compare. For instance,
we are using 50x50 pixels, that means we have to compare 2500 components for each
sample. A large amount of pixels are empty so there are some pixels that are not
necessary to compare so an optimization can be performed. We use PCA to select the n
main eigenvectors of all the training samples and then project all the samples into this
new eigenspace. Now we only compare the n principal components instead of 2500 using
SVM. This method is similar to the eigenfaces proposed by Turk and Pentland[101] and
it is also applied by Man and Bhanu [55] for person identification using GEI.
Each recorded gait sample provides an average of 4 gait cycles. The training process
usually needs a large amount of samples to obtain enough information to perform a
classification. That means we would have to record a lot of gait sequences to obtain a
Gait-A 113
1080 540 270 180
Resolution
Figure 4.31: This figure shows the different synthetic samples obtained from theoriginal (left).
sufficient amount of samples. To solve this issue we propose the use of synthetic samples
to increase the size of the training set.
For such purpose we use the multiresolution approach proposed in 4.5. This method
uses different resolutions to compute the silhouette providing a new synthetic sample for
each subsampled resolution. By obtaining these samples, we provide robustness against
scale variations.
This method transforms the original samples in new synthetic samples by adding arte-
facts that could be produced in reality. We obtain 4 times more samples as shown in
figure 4.31.
4.6.4. Results
We propose different experiments be carried out on our datasets. We first analyse if the
proposed variables are suitable for classifying normal and abnormal gait. To do so we
accordingly performed a 10-fold and leave-one-out cross validations to finely measure
the accuracy of each classifier.
Table 4.6 shows side view results of the three dimensional feature of stride length, cycle
time and speed; the stride time series; and the leg-angle time series. The most accurate
results obtained are those in which we used KNN with leg-angle variables, 100% accuracy
rate. Also a good accuracy rate was obtained by KNN and SVM with three dimensional
features: cycle time, stride length and speed. Poorer results, however, are those offered
by the KNN with stride-width time series in which we tested each cycle separately, but
its level of accuracy was highly increased when we tested each cycle of the same subject
and output the mode class predicted in a number of cycles.
Gait-A 114
Table 4.6: 10-fold and leave-one-out cross-validation results of each classifier. C1:KNN with DTW using stride width testing each cycle, C2: KNN with DTW using legsangle testing each cycle, C3: KNN with DTW using stride width testing each subject,C4: KNN with DTW using legs angle testing each subject, C5, C6 and C7: Bayes,
KNN and SVM respectively using cycle time, stride length and speed
Classifier 10-Fold leave-one-out
C1C2C3C4C5C6C7
71.43%95.71%
80%100%
93.33%96.67%96.67%
74.65%97.18%83.33%100%
93.33%96.67%96.67%
The classification rates exhibit 90% or even 100% of success and suggest that the methods
proposed offer a reliable classification of normal and abnormal gaits even when a single
cycle is classified. This means that the methods are suitable for Ambient Assisted Living
environments where a camera set in a house in a transit area could monitor gait patterns
of every person that comes into the camera’s view.
It is true that the classification results are promising but there is still much work to do
in testing subtle abnormal gait patterns.
In the case of frontal view sequences, the results shown by table 4.7 are not as good,
but are improved by testing all the cycles of the same sample and outputting the class
predicted in most cases.
Table 4.7: 10-fold and leave-one-out cross-validation results of each classifier. C1:KNN with DTW using stride width testing each cycle, C2: KNN with DTW using legsangle testing each cycle, C3: KNN with DTW using stride width testing each subject,
C4: KNN with DTW using legs angle testing each subject.
Classifier 10-Fold leave-one-out
C1C2C3C4
74.64%71.42%
80%87.5%
79.3%77.19%83.72%88.37%
In the last experiment we use the third recorded dataset with side view samples of
five subjects feigning four different pathologies and normal gait. We recorded three
samples of each subject and class which provided a total of 75 samples. The selected
gait pathologies were diplegic, hemiplegic, neuropathic and parkinsonian.
We processed the new dataset with the HSTOD algorithm to obtain the HS to separate
each gait cycle. We first tried the legs-angle time series with KNN like in previous
experiments. Results, shown in table 4.8, were not good as diplegic and hemiplegic are
Gait-A 115
Table 4.8: Confusion matrix of legs-angle with DTW and KNN
Actual
Normal Diplegic Hemiplegic Neuropathic Parkinsonian
Predicted
Normal 89.13% 0.00% 2.35% 18.45% 0.00%Diplegic 6.52% 47.27% 47.06% 4.85% 6.83%Hemiplegic 2.17% 14.55% 32.94% 7.77% 0.00%Neuropathic 2.17% 0.00% 12.94% 62.14% 0.00%Parkinsonian 0.00% 38.18% 4.71% 6.80% 93.17%
0,5
0,55
0,6
0,65
0,7
0,75
5 10 15 20 25 30 35 40 50 100 200 1000
Acc
ura
cy R
ate
Principal Components
Figure 4.32: Classification with different number of principal components
frequently confused and neuropathic is confused with normal gait. Normal gait and
parkinsonian were really well classified confirming the results of Khan et al. [69]. We
then tried a new approach. For each gait cycle, a GEI is obtained as described in the
previous chapter. Figures 4.33, 4.34, 4.35, 4.36 and 4.37 show a GEI of each class. To
assess the accuracy of the classifier we use 5-fold cross-validation using for each fold
all the samples of the same subject as test and the other subjects as training including
synthetic samples in this case as well. This ensures that the subject silhouette has not
been learned by the classifier. We apply PCA to reduce the dimensionality of the input
images. Several classifications using SVM were performed with a different number of
principal components as shown in Figure 4.32. That allowed to determine that the
optimal value is 50 which is the value where the curve stabilises. Therefore, 50 is the
value used in the next experiments.
The confusion matrix in table 4.9 shows an accuracy of 71.17%. It also show that hemi-
plegic and neuropathic gait are frequently confused between each other. Also diplegic
is confused with parkinsonian, the other way around as well but in a lesser degree.
Therefore, a new classifier capable of classifying neuropathic and hemiplegic and an-
other for classifying parkinsonian and diplegic is needed. The fact that the couples
Gait-A 116
Figure 4.33: Normal gait
Figure 4.34: Diplegic gait Figure 4.35: Hemiplegic gait
Figure 4.36: Neuropathic gait Figure 4.37: Parkinsonian gait
Table 4.9: Confusion matrix of PCA with GEI and SVM
Actual
Normal Diplegic Hemiplegic Neuropathic Parkinsonian
Predicted
Normal 89.13% 11.82% 3.53% 9.71% 0.00%Diplegic 0.00% 40.00% 1.18% 4.85% 7.45%Hemiplegic 2.17% 9.09% 62.35% 13.59% 0.00%Neuropathic 8.70% 1.82% 32.94% 71.84% 0.00%Parkinsonian 0.00% 37.27% 0.00% 0.00% 92.55%
hemiplegic-neuropathic and diplegic-parkinsonian are well distinguished provides a way
of developing a hierarchical classifier, the first level classifies between normal, hemiplegic-
neuropathic and diplegic-parkinsonian. Then in the next level two binary classifiers
should deal with each couple.
Both couples cannot be classified with the first classifier because their shape is very
similar. To differentiate hemiplegic from neuropathic we use the legs-angle time series
Gait-A 117
0102030405060708090
100
Classification Results
Legs-angle
GEI
GEI Reclass.
GEI No diplegic
Figure 4.38: Graphic showing the classification rates of the different classifiers used.
Table 4.10: Confusion matrix of PCA with GEI and SVM after applying reclassifica-tion of hemiplegic-neuropathic with legs angle time series and KNN.
Actual
Normal Diplegic Hemiplegic Neuropathic Parkinsonian
Predicted
Normal 89.13% 11.82% 3.53% 9.71% 0.00%Diplegic 0.00% 40.00% 1.18% 4.85% 7.45%Hemiplegic 2.17% 10.00% 78.82% 12.62% 0.00%Neuropathic 8.70% 0.91% 16.47% 72.82% 0.00%Parkinsonian 0.00% 37.27% 0.00% 0.00% 92.55%
because the results of table 4.8 show that those two pathologies are well distinguished
with legs-angle. In the case of diplegic and parkinsonian, we did not find a suitable clas-
sifier to deal with the problem, however, frontal view could provide a way to differentiate
them. After applying the reclassification of hemiplegic-neuropathic, the resultant con-
fusion matrix is as shown in table 4.10 where the accuracy is slightly improved to reach
74.66%. This accuracy can be improved to 80.33% if for each GEI of each test sample we
output the mode class, i.e., the class that is predicted in most cases. Removing diplegic
(Table 4.11), the results improve to 86.52% reaching a 95% applying the mode. Figure
4.38 shows the accuracy of the different classifiers used.
Gait-A 118
Table 4.11: Confusion matrix of PCA with GEI and SVM after applying reclassifi-cation of hemiplegic-neuropathic with legs angle time series and KNN and removing
diplegic gait.
Actual
Normal Hemiplegic Neuropathic Parkinsonian
Predicted
Normal 82.61% 2.35% 0.97% 0.00%Hemiplegic 4.35% 80.00% 15.53% 0.00%Neuropathic 13.04% 16.47% 83.50% 0.00%Parkinsonian 0.00% 1.18% 0.00% 100.00%
4.7. Periodical gait test
The early detection of frailty and dementia could improve the quality of life of elderly
people because an early treatment could slow down the deterioration of gait and the
consequent health decay. However, this is not an easy task since it is difficult to predict
in advance symptoms of frailty or dementia. We need to provide a gait deterioration
value to be able to compare different states of the same subject to view its evolution
over time. By analysing its evolution we could determine where it starts decaying and
take care of the problem as soon as possible.
Only analysing gait we can not establish exactly if the subject is frail or not. The frailty
syndrome is defined as a clinical syndrome in which three or more of the following criteria
are present: unintentional weight loss, self-reported exhaustion, weakness, slow walking
speed, and low physical activity [35]. Of these five criteria, only the last two (maybe
three) could be identified using gait analysis. Therefore we can not establish whether
the subject is frail or not, but we can establish a risk factor providing the geriatric with
an early indication of the presence of possible frail symptoms.
This requires a large amount of time to properly test the results. Since this thesis project
has a time span of 4 years, it is out of the scope of it. A proper test of what we propose
here will be perform as future work.
Chapter 5
Conclusion
This thesis proposes a classification of decisions made by engineers to design solutions
for the engineering problems they face. We have divided the initial problem in three
subproblems:
Model subproblem. It is the part of the problem that belong to the scope of one of
its model and is shared with all the problems of that model.
Specific subproblem. It is the part of the problem unique to it that it is not share
with the rest of the problems that belong to the same model.
Contextual subproblem. It is the part of the problem related to the circumstances
in which the problem is solved, i.e., the project planning.
This division of the problem in three subproblems allowed us to establish a formal basis
to propose a taxonomy of R&D:
Basic research: if it is resolved the model subproblem. The result is theoretical
knowledge.
Applied research: if it is resolved the specific subproblem. Resolving a single
instance of a problem could be considered innovation whether resolving a set of
problems could be considered development. The result is particular knowledge.
Procedural innovation: if it is resolved the contextual subproblem.
We have obtained five classes of design factors:
119
Conclusion 120
A class of decisions directed to a family of the problem, that are abstract design
decisions about the model. We have linked this abstraction to the question what
is the problem, that allow us to know how the problem is.
Three classes of decisions related to solve the specific subproblem through concre-
tion of the model. These are decisions about the architecture, the structure and
the technology, respectively. We have linked this concretion to the sequence of
questions what for is the problem, how and with what is the solution, respectively.
All of these design factors that we have called essential factors have special names:
explicit factors are the ones related to the question what for, which are architec-
tural; implicit factors those related to the question how, which are structural; and
intrinsic factors those related to the question with what, which are technological.
A class of design decisions directed to the implementation of a prototype. Those are
decisions to solve the contextual subproblem. We have linked the circumstances
of implementation to all the other remaining questions: who, how much, when,
where, etc. These design factors has been called contextual factors.
We have ordered the five classes of design factors in the sequence:
model > architecture > structure > technology > context
Therefore, we have established a formal basis and have obtained a method to make
design decisions in a causally coherent way.
The hypothesis represents a version of the problem close to the model domain and the
architecture. Therefore, in order for the design to be causal and formally correct, it’s
necessary to start making decisions related to the model that provides the framework
for the solution. Then continue making decisions sequentially in the order that we have
established. This provides:
– A technique to define a model for a given problem and a set of other related
problems.
– A technique to obtain a functional specification of a given problem based on a
model and a conjunction of architectural objectives. This specification ignores any
constraint coming from the domain of the solution, so it takes into account only
constraints established in the hypothesis, avoiding premature prune in the solution
tree.
– A technique to propose a structural specification of a solution consisting of a tree of
modules and an organizational graph reflecting the relationships among modules.
Conclusion 121
– A discussion about the way to make design decisions related to contextual factors.
The most relevant aspect of those techniques is related to the observational nature of
the involved knowledge, instead of other conventional ways of making design decisions
based on non formal inspirational sources (experience, preferences, etc.). Therefore,
a procedural definition of knowledge creation in terms of human perception has been
proposed.
Within each class of design factors, criteria of precedence to determine which design
decisions have to be made before and which ones have to be subordinated, are dependent
of the optimization objectives and, therefore, of each case. E.g., the design decisions on
the structure (choice of one or the other requirements, structural tree deeper or with less
levels, etc.) are driven to optimization, which is dependent on each solution. This could
be the seed to help engineers on making design decisions by means of computerized and
even artificial intelligent frameworks, which is the future work we plan to develop.
This procedure was validated with a real use case. The chosen use case was the de-
sign and development of a computer vision based system for gait analysis to detect
the syndromes of frailty and senility. A functional specification was given following the
methodology proposed. We started by providing a gait definition for our problem estab-
lishing examples and counterexamples to delimit the model. Then we proceeded with the
functional specification of the problem establishing its objectives and the requirements
that cover them. We recursively defined the other levels of the specification providing a
structural tree. Finally the organizational graph was provided by adding relationships
between leaf nodes.
The resulting system, capable of automatically extracting kinematic features charac-
terising human gait, runs as follows: we first input RGB image sequences and various
classifiers to determine whether the output can be interpreted as normal or abnormal
gait. To meet our objective, we designed a setup to capture images of walking subjects
in a controlled environment. The camera was placed at right angles to the direction
of the gait and in front of it. We then carried out a series of experiments out of the
recorded dataset to validate our approach.
In the first experiment, we determined the accuracy of HSTOD algorithm for both
side and frontal view; in the second, the accuracy of the FASE algorithm; in our third
experiment, the validity of the classifiers we engaged in the process; finally, in the fourth
experiment, we determined the suitability of the real-time capabilities with the IMPReTi
method.
Conclusion 122
The first experiment, testing the HSTOD algorithm, show that, in side view, of the
two filters used, the first delivers less accuracy than the second filter but fewer unde-
tected cases in normal gait analysis. However, given that our objective is to characterise
abnormal gait, it is the second filter that best serves our purposes. In frontal view,
experiment showed that the approach that relies on zero crosses does not provide good
enough results in abnormal gait, this issues is solved with the approach based on local
maxima and minima.
The second experiment, testing the FASE algorithm, show that the algorithm is better
suited to extract head, neck and hip position points than feet and leg position points
due to its inability to distinguish each leg during swing phase.
The third experiment was carried out to test the validity of the classifiers, and show
that, for the task of classifying between normal and abnormal gait, the leg-angle time
series classifier with KNN offers the best classification results. The three-dimensional
feature consisting of cycle time, stride length and speed also offers quite an acceptable
degree of accuracy. The experiment with normal, diplegic, hemiplegic, neuropathic and
parkinsonian gaits shows that diplegic and parkinsonian are confused frequently between
each other. The same problem is present with hemiplegic and neuropathic, but in this
case, a hierarchical classifier is able to improve results up to 74% from the 71% of the
non-hierarchical one.
The fourth experiment determined that the system is robust against smaller input images
and so the IMPReTi optimization allows the system to work up to 100 FPS (with the
tested hardware) with a small reduction in accuracy.
Note finally that the HSTOD algorithm copes well with a delay of 3-4 frames after
applying the filters, and that the FASE algorithm can tackle frame by frame, that
along with the IMPReTi improvement, the system can provide measurements at video
frequency so long as the computation time is lower than frame time.
5.1. Limitations
Although we provided a methodology aimed at reducing the subjectivity of the solution
to a problem, the current state of its development requires a human engineer to make
decisions. The methodology provides tools for the engineer to make design decisions
limiting their arbitrariness, however, as the engineer is human, subjectivity of each
individual is still present. There is still work to do in automatizing the process.
Conclusion 123
Regarding the computer vision system, the current solution presents several limitations.
The main limitation is the environment, side view images need a large room to be able
to obtain a sufficient amount of steps. Frontal view requires less space but provides less
accuracy. The input sequence has to contain a static background with a static camera,
i.e., the subject’s movement is the only movement allowed in the recording. Also a
sufficient contrast between the subject and the background is required to perform the
background subtraction, especially in the area of feet and legs.
The study was perform with a rather small size dataset and the subjects were healthy
people feigning some pathological gait. Although the study was sufficient to assess the
suitability of the proposed system, a larger study with real patients is required to validate
the proposal in a real scenario.
5.2. Future work
The future work will focus in addressing some of the limitations previously mentioned.
The possibility of using a treadmill for recording the gait samples will be asses. This
could potentially reduce the size of the environment needed to record the samples.
Our future work will also revolve around improving the FASE algorithm to enable us to
overcome the problem of readily locating each leg and both arms.
The possibility of real-time processing opens up new possibilities like providing correc-
tions to the subject’s gait in real-time or balance assistance. Regarding the balance
assistance, we are contemplating the possibility of developing a control moment gyro-
scope (CMG) system to reduce the risks of balance loss situations. We based this in the
hypothesis that a group of CMG appropriately oriented could provide a force vector in
any direction to compensate a balance loss or, at least, reduce the falling speed.
Finally, the classification method using GEI is already been tested using Convolutional
Neural Network. Preliminary results obtained show an increase of around 10% in accu-
racy. These preliminary results are expected to improve in the future.
Appendix A
Implementation of Gait-A
The Gait-A prototype has been implemented for two platforms: PC and Android. The
implementation uses OpenCV as the core image processing library. The PC version was
used to perform all the test shown in in this thesis, the Android version was implemented
to test the performance in an embedded device such as a smartphone.
A.1. PC implementation
As previously mentioned, the PC version uses OpenCV 2.4.6 as the main image process-
ing library and it is programmed in Visual C++ 2010 and .NET Framework 4.0. The
implementation follows the module structure described in Chapter 4.
Figure A.1 shows a screen capture of the program running with a smartphone captured
sample. The circles highlight the following elements:
1. A control that allows to flip the sample, i.e., if the subject is walking from right
to left then this control will flip the sample so the subject walks from left to right
or vice-versa.
2. A control to indicate if the sample is frontal view.
3. A control to indicate the desired frame rate. This will activate the IMPReTi
optimization to try to adjust to the selected frame rate.
4. A control to indicate a region of interest, cutting the sample image.
5. The bounding box of the silhouette.
6. Different classifiers output indicating if the gait is normal or abnormal.
125
Implementation of Gait-A 126
1 2 3 4
5
7
8
9
6
1211
10
14
13
Figure A.1: Gait-A PC implementation GUI.
7. Time in milliseconds of each module.
8. Skeleton joints extracted by FASE module.
9. Lean angle in degrees.
10. Bounding box width time series.
11. Toe off detected.
12. Leg-angle in degrees.
13. Tracked heels and toes of each foot.
14. Heel strike detected.
A.2. Android implementation
The Android implementation is programmed in Java using the OpenCV NDK wrapper
provided by the OpenCV 2.4.6 release. The app was developed in a Samsung Galaxy
S3 with Android 4.3 Jelly Bean.
Implementation of Gait-A 127
1
2 3
4
5
6
Figure A.2: Gait-A Android implementation GUI.
Figure A.2 shows a screen capture of the app where the circles highlight the following
elements:
1. Heel gradient.
2. Bounding box of the silhouette.
3. Time in milliseconds of each module.
4. Skeleton joints extracted by FASE module.
5. Toe off detected.
6. Heel strike detected.
Bibliography
[1] Ken Kundert. A formal top-down design process for mixed-signal circuits. Ad-
vances in Analog Circuit Design, 2000.
[2] Ken Kundert. Principles of top-down mixed-signal design. The Designers Guide
Community, 2003.
[3] Valentino Crespi, Aram Galstyan, and Kristina Lerman. Top-down vs bottom-up
methodologies in multi-agent system design. Autonomous Robots, 24(3):303–313,
2008.
[4] Martti Mantyla. A modeling system for top-down design of assembled products.
IBM Journal of Research and Development, 34(5):636–659, 1990.
[5] Xiang Chen, Shuming Gao, Youdong Yang, and Shuting Zhang. Multi-level assem-
bly model for top-down design of mechanical products. Computer-Aided Design,
44(10):1033–1048, 2012.
[6] Douglas C Schmidt. Model-driven engineering. COMPUTER-IEEE COMPUTER
SOCIETY-, 39(2):25, 2006.
[7] Krishnakumar Balasubramanian, Aniruddha Gokhale, Gabor Karsai, Janos Szti-
panovits, and Sandeep Neema. Developing applications using model-driven design
environments. Computer, 39(2):33–40, 2006.
[8] Jon Whittle, John Hutchinson, and Mark Rouncefield. The state of practice in
model-driven engineering. IEEE software, 31(3):79–85, 2014.
[9] Gunter Mussbacher, Daniel Amyot, Ruth Breu, Jean-Michel Bruel, Betty HC
Cheng, Philippe Collet, Benoit Combemale, Robert B France, Rogardt Heldal,
James Hill, et al. The relevance of model-driven engineering thirty years from
now. In International Conference on Model Driven Engineering Languages and
Systems, pages 183–200. Springer, 2014.
[10] Dimitrios S Kolovos, Louis M Rose, Nicholas Matragkas, Richard F Paige, Es-
ther Guerra, Jesus Sanchez Cuadrado, Juan De Lara, Istvan Rath, Daniel Varro,
129
Bibliography 130
Massimo Tisi, et al. A research roadmap towards achieving scalability in model
driven engineering. In Proceedings of the Workshop on Scalability in Model Driven
Engineering, page 2. ACM, 2013.
[11] Junjie Huang and Heming Zhang. Multi-granularity modeling of virtual prototyp-
ing in collaborative product design. In Computer Supported Cooperative Work in
Design, 2008. CSCWD 2008. 12th International Conference on, pages 710–715.
IEEE, 2008.
[12] Chun Zhang, Huachao Mao, Gongzhuang Peng, and Heming Zhang. A novel bom
based multi-resolution model for federated simulation. In Computer Supported
Cooperative Work in Design (CSCWD), 2013 IEEE 17th International Conference
on, pages 178–183. IEEE, 2013.
[13] A Van Wijngaarden. Orthogonal design and description of a formal language.
Stichting Mathematisch Centrum. Rekenafdeling, (MR 76/65):1–25, 1965.
[14] Balanjaninath Edupuganty and Barrett R. Bryant. Two-level grammar as a func-
tional programming language. The Computer Journal, 32(1):36–44, 1989.
[15] Barrett R Bryant. Object-oriented natural language requirements specification. In
Computer Science Conference, 2000. ACSC 2000. 23rd Australasian, pages 24–30.
IEEE, 2000.
[16] Barrett R Bryant and Beum-Seuk Lee. Two-level grammar as an object-oriented
requirements specification language. In System Sciences, 2002. HICSS. Proceed-
ings of the 35th Annual Hawaii International Conference on, pages 3627–3636.
IEEE, 2002.
[17] Dines Bjørner and Cliff B Jones. The vienna development method: The meta-
language. Lecture notes in computer science, 61, 1978.
[18] EH Durr and N Plat. Vdm++ language reference manual. Afrodite (ESPRIT-III
project number 6500), Cap Volmac, 1995.
[19] Gino Brunetti and Borut Golob. A feature-based approach towards an integrated
product model including conceptual design information. Computer-Aided Design,
32(14):877–887, 2000.
[20] Feng Qi. A assembly modeling method based on assembly feature graph-tree
model. In Industrial Engineering and Engineering Management, 2009. IE&EM’09.
16th International Conference on, pages 1657–1662. IEEE, 2009.
Bibliography 131
[21] Otto Harald E. From concepts to consistent object specifications: translation of
a domain-oriented feature framework into practice. Journal of Computer Science
and Technology, 16:208–230, 2001.
[22] J Michael Spivey. The z notation: a reference manual. international series in
computer science, 1992.
[23] Roger Duke, Gordon Rose, and Graeme Smith. Object-z: A specification language
advocated for the description of standards. Computer Standards & Interfaces, 17
(5):511–533, 1995.
[24] Daniel Jackson. Alloy: a lightweight object modelling notation. ACM Transactions
on Software Engineering and Methodology (TOSEM), 11(2):256–290, 2002.
[25] Fernando Gomez, Carlos Segami, and Carl Delaune. A system for the semiau-
tomatic generation of er models from natural language specifications. Data &
Knowledge Engineering, 29(1):57–81, 1999.
[26] Deva Kumar Deeptimahanti and Muhammad Ali Babar. An automated tool for
generating uml models from natural language requirements. In Proceedings of the
2009 IEEE/ACM International Conference on Automated Software Engineering,
pages 680–682. IEEE Computer Society, 2009.
[27] MG Ilieva and Olga Ormandjieva. Automatic transition of natural language soft-
ware requirements specification into formal presentation. In Natural Language
Processing and Information Systems, pages 392–397. Springer, 2005.
[28] Farid Meziane, Nikos Athanasakis, and Sophia Ananiadou. Generating natural
language specifications from uml class diagrams. Requirements Engineering, 13
(1):1–18, 2008.
[29] Jean-Michel Bruel and Robert B France. Transforming uml models to formal
specifications. The Unified Modeling Language, UML, 1998.
[30] Derek Coleman, Patrick Arnold, Stephanie Bodoff, Chris Dollin, Helena Gilchrist,
Fiona Hayes, and Paul Jeremaes. Object-oriented development: the fusion method.
Prentice-Hall, Inc., 1994.
[31] Jacquelin Perry, Jon R Davids, et al. Gait analysis: normal and pathological
function. Journal of Pediatric Orthopaedics, 12(6):815, 1992.
[32] Tim Theologis and Julie Stebbins. The use of gait analysis in the treatment of
pediatric foot and ankle disorders. Foot and ankle clinics, 15(2):365–382, 2010.
Bibliography 132
[33] Florence I Mahoney. Functional evaluation: the barthel index. Maryland state
medical journal, 14:61–65, 1965.
[34] J Waltson and LP Fried. Frailty and the old man. Med Clin North Am, 83(5):
1173–1194, 1999.
[35] Linda P Fried, Catherine M Tangen, Jeremy Walston, Anne B Newman, Calvin
Hirsch, John Gottdiener, Teresa Seeman, Russell Tracy, Willem J Kop, Gregory
Burke, et al. Frailty in older adults evidence for a phenotype. The Journals
of Gerontology Series A: Biological Sciences and Medical Sciences, 56(3):M146–
M157, 2001.
[36] MB van Iersel. The interplay between gait and cognitive function in elderly people.
[Sl: sn], 2007.
[37] Jeffrey M Hausdorff, Galit Yogev, Shmuel Springer, Ely S Simon, and Nir Giladi.
Walking is more like catching than tapping: gait in the elderly as a complex
cognitive task. Experimental Brain Research, 164(4):541–548, 2005.
[38] Theo Mulder, Wiebren Zijlstra, and Alexander Geurts. Assessment of motor re-
covery and decline. Gait & posture, 16(2):198–210, 2002.
[39] LM Waite, DA Grayson, O Piguet, H Creasey, HP Bennett, and GA Broe. Gait
slowing as a predictor of incident dementia: 6-year longitudinal data from the
sydney older persons study. Journal of the neurological sciences, 229:89–93, 2005.
[40] R Camicioli, D Howieson, B Oken, G Sexton, and J Kaye. Motor slowing precedes
cognitive impairment in the oldest old. Neurology, 50(5):1496–1498, 1998.
[41] Joe Verghese, Richard B Lipton, Charles B Hall, Gail Kuslansky, Mindy J Katz,
and Herman Buschke. Abnormality of gait as a predictor of non-alzheimer’s de-
mentia. New England Journal of Medicine, 347(22):1761–1768, 2002.
[42] J Verghese, C Derby, MJ Katz, and RB Lipton. High risk neurological gait syn-
drome and vascular dementia. Journal of neural transmission, 114(10):1249–1252,
2007.
[43] N Scarmeas, M Albert, J Brandt, D Blacker, G Hadjigeorgiou, A Papadimitriou,
B Dubois, M Sarazin, D Wegesin, K Marder, et al. Motor signs predict poor
outcomes in alzheimer disease. Neurology, 64(10):1696–1703, 2005.
[44] Jennifer S Brach, Subashan Perera, Stephanie Studenski, and Anne B Newman.
The reliability and validity of measures of gait variability in community-dwelling
older adults. Archives of physical medicine and rehabilitation, 89(12):2293–2296,
2008.
Bibliography 133
[45] Jeffrey M Hausdorff. Gait variability: methods, modeling and meaning. Journal
of neuroengineering and rehabilitation, 2(1):1, 2005.
[46] Vincenzo Solfrizzi, Emanuele Scafato, Vincenza Frisardi, Davide Seripa, Gian-
carlo Logroscino, Stefania Maggi, Bruno P Imbimbo, Lucia Galluzzo, Marzia
Baldereschi, Claudia Gandin, et al. Frailty syndrome and the risk of vascular
dementia: the italian longitudinal study on aging. Alzheimer’s & Dementia, 9(2):
113–122, 2013.
[47] Gotaro Kojima, Yu Taniguchi, Steve Iliffe, and Kate Walters. Frailty as a predic-
tor of alzheimer disease, vascular dementia, and all dementia among community-
dwelling older people: A systematic review and meta-analysis. Journal of the
American Medical Directors Association, 2016.
[48] Yumi Iwashita, Koji Uchino, and Ryo Kurazume. Gait-based person identification
robust to changes in appearance. Sensors, 13(6):7884–7901, 2013.
[49] Aditi Roy, Shamik Sural, and Jayanta Mukherjee. Gait recognition using pose
kinematics and pose energy image. Signal Processing, 92(3):780–792, 2012.
[50] Yumi Iwashita, Adrian Stoica, and Ryo Kurazume. Gait identification using
shadow biometrics. Pattern Recognition Letters, 33(16):2148–2155, 2012.
[51] Robin M Queen, Benjamin B Haynes, W Mack Hardaker, and William E Garrett.
Forefoot loading during 3 athletic tasks. The American journal of sports medicine,
35(4):630–636, 2007.
[52] Shuai Zheng, Kaiqi Huang, Tieniu Tan, and Dacheng Tao. A cascade fusion scheme
for gait and cumulative foot pressure image recognition. Pattern Recognition, 45
(10):3603–3610, 2012.
[53] Ivan Gonzalez, Jesus Fontecha, Ramon Hervas, and Jose Bravo. An ambulatory
system for gait monitoring based on wireless sensorized insoles. Sensors, 15(7):
16589–16613, 2015.
[54] Michael W Whittle. Gait analysis: an introduction. Butterworth-Heinemann,
2014.
[55] Ju Man and Bir Bhanu. Individual recognition using gait energy image. IEEE
transactions on pattern analysis and machine intelligence, 28(2):316–322, 2006.
[56] Aditi Roy, Shamik Sural, and Jayanta Mukherjee. A hierarchical method com-
bining gait and phase of motion with spatiotemporal model for person re-
identification. Pattern Recognition Letters, 33(14):1891–1901, 2012.
Bibliography 134
[57] Robert T Collins, Ralph Gross, and Jianbo Shi. Silhouette-based human identi-
fication from body shape and gait. In Automatic Face and Gesture Recognition,
2002. Proceedings. Fifth IEEE International Conference on, pages 366–371. IEEE,
2002.
[58] Wei Zeng, Cong Wang, and Feifei Yang. Silhouette-based gait recognition via
deterministic learning. Pattern Recognition, 47(11):3568–3584, 2014.
[59] Adrian Leu, D Ristic-Durrant, and Axel Graser. A robust markerless vision-
based human gait analysis system. In Applied Computational Intelligence and
Informatics (SACI), 2011 6th IEEE International Symposium on, pages 415–420.
IEEE, 2011.
[60] Jang-Hee Yoo and Mark S Nixon. Automated markerless analysis of human gait
motion for recognition and classification. Etri Journal, 33(2):259–266, 2011.
[61] Xu Xu, Raymond W McGorry, Li-Shan Chou, Jia-hua Lin, and Chien-chi Chang.
Accuracy of the microsoft kinect for measuring gait parameters during treadmill
walking. Gait & posture, 2015.
[62] Sruti Das Choudhury and Tardi Tjahjadi. Silhouette-based gait recognition using
procrustes shape analysis and elliptic fourier descriptors. Pattern Recognition, 45
(9):3414–3426, 2012.
[63] David Cunado, Mark S Nixon, and John N Carter. Automatic extraction and
description of human gait models for recognition purposes. Computer Vision and
Image Understanding, 90(1):1–41, 2003.
[64] ChewYean Yam, Mark S Nixon, and John N Carter. Automated person recognition
by walking and running via model-based approaches. Pattern Recognition, 37(5):
1057–1072, 2004.
[65] Sruti Das Choudhury and Tardi Tjahjadi. Gait recognition based on shape and
motion analysis of silhouette contours. Computer Vision and Image Understand-
ing, 117(12):1770–1785, 2013.
[66] Dorthe Meyer, Josef Posl, and Heinrich Niemann. Gait classification with hmms
for trajectories of body parts extracted by mixture densities. In BMVC, pages
1–10. Citeseer, 1998.
[67] Christian Bauckhage, John K Tsotsos, and Frank E Bunn. Automatic detection
of abnormal gait. Image and Vision Computing, 27(1):108–115, 2009.
Bibliography 135
[68] Liang Wang. Abnormal walking gait analysis using silhouette-masked flow his-
tograms. In Pattern Recognition, 2006. ICPR 2006. 18th International Conference
on, volume 3, pages 473–476. IEEE, 2006.
[69] Taha Khan, Jerker Westin, and Mark Dougherty. Motion cue analysis for parkin-
sonian gait recognition. The open biomedical engineering journal, 7:1, 2013.
[70] Rumesh Krishnan, M Sivarathinabala, and S Abirami. Abnormal gait detection
using lean and ramp angle features. In Computational Intelligence in Data Min-
ingVolume 1, pages 325–335. Springer, 2016.
[71] Barry Boehm. Anchoring the software process. IEEE software, 13(4):73–82, 1996.
[72] Jo E Hannay, Dag IK Sjøberg, and Tore Dyba. A systematic review of theory use
in software engineering experiments. Software Engineering, IEEE Transactions
on, 33(2):87–107, 2007.
[73] Dawn G Gregg, Uday R Kulkarni, and Ajay S Vinze. Understanding the philo-
sophical underpinnings of software engineering research in information systems.
Information Systems Frontiers, 3(2):169–183, 2001.
[74] Dag IK Sjoberg, Tore Dyba, and Magne Jorgensen. The future of empirical meth-
ods in software engineering research. In 2007 Future of Software Engineering,
pages 358–378. IEEE Computer Society, 2007.
[75] Steve Easterbrook, Janice Singer, Margaret-Anne Storey, and Daniela Damian. Se-
lecting empirical methods for software engineering research. In Guide to advanced
empirical software engineering, pages 285–311. Springer, 2008.
[76] Klaas-Jan Stol and Brian Fitzgerald. Uncovering theories in software engineer-
ing. In Software Engineering (GTSE), 2013 2nd SEMAT Workshop on a General
Theory of, pages 5–14. IEEE, 2013.
[77] Roel Wieringa, Maya Daneva, and Nelly Condori-Fernandez. The structure of
design theories, and an analysis of their use in software engineering experiments.
In Empirical Software Engineering and Measurement (ESEM), 2011 International
Symposium on, pages 295–304. IEEE, 2011.
[78] Maurice Landry. A note on the concept of’problem’. Organization studies, 16(2):
315–343, 1995.
[79] Gene P Agre. The concept of problem. Educational Studies, 13(2):121–142, 1982.
[80] Alistair M Preston. The problem in and of management information systems.
Accounting, Management and Information Technologies, 1(1):43–69, 1991.
Bibliography 136
[81] Thomas S Kuhn. The structure of scientific revolutions. University of Chicago
press, 2012.
[82] Edsger W Dijkstra. Solution of a problem in concurrent programming control.
In Pioneers and Their Contributions to Software Engineering, pages 289–294.
Springer, 2001.
[83] Werner Heisenberg. Uber den anschaulichen inhalt der quantentheoretischen kine-
matik und mechanik. Zeitschrift fur Physik, 43(3-4):172–198, 1927.
[84] Mark S Nixon and John N Carter. Advances in automatic gait recognition. In
Automatic Face and Gesture Recognition, 2004. Proceedings. Sixth IEEE Interna-
tional Conference on, pages 139–144. IEEE, 2004.
[85] Mario Nieto-Hidalgo, Francisco Javier Ferrandez-Pastor, Rafael J Valdivieso-
Sarabia, Jeronimo Mora-Pascual, and Juan Manuel Garcıa-Chamizo. Vision based
extraction of dynamic gait features focused on feet movement using rgb camera.
In Ambient Intelligence for Health, pages 155–166. Springer, 2015.
[86] Claude Banville and Maurice Landry. Can the field of mis be disciplined? Com-
munications of the ACM, 32(1):48–60, 1989.
[87] Imed Bouchrika and Mark S Nixon. Model-based feature extraction for gait anal-
ysis and recognition. In Computer vision/computer graphics collaboration tech-
niques, pages 150–160. Springer, 2007.
[88] Zoran Zivkovic and Ferdinand van der Heijden. Efficient adaptive density estima-
tion per image pixel for the task of background subtraction. Pattern recognition
letters, 27(7):773–780, 2006.
[89] Faezeh Tafazzoli and Reza Safabakhsh. Model-based human gait recognition using
leg and arm movements. Engineering applications of artificial intelligence, 23(8):
1237–1246, 2010.
[90] Casia gait database. URL http://www.sinobiometrics.com.
[91] Lance Williams. Pyramidal parametrics. In ACM Siggraph Computer Graphics,
volume 17, pages 1–11. ACM, 1983.
[92] Tony Lindeberg. Scale-space theory: A basic tool for analyzing structures at
different scales. Journal of applied statistics, 21(1-2):225–270, 1994.
[93] Tony Lindeberg. Scale-space theory in computer vision, volume 256. Springer
Science & Business Media, 2013.
Bibliography 137
[94] William T Freeman and Edward H Adelson. The design and use of steerable
filters. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE
INTELLIGENCE, 13(9):891, 1991.
[95] Edward H Adelson, Charles H Anderson, James R Bergen, Peter J Burt, and
Joan M Ogden. Pyramid methods in image processing. RCA engineer, 29(6):
33–41, 1984.
[96] Joan M Ogden, Edward H Adelson, James R Bergen, and Peter J Burt. Pyramid-
based computer graphics. RCA Engineer, 30(5):4–15, 1985.
[97] Congyong Su, Yueting Zhuang, Li Huang, and Fei Wu. Steerable pyramid-based
face hallucination. Pattern Recognition, 38(6):813–824, 2005.
[98] Magnus Strengert, Martin Kraus, and Thomas Ertl. Pyramid methods in gpu-
based image processing. Proceedings Vision, Modeling, and Visualization 2006,
pages 169–176, 2006.
[99] Sylvain Paris, Samuel W Hasinoff, and Jan Kautz. Local laplacian filters: edge-
aware image processing with a laplacian pyramid. Communications of the ACM,
58(3):81–91, 2015.
[100] Arvind R Yadav, RS Anand, ML Dewal, and Sangeeta Gupta. Gaussian image
pyramid based texture features for classification of microscopic images of hardwood
species. Optik-International Journal for Light and Electron Optics, 126(24):5570–
5578, 2015.
[101] Matthew Turk and Alex Pentland. Eigenfaces for recognition. Journal of cognitive
neuroscience, 3(1):71–86, 1991.
