Basic  Study  on  Life-‐‑‒logging  Video  Capture  via  Neuraltalk22016/1/15

-‐‑‒ Research  Note  -‐‑‒

About  myself

• Motohiko Takeda（@kusojig /  takeda@rabbitsdata.jp )

• Interests:• Data  Based  Marketing• Machine  Learning,  Data  Mining• Data  Strategy  Planning  &  Execusion

• Currently  freelance  consultant  for  data  utilizationand  sensor  technologies.  Mainly  experienced  consulting  projects  for  mobile,  baking  and  advertising  industries.

Motivation:  to  estimate  the  time  for  housekeeping

• Some  households  have  conflict  about  burdensharing  about  housekeeping.  

• I  spend  a  lot  of  time  in  kitchen…• Thatʼ’s  not  true.  I  also  help  dishes!• Oh  it  doesnʼ’t  take  much  time.  I  pay  more!

• OK,  So  why  donʼ’t  we  analyze  housekeeping  cost  quantitatively  by  using  technologies?

Approach:  Computer  Vision  with  AI  (Neuraltalk2)

• Set  web  camera  at  the  top  of  dinning  room  and  take  photos  for  each  15  seconds.

• Caption  the  picture  by  Neuraltalk2(Neuraltalk2  gives  caption  by  describing  the  picture).

• Estimate  housekeeping  time  by  categorizing  the  caption  data.

Setup  WEB  camera  around  the  celling  and  regulated  by  PC

Kitchen Dinning  table

Refrigerator  

Overview  of  room  and  camera

Camera  keeping• Camera  connected  with  laptop  PC  via  USB

Camera  and  laptop  PC

Caption  results  were  almost  collect  except  detailed  actions• Neuraltalk2  recognized  “a  man/woman   standing   in  a  kitchen”  when  somebody  stands  around  the  kitchen.

• However,   the  same  capture  were  given  when  somebody   is  sitting  in  the  dinning  table  near  the  kitchen.

• It  could  not  recognized  more  detailed  action  like  opening   the  refrigerator.

Estimated   time  of  staying  around  the  kitchen  could  explainthe  amount  of  activity  in  a  day

00 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Estimated   time  of    “man/woman   standing  in  a  kitchen”    per  hour

Prepare  for  tomorrow  meal*  48min/hour  at  a  peak

Wake  up

Prepare  for  a  supper

Lunch  andhaving  a  tea

• Largely  measured   the  day  activity  around  the  kitchen.

Supper

*  Stayed  home  for  whole  day

Issues:  individual  identification  and  intersection

• Issues  1:  individual  identifiation• We  have  not  implemented  individual  identification.• For  identifying  spent  time  for  each  task,  implementation  of  individual  identification  including  detection  of  side-‐‑‒face  is  necessary.

*  In  case  of  husband/wife  identification,  only  gender  identification  might  be  enough.

• Issues  2:  Intersection  • Pictures  in  house  generally  happens  intersection  since  room  has  small  space  compared  with  public  space.

• Attention  towards  the  tilt  of  camera  is  needed.

Issues:  individual  identification  and  intersection

Conclusion  and  future  tasks

• Even  no  customized  AI  program  can  identify  the  activity  time  and  patterns  in  the  room  largely.  

• By  implementing  individual  identification,  time  spent  in  housekeeping  could  be  identified  in  particular.

• By  taking  life-‐‑‒log  in  more  detail,  we  can  develop  the  recommendation  for  daily  life.

• This  project  is  still  on  progress  (Jan.  2016)

For  more  information…

• We  have  developing  the  research  and  development  for  sensor  technologies,  machine  learning  and  deep  learning.

• For  more  information,  please  feel  free  to  contact:  takeda@rabbitsdata.jp