Tangible video communication over the Internet

This document is downloaded from DR‑NTU (https://dr.ntu.edu.sg)Nanyang Technological University, Singapore.

Tangible video communication over the Internet

Guo, Song; Sourin, Alexei

2018

Guo, S., & Sourin, A. (2018). Tangible video communication over the Internet. Proceedingsof the Computer Graphics International 2018, 253‑260. doi:10.1145/3208159.3208183

https://hdl.handle.net/10356/143185

https://doi.org/10.1145/3208159.3208183

© 2018 Association for Computing Machinery. All rights reserved. This paper was publishedin Proceedings of the Computer Graphics International 2018 and is made available withpermission of Association for Computing Machinery.

Downloaded on 21 Dec 2021 14:03:44 SGT

Tangible Video Communication over the Internet

Song Guo Fraunhofer Singapore

Singapore [email protected]

Alexei Sourin Nanyang Technological University

Singapore [email protected]

ABSTRACT Video-conferencing 1 and video phone calls are common nowadays. However more immersion can be achieved if we not only see and hear but also physically feel each other. In contrast to previous attempts which were mostly setting up integral video-haptic communication systems, we propose a different way where we use any existing video communication tools while asynchronously exchanging over the Internet cloud server very little data packets containing only haptic interaction data. In case of desktop haptic devices, we only exchange between the participating computers haptic interface point coordinates, orientation angles of the device handles and computed force vectors. More options for haptic communication appear when we replace common video cameras with depth-sensing cameras and hand-tracking devices. Then, motion of the hand, arm or even the whole body of one party can be both seen and physically felt by the other party participating in such Internet tangible video communication. We validated the proposed way of communication by technical measurements of the transmission times over the LAN and Amazon Cloud server, as well as by conducting a successful user study.

CCS CONCEPTS • Human-centered computing ~ User studies • Human-centered computing ~ Virtual reality

KEYWORDS Haptic interaction, new user-computer interfaces, internet communication, video communication

1 INTRODUCTION Applications like Skype have enabled us to see and talk to people who are thousands of miles away. However, it will be

much more desirable if we can physically feel and interact with each other, just like it is written and shown in science fictions novels and movies. In real life, handshaking, holding hands, touching and hugging are common ways of expressing feelings or indicating affections and trust. In computing, sense of touch is implemented by using various haptic devices which deliver forces to the users through vibrations as well as displacements and rotations of haptic actuators. However, though quite developed for interaction in virtual environments on a single computer, haptic communication using networked computers is a very challenging problem. It is commonly implemented by sharing combined audio-visual and haptic content across the Internet, and this often constitutes a prohibitive overhead to the required 1,000 Hz haptic forces generation frequency. In contrast, our research hypothesis is to use any existing video communication tools on computers and mobile devices while asynchronously exchanging over the Internet cloud very little data packets containing only haptic interaction data.

In Section 2, we are writing about existing attempts of adding the sense of touch to Internet communication and videos. In Section 3, we are presenting our feasibility study on adding the sense of touch to networked haptic communication within local area network. In Section 4, we propose how to perform asynchronous haptic interaction across the Internet using cloud server. In Section 5, we report and analyze the results of the user study on the haptic video communication using the Amazon Cloud server. A discussion on possible serious application of tangible video communication is given in Section 6. Section 7 concludes the paper and outlines the future work.

2 ADDING THE SENSE OF TOUCH TO INTERNET COMMUNICATION AND VIDEO

Delivering haptic sensation during communication over the Internet may be classified into two groups. Either /1/ both video and haptic communications are combined into one pipeline and, eventually, into one final product, or /2/ only haptic interaction over the Internet without involving video/audio communication is used.

The first approach mostly focuses at delivering haptic gestures of affective communication such as handshakes, hugs, kisses, etc. [1—5], however it often requires hardware which is sophisticated and non-intuitive in use. Indirect haptic interaction through avatars in virtual worlds was also considered [6]. Networked collaborative environments with haptic modality were used in education, training, medical therapy, etc. [7, 8]. Besides this, a number of attempts have been made to enable haptic interaction with video content by making objects in a video scene available for haptic exploration by the user. All these methods focused primarily on the mechanisms of haptic content

CGI 2018, June 11–14, 2018, Bintan Island, Indonesia Song Guo and Alexei Sourin

compression and haptic device design. Thus, passive haptic interaction with the color video of the remote networked participant was proposed in [9]. O’Modhrain and Oakley [10] studied how haptic modality can be added to the television viewing experience. They presented an analysis of how active and passive touch can be added to authored content or real-time scenarios. Addition of the haptic modality to the movie watching experience was also proposed in [11] where the video content author had to decide what area of the movie and what frame had to be haptically accessible. Pre-recorded haptic information was then annotated to the video in a time-referenced manner and replayed to the viewer’s device which followed the specific trajectory. In [12], Sung et al. proposed streaming of haptic information in a particular format alongside the video. Cha [13] proposed a comprehensive framework for creating, transmitting and rendering of haptic video based on depth information generated and transmitted to the viewer. Another method that augmented to video function-defined 3D objects following certain objects or regions was presented in [14]. However, in all these methods, making video tangible only allows for feeling it by one side—the viewer. Both parties, while exchanging video streams across the Internet during video conversation and thus seeing each other, cannot have mutual haptic contact, e.g., they feel each other hand motion.

In the second approach, haptic interaction over the Internet without involving video/audio communication is mostly exploring the use of custom-made devices like vibrating vests, rings, etc. Here, both fan applications were proposed as well as serious investigations were done on how, for example, vibrations can help communicate in combat training where voice communication is not permitted. Further example is the work by Israr et al. [15] who proposed a technology for integrating haptic effects into theater seats, gaming chairs, as well as vests, rides, gloves, shoes, etc. In this work, the haptic effects were carefully created and synchronized with the audio-visual cues to deliver an immersive experience. Another example is FeelCraft [16] which is a plugin that monitors activities in media and associates them with expressive tactile patterns from a library of effects. Danieau et al. [17] proposed two models of haptic cinematography based on motion of the camera. The first model made the audience feel the movement of the camera while the second one offered haptic metaphors related to the semantics of the camera effect. Furthermore, Danieau [18] also presented a new device providing 6 degrees of freedom (DOF) motion effects for moving the head and hands of the user along with an authoring tool to generate haptic effects that could be rendered by it. Last but not least, in [19], a method for rendering motion of the user-selected objects on a SPIDAR device was presented.

However, to the best of our knowledge, the problem of adding the sense of touch to video communication is still very much open, and there is no reliable tangible video communication method proposed and implemented. It motivated us to come up with our solution which is presented in this paper.

3 FEASIBILITY STUDY ON TANGIBLE VIDEO INTERACTION

We do not propose to follow the direction of making an integrated solution by combining vision and touching interaction into one pipeline/product, as most researchers did so far. Comparing to the haptic force refresh rate, which is usually as

high as 1,000 Hz, to make the force feedback convincing, video streams are often captured at 60 Hz. Separating haptic and visual pipelines can simplify both processes. We also have no intention to compete with such well-established video communication software like Skype—their users will be reluctant to use anything different from what they already accustomed to. Instead, we propose to use any existing video communication applications on computers or mobile devices without modifications while exchanging between the Internet-connected computers very little data packets containing only haptic interaction data. Hence, video communication and haptic interaction will be performed asynchronously and even using different computational platforms, e.g., video communication may run on mobile devices, while haptic interaction will be done on notebooks with haptic devices. This approach requires from us to come up with an efficient method of exchanging haptic data packets between the involved computers.

To illustrate and prove the correctness of this approach, we have first performed a feasibility study simulating “handshaking” with two networked computers located in different rooms of the same lab. The 3DOF desktop haptic devices (Geomagic Touch) connected to each of them were programmed to follow the other party hand motions. This was done by transmitting via the network (LAN, winsock), only the haptic device coordinates from one computer to another while both devices used the same coordinate system. Based on the proximity of the device coordinates received over the network to the coordinates retrieved from the local haptic device, the remote computer calculated a “magnetic force” that attracted the haptic interface points of the two devices towards each other. In other words, a force field, which existed around the haptic interface point (HIP) position of each device in its coordinate space, was used. As the two force fields virtually overlapped, the remote computer applied the force on the remote device and transmitted updated position via the network to the local computer. The same update rule was applied to the local computer. This means that the two devices became virtually ‘attached’ to each other, and whenever the local device handle was moved, the remote computer enforced motion of its device handle towards the local device handle in the virtual communication space. As a result, motion of the device handle on one computer produced the same motion of the device handle on the remote computer.

Meanwhile, a Skype video connection was established between the two users so that they could see each other, as well as their hands and haptic device handles. During this feasibility study, we used different ways of making Skype video calls, including webcams and mobile phones anticipating that asynchronous haptic and video connections will motivate the future users to enjoy more freedom in choosing the ways of video communication.

We then further verified this interaction method using two different types of desktop haptic devices: 3DOF Geomagic Touch and 6DOF Phantom Premium. While the 3DOF device is able to capture information about the handle position and orientation in 6DOF (position and orientation of the handle), it only allows for producing 3DOF forces relocating the HIP (position of the tip of the handle). In contrast, the 6DOF device allows for both 6DOF capturing and producing 6DOF forces which change both the handle position and orientation. Therefore, in this case of non-homogeneous haptic interaction, not only relocation of the HIP, but also 3 rotation angles of the handle were transmitted from

Tangible video Communication over the Internet CGI 2018, June 11–14, 2018, Bintan Island, Indonesia

3

the 3DOF device to the 6DOF device. This enabled for more possibilities for haptic interaction, as the user with the 6DOF haptic device could feel the displacement and changing orientation of the hand motion of the user with the 3DOF device. See, for example, Fig. 1a, where user “2” with the 6DOF device (marked “b”) feels the direction and rotation of the haptic handle of the 3DOF device (marked “a”) which is moved by user “1”. We anticipated that this method would allow for remote learning hand motor skills.

(a)

(b)

Figure 1: (a) Video chat with virtual handshake using 3DOF and 6DOF haptic devices. (b) Video chat with communication from Creative Senz3D to a 3DOF haptic device.

Furthermore, we considered a possibility of connecting over the network haptic and non-haptic devices. In this case, the communication becomes one-directional and the haptic device side has to act as a receiver, since non-haptic device cannot respond to haptic data transmitted to give haptic feedback to the user. In such communication, the non-haptic device should convert other forms of signals to haptic signals and deliver them to the haptic device. One typical non-haptic user input is hand movement captured by various optical trackers. By obtaining the user’s hand positions, it is possible to make the remote haptic device move accordingly.

We implemented and studied networked haptic interaction where motion of one user hand captured by depth sensors (we used Creative Senz3D camera and Leap Motion Controller) was transferred across the network to another user through the desktop haptic device, while both users were in Skype video communication. For Leap Motion Controller, an extra webcam or

mobile phone was required, while Creative Senz3D camera allowed for simultaneous hand tracking and Skype video call.

In Fig. 1b, user “2” is holding the handle of the 3DOF device (marked “b”) to feel the right hand motion of user “1”. A separate window “c” is shown to user “1” to monitor the tracking status of the hand as seen by the camera (marked “a”).

In contrast to haptic device interaction space, the space in front of the camera is not physically restricted. Therefore we had to set a rule to map the captured hand position in this space to the haptic device’s coordinate system. To achieve natural interaction, we exactly mapped the displacements of the hand to the displacements of the haptic device handle. However, we had to find a way how to ensure that the hand moving in open unrestricted space will always stay in contact with the haptic handle which motion is restricted. To achieve it, we defined a bounding box for the hand motion considering the hand’s initial location to be a center of it. Then, the motion of the hand within this bounding box maps to the motion of the haptic handle. However, if the hand leaves the bounding box, it will follow the hand, and the center of the bounding box will relocate to the new hand position outside the original bounding box. Whenever necessary, like in case of changes in a tracking status, we can also redefine the origin and relocate the bounding box anytime accordingly to expand the effective interaction space.

We have further studied whether Microsoft Kinect can be used for the same camera-to-haptic device communication, however we found out that it is less efficient since Kinect does not capture precisely the hand position and orientation, and such inaccuracy leads to vibrations of haptic device handle. It still however could be used for capturing the whole arm movement with some data smoothing.

For transmitting the haptic data across the network, we defined a protocol for setting up such communication. Like during a phone call, any communication between two parties involves processes like handshaking, termination and exception handling. This process involves defining which of the two computers must initiate different procedures, and how the connection closes, including how the system will handle abrupt loss of connection due to unforeseen circumstances. We assigned one computer to be a host and another one to be a client. A predefined synchronizing message consisting of special characters is exchanged between the two computers periodically to re-synchronize them in case of a significant network delay, e.g., when long distance communication is done. In case of lost connection, each computer still keeps track of missed (un-received) ‘synchronizing’ messages. When a certain number of these messages has been accumulated, the connection is considered as lost and further communication is terminated until the connection will be re-established.

We then performed a successful interim technological user study to validate whether the potential users would accept the proposed way of communication. The results were previously reported in [20, 21], and in this research we used EEG data captured from the user to understand how well the users would accept such kind of interaction compared to common Skype video conversation.

4 MAKING TANGIBLE VIDEO INTERACTION OVER THE CLOUD SERVER

Previous attempts to make haptic interaction over the Internet were based on establishing TCP/IP connections, however it is


not always possible to do due to various VPN and firewalls used, and it is rather complicated for setting up for the end users. Therefore, we considered communication over cloud server like it is done for home IP cameras. To exchange data between devices rapidly, a fast data forwarding server host was needed. Several existing solutions can be used: commercial, like PubNub (www.pubnub.com), as well as some with database storage function, like Redis (redis.io). We chose MQTT broker, which is lightweight open source supporting SSL user authentication. It uses MQTT protocol—a topic-based protocol on top of TCP/IP that commonly used on IoT (Internet of Things) devices, where each device acts as a client and sends/receives messages through the broker (mqtt.org). The MQTT broker that we use is mosquito (mosquitto.org). It is also freeware and can run on both Windows and Linux-based systems.

We deployed the broker on the Amazon Web Service EC2 instance based on the physical location of device for lower latency in communication. However, there are many other cloud service providers and data exchange protocols, like, for example, Microsoft Azure Cloud, and besides this it is also possible to setup a private cloud server.

Figure 2: Structure of various tangible video communications.

As illustrated in the upper part of Fig. 2, once a connection of a pair of devices to the broker was established, we made one device subscribe to its remote target’s topic (i.e. listen to and receive data from) and publish its own data to another topic that the remote side subscribes to. Each pair of two haptic devices is then linked by two separate topics. This is done to avoid extra information needed to identify source and target devices in data communication, as each pair of topics is assigned during establishment of connection, and no headers are needed like those in IP packets. However, as illustrated in the lower part of Fig. 2, only one topic is needed in case of a one-directional camera-to-haptic communication.

After implementation of the proposed cloud communication, we then performed a technological study of the impact of introducing haptic communication to computer by measuring resources required.

To reduce network load, we only transferred data required for haptic device movement which was positional data of a handle or a hand. In further applications, rotational data was also added. Such data could either be encapsulated in double

precision or single precision floating-number format. Network load caused by both precisions were examined for comparison.

With reference to Table 1, we observed that haptic communication consumed much lower network resource than video chatting. Moreover, the bandwidth required for sending data using double precision was slightly more than twice of that of single precision. On the other hand, the amount of memory used for haptic communication was relatively large, especially for camera-to-haptic device connection. Most probably this was caused by the high throughput of the data transmission.

Table 1: Resource consumption

Interface Precision Network (total, KBps) RAM (MB)

FaceTime NA 890 NA Skype NA 370 145

Haptic client double 132.9 217.8 Haptic client float 82.6 112.2

Camera Client double 68.8 253.1 Camera Client float 45.1 239.6

As a result, we made the precision of data adjustable

according to the real-time server status, like bandwidth and latency, to ensure a stable connection under different network conditions.

Unlike in the direct network communication, the service hosted in the cloud server is completely asynchronous. Therefore, it is necessary to monitor connections of both parties in a separate ‘Control’ topic. When clients connect to or disconnect from the server, they must broadcast their status. To link two devices, one client initiates with a handshake message to request a connection to another device. If the target device accepts the request, the communication is started and the remote side begins listening and returning data. Otherwise, a message of rejection can be sent and no communication will happen. Any party can terminate the connection at any time, and the remote side would close the connection when a notification is received. If high latency or even loss of connection is detected, the clients can send warning message to the control topic, so that the remote side can close connection and stop transmitting data whenever necessary to prevent meaningless traffic load for the MQTT broker.

5 USER STUDY After successful feasibility study of the proposed tangible video communication over the LAN and collecting technical measurements of the transmission times over the Internet cloud server, we implemented and tested tangible Skype communication over the Internet.

We used two computers (Intel i7 6700, with 32GB RAM, running Windows 10) located in different physical locations on the university campus and connected to the Internet. Both were equipped with 3DOF desktop haptic device Geomagic Touch and depth-sensing camera Creative Senz3D.

We have developed a pilot user interface for connecting devices to the Internet Cloud server. The interface was designed in such a way that all the connections could be done intuitively and in a few mouse clicks only. The users had to first establish Skype communication, then they had to register in the cloud (Go Online/Offline button in Fig. 3a). By clicking an available remote device’s icon, they could initiate a connection, while the other


5

participant received a notification to answer the request (pop-up window in Fig. 3b). They could either connect two haptic devices or a haptic device with a depth-sensing camera.

(a)

(b)

Figure 3: User Interface. (a) Cloud registration (b) Pairing Two haptic interaction modes were provided: normal and

mirrored. In the normal mode, the device handle seen in Skype video from remote side acted as the user’s handle/hand, which means it moved forward when the user moved handle or hand forward. In the mirror mode, the device handle of the remote side observed in Skype looked like a mirror image of the user, since moving forward leads to a backward movement of the remote device handle. In case of a hand-tracking camera used in place of the haptic device, a new window popped up displaying the location of the hand being tracked with reference to the currently used bounding box matching the remote haptic space. As it was discussed in Section 3, if the hand leaves the bounding box (for more than 2 sec), the box will follow the hand and the center of the bounding box will relocate to the new hand position outside the original bounding box.

We used 24 participants in the experiment: 18 males and 6 females, aged from 21 to 38 (Mean=27.8 SD=4.25). They were undergraduate and graduate students as well as researchers and engineers, hence having the same or matching educational background. 17 were experienced haptic users while others had little or no such experience and therefore were given 5 minutes of induction haptic exercises. The participants were grouped into pairs based on their cultural, educational and language similarities to avoid possible social awkwardness that could affect the experiment.

In each session of the user study, a pair of the participants was asked to go through four interaction tasks:

Task 1. The participants greeted and talked to each other on

Skype for a few minutes. Task 2. Haptic communication using 3DOF haptic devices for

both participants was added to the Skype video chat (Fig. 4a). The participants were also exploring how to toggle between two modes of interaction: normal and mirrored.

Task 3. One of the users switched to use camera for hand tracking in place of the haptic device while the other user kept receiving positional data on his haptic device (Fig. 4 b, c, d).

Task 4. The users switched the ways of using the devices (user 1 – haptic device, user 2 – hand tracking camera) so that both could try both modes of operation: with the haptic device and with the camera.

After the experiment, the questionnaires with the following 4 statements were given to the participants:

QUESTIONNARIE 1: Video chat with Haptic-to-Haptic communication is more

interesting than pure video chat. 2: Video chat with Haptic-to-Camera communication is more

interesting than pure video chat. 3: Video chat with Camera-to-Haptic communication is more

interesting than pure video chat. 4: Function of toggling between Normal Mode and Mirror Mode

is helpful.

In this questionnaire we only looked at the evaluation of the overall experience and its comparison with the common Skype conversation. The participants were asked to rate their experience on the five-point Likert scale according to the extent how they agree with the statements, where 5 indicates agree and 1 means disagree. The analyzed statistics obtained in the user study is shown in Table 2 and Fig. 5.

Table 2: User study analysis

Statement 1 2 3 4

Average score 4.25 4.083333 4.041667 4.125000

Std Deviation 0.944089 0.880547 0.999094 1.115601

95% Conf Interval 0.398654 0.371822 0.42188 0.471077

To conclude whether the obtained results are statistically

significant, we considered that the null hypothesis of our study is a score equal or smaller than 3, which would mean that bringing haptic experience into video chat has no positive effect in terms of joy for the users. As it can be seen in Fig. 5, all the error bars that depicts 95% confidence intervals do not include the null hypothesis, i.e. any value smaller than 3. Therefore, the null hypothesis can be rejected. In other words, the results are significant.


(a)

(b)

(c)

(d)

Figure 4: User study of haptic video interaction over the Internet cloud server. (a) Mutual haptic contact (b) “High-five” from the depth camera to the haptic device (c) “Gentle touch” from the depth camera to the haptic device

(d) An attempt to scratch the back with the haptic device handle.

Figure 5: User study analysis chart based on a 5-point scale.

Besides scoring the statements in the questionnaire, some of the participants provided useful comments which: • Requested for a more natural device like a haptic glove:

“It would be better if it is something wearable like a glove.”, “It would be more interesting if the haptic device can be modified to other types.”

• Provided comments how to improve the interaction: “Camera to haptic should have an option to be scaled. Big hand movements to smaller device movements, so that you don’t leave the interaction box.” “I wonder if the dot (that represents the hand) could change in radias based on depth towards screen.” “May take some time to get used to the motion of the device in relation to your hand.” “3D mapping to be improved.” “Arm fatigue, not sure what to do.”

• Evaluated the overall experience: “Very interesting! Depending on usage of device writing vs other operations” “Camera tends to override user with haptic” “Good when A needs B to see from A's point of view”.

Indeed, the available haptic devices are unfortunately artifacts of last century technology, and a technological breakthrough has to be done to bring them to every home. We have also considered using Novint Falcon however its limited manipulation space only allowed for implementing “handshake” across the internet and an ability to feel hand vibrations of the remote party. The users still commented that they would be happy to use this way of Internet communication if the haptic device would be provided to them.

Experience arm fatigue is a common side effect in haptic interaction. However it can be easily learnt how to rest an elbow on the desk to avoid it.

Other comments left room for further improvements but overall this final user study has further confirmed that our method of adding haptic communications to pure video chat can indeed add more fun to the users. We then explored how much educational it can be if serious applications are considered.

Statement number

Average scores and 95% confidence intervals

Scor

es


7

6 TOWARDS SERIOUS APPLICATIONS OF TANGIBLE VIDEO COMMUNICATION

Besides potential social application like handshaking and touching over the Internet, we also considered possible serious application scenarios which may require complex hand manipulation.

Thus, in medical area, spinal injections involve injecting anesthetic and anti-inflammatory medication into designated regions on the back of the patient. Common training for this procedure is based on apprenticeship model where trainees observe the doctor, which require them to be in the same room. With the 6DOF haptic device used in the training classroom, trainees can be located elsewhere and not only observe the correct movements on the video monitor, but also directly feel them.

We have performed a feasibility study on such remote training using the same settings as in Section 5. The application was developed using Unity 3D to reconstruct the 3D scene and the receiver had to follow the instructor’s movement. The required data was recorded and sent to the server in real-time. On the visual receiver side, the Unity 3D program with an integrated MQTT client was able to receive the data and to apply the transformations accordingly on the 3D model of a needle to render the real handle’s pose in the virtual space. Meanwhile, the movement receiver was equipped with MQTT client that listens for instructions and moves the handle accordingly. There was no need to additionally transmit or display the video scene since it is already seen in the Skype window.

(a)

(b)

Figure 5: Spinal injection simulation. a) Host is sending positions and rotations of a 3DOF device’s handle. b)

Haptic data is received by a 6DOF device which handle follows the motion of the instructor’s handle while visual demonstration is shown on screen.

In this feasibility study, the instructor used a 3DOF device to perform the virtual injections, which was recorded both for position and rotation of the handle. As the handle moved, the receiving 6DOF device’s handle copied its movement (Fig. 5). In the same way many minimally invasive surgeries can be also simulated and remotely trained, which we are going to verify in our future work based on our previous research in this area where virtual surgical raining was based on using the actual surgical videos [22].

7 CONCLUSION In this paper, we have proposed a way of making common video communication more immersive by adding touch modality to it. In contrast to the previous attempts, we use any existing video communication tools while asynchronously exchanging over the Internet cloud server very little data packets containing only haptic interaction data. To increase usability for the users and reduce required network bandwidth, we proposed a communication structure and designed the connection control logic. The method has been proved to be effective in making video communication more interesting.

We have performed the user study with the computers physically located in the same country while being connected over the Internet cloud server. However the developed software still requires further testing of the tangible video communication between different countries and with different cloud servers. This is what we are going to do in our immediate future work.

ACKNOWLEDGEMENTS This research is supported by the National Research Foundation, Prime Minister’s Office, Singapore under its International Research Centers in Singapore Funding Initiative.

REFERENCES [1] Mueller F. F., Vetere F., Gibbs M. R., Kjeldskov J., Pedell S., and Howard S.

2005. Hug Over a Distance. In Proceedings of CHI '05 Extended Abstracts on Human Factors in Computing Systems, Portland, OR, USA.

[2] Jongeun C., Eid M., Rahal L., and Saddik A. E., HugMe. 2008. An Interpersonal Haptic Communication System. In Proceedings of Haptic Audio visual Environments and Games, 2008. HAVE 2008. IEEE International Workshop on, pp. 99-102.

[3] Tsetserukou D., HaptiHug, 2010. A Novel Haptic Display for Communication of Hug Over a Distance. Haptics: Generating and Perceiving Tangible Sensations. vol. 6191, A. L. Kappers, J. F. van Erp, W. Bergmann Tiest, and F. T. van der Helm, Eds., ed: Springer Berlin Heidelberg, pp. 340-347.

[4] Saadatian E., Samani H., Parsani R., Pandey A. V., Li J., Tejada, L. et al. 2014. Mediating Intimacy in Long-distance Relationships Using Kiss Messaging, International Journal of Human-Computer Studies, vol. 72, pp. 736-746, 10.

[5] Nakanishi H., Tanaka K., and Wada Y. 2014. Remote Handshaking: Touch Enhances Video-mediated Social Telepresence, in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Toronto, Ontario, Canada.

[6] Tsetserukou D., Neviarouskaya A., Prendinger H., Ishizuka M., and Tachi S. 2010. iFeel_IM: Innovative Real-Time Communication System with Rich Emotional and Haptic Channels. In Proceedings of CHI '10 Extended Abstracts on Human Factors in Computing Systems, Atlanta, Georgia, USA.

[7] Wei L., Sourin A. 2011. Function-based Approach to Mixed Haptic Effects Rendering. The Visual Computer. Springer, 27(4):321-332.

[8] Flowers M. G. and Aggarwal R. 2014. Second Life™: A Novel Simulation Platform for the Training of Surgical Residents. Expert Review of Medical Devices, vol. 11, pp. 101-103, 2014/03/01.


[9] Rasool S. and Sourin A. 2014. Image-Driven Haptic Rendering. Transactions on Computational Science XXIII. vol. 8490, M. Gavrilova, C. J. K. Tan, X. Mao, and L. Hong, Eds., ed: Springer Berlin Heidelberg, pp. 58-77.

[10] O'Modhrain S. and Oakley I. 2004. Adding Interactivity: Active Touch in Broadcast Media. In Proceedings of the 12th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, 2004. HAPTICS '04. , Chicago, USA, 293-294.

[11] Gaw D., Morris D., and Salisbury K. 2006. Haptically Annotated Movies: Reaching out and Touching the Silver Screen. In Proceedings of the 14th Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, Alexandria, USA, 287-288.

[12] Sung M.Y., Jun K., Ji D., Lee H., and Kim K. 2009. Touchable Video And Tactile Audio. In Proceedings of the 11th IEEE International Symposium on Multimedia. ISM '09, San Diego, USA, 425-431.

[13] Cha J., Eid M., and El Saddik A. 2009. Touchable 3D Video System. ACM Trans. Multimedia Comput. Commun. Appl., TOMMCAP 5, 4 October, 1-25.

[14] Rasool S. and Sourin A. 2010. Towards Tangible Images And Video in Cyberworlds--Function-based Approach. In Proceedings of the 2010 International Conference on Cyberworlds (CW), Singapore, 92-96.

[15] Israr A., Kim S.-C., Stec J., and Poupyrev I. 2012. Surround Haptics: Tactile Feedback for Immersive Gaming Experiences. In Proceedings of the CHI '12 Extended Abstracts on Human Factors in Computing Systems, ACM, 1087-1090.

[16] Schneider O., Zhao S., and Israr A.: FeelCraft. 2015. User-Crafted Tactile Content. Haptic Interaction. vol. 277, H. Kajimoto, et al., Eds., Springer Japan, 253-259.

[17] Danieau F., Fleureau J., Guillotel P., Mollet N., Christie M., and Lecuyer A. 2014. Toward Haptic Cinematography: Enhancing Movie Experiences with Camera-Based Haptic Effects. MultiMedia, IEEE 21, 2, 11-21.

[18] Danieau F. 2014. Contribution to the Study of Haptic Feedback for Improving The Audiovisual Experience. PhD Dissertation, University of Rennes 1, France.

[19] Jayasiri A., Honda K., Akahane K., and Sato M. 2013. Feeling Wind: An Interactive Haptization System for Motion Rendering in Video Contents using SPIDAR. In Proceedings of the Artificial Reality and Telexistence (ICAT), 2013 23rd International Conference on, 54-60.

[20] Rasool, S., Hou, X., Liu, Y., Sourin, A., Sourina, O. 2016. Neurocognitive Tools for Assessing Haptic Interaction with Tangible Images, International Journal of Psychophysiology, Elsevier, v. 108, p.147.

[21] Rasool, S., Hou, X., Liu, Y., Sourin, A., Sourina, O. 2016. Assessing Haptic Video Interaction with Neurocognitive Tools, In Proceedings of the 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC2016), Budapest, 3057-3062.

[22] Rasool, S., Sourin, A. 2013. Image-driven virtual simulation of arthroscopy, The Visual Computer, Springer, 29(5): 333-355.

Documents

Tangible video communication over the Internet