Eye-hand coordination during dynamic visuomotor rotations

Abstract

Background: for many technology-driven visuomotor tasks such as tele-surgery, human operators face situations in which the frames of reference for vision and action are misaligned and need to be compensated in order to perform the tasks with the necessary precision. The cognitive mechanisms for the selection of appropriate frames of reference are still not fully understood. This study investigated the effect of changing visual and kinesthetic frames of reference during wrist pointing, simulating activities typical for tele-operations.

Methods: using a robotic manipulandum, subjects had to perform center-out pointing movements to visual targets presented on a computer screen, by coordinating wrist flexion/extension with abduction/adduction. We compared movements in which the frames of reference were aligned (unperturbed condition) with movements performed under different combinations of visual/kinesthetic dynamic perturbations. The visual frame of reference was centered to the computer screen, while the kinesthetic frame was centered around the wrist joint. Both frames changed their orientation dynamically (angular velocity = 36 degrees /s) with respect to the head-centered frame of reference (the eyes). Perturbations were either unimodal (visual or kinesthetic), or bimodal (visual+kinesthetic). As expected, pointing performance was best in the unperturbed condition. The spatial pointing error dramatically worsened during both unimodal and most bimodal conditions. However, in the bimodal condition, in which both disturbances were in phase, adaptation was very fast and kinematic performance indicators approached the values of the unperturbed condition.

Conclusions: this result suggests that subjects learned to exploit an "affordance" made available by the invariant phase relation between the visual and kinesthetic frames. It seems that after detecting such invariance, subjects used the kinesthetic input as an informative signal rather than a disturbance, in order to compensate the visual rotation without going through the lengthy process of building an internal adaptation model. Practical implications are discussed as regards the design of advanced, high-performance man-machine interfaces.

Conflict of interest statement

Competing Interests: The authors have declared that no competing interests exist.

Figures

Figure 1. Apparatus and experimental procedures.

(A): Wrist robot (WR) and (B): Visual Environment (VE) task is to perform center-out pointing movements, using the F/E and Ab/Ad DOFs, to each one of the four different targets: a central target, which corresponds to the neutral wrist position () and four peripheral targets equally spaced on the upper semi-circle. (C):the experimental protocol the x and y coordinates of VR (FRVIS) always correspond to movements of the F/E (flexion/extension) and Ab/Ad (abduction/adduction) wrist DOFs (FRKIN) respectively. (D): unimodal and bimodal disturbances: F condition; K condition; V condition; VK+ or VK− or VKP condition. The red circle identifies the target and the yellow circle the wrist end effector. The orientation of the visual scene is identified by the stripe pattern. In the case of kinesthetic disturbance K the P/S DOF was driven by a position servo with no effect on the VR. In the case of the visual disturbance V, the visual scene was rotated with respect to the computer screen. Bimodal conditions are a combination of both visual and kinesthetic disturbances. The VK- and VKP conditions are similar bimodal perturbations as VK+ but with a variable misorientation of the visual frame FRVIS and the wrist frame FRKIN.

Figure 2. Pointing trajectories.

For one of the subjects, the figure shows pointing trajectories in the 6 experimental conditions (F, V, K, VK+, VK−, VKP). Black trajectories correspond to center-out movements. Grey trajectories, which are displayed in a mirror way for graphical clarity sake correspond to return movements. Abscissas: F/E rotations or movements along x-axes; Ordinates: Ab/Ad rotations or movement along y-axes. The scale bars correspond to 2.5 cm on the computer screen or 0.1 rad in terms of wrist rotation.

Figure 3. Results of kinematic analysis.

a: lateral deviation of the pointing movements as a function of the different movement sets or experimental conditions. b: mean speed of the pointing movements (deg/s) as a function of the different movement sets or experimental conditions. c: Jerk index (rad/s3) of the pointing movements as a function of the different movement sets or experimental conditions. d: duration time (s) of the pointing movements as a function of the different movement sets or experimental conditions. *p<0.05 indicates a significant difference.

Figure 4. Aiming error adaptation.

Aiming error (deg) of the pointing movements, at 300 ms after movement onset or experimental conditions, as a function of the different movement sets. The aiming error was evaluated at the first 15 trials and last 15 trials to see if an adaptation occurs during the different target sets.

Figure 5. Aiming error as function of instantaneous visuo-kinesthetic rotational misalignment.

Scatter diagram, for the whole population of subjects, of the aiming error 300 ms after movement onset as a function of the rotational misalignment between the visual disturbance (θvis) and the kinaesthetic disturbance (θkin). The misalignment is null by definition in the F and VK+ conditions; it is randomly distributed across the whole range of possible values in all the other conditions. The slopes of the regression lines have the following values, for the K, V, VK− and VKP conditions, respectively: 0.749, 0.799, 0,697, 0.679; these values all differ significantly from 1.

Figure 6. Instantaneous mutual orientation of kinaesthetic and visual frames and trajectory generation.

(a): screen visualisation of the intended movement towards the target (blue circle) using the cursor (red circle); (b): wrist movement in order to reach the target in different experimental conditions. The curved path wrist movement are mapped on the screen rotated according the visuomotor transformation. The inability of the subject to move the wrist along the desired direction dkin, which would correspond to the straight path to the target dvis, is caused by the lack of capacity in mentally rotating the kinaesthetic-wrist-centered FRKIN to macth the visual-virtual-reality FRVIS map. (c): orientation of the visual-environment frame FRVIS, the kinaesthetic-wrist-centered frame FRKIN and the head-centerd frame FRH during a pointing movement in different experimental conditions. The aiming error is due to the instantaneous angular mismatch between the two frames of reference.