Does It Par-?: Investigating the Relationship Between Mid-Air Haptics and Visual Representations of Surface Textures

Mid-air haptic feedback technology produces tactile sensations that are felt without the need for physical interactions. However, mid-air haptic experiences need to be congruent with visual cues to reflect user expectations. To overcome this, we investigate how to visually present properties of objects, so that what one feels is a more accurate prediction of what one sees. Specifically, this paper investigates the relationship between 8 visual parameters of a point-cloud representation of a surface (particle color, size, distribution, etc.) and 4 mid-air haptic spatial modulation frequencies (20, 40, 60 and 80 Hz). Our results and analysis reveal a statistical significance between low and high-frequency modulations and particle density, particle bumpiness (depth) and particle arrangement (randomness).

Abstract-Mid-air haptic feedback technology produces tactile sensations that are felt without the need for physical interactions.However, mid-air haptic experiences need to be congruent with visual cues to reflect user expectations.To overcome this, we investigate how to visually present properties of objects, so that what one feels is a more accurate prediction of what one sees.Specifically, this paper investigates the relationship between 8 visual parameters of a point-cloud representation of a surface (particle color, size, distribution, etc.) and 4 mid-air haptic spatial modulation frequencies (20, 40, 60 and 80 Hz).Our results and analysis reveal a statistical significance between low and high-frequency modulations and particle density, particle bumpiness (depth) and particle arrangement (randomness).
What is common in all previous research on the topic of midair haptic textures is that the haptic rendering method [10] was a function of the graphical texture (i.e., graphics to haptics).For example, Beattie et al. [5], [6] described an algorithm that uses the displacement map of a textured graphical image (e.g., a picture of a carpet or of a plank of wood) to dynamically adjust Fig. 1.Overview of the experimental GUI, with 8 visual parameters/sliders and the particle surface that users interact with.Also shown are the user's avatar hands captured by the hand tracking camera.
the properties of a mid-air haptic stimuli.In contrast, in this paper, we investigate the opposite (i.e., haptics to graphics).We aimed to bridge the gap, by visualising the experiential qualities of the pattern rather than its spatial properties.Our motivation is to better understand the perceptual space of the mid-air haptic stimuli and also to provide some design insights on how to best achieve visuo-haptic (i.e., cross-modal) congruence.
Existing mid-air haptic technology may not adequately reflect the user's expectation of haptic feedback that matches the visual properties of solid objects.Martinez et al. [11] proposed that there is a mismatch between seeing a solid-looking virtual object, and feeling mid-air haptic feedback (which lacks force feedback), resulting in a sub-optimum user experience.User expectation is key in providing a good user experience, and when it comes to experiences of everyday objects in virtual environments, people typically bring their real-world expectations with them.Therefore, the challenge for mid-air haptic technology is how to prepare and adjust the user's expectations a priori to the tactile experience.
To that end, in this paper we have designed and performed a user study whereby 21 participants adjusted 8 properties of a digital point-cloud surface (Color, Saturation, Sheen, Size, Shape, Density, Distribution, Surface) made up of hundreds of particles (i.e., a point-cloud) as shown in Figs. 1 and 2, in response to four different mid-air haptic stimuli.Namely, a spatio-temporally modulated (STM) focal point targeted onto the user's palm rapidly tracing out a small haptic circle of perimeter 20 cm with rotation frequency of 20, 40, 60, and 80 Hz.We chose these four stimuli since Ablart et al. [4] had previously identified an inverse relationship between STM frequency and 1939-1412 © 2023 IEEE.Personal use is permitted, but republication/redistribution requires IEEE permission.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.roughness (higher frequency feels less rough).The motivation to employ a "passive" approach for the evaluation of surface roughness was also inspired by the Ablart's study.Through several significance tests, a principal component analysis, a participant questionnaire feedback, and a thematic analysis, we demonstrate that Density, Distribution, Surface are the three most important visual parameters that can influence visuo-haptic congruence.

A. Visual Textures and Point-Clouds
Surface texture can be represented as a point-cloud, which can be rendered directly or transformed into mesh models for 3D graphics processing.They are commonly used in applications like visualization, animation, rendering, and mass customization [12].Martinez et al. proposed creating visual textures based on point-clouds to represent mid-air haptic textures, while Bhardwaj et al. proposed the use of mid-air haptics to explore cluster density and volume information of data clouds [11], [13].

B. Haptic Texture Perception
Texture perception is a combined sensory experience where visual and tactile senses work together to interpret three textural dimensions: roughness, hardness, and slipperiness [14].Tactile exploration, whether dynamic or static, is required for full haptic perception [15].Haptic virtual texture has been studied using various apparatus, including force feedback devices, pin-arrays, vibrotactile actuators, touchscreen haptic surfaces, and mid-air haptics.Researchers have investigated numerous characteristics to alter the perceived texture of tactile inputs, including frequency and waveform, which have the greatest impact on tactile texture perception [16].Pseudo-haptic sensory supplementation techniques have also been explored [17].

C. Mid-Air Haptic Textures
Freeman et al. [3] used a mapping technique for mid-air haptic texture creation, while Ablart et al. [4] studied roughness perception based on spatiotemporal frequency modulation.Beattie et al. [5] used ML algorithms for incorporating roughness perception, Matsubayashi et al. [7] focused on rendering softness, and Morisaki et al. [8] created a visual-haptic prototype demo.Freeman [9] used audio sounds to influence roughness perception in mid-air haptic sensations.

D. Visuo-Haptic Congruence and Perception
There is agreement that haptic feedback improves immersion and UX in virtual environments [18], [19].Sensory information congruency between visual and haptic senses determines a user's level of immersion [20].Congruent visuo-haptic inputs result in more accurate perception of stimuli wheras incongruent visuohaptic information deteriorates presence in VR [21], [22].

III. USER STUDY
The purpose of this experimental research was to investigate and define what parameters of the visual point-cloud surface are significantly related to mid-air haptic stimuli of varying roughness ratings.To that end, we describe below a within-subjects user study comprising of four haptic conditions followed by a post-evaluation questionnaire.

A. Participants
21 participants were recruited from University of Bath staff and students, Ultraleap employees, and members of their families, with an age range of 18 to 54 (mean 33, SD 8 years), 5 identified as women and 16 identified as men.All participants were right-handed and 18 described themselves as familiar with mid-air haptic technology.Participants came from diverse backgrounds with a good grasp of English language.

B. Haptic Stimuli
Four mid-air haptic stimuli were presented, each comprising of a focal point that traces a small haptic circle of perimeter 20 cm with rotation frequency of 20, 40, 60, and 80 Hz (motivated by Ablart et al. [4]).The stimuli were targeted at the centre of the user's palm using the Leap Motion hand tracking API.Therefore, the participants did not need to hold their hand completely still but could move it freely within the tracking range of the device.Participants were however advised to keep their hand relatively steady, centred, and approximately 20 cm above the haptic display for optimal performance.We note that the displayed pointcloud had a square tract (see Fig. 1) in order to resemble the Ultraleap device and hence confine the possible user motion above the device and near its optimal operating region.

C. Protocol
Each participant experienced the 4 haptic conditions, 3 times each, in a counterbalanced order using a Latin square.Eight dependent variables could be adjusted using a mouse pointer that would move the respective sliders smoothly from left to right on the screen.Each of these represented a different visual feedback and would dynamically alter the properties of the graphical point-cloud in an obvious and intuitive way.
1) Color: this would adjust the color of the particles by cycling through the hue spectrum.2) Saturation: this would adjust the brightness of the particles.3) Sheen: this would adjust the glossiness of the particles.4) Size: this would adjust the size of each particle.5) Shape: this would morph the particles from sphere, to cube, to cylinder, to a pyramid to triangle.6) Density: this would change the number of particles displayed per unit area.7) Distribution: this would adjust the strength of random perturbations applied to a uniform grid of points.A stronger perturbation would lead to a random distribution of points.8) Surface: This would apply random perturbations in the z direction thus affecting the visual bumpiness of the pointcloud surface.A ninth slider was included for participants to rank their confidence in the visuo-haptic congruence achieved by their selections.In total we collected 4×3×9×21 data points.

D. Procedure
Due to Covid-19 restrictions, the study was conducted independently by participants on laptops and an Ultraleap Stratos Explorer Device during a Zoom video call with the researcher.Participants signed an information sheet and consent form, and an experimental Unity3D-developed GUI was shared as an executable for them to play the particle demo shown in Fig. 1.The particle point clouds were rendered using standard parameters available for point cloud generation, using existing Unity 3D VFX packages.Participants were briefed about the purpose of the study and allowed one trial session where they were familiarised with the Ultraleap device, the experimental GUI interface, and how the different sliders could be adjusted to change the particle visual representations.During the experiment, participants were instructed to use ear defenders to ensure minimum distraction from the outside environment, and were asked to adjust the visual representation of the particles to best match the mid-air haptic stimulus, one at a time, using all the sliders, and save the parameters before feeling the next stimulus.To reduce bias, after each sensation, the sliders were reset to 0 and the haptic sensation stopped playing for 3 seconds to allow for the hand to rest.In addition, participants were asked to use a confidence slider to show how well they thought they were able to recreate the sensation using the sliders.
After the haptic experiment, participants were debriefed and asked to fill out a post-evaluation online questionnaire, which consisted of 26 questions regarding their overall experience, interaction with the particles GUI and a 7-point Likert scale for each dependent variable (ranging from not useful (1), to very useful ( 7)) followed by a short explanation of why the participant found that control parameter useful or not useful when trying to visually recreate the mid-air haptic stimulus.The Zoom calls lasted approximately 50 minutes, while the experimental part of it took about 30 minutes.All collected data was anonymized, stored securely, and prepared for post-processing and statistical analysis.

A. Quantitative Analysis
Fig. 3 shows box plots of the nine sliders scaled onto the unit interval for the four haptic stimuli conditions (20, 40, 60, 80 Hz), demonstrating the locality, spread and skewness of the data.To aid visual comparison, Fig. 4 plots the median of the nine sliders for the four haptic stimuli conditions.We plot the median since the mean can be biased by outlier data rather than the typical values.Also plotted in Fig. 4 is the standard deviation σ of the medians for each of the nine input sliders across the four conditions.We observe that participants rated their answers to the 20 and 80 Hz conditions slightly higher median confidence levels.We also observe that the standard deviations σ of the medians of the Surface, Density and Distribution are much larger that any of the other five dependent variables (Color, Saturation, Sheen, etc.) as can be visually seen by the spread of their medians.We proceed with further statistical tests.
Each frequency data set was divided into a data set of its own, with every parameter measured against its counterparts from the other groups.Due to data violating normality following a Shapiro Wilk test, frequency group differences were measured using other non-parametric tests.Both Friedman and Wilcoxon showed significant differences with χ(4) = 43.788,and p < 0.001 for Surface and Distribution, and χ(4) = 26.565,and p < 0.001 for Density.A post hoc analysis with Wilcoxon signed-rank tests was conducted with a Bonferroni correction applied, resulting in a significance level set at p < 0.0083 for the following samples.Table I presents the significant differences between test scores between various pairs of frequency conditions calculated using the Wilcoxon Signed-Rank Test for dependent parameters Density, Surface, and Distribution.We did not find statistical significance for Size (χ(4) = 1.59, p = 0.6), Sheen (χ(4) = 0.38, p = 0.9), Shape (χ(4) = 4.0, p = 0.2), Color (χ(4) = 2.9, p = 0.4, or Saturation (χ(4) = 1.12, p = 0.7).
Fig. 5 shows the resulting user-generated point-cloud surface visualisations that best match the four haptic stimulus conditions.Indeed, we can visually observe that the lower frequency stimulus appears more sparse, random, and bumpy compared to the higher frequency ones.

B. Principal Component Analysis
In this subsection, we perform further data analysis on the participant data in the form of principal component analysis (PCA).PCA is often used in exploratory data analysis and for dimensionality reduction by projecting each data point onto only the first few principal components to obtain a lower-dimensional dataset while preserving as much of the data's variation as possible.We will show that 99.9% of our original 8-dimensional data variability can be well described by just 3 dimensions (principal components) that are a linear combination of the original 8 visual attributes (Color, Saturation, Sheen, Size, Shape, Density, Distribution, Surface).
We start by standardizing the dataset by rescaling the userratings onto the unit interval and taking the median values (as shown in Fig. 4).We can represent this as a 4 × 8 matrix X, where each row corresponds to the 4 haptic conditions, and each column corresponds to the 8 dependent variables.We then calculate the 8 × 8 covariance matrix C = cov(X), and obtain its respective eigenvalues {u 1 , u 2 , . . ., u 8 } and eigenvectors {v 1 , v 2 , . . ., v 8 } which we identify as the principal components (PCs) of our data.Recall that eigenvalues satisfy u 1 ≥ u 2 ≥ . . .≥ u 8 ≥ 0, and that eigenvectors have unit length |v i | 2 = 1, by definition.The proportion of variance explained (PVE) e i by eigenvector v i can be used to identify the optimal number of PCs to keep based on the total variability that we would like to account for in our data, and is given by e i = u i / j u j .We find that e 1 = 91.3%, e 2 = 5.4% and e 3 = 3.2%, while e 4 ≈ 0. It can therefore be noted that the first three eigenvalues account for 99.9% of the variability of the participant data, implying that the 8-dimensional dependent variable space can be reduced and adequately described by a 3 dimensional one comprising of the three first PCs.We thus set v 1 , v 2 , and v 3 as the three PCs which we use to visualize and interpret our haptics-to-graphics user-generated data set PC1 : v 1 = (0.1, 0, 0, 0.1, 0, 0.4, −0.4,−0.8) (1) PC2 : v 2 = (0.3, 0, 0.4, −0.4,0.6, 0.5, 0, 0.3) PC3 : v 3 = (−0.1,−0.3, 0.5, 0.4, 0.4, −0.5, −0.3, 0) (3) Fig. 6 shows how the eight different features contribute to PC1, PC2 and PC3.We observe that most of the features have very low weights along PC1, with the exception of Density, Distribution, and Surface (similarly to the significance results of the previous subsection).
Projecting the 4 haptic stimuli onto the dimensionally reduced principal component space by using a dot product X • v i allows us to better visualize their differences, as shown in Fig. 7. Distances between points represent how different those stimuli are in terms of the PCs.We observe that PC1 manages to sort the stimuli conditions in terms of their frequency.This is quite remarkable, since we know from Ablart et al. [4] that stimulus frequency is inversely proportional to perceived texture roughness, meaning that we can identify PC1 as a smoothness scale.Thus, a higher frequency stimulus is perceived as smoother and is visuo-haptic congruence is positively correlated with pointcloud Density (more particles), but negatively with point-cloud Distribution (more uniform particles) and point-cloud Surface (more flat), and vice versa.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.Also, from Fig. 7, we can see that PC1 does not adequately separate the 40 and 60 Hz conditions.For that, one needs to look at PC2 which suggests that these two conditions differ mostly in terms of Shape, Density, Size and Sheen, however we cannot provide any further intuition on the meanings of PC2 and PC3.
Using v 1 , v 2 , and v 3 , one can try to guess the solution of the inverse problem: given a haptic stimulus frequency, what should the point-cloud Density, Distribution, and Surface sliders be?To see this, we first construct the slider vector s = (0.2, 0.25, 0.24, 0.48, 0.23, s 6 , s 7 , s 8 ), where the first 5 entries comprise of the median participant ratings for point-cloud Color, Saturation, Sheen, Size, and Shape, respectively, while s 6 , s 7 , and s 8 correspond to the unknown slider values for Density, Distribution, and Surface, respectively, that we want to find.Next, we select a coordinate point f = (f 1 , f 2 , f 3 ) in PC1-PC3 space from the 3D plot in Fig. 7 that we want to find its likely slider values.To do so, we solve the equation (v 1 , v 2 , v 3 ) • s = f to obtain that s 6 = 0.13 + 0.4f 1 + 1.14f 2 − 0.71f 3 , s 7 = 0.86 − 0.56f 1 − 1.73f 2 − 2.2f 3 and s 8 − 0.35 − 0.75f 1 + 1.56f 2 + 0.86f 3 .For example, we could try and guess that a 30 Hz haptic condition might have coordinates near f = (−0.3,0.25, 0.15), since it should be approximately at the midpoint between the 20 and 40 Hz conditions in Fig. 7.We can then calculate that {s 6 , s 7 , s 8 } = {0.18,0.30, 0.38}.While we cannot verify this prediction, our approach is an educated best guess made possible by PCA.

C. Usefulness Ratings
The post-evaluation online questionnaire was completed by participants after the Zoom call at their own time.First, participants were asked to score from 1 to 7 the usefulness of the 8 sliders they used during the experiment when trying to match point-cloud properties to the haptic stimuli.In descending order of reported usefulness, their responses were: Density (6.1), Size (5.8), Surface (5.5), Distribution (5), Shape (3.8), Sheen (2.5), Saturation (2.2), and Color (1.5).This ordering is mostly in line with the results reported in the previous section, with the exception of Size, where Density, Surface, and Distribution were identified as significant differentiators of the four haptic conditions presented.Fig. 8 shows the participant responses with regards to slider usefulness.
We provide some of the participant explanations or reasoning of the above ranking.Interestingly, we see some answers referring to multiple features thereby implying that haptic sensations cannot be described by a single visual parameter.Density -P16: "Particle amount was the most intuitive parameter.It was easier to tell if there were lots or few compared to some of the other parameters."Size -P1: "The sensation sometimes felt like big pulses on the hand, or the sensation felt smooth with no noticeable pulses.Setting the particle size to small meant I could get a smooth flat surface, or a slightly bumpy surface.A big particle size meant I could recreate the big pulsing sensations."Surface -P18: "I felt like for sensations that had larger bumps or particles, it is useful, as these are more pronounced and I could feel some level of elevation difference.For sensations that felt like small particles, I found it hard to tell if it was a difference in shape or density I was feeling."Distribution -P8: "When it was random, I could really feel the "no sensation" gaps when I ran hand over places of no particles."Shape -P4: "It helped me think about the haptics when I was feeling them, but I couldn't differentiate between the shapes."Sheen -P15: "It was useful just for the sake of the imagination, For example if the stimulus was bubbly, it was easier for me to perceive a colder material which pointed me to choose a glossy effect (metallic effect) vs matte."Saturation -P7: "I couldn't associate it with a sensation feature."Color -P16: "It was difficult to associate any different colours -I think the temperature was more influencing my choice of colour."

D. Thematic Analysis
We looked at the frequency counts of different words and adjectives used by participants in their questionnaire responses to describe the haptic stimuli.The most used adjectives for a description of mid-air haptic sensations were either "bumpy, bubbly, round" (10) or "rough" (5) (for low frequencies conditions) "smooth, fuzzy" (7) (for high frequencies) or other synonyms of those.Other commonly used adjectives were "enjoyable" (3), "playful" (2), "soft" (3)."spiky, scratchy" (2).

E. General Participant Comments
Overall, 78% of the participants thought that particles was a good way to represent texture, 57% thought that particle was a good way to represent bumpy or rough surfaces, but only 37% that it was good to represent smooth or flat surfaces, while 32% did not think it was a good way to represent any surfaces at all.47% of the participants said that they found it difficult to recreate the visual for at least some stimuli, and 16% said that it would have been easier if fewer slider options were available.When asked about using particles to represent objects in virtual reality (VR), 42% of the participants thought it would be a good idea (but depending on the context e.g., volumetric items, curtains, rucksacks) while 16% participants did not feel it was a good way to represent solid objects in VR, and the rest of the participants were unsure.
V. DISCUSSION 1) Limitations: COVID-19 restrictions resulted in an uncontrolled study as participants took the study online, potentially introducing bias.The results are limited in their generalizability to 3D graphics, as most are polygonal meshes rather than point-clouds.Similarly, the findings cannot be transferred to other haptic stimuli such as AM, DTP, LM, and other STM shapes.The study's results may have also been impacted by participants' varying levels of experience with mid-air haptic technology.[10].The study's sample size of 21 test persons may limit the generalizability of the findings.Additionally, the complexity of the interface as well as personal preferences, which includes eight variables, could pose challenges for users and impact the overall convergence to a consensus.
2) Importance: Existing research has demonstrated the importance of visuo-haptic congruence in applications and use cases.The results of our study align with this notion, as participants agreed that particles allows for a better representation of mid-air haptic sensations and for creating congruent mid-air textures/surfaces visualized through particles.In addition, our research also calls for more multisensory research to better inform on how graphical properties should be adjusted to better match mid-air haptics.These findings hold significant implications for how visuals should be adjusted in order to create a more efficient and immersive user experience.
3) Future Work: Visuo-haptic congruence is a key factorfor successful mid-air haptic interfaces, such as widget and buttons.Established design guidelines are needed to create visually appealing and functionally effective interfaces.It is becoming clear that adherence to these guidelines is necessary to achieve optimal outcomes.As future work, stimuli could be designed to move in a circulating trajectory for participants to perceive the entire texture.Further research can build on the "haptics to graphics" approach by incorporating dynamic haptic and visual stimuli.Additioanlly, conducting a study with fewer parameters may increase confidence in the findings.Looking at the results, intermediate values (40 and 60) may have been harder for participants to differentiate than the extremes.This has implications for future studies, which should consider optimizing the number and distribution of stimuli to minimize confusion.

VI. CONCLUSION
We conducted a user study exploring the relationship between 8 visual point-cloud parameters and 4 mid-air haptic stimuli of varying modulation frequencies.Significant relationships were found between 3 of the visual parameters, which were corroborated by principal component analysis.A post-evaluation questionnaire provided insights into which visual properties of a point-cloud were considered useful in achieving visuo-haptic congruence.Our findings align with previous research on haptic textures [23].Future studies can further investigate the use of particles for visualizing other mid-air haptic sensations and improving audio-haptic congruence.

Does
It Par-Tickle?: Investigating the Relationship Between Mid-Air Haptics and Visual Representations of Surface Textures Katarzyna Wojna , Orestis Georgiou , David Beattie , Michael Wright , and Christof Lutteroth

Fig. 2 .
Fig. 2. Examples of potential point-cloud surfaces made up by using different visual attributes during the study.

Fig. 3 .
Fig. 3. Box plots of the 9 sliders for the four different frequency conditions.Each slider output was re-scaled onto the unit interval for the sake of comparison.

Fig. 4 .
Fig. 4. Reported median values of each variable for the four conditions.Also plotted is the 1 minus the standard deviation (1 − σ) of the reported medians of each condition.

Fig. 5 .
Fig. 5. Example particle surfaces using the medians of the three significant parameters (Density, Distribution, Surface) for the four haptic conditions 20 Hz, 40 Hz, 60 Hz, and 80 Hz (from top left to bottom right).

Fig. 6 .
Fig. 6.Feature contribution for principal components one, two, and three.The box ratio is scaled according to the PVE of each PC.

Fig. 7 .
Fig. 7. Re-projection of the four haptic conditions (20, 40, 60 and 80 Hz) onto the principal component 3D space.The box ratio is scaled according to the PVE of each PC.

Fig. 8 .
Fig. 8. Box plots of the usefulness ratings associated to the 8 graphical sliders.

TABLE I SIGNIFICANT
DIFFERENCES FOR TEST SCORES BETWEEN PAIRS OF FREQUENCY, CALCULATED USING THE WILCOXON SIGNED-RANK TEST