Paper Review

#### Eavesdropping on Controller Acoustic Emanation for Keystroke Inference Attack in Virtual Reality

Shiqing Luo, Anh Hguyen, Hafsa Farooq, Kun Sun, Zhisheng Yan
<hr>
Presented By: Ryan Schanzenbacher

---

### Background Terms

- VR: Virtual Reality
- HMD: Head-Mounted Display
- DOA: Direction of Arrival
- Altitude: Angle from attacker's phone on vertical axis
- Azimuth: Angle from attacker's phone on horizontal axis

---

### Introduction

Virtual Reality headsets are becoming more ubiquitous as time goes on

----

### Introduction

Virtual Reality headsets are becoming more ubiquitous as time goes on

VR's extensive personal data collection poses new and unique **privacy challenges**

----

### Motivation

- Previous works focused on <strong>strengthening the privacy and security of the HMD</strong>
    - Focused on knowledge/biometric-based authentication controls
- No previous research on privacy-leaking data from <strong>side-channel signals</strong>
    - Various side channels can be used, but many have limitations.

Notes:
Motivation FOR THIS PAPER

Increasing security of HMD-- biometric/other authentication, usual protections for devices

----

### Side-Channels available

#### Visual Side Channel

- Record a user's hand motions and derive keys being pressed
- Con: Requires specific camera placement
- Con: Requires no obstruction or occlusions of victim
</div>
<div class="col" style="border-left: 2px solid #e7d6ff;">

#### Wireless Side Channel

- Analyze variations in wireless signals from user input
- Con: Requires precise placement of transmitter and receiver with user directly between them
- Con: User can protect against attack by going in location that cannot support this setup (corner)
</div>
</div>

Notes:
Camera side channel: Camera MUST be directly in front of user, works since vision is occluded

----

### Audio Side Channel

- Main idea focused on in this paper
    - Derive keystrokes from 3d audio capture
    - Attacker has flexible placement

Notes:
Attacker does not need to train/attack in certain position

---

### Heimdall

Placement-flexible audio based VR keystroke inference attack framework

----

### Challenges

- Previous work on similar side-channels against physical and virtual phone keyboards exist
    - Only need to focus on resolving positions on a 2D plane, not 3D space.
    - Commodity phones have <strong>omni-directional</strong> microphones.

Notes:
Omni-directional mics do not allow for directional inference

----

### Challenges

- Unlike previous works, the attacker's microphone may be placed at varying distances and orientations
    - This means pre-trained keystroke sounds cannot be used verbatim.
- The noise differences between different keystrokes in VR is very subtle
    - Decoding errors may occur if a user rotates the controller rather than translate it

----

#### Example Setup / Threat Model

![](csec759_paper_review/example01.png)

</div>
<div class="col">

![](csec759_paper_review/example02.png)

</div>
</div>
<div class="container" style="flex: 1;">
<div class="col">

<small>Abstract example showing setup</small>

</div>
<div class="col">

<small>Real world example set-up within library</small>

</div>
</div>

----

----

![](csec759_paper_review/example03.png)

</div>
<div class="col">

![](csec759_paper_review/example04.png)

</div>
</div>
<div class="container" style="flex: 1;">
<div class="col">

<small>Strokes for keys vary between users, but form clusters</small>

</div>
<div class="col">

<small>Sounds are distinct per key, but remain consistent between users</small>

</div>
</div>

Notes:
The researchers noticed something...

----

#### Architecture

![](csec759_paper_review/arch.png)

Three distinct modules that solve each of the challenges presented

---

### Challenge 1: Directional Acoustic Signal Acquisition

<hr>

Simple modifications made to commodity phones allow omni-directional microphones to become directional

----

#### The Concept

![](csec759_paper_review/tubes.png)

Porous tubes added to both microphones present on a phone

<small>Contains side slots to produce uniquely distorted signals from different DOAs</small>

Notes:
made of plastic

modelled to mimic human ear function

----

#### The Concept

- Authors made a requirement of no specialized modifications; anyone can manufacture these
- Demonstration units were 3D-printed
- Theoretical resolution of <i>7mm</i> between sound sources can be achieved
    - <strong>Assumes a sampling rate of 48kHz</strong>

Notes:

- 7mm is approx 2/3 the width of the head of my Yubikey

----

#### Improving keystroke recognition

Authors implement a background-noise reduction filter to remove noises such as bystanders.

The authors achieve this in two steps

----

#### Improving keystroke recognition

<hr>
<small>Maximal Overlap Discrete Wavelet Transform</small>

$\alpha\_i^{(J)} = \sum\_{n \in \mathbb{Z}} X\_n \bar{g}\_{n-2^j i}^{(J)} \quad \beta\_i^{(j)} = \sum\_{n \in \mathbb{Z}} X\_n \bar{h}\_{n-2^j i'}^{(j)}$

<hr>
...where $\alpha_i$ and $\beta_i$ are the detail coefficients, $J \in \mathbb{Z}$,$j \in \{1,2,..J\}$ are the levels and $\bar{g}$ and $\bar{h}$ are the discrete orthogonal wavelet functions

Notes:
BayesShrink thresholding applied after this initial tranform

----

#### Improving keystroke recognition

<hr>
<small>Reconstruction using inverse transform</small>

$X\_n = \sum\_{i \in \mathbb{Z}} \alpha\_i^{(J)} \bar{g}\_{n-2^J i}^{(J)} + \sum\_{j=1}^{(J)} \sum\_{i \in \mathbb{Z}} \beta\_i^{(j)} \bar{h}\_{n-2^j i'}^{(j)}$

----

#### The result

![](csec759_paper_review/audio_processing.png)

----

#### Other details to note

![](csec759_paper_review/threshold1.png)

</div>
<div class="col">

![](csec759_paper_review/threshold2.png)

</div>
</div>
<div class="container" style="flex: 1;">
<div class="col">

<small>Keystrokes can be identified by applying a magnitude threshold to the waveform. The authors found that 200ms segments were sufficient through a pilot study.</small>

</div>
<div class="col">

<small>Keystroke session can be identified and isolated by identifying sessions of regular, faster-paced clicking</small>

</div>
</div>

----

#### Effectiveness

- Average session in study: 3.5 minutes
- True Positive Rate: 0.98
- False Positive Rate: 0.04

Notes:
This is in terms of detecting keystrokes successfully

----

At this point, we have discrete keystrokes that we can attempt to solve the 3D position of.

---

### Challenge 2: Adaptive DOA-Key Mapping

<hr>

Various novel techniques allow keys pressed in VR to be derived

----

#### Solving the position problem

1. Collect Base-line DOA-key Mappings 
    - Keys are entered on the VR OS and DOAs are collected and translated to the attacker coordinate system 
1. Derive the DOAs 
1. Update DOA-Key mappings 
    - Optimize for potential rotations and translations of the phone, attempt to find DOA that matches closest after brute-force searching

Notes:
Step 1 is pre-training

Step 2 and 3 are done when attacking

----

#### Update DOA-Key mappings

<blockquote>Overall, any displacement of the DOA in the attack can be represented by a combination of rotation and translation</blockquote>

<hr>

$\pi = \underset{\delta\_x, \delta\_y, \delta\_z, \gamma}{\arg\min} \sum\_{s \in S} \Delta(d\_{k,atk}(\delta\_x, \delta\_y, \delta\_z, \gamma), d\_{s,atk})$

Notes:

pi in this case is the optimal displacement parameters for delta dk,atk(theta_x,theta_y,theta_z, gamma)

----

$\pi = \underset{\delta\_x, \delta\_y, \delta\_z, \gamma}{\arg\min} \sum\_{s \in S} \Delta(d\_{k,atk}(\delta\_x, \delta\_y, \delta\_z, \gamma), d\_{s,atk})$

<hr>

<blockquote>By searching a finite range of displacement parameters, with δx, δy , δz from -1 meter to 1 meter at a step size of 0.05 meter, and γ from 0◦ to 359◦ at a step size of 1◦, we can solve [the above equation].</blockquote>

Notes:
Essentially a brute-force

----

![](csec759_paper_review/solve_doas.png)

Rotation of 45$\degree$ in azimuth

----

This results in us determining the DOA-Key for each keypress, allows possible mappings, and will give us a direction between each DOA-Key.

Notes:
9 total states

up,up-right,right,down-right,down,down-left,left,up-left,no-move

---

### Challenge 3: Inter-Key Relation-based Calibration

<hr>

Using a <i>Hidden Markov Model</i> (HMM), keys can be predicted with greater accuracy

----

#### Why do this?

Many of the errors encountered during key mapping was when the user drastically rotated their wrist with minimal translation

----

#### How?

The authors explore the <i>relationship between keystroke transitions</i>.

Namely, letters that are closer together take less time to press in succession.

----

![](csec759_paper_review/timing_differentials.png)

----

#### Hidden Markov Model

Used to help predict what possible combinations for the next letter may be based on distance and <i>keystroke transition</i>

Notes:
That direction we derived from before is the transition

----

$P_0(q) = \frac{1}{(26+10)\cdot(26+10)}$

Notes:
This is the chance of any one key being entered. a-z0-9

----

Keystroke transitions must originate from where previous one ended.

A->B to C->D = 0

A->B to B->C = $\frac{1}{26+10}$

----

Given a <i>direction</i> and <i>time interval</i>, the authors can predict possible mappings, even if the user's movement was minimal

---

### Study Results

- 30 university students participated
- 274,050 attacks trialed

Notes:
29 victims x 45 passwords x 7 placements x 30 rounds

----

![](csec759_paper_review/prec_recall.png) 
<small>Good performance on all 36 characters</small>

</div>
<div class="col">

![](csec759_paper_review/acc_length.png)
<small>Accuracy decreased for longer passwords</small>

</div>
</div>
<div class="container" style="flex: 1;">
<div class="col">

![](csec759_paper_review/acc_distance.png)
<small>Accuracy increased for larger inter-key distances</small>

</div>
<div class="col">

![](csec759_paper_review/acc_candidates.png)
<small>Accuracy increased with more guesses</small>

</div>
</div>

Notes:
Best length was ~6 chars, as it had the best balance between ease of guessing and info to guess

top-w: successful if one of the w candidates matched exactly

distance: long if inter-key distance is 5 or more, short if less, mixed if mixed

4-8 character password has 95% acc in 20 attempts

Acc converges ~98% @ 50 candidates

----

![](csec759_paper_review/acc_phone_distance.png)

</div>
<div class="col">

![](csec759_paper_review/acc_denoise.png)

</div>
</div>
<div class="container" style="flex: 1;">
<div class="col">

<small>Notable decrease in accuracy at larger distances, however key inferencing accuracy is still ~85%</small>

</div>
<div class="col">

<small>Accuracy with various noise was unchanged, proving effectiveness of the denoising algorithm</small>

</div>
</div>

Notes:
This refers to PHONE distances

---

### Countermeasures

- Vary typing patterns when inputting sensitive information
- Authors mention using mechanical keyboards
    - Vulnerable to shoulder-surfing attacks
    - Vulnerable to KeyTap-style vulnerabilities (<a target="_blank" rel="noopener noreferrer" href="https://keytap.ggerganov.com">keytap.ggerganov.com</a>)

----

#### Key Takeaways

- Introduces Heimdal
    - Directional acoustic acquisition framework
    - Adaptive DOA-Key Mapping scheme
    - Inter-key relation-based calibration model to correct mapping errors
    - Evaluation of implementation

----

#### Key Takeaways

- Achieved an overall inference accuracy of 96.51%
    - top-5 accuracy of 85.14-91.22%

---

# Questions?