In this project, we propose the use of policy gradient methods to perform spoofing of devices in wireless networks by impersonating their wireless transmission fingerprints. Physical layer authentication relies on detecting unique imperfections in signals transmitted by radio devices in order to obtain their fingerprint and identify them. These imperfections are present within analog components of a radio device and can differentiate radio devices even if their manufacturer and make/model are identical. Radio fingerprints are usually considered hard to reproduce or replay because the replicating or replaying device suffers from its own impairments which disturb the features in the RF fingerprint.

We consider a wireless environment in which there are $|T|$ transmitters $T$ which are authorized to transmit to a single receiver $R$. $R$ is equipped with a pre-trained neural network-based authenticator $D_R$ that uses raw IQ samples of the received signals to perform a binary authentication decision at the physical layer, denoting whether the signal under consideration is from an authorized transmitter or not. There is an adversarial transmitter $T_A$ that wants to communicate with $R$, and it tries to do this by impersonating one of the $N$ authorized transmitters.

In a wireless communication system, there are three main sources of non-linearities that are imparted on the intended transmitted signal: if $x(t)$ is the signal at the beginning of the transmitter chain, the signal at the end of the receiver chain will be of the form $y(t)=f_R(f_C(f_T(x(t))))$, where $f_R$, $f_C$ and $f_T$ are fingerprints introduced by the receiver hardware, channel and transmitter hardware respectively. Physical-layer based wireless authentication systems in the literature are mostly designed to differentiate transmitters based on $f_T$ (for example, in [1]), so we will try to emulate a similar setting in our approach as well. We assume that we can place an adversarial receiver $R_A$ close-enough to $R$ such that the channel from $T_A$ to $R_A$ is similar to the channel from $T$ to $R$. So $R_A$ receives signals with a similar $f_C$ to signals that $R$ receives. Furthermore, as a simplification, in this project we will assume that we can find an $R_A$ device with a similar $f_R$ to $R$, which is not unreasonable since high quality wireless receivers of the same make/model will have a smaller variance of $f_R$ [2]. This means that effectively, $D_R$ and $D$ will have learned to discriminate based on the same $f_T$.

Now $R_A$ builds a discriminator $D$ that tries to distinguish between the signals it receives from $T$ and the signals it receives from $T_A$. Note that $R_A$ will have a ground truth since $T_A$ includes a flag in its transmitted signals. For each received signal from $T_A$, $R_A$ transmits its classification decision back to $T_A$ as feedback. $T_A$ tries to build a generator $G$ whose purpose is to distort the complex IQ samples of input discrete time signal $z(n)$ at $T_A$ such that after it is transmitted, it will be classified as authenticated at $D$. At convergence, $T_A$ should be able to generate signals that are good enough to fool $D_R$ (at $R$) into believing that they are from $T$.

The idea is to model $D$ and $G$ as neural networks. $D$ takes $N_S$ most recent samples of the sampled signal $y(n)$ and outputs a scalar value through a sigmoid that can be thresholded to get a binary value, representing whether the signal is authorized or not. We also plan to attempt using $|T|+1$ classes, with $|T|$ corresponding to $|T|$ authorized transmitters and a single class corresponding to non-authorized transmitters. Since $D$ is aware of the ground-truth, it is straightforward to train $D$ through gradient descent in a classical supervised learning fashion. However, this supervised method is not available for $G$ as there is no ground-truth. So we propose to train it using policy gradient methods.

We can model the neural network $G$ as the policy $\pi$ of a Markov Decision Process.

The action $a_n\in {ℝ}^2$ is the output of the neural network $\pi (s_n)$, where $s_n$ is the state at time $n$. The generator $G$ can then be viewed as the source of the signal $a(n)=a_n[0]+j{a}_n[1]$. The signal $a(n)$ is transmitted to the receiver to obtain $y(n)=f_R(f_C(f_T(a(n))))$. The reward $r_n$ at time $n$ will be the feedback from $D$ based on $\{y(n),y(n-1),\mathrm{\dots },y(n-N_s)\}$. Depending on the main type of distortion that the impersonator tries to mimic, different definitions of the state $s_n$ can be used.

1. 1.

   The state is a vector $s_n=[\text{Re}\{z(n)\},\text{Im}\{z(n)\}]$ containing the real and imaginary part of the most recent sample of the signal $z(n)$. This is applicable when the distortion over each sample is independent of the other samples. For example, the distortion imparted by the power amplifier in the RF chain will have this property [1].

2. 2.

   The state is a sequence of length $N_P$, $s_n=\{[\text{Re}\{z(n)\},\text{Im}\{z(n)\}],a_{n-1},\mathrm{\dots },a_{n-N_P-1}\}$. $N_P$ is a hyperparamter. This state is applicable when the distortion of the most recent sample of the signal is dependent on previously transmitted samples and the current sample.

3. 3.

   The state $s_n$ is $\{[\text{Re}\{z(n)\},\text{Im}\{z(n)\}],H_{n-1}\}$, where is $H_{n-1}$ is the hidden state of $G(s_{n-1})$, when it is modeled as a recurrent neural network. This state can in theory apply to any type of non-linearity.

Assume we have collected a trajectory $\tau$ defined as a sequence of states, actions, and rewards, $\{s_0,a_0,r_0,s_1,a_1,r_1,\mathrm{\dots },s_T\}$. Now, the goal is to tune the parameters $\theta$ (weights and biases) of $G$:

To solve this problem, we can use a policy gradient method: we repeatedly estimate the gradient of the policy’s performance with respect to its parameters and use that to update its parameters. To estimate the gradients, we will use a score function gradient estimator. With the introduction of a baseline $b(s)$ to reduce variance, an estimate $\widehat{g}$ for ${\nabla }_{\theta }{𝔼}_{\tau }[r_n|\theta ]$ is [3]

Now, the policy update can be performed with stochastic gradient ascent $\theta \leftarrow \theta +ϵ\widehat{g}$. This process can be repeated for a number of iterations in an algorithm termed Vanilla Policy Gradient, with a trajectory collected for each iteration, until $G$ converges to a satisfactory state. This algorithm also allows for $b(s)$ to be trained along with $\theta$ [3].

In our case, $N_s$ symbols of the signal have to be processed (perturbed) by the generator before transmitting, and hence a reward $r_n$ is not immediately available for every state $s_n$. To estimate $r_n$, we propose to perform a Monte-Carlo search from $s_n$ until the final state, using a roll-out policy $G_{\beta }$, which may or may not be equal to $G$ [4].