LHMap-loc: Cross-Modal Monocular Localization Using LiDAR Point Cloud Heat Map

III-B Offline lhMap Generation Network

As shown in Fig. 2,we is use use the offline lhMap generation network to compress the pre – build lidar point cloud map . It is realize through two stage .

In the first stage ,we is realize realize the point selection to compress the dense map andpose supervision to refine the generate local map .
to satisfy the requirement for point selection andmap compression ,the project lidar depth$d_{gt}$ is used to construct an evaluation system for point cloud . It is calculate as :

here ,$(x,y,z)\in p$,$k$ represents the camera intrinsics,and $t_{gt}\in sE\left(3\right)$ represents the ground truth camera pose at each frame.
Additionally,the offline RGB image $I_{offline}$ andprojected lidAR depth $d_{init}$ are used to perform pose supervision.
Based on the initial rough camera pose $t_{init}\in sE\left(3\right)$ at each frame,which can be acquired by Gps or visual odometry,$d_{init}$ is calculate as :

here ,${(x,y,z)}^t\in p$.
Both $d_{gt}$ and$d_{init}$ contain only the depth information of point clouds.

Firstly,feature maps $F_I,F_{d1}$,and $F_{d2}$ with different scales are extracted from $I_{offline}$,$d_{gt}$,and $d_{init}$ respectively,through convolutional neural networks (CNN). $F_{d1}$,the CNN feature of the projected lidAR depth $d_{gt}$ is used to generate the heat feature $h_c$.
the point clouds are selected by evaluating heat value. heat value is calculated by heat feature $h_c$ which is generated by $F_{d1}$. Each element$h_k^{i,j}\in h_c(i\in \{1,2,\dots ,h\},j\in \{1,2,\dots ,w\}$) is used to calculate heat value$h^{i,j}$ for point clouds evaluation as:

here ,$C$ represents the number of channels of $h_c$.
subsequently,points exhibiting the highest heat values are selected to constitute the coarse local lhMap,denoted as
$M_c$.

during the generation of $M_c$,the pose supervision is adopted to guide the procedure. the pose supervision module incorporates two inputs: the heat feature $h_c$,and the optical flow embedding $E_d$,which is derived from $F_{d2}$ and$F_I$ based on the iterative optimization structure of pWCNet[ 29 ]. pose supervision is realized by pose calculation module,detailed in sec. III – C.

the single stage 1 learning fails to converge. therefore,we propose the second stage to refine lhMap.
In the second stage,we apply $\Delta t=t_{init}^{-1}\cdot t_{gt}$ to the coarse local lhMap $M_c$ to recover the initial localization results.
the initial coarse local lhMap $M_c^{init}$ andthe offline RGB image $I_{offline}$ are used for further pose supervision.
Because both stages share the same offline RGB image,they share the same feature maps $F_I$ of the RGB image naturally,while the feature maps $F_M$ of the initial coarse local lhMap $M_c^{init}$ are regenerated. then,the heat feature $h_M$ is generated by $F_M$ andthe flow embedding $E_M$ is generated by $F_M$ and$F_I$. At last,they work together for pose supervision.
pose supervision is realized by the pose calculation module which is introduced in sec. III – C.
In this stage,we regress another set of 6-doF pose $q_1,t_1$. Both $q_0,t_0$ and$q_1,t_1$ refine the local lhMap by optimising the heat feature $h_c$.

the output of this network is the lidAR point cloud heat Map (lhMap) combined by the refined local lhMap at each frame. though the local lhMap contains only the depth information,by taking the inverse of the projection formulation,we can obtain the 3d coordinates information $p_k$ at each frame $k$. With the knowledge of the ground truth camera pose${}_{}{}^{w}t_k^{}$ at frame $k$ andthe point$p_k$ of frame $k$,we can convert $p_k$ to the world frame:

here ,${}_{}{}^{w}p_k^{}$ represents the points at the frame $k$ in the world coordinate system. the lhMap is constructed by uniting all the points ${}_{}{}^{w}p_k^{}$ together through an union operation ${\cup }_k$:

the loss function of the offline heat map generation network is similar to CMRNet[ 16 ].
let $q_{gt}$ and$t_{gt}$ represent the ground truth camera pose. the angular distance $l_q$ between quaternions is used to evaluate the rotation loss. the l1-smooth loss $l_t$ is used to evaluate the translation loss,which is defined as:

here ,$\{a,b,c,d\}$ are the components of quaternion $q$ and$⨂$ is the multiplicative operation between two quaternions.
the pose loss is defined as:

the pose $t_0,q_0$ regressed by the pose supervision module in stage 1 andthe pose $t_1,q_1$ regressed by the pose supervision module in stage 2 are both taken into account for better supervision. therefore,the total loss is defined as:

III – C Online pose Regression Network

this network is used for real-time monocular localization. the inputs are the online RGB image $I_{online}$ andthe real-time lhMap $M^r$. the $M^r$ is constructed by projecting ${}_{}{}^{t}p$ at each local lhMap stored in the first network to the image plane according to the function in (4).

Firstly,feature maps $F_I^r$ and$F_M^r$ are extracted from both inputs $I_{online}$ andreal-time local lhMap $M^r$ through convolutional neural network ( CNN ) .

then,the feature maps $F_M^r$ are used to calculate the 2d flow embedding $E_M^r$ along with the RGB image feature maps $F_I^r$ andto generate the heat feature $h_M^r$ alone. $E_M^r$ here is calculated the same as pWCNet[ 29 ]. the usage of the heat feature $h_M^r$ enables the pose regression to focus on effective features. therefore,the supervision of the 2d flow embedding $E_M^r$ andthe regression of 6-doF pose can achieve better performance. the cost volume $v$ is then calculate by feed$h_M^r$ into the softmax layer to generate the coefficients andmultiplying the coefficients with $E_M^r$. the cost volume $v$ is calculate as :

where$\odot$ means element-wise product,$softma{x}_{h\times w}$ means apply softmax to height andwidth dimensions of $h_M^r$.

At last,the cost volume is fed into separate Mlps for pose regression:

the pose regression is realized by the pose calculation module as shown in Fig. 3. the resolution of the flow feature may be different from that of the heat feature. therefore,the flow feature is transferred to the up-sampled layers to maintain the same resolution as the heat feature before being multiplied with it. the multiplication result then accumulates all the elements across the height andwidth dimensions before being fed into fully connected layers,which are denoted as $Ml{p}_q$ and$Ml{p}_t$. the outputs of this network are 6-doF poses $t_o,q_o$.

the loss function used here follows [ 30 ]. Adding two trainable parameters $w_x$ and$w_q$,the loss function is defined as:

Figure 3: the details of the regression part. Multiply flow embedding $E$ andup-sampled heat feature $h$ as inputs,and then calculate weighted features. the result is fed into $Ml{p}_q$ and$Ml{p}_t$ to regress 6-doF poses