Overview

D-SLAM is a Visual SLAM system using monocular RGB inputs and estimated depth inputs to construct a 3D mapping. The goal of this project is to create a well-performing SLAM system under contrained resources setup while aiming for real-time performance. The D-SLAM system can supports stream of RGB images either from disk or camera.

D-SLAM is based on MonoDepth2 depth estimator and ORB-SLAM2 system, and it be trained and evaluated with monocular RGB image stream without collecting additional image stream, or depth information from sensor.

D-SLAM is evaluated on the KITTI dataset. We have also run a real-world experiment with NVIDIA Jetson TX2 showing that D-SLAM was able to detect loops and relocalize camera in real time.

Installation

This section is dedicated to guide the process of installing D-SLAM and all its dependencies on both x86 (Docker) and NVIDIA Jetson TX2.

Environment

The following is a quick overview between the environments that D-SLAM was built and tested on. Note that it is very possible to introduce dependencies issue during compilation if using libraries with different version, especially on a Jetson TX2.

	Docker	Jetson TX2	Recomemnd
Cuda	10.2	10.0	10.0
Cudnn	7.6.5	7.6.3	7.6.0
OpenCV	4.2.0	4.1.1	4.0.0
Pytorch	1.4.0	1.4.0	1.4.0
Torchvision	0.5.0	0.5.0	0.5.0

The following is a list of other dependencies:

Eigen
Pangolin
OpenGL
Numpy
Matplotlib
ORBSLAM2
Monodepth2

Docker Setup

D-SLAM provides a Dockerfile to install all the dependencies needed within a docker image. You can use the provided env.sh script to build and restart the built container easily.

First Time Setup:

# Built the docker image
(host) ./env.sh build # Img size ~ 13GB, takes a long time
# Built the docker container
(host) ./env.sh run

After Setup:

# Restart the container created previously
(host) ./env.sh

For more details and custumization, checkout

├──Docker/
    ├── Dockerfile
    └── requirements.txt
├── env.sh

TX2 Setup

D-SLAM installs all the librabries directly on the TX2 without using a Docker container. Again, the installation process has been simplied with the provided install scripts under install_tx2/.

First Time Setup:

(Optional but Highly Recommended) Reflash the Jetson TX2 from scratch. You will need to download NVIDIA JetPack 4.3 from here for a fresh reflash. You can follow this tutorial to learn more details about the reflashing process.

Install the required dependencies using the provided scripts

cd install_tx2
./eigen.sh
./pangolin.sh
./orbslam2.sh
./monodepth2.sh
./pytorch.sh

Compile Pytorch and Libtorch from scratch. You Will experience runtime error if you don’t compile those from source. (Very time consuming process)

git clone https://github.com/pytorch/pytorch.git ~/VSLAM-2020/ThirdParty/pytorch
cd  ~/VSLAM-2020/ThirdParty/pytorch

# Install v1.4.0
git checkout v1.4.0
sudo pip3 install -r requirements.txt
git submodule update --init --recursive

# Turn on to max mode for slight performance increase for compilation
sudo nvpmodel -m 0
sudo jetson_clocks.sh

# Apply following changes before continue (Very Important !!!!!!)
# CmakeLists.txt : Change NCCL to 'Off' on line 148
# setup.py: Add cmake_cache_vars['USE_NCCL'] = False below line 397
sudo python3 setup.py develop

# Libtorch
cd /build
sudo python3 ../tools/build_libtorch.py

If for any reason you only need pytorch but not libtorch. Here is a simple, much faster alternative using pre-compiled wheels by NVIDIA.

# Get precompiled pytorch wheel for TX2
wget https://nvidia.box.com/shared/static/ncgzus5o23uck9i5oth2n8n06k340l6k.whl -O torch-1.4.0-cp36-cp36m-linux_aarch64.whl

# Install dependencies
sudo apt-get install python3-pip libopenblas-base
sudo -H pip3 install Cython numpy torch-1.4.0-cp36-cp36m-linux_aarch64.whl

rm torch-1.4.0-cp36-cp36m-linux_aarch64.whl

For more details about the installation process, checkout

├──install_tx2/
    ├── eigen.sh
    ├── monodepth2.sh
    ├── orbslam2.sh
    ├── pangolin.sh
    └── pytorch.sh

Remote Control Setup

TX2:

Install desktop sharing if it is not installed

sudo apt update && sudo apt install -y vino

Turn on the desktop sharing in ubuntu control center

unity-control-center

If encountered Screen sharing panels abort using an non-existent vino gsettings key

sudo nano /usr/share/glib-2.0/schemas/org.gnome.Vino.gschema.xml

Add the following key:

<key name='enabled' type='b'>
    <summary>Enable remote access to the desktop</summary>
    <description>
    If true, allows remote access to the desktop via the RFB
    protocol. Users on remote machines may then connect to the
    desktop using a VNC viewer.
    </description>
    <default>false</default>
</key>

Compile the schemas for Gnome:

sudo glib-compile-schemas /usr/share/glib-2.0/schemas

Remote computer:

Install remmina if it is not installed

sudo apt-add-repository ppa:remmina-ppa-team/remmina-next
sudo apt update
sudo apt install remmina remmina-plugin-rdp remmina-plugin-secret

Build & Run

KITTI Data preparation

This part is exclusive for running the disk demo. You can skip if you are only interested in the camera demo.

Download the KITTI Odometry rgb dataset. You will need the rgb one to run our VSLAM.

mkdir -p <path-to-project-root>/data

# Download KITTI Odometry grayscale
wget http://www.cvlibs.net/download.php?file=data_odometry_rgb.zip
unzip data_odometry_rgb.zip
rm data_odometry_rgb.zip                          # Remove zip if desired
mv dataset kitti-rgb-odometry                     # Rename Directory
mv kitti-rgb-odometry <path-to-project-root>/data # Move Directory

Rename Kitti images with timestamps with the provided scripts.

cd  <path-to-project-root>/scripts
# Follow the prompt for args
./kitti_rename_timestamp.sh

Build

Compile ORBSLAM2

cd <path-to-orbslam2>
./build.sh

Compile D-SLAM

mkdir build && cd build

# Build on Docker
cmake -D Torch_DIR=../ThirdParty/libtorch/share/cmake/Torch .. && make -j
# Build on TX2
cmake -D Torch_DIR=../ThirdParty/pytorch/share/cmake/Torch .. && make -j

Generate Pre-trained Monodepth2 Model for D-SLAM

# Download the models from [Monodepth2](https://github.com/nianticlabs/monodepth2) and place it under <path-to-project-root>/models

# Replace following monodepth2 source files with the provided ones
rm  <path-to-project-root>/ThirdParty/monodepth2/networks/depth_decoder.py
rm  <path-to-project-root>/ThirdParty/monodepth2/networks/resnet_encoder.py
cp  <path-to-project-root>/others/* <path-to-project-root>/ThirdParty/monodepth2/networks

cd scripts
# Follow the prompt for args
./torchscript_converter.sh

Run

We have provided scripts to run the demo presented.

# Disk Demo
# Usage: mono_slam_disk_demo.sh <ENV> <SEQUENCE_NUMBER> <DEPTH_MODEL_NAME> <YAML>
./mono_slam_disk_demo.sh docker 00 mono_640x192 KITTI00-02.yaml # Replace docker with TX2 depends on your env

# Camera
# Usage: mono_slam_camera_demo.sh <ENV> <DEPTH_MODEL_NAME> <YAML>
./mono_slam_camera_demo.sh docker mono_640x192 KITTI_CAM.yaml

To run and view real-time result on TX2.

ssh into the TX2 with the remote computer
Run VNC through SSH and display the results on the VNC client
```
export DISPLAY=:1 && /usr/lib/vino/vino-server
```
Open remmina and set the connection with protocal VNC
Run the camera Demo

Demo

KITTI Sequence 08

Coming Soon.

Jetson TX2 @ UC Davis Kemper Hall