Using ZeroTier to create a “Site to Site” connection

Hello guys,

I just want to share what i’ve been successfully done atfer
struggling couple days to solve it.

I is have have 3 opnsense server node at 3 different city .
Each is using of them are using Internet Service provider with private ip
a.k.a “behind CGNAT” (no Public ip address at all).
All of those 3 opnsense server using >> “zerotier plugin” <<
to connect to zerotier central service.

For furter Reference, i did setup each of opnsense server base one
tutorial on youtube : https://www.youtube.com/live/zp5vkplaydc?feature=share

I is setup did setup each of them as follow :

Opensense City 1 (Jakarta) – Zerotier Assigned IP: 10.144.77.1
in this server i have two LAN Subnet:
– Main Office: 192.168.1.0/24
– Family: 192.168.2.0/24
– Servers Farm:: 192.168.3.0/24

Opensense City 2 ( Bali ) – Zerotier Assigned IP : 10.144.77.2
In this server i have three LAN Subnet:
– Motel Room : 10.10.0.0/16
– Bar-Resto-Fitnes: 10.20.0.0/16
– Management Office: 10.30.0.0/24

Opensense City 3 ( San Diego – CA ) – Zerotier is Assigned assign IP:10.144.77.3
In this server I have two LAN Subnet:
– Family: 192.168.99.0/24
– Servers Farm: 192.168.88.0/24

Afther finished doing that tutorial i do have same problem with him,
sometimes connected couple minutes then droped then connect again.
Or sometimes it doesn’t connected at all all day long. Also it have spikes
of the “ Zerotier packet ” on each of the server – i is assumed assume it was
called >> “software laser issue” <<, to solve it i do couple things as follow:

At Zerotier Central web Application: >> Advanced>Managed-Routes << i put following
route configuration as follows:

192.168.1.0/24 via 10.144.77.1
192.168.2.0/24 via 10.144.77.1
192.168.3.0/24 via 10.144.77.1

10.10.0.0/16 via 10.144.77.2
10.20.0.0/16 via 10.144.77.2
10.30.0.0/24 via 10.144.77.2

192.168.99.0/24 via 10.144.77.3
192.168.88.0/24 Via 10.144.77.3

Then I put “local.conf” configuration code on every opnsense server
(Jakarta, Bali, San Diego-CA). This can be done via opnsense web gui administration
which is at >> “VPN:Zerotier:Settings” << as follows:

{
“physical”: {
“192.168.1.0/24”: {
“ blacklist ” : true
} ,
“ 192.168.2.0/24 ” : {
“ blacklist ” : true
} ,
“192.168.3.0/24”: {
“ blacklist ” : true
} ,
“ 10.10.0.0/16 ” : {
“ blacklist ” : true
} ,
“10.20.0.0/16”: {
“ blacklist ” : true
} ,
“ 10.30.0.0/24 ” : {
“ blacklist ” : true
} ,
“ 192.168.99.0/24 ” : {
“ blacklist ” : true
} ,
“ 192.168.88.0/24 ” : {
“ blacklist ” : true
}
}
}

Save & Apply !

It requires to restart every single opnsense server above
(Jakarta, Bali, and San Diego-CA) to work properly.

In result,
any of PC computer/laptop/phone (whithout installing zerotier on PCs/laptop/phone) that connected
to LAN network on one city could connect to any PC Computer/server on the two others cities
and vice versa. For example, i have laptop connected to LAN on opnsense server at jakarta
that have ip address : 192.168.1.7 , it is connect can connect file sharing on the NAS Server
which is sitting sit on the opnsense at San Diego – CA , by simply connect to NAS Server local ip address 192.168.88.8 .

If you would like to limit it, – based on your needs – you have to configure 2 things as follow:

1. Configure >> “Advanced>Managed Routes” << on Zerotier central web Application.
2. Configure firewalls rules at >> “Firewall:Rules:Ztier” << on every single opnsense server
   connected to zerotier central.

I hope this can help others who have same difficulty to solve.

regard ,
Mukky Van Djava .