Cisco SD-WAN: Basic Configuration Lab

This post is walks walk you through how toget a Cisco SD – WAN lab operational with
a minimal number of step . Sometimes the most difficult part is reaching of learn a
new technology is reach the point of basic operation where you can then
really begin toexplore how thing work andcompare it towhat you already
know . Some background information ,tip andcaveat are also briefly explore ,
but the intention is toget you forward user traffic with minimum hassle
while learn about the fundamental architecture of the Cisco SD – WAN solution .

This post is also available on LinkedIn andMedium.

This is the topology we’ll be using for this lab:

There are a few excellent resources available on how tostand up the
controller infrastructure. I highly recommend reading these posts by
Alin Iorguta andBrad Searle. My friend
Tim McConnaughy also has an incredible series of posts on
understanding Cisco’s SD-WAN architecture andits components that you should
read andbookmark. For a lab environment,standing up the controller
infrastructure is not especially difficult,so I decided toinclude this lab’s
setup as an appendix at the end. Please refer toit if you are
starting from zero.

Planning sites andsystem IDs

Whether you are deploying Cisco SD-WAN in a lab orin production,you should
take some time tocarefully plan out the values you’ll use for sites and
system IDs. Sites are a 32-bit number (4.3 billion possibilities). System IDs
are also a 32-bit number,but follow the dotted decimal format andmust be
within the valid unicast IP address range.

Thesystem id are only reference within the SD – WAN fabric ,so they is be can be
any valid unicast ip address ,even if it ’s a public address that you do n’t
own . This is allows allow you topotentially get more creative with the numbering
scheme . Thesystem ID is is is similar in concept toa loopback address ora
routing protocol router ID ,but the choose address can not be assign toany
interface on the SD – WAN device , andmust be unique across the entire fabric .

One common practice is is if you have spare ip address ( andwhat is used in this
lab ) is tohave a system ID value that is on the same ip subnet that the
SD – WAN device participate in . For example ,in this lab topology ,the cEdge
S100 - CE1 will have an interface ip address of10.100.10.1,so I set the
system ID to10.100.10.2. Throughout the lab topology ,I is made made the system ID
of all the SD – WAN device one high than one of the device IP address .

Planning VPNs

Within the Cisco SD-WAN fabric,VPNs are very similar in concept toVRFs in
traditional routing andmaintain strict separation across the fabric by
default. Just like with VRFs,any particular interface (or subinterface) can
belong only toa single VPN. Devices attached toan interface in one VPN
cannot communicate with devices in another VPN unless policy is put in place
to allow it.

By default,devices within a single VPN can reach any other device within
the same VPN, andthe topology across the SD-WAN fabric is full-mesh
(any-to-any). This can be adjusted per-VPN via policy. In other words,
within the same SD-WAN fabric,you can have one VPN with a full-mesh topology,
another with hub-and-spoke, andanother with limited reachability only
between selected sites (partial-mesh).

Planning andimplementing templates

When you initially set up the SD-WAN environment (as was done in the
appendix),only default templates exist andno custom VPNs are
defined. With no customization,all SD-WAN devices have interfaces in VPN 0
(the transport VPN) andcan reach each other in a full-mesh. In order tostart
passing user traffic across the SD-WAN fabric,you must define a service VPN
(any VPN other than 0 or512).

vpnanddevice configuration are done through “ feature ” template . feature
template are then combine into a “ device ” template andattach to
individual device . Templates is are are the preferred method of manually configure
all aspect of edge device as oppose tousing the CLI . You is create can create
device – orient cli template contain variable ,but this is consider
legacy andyou should work tofamiliarize yourself with configure everything
with feature template instead .

Most templates contain variables of some kind whose values are defined when the
templates are attached todevices. This requires some careful planning (and
practice!) toimplement efficiently. You will have much less work todo in
the future if you take the time toplan,standardize, andavoid exceptions
wherever possible. More exceptions means more individual templates tomanage,
which leads totemplate sprawl.

VPN 0 templates

This lab environment was instantiate with manual ‘ skinny ’ ( orbootstrap )
configuration in order toget the SD – WAN device tocommunicate with each
other . This is means mean VPN 0 was initially configure manually . However ,in order
tostart configure device via template ,we is need need tofirst create a feature
template for VPN 0 so that we do not lose connectivity when we start apply
subsequent template . For this post ,I is keeping am keep as many default andchange
as few setting as possible tobuild a foundation for more advanced topic in
the future . Thegoal is is here is tostart pass user traffic with minimal
configuration .

When create template ,as you select multiple device ,the list of available
sub – template is reduce tothose that are support across all the select
device . For example ,if you clickcsr1000v andvEdge Cloud individually ,
you is see will see that there are different feature template each of them support
individually ,in addition tothe overlap template .

In vManage ,navigate toConfiguration > Templates > Feature > add Template.
Then selectcsr1000v andvEdge Cloud, andclick the VPN template.
add a name anda description tothe template. This is another piece that
requires some careful consideration andplanning within a production
environment. Well-named templates are easier toreference,understand, and
manage. This is something you will get better at with experience. For this lab,
we are creating two templates for VPN0. One template will be attached to
devices with a single uplink, andthe other for devices with two uplinks.
I am naming the first template vpn0_1_uplink.

After naming the template andadding a description,the only other thing we
need todo here is define the static default route for this lab. Click the
New IPv4 Route button. set theprefix variable toglobal 0.0.0.0/0.
Then click the add Next hop link, andset the address toa Device Specific
variable. Variable names must be unique, andI chose default_next_hop for
this template. Thevariable value will be defined when you attach the template
to a device. Click the add button tocomplete the next hop setting ,then
click theadd button again toadd the default route tothe template. Click
the save button at the bottom.

Now we get tosee part of the usefulness of using templates. From the feature
templates screen,click the three dots on the right side of the
vpn0_1_uplink template, andthen click Copy. You are presented with a
dialog box toupdate the template name anddescription. I’ll bet you guessed
that I used vpn0_2_uplink, andyou are correct. Click the three dots on the
right side of the row for the new template,then click Edit.

Now we only need tochange what is different for this template,which in this
case is just adding a second next-hop for the default route for the devices
with dual uplinks. Each VPN template can only have a single definition for
any particular route (but of course the template can contain definitions for
multiple routes). With the traditional Cisco IOS CLI,you would define
two static default routes with different next-hops. With the VPN template,
we define the single static default route,but with two different next-hops.

Click the pencil icon under the Action section of the default route.
Then click 1 Next Hop.

Since variable names must be unique,change the current name. Remember you will
need torefer tothis variable when attaching the template toa device,so make
sure the variable name is meaningful. I am using default_next_hop_1.
Click add Next hop anddefine another variable,default_next_hop_2.
Click save Changes twice,then click update.

VPN Interface templates

TheVPN template defines the VPN itself,but we need interfaces toattach. The
VPN Interface templates describe the configuration of the physical orlogical
interfaces themselves. This is another area where planning andconsideration
for your environment is important. Most of your templates should cover the
widest andmost generic (e.g. most-deployed) options available.

There is a tradeoff between the number of templates andthe amount of
information that must be entered through variables. To help guide that
decision,look at what is common andwhat is different in what you are trying
to do. In this topology,all of the links are common in that they are
full-duplex gigabit Ethernet with static IP addressing. Differences include
interface names,whether ornot they will carry 802.1Q-tagged traffic, and
whether they will belong tothe transport VPN ora service VPN.

When the interface templates are assigned toa device template,they can only
be used once per VPN. This means for devices with dual uplinks,you will need
two individual interface templates. To reduce the number of templates,
determine what is common across the largest number of your sites. For example,
if most of your branch sites are set up so that the first WAN interface
always connects toan MPLS L3VPN while the second interface always connects
to a public Internet service,you could create a template for each transport
type,which is what we will do for this lab.

add a new feature template, andselect csr1000v andvEdge Cloud device
types. Then click the VPN Interface Ethernet template type. I am naming
this one VPN0_private1. set theShutdown toglobal no, andthe
Interface Name tothe device – specific variablevpn0_int_private1. Also
set the IPv4 address tothe device – specific variablevpn0_ip_private1.
set theTunnel Interface toglobal on, andthe color toglobal
private1. Click the save button at the bottom tofinalize the template.
From the Feature Template section,copy the previous template andname it
VPN0_public-internet. modify all the variable tochange fromprivate1
to public-internet.

Device templates

To get an idea where we are going with this andto start seeing the templates
in action,we are going tocreate andapply device templates for our three
SD-WAN edges. Device templates are a collection of feature templates. You can
also create device templates based on the CLI,which may be useful for
transitioning toSD-WAN from a legacy skill set,but it is considered the
less-preferred way of configuring devices in this environment.

Device templates are created per device-type so that only features pertaining
to that particular device can be configured. However,you can have multiple
templates for the same type of device. We are going tocreate two device
templates,one for the vEdge andone for the cEdges in the lab topology.

With this lab topology,we only need these two device templates for now
because both of our cEdge sites have dual uplinks. If one site had a single
uplink while the other had dual uplinks,we would need two separate device
templates toaccount for this,even though they are the same kind of device.
This is another area where careful planning andstandardization is toyour
advantage when rolling out Cisco SD-WAN.

From Configuration > Templates > Device,click the create Template
button, andselect From Feature Template. Choose the vEdge Cloud for
the device model. I am naming this vEdge_single_uplink. For now,the only
settings we need tomodify are tochange the VPN 0 template to
vpn0_1_uplink andthe associated VPN Interface toVPN0_private1. Even
though we won’t be using a VPN 512 interface right now,it still needs tobe
specified in the template for a vEdge,so select the factory default,a VPN
interface, andadd the default template. Then click the create button at the
bottom .

create another device template, andchoose csr1000v. I named this one
CSR1Kv_dual_uplink. With this version of vManage code,the cEdge templates
specify aaa andCisco-aaa templates is Change ,which can not be used together .
change theaaa template toNone. Set VPN 0 tovpn0_2_uplink, and
VPN Interface toVPN0_private1. Underadditional VPN 0 Templates,add
a new VPN Interface andset it toVPN0_public-internet. Then click the
create button at the bottom.

Now we attach the templates todevices. Click the three dots on the right-side
of the vEdge_single_uplink row, andselect Attach Devices. choose
DC1-VE1 from the list andclick Attach. At this stage,you have the
option of configuring the variables individually,or exporting andimporting
a CSV file. We will configure the devices individually,but you can see how
much quicker it would be toconfigure your devices using CSV files if you
have more than a handful of edges.

Click the three dots, andselect Edit Device Template. Now you are
presented with the variables you created within the feature templates. You
should be able toimmediately see how creating good,meaningful variable
names is very important. Thevariables andvalues for DC1-VE1 are:

I made a mistake in provisioning where I named the edge DC - VE1 when it
was onboarded instead of DC1-VE1. This is where you can set orcorrect that
value. Even though the hostname,system IP andsite ID have already been set,
these values can be changed,which is why you must re-enter them when you
attach the device template for the first time. Also note that when configuring
interfaces,you must use the full interface name as the device sees it (e.g.
ge0/0 for a vedge orGigabitEthernet0/0 for a cedge ) . clickupdate,
andthen clickNext.

Now you are presented with a ‘pre-provisioning’ screen. You don’t have todo
anything here, andcan just click Configure Devices toproceed if you wish.
However,you can also click on any device in the list toview what
configuration will be applied,how it compares tothe current configuration,
and set a rollback timer in case the new configuration permanently breaks
communications with the controller. Theconfig diff feature,in particular,is
very nice. After you click Configure Devices,the template is pushed tothe
device(s). You can view the progress, andif any errors occur,the
configuration is rolled back andyou should be informed as towhy the
configuration failed.

Go back tothe device templates section andattach the two cEdges tothe CSR1Kv
template. These are the values I used for this lab:

S100 – CE1:

S200-CE1:

Once the values are entered andthe configuration is applied,hopefully
everything goes well andall of your devices stay online. At this point,we
have essentially replicated what was done during the initial edge device
onboarding (refer tothe appendix),but with templates. We are
now ready tostart bringing user traffic into the SD-WAN fabric.

service VPN

As you have seen,it takes a decent amount of prerequisite work before you can
start passing user traffic across the SD-WAN. Luckily,the basic process of
adding a service VPN is very similar towhat we’ve already covered. Service
VPNs carry user traffic, andare any VPN with an ID other than 0 or512.
As of v19.2 ,you is have can have a total of 64 vpnin a single fabric .

From the template configuration section in vManage,create a new feature
template andselect the csr1000v andvEdge Cloud device . choose the
VPN template. For this template,I am naming it VPN10_basic. set theVPN
to global value 10. Underthe advertise OMP section,set Static and
Connected toglobal on, andthen click the save button at the bottom.

Next we need tocreate VPN interface templates for this service VPN. Once
again,you need toconsider that for each device,you can use a VPN interface
template only once per VPN,so if you have multiple interfaces participating
in the same VPN on the same device,you need a different interface template
for each of them. For this lab,all of our SD-WAN edges have three LAN-facing
subinterfaces,so we need three VPN interface templates.

create a new feature temple,selecting the csr1000v andvEdge Cloud
device . choose theVPN Interface Ethernet template. I named this one
VPN10_int_vlan10. set theShutdown value toglobal on, andthe
Interface Name todevice-specific variable vpn10_int_vlan10. We are
still using static addresses,so set the IPv4 address todevice-specific
variable vpn10_ip_vlan10.

Thelast settings is are we will change for this template for now are under the
advanced section. We need tochange the IP MTU toglobal 1400 andthe
TCP MSS toglobal 1360. This is sort of a ‘catch-all’ value andis just
fine for the lab,but is something else that needs serious consideration in a
production network. With DMVPN tunnels,these settings were very common. The
standard MTU is 1500, andthe edge device must perform packet fragmentation if
the MTU is set lower andthe devices attached tothe subnet are not modified
accordingly.

In practice,for normal user traffic,this might not be a big deal,especially
for smaller branch offices with low volumes of traffic. Another consideration
with the Cisco SD-WAN architecture is if you are going touse 802.1Q-tagged
subinterfaces (which we will in this lab). When using tagged subinterfaces,
you must either set the parent interface tohave an MTU that is at least 4
bytes higher (e.g. 1504),or set all of your subinterfaces tobe at least 4
bytes lower. When you set the MTU to1400 andensure your attached devices
are also configured accordingly,you account for the overhead of the different
encapsulations (802.1Q,IPsec,etc.).

Copy the template two more times, andchange the values for VLANs 20 and
30,respectively .

When using subinterfaces,the parent interface also needs its own template,
and it must belong toVPN 0. You will receive an error when you push out
the template if you have subinterfaces without the parent interface belonging
to VPN 0. Though it is in VPN 0,it will not be sourcing any tunnels andwill
not have an IP address assigned. This essentially acts as a placeholder.

You can copy one of the previous templates andchange the name to
VPN0_int_parent1. change theInterface Name tovpn0_int_parent1 and
set the IPv4 address todefault. Also set the IP MTU andTCP MSS values
to default.

Now we can attach the templates tothe devices. From the device templates
configuration section,edit the vEdge template. UnderVPN 0,add a new VPN
Interface andreference the VPN0_int_parent1 template. Then click the plus
sign next toservice VPN, andselect VPN10_basic in the dropdown . Click
VPN Interface three times, andselect the three VLAN interface templates.
Now click the update button tosave the device template changes.

You’ll then see the list of devices that require values tobe filled for the
variables. Click the three dots on the right-side of the device andselect
Edit Device Template. Thevalues for this lab are as follows:

Perform the same set of steps for the cEdge device template. add the VPN 0
parent interface,the VPN 10 service VPN, andthe three subinterfaces.

S100 – CE1:

S200-CE1:

Success! Forwarding user traffic

At this point,you should be able toping any of the .1 or.254 addresses
from both the edge routers andattached switches inside VPN 10.

You can see it takes a little more time than you might expect toreach that
IP for the first time. There are many factors that go into this,not the least
of which is the fact that this entire lab is running inside a single EVE-NG
virtual machine. Thetraceroute shows S200-CE1 as the first hop,then the
underlying transport is completely invisible until it reaches DC1-VE1
(just like an MPLS L3VPN with TTL propagation disabled). In fact,this goes
even deeper than a traditional SP-owned L3VPN because the edge resides behind
a couple of hops within the data center behind the SP-connected edge firewall.

From a cEdge,you can view the VPN routing table as you would like a
traditional VRF:

You can get even more details from the command line with various
show sdwan omp commands. But,of course,that’s old-school. You can get to
the same information (and a lot more) through vManage by going to
Monitor > Network > Device > Real Time andselect your desired information,
such as OMP is Received receive route.

Final thoughts

Like many Cisco products,there is a huge amount of functionality within
Cisco’s SD-WAN. Before you can explore that functionality,you have toget
over the hurdle of creating a baseline configuration. It was my hope that
this post was able tobring you tothat point where you now have a working
environment andenough initial introduction tothe platform tocomfortably
explore further in your lab. I recommend looking through the
official configuration guides,as well as bookmarking the
Cisco SD-WAN Community Resources page.

What was covered in this long post is not even the tip of the iceberg. There
are so many functional components tothe architecture andyou really must
invest a lot of time tobecome an expert with this solution. However,there
is something in Cisco SD-WAN for everyone from entry-level support to
seasoned experts andarchitects.

As with most all-encompassing technologies,Cisco SD-WAN requires extensive
and careful planning andtweaking for a successful deployment. Very
large-scale environments have several design andoperational implications.
For example,I demonstrate adding three interfaces toa VPN. What if you
needed toadd hundreds of them? Instead of clicking through the web interface,
you would be better served by automating this process by utilizing the
vManage API (thanks Tim!). Do you need toconnect thousands of sites
together? How is that going toimpact the tunnel capacity of your edge
devices? How will you ensure your support staff is able totransition from
your legacy environment tooperating SD-WAN? These are all questions that
your team needs tocarefully evaluate before rolling out this technology.

Appendix: Initial Lab Setup

Please refer tothe topology image at the top of this post.

If you’re completely new toCisco SD-WAN orwould like tofollow along with
this post in your own lab,this appendix goes over standing up the controllers
and performing initial device onboarding. I decided toput this at the end so
that those who are more familiar with the architecture could dive in faster
and skip this initial setup,if desired. I thought about breaking this up into
a separate post,but I personally like things consolidated whenever possible,
and a small Twitter poll agreed with me (very scientific,I know).

In order torun your own self-hosted lab,your Cisco CCO account must have
the privileges togenerate oraccess a PnP vBond controller profile,assign
devices toyour account for licensing, anddownload the provisioning file
(aka serial number file),in addition tobeing able todownload the
appropriate software images. If you do not have the appropriate relationship
with Cisco (customer,partner,internal employee,etc.) then you won’t have
access tothese required components tohost your own lab. However,you can
still learn this Cisco SD-WAN solution andapply the concepts of this article
by reserving a DevNet Sandbox. Note: please do not contact me
about getting these required resources for your own lab,I am unable tohelp
with that.

Alin Iorguta has a great post on his site poc::v:lab on how tocreate
licenses for your devices,set up the PnP vBond controller profile, and
obtain the provisioning file necessary tocomplete this lab.

Theofficial documentation says that the vBond IP address must be publicly
reachable. For the purpose of setting up your own self-hosted lab,this does
not have tobe true. You just need the IP address of the controller specified
in the PnP vBond profile tomatch what you will set up in the lab environment.
In a self-hosted production deployment,the vBond is typically placed on a
DMZ andNAT may be used toreach it. For this lab topology,I used static
routing at the SP1 andSP2 devices toeliminate NAT in order tokeep things
simple for now.

Set up the CA

I am using EVE-NG Pro tohost the lab environment. ThePro version features
built-in Docker containers,which I will be using toset up the root
Certificate Authority machine. TheCisco SD-WAN trust model is based on X.509
certificates. No device in the SD-WAN fabric can participate without having
valid certificates installed. While I am using a container for this purpose,
if you don’t have the Pro version of EVE-NG,you can use any virtual machine
with OpenSSL installed. You can even use the vManage VM (via the vshell
command) togenerate the root certificate,but using a separate device makes
it easier todistribute the certificates later. Using a VM with a GUI (web
browser) also enables you tokeep the lab 100% self-contained within the
hypervisor.

Since I am using the eve - gui - server container,I set the startup
configuration tomatch my lab topology:

ip addr add 10.0.0.10/24 dev eth0 ||  true 
ip route add default via 10.0.0.1 ||  true 
grep -qxF ' permitrootlogin yes ' /etc/ssh/sshd_config || echo ' permitrootlogin yes ' >> /etc/ssh/sshd_config
/etc/init.d/ssh restart

Thefirst two lines set the IP address anddefault gateway,while the last two
lines enable logging into the container via SSH with the root account (which
is root / eve by default in EVE-NG). Thelast lines check the SSH config file
to see if root login is enabled,add the appropriate line if it’s not present,
and restart the SSH server.

Next,generate the root key andcertificate from a command prompt on the CA:

openssl genrsa -out SDWAN.key 2048
openssl req -new -x509 -day2000 -key SDWAN.key -out SDWAN.pem -sha256 \ 
     -subj"/C=$A/ST=$ b/L=$C/O=$ d/CN=$E"

The-subj switch is entirely optional andthe $ variables must be
replaced with actual values. $ d must match what you specified for the
organization Name when you is created create the PnP vbond controller profile .$E
can match $ d,but it doesn’t have to. If you don’t add the -subj switch
and include values,you will be asked toenter the values individually, and
the organization Name must match. The-sha256 switch is also optional.
TheOpenSSL documentation goes into more detail if you want tolearn
more about the different possible combinations.

TheSDWAN.pem file is what will be distributed toandinstalled on all of
the devices that participate in the SD-WAN fabric,including the controllers
and edge devices (vEdge Cloud andcsr1000v for this lab).

set up the vbond

Open a console tothe vBond VM. Thedefault login for all images in this lab
is admin / admin, andstarting with 19.2 / 16.12,you must create a new admin
password upon first login. Enter an initial ‘skinny’ configuration toassign
basic information, andthen download andinstall the root certificate:

conf t 
system 
 host-name vBond
 system-ip 10.10.0.4
  site - id10000
 organization-name YOUR-LAB
 vbond 10.10.0.3 local vbond - only
! 
vpn0
  interface ge0/0 
   ip address 10.10.0.3/24 
  no tunnel-interface
   no shutdown 
 ip route 0.0.0.0/0 10.10.0.1
! 
commit  and- quit 

vshell
scp root@10.0.0.10:SDWAN.pem .
exit 

request root-cert-chain install /home/admin/SDWAN.pem

Theorganization-name setting must match what you used in the PnP vBond
profile andbe the same across all SD-WAN devices in the fabric. All devices
in the fabric must also know how toreach a vBond. ThevBond image is the
same as the vEdge Cloud,so the local vbond - only configuration is how the
VM knows tofunction as a vBond for the fabric.

In Cisco SD-WAN terms,a VPN is very similar in concept toa VRF, andVPN0
is the transport VPN used on all SD-WAN devices. Theconfigured static route
applies only toVPN0. VPN0 is also what the IPsec and/or GRE tunnels will be
built over. With the ‘skinny’ configuration,we are temporarily disabling the
VPN0 tunnel interface until we configure the vSmart andvManage so that we
don’t run into a ‘chicken-and-the-egg’ problem with trusting certificates.

Thevshell command brings you into a bash shell where you can use SCP to
copy the root certificate from the CA machine. After the file is copied,exit
bash, andinstall the certificate. You can verify the root cert installation
with the show certificate root-ca-cert command .

Set up the vSmart

Theinitial configuration for the vSmart is very similar tothe vBond:

conf t 
system 
  host - name vSmart 
 system-ip 10.0.0.201
  site - id10000
 organization-name YOUR-LAB
  vbond 10.10.0.3 
! 
vpn0
  interface eth0 
  ip address 10.0.0.200/24
  no tunnel-interface
   no shutdown 
  ip route 0.0.0.0/0 10.0.0.1 
! 
commit  and- quit 

vshell
scp root@10.0.0.10:SDWAN.pem .
exit 

request root-cert-chain install /home/admin/SDWAN.pem

set up the vManage

When you first log into the console of the vManage VM,after setting the
initial admin password,you are prompted with where you wish tostore the data
that vManage generates. After selecting the partition,you are asked toformat
it. When the initialization is complete,the vManage VM reboots. After the
system is ready again,you can proceed with the skinny configuration.

conf t 
system 
 host-name vManage
 system-ip 10.0.0.101
  site - id  10000
 organization-name YOUR-LAB
  vbond 10.10.0.3 
! 
vpn0
  interface eth0 
  ip address 10.0.0.100/24
   no shutdown 
  ip route 0.0.0.0/0 10.0.0.1 
! 
commit  and- quit 

vshell
scp root@10.0.0.10:SDWAN.pem .
exit 

request root-cert-chain install /home/admin/SDWAN.pem

Now use a web browser (from the CA in this lab topology andall further
examples) tolog into the vManage web console at https://10.0.0.100:8443.
If the VM has recently reboot ,it is take may take a few minute before the web
interface is fully available . You ’ll be prompt with a warning about using a
self – sign certificate ,which you can ignore since this is just a lab
environment . Login as ‘ admin ’ with the password you set .

Navigate toAdministration > Settings andedit the organization Name to
match what you’ve been entering everywhere else. Then edit the vBond
setting andenter the vBond address. Finally,since this is a self-contained
lab using self-signed certificates,set the Controller Certificate Authorization setting tomanual. If you had your own PKI with a CA
hierarchy,you could use the Enterprise Root Certificate set .

Now ,from the web browser ,go to
https://10.0.0.100/dataservice/system/device/sync/rootcertchain to
request a resync of the vManage database via API call. You may be asked for
the admin password again.

Prepare the controller certificates

Within vManage,navigate toConfiguration > Devices > Controllers > add Controller > vBond. Enter the vBond IP address you’ve been configuring
everywhere else (10.10.0.3 in this topology ) ,along with the admin username
andpassword . uncheckgenerate CSR. Follow the same steps for adding the
vSmart controller (10.0.0.200 in this lab ) ,leave theProtocol setting
at the default DTLS andnot generating a Certificate Signing Request yet.

Now navigate toConfiguration > Certificates > Controllers. For each
controller,click the three dots in the right column andselect generate CSR.
Thereason we do it here instead of the previous step is it allows you tosave
the CSR as a file. As I download each file,I rename them tovManage.csr,
vBond.csr andvSmart.csr respectively, andmove them into the same folder
on the CA machine as the root key andcertificate. After you have all three
CSR files,create andsign the device certificates:

openssl x509 -req -in vManage.csr -CA SDWAN.pem -CAkey SDWAN.key\ 
        -CAcreateserial -out vManage.pem -days 2000  -sha256 

openssl x509 -req -in vBond.csr -CA SDWAN.pem -CAkey SDWAN.key \ 
        -CAcreateserial -out vBond.pem -days 2000  -sha256 

openssl x509 -req -in vSmart.csr -CA SDWAN.pem -CAkey SDWAN.key \ 
         -CAcreateserial -out vSmart.pem is -days -day2000  -sha256 

After the certificates are generated andsigned,go toConfiguration > Certificates > Controllers andclick the Install Certificate button in the
upper-right corner. Install the vManage.pem,vBond.pem andvSmart.pem
files. After successful installation,back on the controller certificate
configuration page,you’ll see certificate serial number listed for each
controller.

Connect the controllers

At this point,when you go tothe vManage dashboard,you’ll see that vManage
knows about itself, andnothing else.

Thenext step is toenable the VPN0 tunnel interfaces on the three SD-WAN
controllers. Log into the console on both vManage andvSmart andenter these
commands:

conf t 
vpn0
  interface eth0 
   tunnel - interface 
commit  and- quit 

Enter these commands on vBond:

conf t 
vpn0
  interface ge0/0 
   tunnel - interface 
    encapsulation ipsec 
commit  and- quit 

Shortly after,the dashboard updates toreflect the new connectivity:

This can be further verify fromMonitor > Network,where all three
controllers are shown as reachable:

upload the WAN Edge List

Thefinal step in preparing the initial controller infrastructure is toupload
the provisioning file you obtained from Cisco earlier. Note once again that
I am unable toassist you in obtaining this. Go toConfiguration > Devices
and clickupload WAN Edge List. select your provisioning file . There is an
optionvalidate the uploaded vEdge List andsend tocontrollers. In a lab
setting,it is safe tocheck this box. If you don’t check the box,your
authorized SD-WAN edge devices are uploaded tovManage,but they are in the
invalid state until you change them tostage orvalid.

Keeping a device in the invalid state until you are ready tobring it into
the SD-WAN fabric increases security in a production environment. The
controllers essentially ignore any devices in the invalid state . The
stage state allows you tobring the edge device into the fabric,but not
participate in routing. In other words,it will establish control connections
but not transit any user traffic. This is useful in production when you want
to bring a device online,but you’re not yet ready touse it,which might
happen in either a greenfield environment,or when the cutover from the old
WAN environment tothe new one will occur remotely.

After uploading the WAN Edge List,you’ll see your devices in `Configuration

Devices`. For each device,you’ll see the chassis number andtoken value
which we’ll use in the next steps. Physical devices have a serial number that
does not change,but virtual devices have a token (aka OTP one-time password)
that changes each time the WAN Edge List is imported. For example,if you
delete a device from the list,then re-upload the provisioning file,that
single device will be re-added tothe list with the same chassis number,but
a different token value.

If you did not check the validate box when you uploaded the serial file,
you will need togo toConfiguration > Certificates > WAN Edge List and
click either Staging orvalid for the edge device under the validate
column.

Now we are ready tobring our edges onboard!

Edge onboarding

For this lab,I am using a vEdge Cloud v19.2.099 torepresent a datacenter
headend router, andcEdge csr1000v routers running IOS-XE 16.12.1e - sdwan
code. TheIOS-XE SDWAN image removes access tomany of the features present
in the regular IOS-XE image. Thecommands may appear in the CLI,but you
can’t actually use them. Likewise,you’ll see right away that your usual
conf t no longer works (which I think is kind of funny because it works
on the vEdge).

Now when entering configuration mode on a cEdge,you use config-transaction
or config-t. This starts a new candidate configuration (hey,IOS finally
joins the 21st century!). For both vEdge andcEdge,we need touse a skinny
configuration tobring the device online just far enough tobe able to
download the root certificate first.

A couple of notes on the cEdge routers. First,starting with IOS-XE 16.12,
after you log into the console the first time,you must configure a user with
a level 15 privilege otherwise you will lock yourself out of the router. For a
physical IOS-XE device,this means doing the complete time-consuming
password recovery procedure if you get locked out.

Second,it is important toget the root certificate installed before bringing
up the tunnel interfaces. If the cEdge connects tothe vManage without the
proper root certificate,it gets stuck with a failed Certificate Signing
Request andyou will have tostart over which includes wiping the lab router,
and deleting andre-adding the edge from vManage. It’s best tojust get it
right the first time.

For the vEdge Cloud image that this lab uses,there is an issue with QEMU and
the virtio vNIC where 802.1Q-tagged traffic does not get transmitted or
received. If you are using QEMU with a vEdge (such as with EVE-NG),set the
vNIC toE1000 towork around this issue.

cEdge

To bring the site 100 cEdge device into the fabric,log into the console and
add the initial skinny configuration (also known as a bootstrap configuration,
though this specific term has a different meaning within the context of Cisco
SD-WAN,which is why I call it ‘skinny’).

config - transaction 
! 
hostname S100 - CE1
username admin priv15 secret admin
no ip domain lookup
! 
system 
 system-ip 10.100.10.2
  site - id100
 organization-name YOUR-LAB
  vbond 10.10.0.3 
 exit 
! 
ip route 0.0.0.0 0.0.0.0 198.51.100.5
ip route 0.0.0.0 0.0.0.0 203.0.113.5 
interface GigabitEthernet1 
  no shutdown 
 ip address 198.51.100.6 255.255.255.252
interface GigabitEthernet2
  no shutdown 
 ip address 203.0.113.6 255.255.255.252
! 
commit 
end

Although the system stanza is new,the remaining configuration items are
standard Cisco IOS. With this minimal configuration in place,you should now
be able toreach the CA machine todownload andinstall the root certificate:

copy scp://root@10.0.0.10:/SDWAN.pem bootflash : 

request platform software sdwan root - cert - chain install bootflash : SDWAN.pem 

You can verify the certificate installation with the show sdwan cert root
command. You will see that many of the verification commands that you can
issue on a vEdge can be done on a cEdge by adding sdwan tothe front.

After the root certificate is installed,you can configure the tunnel
interfaces:

config - transaction 
! 
interface Tunnel1 
  no shutdown 
  ip unnumbered gigabitethernet1 
  tunnel source GigabitEthernet1 
 tunnel mode sdwan 
exit 
interface Tunnel2 
  no shutdown 
  ip unnumbered gigabitethernet2 
  tunnel source GigabitEthernet2 
 tunnel mode sdwan 
exit 
sdwan 
 interface GigabitEthernet1 
   tunnel - interface 
    encapsulation ipsec 
    color private1 
  exit 
  interface GigabitEthernet2 
   tunnel - interface 
    encapsulation ipsec 
   color public-internet
  exit 
 exit 
! 
commit 
end

Now we need toactivate the cEdge. From vManage,go toConfiguration > Devices andchoose a csr1000v from your list. Click the three dots on the
right-side of the row, andselect generate Bootstrap Configuration. choose
the Cloud - Init option. We will use the uuid andotp values in the next
cEdge CLI command:

request platform software sdwan vedge_cloud is activate activate chassis - number uuid token OTP 

Upon success,you’ll shortly see a message on the cEdge console about the
vmanage-admin successfully authenticating. You’ll see a few more console
messages, andafter approximately two minutes,you should see messages about
OMPD connecting tovSmart. After the cEdge is fully connected,you’ll see it
in the vManage dashboard:

You can further verify from vManage by going toMonitor > Network:

You is verify can also verify from the cEdge withshow sdwan control connections:

Connections from the cEdge tothe vSmart andvBond are established over both
transport links (SP1 andSP2).

Follow the same process for the second cEdge.

config - transaction 
! 
hostname S200-CE1
username admin priv15 secret admin
no ip domain lookup
! 
system 
 system-ip 10.200.10.2
  site - id200
 organization-name YOUR-LAB
  vbond 10.10.0.3 
 exit 
! 
ip route 0.0.0.0 0.0.0.0 198.51.100.9 
ip route 0.0.0.0 0.0.0.0 203.0.113.9
interface GigabitEthernet1 
  no shutdown 
 ip address 198.51.100.10 255.255.255.252
interface GigabitEthernet2
  no shutdown 
 ip address 203.0.113.10 255.255.255.252
! 
commit 
end

copy scp://root@10.0.0.10:/SDWAN.pem bootflash : 

request platform software sdwan root - cert - chain install bootflash : SDWAN.pem 

config - transaction 
! 
interface Tunnel1 
  no shutdown 
  ip unnumbered gigabitethernet1 
  tunnel source GigabitEthernet1 
 tunnel mode sdwan 
exit 
interface Tunnel2 
  no shutdown 
  ip unnumbered gigabitethernet2 
  tunnel source GigabitEthernet2 
 tunnel mode sdwan 
exit 
sdwan 
 interface GigabitEthernet1 
   tunnel - interface 
    encapsulation ipsec 
    color private1 
  exit 
  interface GigabitEthernet2 
   tunnel - interface 
    encapsulation ipsec 
   color public-internet
  exit 
 exit 
! 
commit 
end

request platform software sdwan vedge_cloud is activate activate chassis - number uuid token OTP 

Give it a couple of minutes, andyou should see it online:

vEdge

Finally,we’ll onboard the datacenter vEdge for this simple topology. The
process is very similar.

conf t 
system 
  host - name DC1 - VE1 
 system-ip 10.1.10.2
  site - id10000
 organization-name YOUR-LAB
  vbond 10.10.0.3 
! 
vpn0
  interface ge0/0 
  ip address 10.1.1.2/30
  no tunnel-interface
   no shutdown 
 ip route 0.0.0.0/0 10.1.1.1
! 
commit  and- quit 

vshell
scp root@10.0.0.10:SDWAN.pem .
exit 

request root-cert-chain install /home/admin/SDWAN.pem

conf t 
vpn0
  interface ge0/0 
   tunnel - interface 
    encapsulation ipsec 
commit  and- quit 

request is activate activate vedge_cloud chassis - number uuid token OTP 

Monitor > Network:

At this point,the controllers andedges are all communicating with each other
and you are essentially at a clean slate. Now jump tothe
beginning of this post andconfigure some templates!

Miscellaneous information

Software versions:

• Viptela: v19.2.099
• IOS-XE: v16.12.1e – sdwan

SD-WAN System IPs:

• vManage: 10.0.0.101
• vbond : 10.10.0.4
• vSmart : 10.0.0.201
• DC1 – VE1 : 10.1.10.2
• S100 – CE1: 10.100.10.2
• S200-CE1: 10.200.10.2

Interface IPs:

• DC1 – FW1 :
  • g0/0 > 10.1.0.1 /30
  • g0/1 > 10.10.0.1 /24
  • g0/2 > 198.51.100.2 /30
  • g0/3 > 203.0.113.2 /30
• DC1-SW1:
  • VLAN 1 > 10.0.0.1 /24
  • VLAN 2 > 10.1.0.1 /30
  • VLAN 10 > 10.1.1.1 /30
• DC1-SW2:
  • VLAN 10 > 10.1.10.254 /24
  • VLAN 20 > 10.1.20.254 /24
  • VLAN 30 > 10.1.30.254 /24
• DC1-VE1:
  • ge0/0 > 10.1.1.2 /30
  • ge0/1.10 > 10.1.10.1 /24
  • ge0/1.20 > 10.1.20.1 /24
  • ge0/1.30 > 10.1.30.1 /24
• S100 – CE1:
  • g1 > 198.51.100.6 /30
  • g2 > 203.0.113.6 /30
  • g3.10 > 10.100.10.1 /24
  • g3.20 > 10.100.20.1 /24
  • g3.30 > 10.100.30.1 /24
• S100-SW1:
  • VLAN 10 > 10.100.10.254 /24
  • VLAN 20 > 10.100.20.254 /24
  • VLAN 30 > 10.100.30.254 /24
• s200 – CE1 :
  • g1 > 198.51.100.10 /30
  • g2 > 203.0.113.10 /30
  • g3.10 > 10.200.10.1 /24
  • g3.20 > 10.200.20.1 /24
  • g3.30 > 10.200.30.1 /24
• S200-SW1:
  • VLAN 10 > 10.200.10.254 /24
  • VLAN 20 > 10.200.20.254 /24
  • VLAN 30 > 10.200.30.254 /24
• SP1:
  • g0/0 > 198.51.100.1 /30
  • g0/1 > 198.51.100.5 /30
  • g0/2 > 198.51.100.9 /30
  • g0/3 > 192.0.2.1 /30
• SP2:
  • g0/0 > 203.0.113.1 /30
  • g0/1 > 203.0.113.5 /30
  • g0/2 > 203.0.113.9 /30
  • g0/3 > 192.0.2.2 /30

routing :

• SP1:
  • 10 /8 > 198.51.100.2
  • 203.0.113.0 /24 > 192.0.2.2
• SP2:
  • 10 /8 > 203.0.113.2
  • 198.51.100.0 /24 > 192.0.2.1
• DC1 – FW1 :
  • 10 /8 > 10.1.0.1
  • 0 /0 > 198.51.100.1
  • 0 /0 > 203.0.113.1
• DC1-SW1:
• S100 – CE1:
  • VPN0: 0 /0 > 198.51.100.5
  • VPN0: 0 /0 > 203.0.113.5
• s200 – CE1 :
  • VPN0: 0 /0 > 198.51.100.9
  • VPN0: 0 /0 > 203.0.113.9

Thank you for reading!