<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.4.1">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2025-11-14T21:38:11+00:00</updated><id>/feed.xml</id><title type="html">2stacks.net</title><subtitle>A Collection of Projects and Interests.</subtitle><author><name>2stacks</name></author><entry><title type="html">Getting Started with FRRouting</title><link href="/blog/getting-started-with-frr-on-eveng/" rel="alternate" type="text/html" title="Getting Started with FRRouting" /><published>2019-11-05T00:00:00+00:00</published><updated>2019-11-05T00:00:00+00:00</updated><id>/blog/getting-started-with-frr-on-eveng</id><content type="html" xml:base="/blog/getting-started-with-frr-on-eveng/"><![CDATA[<h1 id="introduction">Introduction</h1>
<p>If you’ve been around networking long enough chances are you have heard of the open source routing software <a href="http://docs.frrouting.org/en/latest/zebra.html">Zebra</a> or 
<a href="https://www.quagga.net/">Quagga</a>.  You may have used them without knowing, if you ever worked with Sidewinder firewalls or Brocade’s Vyatta.  If 
you :heart: Linux and open source routing you may have heard of the next evolution in open source routing <a href="https://frrouting.org/">FRRouting</a> 
aka Free Range Routing (FRR).  If you haven’t;</p>

<blockquote>
  <p>FRR is a routing software package that provides TCP/IP based routing services with routing protocols support such as 
BGP, RIP, OSPF, IS-IS and more.  It uses the Linux kernel’s routing stack for packet forwarding.</p>
</blockquote>

<p>FRR has been around since 2016 when it was forked from the Quagga project.  It has a growing number of contributors and 
support from various networking vendors such as Cumulus, VMware, Volta, Orange and 6wind with the common goal of creating 
a “world class routing stack.”  The FRR project is currently licensed under the <a href="https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html">GPLv2+</a> and is overseen by <a href="https://www.linuxfoundation.org/">The Linux 
Foundation</a> in an effort to ensure the project’s growth and sustainability.</p>

<p>In this post I’m going to demonstrate how to install FRR on the popular network modeling platform <a href="http://www.eve-ng.net/">Eve-NG</a>.  This will 
allow you to evaluate FRR and its many features and hopefully inspire you to get more involved with open source 
routing.</p>

<h1 id="prerequisites">Prerequisites</h1>
<p>If you are not familiar with <a href="http://www.eve-ng.net/">Eve-NG</a> or Emulated Virtual Environment (EVE), it is a platform for designing, testing and
training with virtual appliances and network emulation software.  Although geared more towards networking, you can emulate 
almost anything in Eve-NG that can run on the <a href="https://www.linux-kvm.org/page/Main_Page">KVM</a> hypervisor.  Since FRR is a routing platform Eve-NG provides everything
you need to start creating labs and proof of concepts using FRR.</p>

<p>Eve-NG can run on bare metal or as a virtual appliance on KVM and VMware.  There are a few different <a href="http://www.eve-ng.net/documentation/installation/system-requirement">installation</a> methods
so I’m not going to cover that in this post.  Once you have a running instance of Eve-NG you can follow along
with how to set up FRR for use within Eve-NG.</p>

<h2 id="client-tools">Client Tools</h2>
<p>Whether you choose to work from a Windows, Mac or Linux OS you’ll need the following tools to work with Eve-NG and to install
FRR.</p>
<ul>
  <li>VNC</li>
  <li>Telnet</li>
  <li>SSH</li>
</ul>

<p>It is highly recommended that you install the Eve-NG <a href="https://github.com/SmartFinn/eve-ng-integration">Client Integration</a> tools for your specific OS.</p>

<h1 id="installation">Installation</h1>
<p>Before we start installing FRR its important to check the FRR <a href="https://github.com/FRRouting/frr/releases">Releases</a> page.  Here you’ll find a list of versions and
their supported features as well as the different installation methods for your chosen base operating system.</p>

<p>If you are interested in testing a specific feature you should also review the <a href="http://docs.frrouting.org/en/latest/overview.html#supported-protocols-vs-platform">Supported Protocols</a> section of the FRR
documentation.  Here you’ll find the OS and kernel version requirements for all of FRR’s supported features.</p>

<p>It’s important to note that some of the newest features of FRR require the latest linux kernel versions.  I’m going to 
start with a standard installation of the <a href="https://ubuntu.com/download/server">Ubuntu Server</a> version 18.04.3 LTS and demonstrate how to upgrade to the most 
recent kernel versions.</p>

<p>The process I’m going to use is well documented in the Eve-NG <a href="http://www.eve-ng.net/documentation/howto-s/106-howto-create-own-linux-image">Linux How-To</a> documentation.  I’ve only added a few steps
I’ve found useful for working with FRR.</p>

<h2 id="get-ubuntu-server-iso">Get Ubuntu Server ISO</h2>
<p>Using SSH or the console, log in as <code class="language-plaintext highlighter-rouge">root</code> to your Eve-NG server and download the latest LTS version of the <a href="https://ubuntu.com/download/server">Ubuntu Server</a>.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@eve-ng:~# wget http://releases.ubuntu.com/18.04.3/ubuntu-18.04.3-live-server-amd64.iso
</code></pre></div></div>

<h2 id="create-qemu-directories-and-images">Create qemu directories and images</h2>
<p>From your Eve-NG server create the required directories and image files.  Eve-NG requires the directory be prefixed with
<code class="language-plaintext highlighter-rouge">linux-</code>.  I’m going to install FRR using the latest available Debian package version 7.1.  I’ve chosen a base disk size
of 16GB but you can use any size you like so long as it meets the Ubuntu server <a href="https://help.ubuntu.com/lts/serverguide/preparing-to-install.html#system-requirements">minimum requirements</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /opt/unetlab/addons/qemu/
<span class="nb">mkdir </span>linux-frr-7.1 <span class="o">&amp;&amp;</span> <span class="nb">cd</span> ./linux-frr-7.1
<span class="nb">cp</span> /root/ubuntu-18.04.3-live-server-amd64.iso cdrom.iso
/opt/qemu/bin/qemu-img create <span class="nt">-f</span> qcow2 virtioa.qcow2 16G
</code></pre></div></div>

<p>Before continuing verify the creation of the <a href="https://www.qemu.org/">qemu</a> image file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>root@eve-ng:/opt/unetlab/addons/qemu/linux-frr-7.1# /opt/qemu/bin/qemu-img info virtioa.qcow2  
image: virtioa.qcow2
file format: qcow2
virtual size: 16G <span class="o">(</span>17179869184 bytes<span class="o">)</span>
disk size: 196K
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: <span class="nb">false
    </span>refcount bits: 
</code></pre></div></div>

<h2 id="create-eve-ng-lab">Create Eve-NG Lab</h2>
<p>Now log in to the web interface of your Eve-NG server.</p>

<p><img src="/assets/images/20191105/eve_login.png" alt="Eve-NG Login" title="Eve-NG Login" /></p>

<p>From the root of the file manager click the icon to add a new lab.</p>

<p><img src="/assets/images/20191105/add_lab.png" alt="Add Lab" title="Add Lab" /></p>

<p>Give the lab a name and description similar to <code class="language-plaintext highlighter-rouge">FRR Test</code>.</p>

<p><img src="/assets/images/20191105/add_lab_1.png" alt="Add Lab" title="Add Lab" /></p>

<p>Right click anywhere inside the new lab and select <strong>Node</strong> from the new object menu.</p>

<p><img src="/assets/images/20191105/add_node.png" alt="Add Node" title="Add Node" /></p>

<p>Scroll down the new node list and select <strong>Linux</strong></p>

<p><img src="/assets/images/20191105/add_node_1.png" alt="Add Node" title="Add Node" /></p>

<p>If this is the first linux image you’ve added to Eve-NG you should only see the <strong>linux-frr-7.1</strong> in the list.  Otherwise
choose the FRR image from the image list.</p>

<p><img src="/assets/images/20191105/add_node_2.png" alt="Add Node" title="Add Node" /></p>

<p>To make the installation faster you can add more CPU and RAM to the node settings.
All other settings can be left at their defaults.  Click <strong>Save</strong> once you’ve made your changes.</p>

<p><img src="/assets/images/20191105/add_node_3.png" alt="Add Node" title="Add Node" /></p>

<p>In order to install updates and the FRR packages to the instance, you’ll need to connect it to your local network.  Eve-NG
uses <strong>Cloud</strong> networks to bridge virtual instances to interfaces on your Eve-NG server.  This allows you to connect your
lab instances to the same network used by Eve-NG for access to the Internet.</p>

<p>Right click in the open space of your lab and select <strong>Network</strong> from the new object menu.</p>

<p><img src="/assets/images/20191105/add_network.png" alt="Add Network" title="Add Network" /></p>

<p>Give the network a name and change the <strong>Type</strong> to <strong>Management(Cloud0)</strong>.</p>

<blockquote>
  <p>This will bridge the instance of the Ubuntu
server to the same network used by your Eve-NG server.  If this network is not enabled with DHCP you will have to configure
a static IP on the Ubuntu server during the installation process.</p>
</blockquote>

<p><img src="/assets/images/20191105/add_network_1.png" alt="Add Network" title="Add Network" /></p>

<p>To connect the node to the newly created cloud network, mouse over the node and drag the orange plug icon from the node
to the cloud network icon.</p>

<p><img src="/assets/images/20191105/connect_node.png" alt="Connect Node" title="Connect Node" /></p>

<p>The <strong>Add Connection</strong> menu will appear and allow you to specify which interface on your Ubuntu server instance you want
to connect to the cloud network.  Unless you added additional interfaces when you added your node you should only see an
option for <strong>e0</strong>.</p>

<p><img src="/assets/images/20191105/connect_node_1.png" alt="Connect Node" title="Connect Node" /></p>

<p>Click <strong>Save</strong> to complete the connection.</p>

<p><img src="/assets/images/20191105/connect_node_2.png" alt="Connect Node" title="Connect Node" /></p>

<p>Now right click on the node and select <strong>Start</strong> from the node menu.</p>

<p><img src="/assets/images/20191105/start_node.png" alt="Start Node" title="Start Node" /></p>

<p>If you’ve installed the Eve-NG <a href="https://github.com/SmartFinn/eve-ng-integration">Client Integration</a> tools you should now be able to click on the node to connect with <a href="https://wiki.gnome.org/Apps/Vinagre">VNC</a>.  If
you did not you’ll need to open your VNC client and manually connect to the instance.  You can find the VNC connection URL
by mousing over the node and looking in the bottom left hand corner of your browser window.</p>

<p><img src="/assets/images/20191105/connect_to_node.png" alt="Connect To Node" title="Connect To Node" /></p>

<h2 id="install-ubuntu-server">Install Ubuntu Server</h2>

<p>Once you’re connected to the instance with VNC proceed with the installation of the Ubuntu Server OS.</p>

<p><img src="/assets/images/20191105/install_os.png" alt="Install OS" title="Install OS" /></p>

<p>Accept all of the defaults until you reach the <strong>Filesystem Setup</strong> screen.  Here you should choose the <strong>Manual</strong> option.</p>

<p><img src="/assets/images/20191105/install_os_1.png" alt="Install OS" title="Install OS" /></p>

<p>Select <strong>/dev/vda</strong> from the list of available devices.</p>

<p><img src="/assets/images/20191105/install_os_2.png" alt="Install OS" title="Install OS" /></p>

<p>Now select <strong>Add Partition</strong>.</p>

<p><img src="/assets/images/20191105/install_os_3.png" alt="Install OS" title="Install OS" /></p>

<p>Enter the max size available based on the size of the qemu image you created earlier and select <strong>Create</strong></p>

<p><img src="/assets/images/20191105/install_os_4.png" alt="Install OS" title="Install OS" /></p>

<blockquote>
  <p>The max size should be just shy of the total disk size as the installation has already reserved space for <a href="https://www.gnu.org/software/grub/">Grub</a>.</p>
</blockquote>

<p>Now select <strong>Done</strong>.</p>

<p><img src="/assets/images/20191105/install_os_5.png" alt="Install OS" title="Install OS" /></p>

<p>Select <strong>Continue</strong> to begin the installation.</p>

<p><img src="/assets/images/20191105/install_os_6.png" alt="Install OS" title="Install OS" /></p>

<p>When you reach the <strong>Profile Setup</strong> menu enter a default username and password to be used to access the FRR instance.</p>

<p><strong>NOTE: Do not use <code class="language-plaintext highlighter-rouge">frr</code> as the username as this will break the FRR installation.</strong></p>

<p><img src="/assets/images/20191105/install_os_7.png" alt="Install OS" title="Install OS" /></p>

<p>Select the option to enable the OpenSSH server.  You can optionally import SSH keys if you know what you are doing.</p>

<p><img src="/assets/images/20191105/install_os_8.png" alt="Install OS" title="Install OS" /></p>

<p>Do not select any additional Snap packages for installation, scroll to the bottom of the screen and select <strong>Done</strong>.</p>

<p><img src="/assets/images/20191105/install_os_10.png" alt="Install OS" title="Install OS" /></p>

<p>Allow the system to fully install and update packages.  Do not select the option to <strong>Cancel update and reboot</strong>.</p>

<p><img src="/assets/images/20191105/install_os_11.png" alt="Install OS" title="Install OS" /></p>

<p>Once the installation and OS update have completed select the option to <strong>Reboot</strong>.</p>

<p><img src="/assets/images/20191105/install_os_12.png" alt="Install OS" title="Install OS" /></p>

<p>The reboot will pause and prompt you to remove the installation medium.</p>

<p><img src="/assets/images/20191105/install_os_13.png" alt="Install OS" title="Install OS" /></p>

<p>At this point you should connect to your Even-NG server and remove the <code class="language-plaintext highlighter-rouge">cdrom.iso</code> file you created from the Ubuntu Server 
installation ISO.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /opt/unetlab/addons/qemu/linux-frr-7.1
<span class="nb">rm</span> <span class="nt">-rf</span> cdrom.iso
</code></pre></div></div>

<p>Now from the Eve-NG GUI right click on your node and select <strong>Stop</strong> from the node menu.</p>

<p><img src="/assets/images/20191105/stop_node.png" alt="Stop Node" title="Stop Node" /></p>

<p>Right click on the node again and select <strong>Start</strong> from the node menu.</p>

<blockquote>
  <p>It is important to fully stop and then restart the node to ensure that it does not boot from a cached copy of the 
installation ISO.</p>
</blockquote>

<p><img src="/assets/images/20191105/restart_node.png" alt="Restart Node" title="Restart Node" /></p>

<p>Now click on the node or manually launch your VNC client as before to connect to the console of the node.</p>

<p><img src="/assets/images/20191105/login_to_node.png" alt="Log In to Node" title="Log In to Node" /></p>

<h2 id="upgrade-linux-kernel">Upgrade Linux Kernel</h2>
<p>To get the latest 5.0 kernel versions in the LTS release of Ubuntu Server you’ll need to set up the <a href="https://wiki.ubuntu.com/Kernel/LTSEnablementStack">LTS Enablement Stack</a>.
You can read more about it in this <a href="https://itsfoss.com/ubuntu-hwe-kernel/">HWE Kernel Install</a> guide or just follow the steps below.</p>

<blockquote>
  <p>By default, Ubuntu LTS releases stay on the same Linux kernel they were released with. The hardware enablement stack 
(HWE) provides newer kernel and xorg support for existing Ubuntu LTS releases.</p>
</blockquote>

<p>Login to the FRR system using the username and password you created during the Ubuntu Server installation.</p>

<p>First verify the current kernel version.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>lsb_release <span class="nt">-a</span>
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.3 LTS
Release:        18.04
Codename:       bionic

~<span class="nv">$ </span><span class="nb">uname</span> <span class="nt">-a</span>
Linux frr-01 4.15.0-66-generic <span class="c">#75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux</span>
</code></pre></div></div>

<p>To upgrade to latest kernel version run the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-get <span class="nb">install</span> <span class="nt">--install-recommends</span> linux-generic-hwe-18.04
<span class="nb">sudo </span>apt-get update
<span class="nb">sudo </span>apt-get dist-upgrade
<span class="nb">sudo </span>reboot
</code></pre></div></div>

<p>Now verify that you are on the latest 5.0 kernel version.  If the kernel wasn’t updated run the <code class="language-plaintext highlighter-rouge">dist-upgrade</code> and <code class="language-plaintext highlighter-rouge">reboot</code> again.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span><span class="nb">uname</span> <span class="nt">-a</span>
Linux frr-01 5.0.0-32-generic <span class="c">#34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux</span>
</code></pre></div></div>

<h2 id="add-debian-repo-and-install-packages">Add Debian Repo and Install packages</h2>
<p>Now install FRR from the <a href="https://deb.frrouting.org/">FRR Debian Repository</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl <span class="nt">-s</span> https://deb.frrouting.org/frr/keys.asc | <span class="nb">sudo </span>apt-key add -
<span class="nv">FRRVER</span><span class="o">=</span><span class="s2">"frr-stable"</span>
<span class="nb">echo </span>deb https://deb.frrouting.org/frr <span class="si">$(</span>lsb_release <span class="nt">-s</span> <span class="nt">-c</span><span class="si">)</span> <span class="nv">$FRRVER</span> | <span class="nb">sudo tee</span> <span class="nt">-a</span> /etc/apt/sources.list.d/frr.list
<span class="nb">sudo </span>apt update <span class="o">&amp;&amp;</span> <span class="nb">sudo </span>apt <span class="nb">install </span>frr frr-pythontools
</code></pre></div></div>

<h2 id="add-users-to-frrvty-group">Add Users to frrvty Group</h2>
<p>To allow access to FRR’s <a href="http://docs.frrouting.org/projects/dev-guide/en/latest/vtysh.html#vtysh">vtysh</a> configuration utility you must add users to the <strong>frrvty</strong> group.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>usermod <span class="nt">-a</span> <span class="nt">-G</span> frr,frrvty &lt;your_username&gt;
</code></pre></div></div>

<h2 id="enable-daemons">Enable Daemons</h2>
<p>By default only <a href="http://docs.frrouting.org/en/latest/zebra.html">zebra</a> and <a href="http://docs.frrouting.org/en/latest/static.html">staticd</a> are enabled.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>ps <span class="nt">-ef</span> | <span class="nb">grep </span>frr
root       973     1  0 21:48 ?        00:00:00 /usr/lib/frr/watchfrr <span class="nt">-d</span> zebra staticd
frr        999     1  0 21:48 ?        00:00:00 /usr/lib/frr/zebra <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1 <span class="nt">-s</span> 90000000
frr       1056     1  0 21:48 ?        00:00:00 /usr/lib/frr/staticd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
</code></pre></div></div>

<p>To enable additional daemons edit the <code class="language-plaintext highlighter-rouge">/etc/frr/daemons</code> file with your preferred text editor.  Change the <code class="language-plaintext highlighter-rouge">no</code> to <code class="language-plaintext highlighter-rouge">yes</code> 
for each daemon you want to enable.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span><span class="nb">sudo </span>vi /etc/frr/daemons

<span class="nv">bgpd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">ospfd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">ospf6d</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">ripd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">ripngd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">isisd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">pimd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">ldpd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">nhrpd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">eigrpd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">babeld</span><span class="o">=</span>no
<span class="nv">sharpd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">pbrd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">bfdd</span><span class="o">=</span><span class="nb">yes
</span><span class="nv">fabricd</span><span class="o">=</span><span class="nb">yes</span>
</code></pre></div></div>

<p>Restart frr with systemd.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>systemctl restart frr.service
</code></pre></div></div>

<p>Verify that the new daemons have started.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>ps <span class="nt">-ef</span> | <span class="nb">grep </span>frr
root      2149     1  0 00:49 ?        00:00:00 /usr/lib/frr/watchfrr <span class="nt">-d</span> zebra bgpd ripd ripngd ospfd ospf6d isisd pimd ldpd nhrpd eigrpd pbrd staticd bfdd fabricd
frr       2208     1  0 00:49 ?        00:00:00 /usr/lib/frr/zebra <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1 <span class="nt">-s</span> 90000000
frr       2213     1  0 00:49 ?        00:00:00 /usr/lib/frr/bgpd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2221     1  0 00:49 ?        00:00:00 /usr/lib/frr/ripd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2225     1  0 00:49 ?        00:00:00 /usr/lib/frr/ripngd <span class="nt">-d</span> <span class="nt">-A</span> ::1
frr       2230     1  0 00:49 ?        00:00:00 /usr/lib/frr/ospfd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2235     1  0 00:49 ?        00:00:00 /usr/lib/frr/ospf6d <span class="nt">-d</span> <span class="nt">-A</span> ::1
frr       2239     1  0 00:49 ?        00:00:00 /usr/lib/frr/isisd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2243     1  0 00:49 ?        00:00:00 /usr/lib/frr/pimd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2248     1  0 00:49 ?        00:00:00 /usr/lib/frr/ldpd <span class="nt">-L</span>
frr       2249     1  0 00:49 ?        00:00:00 /usr/lib/frr/ldpd <span class="nt">-E</span>
frr       2250     1  0 00:49 ?        00:00:00 /usr/lib/frr/ldpd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2256     1  0 00:49 ?        00:00:00 /usr/lib/frr/nhrpd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2260     1  0 00:49 ?        00:00:00 /usr/lib/frr/eigrpd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2281     1  0 00:49 ?        00:00:00 /usr/lib/frr/bfdd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2284     1  0 00:49 ?        00:00:00 /usr/lib/frr/pbrd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2287     1  0 00:49 ?        00:00:00 /usr/lib/frr/fabricd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
frr       2291     1  0 00:49 ?        00:00:00 /usr/lib/frr/staticd <span class="nt">-d</span> <span class="nt">-A</span> 127.0.0.1
</code></pre></div></div>

<blockquote>
  <p>Keep in mind that each daemon requires an additional <a href="https://en.wikipedia.org/wiki/POSIX_Threads">pthread</a> and memory and BGP requires more than one additional
thread.  In production is is not unreasonable to have a CPU per enabled daemon.  At minimum BGP would require 2 CPUs and
~600MB of memory (per table) for a full Internet table of routes.</p>
</blockquote>

<h2 id="enable-kernel-forwarding">Enable Kernel Forwarding</h2>
<p>IP forwarding is disabled by default on most linux distributions so in order to perform any routing you must enable ip 
forwarding in the kernel by editing <code class="language-plaintext highlighter-rouge">/etc/sysctl.conf</code> and uncommenting the following lines.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>net.ipv4.ip_forward<span class="o">=</span>1
net.ipv6.conf.all.forwarding<span class="o">=</span>1
</code></pre></div></div>

<blockquote>
  <p>Some documentation lists the setting to enable IPv4 forwarding as <code class="language-plaintext highlighter-rouge">net.ipv4.conf.all.forwarding=1</code>.  Verify which setting
is appropriate for you system.</p>
</blockquote>

<h3 id="surviving-reboots">Surviving Reboots</h3>
<p>At the time of this writing there is a bug in Ubuntu that prevents some sysctl settings from loading after reboot.</p>

<p>One workaround is to create a new file <code class="language-plaintext highlighter-rouge">/etc/rc.local</code> and add the following contents:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#!/bin/bash</span>
<span class="c"># /etc/rc.local</span>

<span class="c"># Load kernel variables from /etc/sysctl.d</span>
/etc/init.d/procps restart

<span class="nb">exit </span>0
</code></pre></div></div>

<p>Make sure that the file is executable by running;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo chmod </span>755 /etc/rc.local
</code></pre></div></div>

<h2 id="optional-settings">Optional Settings</h2>
<p>Additional <a href="http://docs.frrouting.org/en/latest/installation.html#linux-sysctl-settings-and-kernel-modules">sysctl settings and kernel modules</a> are required for <a href="http://docs.frrouting.org/en/latest/zebra.html#mpls-commands">MPLS</a> and <a href="http://docs.frrouting.org/en/latest/zebra.html#virtual-routing-and-forwarding">VRFs</a>.  Please see the FRR installation
documentation for these settings.</p>

<h1 id="initial-frr-configuration">Initial FRR Configuration</h1>
<p>If you want to set some basic defaults in FRR that will be applied to all new instances created in Eve-NG, you should 
configure them prior to committing any changes back to the base qemu image.</p>

<p>You can configure FRR directly via its config files or interactively using <a href="http://docs.frrouting.org/en/latest/basic.html#virtual-terminal-interfaces">vty</a> or <a href="http://docs.frrouting.org/projects/dev-guide/en/latest/vtysh.html#vtysh">vtysh</a>.</p>

<p>To start an interactive shell using vtysh run the vtysh command.  Keep in mind that the currently logged in user must be 
a member of the frrvty group in linux.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>vtysh

Hello, this is FRRouting <span class="o">(</span>version 7.1<span class="o">)</span><span class="nb">.</span>
Copyright 1996-2005 Kunihiro Ishiguro, et al.

Router#
</code></pre></div></div>

<p>In order to use vty you’ll need to configure a vty and enable password.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>vtysh

Hello, this is FRRouting <span class="o">(</span>version 7.1<span class="o">)</span><span class="nb">.</span>
Copyright 1996-2005 Kunihiro Ishiguro, et al.

Router# configure terminal 
Router<span class="o">(</span>config<span class="o">)</span><span class="c"># password zebra</span>
Router<span class="o">(</span>config<span class="o">)</span><span class="c"># enable password zebra</span>
Router<span class="o">(</span>config<span class="o">)</span><span class="c"># exit</span>
Router# write memory 
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Integrated configuration saved to /etc/frr/frr.conf
<span class="o">[</span>OK]
Router# <span class="nb">exit</span>
</code></pre></div></div>

<p>Now you can also access the CLI using the vty interface.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>telnet localhost 2601
Trying 127.0.0.1...
Connected to localhost.
Escape character is <span class="s1">'^]'</span><span class="nb">.</span>

Hello, this is FRRouting <span class="o">(</span>version 7.1<span class="o">)</span><span class="nb">.</span>
Copyright 1996-2005 Kunihiro Ishiguro, et al.


User Access Verification

Password: zebra
Router&gt; <span class="nb">enable
</span>Password: zebra
Router#
</code></pre></div></div>

<h2 id="disable-integrated-configuration-file-optional">Disable integrated configuration file (optional)</h2>
<p>By default FRR uses an <a href="http://docs.frrouting.org/en/latest/vtysh.html#integrated-configuration-mode">Integrated configuration mode</a> where all configs are kept in a single <code class="language-plaintext highlighter-rouge">frr.conf</code> file.  Though 
optional, it is recommended to use a configuration file per daemon.  To disable the use of the integrated <code class="language-plaintext highlighter-rouge">frr.conf</code> and 
instead use a separate configuration file per daemon, first edit the <code class="language-plaintext highlighter-rouge">vtysh.conf</code>file.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo sed</span> <span class="nt">-i</span> <span class="s1">'s/^service/no service/g'</span> /etc/frr/vtysh.conf
</code></pre></div></div>

<p>Now login with <code class="language-plaintext highlighter-rouge">vtysh</code> and save the configuration.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>vtysh

Hello, this is FRRouting <span class="o">(</span>version 7.1<span class="o">)</span><span class="nb">.</span>
Copyright 1996-2005 Kunihiro Ishiguro, et al.

Router# wr mem
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Configuration saved to /etc/frr/zebra.conf
Configuration saved to /etc/frr/ripd.conf
Configuration saved to /etc/frr/ripngd.conf
Configuration saved to /etc/frr/ospfd.conf
Configuration saved to /etc/frr/ospf6d.conf
Configuration saved to /etc/frr/ldpd.conf
Configuration saved to /etc/frr/bgpd.conf
Configuration saved to /etc/frr/isisd.conf
Configuration saved to /etc/frr/pimd.conf
Configuration saved to /etc/frr/nhrpd.conf
Configuration saved to /etc/frr/eigrpd.conf
Configuration saved to /etc/frr/fabricd.conf
Configuration saved to /etc/frr/pbrd.conf
Configuration saved to /etc/frr/staticd.conf
Configuration saved to /etc/frr/bfdd.conf
</code></pre></div></div>

<p>Be sure to remove the combined configuration files.  All daemons check for the existence of this file at startup, and if 
it exists will not load their individual configuration files.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo rm</span> <span class="nt">-rf</span> /etc/frr/frr.conf /etc/frr/frr.conf.sav
</code></pre></div></div>

<h2 id="enable-eve-ng-telnet-access-to-frr-nodes">Enable Eve-NG Telnet access to FRR nodes</h2>
<p>You may prefer to connect to your new FRR instances using telnet instead of VNC.  This allows you to use client tools such
as Putty and SecureCRT.</p>

<p><strong>Warning !!! Use the following command INSIDE the FRR instance.  DO NOT execute this on your Eve-NG server or your workstation.</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo sed</span> <span class="nt">-i</span> <span class="s1">'s/GRUB_CMDLINE_LINUX=.*/GRUB_CMDLINE_LINUX="console=ttyS0,115200 console=tty0"/'</span> /etc/default/grub
<span class="nb">sudo </span>update-grub
</code></pre></div></div>

<h1 id="commit-changes-to-qemu-image">Commit changes to qemu image</h1>
<p>To preserve the installation and the default changes you’ve made you’ll need to commit them to the base qemu image before
creating new instances of FRR inside of Eve-NG.</p>

<p>First shutdown the FRR node from inside the VNC connection.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>shutdown <span class="nt">-h</span> now
</code></pre></div></div>

<p>The changes we have made up to this point are all stored in a temporary directory inside Eve-NG.  To locate the directory,
on the left side-bar within the EVE lab interface, choose <strong>Lab Details</strong> to get your lab’s UUID.</p>

<p><img src="/assets/images/20191105/lab_details.png" alt="Lab Details" title="Lab Details" /></p>

<p>Make note of the <strong>ID</strong>. This will be part of the directory path to your temporary image file.</p>

<p><img src="/assets/images/20191105/lab_details_2.png" alt="Lab Details" title="Lab Details" /></p>

<blockquote>
  <p>The UUID associated with my test lab is ID: 90d63733-6446-4437-a587-38050af45862</p>
</blockquote>

<p>Next, find the POD ID of your user and the Node ID of your newly installed node.  From the left side menu select <strong>Status</strong>.</p>

<p><img src="/assets/images/20191105/lab_status.png" alt="Lab Status" title="Lab Status" /></p>

<p>The Status page will show your current POD ID.</p>

<p><img src="/assets/images/20191105/lab_status_1.png" alt="Lab Status" title="Lab Status" /></p>

<blockquote>
  <p>Unless you have created multiple Eve-NG users the default Pod ID should be 0.</p>
</blockquote>

<p>To get the Node ID right click on the node you created in the lab.</p>

<p><img src="/assets/images/20191105/node_details.png" alt="Node Details" title="Node Details" /></p>

<blockquote>
  <p>If this is the first/only node you’ve created in this lab the ID should be 1.</p>
</blockquote>

<p>Using this information, connect to your Eve-NG server and locate the directory that contains the changes you made to the 
default image.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> /opt/unetlab/tmp/&lt;pod_id&gt;/&lt;lab_uuid&gt;/node_id/
</code></pre></div></div>

<p>Verify that you have found the correct image with;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>/opt/qemu/bin/qemu-img info virtioa.qcow2
image: virtioa.qcow2
file format: qcow2
virtual size: 16G <span class="o">(</span>17179869184 bytes<span class="o">)</span>
disk size: 3.3G
cluster_size: 65536
backing file: /opt/unetlab/addons/qemu/linux-frr-7.1/virtioa.qcow2
Format specific information:
    compat: 1.1
    lazy refcounts: <span class="nb">false
    </span>refcount bits: 16
    corrupt: <span class="nb">false</span>
</code></pre></div></div>

<blockquote>
  <p>You should see that the backing file points to the original qemu image we created.</p>
</blockquote>

<p>Now commit the changes back to the base image;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/opt/qemu/bin/qemu-img commit virtioa.qcow2
</code></pre></div></div>

<p>Once the command completes run the included Eve-NG script to check and fix file permissions.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/opt/unetlab/wrappers/unl_wrapper <span class="nt">-a</span> fixpermissions
</code></pre></div></div>

<h1 id="testing-new-frr-images">Testing new FRR images</h1>
<p>Now that we have saved our FRR image we can launch multiple instances and create additional labs to test FRR’s features.  I’ll
demonstrate a simple OSPF routing lab and point out additional settings/options you may wish to use while running FRR in
Eve-NG.</p>

<h2 id="add-nodes">Add nodes</h2>
<p>First we’ll update the existing node in the lab we’ve already created.  Make sure the node has been stopped, right click
on the node and select <strong>Edit</strong>.</p>

<p><img src="/assets/images/20191105/edit_node.png" alt="Edit Node" title="Edit Node" /></p>

<p>Change the nodes name to something similar to <code class="language-plaintext highlighter-rouge">frr-01</code>.  Update the CPU and RAM settings as needed and change the number
of <strong>Ethernets</strong> to <code class="language-plaintext highlighter-rouge">3</code>.</p>

<p><img src="/assets/images/20191105/edit_node_1.png" alt="Edit Node" title="Edit Node" /></p>

<p>If you prefer to use Telnet instead of VNC, change the <strong>Console</strong> type to <code class="language-plaintext highlighter-rouge">telnet</code>.  Click <strong>Save</strong> to update the node.</p>

<p><img src="/assets/images/20191105/edit_node_2.png" alt="Edit Node" title="Edit Node" /></p>

<p>Now add an additional FRR node by right clicking in the open area of the lab and selecting <strong>Node</strong> as shown in
<a href="#create-eve-ng-lab">previous steps</a>.</p>

<p><img src="/assets/images/20191105/add_2nd_node.png" alt="Add Node" title="Add Node" /></p>

<p>Connect the second node to the existing cloud network using its <code class="language-plaintext highlighter-rouge">e0</code> interface.</p>

<p><img src="/assets/images/20191105/connect_2nd_node.png" alt="Connect Node" title="Connect Node" /></p>

<p>Connect the two FRR instances together by dragging the connection icon from one instance to the other.</p>

<p><img src="/assets/images/20191105/connect_2nd_node_1.png" alt="Connect Node" title="Connect Node" /></p>

<p>In the <strong>Add Connection</strong> window choose interface <code class="language-plaintext highlighter-rouge">e1</code> for both FRR instances.</p>

<p><img src="/assets/images/20191105/connect_2nd_node_2.png" alt="Connect Node" title="Connect Node" /></p>

<p>Now we’ll add two <a href="https://sourceforge.net/projects/vpcs/">Virtual PC Simulator</a> (VPCS) instances to act as end hosts connected to our FRR OSPF network.  Right
click on the lab and select <strong>Node</strong> again.</p>

<p><img src="/assets/images/20191105/add_vpcs.png" alt="Add VPCS" title="Add VPCS" /></p>

<p>Scroll to the bottom of the list and select <strong>Virtual PC (VPCS)</strong>.</p>

<p><img src="/assets/images/20191105/add_vpcs_1.png" alt="Add VPCS" title="Add VPCS" /></p>

<p>In the <strong>Add Node</strong> window, set the number of nodes to add to <code class="language-plaintext highlighter-rouge">2</code>, set a name/prefix like <code class="language-plaintext highlighter-rouge">VPC</code> and click <strong>Save</strong>.</p>

<p><img src="/assets/images/20191105/add_vpcs_2.png" alt="Add VPCS" title="Add VPCS" /></p>

<p>Now connect each VPCS to a respective instance of FRR.</p>

<p><img src="/assets/images/20191105/connect_vpcs.png" alt="Connect VPCS" title="Connect VPCS" /></p>

<p>Connect the VPCS <code class="language-plaintext highlighter-rouge">eth0</code> interface to the FRR <code class="language-plaintext highlighter-rouge">e2</code> interface.</p>

<p><img src="/assets/images/20191105/connect_vpcs_1.png" alt="Connect VPCS" title="Connect VPCS" /></p>

<p>At this point your lab should look similar to the following.</p>

<p><img src="/assets/images/20191105/lab_final.png" alt="Final Lab" title="Final Lab" /></p>

<p>To start all nodes simultaneously, select <strong>More Actions</strong> from the left side menu.</p>

<p><img src="/assets/images/20191105/start_all_nodes.png" alt="Start Nodes" title="Start Nodes" /></p>

<p>Now select <strong>Start all nodes</strong> from the actions menu.</p>

<p><img src="/assets/images/20191105/start_all_nodes_1.png" alt="Start Nodes" title="Start Nodes" /></p>

<p>Now connect to all nodes by click on each inside the lab.</p>

<blockquote>
  <p>All of my nodes are configured to use telnet and the Eve-NG client integration tools are configured to open my default 
terminal application.</p>
</blockquote>

<p><img src="/assets/images/20191105/connect_to_all_nodes.png" alt="Connect to Nodes" title="Connect to Nodes" /></p>

<h2 id="configure-ospf-routing-on-frr">Configure OSPF Routing on FRR</h2>
<p>Apply the following configurations to the FRR nodes in the lab.</p>

<p><strong>frr-01</strong></p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>vtysh

configure terminal
interface ens4
 no shutdown
 ip address 10.0.0.1/30
interface ens5
 no shutdown
 ip address 10.0.1.1/24
router ospf
 ospf router-id 1.1.1.1
 network 0.0.0.0/0 area 0.0.0.0
<span class="nb">exit
exit
</span>write memory
</code></pre></div></div>

<p><strong>frr-02</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>vtysh

configure terminal
interface ens4
 no shutdown
 ip address 10.0.0.2/30
interface ens5
 no shutdown
 ip address 10.0.2.1/24
router ospf
 ospf router-id 2.2.2.2
 network 0.0.0.0/0 area 0.0.0.0
<span class="nb">exit
exit
</span>write memory
</code></pre></div></div>

<h2 id="configure-vpcs-networking">Configure VPCS Networking</h2>
<p>Now add an IP and default gateway to each of the VPCS instances.</p>

<p><strong>vpcs1</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ip 10.0.1.10/24 10.0.1.1
</code></pre></div></div>

<p><strong>vpcs2</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ip 10.0.2.10/24 10.0.1.1
</code></pre></div></div>

<h2 id="verify-routing">Verify Routing</h2>
<p>Now verify OSPF routing and reachability between the two VPCS instances.</p>

<p><strong>frr-01</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>sho ip ospf interface ens4
ens4 is up
  ifindex 3, MTU 1500 bytes, BW 4294967295 Mbit &lt;UP,BROADCAST,RUNNING,MULTICAST&gt;
  Internet Address 10.0.0.1/30, Broadcast 10.0.0.3, Area 0.0.0.0
  MTU mismatch detection: enabled
  Router ID 1.1.1.1, Network Type BROADCAST, Cost: 1
  Transmit Delay is 1 sec, State Backup, Priority 1
  Backup Designated Router <span class="o">(</span>ID<span class="o">)</span> 1.1.1.1, Interface Address 10.0.0.1
  Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters
  Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5
    Hello due <span class="k">in </span>5.103s
  Neighbor Count is 1, Adjacent neighbor count is 1
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>sho ip ospf neighbor 

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
2.2.2.2           1 Full/DR           38.920s 10.1.0.65       ens3:10.1.0.64           0     0     0
2.2.2.2           1 Full/DR           38.922s 10.0.0.2        ens4:10.0.0.1            0     0     0
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>sho ip route ospf
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       <span class="o">&gt;</span> - selected route, <span class="k">*</span> - FIB route, q - queued route, r - rejected route

O   10.0.0.0/30 <span class="o">[</span>110/1] is directly connected, ens4, 00:02:08
O   10.0.1.0/24 <span class="o">[</span>110/1] is directly connected, ens5, 00:05:23
O&gt;<span class="k">*</span> 10.0.2.0/24 <span class="o">[</span>110/2] via 10.0.0.2, ens4, 00:01:58
  <span class="k">*</span>                     via 10.1.0.65, ens3, 00:01:58
O   10.1.0.0/24 <span class="o">[</span>110/1] is directly connected, ens3, 00:05:23
</code></pre></div></div>

<p><strong>frr-01</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>sho ip ospf interface ens4
ens4 is up
  ifindex 3, MTU 1500 bytes, BW 4294967295 Mbit &lt;UP,BROADCAST,RUNNING,MULTICAST&gt;
  Internet Address 10.0.0.2/30, Broadcast 10.0.0.3, Area 0.0.0.0
  MTU mismatch detection: enabled
  Router ID 2.2.2.2, Network Type BROADCAST, Cost: 1
  Transmit Delay is 1 sec, State DR, Priority 1
  Backup Designated Router <span class="o">(</span>ID<span class="o">)</span> 1.1.1.1, Interface Address 10.0.0.1
  Multicast group memberships: OSPFAllRouters OSPFDesignatedRouters
  Timer intervals configured, Hello 10s, Dead 40s, Wait 40s, Retransmit 5
    Hello due <span class="k">in </span>3.541s
  Neighbor Count is 1, Adjacent neighbor count is 1
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>sho ip ospf neighbor 

Neighbor ID     Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
1.1.1.1           1 Full/Backup       38.479s 10.1.0.64       ens3:10.1.0.65           0     0     0
1.1.1.1           1 Full/Backup       38.479s 10.0.0.1        ens4:10.0.0.2            0     0     0
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>sho ip route ospf
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       <span class="o">&gt;</span> - selected route, <span class="k">*</span> - FIB route, q - queued route, r - rejected route

O   10.0.0.0/30 <span class="o">[</span>110/1] is directly connected, ens4, 00:22:33
O&gt;<span class="k">*</span> 10.0.1.0/24 <span class="o">[</span>110/2] via 10.0.0.1, ens4, 00:22:23
  <span class="k">*</span>                     via 10.1.0.62, ens3, 00:22:23
O   10.0.2.0/24 <span class="o">[</span>110/1] is directly connected, ens5, 00:22:39
O   10.1.0.0/24 <span class="o">[</span>110/1] is directly connected, ens3, 00:22:39
</code></pre></div></div>

<p><strong>vpcs1</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VPCS&gt; ping 10.0.2.10   

84 bytes from 10.0.2.10 <span class="nv">icmp_seq</span><span class="o">=</span>1 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>3.041 ms
84 bytes from 10.0.2.10 <span class="nv">icmp_seq</span><span class="o">=</span>2 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>2.739 ms
84 bytes from 10.0.2.10 <span class="nv">icmp_seq</span><span class="o">=</span>3 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>4.734 ms
84 bytes from 10.0.2.10 <span class="nv">icmp_seq</span><span class="o">=</span>4 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>17.732 ms
84 bytes from 10.0.2.10 <span class="nv">icmp_seq</span><span class="o">=</span>5 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>2.129 ms
</code></pre></div></div>

<p><strong>vpcs2</strong></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>VPCS&gt; ping 10.0.1.10

84 bytes from 10.0.1.10 <span class="nv">icmp_seq</span><span class="o">=</span>1 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>6.083 ms
84 bytes from 10.0.1.10 <span class="nv">icmp_seq</span><span class="o">=</span>2 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>2.504 ms
84 bytes from 10.0.1.10 <span class="nv">icmp_seq</span><span class="o">=</span>3 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>8.699 ms
84 bytes from 10.0.1.10 <span class="nv">icmp_seq</span><span class="o">=</span>4 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>2.528 ms
84 bytes from 10.0.1.10 <span class="nv">icmp_seq</span><span class="o">=</span>5 <span class="nv">ttl</span><span class="o">=</span>62 <span class="nb">time</span><span class="o">=</span>4.602 ms
</code></pre></div></div>

<h1 id="conclusion">Conclusion</h1>
<p>Now that you have tested the basic functionality of your new FRR image you can proceed to create more complex labs and test
additional features of FRR.  Whether you’re new to open source routing or you have plans to write your own <a href="https://en.wikipedia.org/wiki/Network_operating_system">Network
Operating System</a> (NOS) based on FRR, I recommend you check out 
the <a href="https://frrouting.org/#participate">participate</a> section of the FRR website to find out how you can become more 
involved with the project.</p>]]></content><author><name>2stacks</name></author><category term="Blog" /><category term="FRRouting" /><category term="Eve-NG" /><category term="Open Source" /><summary type="html"><![CDATA[Installing and Running FRR on Eve-NG]]></summary></entry><entry><title type="html">Bare Metal to Kubernetes-as-a-Service - Part 3</title><link href="/blog/bare-metal-to-kubernetes-part-3/" rel="alternate" type="text/html" title="Bare Metal to Kubernetes-as-a-Service - Part 3" /><published>2019-10-08T00:00:00+00:00</published><updated>2019-10-08T00:00:00+00:00</updated><id>/blog/bare-metal-to-kubernetes-part-3</id><content type="html" xml:base="/blog/bare-metal-to-kubernetes-part-3/"><![CDATA[<h1 id="introduction">Introduction</h1>
<p>I started this series of posts with the assertion that, although a lot of effort has gone in to making Kubernetes easier
to deploy and operate, Kubernetes is still hard.  Simply showing how to deploy a Kubernetes cluster is not the 
aim.  There are thousands of how-to’s on the web covering the basics.  The ultimate point of these posts is to demonstrate 
at least one viable option for deploying a capability similar to a public cloud Kubernetes-as-a-service solution.  This 
means having the ability to provision multiple, production-ready clusters complete with features such as <a href="https://en.wikipedia.org/wiki/Multitenancy">multitenancy</a>, 
role based access control (<a href="https://en.wikipedia.org/wiki/Role-based_access_control">RBAC</a>), <a href="https://istio.io/docs/concepts/what-is-istio/#what-is-a-service-mesh">service mesh</a>, <a href="https://en.wikipedia.org/wiki/CI/CD">CI/CD</a> etc.  When you hear “Kubernetes is hard” it’s not usually in 
reference to simply setting up a functioning cluster.  It is usually a commentary on the challenges associated with these 
additional features which are necessary for running Kubernetes in production.</p>

<h1 id="objective">Objective</h1>
<p>I mentioned in <a href="/blog/bare-metal-to-kubernetes-part-1/">Part 1</a> of this series some of the available 
solutions for deploying Kubernetes in private clouds.  After some extensive research and experimentation with a few of 
the more popular solutions I decided to go with <a href="https://rancher.com/">Rancher’s</a> Kubernetes management platform.  This is not an endorsement of 
one product over another and I don’t intend to do a side-by-side comparison of all of the platforms I tested.  For me,
Rancher simply shortened the steep learning curve associated with Kubernetes and met or exceeded all of the requirements 
I had for running production clusters with limited experience.</p>

<p>In this final post I’m going to demonstrate how to integrate Rancher with Openstack.  Integrating these two technologies 
simplifies the deployment and management of production ready Kubernetes clusters.  Rancher abstracts and automates the 
process of creating the underlying infrastructure and provides a unified management platform for running Kubernetes at 
scale on any cloud, public or private.  Once completing this demonstration we’ll have a self-service platform from which 
to create and manage multiple Kubernetes clusters with the features necessary for production environments.</p>

<h1 id="prerequisites">Prerequisites</h1>
<p>In <a href="/blog/bare-metal-to-kubernetes-part-1/">Part 1</a> of this series we deployed <a href="https://maas.io/">MAAS</a> to automate the
provisioning and management of our physical compute resources.  In <a href="/blog/bare-metal-to-kubernetes-part-2/">Part 2</a>
we deployed <a href="https://www.openstack.org/">Openstack</a> as our Infrastructure-as-a-Service (IaaS) platform for managing virtual machines, networks and
storage.  These prerequisites provide the private cloud infrastructure on which Rancher will automate self-service Kubernetes
deployments.  If you’ve been following along up till now you should already have everything you need to complete this final
demonstration.</p>

<h1 id="install-rancher">Install Rancher</h1>
<p>Rancher was designed to be <a href="https://opensource.com/article/18/7/what-are-cloud-native-apps">cloud-native</a> and is intended to be run as a distributed, highly available (HA) application on top of
Kubernetes.  That said, getting started with Rancher is as simple as launching a single <a href="https://techcrunch.com/2016/10/16/wtf-is-a-container/">container</a>.  For the purposes of this demonstration I’ll be using a 
single node deployment.  For additional information on running Rancher in HA please reference the <a href="https://rancher.com/docs/rancher/v2.x/en/">Rancher Documentation</a>.</p>

<h2 id="create-cloud-init-file">Create Cloud-init file</h2>
<p>To automate the creation and installation of the Rancher server we’ll use a <a href="https://cloudinit.readthedocs.io/en/latest/">cloud-init</a> file with the Openstack API.</p>

<blockquote>
  <p>Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialization. It is 
supported across all major public cloud providers, provisioning systems for private cloud infrastructure, and bare-metal 
installations.</p>
</blockquote>

<p>First we’ll need to retrieve the SSH public key from the keypair that was created in <a href="https://www.2stacks.net/blog/bare-metal-to-kubernetes-part-2/#add-ssh-keys">Part 2</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack keypair show <span class="nt">--public-key</span> os_rsa
</code></pre></div></div>

<p>Next using your preferred text editor create a file named <code class="language-plaintext highlighter-rouge">cloud-init</code> with the following contents.  Be sure to replace
<code class="language-plaintext highlighter-rouge">&lt;your public key&gt;</code> with the public key retrieved in the previous step.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">#cloud-config</span>
<span class="nb">groups</span>:
  - ubuntu
  - docker
<span class="nb">users</span>:
  - name: ubuntu
    shell: /bin/bash
    <span class="nb">sudo</span>: <span class="o">[</span><span class="s1">'ALL=(ALL) NOPASSWD:ALL'</span><span class="o">]</span>
    primary-group: ubuntu
    <span class="nb">groups</span>: <span class="nb">sudo</span>, docker
    lock_passwd: <span class="nb">true
    </span>ssh-authorized-keys:
      - ssh-rsa &lt;your public key&gt;
apt:
  sources:
    docker:
      keyid: <span class="s2">"0EBFCD88"</span>
      <span class="nb">source</span>: <span class="s2">"deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"</span> 
package_upgrade: <span class="nb">true
</span>packages: 
  - apt-transport-https 
  - docker-ce
runcmd:
  - <span class="s2">"docker run -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher"</span>
</code></pre></div></div>
<p>This cloud-init file will instruct the Ubuntu virtual machine, that we’ll launch in the next step, to install Docker, then
pull and run the Rancher server container.</p>

<h2 id="create-rancher-server">Create Rancher Server</h2>
<p>You should have already created the required Openstack flavors, images and networks needed to launch the server.  To launch
the server with the custom cloud-init file run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack server create <span class="nt">--image</span> bionic <span class="nt">--flavor</span> m1.medium <span class="se">\</span>
    <span class="nt">--nic</span> net-id<span class="o">=</span><span class="si">$(</span>openstack network list | <span class="nb">grep </span>int_net | <span class="nb">awk</span> <span class="s1">'{ print $2 }'</span><span class="si">)</span> <span class="se">\</span>
    <span class="nt">--user-data</span> cloud-init rancher
</code></pre></div></div>

<h2 id="add-floating-ip">Add Floating IP</h2>
<p>To enable SSH access from external hosts we’ll need to assign the instance a floating ip in the external subnet.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">floating_ip</span><span class="o">=</span><span class="si">$(</span>openstack floating ip create <span class="nt">-f</span> value <span class="nt">-c</span> floating_ip_address ext_net<span class="si">)</span>
openstack server add floating ip rancher <span class="k">${</span><span class="nv">floating_ip</span><span class="k">}</span>
</code></pre></div></div>

<h1 id="security-groups">Security Groups</h1>
<p>Kubernetes has many port and protocol requirements for communicating with cluster nodes.  There are 
several different ways to implement these requirements depending upon the type of public or private cloud you are deploying 
to.  Make sure you review these <a href="https://rancher.com/docs/rancher/v2.x/en/installation/requirements/">port and protocol</a> requirements before running Rancher in production.  For demonstration 
purposes I’m going to update the existing default security group in Openstack to allow all traffic.  It should go without 
saying that this is not recommended outside of a lab environment.</p>

<p>To update the default security group run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">project_id</span><span class="o">=</span><span class="si">$(</span>openstack project show <span class="nt">--domain</span> admin_domain admin <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span><span class="si">)</span>
<span class="nv">secgroup_id</span><span class="o">=</span><span class="si">$(</span>openstack security group list <span class="nt">--project</span> <span class="k">${</span><span class="nv">project_id</span><span class="k">}</span> | <span class="nb">awk</span> <span class="s1">'/default/ {print $2}'</span><span class="si">)</span>
openstack security group rule create <span class="nv">$secgroup_id</span> <span class="nt">--protocol</span> any <span class="nt">--ethertype</span> IPv6 <span class="nt">--ingress</span>
openstack security group rule create <span class="nv">$secgroup_id</span> <span class="nt">--protocol</span> any <span class="nt">--ethertype</span> IPv4 <span class="nt">--ingress</span>
</code></pre></div></div>

<h1 id="connect-to-rancher-ui">Connect to Rancher UI</h1>
<p>Our Rancher server should now be remotely reachable via the floating IP address we added earlier.  To get the IP run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack server show rancher <span class="nt">-f</span> value <span class="nt">-c</span> addresses
</code></pre></div></div>

<p>Now open a web browser and launch the Rancher User Interface (UI) at <code class="language-plaintext highlighter-rouge">https://&lt;floating_ip&gt;</code>.</p>

<p><img src="/assets/images/20191008/rancher_ui.png" alt="Rancher UI" title="Rancher UI" /></p>

<h1 id="configure-rancher">Configure Rancher</h1>
<p>Upon connecting to the Rancher UI you’ll be prompted to set a new password for the default <strong>admin</strong> user.</p>

<p>After selecting <strong>Continue</strong> you’ll be prompted to set the Rancher server URL.  It’s OK to leave the default for now but 
typically this would be the fully qualified domain name of the server (<a href="https://en.wikipedia.org/wiki/Fully_qualified_domain_name">FQDN</a>).</p>

<blockquote>
  <p>Note: all cluster nodes will need to be able to reach this URL and it can be changed later from within the UI.</p>
</blockquote>

<p><img src="/assets/images/20191008/rancher_url.png" alt="Rancher URL" title="Rancher URL" /></p>

<p>After selecting <strong>Save URL</strong> you’ll be logged in to the default <strong>Clusters</strong> view of the UI.  Since no clusters have been
added, only an <strong>Add Cluster</strong> button is visible.</p>

<h2 id="enable-openstack-node-driver">Enable Openstack Node Driver</h2>
<p>Before adding our first cluster we need to configure settings that allow Rancher to automate the provisioning of Openstack 
resources.  The first step is to enable the Openstack <a href="https://rancher.com/docs/rancher/v2.x/en/admin-settings/drivers/node-drivers/">Node Driver</a>.</p>

<blockquote>
  <p>Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters.</p>
</blockquote>

<p>To enable the Openstack Node Driver navigate to <strong>Tools</strong> then <strong>Drivers</strong> in the UI navigation.</p>

<p><img src="/assets/images/20191008/node_driver_1.png" alt="Enable Node Driver" title="Enable Node Driver" /></p>

<p>On the Drivers page, click the tab for <strong>Node Drivers</strong>, select the check box next to <strong>OpenStack</strong> and then click the <strong>Activate</strong>
button.</p>

<p><img src="/assets/images/20191008/node_driver_2.png" alt="Enable Node Driver" title="Enable Node Driver" /></p>

<h2 id="add-openstack-node-template">Add Openstack Node Template</h2>
<p>Once we’ve enabled the Openstack Node Driver we can create a <a href="https://rancher.com/docs/rancher/v2.x/en/cluster-provisioning/rke-clusters/node-pools/#node-templates">Node Template</a> that will specify the settings used to create
nodes/hosts within Openstack.</p>

<p>To create the node template click the drop down in the top right of the UI next to the admin user’s avatar and select 
<strong>Node Templates</strong>.</p>

<p><img src="/assets/images/20191008/node_template_1.png" alt="Enable Node Template" title="Enable Node Template" /></p>

<p>Next click the <strong>Add Template</strong> button.</p>

<p><img src="/assets/images/20191008/node_template_4.png" alt="Enable Node Template" title="Enable Node Template" /></p>

<p>On the <strong>Add Node Template</strong> page first select the option for Openstack.  Before filling out the template data we’ll need to
retrieve our settings from the Openstack API.</p>

<p><img src="/assets/images/20191008/node_template_2.png" alt="Enable Node Template" title="Enable Node Template" /></p>

<h3 id="obtain-template-settings">Obtain Template Settings</h3>
<p>The script for obtaining the default API credentials was included in the Openstack bundle we downloaded in 
<a href="https://www.2stacks.net/blog/bare-metal-to-kubernetes-part-2/#configure-openstack-bundle">Part 2</a>.  If they are not already
loaded in your environment change to the <strong>openstack-base</strong> directory and run the following.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span><span class="nb">source </span>openrc
~<span class="nv">$ </span><span class="nb">env</span> | <span class="nb">grep </span>OS_

<span class="nv">OS_USER_DOMAIN_NAME</span><span class="o">=</span>admin_domain
<span class="nv">OS_PASSWORD</span><span class="o">=</span>Uafe7chee6eigedi
<span class="nv">OS_AUTH_URL</span><span class="o">=</span>http://10.1.20.32:5000/v3
<span class="nv">OS_USERNAME</span><span class="o">=</span>admin
</code></pre></div></div>
<blockquote>
  <p>Only the settings needed for the node template are shown above.</p>
</blockquote>

<p>After loading the API credentials in your environment, you’ll need to obtain the default <strong>domain</strong> and <strong>tenant</strong> id from the 
Openstack API.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack project show admin <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span>
</code></pre></div></div>

<p>Once you have these settings you should have all of the information needed to fill out the node template.</p>

<blockquote>
  <p>Several of the settings shown below were created in <a href="https://www.2stacks.net/blog/bare-metal-to-kubernetes-part-2/#validate-deployment">Part 2</a>
in order to validate the Openstack installation.  Some of these settings are shown with the values specific to my environment.</p>
</blockquote>

<table>
  <thead>
    <tr>
      <th>Option</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>authUrl</strong></td>
      <td>http://10.1.20.32:5000/v3</td>
    </tr>
    <tr>
      <td><strong>domainName</strong></td>
      <td>admin_domain</td>
    </tr>
    <tr>
      <td><strong>flavorName</strong></td>
      <td>m1.large</td>
    </tr>
    <tr>
      <td><strong>floatingipPool</strong></td>
      <td>ext_net</td>
    </tr>
    <tr>
      <td><strong>imageName</strong></td>
      <td>bionic</td>
    </tr>
    <tr>
      <td><strong>insecure</strong></td>
      <td>enable</td>
    </tr>
    <tr>
      <td><strong>keypairName</strong></td>
      <td>os_rsa</td>
    </tr>
    <tr>
      <td><strong>netName</strong></td>
      <td>int_net</td>
    </tr>
    <tr>
      <td><strong>password</strong></td>
      <td>Uafe7chee6eigedi</td>
    </tr>
    <tr>
      <td><strong>privateKeyFile</strong></td>
      <td>&lt;contents of ~/.ssh/os_rsa&gt;</td>
    </tr>
    <tr>
      <td><strong>secGroups</strong></td>
      <td>default</td>
    </tr>
    <tr>
      <td><strong>sshUser</strong></td>
      <td>ubuntu</td>
    </tr>
    <tr>
      <td><strong>tenantid</strong></td>
      <td>ca52ba242b104464b15811d4589f836e</td>
    </tr>
    <tr>
      <td><strong>username</strong></td>
      <td>admin</td>
    </tr>
  </tbody>
</table>

<blockquote>
  <p>Be sure to use tenantId instead of tenantName. Using tenantName will cause docker-machine to use the wrong Openstack API
version.</p>
</blockquote>

<p>Once you’ve added all of your relevant settings to the template click <strong>Create</strong> to save the node template.</p>

<p><img src="/assets/images/20191008/node_template_3.png" alt="Enable Node Template" title="Enable Node Template" /></p>

<h1 id="add-cluster">Add Cluster</h1>
<p>Now that we’ve configured the settings Rancher will use to create nodes within Openstack, we can add our first Kubernetes
cluster.</p>

<p>Click on <strong>Custers</strong> in the UI navigation and then click the <strong>Add Cluster</strong> button.</p>

<p><img src="/assets/images/20191008/add_cluster_1.png" alt="Add Cluster" title="Add Cluster" /></p>

<p>On the <strong>Add Cluster</strong> page select <strong>Openstack</strong> from the list of infrastructure providers.</p>

<p><img src="/assets/images/20191008/add_cluster_2.png" alt="Add Cluster" title="Add Cluster" /></p>

<p>Continue configuring the cluster with the following settings;</p>

<table>
  <thead>
    <tr>
      <th>Option</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Cluster Name</strong></td>
      <td>os1</td>
    </tr>
    <tr>
      <td><strong>Name Prefix</strong></td>
      <td>os1-all-</td>
    </tr>
    <tr>
      <td><strong>Count</strong></td>
      <td>3</td>
    </tr>
    <tr>
      <td><strong>Template</strong></td>
      <td>Openstack</td>
    </tr>
  </tbody>
</table>

<p>For demonstration purposes select the check boxes next to <a href="https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/">etcd</a>, <a href="https://kubernetes.io/docs/concepts/#kubernetes-control-plane">control plane</a> and <a href="https://kubernetes.io/docs/concepts/architecture/nodes/">worker</a>.</p>

<p><img src="/assets/images/20191008/add_cluster_3.png" alt="Add Cluster" title="Add Cluster" /></p>

<p>These setting will create a three node cluster named <strong>os1</strong> using the previously created Openstack node template.  Each 
node will be configured to run the Kubernetes etcd, control plane and worker services.</p>

<blockquote>
  <p>Note: it is recommended to separate the worker nodes from other services in production clusters.</p>
</blockquote>

<h2 id="configure-the-cloud-provider">Configure the Cloud Provider</h2>
<p>Before launching the cluster you’ll need to configure the Kubernetes <a href="https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack">Cloud Provider</a> for Openstack.</p>

<blockquote>
  <p>Cloud Providers enable a set of functionality that integrate with the underlying infrastructure provider, a.k.a the cloud 
provider. Enabling this integration unlocks a wide set of features for your clusters such as: node address &amp; zone 
discovery, cloud load balancers for Services with Type=LoadBalancer, IP address management, and cluster networking via 
VPC routing tables.</p>
</blockquote>

<p>Under the <strong>Cloud Provider</strong> section select the <strong>Custom</strong> option.  A warning message will appear that states that the 
prerequisites of the underlying cloud provider must be met before enabling this option.</p>

<p><img src="/assets/images/20191008/cloud_provider_1.png" alt="Cloud Provider" title="Cloud Provider" /></p>

<blockquote>
  <p>All of the prerequisite configurations for Openstack were applied in <a href="/blog/bare-metal-to-kubernetes-part-2/">Part 2</a> of this series.</p>
</blockquote>

<p>Also notice the message instructing you to edit the cluster’s <a href="https://yaml.org/">YAML</a> configuration in order to enable the custom
cloud provider.  To begin editing the cluster configuration click the <strong>Edit as YAML</strong> link at the top of the <strong>Cluster
Options</strong> section.</p>

<p><img src="/assets/images/20191008/cloud_provider_2.png" alt="Cloud Provider" title="Cloud Provider" /></p>

<h3 id="obtain-cloud-provider-settings">Obtain Cloud Provider Settings</h3>
<p>To configure the Openstack cloud provider we need to add a section of YAML to the top of the configuration.  The YAML
consists of settings similar to those used in the node template.</p>

<p>First we’ll need the Openstack API credentials.  As shown <a href="#obtain-template-settings">above</a> they can be obtained by running.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span><span class="nb">source </span>openrc
~<span class="nv">$ </span><span class="nb">env</span> | <span class="nb">grep </span>OS_

<span class="nv">OS_PASSWORD</span><span class="o">=</span>Uafe7chee6eigedi
<span class="nv">OS_AUTH_URL</span><span class="o">=</span>http://10.1.20.32:5000/v3
<span class="nv">OS_USERNAME</span><span class="o">=</span>admin
</code></pre></div></div>

<p>The remaining configuration settings can be obtained from the Openstack API.  Run the following commands and record their
output.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># tenant-id</span>
openstack project show admin <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span>

<span class="c"># domain-id</span>
openstack project show admin <span class="nt">-f</span> value <span class="nt">-c</span> domain_id

<span class="c"># subnet-id</span>
openstack subnet show int_subnet <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span>

<span class="c"># floating-network-id</span>
openstack network show ext_net <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span>

<span class="c"># router-id</span>
openstack router show rtr-01 <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span>
</code></pre></div></div>

<p>Now that we have the needed credentials and settings we can add them to a YAML formatted block of text as shown below.</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">cloud_provider</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">openstack</span>
  <span class="na">openstackCloudProvider</span><span class="pi">:</span>
    <span class="na">global</span><span class="pi">:</span> 
      <span class="na">username</span><span class="pi">:</span> <span class="s">admin</span>
      <span class="na">password</span><span class="pi">:</span> <span class="s">Uafe7chee6eigedi</span>
      <span class="na">auth-url</span><span class="pi">:</span> <span class="s2">"</span><span class="s">http://10.1.20.32:5000/v3"</span>
      <span class="na">tenant-id</span><span class="pi">:</span> <span class="s">ca52ba242b104464b15811d4589f836e</span>
      <span class="na">domain-id</span><span class="pi">:</span> <span class="s">3f7614e901ef4adc8e14014ae9a5d8e6</span>
    <span class="na">load_balancer</span><span class="pi">:</span>
      <span class="na">use-octavia</span><span class="pi">:</span> <span class="no">true</span>
      <span class="na">subnet-id</span><span class="pi">:</span> <span class="s">a114a47d-e7d5-4e98-8a5a-26b2a6692017</span>
      <span class="na">floating-network-id</span><span class="pi">:</span> <span class="s">e499e0b6-fa1e-40b0-abcd-3f9299fd1094</span>
    <span class="na">route</span><span class="pi">:</span>
      <span class="na">router-id</span><span class="pi">:</span> <span class="s">498ce328-6ba2-4a09-b421-71e28028b4fa</span>
</code></pre></div></div>

<p>Paste this configuration (substituting your values) in to the top of the Cloud Provider YAML configuration as shown below.</p>

<p><img src="/assets/images/20191008/cloud_provider_3.png" alt="Cloud Provider" title="Cloud Provider" /></p>

<p>Now click the <strong>Create</strong> button to launch the Kubernetes cluster.</p>

<blockquote>
  <p>For more information on configuring the cloud provider please see the Kubernetes <a href="https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#openstack">Cloud Provider</a> or the <a href="https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/openstack/">Rancher</a> documentation.</p>
</blockquote>

<h2 id="validate-cluster">Validate Cluster</h2>
<p>As the cluster is deploying we can monitor its progress and status in the Rancher UI by navigating to the <strong>Clusters</strong> page
and selecting the link to <strong>os1</strong> under the <strong>Cluster Name</strong> list.  From there click on <strong>Nodes</strong> in the UI navigation.</p>

<p><img src="/assets/images/20191008/validate_cluster_1.png" alt="Validate Cluster" title="Validate Cluster" /></p>

<p>From here status and error messages will be displayed as the cluster is being deployed.</p>

<h3 id="rancher-logs">Rancher Logs</h3>
<p>You can also monitor the cluster build process by connecting to the Rancher instance with SSH.  This is useful for debugging 
any error messages produced by Rancher.</p>

<p>First SSH to the Rancher server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ssh <span class="nt">-i</span> ~/.ssh/os_rsa ubuntu@&lt;rancher_ip&gt;
</code></pre></div></div>

<p>Next get the rancher Container ID.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ubuntu@rancher:~<span class="nv">$ </span>docker ps
CONTAINER ID        IMAGE                    COMMAND             CREATED             STATUS              PORTS                                      NAMES
21792badbb1c        rancher/rancher:v2.2.8   <span class="s2">"entrypoint.sh"</span>     19 hours ago        Up 19 hours         0.0.0.0:80-&gt;80/tcp, 0.0.0.0:443-&gt;443/tcp   upbeat_chaplygin
</code></pre></div></div>

<p>Run the following to watch the logs of the rancher container.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker logs <span class="nt">-f</span> 21792badbb1c
</code></pre></div></div>

<h3 id="openstack-api">Openstack API</h3>
<p>You can validate the creation of each cluster node and its associated settings from the Openstack API.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>openstack server list
+--------------------------------------+-----------+--------+------------------------------------+--------+-----------+
| ID                                   | Name      | Status | Networks                           | Image  | Flavor    |
+--------------------------------------+-----------+--------+------------------------------------+--------+-----------+
| 6a54ade8-1c93-486e-99d9-183b401131a7 | os1-all-2 | ACTIVE | <span class="nv">int_net</span><span class="o">=</span>10.0.0.127, 192.168.10.229 | bionic | m1.large  |
| be07abea-fab0-4ffd-8098-0e6c4b69a86b | os1-all-1 | ACTIVE | <span class="nv">int_net</span><span class="o">=</span>10.0.0.41, 192.168.10.180  | bionic | m1.large  |
| 8a281b10-2ab9-4573-b1bc-62910622fb46 | os1-all-3 | ACTIVE | <span class="nv">int_net</span><span class="o">=</span>10.0.0.81, 192.168.10.204  | bionic | m1.large  |
| 982c2885-55fb-4a5b-a9ee-8bd4b9d2a771 | rancher   | ACTIVE | <span class="nv">int_net</span><span class="o">=</span>10.0.0.238, 192.168.10.234 | bionic | m1.medium |
+--------------------------------------+-----------+--------+------------------------------------+--------+-----------+
</code></pre></div></div>

<p>Make sure that the status of all hosts created in Openstack is <strong>ACTIVE</strong> and that each is associated with a floating IP 
address.</p>

<h3 id="rancher-ui">Rancher UI</h3>
<p>Once the cluster nodes have been deployed and Rancher has completed the Kubernetes installation process the <strong>State</strong> of 
each node in the Rancher UI should change to <strong>Active</strong>.</p>

<p><img src="/assets/images/20191008/validate_cluster_2.png" alt="Validate Cluster" title="Validate Cluster" /></p>

<h1 id="deploy-application">Deploy Application</h1>
<p>Now that the cluster has been deployed we can validate its functionality by deploying a simple test application.</p>

<p>First navigate to the cluster’s default <a href="https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/">namespace</a> by selecting the cluster name <strong>os1</strong> in the top left of the navigation 
and then selecting <strong>Default</strong>.</p>

<p><img src="/assets/images/20191008/deploy_app_1.png" alt="Deploy App" title="Deploy App" /></p>

<p>From the <a href="https://rancher.com/docs/rancher/v2.x/en/k8s-in-rancher/workloads/">Workloads</a> view click the <strong>Deploy</strong> button to configure our test application.</p>

<p><img src="/assets/images/20191008/deploy_app_2.png" alt="Deploy App" title="Deploy App" /></p>

<h2 id="configure-workload">Configure Workload</h2>
<p>First set the following name, type and image for the workload.</p>

<table>
  <thead>
    <tr>
      <th>Option</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Name</strong></td>
      <td>webapp</td>
    </tr>
    <tr>
      <td><strong>Workload Type</strong></td>
      <td>3 pods</td>
    </tr>
    <tr>
      <td><strong>Docker Image</strong></td>
      <td><a href="https://github.com/leodotcloud/swiss-army-knife">leodotcloud/swiss-army-knife</a></td>
    </tr>
  </tbody>
</table>

<p><img src="/assets/images/20191008/deploy_app_3.png" alt="Deploy App" title="Deploy App" /></p>

<p>Next click the <strong>Add Port</strong> button and configure the port, protocol and <a href="https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types">service type</a>.</p>

<table>
  <thead>
    <tr>
      <th>Option</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Container Port</strong></td>
      <td>80</td>
    </tr>
    <tr>
      <td><strong>Protocol</strong></td>
      <td>TCP</td>
    </tr>
    <tr>
      <td><strong>Service Type</strong></td>
      <td>Layer-4 Load Balancer</td>
    </tr>
    <tr>
      <td><strong>Listening Port</strong></td>
      <td>80</td>
    </tr>
  </tbody>
</table>

<p>Next add an environment variable to be used by the application by expanding the <strong>Environment Variables</strong> section and 
clicking the <strong>Add Variable</strong> button.</p>

<p>Add the following environment variable then click the <strong>Launch</strong> button to deploy the application.</p>

<table>
  <thead>
    <tr>
      <th>Variable</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>ALPHABET</strong></td>
      <td>A</td>
    </tr>
  </tbody>
</table>

<p><img src="/assets/images/20191008/deploy_app_4.png" alt="Deploy App" title="Deploy App" /></p>

<blockquote>
  <p>Adding the environment variable is optional but is useful for testing additional Kubernetes functionality.</p>
</blockquote>

<h1 id="validate-application">Validate Application</h1>
<p>The workload we created will launch three <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">pods</a> each with a single instance of the <a href="https://github.com/leodotcloud/swiss-army-knife">leodotcloud/swiss-army-knife</a>
container.  This container contains various tools and a simple web server that can be used to test your Kubernetes 
environment.</p>

<h2 id="view-application-status">View Application Status</h2>
<p>While the workload is is being deployed we can monitor its status and progress from the <strong>Workloads</strong> view in Rancher.</p>

<p><img src="/assets/images/20191008/validate_app_1.png" alt="Validate App" title="Validate App" /></p>

<p>You can change the default view to <strong>Group By Node</strong> to see the placement of each Pod on the nodes in our Kubernetes 
cluster.</p>

<p><img src="/assets/images/20191008/validate_app_2.png" alt="Validate App" title="Validate App" /></p>

<h2 id="view-load-balancer-status">View Load Balancer Status</h2>
<p>Since we chose a service type of <strong>Layer-4 Load Balancer</strong> Rancher will use the previously configured cloud provider to 
create an instance of the Openstack Octavia load balancer.  It will further configure all of the necessary settings for the
load balancer to direct traffic to our new application.</p>

<p>You can view the deployment status and configuration of the load balancer from the <strong>Load Balancing</strong> view of the Rancher UI.</p>

<p><img src="/assets/images/20191008/validate_lb_1.png" alt="Validate LB" title="Validate LB" /></p>

<blockquote>
  <p>It can take some time for Openstack to create the Octavia virtual machine instance so be patient.</p>
</blockquote>

<p>You can validate from the Openstack API that the load balancer instance was created successful by running the following command.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>openstack loadbalancer list                             
+--------------------------------------+----------------------------------+----------------------------------+-------------+---------------------+----------+
| <span class="nb">id</span>                                   | name                             | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+----------------------------------+----------------------------------+-------------+---------------------+----------+
| 1301cf7b-9319-4222-a0a0-bcf42fe6d2f0 | ad023b4dbda2e11e98886fa163ea07cc | ca52ba242b104464b15811d4589f836e | 10.0.0.50   | ACTIVE              | amphora  |
+--------------------------------------+----------------------------------+----------------------------------+-------------+---------------------+----------+
</code></pre></div></div>

<p>The <strong>provisioning_status</strong> in the Openstack API should eventually change to <strong>ACTIVE</strong>.  You can further validate that the
<strong>State</strong> shown in the Rancher UI has changed from <strong>Pending</strong> to <strong>Active</strong>.</p>

<p><img src="/assets/images/20191008/validate_lb_2.png" alt="Validate LB" title="Validate LB" /></p>

<h2 id="test-application">Test Application</h2>
<p>Now that the workload and it’s load balancer have been deployed we can connect to the web application included in the 
swiss-army-knife container.</p>

<p>From the <strong>Workloads</strong> view of the Rancher UI click the <strong>80/tcp</strong> link below the application’s name.  This should launch 
a web browser and connect you to the floating IP address that was assigned to the load balancer.</p>

<p><img src="/assets/images/20191008/test_app_1.png" alt="Test App" title="Test App" /></p>

<p>The browser window should display the floating IP of the load balancer in the address bar. The page should show the hostname 
and IP address of the container instance that the load balancer forwarded you to.</p>

<p><img src="/assets/images/20191008/test_app_2.png" alt="Test App" title="Test App" /></p>

<p>If you refresh the browser periodically you should see that the load balancer forwards you to each of the three containers
that were launched as part of the workload.</p>

<p><img src="/assets/images/20191008/test_app_3.png" alt="Test App" title="Test App" /></p>

<h1 id="conclusion">Conclusion</h1>
<p>Once you’ve successfully completed your validation you should have the basic functionality needed for delivering Kubernetes
as a service.  Though this demonstration only walked you through the creation of a single cluster it is trivial to create
additional clusters by reproducing these steps.  By leveraging additional features in Rancher, creating additional host and
cloud templates and by utilizing the Rancher API you can streamline your process for abstracting the creation of these clusters.
Additional efficiency can be added by using DevOps tools such as HashiCorp’s <a href="https://www.hashicorp.com/products/terraform/">Terraform</a> to fully automate Rancher configurations
and deployments.</p>]]></content><author><name>2stacks</name></author><category term="Blog" /><category term="IaaS" /><category term="Multi-Cloud" /><category term="MaaS" /><category term="Juju" /><category term="OpenStack" /><category term="Kubernetes" /><category term="Rancher" /><summary type="html"><![CDATA[Using Rancher to deploy self-service Kubernetes clusters.]]></summary></entry><entry><title type="html">Bare Metal to Kubernetes-as-a-Service - Part 2</title><link href="/blog/bare-metal-to-kubernetes-part-2/" rel="alternate" type="text/html" title="Bare Metal to Kubernetes-as-a-Service - Part 2" /><published>2019-09-06T00:00:00+00:00</published><updated>2019-09-06T00:00:00+00:00</updated><id>/blog/bare-metal-to-kubernetes-part-2</id><content type="html" xml:base="/blog/bare-metal-to-kubernetes-part-2/"><![CDATA[<h1 id="introduction">Introduction</h1>
<p>In <a href="/blog/bare-metal-to-kubernetes-part-1/">Part 1</a> of this series I demonstrated the installation and
configuration of <a href="https://maas.io/">MAAS</a>.  In this post I’m going to show how to use Canonical’s <a href="https://jaas.ai/openstack-base/bundle/61">Juju</a> to deploy an <a href="https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html#deploy-openstack">Openstack</a> cloud on top
of hosts managed by MAAS.</p>

<blockquote>
  <p>Juju is an open source application modelling tool that allows you to deploy, configure, scale and operate cloud 
infrastructures quickly and efficiently on public clouds such as AWS, GCE, and Azure along with private ones such as 
MAAS, OpenStack, and VSphere.</p>
</blockquote>

<h1 id="prerequisites">Prerequisites</h1>
<p>If you’ve been following along from the previous post you should have a working MAAS server with at least one physical 
machine in its default resource pool.  Refer to the <a href="https://www.2stacks.net/blog/bare-metal-to-kubernetes-part-1/#prerequisites">prerequisites</a> defined in Part 1 for a complete list of requirements.</p>

<p>Before continuing you’ll need to add at least two more machines to MAAS.  I’ve added an additional physical machine as 
well as a virtual machine composed from a <a href="https://maas.io/docs/kvm-introduction">KVM Pod</a> on which I’ll install my Juju controller.
<img src="/assets/images/20190906/maas_inventory.png" alt="MAAS Inventory" title="MAAS Inventory" /></p>

<blockquote>
  <p>To deploy Openstack you must have a bare minimum of two physical machines available in MAAS.  If you don’t have an extra 
machine to add to MAAS for the Juju controller you can deploy a <a href="https://jaas.ai/docs/clouds#heading--managing-multiple-clouds-with-one-controller">multi-cloud controller</a> on your workstation using <a href="https://linuxcontainers.org/lxd/introduction/">LXD</a>.</p>
</blockquote>

<h1 id="install-juju">Install Juju</h1>
<p>There are two methods of installing Juju on Debian based Linux, <a href="https://snapcraft.io/docs/installing-snapd">Snapd</a> and <a href="https://launchpad.net/~juju/+archive/ubuntu/stable">PPA</a>.  Snap packages are also available for
other Linux distributions.  For MacOS and Windows reference the Juju installation <a href="https://jaas.ai/docs/installing">documentation</a>.  I’ll be using the snap 
package with a Debian based distribution of Linux.</p>

<p>To install the snap run the following</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>snap <span class="nb">install </span>juju <span class="nt">--classic</span>
</code></pre></div></div>

<h1 id="configure-juju">Configure Juju</h1>
<p>Before we can use Juju to model and deploy workloads we must first configure a few prerequisites.</p>
<ul>
  <li><strong>Clouds</strong> - Define the underlying provider of bare metal and/or virtual machines.</li>
  <li><strong>Credentials</strong> - Keys, user names, passwords etc. used to authenticate with each cloud.</li>
  <li><strong>Controllers</strong> - Machines deployed within a cloud to manage configuration and resources.</li>
  <li><strong>Models</strong> - An environment associated with a controller in which workloads are deployed.</li>
</ul>

<h2 id="add-clouds">Add Clouds</h2>
<p>Juju works on top of many different types of <a href="https://jaas.ai/docs/clouds">clouds</a> and has built in support for MAAS.  To see the available clouds run
the following from you workstation console;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju clouds <span class="nt">--local</span>

Cloud           Regions  Default          Type        Description
aws                  18  us-east-1        ec2         Amazon Web Services
aws-china             2  cn-north-1       ec2         Amazon China
aws-gov               2  us-gov-west-1    ec2         Amazon <span class="o">(</span>USA Government<span class="o">)</span>
azure                27  centralus        azure       Microsoft Azure
azure-china           2  chinaeast        azure       Microsoft Azure China
cloudsigma           12  dub              cloudsigma  CloudSigma Cloud
google               18  us-east1         gce         Google Cloud Platform
joyent                6  us-east-1        joyent      Joyent Cloud
oracle                4  us-phoenix-1     oci         Oracle Cloud Infrastructure
oracle-classic        5  uscom-central-1  oracle      Oracle Cloud Infrastructure Classic
rackspace             6  dfw              rackspace   Rackspace Cloud
localhost             1  localhost        lxd         LXD Container Hypervisor
</code></pre></div></div>

<p>To add add a new MAAS cloud run the <code class="language-plaintext highlighter-rouge">add-cloud</code> command and enter the details for your environment.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju add-cloud

Cloud Types
  lxd
  maas
  manual
  openstack
  vsphere

Select cloud <span class="nb">type</span>: maas

Enter a name <span class="k">for </span>your maas cloud: maas-01

Enter the API endpoint url: http://10.1.20.5:5240/MAAS/

Cloud <span class="s2">"maas-01"</span> successfully added

You will need to add credentials <span class="k">for </span>this cloud <span class="o">(</span><span class="sb">`</span>juju add-credential maas-01<span class="sb">`</span><span class="o">)</span>
before creating a controller <span class="o">(</span><span class="sb">`</span>juju bootstrap maas-01<span class="sb">`</span><span class="o">)</span><span class="nb">.</span>
</code></pre></div></div>

<h2 id="add-credentials">Add Credentials</h2>
<p>As the output shows we’ll need to add our MAAS server <a href="https://jaas.ai/docs/credentials">credentials</a> before Juju can interact with the MAAS API.  To obtain
your MAAS API key, log in to MAAS and click on your username in the top right of the MAAS GUI.
<img src="/assets/images/20190906/obtain_credentials.png" alt="Obtain Credentials" title="Obtain Credentials" /></p>

<p>Copy the API key shown or generate a new key.
<img src="/assets/images/20190906/copy_key.png" alt="Copy Key" title="Copy Key" /></p>

<p>Now run the following and enter the copied API key when prompted.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju add-credential maas-01

Enter credential name: maas-01_creds

Using auth-type <span class="s2">"oauth1"</span><span class="nb">.</span>

Enter maas-oauth: 

Credential <span class="s2">"maas-01_creds"</span> added locally <span class="k">for </span>cloud <span class="s2">"maas-01"</span><span class="nb">.</span>
</code></pre></div></div>

<p>To validate/list the added credentials run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju credentials

Cloud    Credentials
maas-01  maas-01_creds
</code></pre></div></div>

<h2 id="add-controllers">Add Controllers</h2>
<p>Now that we have added a cloud and its associated credentials we can add a Juju <a href="https://jaas.ai/docs/creating-a-controller">controller</a>.</p>
<blockquote>
  <p>A controller is the management node of a Juju cloud environment. It runs the API server and the underlying database. 
Its overall purpose is to keep state of all the models, applications, and machines in that environment.</p>
</blockquote>

<p>The controller only requires a minimum of 3.5 GiB of memory and 1vCPU so to ensure that Juju deploys the controller on the
virtual machine I’ve added to MAAS I’m going to add a <a href="https://jaas.ai/docs/constraints-reference">Tag</a> to the machine that Juju can use as a deployment <a href="https://jaas.ai/docs/constraints">constraint</a>.</p>

<p>To add a Tag to a machine go to the Machines tab of the MAAS GUI and click on the name of the machine you would like to
tag.  Next click the <strong>Edit</strong> link under the <strong>Tags</strong> section.
<img src="/assets/images/20190906/add_tag.png" alt="Add Tag" title="Add Tag" /></p>

<p>Click the <strong>Edit</strong> button to the right of the <strong>Machine configuration</strong>
<img src="/assets/images/20190906/edit_config.png" alt="Edit Config" title="Edit Config" /></p>

<p>Add <strong>juju</strong> to the <strong>Tags</strong> section and click <strong>Save changes</strong>
<img src="/assets/images/20190906/save_tags.png" alt="Save Tags" title="Save Tags" /></p>

<p>To bootstrap the controller on MAAS execute the following from you workstation;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju bootstrap <span class="nt">--constraints</span> <span class="nv">tags</span><span class="o">=</span>juju maas-01 juju-01

Creating Juju controller <span class="s2">"juju-01"</span> on maas-01
Looking <span class="k">for </span>packaged Juju agent version 2.6.8 <span class="k">for </span>amd64
Launching controller instance<span class="o">(</span>s<span class="o">)</span> on maas-01...
 - eppk7s <span class="o">(</span><span class="nb">arch</span><span class="o">=</span>amd64 <span class="nv">mem</span><span class="o">=</span>4G <span class="nv">cores</span><span class="o">=</span>1<span class="o">)</span>  
Installing Juju agent on bootstrap instance
Fetching Juju GUI 2.15.0
</code></pre></div></div>

<p>This will bootstrap a controller named <code class="language-plaintext highlighter-rouge">juju-01</code> to a machine with the tag <code class="language-plaintext highlighter-rouge">juju</code> on the <code class="language-plaintext highlighter-rouge">maas-01</code> cloud.  You can verify 
in MAAS that your machine is being deployed.
<img src="/assets/images/20190906/deploy_controller.png" alt="Deploy Controller" title="Deploy Controller" /></p>

<p>After a few minutes your terminal will return the deployment status and you can verify the deployment with the <code class="language-plaintext highlighter-rouge">juju controllers</code>
command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Waiting <span class="k">for </span>address
Attempting to connect to 10.1.20.13:22
Connected to 10.1.20.13
Running machine configuration script...
Bootstrap agent now started
Contacting Juju controller at 10.1.20.13 to verify accessibility...

Bootstrap <span class="nb">complete</span>, controller <span class="s2">"juju-01"</span> now is available
Controller machines are <span class="k">in </span>the <span class="s2">"controller"</span> model
Initial model <span class="s2">"default"</span> added

~<span class="nv">$ </span>juju controllers

Use <span class="nt">--refresh</span> option with this <span class="nb">command </span>to see the latest information.

Controller  Model    User   Access     Cloud/Region  Models  Nodes    HA  Version
juju-01<span class="k">*</span>    default  admin  superuser  maas-01            2      1  none  2.6.8 
</code></pre></div></div>

<h3 id="juju-gui">Juju GUI</h3>
<p>In addition to the API the Juju controller has a web interface that makes it easier to experiment with with Juju’s modeling 
and automation capabilities.  To access the GUI run the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju gui

GUI 2.15.0 <span class="k">for </span>model <span class="s2">"admin/default"</span> is enabled at:
  https://10.1.20.13:17070/gui/u/admin/default
Your login credential is:
  username: admin
  password: 85faa8d324ce26300d8aa6b43b8fd794
</code></pre></div></div>

<p>Open a browser and enter the url and credentials provided in the output.</p>

<p><img src="/assets/images/20190906/juju_gui.png" alt="Juju GUI" title="Juju GUI" /></p>

<p>You can play around with the default model from here, click the green circle in the center of the model to add a workload 
or use the search in the top right to explore the available <a href="https://jaas.ai/docs/charm-writing">Charms</a> in the Charm <a href="https://jaas.ai/store">Store</a>.
<img src="/assets/images/20190906/juju_gui_2.png" alt="Juju GUI" title="Juju GUI" /></p>

<blockquote>
  <p>The rest of this demonstration will use the controller API however it is useful to monitor the status of your models in 
the GUI as you continue the deployment.</p>
</blockquote>

<h2 id="add-models">Add Models</h2>
<p>When a controller is bootstrapped two <a href="https://jaas.ai/docs/configuring-models">models</a> are automatically added, <strong>controller</strong> and <strong>default</strong>.  The controller 
model is used for internal Juju management and is not intended for general workloads. The default model can be used to
deploy any supported software but is typically used for experimentation purposes.</p>

<p>Before deploying Openstack we’ll create a new model by running the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju add-model openstack

Added <span class="s1">'openstack'</span> model with credential <span class="s1">'maas-01_creds'</span> <span class="k">for </span>user <span class="s1">'admin'</span>
</code></pre></div></div>

<p>To verify that the model was created and selected as the active model run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>juju models

Controller: juju-01

Model       Cloud/Region  Type  Status     Machines  Cores  Access  Last connection
controller  maas-01       maas  available         1      1  admin   just now
default     maas-01       maas  available         0      -  admin   5 minutes ago
openstack<span class="k">*</span>  maas-01       maas  available         0      -  admin   never connected
</code></pre></div></div>

<h1 id="deploy-openstack">Deploy Openstack</h1>
<p>Both the <a href="https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-openstack.html#deploy-openstack">Openstack</a> and <a href="https://jaas.ai/openstack-base/bundle/61">Juju</a> documentation contain detailed, step by step instructions for installing the core components 
of Openstack with Juju.  Instead of just providing those as reference I wanted to show how you can customize the 
installation to reduce the number of servers needed as well as how to include <a href="https://jaas.ai/octavia/11">Octavia</a> (Openstacks’ LBaaS implementation).  In
my next post I’ll demonstrate how Kubernetes interacts directly with Octavia to provide L4 load balancing services for 
container workloads.</p>

<blockquote>
  <p>The Juju and Openstack documentation both utilize a minimum of four servers in their examples.  My demonstration will show
how to deploy on a minimum of two servers.  Deploying on a single server is only possible with <a href="https://linuxcontainers.org/lxd/introduction/">LXD</a> as <a href="https://jaas.ai/neutron-gateway">Neutron Gateway</a> 
and <a href="https://jaas.ai/nova-compute">Nova Compute</a> can not run on the same bare metal server.</p>
</blockquote>

<h2 id="configure-openstack-bundle">Configure Openstack Bundle</h2>
<p>Before we deploy the <a href="https://jaas.ai/openstack-base/bundle/61">Openstack Base</a> bundle we need to make modifications to match our lab environment.  There’s a charm 
development tool aptly named <em>charm</em> that can be installed as a snap.  To install the snap run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>snap <span class="nb">install </span>charm <span class="nt">--classic</span>
</code></pre></div></div>
<p>For other distributions please reference the Charm Tools Project <a href="https://jaas.ai/docs/charm-tools">documentation</a>.</p>

<p>Now we’ll pull down the current version of the Openstack bundle with;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>charm pull cs:openstack-base
</code></pre></div></div>

<p>Change to the newly created directory and create a backup of the bundle.yaml file;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>openstack-base
<span class="nb">cp </span>bundle.yaml bundle.yaml.bak
</code></pre></div></div>

<p>If you’ve been following my examples you can download a copy of the bundle file I’m using or edit the bundle.yaml file to
match your own environment;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget https://gist.githubusercontent.com/2stacks/d0b4b4b81df4a835934bbd9b8543ad2e/raw/aa98813d2dc260b188931c5df58066db3b54c4df/bundle.yaml
</code></pre></div></div>

<p>You can diff my version with the original file to see what I’ve changed.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>diff bundle.yaml bundle.yaml.bak
</code></pre></div></div>
<p>In short, I’m placing Neutron Gateway and all of the other Openstack dependencies on one host and reserving the remaining 
host for Nova Compute.  Remember that Neutron Gateway and Nova Compute can not run on the same bare metal machine.  In 
order to have as much capacity as possible for running virtual machines I’m not running any additional services on the 
host dedicated to Nova Compute.</p>

<h2 id="configure-octavia-overlay">Configure Octavia Overlay</h2>
<p>Next we’ll download and modify the Octavia overlay used to deploy the resources required by Octavia.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl https://raw.githubusercontent.com/openstack-charmers/openstack-bundles/master/stable/overlays/loadbalancer-octavia.yaml <span class="nt">-o</span> loadbalancer-octavia.yaml
</code></pre></div></div>

<p>By default this overlay expects a minimum of four machines so it will need to be modified to run with fewer machines.  As
before I’ve modified the file to deploy all of the resources on the host running Neutron Gateway.  You can download my 
version of the file by running;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mv </span>loadbalancer-octavia.yaml loadbalancer-octavia.yaml.bak
wget https://gist.githubusercontent.com/2stacks/21ba79de48e38b3588868cd033675d1a/raw/2eac80acffc6173b996cc30862da7a02317ce9a3/loadbalancer-octavia.yaml
</code></pre></div></div>

<h2 id="deploy-openstack-bundle">Deploy Openstack Bundle</h2>
<p>Once we’ve made the necessary changes or downloaded the modified files we can start the deployment of Openstack.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>juju deploy ./bundle.yaml <span class="nt">--overlay</span> loadbalancer-octavia.yaml
</code></pre></div></div>

<p>The deployment time depends on the speed of your hardware but in my environment it takes approximately 15 minutes.  You 
can monitor the deployment status by running the following command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>watch <span class="nt">-c</span> juju status <span class="nt">--color</span>
</code></pre></div></div>
<p>You can also monitor the status of the deployment and view the model of the Openstack installation that has been constructed 
in the <a href="#juju-gui">Juju GUI</a>.  While Juju is deploying the Openstack workloads and resources we can continue to apply 
additional configurations required by Octavia.</p>

<h2 id="configure-octavia">Configure Octavia</h2>
<p>Octavia consists of an Openstack controller and load balancer instances that deploy as virtual machines on Nova Compute.  The 
communication between the controller and each instance is secured with bi-directional certificate authentication over an 
out-of-band management network.  The certificate, network and image resources must be configured before you can deploy 
Octavia load balancer instances.</p>

<blockquote>
  <p>For additional information on configuring the Octavia Charm please reference the <a href="https://jaas.ai/octavia/11">documentation</a></p>
</blockquote>

<h3 id="generate-certificates">Generate Certificates</h3>
<p>Below is an example of how to create self-signed certificates for use by Octavia.  <strong>Note:</strong> This example is not meant to 
be used in production.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir</span> <span class="nt">-p</span> demoCA/newcerts
<span class="nb">touch </span>demoCA/index.txt
<span class="nb">touch </span>demoCA/index.txt.attr
openssl genrsa <span class="nt">-passout</span> pass:foobar <span class="nt">-des3</span> <span class="nt">-out</span> issuing_ca_key.pem 2048
openssl req <span class="nt">-x509</span> <span class="nt">-passin</span> pass:foobar <span class="nt">-new</span> <span class="nt">-nodes</span> <span class="nt">-key</span> issuing_ca_key.pem <span class="se">\</span>
    <span class="nt">-config</span> /etc/ssl/openssl.cnf <span class="se">\</span>
    <span class="nt">-subj</span> <span class="s2">"/C=US/ST=Somestate/O=Org/CN=www.example.com"</span> <span class="se">\</span>
    <span class="nt">-days</span> 365 <span class="se">\</span>
    <span class="nt">-out</span> issuing_ca.pem
openssl genrsa <span class="nt">-passout</span> pass:foobar <span class="nt">-des3</span> <span class="nt">-out</span> controller_ca_key.pem 2048
openssl req <span class="nt">-x509</span> <span class="nt">-passin</span> pass:foobar <span class="nt">-new</span> <span class="nt">-nodes</span> <span class="se">\</span>
    <span class="nt">-key</span> controller_ca_key.pem <span class="se">\</span>
    <span class="nt">-config</span> /etc/ssl/openssl.cnf <span class="se">\</span>
    <span class="nt">-subj</span> <span class="s2">"/C=US/ST=Somestate/O=Org/CN=www.example.com"</span> <span class="se">\</span>
    <span class="nt">-days</span> 365 <span class="se">\</span>
    <span class="nt">-out</span> controller_ca.pem
openssl req <span class="se">\</span>
    <span class="nt">-newkey</span> rsa:2048 <span class="nt">-nodes</span> <span class="nt">-keyout</span> controller_key.pem <span class="se">\</span>
    <span class="nt">-subj</span> <span class="s2">"/C=US/ST=Somestate/O=Org/CN=www.example.com"</span> <span class="se">\</span>
    <span class="nt">-out</span> controller.csr
openssl ca <span class="nt">-passin</span> pass:foobar <span class="nt">-config</span> /etc/ssl/openssl.cnf <span class="se">\</span>
    <span class="nt">-cert</span> controller_ca.pem <span class="nt">-keyfile</span> controller_ca_key.pem <span class="se">\</span>
    <span class="nt">-create_serial</span> <span class="nt">-batch</span> <span class="se">\</span>
    <span class="nt">-in</span> controller.csr <span class="nt">-days</span> 365 <span class="nt">-out</span> controller_cert.pem
<span class="nb">cat </span>controller_cert.pem controller_key.pem <span class="o">&gt;</span> controller_cert_bundle.pem
</code></pre></div></div>

<h3 id="apply-certificates">Apply Certificates</h3>
<p>To apply the certificates to the Octavia controller run the following command.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>juju config octavia <span class="se">\</span>
    lb-mgmt-issuing-cacert<span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">base64 </span>controller_ca.pem<span class="si">)</span><span class="s2">"</span> <span class="se">\</span>
    lb-mgmt-issuing-ca-private-key<span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">base64 </span>controller_ca_key.pem<span class="si">)</span><span class="s2">"</span> <span class="se">\</span>
    lb-mgmt-issuing-ca-key-passphrase<span class="o">=</span>foobar <span class="se">\</span>
    lb-mgmt-controller-cacert<span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">base64 </span>controller_ca.pem<span class="si">)</span><span class="s2">"</span> <span class="se">\</span>
    lb-mgmt-controller-cert<span class="o">=</span><span class="s2">"</span><span class="si">$(</span><span class="nb">base64 </span>controller_cert_bundle.pem<span class="si">)</span><span class="s2">"</span>
</code></pre></div></div>

<h3 id="create-openstack-resources">Create Openstack Resources</h3>
<p>To initiate the creation of the Openstack network, router and security group used by Octavia instances run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>juju run-action <span class="nt">--wait</span> octavia/0 configure-resources
</code></pre></div></div>

<h3 id="deploy-amphora-image">Deploy Amphora Image</h3>
<p>Finally to create the disk image that will be used by load balancer instances run the following.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>juju run-action <span class="nt">--wait</span> octavia-diskimage-retrofit/leader retrofit-image
</code></pre></div></div>
<blockquote>
  <p>By default this will create an Ubuntu bionic image preconfigured with <a href="http://www.haproxy.org/">HAProxy</a>.This command 
can take a long time to finish so be patient.  It should eventually create an image with a prefix of <em>amphora-haproxy</em>. We’ll 
validate that this image was successfully created in the next section.</p>
</blockquote>

<h1 id="validate-deployment">Validate Deployment</h1>
<p>Once everything has been configured and Juju has finished it’s deployment we can begin validating the installation by creating
resources within Openstack.</p>

<blockquote>
  <p>The output of <code class="language-plaintext highlighter-rouge">juju status</code> should show everything as <strong>active</strong> with the exception of <a href="https://www.vaultproject.io/">Vault</a>.  It’s ok to leave this as 
is for now but if you’d like additional information on setting up Vault please reference <a href="https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-vault.html">Appendix C</a> of the Openstack 
deployment documentation.</p>
</blockquote>

<h2 id="api-access">API Access</h2>
<p>The Openstack installation should now be accessible via it’s API and built in web interface.  To begin utilizing the API
we need to install the <a href="https://docs.openstack.org/python-openstackclient/pike/">OpenStackClient</a> package.</p>

<blockquote>
  <p>OpenStackClient (aka OSC) is a command-line client for OpenStack that brings the command set for Compute, Identity, Image, 
Object Storage and Block Storage APIs together in a single shell with a uniform command structure.</p>
</blockquote>

<p>There are several ways to install the client package but for most linux distributions the snap package is recommended.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>snap <span class="nb">install </span>openstackclients
</code></pre></div></div>

<p>The credentials required to access the API can be obtained and set as environment variables by running the rc script included
in the <strong>openstack-base</strong> directory.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">source </span>openrc
</code></pre></div></div>

<p>To view the credentials run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span><span class="nb">env</span> | <span class="nb">grep </span>OS_

<span class="nv">OS_REGION_NAME</span><span class="o">=</span>RegionOne
<span class="nv">OS_USER_DOMAIN_NAME</span><span class="o">=</span>admin_domain
<span class="nv">OS_PROJECT_NAME</span><span class="o">=</span>admin
<span class="nv">OS_AUTH_VERSION</span><span class="o">=</span>3
<span class="nv">OS_IDENTITY_API_VERSION</span><span class="o">=</span>3
<span class="nv">OS_PASSWORD</span><span class="o">=</span>MaemumeFoolah1ae
<span class="nv">OS_AUTH_TYPE</span><span class="o">=</span>password
<span class="nv">OS_AUTH_URL</span><span class="o">=</span>http://10.1.20.36:5000/v3
<span class="nv">OS_USERNAME</span><span class="o">=</span>admin
<span class="nv">OS_PROJECT_DOMAIN_NAME</span><span class="o">=</span>admin_domain
</code></pre></div></div>

<p>To verify access to the API run the following commands.  They should output a list of all of the Openstack API endpoints 
and services created by Juju.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack catalog list
openstack endpoint list
</code></pre></div></div>

<h2 id="web-dashboard-access">Web Dashboard Access</h2>
<p>To access the Openstack web interface, retrieve the IP address of the LXD container that the Openstack Dashboard was deployed on.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>juju status openstack-dashboard/0<span class="k">*</span> | <span class="nb">grep </span>Container | <span class="nb">awk</span> <span class="s1">'{print $3}'</span>
</code></pre></div></div>

<p>Open a browser and use the returned IP address in the following URL;</p>

<p><code class="language-plaintext highlighter-rouge">http://&lt;dashboard_ip&gt;/horizon</code></p>

<p>The credentials needed to log in to the browser are the same as previously obtained with the <strong>openrc</strong> script.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">OS_USER_DOMAIN_NAME</span><span class="o">=</span>admin_domain
<span class="nv">OS_PROJECT_NAME</span><span class="o">=</span>admin
<span class="nv">OS_PASSWORD</span><span class="o">=</span>MaemumeFoolah1ae
</code></pre></div></div>

<p><img src="/assets/images/20190906/openstack_gui.png" alt="Openstack GUI" title="Openstack GUI" /></p>

<h2 id="add-images">Add Images</h2>
<p>During deployment the Juju <a href="https://jaas.ai/glance-simplestreams-sync/21">glance-simplestreams</a> charm should have added a few default disk images in addition to the 
<a href="https://docs.openstack.org/octavia/stein/reference/glossary.html">Amphora</a> image used by Octavia.  To verify that these images were created run the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack image list

+--------------------------------------+-----------------------------------------------------------------+--------+
| ID                                   | Name                                                            | Status |
+--------------------------------------+-----------------------------------------------------------------+--------+
| 86c318f3-76f7-4311-b146-cc00e79e9406 | amphora-haproxy-x86_64-ubuntu-18.04-20190813.1                  | active |
| 40aca86a-9177-4b94-afe3-fee448abee4c | auto-sync/ubuntu-bionic-18.04-amd64-server-20190813.1-disk1.img | active |
| 4b238000-ba59-4470-bcea-93dd8761c40e | auto-sync/ubuntu-trusty-14.04-amd64-server-20190514-disk1.img   | active |
| d87ed10a-5d86-409c-9fa5-7cf59e80a258 | auto-sync/ubuntu-xenial-16.04-amd64-server-20190814-disk1.img   | active |
+--------------------------------------+-----------------------------------------------------------------+--------+
</code></pre></div></div>

<p>You can use the <strong>auto-sync</strong> images in the following examples or create your own via the API.  The following command will
create an image named <em>bionic</em> using the current Ubuntu 18.04 cloud-image.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>curl http://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img | <span class="se">\</span>
openstack image create <span class="nt">--public</span> <span class="nt">--min-disk</span> 3 <span class="nt">--container-format</span> bare <span class="nt">--disk-format</span> qcow2 <span class="se">\</span>
    <span class="nt">--property</span> <span class="nv">architecture</span><span class="o">=</span>x86_64 <span class="nt">--property</span> <span class="nv">hw_disk_bus</span><span class="o">=</span>virtio <span class="nt">--property</span> <span class="nv">hw_vif_model</span><span class="o">=</span>virtio bionic
</code></pre></div></div>

<p>If you run <code class="language-plaintext highlighter-rouge">openstack image list</code> again you should see the newly created image in the list.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack image list

+--------------------------------------+-----------------------------------------------------------------+--------+
| ID                                   | Name                                                            | Status |
+--------------------------------------+-----------------------------------------------------------------+--------+
| 86c318f3-76f7-4311-b146-cc00e79e9406 | amphora-haproxy-x86_64-ubuntu-18.04-20190813.1                  | active |
| 40aca86a-9177-4b94-afe3-fee448abee4c | auto-sync/ubuntu-bionic-18.04-amd64-server-20190813.1-disk1.img | active |
| 4b238000-ba59-4470-bcea-93dd8761c40e | auto-sync/ubuntu-trusty-14.04-amd64-server-20190514-disk1.img   | active |
| d87ed10a-5d86-409c-9fa5-7cf59e80a258 | auto-sync/ubuntu-xenial-16.04-amd64-server-20190814-disk1.img   | active |
| b2797730-70ca-4e91-8b30-26cf2e6c7968 | bionic                                                          | active |
+--------------------------------------+-----------------------------------------------------------------+--------+
</code></pre></div></div>

<h2 id="add-flavors">Add Flavors</h2>
<p>Next we’ll create <a href="https://docs.openstack.org/nova/stein/user/flavors.html">flavors</a> that determine the amount of resources given to virtual machine instances.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack flavor create <span class="nt">--ram</span> 1024 <span class="nt">--disk</span> 10 m1.small
openstack flavor create <span class="nt">--vcpus</span> 2 <span class="nt">--ram</span> 2048 <span class="nt">--disk</span> 20 m1.medium
openstack flavor create <span class="nt">--vcpus</span> 4 <span class="nt">--ram</span> 4096 <span class="nt">--disk</span> 40 m1.large
</code></pre></div></div>

<p>To list the newly created instance flavors run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack flavor list 

+--------------------------------------+-----------+------+------+-----------+-------+-----------+
| ID                                   | Name      |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+-----------+------+------+-----------+-------+-----------+
| 8dc59145-1ba3-4709-b037-6806a38d598c | m1.large  | 4096 |   20 |         0 |     4 | True      |
| c14adc69-d72a-4ed2-b49a-d010a5e58fa3 | m1.small  | 1024 |   10 |         0 |     1 | True      |
| e426aa37-2d29-4ece-8e3c-c1fb833f9308 | m1.medium | 2048 |   20 |         0 |     2 | True      |
+--------------------------------------+-----------+------+------+-----------+-------+-----------+
</code></pre></div></div>

<h2 id="add-networks-and-subnets">Add Networks and Subnets</h2>
<p>We need to create at least two networks (internal and external) to verify the functionality of Openstack virtual routers 
and load balancers.  The two types of Openstack networks we’ll create are;</p>

<ul>
  <li><a href="https://docs.openstack.org/mitaka/networking-guide/intro-os-networking.html#provider-networks">provider-network</a></li>
  <li><a href="https://docs.openstack.org/mitaka/networking-guide/intro-os-networking.html#self-service-networks">self-service network</a></li>
</ul>

<p>The first network type we’ll create is a provider-network.  This network will provide ingress and egress 
communication between compute instances and the outside world.  Subnets created within this network can be used to assign 
floating IPs which will allow you to run public services and remotely reach your Openstack instances.</p>

<p>To create the network run the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack network create ext_net <span class="nt">--share</span> <span class="nt">--external</span> <span class="nt">--provider-network-type</span> flat <span class="nt">--provider-physical-network</span> physnet1
</code></pre></div></div>

<p>To create a subnet inside the network for allocation of floating IPs run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack subnet create ext_subnet <span class="nt">--no-dhcp</span> <span class="nt">--allocation-pool</span> <span class="nv">start</span><span class="o">=</span>192.168.10.101,end<span class="o">=</span>192.168.10.254 <span class="se">\</span>
    <span class="nt">--subnet-range</span> 192.168.10.0/24 <span class="nt">--gateway</span> 192.168.10.1 <span class="nt">--dns-nameserver</span> 192.168.10.1 <span class="nt">--network</span> ext_net
</code></pre></div></div>

<blockquote>
  <p>Refer to <a href="/blog/bare-metal-to-kubernetes-part-1/">Part 1</a> of this series for a description of the networks 
and interfaces required to support the Openstack installation.  The subnet you create here must be routable in your 
environment.</p>
</blockquote>

<p>The second network type we’ll create is a self-service network.  These networks allow non privileged users of the Openstack
installation to create project specific networks.  Typically they employ GRE or VXLAN as overlays and require an Openstack 
virtual router to connect to external networks.</p>

<p>To create the network run the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack network create int_net
</code></pre></div></div>

<p>To create a subnet inside the network;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack subnet create int_subnet <span class="nt">--allocation-pool</span> <span class="nv">start</span><span class="o">=</span>10.0.0.11,end<span class="o">=</span>10.0.0.254 <span class="se">\</span>
    <span class="nt">--subnet-range</span> 10.0.0.0/24 <span class="nt">--gateway</span> 10.0.0.1 <span class="nt">--dns-nameserver</span> 192.168.10.1 <span class="nt">--network</span> int_net
</code></pre></div></div>

<blockquote>
  <p>This will deploy a virtual network within the default <em>admin</em> project using the default overlay type of GRE.  Since this 
subnet is not exposed outside of Openstack by default, you can configure any subnet and IP range you like.  Note that I 
have configured a dns-nameserver external to Openstack for external name resolution.</p>
</blockquote>

<p>To verify the networks and subnets we just created run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack network list

+--------------------------------------+-------------+--------------------------------------+
| ID                                   | Name        | Subnets                              |
+--------------------------------------+-------------+--------------------------------------+
| 1730fd08-c48c-4591-993c-7d5f90ea5578 | ext_net     | 3d283801-faee-44c1-b51b-5c69f76ef61f |
| 695d5dab-a12a-4471-841b-566c08ea732d | lb-mgmt-net | 525306b4-5653-4b80-bbe9-facf3d900bca |
| a606266e-3c10-4962-9588-71a7989c78ac | int_net     | c4a1bfd3-463d-44a0-b4ca-9748e76f90e6 |
+--------------------------------------+-------------+--------------------------------------+

~<span class="nv">$ </span>openstack subnet list

+--------------------------------------+------------------+--------------------------------------+-------------------------+
| ID                                   | Name             | Network                              | Subnet                  |
+--------------------------------------+------------------+--------------------------------------+-------------------------+
| 3d283801-faee-44c1-b51b-5c69f76ef61f | ext_subnet       | 1730fd08-c48c-4591-993c-7d5f90ea5578 | 192.168.10.0/24         |
| 525306b4-5653-4b80-bbe9-facf3d900bca | lb-mgmt-subnetv6 | 695d5dab-a12a-4471-841b-566c08ea732d | fc00:566c:8ea:732d::/64 |
| c4a1bfd3-463d-44a0-b4ca-9748e76f90e6 | int_subnet       | a606266e-3c10-4962-9588-71a7989c78ac | 10.0.0.0/24             |
+--------------------------------------+------------------+--------------------------------------+-------------------------+
</code></pre></div></div>

<p>For additional network options and settings consult the <a href="https://docs.openstack.org/neutron/stein/admin/">Openstack Networking</a> documentation or run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack network create <span class="nt">--help</span>
</code></pre></div></div>

<h2 id="add-a-router">Add a Router</h2>
<p>Openstack virtual routers provide layer-3 services such as routing and NAT.  They are required for routing between 
self-service networks within the same project as well as for communication between self-service and provider networks.</p>

<p>To create a router within the current domain/project run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack router create rtr-01
</code></pre></div></div>

<p>To attach a router interface to the previously created subnet within our provider network run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack router <span class="nb">set </span>rtr-01 <span class="nt">--external-gateway</span> ext_net
</code></pre></div></div>

<p>And finally to connect our internal self-service subnet to the external provider subnet run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack router add subnet rtr-01 int_subnet
</code></pre></div></div>

<p>To verify that the router was created and configured run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack router list
openstack router show rtr-01
</code></pre></div></div>

<p>You can also verify via the Openstack web interface by navigating to;</p>

<p><code class="language-plaintext highlighter-rouge">http://&lt;dashboard_ip&gt;/horizon/project/network_topology/</code></p>

<p>Here openstack will display a topology of the existing routers/networks/instances created within the current project.
<img src="/assets/images/20190906/network_topology.png" alt="Network Topology" title="Network Topology" /></p>

<p>If all has gone well up to now you should be able to ping the external interface of the router from your workstation.  To
get the external IP of the router run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">router_ip</span><span class="o">=</span><span class="si">$(</span>openstack port list <span class="nt">--router</span> rtr-01 <span class="nt">--network</span> ext_net <span class="nt">-f</span> value | <span class="nb">awk</span> <span class="nt">-F</span> <span class="s2">"'"</span> <span class="s1">'{print $2}'</span><span class="si">)</span>
</code></pre></div></div>

<p>Now try pinging the returned IP from your workstation.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>ping <span class="nv">$router_ip</span>

PING 192.168.10.214 <span class="o">(</span>192.168.10.214<span class="o">)</span> 56<span class="o">(</span>84<span class="o">)</span> bytes of data.
64 bytes from 192.168.10.214: <span class="nv">icmp_seq</span><span class="o">=</span>1 <span class="nv">ttl</span><span class="o">=</span>64 <span class="nb">time</span><span class="o">=</span>1.17 ms
64 bytes from 192.168.10.214: <span class="nv">icmp_seq</span><span class="o">=</span>2 <span class="nv">ttl</span><span class="o">=</span>64 <span class="nb">time</span><span class="o">=</span>0.292 ms
</code></pre></div></div>

<h2 id="add-security-groups">Add Security Groups</h2>
<p>To allow access to our compute instances we’ll need to create security groups or add rules to the existing security groups 
that were created during the installation of Openstack.  To view a list of the existing security groups run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack security group list
</code></pre></div></div>

<p>Next we’ll add rules to the default security group to allow ICMP and SSH access from external hosts.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">project_id</span><span class="o">=</span><span class="si">$(</span>openstack project show <span class="nt">--domain</span> admin_domain admin <span class="nt">-f</span> value <span class="nt">-c</span> <span class="nb">id</span><span class="si">)</span>
<span class="nv">secgroup_id</span><span class="o">=</span><span class="si">$(</span>openstack security group list <span class="nt">--project</span> <span class="k">${</span><span class="nv">project_id</span><span class="k">}</span> | <span class="nb">awk</span> <span class="s1">'/default/ {print $2}'</span><span class="si">)</span>
openstack security group rule create <span class="k">${</span><span class="nv">secgroup_id</span><span class="k">}</span> <span class="nt">--protocol</span> icmp <span class="nt">--remote-ip</span> 0.0.0.0/0
openstack security group rule create <span class="k">${</span><span class="nv">secgroup_id</span><span class="k">}</span> <span class="nt">--protocol</span> tcp <span class="nt">--remote-ip</span> 0.0.0.0/0 <span class="nt">--dst-port</span> 22
</code></pre></div></div>

<p>You can validate that the rules were added to the security group by running;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack security group show <span class="k">${</span><span class="nv">secgroup_id</span><span class="k">}</span>
</code></pre></div></div>

<h2 id="add-ssh-keys">Add SSH Keys</h2>
<p>To be able to access our compute instances remotely via ssh we’ll need to either create a new SSH keypair or upload the 
public key of an existing keypair.</p>

<p>To create a new keypair run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">touch</span> ~/.ssh/os_rsa
<span class="nb">chmod </span>600 ~/.ssh/os_rsa
openstack keypair create os_rsa <span class="o">&gt;</span> ~/.ssh/os_rsa
</code></pre></div></div>

<p>Or to upload an existing public key run;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack keypair create <span class="nt">--public-key</span> ~/.ssh/id_rsa.pub id_rsa
</code></pre></div></div>

<h2 id="add-an-instance">Add an Instance</h2>
<p>The following command will add a new compute instance named <em>bionic-test</em> using the Ubuntu 18.04 cloud image.  The configured
flavor will give it 1 vCPU and 1G of RAM.  It will be configured with an IP in the internal subnet we created, be 
assigned to the default security group and will allow SSH access using the os_rsa keypair.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack server create <span class="se">\</span>
    <span class="nt">--image</span> bionic <span class="nt">--flavor</span> m1.small <span class="nt">--key-name</span> os_rsa <span class="se">\</span>
    <span class="nt">--nic</span> net-id<span class="o">=</span><span class="si">$(</span>openstack network list | <span class="nb">grep </span>int_net | <span class="nb">awk</span> <span class="s1">'{ print $2 }'</span><span class="si">)</span> <span class="se">\</span>
    bionic-test
</code></pre></div></div>

<h3 id="add-floating-ip">Add Floating IP</h3>
<p>To enable SSH access from external hosts we’ll need to assign the instance a floating ip in the external subnet.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">floating_ip</span><span class="o">=</span><span class="si">$(</span>openstack floating ip create <span class="nt">-f</span> value <span class="nt">-c</span> floating_ip_address ext_net<span class="si">)</span>
openstack server add floating ip bionic-test <span class="k">${</span><span class="nv">floating_ip</span><span class="k">}</span>
</code></pre></div></div>

<h3 id="test-reachability">Test Reachability</h3>
<p>Once the <strong>Status</strong> of <code class="language-plaintext highlighter-rouge">openstack server list</code> shows that the instance is <strong>ACTIVE</strong> we can attempt to access it via ssh.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack server list

+--------------------------------------+-------------+--------+-------------------+--------+----------+
| ID                                   | Name        | Status | Networks          | Image  | Flavor   |
+--------------------------------------+-------------+--------+-------------------+--------+----------+
| 946b52a2-bf60-4063-966c-f0d1f4828777 | bionic-test | ACTIVE | <span class="nv">int_net</span><span class="o">=</span>10.0.0.35 | bionic | m1.small |
+--------------------------------------+-------------+--------+-------------------+--------+----------+
</code></pre></div></div>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>ssh <span class="nt">-i</span> ~/.ssh/os_rsa ubuntu@<span class="nv">$floating_ip</span>

Welcome to Ubuntu 18.04.3 LTS <span class="o">(</span>GNU/Linux 4.15.0-60-generic x86_64<span class="o">)</span>

ubuntu@bionic-test:~<span class="err">$</span>
</code></pre></div></div>

<h2 id="add-a-load-balancer">Add a Load Balancer</h2>
<p>The last thing we’ll test is the functionality of the Octavia load balancing service.  I’m not going set up multiple web servers
and verify full functionality for now.  The goal of this test is to verify than an Amphora instance is created and
that it can properly route traffic to the Ubuntu instance we just crated.</p>

<p>To create the load balancer with an interface in our internal subnet run the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">lb_vip_port_id</span><span class="o">=</span><span class="si">$(</span>openstack loadbalancer create <span class="nt">-f</span> value <span class="nt">-c</span> vip_port_id <span class="nt">--name</span> lb-01 <span class="nt">--vip-subnet-id</span> int_subnet<span class="si">)</span>
</code></pre></div></div>

<p>Continue to run the following until lb-01 shows status of <strong>ACTIVE</strong> and <strong>ONLINE</strong>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack loadbalancer show lb-01
</code></pre></div></div>

<p>The instance can take some time to create so be patient.  If the image creation fails you can check the worker logs on the
Octavia controller.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>juju ssh octavia/0
more /var/log/octavia/octavia-worker.log 
</code></pre></div></div>

<h3 id="configure-the-load-balancer">Configure the Load Balancer</h3>
<p>Once the load balancer has been created we can configure vips/pools/monitors etc.  For testing purposes, I’m only going 
to show how to create a simple service to forward ssh connections to the <em>bionic-test</em> instance we created.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>openstack loadbalancer listener create <span class="nt">--name</span> listener1 <span class="nt">--protocol</span> tcp <span class="nt">--protocol-port</span> 22 lb-01
openstack loadbalancer pool create <span class="nt">--name</span> pool1 <span class="nt">--lb-algorithm</span> ROUND_ROBIN <span class="nt">--listener</span> listener1 <span class="nt">--protocol</span> tcp
openstack loadbalancer healthmonitor create <span class="nt">--delay</span> 5 <span class="nt">--timeout</span> 5 <span class="nt">--max-retries</span> 3 <span class="nt">--type</span> TCP pool1
<span class="nv">instance_ip</span><span class="o">=</span><span class="si">$(</span>openstack server show bionic-test <span class="nt">-f</span> value <span class="nt">-c</span> addresses | <span class="nb">sed</span> <span class="nt">-e</span> <span class="s1">'s/int_net=//'</span> | <span class="nb">awk</span> <span class="nt">-F</span> <span class="s2">","</span> <span class="s1">'{print $1}'</span><span class="si">)</span>
openstack loadbalancer member create <span class="nt">--subnet-id</span> int_subnet <span class="nt">--address</span> <span class="k">${</span><span class="nv">instance_ip</span><span class="k">}</span> <span class="nt">--protocol-port</span> 22 pool1
</code></pre></div></div>

<p>Now verify the status of the health monitor and pool member.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>openstack loadbalancer member list pool1

+--------------------------------------+------+----------------------------------+---------------------+-----------+---------------+------------------+--------+
| <span class="nb">id</span>                                   | name | project_id                       | provisioning_status | address   | protocol_port | operating_status | weight |
+--------------------------------------+------+----------------------------------+---------------------+-----------+---------------+------------------+--------+
| b332c304-3695-4e83-a57f-7bcd2bf76f3d |      | 55d6e3168bc24e0397c1185a09500b78 | ACTIVE              | 10.0.0.35 |            22 | ONLINE           |      1 |
+--------------------------------------+------+----------------------------------+---------------------+-----------+---------------+------------------+--------+
</code></pre></div></div>

<h3 id="add-floating-ip-1">Add Floating IP</h3>
<p>To reach our load balancer externaly we’ll need to create and assign a floating IP address.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">floating_ip</span><span class="o">=</span><span class="si">$(</span>openstack floating ip create <span class="nt">-f</span> value <span class="nt">-c</span> floating_ip_address ext_net<span class="si">)</span>
openstack floating ip <span class="nb">set</span> <span class="nt">--port</span> <span class="nv">$lb_vip_port_id</span> <span class="nv">$floating_ip</span>
</code></pre></div></div>

<h3 id="test-reachability-1">Test Reachability</h3>
<p>Now ssh to the newly created floating IP.  If the load balancer was created correctly it should forward your connection 
to your internal instance.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>ssh <span class="nt">-i</span> ~/.ssh/os_rsa ubuntu@<span class="nv">$floating_ip</span>

Welcome to Ubuntu 18.04.3 LTS <span class="o">(</span>GNU/Linux 4.15.0-60-generic x86_64<span class="o">)</span>

ubuntu@bionic-test:~<span class="err">$</span>
</code></pre></div></div>

<h1 id="conclusion">Conclusion</h1>
<p>At this point we have deployed and validated the essential components of Openstack necessary to complete our 
Kubernetes-as-a-Service solution.  Openstack is a very robust and capable IaaS solution for building public and private 
clouds.  This post is only intended to demonstrate a subset of it’s capabilities and how to use open source orchestration
to simplify what is ordinarily a very complex and labor intensive endeavor.  In my next post I’ll continue with this trend of 
simplifying complex deployments through the use of open source solutions and demonstrate how to automate Kubernetes
deployments on top of Openstack.</p>]]></content><author><name>2stacks</name></author><category term="Blog" /><category term="IaaS" /><category term="Multi-Cloud" /><category term="MaaS" /><category term="Juju" /><category term="OpenStack" /><category term="Kubernetes" /><summary type="html"><![CDATA[Using JuJu and MAAS to deploy Openstack.]]></summary></entry><entry><title type="html">Bare Metal to Kubernetes-as-a-Service - Part 1</title><link href="/blog/bare-metal-to-kubernetes-part-1/" rel="alternate" type="text/html" title="Bare Metal to Kubernetes-as-a-Service - Part 1" /><published>2019-08-05T00:00:00+00:00</published><updated>2019-08-05T00:00:00+00:00</updated><id>/blog/bare-metal-to-kubernetes-part-1</id><content type="html" xml:base="/blog/bare-metal-to-kubernetes-part-1/"><![CDATA[<h1 id="introduction">Introduction</h1>

<blockquote>
  <p>“You can think of Kubernetes as a platform for application patterns. The patterns make your application easy to deploy, 
easy to run, and easy to keep running.” <cite><a href="https://www.zdnet.com/article/what-kubernetes-really-is-and-how-orchestration-redefines-the-data-center/">Janet Kuo</a></cite></p>
</blockquote>

<p>The “ease” with which Google Engineer Janet Kuo speaks has led to the rapid adoption of Kubernetes by organizations of 
all sizes and across all industries.  However, “easy” is relative.  For all of the gains made by Kubernetes, as it relates 
to managing applications, there is a general consensus that deploying and managing Kubernetes itself is hard.  To address 
this problem, all of the major public cloud companies (Google <a href="https://cloud.google.com/kubernetes-engine/">GKE</a>, Amazon <a href="https://aws.amazon.com/eks/">EKS</a>, Microsoft <a href="https://azure.microsoft.com/en-us/services/kubernetes-service/">AKS</a>, etc.) are now offering 
Kubernetes-as-a-Service.  Similar to Infrastructure-as-a-Service (<a href="https://en.wikipedia.org/wiki/Infrastructure_as_a_service">IaaS</a>) these offerings promise to abstract the functionality 
of Kubernetes from the underlying physical resources, tooling and expertise needed to support it.  If your organization 
is leveraging public cloud services then problem solved.  But what about organizations in search of private cloud Kubernetes 
offerings?  Although there are a growing number of options available in this space (OpenShift, Rancher, PKS, Platform9 etc.) 
the degree to which they make Kubernetes “easy” is still relative.</p>

<h2 id="objective">Objective</h2>
<p>Organizations looking to adopt an on-premise as-a-service Kubernetes solution face implementation challenges and choices. This 
lab deployment demonstrates how anyone can build their own Kubernetes-as-a-Service offering using readily available open 
source solutions. I’m going to cover this process in a series of posts each one building upon the previous.  The final 
solution will consist of a private cloud, built with <a href="https://maas.io/">MAAS</a> (Metal-as-a-Service) and <a href="https://www.openstack.org/">Openstack</a>, on which self-service 
Kubernetes clusters can be deployed.</p>

<p>This first post will cover the deployment of MAAS as the base building block of this on-premises solution.  MAAS is 
<a href="https://canonical.com/">Canonical’s</a> open source solution for building a self-service cloud of bare metal servers.  It allows you to deploy and 
manage physical servers as though they were virtual instances in a public cloud.</p>

<h1 id="prerequisites">Prerequisites</h1>
<p>In order to replicate the configurations that I’m going to demonstrate, the following minimum requirements must be met;</p>
<ul>
  <li>1 x Physical host or VM to run MAAS. (see <a href="https://maas.io/docs/maas-requirements">https://maas.io/docs/maas-requirements</a>)</li>
  <li>1 x Physical host or VM to run the <a href="https://jaas.ai/">Juju</a> controller. (We’ll use this to deploy Openstack on MAAS)</li>
  <li>2 x Physical hosts with an <a href="https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface">IPMI</a> capable <a href="https://en.wikipedia.org/wiki/Intelligent_Platform_Management_Interface#Baseboard_management_controller">BMC</a>. (Bare minimum for this demonstration - Quad Core, 8G RAM, two network 
interfaces and two storage disks)</li>
  <li>1 x Switch with VLAN support.</li>
  <li>1 x Router/Firewall to provide inter-vlan routing and Internet access.</li>
</ul>

<h2 id="lab-setup">Lab Setup</h2>
<p>My lab environment is representative of a typical <a href="https://en.wikipedia.org/wiki/Brownfield_(software_development)">brownfield</a> deployment so disregard the randomness of names, interfaces, 
vlans etc.  You can substitute VLAN IDs, Subnets, Names etc. to match your own environment.</p>

<h3 id="physical-configuration">Physical Configuration</h3>
<p>On each of the physical servers that will be managed by MAAS, you’ll need a BMC interface, two network interfaces and two storage 
disks.  The network and storage requirements are set by the Openstack installation I’ll demonstrate in a later blog post.  Below
is the configuration I’m using in my environment.</p>

<h4 id="network">Network</h4>
<ul>
  <li><strong>eno1</strong> - PXE Boot and Host Management</li>
  <li><strong>eno2</strong> - Dedicated to Openstack Networking</li>
  <li><strong>mgmt0</strong> - iDrac interface used for IPMI power management</li>
</ul>

<p><img src="/assets/images/20190805/physical.png" alt="alt text" title="Lab Setup" /></p>

<h4 id="storage">Storage</h4>
<ul>
  <li><strong>/dev/sda</strong> - Used for OS Installation</li>
  <li><strong>/dev/sdb</strong> - Dedicated to Openstack Storage</li>
</ul>

<h3 id="logical-configuration">Logical Configuration</h3>
<p>I’m using the following vlans in my lab for this demonstration.  You may substitute these as needed however, it is 
recommended that you use at least three separate vlans.</p>
<ul>
  <li><strong>v11</strong> - IPMI for out of band access to the physical hosts</li>
  <li><strong>v20</strong> - MAAS for PXE Boot and host management</li>
  <li><strong>v193</strong> - IaaS for Openstack virtual machine instances</li>
</ul>

<p><img src="/assets/images/20190805/maas_lab.png" alt="alt text" title="Lab Setup" /></p>

<blockquote>
  <p>Note: I will not be covering the creation of interfaces and vlans on the physical switch or router. Please consult your 
device documentation for guidance on implementing these configurations.</p>
</blockquote>

<h1 id="maas-deployment">MAAS Deployment</h1>
<h2 id="install-maas">Install MAAS</h2>
<p>The installation of MAAS is very simple.  For lab purposes we’ll be installing both the <a href="https://maas.io/docs/introduction-to-controllers">Region Controller</a> and 
<a href="https://maas.io/docs/introduction-to-controllers">Rack Controller</a> service on the same host.  I recommend you start with a clean install of Ubuntu Server 18.04.  For any 
other OS/version please check the MAAS documentation for compatibility.  To get started you can use any of the following 
options;</p>
<ul>
  <li>Quick Start - <a href="https://maas.io/install">https://maas.io/install</a></li>
  <li>Documentation - <a href="https://maas.io/docs/install-from-packages">https://maas.io/docs/install-from-packages</a></li>
</ul>

<p>Or for the impatient, ssh to the machine you’ve dedicated to run MAAS and execute the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>maas-01:~<span class="nv">$ </span><span class="nb">sudo </span>apt-add-repository <span class="nt">-yu</span> ppa:maas/stable
maas-01:~<span class="nv">$ </span><span class="nb">sudo </span>apt update
maas-01:~<span class="nv">$ </span><span class="nb">sudo </span>apt <span class="nb">install </span>maas
maas-01:~<span class="nv">$ </span><span class="nb">sudo </span>maas init
Create first admin account
Username: admin
Password: 
Again: 
Email: admin@localhost
Import SSH keys <span class="o">[]</span> <span class="o">(</span>lp:user-id or gh:user-id<span class="o">)</span>: &lt;your_Launchpad_or_Github_name&gt;
</code></pre></div></div>

<p>We won’t be using the MAAS CLI for now but if you find you have issues running MAAS commands as a non root user you may 
also need to do the following;</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>maas-01:~<span class="nv">$ </span>maas <span class="nt">--help</span>
Traceback <span class="o">(</span>most recent call last<span class="o">)</span>:
  File <span class="s2">"/usr/lib/python3/dist-packages/maascli/config.py"</span>, line 109, <span class="k">in </span>open
    cls.create_database<span class="o">(</span>dbpath<span class="o">)</span>
  File <span class="s2">"/usr/lib/python3/dist-packages/maascli/config.py"</span>, line 93, <span class="k">in </span>create_database
    os.close<span class="o">(</span>os.open<span class="o">(</span>dbpath, os.O_CREAT | os.O_APPEND, 0o600<span class="o">))</span>
PermissionError: <span class="o">[</span>Errno 13] Permission denied: <span class="s1">'/home/&lt;your_user&gt;/.maascli.db'</span>

maas-01:~<span class="nv">$ </span><span class="nb">sudo chown</span> &lt;your_user&gt;:&lt;your_user&gt; .maascli.db
</code></pre></div></div>

<p>You should now be able to login to the MAAS UI at http://(your_maas_ip):5240/MAAS/</p>

<h2 id="configure-maas">Configure MAAS</h2>
<p>The first time you log in to MAAS you’ll be prompted to verify/change some initial settings. Of these the following two
need to be configured for your environment.</p>

<h3 id="dns-forwarder">DNS forwarder</h3>
<p>By default this is set to dns servers used by the host where MAAS is running.</p>

<p><img src="/assets/images/20190805/dns_forwarder.png" alt="alt text" title="DNS Forwarder" /></p>

<h3 id="ssh-keys">SSH Keys</h3>
<p>If you did not specify a Github or Launchpad User-ID during initialization you should upload an RSA public key now.</p>

<p><img src="/assets/images/20190805/upload_ssh_key.png" alt="alt text" title="Upload SSH Key" /></p>

<p>You can make other changes as desired but for now we will leave the defaults.</p>

<p>Now click the <strong>Go to dashboard</strong> button to continue configuring MASS.
<img src="/assets/images/20190805/go_to_dashboard.png" alt="alt text" title="Go to Dashboard" /></p>

<p>Next you will be taken to the <strong>Machines</strong> section of the dashboard where you will be presented with a warning that DHCP 
is not enabled.
<img src="/assets/images/20190805/dhcp_warning.png" alt="alt text" title="DHCP Warning" /></p>

<p>The next step in deploying MAAS is to set up the DHCP configuration that will be used to PXE boot hosts.  Before I cover 
these steps it’s important to review some of MAAS’ <a href="https://maas.io/docs/concepts-and-terms">Concepts and Terms</a> and the different types of DHCP scopes that can 
be configured.  I found the documentation to be a little confusing as it pertains to the setup and configuration of DHCP 
so hopefully this summary will be of use to those deploying MAAS for the first time.</p>

<h3 id="maas-concepts-and-terms">MAAS Concepts and Terms</h3>
<p>For complex, highly available deployments, MAAS supports concepts similar to those found in public clouds such as <em>Regions</em>
and <em>Availability Zones</em>.  For the purposes of this post we will not be focused on those for now.  If you would like 
more information on configuring these features please read the <a href="https://maas.io/docs">MAAS Documentation</a>.</p>

<h4 id="fabric">Fabric</h4>
<p>Is synonymous with a Layer 2 Fabric.  It can be represented by a single switch or cluster of connected switches.
It is assumed that vlan IDs will not be duplicated within a single Fabric.</p>

<h4 id="spaces">Spaces</h4>
<p>Are groupings of logical networks that serve a similar purpose and that may or may not be a part of the same Fabric, Zone 
or Region.  An example use of Spaces would be to restrict the deployment of hosts to specific security zones. We won’t be 
configuring any spaces for now but I wanted to mention their purpose as its a concept unique to MAAS.</p>

<p><img src="/assets/images/20190805/maas_architecture.png" alt="alt text" title="MAAS Networking Concepts" /></p>

<h4 id="machines">Machines</h4>
<p>A machine is a host that can be deployed by MAAS. This can be a physical host or a virtual machine belonging to a <a href="https://www.linux-kvm.org/page/Main_Page">KVM</a> 
host or <a href="https://maas.io/docs/manage-composable-machines">POD</a>.</p>

<h4 id="dhcp-and-reserved-ranges">DHCP and Reserved Ranges</h4>
<p>This to me was one of the harder configuration concepts to work with in MAAS.  After deploying a
few times and trying different options I was finally able to make sense of the documentation.  The following is a summary
of the concepts as I understand them.</p>

<ul>
  <li><strong>DHCP</strong> - At least one <a href="https://maas.io/docs/ip-ranges">Reserved Dynamic Range</a> is needed by MAAS for the <a href="https://maas.io/docs/add-nodes">Enlistment</a> and <a href="https://maas.io/docs/commission-nodes">Commissioning</a> process 
used to discover and manage machines.  The easiest and recommended setup is to enable DHCP on the untagged vlan in which 
the MAAS server resides.</li>
  <li><strong>Managed Subnet</strong> - By default all subnets added to MAAS are considered <em>Managed</em> regardless of whether you choose
to enable MAAS managed DHCP for them.
    <ul>
      <li><strong>Reserved Ranges</strong> - A reserved range in a <em>Managed</em> subnet will never be utilized by MAAS.  However, any IP in the subnet that is 
  outside of the reserved ranges can be statically assigned to hosts when they are deployed.  Reserved ranges in 
  this case are to be used to exclude the assignment of IPs that may be in use by routers, switches and other infrastructure 
  devices within the subnet.</li>
      <li><strong>Reserved Dynamic Ranges</strong> - A dynamic range in a <em>Managed</em> subnet can be used for <em>Enlistment</em> and <em>Commissioning</em> as previously 
  mentioned but can also be used to assign DHCP addresses to hosts during deployment.  This will actually set the deployed 
  hosts’ /etc/network/interfaces or <a href="https://netplan.io/">Netplan</a> configuration to use DHCP.</li>
    </ul>
  </li>
  <li><strong>Unmanaged Subnet</strong> - You must explicitly configure a subnet added to MAAS as <em>Unmanaged</em>.  This assumes that there 
may be an external DHCP server allocating addresses for this subnet.
    <ul>
      <li><strong>Reserved Ranges</strong> - Reserved ranges in an <em>Unmanaged</em> Subnet constitute the only addresses that may be statically assigned by MAAS 
  during host deployment.  You must also add this same range to a DHCP exclusion list in any external DHCP server used 
  for this subnet.</li>
      <li><strong>Reserved Dynamic Ranges</strong> - Dynamic ranges can not be configured in an <em>Unmanaged</em> subnet.</li>
    </ul>
  </li>
</ul>

<blockquote>
  <p>Until you feel you’ve mastered the difference between these concepts it is recommended that you start with a single DHCP
range managed by MAAS for <em>Enlistment</em> and <em>Commissioning</em> purposes.</p>
</blockquote>

<h3 id="configure-dhcp">Configure DHCP</h3>
<p>To configure the DHCP requirements we need to enable MAAS managed DHCP on the default untagged vlan utilized by the MAAS 
server.</p>

<blockquote>
  <p>This vlan needs to be configured in your physical switch(s) as the access or <a href="https://en.wikipedia.org/wiki/IEEE_802.1Q">native vlan</a> (untagged) to all hosts that will be 
managed by MAAS.  If it is not configured as untagged to your physical hosts the PXE boot process may not work.</p>
</blockquote>

<p>Click on the <strong>Subnets</strong> tab in the MAAS GUI navigation, then click the link for the vlan named <strong>untagged</strong>
<img src="/assets/images/20190805/configure_dhcp_1.png" alt="alt text" title="Configure DHCP" /></p>

<p>Click the <strong>Enable DHCP</strong> button
<img src="/assets/images/20190805/configure_dhcp_2.png" alt="alt text" title="Configure DHCP" /></p>

<p>To configure the <em>Reserved Dynamic Range</em> enter a start and end IP address for the range that MAAS will use for DHCP.
I’ve added a comment as a reminder that this range is primarily used for enlistment and commissioning. Click the <strong>Configure DHCP</strong> 
button to apply the settings.
<img src="/assets/images/20190805/configure_dhcp_3.png" alt="alt text" title="Configure DHCP" /></p>

<p>Now under the <strong>Reserved ranges</strong> section click the drop down menu to the right and select <strong>Reserve range</strong>.<br />
<img src="/assets/images/20190805/configure_dhcp_4.png" alt="alt text" title="Configure DHCP" /></p>

<p>Since this subnet is a <em>Managed</em> subnet this range of IPs will never be assigned (statically or dynamically) by MAAS.<br />
We are going to use this range to block off the first ten IPs of the subnet for use by network infrastructure devices.
Enter the range of IPs and then click the <strong>Reserve</strong> button.
<img src="/assets/images/20190805/configure_dhcp_5.png" alt="alt text" title="Configure DHCP" /></p>

<p>At this point we have configured MAAS to manage the allocation of the default vlan/subnet as follows;</p>
<ul>
  <li><strong>10.1.20.1-10.1.20.10</strong> - MAAS will never allocate IPs in this range.</li>
  <li><strong>10.1.20.11-10.1.20.199</strong> - MAAS may use this range to allocate static IPs to deployed hosts.</li>
  <li><strong>10.1.20.200-10.1.20.254</strong> - MAAS will use this range for Enlistment and Commissioning and may use it to assign DHCP 
addresses to deployed hosts.</li>
</ul>

<h1 id="machine-deployment">Machine Deployment</h1>
<p>Now that we have completed the initial configuration of MAAS we can prepare the physical hosts/machines that will be 
managed by MAAS.</p>

<h2 id="configure-hosts">Configure Hosts</h2>
<h3 id="physical-networking">Physical Networking</h3>
<p>You should verify your physical host cabling and switch configurations to ensure compatibility with the vlans and subnets
that we have added to MAAS.</p>

<h3 id="physical-storage">Physical Storage</h3>
<p>If you have a hardware RAID controller in any of your hosts it will need to be configured prior to adding the host to MAAS.
If this is not done MAAS may not be able to detect storage disks during Enlistment or Commissioning.</p>

<h3 id="pxe-boot">PXE Boot</h3>
<p>At least one interface on your physical hosts should be enabled for PXE boot.  As mentioned before this interface should 
be assigned to an untagged switch port in the vlan in which the MAAS server resides.</p>

<h4 id="enable-pxe-boot-in-bios">Enable PXE Boot in BIOS</h4>
<p>I’m using older Dell R610s with iDrac6 in my lab and will use one of the on board NICs to PXE boot the servers.
<img src="/assets/images/20190805/pxe_boot_1.png" alt="alt text" title="Enable PXE Boot" /></p>

<h4 id="configure-bios-boot-order">Configure BIOS Boot Order</h4>
<p>It is important that the PXE enabled interface be placed higher in the boot order than any physical disks.  MAAS will take 
care of ensuring that the server boots from network or disk as needed.
<img src="/assets/images/20190805/pxe_boot_2.png" alt="alt text" title="Configure Boot Order" /></p>

<h3 id="ipmi">IPMI</h3>
<p>If your servers have an onboard BMC similar to Dell’s <a href="https://en.wikipedia.org/wiki/Dell_DRAC">DRAC</a> or HP’s <a href="https://en.wikipedia.org/wiki/HP_Integrated_Lights-Out">iLO</a> you can take full advantage of MAAS’ power 
control and zero-touch-provisioning capabilities.</p>

<h4 id="ipmi-settings">IPMI Settings</h4>
<p>For my Dell servers all I had to do was enable IPMI in the DRAC management interface.
<img src="/assets/images/20190805/ipmi_settings.png" alt="alt text" title="IPMI Settings" /></p>

<h4 id="ipmi-testing">IPMI Testing</h4>
<p>In order for MAAS to control the power of your servers with IPMI, MAAS will need to be able to reach the servers BMC 
interface via UDP port 623.  You can test that this is working before adding your servers to MAAS by installing <strong>ipmitool</strong> 
on your MAAS server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>maas-01:~<span class="nv">$ </span><span class="nb">sudo </span>apt <span class="nb">install </span>ipmitool
maas-01:~<span class="nv">$ </span>ipmitool <span class="nt">-I</span> lanplus <span class="nt">-H</span> &lt;host_ip&gt; <span class="nt">-U</span> &lt;user&gt; <span class="nt">-P</span> &lt;password&gt; sdr elist all
Temp             | 01h | ns  |  3.1 | No Reading
Temp             | 02h | ns  |  3.2 | No Reading
Temp             | 05h | ns  | 10.1 | No Reading
Ambient Temp     | 07h | ns  | 10.1 | Disabled
Temp             | 06h | ns  | 10.2 | No Reading
Ambient Temp     | 08h | ns  | 10.2 | Disabled
Ambient Temp     | 0Eh | ns  |  7.1 | No Reading
Planar Temp      | 0Fh | ns  |  7.1 | No Reading
CMOS Battery     | 10h | ns  |  7.1 | No Reading
....
...
..
</code></pre></div></div>
<p>To use ipmitool, you must use a username and password that has already been configured in your BMC and that has permissions 
to query IPMI information over LAN.  By default MAAS will create its own user account during the Enlistment phase.</p>

<h2 id="add-hosts-to-maas">Add Hosts to MAAS</h2>
<p>MAAS manages physical and virtual machines using what it calls a <a href="https://maas.io/how-it-works">node lifecycle</a>.  I’m going to focus mainly on the the 
following phases of this lifecycle for now.</p>
<ul>
  <li>Enlistment</li>
  <li>Commissioning</li>
  <li>Deployment</li>
  <li>Release</li>
</ul>

<p>In both the Enlistment and Commissioning phases a machine will undergo the following process;</p>
<ol>
  <li>DHCP server is contacted</li>
  <li>kernel and initrd are received over TFTP</li>
  <li>machine boots</li>
  <li>initrd mounts a Squashfs image ephemerally over HTTP</li>
  <li>cloud-init runs enlistment and built-in commissioning scripts</li>
  <li>machine shuts down</li>
</ol>

<p>The key difference between the two phases is Enlistment is primarily used for initial discovery and Commissioning allows
for additional hardware customization and testing.</p>

<h3 id="enlistment">Enlistment</h3>
<p>Adding a host to MAAS is typically done via a combination of DHCP, TFTP and PXE. This unattended manner of adding a host 
is called <em>Enlistment</em>.</p>

<blockquote>
  <p>Note: MAAS runs built-in commissioning scripts during the enlistment phase so that when you commission a host, any customised 
commissioning scripts you add will have access to data collected during enlistment.</p>
</blockquote>

<p>If everything has been configured correctly the only required step to start the Enlistment process is to power on the machine
you are adding to MAAS.  You can watch the Enlistment process discovering a new host in the following video;</p>

<!-- Courtesy of embedresponsively.com -->

<div class="responsive-video-container">
    <iframe src="https://www.youtube-nocookie.com/embed/jj1M-YyCgD4" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
  </div>

<h4 id="ipmi-setup-verification-in-maas">IPMI Setup Verification in MAAS</h4>
<p>Once the Enlistment process is complete you can verify that MAAS was able to discover/configure the IPMI information needed 
to manage power settings on your host.</p>

<p>Navigate to <strong>Machines</strong> in the MAAS GUI menu.  Click the automatically assigned host name of the newly discovered host,
(we’ll change this later) then navigate to the <strong>Configuration</strong> tab.
<img src="/assets/images/20190805/ipmi_verification_1.png" alt="alt text" title="IPMI Verification" /></p>

<blockquote>
  <p>Here you can see that the IP address and power management type have been discovered and that MAAS has configured a username
and password to authenticate with the host’s BMC.</p>
</blockquote>

<h4 id="ipmi-setup-verification-on-host">IPMI Setup Verification on Host</h4>
<p>You can also confirm that a <strong>maas</strong> user was automatically created in your hosts BMC management interface.
<img src="/assets/images/20190805/ipmi_verification_2.png" alt="alt text" title="IPMI Verification" /></p>

<h3 id="commissioning">Commissioning</h3>
<p>Once a host has been added to MAAS via the Enlistment process, the next step is to Commission it.
You have the option of selecting some extra parameters (like whether to leave the host running and accessible via ssh) 
and can perform additional hardware tests during this phase.</p>

<p>Prior to Commissioning a host I prefer to change the automatically assigned hostname.  This is purely optional but if you
would like to change the hostname, click the current hostname in the top left corner and set the name to something more 
identifiable for your environment.
<img src="/assets/images/20190805/change_name.png" alt="alt text" title="Configure Hostname" /></p>

<p>To start the commissioning process, click the <strong>Take Action</strong> button in the top right corner and select <strong>Commission</strong> 
from the drop down list.
<img src="/assets/images/20190805/commission_1.png" alt="alt text" title="Commission Host" /></p>

<p>Now you can specify a few configuration options as well as select specific hardware tests to run.  You can leave the default
settings for now and click the <strong>Commission machine</strong> button.
<img src="/assets/images/20190805/commission_2.png" alt="alt text" title="Commission Host" /></p>

<p>You can watch the Commissioning process run its scripts and
perform hardware tests in the following video;</p>

<!-- Courtesy of embedresponsively.com -->

<div class="responsive-video-container">
    <iframe src="https://www.youtube-nocookie.com/embed/k-9VHZg_qoo" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
  </div>

<blockquote>
  <p>Once a host is commissioned its status will change to Ready.</p>
</blockquote>

<h2 id="deploy-hosts">Deploy Hosts</h2>
<p>During the deployment phase MAAS utilizes a process similar to Commissioning to deploy an operating system with <a href="https://curtin.readthedocs.io/en/latest/">Curtain</a>. 
Network and Storage configurations are also applied as part of this process.</p>
<ol>
  <li>DHCP server is contacted</li>
  <li>kernel and initrd are received over TFTP</li>
  <li>machine boots</li>
  <li>initrd mounts a Squashfs image ephemerally over HTTP</li>
  <li>cloud-init triggers deployment process
    <ol>
      <li>curtin installation script is run</li>
      <li>Squashfs image (same as above) is placed on disk</li>
    </ol>
  </li>
</ol>

<h3 id="maas-network-model">MAAS Network Model</h3>
<p>Once a machine is in the Ready state (post-commissioning) it’s intended network and storage configuration can be defined 
in MAAS.  Interfaces can be added/removed, attached to a fabric, linked to a subnet, and provided an IP assignment mode.<br />
This is done using a model that represents the desired network configuration for a deployed host.  This allows MAAS to 
deploy consistent networking across multiple operating systems regardless of whether they employ <a href="https://en.wikipedia.org/wiki/NetworkManager">Network Manager</a>, 
<a href="http://man7.org/linux/man-pages/man8/systemd-networkd.service.8.html">systemd-networkd</a> or <a href="https://netplan.io/">Netplan</a>.</p>

<p>MAAS will automatically discover the host interfaces and the vlan/subnet configuration used by the PXE enabled interface.  The
default configuration is all that is needed for this demonstration but MAAS is capable of provisioning advanced network
settings such as LACP Bonds, VLAN interfaces and bridges.</p>

<p>To view the current network configuration Navigate to the <strong>Interfaces</strong> tab of the host.  Notice that the <strong>IP Mode</strong> is
set to <strong>Auto assign</strong> this means that when the host is deployed MAAS will automatically configure a static IP address on
the host.  This is not the same as enabling DHCP in the post-deployment configuration.</p>

<p><img src="/assets/images/20190805/network_model.png" alt="alt text" title="Network Model" /></p>

<p>If you prefer you can also change
this setting to <strong>Static assign</strong> and manually set the preferred IP address yourself.</p>

<p><img src="/assets/images/20190805/static_assign.png" alt="alt text" title="Static Assign" /></p>

<blockquote>
  <p>No configuration is needed in MAAS for eno2.  However, you need to make sure that this interface is assigned to the vlan 
you want to use with Openstack on your physical switch.</p>
</blockquote>

<h3 id="configure-maas-storage-model">Configure MAAS Storage Model</h3>
<p>The last requirement before deploying a host is to configure the storage disks and filesystems.</p>

<p>Navigate to the <strong>Storage</strong> tab of the hosts configuration menu and review any existing configuration that may have been 
discovered when the host was added to MAAS.  If no existing partitions and filesystems exist this tab will only show the
disks that were discovered.</p>

<blockquote>
  <p>For simplicity I’m going to keep the default storage layout setting of <strong>Flat</strong>.</p>
</blockquote>

<p>Next, click the button under <strong>Actions</strong> to the right of the disk you want to configure and select <strong>Add partition</strong>.
<img src="/assets/images/20190805/configure_disk_1.png" alt="alt text" title="Configure Disks" /></p>

<p>Configure the <strong>Size</strong>, <strong>Filesystem</strong> type and <strong>Mount point</strong> you to want provision.  For lab purposes I’m going to use 
the simplest possible configuration of a single ext4 partition mounted at the root of the filesystem.
<img src="/assets/images/20190805/configure_disk_2.png" alt="alt text" title="Configure Disks" />
Click the <strong>Add partition</strong> button after filling in these settings.</p>

<blockquote>
  <p>Do not configure any partitions or filesystems on the remaining disk (sdb).  The Openstack deployment will fail if it 
can not detect an unused disk.</p>
</blockquote>

<h3 id="deployment">Deployment</h3>
<p>To deploy the host click the <strong>Take action</strong> button in the top right corner and select <strong>Deploy</strong> from the drop down list.
You can watch as the Deployment process provisions the storage, OS and network configurations in the following video;</p>

<!-- Courtesy of embedresponsively.com -->

<div class="responsive-video-container">
    <iframe src="https://www.youtube-nocookie.com/embed/Z4DYUagjjcA" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>
  </div>

<h3 id="verify-deployment">Verify Deployment</h3>
<p>Once the Deployment process has completed your host should be left powered on with a newly provisioned 
operating system (Ubuntu 18.04 unless otherwise specified).  If the host was deployed successfully, it’s status on the
Machines page of the MAAS GUI should show the version of the operating system that was deployed.
<img src="/assets/images/20190805/os_deployed.png" alt="alt text" title="Verify Deployment" /></p>

<p>You should also be able to ssh to the host using the default 
username <strong>ubuntu</strong> and the SSH key you specified during the initial installation of the MAAS server.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>host:~<span class="nv">$ </span>ssh <span class="nt">-i</span> ~/.ssh/id_rsa ubuntu@10.1.20.11
</code></pre></div></div>

<p>Once you log in you should verify that your partitions were created as specified and that the MAAS network model properly deployed your 
intended network configuration.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ubuntu@metal-03:~<span class="nv">$ </span><span class="nb">df</span> <span class="nt">-h</span>
Filesystem      Size  Used Avail Use% Mounted on
udev             24G     0   24G   0% /dev
tmpfs           4.8G  1.2M  4.8G   1% /run
/dev/sda1        67G  9.8G   54G  16% /
tmpfs            24G     0   24G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs            24G     0   24G   0% /sys/fs/cgroup
tmpfs           4.8G     0  4.8G   0% /run/user/1000
</code></pre></div></div>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="s">ubuntu@metal-03:~$ cat /etc/netplan/50-cloud-init.yaml</span> 

<span class="nn">---</span>
<span class="na">network</span><span class="pi">:</span>
    <span class="na">ethernets</span><span class="pi">:</span>
        <span class="na">eno1</span><span class="pi">:</span>
            <span class="na">addresses</span><span class="pi">:</span>
            <span class="pi">-</span> <span class="s">10.1.20.13/24</span>
            <span class="na">gateway4</span><span class="pi">:</span> <span class="s">10.1.20.1</span>
            <span class="na">match</span><span class="pi">:</span>
                <span class="na">macaddress</span><span class="pi">:</span> <span class="s">f0:4d:a2:07:c9:2d</span>
            <span class="na">mtu</span><span class="pi">:</span> <span class="m">1500</span>
            <span class="na">nameservers</span><span class="pi">:</span>
                <span class="na">addresses</span><span class="pi">:</span>
                <span class="pi">-</span> <span class="s">10.1.20.5</span>
                <span class="na">search</span><span class="pi">:</span>
                <span class="pi">-</span> <span class="s">maas</span>
            <span class="na">set-name</span><span class="pi">:</span> <span class="s">eno1</span>
        <span class="na">eno2</span><span class="pi">:</span>
            <span class="na">match</span><span class="pi">:</span>
                <span class="na">macaddress</span><span class="pi">:</span> <span class="s">f0:4d:a2:07:c9:2f</span>
            <span class="na">mtu</span><span class="pi">:</span> <span class="m">1500</span>
            <span class="na">set-name</span><span class="pi">:</span> <span class="s">eno2</span>
    <span class="na">version</span><span class="pi">:</span> <span class="m">2</span>
</code></pre></div></div>

<h3 id="release">Release</h3>
<p>Now that I have demonstrated the entire deployment process I’m going to release the host back in to MAAS’ pool of available
machines.  In the next post I’m going to demonstrate how to use Canonical’s <a href="https://jaas.ai/">Juju</a> to deploy a two node Openstack cloud 
with MAAS.</p>

<p>To release the host we just provisioned, go to the the <strong>Machines</strong> page and click the name of the host.  Next click the 
<strong>Take action</strong> button in the top right corner and select <strong>Release</strong> from the drop down list.
<img src="/assets/images/20190805/release_machine.png" alt="alt text" title="Release Machine" /></p>

<blockquote>
  <p>Releasing a host back into the pool of available machines changes a host’s status from ‘Deployed’ to ‘Ready’.
This process includes the ‘Power off’ action. The user has the opportunity to erase the host’s storage (disks) before confirming
the action.</p>
</blockquote>

<h1 id="conclusion">Conclusion</h1>
<p>The steps and examples demonstrated in this post can be used to construct an entire data center of bare metal resources.  According 
to the documentation, the single MAAS rack controller we deployed can service as many as 1000 hosts.  A distributed, highly-available 
deployment of MAAS is designed to support multiple data centers across multiple regions.  This level of scalability makes
it possible to rapidly build a self managed cloud of physical resources much like those used to power the services of existing public 
cloud vendors.</p>]]></content><author><name>2stacks</name></author><category term="Blog" /><category term="IaaS" /><category term="Multi-Cloud" /><category term="MAAS" /><category term="Juju" /><category term="OpenStack" /><category term="Kubernetes" /><summary type="html"><![CDATA[Using MAAS (metal-as-a-service) to build self-service private clouds.]]></summary></entry><entry><title type="html">Rancher 2 and Letsencrypt</title><link href="/blog/rancher-2-and-letsencrypt/" rel="alternate" type="text/html" title="Rancher 2 and Letsencrypt" /><published>2019-05-24T00:00:00+00:00</published><updated>2019-05-24T00:00:00+00:00</updated><id>/blog/rancher-2-and-letsencrypt</id><content type="html" xml:base="/blog/rancher-2-and-letsencrypt/"><![CDATA[<p>I decided to write this post to help with the <a href="https://forums.rancher.com/t/rancher-2-and-letsencrypt/10492">discussion on the Rancher Forum</a> regarding the difficulties many were having
trying to setup Letsencrypt certificates with cert-manager.  I’ve borrowed and owe credit to work that’s already been
documented <a href="https://www.idealcoders.com/posts/rancher/2018/06/rancher-2-x-and-lets-encrypt-with-cert-manager-and-nginx-ingress/">here</a> and I’ll try to stick to the steps I took to enable the full automation of the certificate process.</p>

<blockquote>
  <p>NOTE: This post has been updated to demonstrate usage of a newer version of cert-manager provided by JetPack. Notices
from Letsencrypt have been sent regarding the blocking of versions older than v0.8.0.</p>
</blockquote>

<h2 id="prerequisites">Prerequisites</h2>

<p>I’m going to assume a few things for brevity.</p>

<ul>
  <li>You have a functioning Rancher 2.0 Cluster</li>
  <li>You have kubectl set up with your <a href="https://rancher.com/docs/rancher/v2.x/en/cluster-admin/kubectl/#accessing-clusters-with-kubectl-and-a-kubeconfig-file">Rancher Kubeconfig File</a></li>
  <li>You have a publicly reachable dns service that points the domain, for which you want to issue certificates, to your cluster nodes, loadbalancer or port forwarder if using NAT.</li>
  <li>If your cluster is behind NAT you have set up split DNS. (See section on DNS)</li>
</ul>

<h2 id="installation">Installation</h2>
<h3 id="enable-jetpack-helm-repository">Enable JetPack Helm Repository</h3>

<p>The default library repository in Rancher only includes cert-manager versions up to v0.5.2. The first thing we need to
do is add the JetPack repository to Rancher.</p>

<p>In the Rancher UI navigation go to <strong>Tools</strong> and select <strong>Catalogs</strong>.</p>

<p><img src="/assets/images/20190524/Selection_001.png" alt="Add Catalog" title="Add Catalog" /></p>

<p>Next click the <strong>Add Catalog</strong> button.</p>

<p><img src="/assets/images/20190524/Selection_002.png" alt="Add Catalog" title="Add Catalog" /></p>

<p>Give the new catalog a name like <code class="language-plaintext highlighter-rouge">jetstack</code> and configure <code class="language-plaintext highlighter-rouge">https://charts.jetstack.io</code> as the catalog URL.</p>

<p><img src="/assets/images/20190524/Selection_003.png" alt="Add Catalog" title="Add Catalog" /></p>

<h3 id="install-cert-manager">Install cert-manager</h3>

<p>Lets start by ensuring we are working from a clean slate.  If you have attempted to install cert-manager before remove any
existing resources and verify that the required name space isn’t in use.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>kubectl get all <span class="nt">-n</span> cert-manager
No resources found.
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>kubectl describe clusterissuers letsencrypt-staging
error: the server doesnt have a resource <span class="nb">type</span> <span class="s2">"clusterissuers"</span>
</code></pre></div></div>

<p>Before installing the cert-manager application in Rancher we need to first add the <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#customresourcedefinitions">Customer Resource Definition</a>. From
your workstation execute the following.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl apply <span class="nt">-f</span> https://raw.githubusercontent.com/jetstack/cert-manager/release-0.9/deploy/manifests/00-crds.yaml
</code></pre></div></div>

<p>Now label the kube-system namespace to disable resource validation.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl label namespace kube-system certmanager.k8s.io/disable-validation<span class="o">=</span><span class="nb">true</span>
</code></pre></div></div>

<p>Next navigate to the <strong>Apps</strong> section of the Rancher <strong>System</strong> Project.</p>

<p><img src="/assets/images/20190524/nav_to_apps.png" alt="alt text" title="Navigate to Apps" /></p>

<p>Next click the <strong>Launch</strong> button and type <strong>cert</strong> in the search menu. Click on the cert-manager provided by the
JetStack catalog.</p>

<p><img src="/assets/images/20190524/Selection_009.png" alt="alt text" title="Launch cert-manager" /></p>

<p>Change the namespace to <strong>kube-system</strong> and set the template version to <strong>v0.9.1</strong>.</p>

<blockquote>
  <p>Previous versions of cert-manager used the namespace ‘cert-manager’. Make sure you deploy to kube-system or the
installation will fail.</p>
</blockquote>

<p><img src="/assets/images/20190524/Selection_010.png" alt="alt text" title="Configure cert-manager" /></p>

<h3 id="verify-installation">Verify Installation</h3>

<p>Now verify your app deployment with kubectl.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>kubectl get all <span class="nt">-n</span> kube-system | <span class="nb">grep </span>cert-manager
pod/cert-manager-5b9ff77b7-lhsb4             1/1     Running     0          165m
pod/cert-manager-cainjector-59d69b9b-9f5ng   1/1     Running     0          165m
pod/cert-manager-webhook-cfd6587ff-fz2cv     1/1     Running     0          125m
service/cert-manager-webhook   ClusterIP   10.43.104.116   &lt;none&gt;        443/TCP                  165m
deployment.apps/cert-manager              1/1     1            1           165m
deployment.apps/cert-manager-cainjector   1/1     1            1           165m
deployment.apps/cert-manager-webhook      1/1     1            1           165m
replicaset.apps/cert-manager-5b9ff77b7             1         1         1       165m
replicaset.apps/cert-manager-cainjector-59d69b9b   1         1         1       165m
replicaset.apps/cert-manager-webhook-cfd6587ff     1         1         1       165m
</code></pre></div></div>

<p>Unlike previous versions of the Rancher cert-manager application, you’ll need to create your own <a href="https://docs.cert-manager.io/en/release-0.9/reference/clusterissuers.html">Cluster Issuer</a>. First
create a file similar to the following.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span><span class="nb">cat </span>cluster-issuer.yaml 
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    <span class="c"># The ACME server URL</span>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    <span class="c"># Email address used for ACME registration</span>
    email: 2stacks@2stacks.net
    <span class="c"># Name of a secret used to store the ACME account private key</span>
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    <span class="c"># Enable HTTP01 validations</span>
    http01: <span class="o">{}</span>
</code></pre></div></div>

<p>Now apply the configuration with;</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl create <span class="nt">-f</span> cluster-issuer.yaml
</code></pre></div></div>

<p>Verify the creation of the Cluster Issuer with;</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl describe clusterissuers letsencrypt-
</code></pre></div></div>

<p>The important thing to note is that the cert-manager pod is running and that your email account was successfully
registered with the ACME server API. If your output isn’t similar to above check the logs of the cert-manager pod for
any issues.</p>

<p><img src="/assets/images/20190524/view_logs.png" alt="alt text" title="View Logs" /></p>

<p>Notice in the logs below the message that says <code class="language-plaintext highlighter-rouge">Not syncing ingress default/nginx as it does not contain necessary annotations</code>.
I precreated a test nginx deployment which we are going to use to test the <code class="language-plaintext highlighter-rouge">ingress-shim</code> functionality of cert-manager.</p>

<p><img src="/assets/images/20190524/logs.png" alt="alt text" title="Logs" /></p>

<h2 id="configuring-ingress">Configuring Ingress</h2>
<h3 id="deploy-a-test-workload">Deploy a Test Workload</h3>

<p>I chose to deploy an nginx container as a test since it provides the default server and nginx welcome page without any
configuration.</p>

<p>From the <code class="language-plaintext highlighter-rouge">Workloads</code> section of your chosen Rancher Project click the <code class="language-plaintext highlighter-rouge">Deploy</code> button.  Give the workload a name, choose
the nginx <code class="language-plaintext highlighter-rouge">Docker Image</code> of your choice, leave the NameSpace set to default.  Add a <code class="language-plaintext highlighter-rouge">Port Mapping</code> for port 80 and publish
the service as a <code class="language-plaintext highlighter-rouge">Cluster IP (Internal only)</code>.  Click <code class="language-plaintext highlighter-rouge">Launch</code> to create the new workload.</p>

<p><img src="/assets/images/20190524/new_workload.png" alt="alt text" title="Launch Workload" /></p>

<h3 id="create-an-ingress">Create an Ingress</h3>

<p>Next from the <code class="language-plaintext highlighter-rouge">Load Balancing</code> menu, click the <code class="language-plaintext highlighter-rouge">Add Ingress</code> button.</p>

<p>In the <code class="language-plaintext highlighter-rouge">Add Ingress</code> configuration page, give the Ingress a name and leave the Namespace set to default.  Under the <code class="language-plaintext highlighter-rouge">Rules</code>
section select the option for <code class="language-plaintext highlighter-rouge">Specify a hostname to use</code>.  My test lab is setup to use the domain bsptn.xyz so I have 
configured the <code class="language-plaintext highlighter-rouge">Request Host</code> name as “nginx.bsptn.xyz”</p>

<p>By default Rancher chooses a <code class="language-plaintext highlighter-rouge">Workload</code> as the default for the <code class="language-plaintext highlighter-rouge">Target Backend</code>.  We want to use the service that was
automatically created when we deployed our nginx workload.  Click the minus sign button to the right of the <code class="language-plaintext highlighter-rouge">Port</code> field
to remove the existing Target Backend.</p>

<p><img src="/assets/images/20190524/configure_ingress.png" alt="alt text" title="Configure Ingress" /></p>

<p>Now click the <code class="language-plaintext highlighter-rouge">Service</code> button next to <code class="language-plaintext highlighter-rouge">Target Backend</code>. Set the <code class="language-plaintext highlighter-rouge">Path</code> to “/” and in the <code class="language-plaintext highlighter-rouge">Target</code> drop down select the nginx
service.</p>

<p><img src="/assets/images/20190524/target_backend.png" alt="alt text" title="Target Backend" /></p>

<p>At this point you should save the ingress without configuring any SSL Certificates or Annotations.  You should verify that
you nginx deployment is reachable via the ingress from both the Internet and your internal network.  If you can not then
you have more work to do with DNS, Load Balancing, NAT etc. before you can proceed to the next step.</p>

<p><img src="/assets/images/20190524/nginx_http.png" alt="alt text" title="Nginx Http" /></p>

<h3 id="edit-the-ingress">Edit the Ingress</h3>

<p>At this point if you have verified that your ingress service is reachable you can proceed to adding the annotations required
to automatically request and deploy a certificate.  From the <code class="language-plaintext highlighter-rouge">Load Balancing</code> menu click the drop down to the far right of
the nginx ingress and then select <code class="language-plaintext highlighter-rouge">edit</code></p>

<p><img src="/assets/images/20190524/edit_ingress.png" alt="alt text" title="Edit Ingress" /></p>

<p>Scroll to the bottom of the page and expand the <code class="language-plaintext highlighter-rouge">SSL/TLS Certificates</code> and <code class="language-plaintext highlighter-rouge">Labels &amp; Annotations</code>  sections.  First click
the <code class="language-plaintext highlighter-rouge">Add Certificate</code> button under the SSL/TLS section.  Leave the option set for <code class="language-plaintext highlighter-rouge">Use default ingress controller certificate</code>.
Don’t worry we will manually edit this in the Yaml later.  Set the <code class="language-plaintext highlighter-rouge">Host</code> section to the FQDN of your service.</p>

<p><img src="/assets/images/20190524/add_cert.png" alt="alt text" title="Add Certificate" /></p>

<p>Now you need to add the annotations as per the <a href="https://cert-manager.readthedocs.io/en/latest/tasks/issuing-certificates/ingress-shim.html#supported-annotations">ingress-shim</a> documentation.  If you forget the required annotations you
can view the docs.  They are also provided in the <code class="language-plaintext highlighter-rouge">Notes</code> section when you first launched the cert-manager app.</p>

<p><img src="/assets/images/20190524/shim_settings.png" alt="alt text" title="Ingress-shim" /></p>

<p>Under the <code class="language-plaintext highlighter-rouge">Labels &amp; Annotations</code> section click the <code class="language-plaintext highlighter-rouge">Add Annotation</code> button twice and add the following annotations.</p>
<ul>
  <li><code class="language-plaintext highlighter-rouge">kubernetes.io/tls-acme: "true"</code></li>
  <li><code class="language-plaintext highlighter-rouge">certmanager.k8s.io/cluster-issuer: letsencrypt-staging</code></li>
</ul>

<p>Now click <code class="language-plaintext highlighter-rouge">Save</code> to update the Ingress.</p>

<p><img src="/assets/images/20190524/save_ingress.png" alt="alt text" title="Save Ingress Changes" /></p>

<p>The final step can not be performed through the Rancher Gui at this time.  We’ll need to manually edit the Yaml of the Ingress
we just created.</p>

<p>From the <code class="language-plaintext highlighter-rouge">Load Balancing</code> menu click the drop down to the far right of the nginx ingress and then select <code class="language-plaintext highlighter-rouge">View/Edit YAML</code>.</p>

<p><img src="/assets/images/20190524/edit_yaml.png" alt="alt text" title="Edit YAML" /></p>

<p>Scroll the bottom of the Yaml config and under <code class="language-plaintext highlighter-rouge">spec -&gt; tls -&gt; hosts</code> add the <code class="language-plaintext highlighter-rouge">secretName</code> definition with the resource 
name you want the certificate to be saved with.  I’ve chosen <code class="language-plaintext highlighter-rouge">nginx-bsptn-xyz-crt</code> for my implementation.  Now click the
save button.</p>

<p><img src="/assets/images/20190524/add_secretname.png" alt="alt text" title="Add SecretName" /></p>

<p>If everything to this point has gone well and if you watch carefully, cert-manager will temporarily create a new Ingress
for the purposes of performing HTTP01 challenge verification.</p>

<p><img src="/assets/images/20190524/challenge_ingress.png" alt="alt text" title="Challenge Ingress" /></p>

<p>If you missed it or if something went wrong now is a good time to review the cert-manager logs.  I’ve included my logs
from the last time the ingress-shim failed due to missing configurations up until the certificate is pulled and the temporary
HTTP01 challenge Ingress is removed.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>E0530 23:01:37.571902 1 controller.go:177] ingress-shim controller: Re-queuing item <span class="s2">"default/nginx"</span> due to error processing: TLS entry 0 <span class="k">for </span>ingress <span class="s2">"nginx"</span> must specify a secretName
I0530 23:02:37.572406 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/nginx'</span>
I0530 23:02:37.598364 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/nginx"</span>
I0530 23:02:37.598406 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/nginx'</span>
I0530 23:02:37.598430 1 sync.go:140] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> already exists
I0530 23:02:37.598462 1 sync.go:143] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> is up to <span class="nb">date
</span>I0530 23:02:37.598480 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/nginx"</span>
I0530 23:02:39.598243 1 controller.go:171] certificates controller: syncing item <span class="s1">'default/nginx-bsptn-xyz-crt'</span>
I0530 23:02:39.598704 1 sync.go:274] Preparing certificate default/nginx-bsptn-xyz-crt with issuer
I0530 23:02:39.599901 1 prepare.go:263] Cleaning up previous order <span class="k">for </span>certificate default/nginx-bsptn-xyz-crt
I0530 23:02:39.599939 1 prepare.go:279] Cleaning up old/expired challenges <span class="k">for </span>Certificate default/nginx-bsptn-xyz-crt
I0530 23:02:39.599998 1 logger.go:38] Calling CreateOrder
I0530 23:02:40.148955 1 acme.go:126] Created order <span class="k">for </span>domains: <span class="o">[{</span>dns nginx.bsptn.xyz<span class="o">}]</span>
I0530 23:02:40.149074 1 logger.go:73] Calling GetAuthorization
I0530 23:02:40.212749 1 logger.go:93] Calling HTTP01ChallengeResponse
I0530 23:02:40.212917 1 prepare.go:279] Cleaning up old/expired challenges <span class="k">for </span>Certificate default/nginx-bsptn-xyz-crt
I0530 23:02:40.212949 1 logger.go:68] Calling GetChallenge
I0530 23:02:40.340693 1 pod.go:65] No existing HTTP01 challenge solver pod found <span class="k">for </span>Certificate <span class="s2">"default/nginx-bsptn-xyz-crt"</span><span class="nb">.</span> One will be created.
I0530 23:02:40.394210 1 service.go:51] No existing HTTP01 challenge solver service found <span class="k">for </span>Certificate <span class="s2">"default/nginx-bsptn-xyz-crt"</span><span class="nb">.</span> One will be created.
I0530 23:02:40.506689 1 ingress.go:49] Looking up Ingresses <span class="k">for </span>selector certmanager.k8s.io/acme-http-domain<span class="o">=</span>806880787,certmanager.k8s.io/acme-http-token<span class="o">=</span>1172442703
I0530 23:02:40.506761 1 ingress.go:102] No existing HTTP01 challenge solver ingress found <span class="k">for </span>Certificate <span class="s2">"default/nginx-bsptn-xyz-crt"</span><span class="nb">.</span> One will be created.
I0530 23:02:40.595694 1 helpers.go:194] Setting lastTransitionTime <span class="k">for </span>Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> condition <span class="s2">"Ready"</span> to 2019-05-30 23:02:40.595666151 +0000 UTC <span class="nv">m</span><span class="o">=</span>+8461.551329830
I0530 23:02:40.595767 1 sync.go:276] Error preparing issuer <span class="k">for </span>certificate default/nginx-bsptn-xyz-crt: http-01 self check failed <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span>
E0530 23:02:40.595863 1 sync.go:197] <span class="o">[</span>default/nginx-bsptn-xyz-crt] Error getting certificate <span class="s1">'nginx-bsptn-xyz-crt'</span>: secret <span class="s2">"nginx-bsptn-xyz-crt"</span> not found
E0530 23:02:40.711924 1 controller.go:180] certificates controller: Re-queuing item <span class="s2">"default/nginx-bsptn-xyz-crt"</span> due to error processing: http-01 self check failed <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span>
I0530 23:02:40.721555 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/nginx'</span>
I0530 23:02:40.723070 1 sync.go:140] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> already exists
I0530 23:02:40.723508 1 sync.go:143] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> is up to <span class="nb">date
</span>I0530 23:02:40.723762 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/nginx"</span>
I0530 23:02:44.713550 1 controller.go:171] certificates controller: syncing item <span class="s1">'default/nginx-bsptn-xyz-crt'</span>
I0530 23:02:44.713701 1 sync.go:274] Preparing certificate default/nginx-bsptn-xyz-crt with issuer
I0530 23:02:44.714024 1 logger.go:43] Calling GetOrder
I0530 23:02:44.808517 1 logger.go:73] Calling GetAuthorization
I0530 23:02:44.897577 1 logger.go:93] Calling HTTP01ChallengeResponse
I0530 23:02:44.897678 1 prepare.go:279] Cleaning up old/expired challenges <span class="k">for </span>Certificate default/nginx-bsptn-xyz-crt
I0530 23:02:44.897711 1 logger.go:68] Calling GetChallenge
I0530 23:02:44.973655 1 http.go:134] wrong status code <span class="s1">'503'</span>
I0530 23:02:44.974074 1 ingress.go:49] Looking up Ingresses <span class="k">for </span>selector certmanager.k8s.io/acme-http-domain<span class="o">=</span>806880787,certmanager.k8s.io/acme-http-token<span class="o">=</span>1172442703
I0530 23:02:44.974202 1 helpers.go:201] Found status change <span class="k">for </span>Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> condition <span class="s2">"Ready"</span>: <span class="s2">"False"</span> -&gt; <span class="s2">"False"</span><span class="p">;</span> setting lastTransitionTime to 2019-05-30 23:02:44.974192204 +0000 UTC <span class="nv">m</span><span class="o">=</span>+8465.929855813
I0530 23:02:44.974303 1 sync.go:276] Error preparing issuer <span class="k">for </span>certificate default/nginx-bsptn-xyz-crt: http-01 self check failed <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span>
E0530 23:02:44.974380 1 sync.go:197] <span class="o">[</span>default/nginx-bsptn-xyz-crt] Error getting certificate <span class="s1">'nginx-bsptn-xyz-crt'</span>: secret <span class="s2">"nginx-bsptn-xyz-crt"</span> not found
I0530 23:02:45.004974 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/nginx'</span>
I0530 23:02:45.005147 1 sync.go:140] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> already exists
I0530 23:02:45.005183 1 sync.go:143] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> is up to <span class="nb">date
</span>I0530 23:02:45.005248 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/nginx"</span>
E0530 23:02:45.008794 1 controller.go:180] certificates controller: Re-queuing item <span class="s2">"default/nginx-bsptn-xyz-crt"</span> due to error processing: http-01 self check failed <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span>
I0530 23:02:45.599202 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/cm-acme-http-solver-wtcbm'</span>
I0530 23:02:45.599414 1 sync.go:65] Not syncing ingress default/cm-acme-http-solver-wtcbm as it does not contain necessary annotations
I0530 23:02:45.599638 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/cm-acme-http-solver-wtcbm"</span>
I0530 23:03:01.005455 1 controller.go:171] certificates controller: syncing item <span class="s1">'default/nginx-bsptn-xyz-crt'</span>
I0530 23:03:01.006291 1 sync.go:274] Preparing certificate default/nginx-bsptn-xyz-crt with issuer
I0530 23:03:01.007303 1 logger.go:43] Calling GetOrder
I0530 23:03:01.187043 1 logger.go:73] Calling GetAuthorization
I0530 23:03:01.271914 1 logger.go:93] Calling HTTP01ChallengeResponse
I0530 23:03:01.272196 1 prepare.go:279] Cleaning up old/expired challenges <span class="k">for </span>Certificate default/nginx-bsptn-xyz-crt
I0530 23:03:01.272583 1 logger.go:68] Calling GetChallenge
I0530 23:03:11.760008 1 prepare.go:488] Accepting challenge <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span>
I0530 23:03:11.760125 1 logger.go:63] Calling AcceptChallenge
I0530 23:03:12.179202 1 prepare.go:500] Waiting <span class="k">for </span>authorization <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span>
I0530 23:03:12.179361 1 logger.go:78] Calling WaitAuthorization
I0530 23:03:14.383630 1 prepare.go:510] Successfully authorized domain <span class="s2">"nginx.bsptn.xyz"</span>
I0530 23:03:14.383916 1 prepare.go:303] Cleaning up challenge <span class="k">for </span>domain <span class="s2">"nginx.bsptn.xyz"</span> as part of Certificate default/nginx-bsptn-xyz-crt
I0530 23:03:14.642912 1 ingress.go:49] Looking up Ingresses <span class="k">for </span>selector certmanager.k8s.io/acme-http-domain<span class="o">=</span>806880787,certmanager.k8s.io/acme-http-token<span class="o">=</span>1172442703
I0530 23:03:14.674932 1 sync.go:281] Issuing certificate...
I0530 23:03:14.675267 1 logger.go:43] Calling GetOrder
I0530 23:03:15.898507 1 logger.go:58] Calling FinalizeOrder
I0530 23:03:16.872750 1 issue.go:196] successfully obtained certificate: <span class="nv">cn</span><span class="o">=</span><span class="s2">"nginx.bsptn.xyz"</span> <span class="nv">altNames</span><span class="o">=[</span>nginx.bsptn.xyz] <span class="nv">url</span><span class="o">=</span><span class="s2">"https://acme-staging-v02.api.letsencrypt.org/acme/order/9448784/35891936"</span>
I0530 23:03:16.931122 1 sync.go:300] Certificate issued successfully
I0530 23:03:16.931385 1 helpers.go:201] Found status change <span class="k">for </span>Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> condition <span class="s2">"Ready"</span>: <span class="s2">"False"</span> -&gt; <span class="s2">"True"</span><span class="p">;</span> setting lastTransitionTime to 2019-05-30 23:03:16.931293187 +0000 UTC <span class="nv">m</span><span class="o">=</span>+8497.886956711
I0530 23:03:16.932560 1 sync.go:206] Certificate default/nginx-bsptn-xyz-crt scheduled <span class="k">for </span>renewal <span class="k">in </span>1438 hours
I0530 23:03:16.951754 1 controller.go:185] certificates controller: Finished processing work item <span class="s2">"default/nginx-bsptn-xyz-crt"</span>
I0530 23:03:16.952862 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/nginx'</span>
I0530 23:03:16.953088 1 sync.go:140] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> already exists
I0530 23:03:16.953906 1 sync.go:143] Certificate <span class="s2">"nginx-bsptn-xyz-crt"</span> <span class="k">for </span>ingress <span class="s2">"nginx"</span> is up to <span class="nb">date
</span>I0530 23:03:16.954138 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/nginx"</span>
I0530 23:03:18.953126 1 controller.go:171] certificates controller: syncing item <span class="s1">'default/nginx-bsptn-xyz-crt'</span>
I0530 23:03:18.954881 1 sync.go:206] Certificate default/nginx-bsptn-xyz-crt scheduled <span class="k">for </span>renewal <span class="k">in </span>1438 hours
I0530 23:03:18.955045 1 controller.go:185] certificates controller: Finished processing work item <span class="s2">"default/nginx-bsptn-xyz-crt"</span>
I0530 23:03:19.674777 1 controller.go:168] ingress-shim controller: syncing item <span class="s1">'default/cm-acme-http-solver-wtcbm'</span>
E0530 23:03:19.674894 1 controller.go:198] ingress <span class="s1">'default/cm-acme-http-solver-wtcbm'</span> <span class="k">in </span>work queue no longer exists
I0530 23:03:19.674942 1 controller.go:182] ingress-shim controller: Finished processing work item <span class="s2">"default/cm-acme-http-solver-wtcbm"</span>
</code></pre></div></div>
<p><br /></p>

<h2 id="verification">Verification</h2>
<p>If you get a log message similar to <code class="language-plaintext highlighter-rouge">issue.go:196] successfully obtained certificate: cn="nginx.bsptn.xyz"</code> chances are
you are in good shape.  I’ll run through just a couple of things to verify everything is working.</p>

<p>Within the project in which you created your Ingress navigate to <code class="language-plaintext highlighter-rouge">Resources</code> -&gt; <code class="language-plaintext highlighter-rouge">Certificates</code> and verify that your certificate
resource has been created in the cluster.</p>

<p><img src="/assets/images/20190524/verify_certificate.png" alt="alt text" title="Verify Certificate Resource" /></p>

<p>The Rancher UI does’t give a lot of information about the certificate so to view it’s details we’ll need to use Kubectl.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>kubectl get certificates
NAME                  AGE
nginx-bsptn-xyz-crt   26m
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~<span class="nv">$ </span>kubectl describe certificates nginx-bsptn-xyz-crt 
Name:         nginx-bsptn-xyz-crt
Namespace:    default
Labels:       &lt;none&gt;
Annotations:  &lt;none&gt;
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-05-30T23:02:37Z
  Generation:          4
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  <span class="nb">true
    </span>Controller:            <span class="nb">true
    </span>Kind:                  Ingress
    Name:                  nginx
    UID:                   &lt;some_uid&gt;
  Resource Version:        1023500
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/nginx-bsptn-xyz-crt
  UID:                     &lt;some_uid&gt;
Spec:
  Acme:
    Config:
      Domains:
        nginx.bsptn.xyz
      Http 01:
        Ingress:  
  Dns Names:
    nginx.bsptn.xyz
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  nginx-bsptn-xyz-crt
Status:
  Acme:
    Order:
      URL:  https://acme-staging-v02.api.letsencrypt.org/acme/order/&lt;number&gt;/&lt;number&gt;
  Conditions:
    Last Transition Time:  2019-05-30T23:03:16Z
    Message:               Certificate issued successfully
    Reason:                CertIssued
    Status:                True
    Type:                  Ready
    Last Transition Time:  &lt;nil&gt;
    Message:               Order validated
    Reason:                OrderValidated
    Status:                False
    Type:                  ValidateFailed
Events:
  Type    Reason          Age   From          Message
  <span class="nt">----</span>    <span class="nt">------</span>          <span class="nt">----</span>  <span class="nt">----</span>          <span class="nt">-------</span>
  Normal  CreateOrder     26m   cert-manager  Created new ACME order, attempting validation...
  Normal  DomainVerified  25m   cert-manager  Domain <span class="s2">"nginx.bsptn.xyz"</span> verified with <span class="s2">"http-01"</span> validation
  Normal  IssueCert       25m   cert-manager  Issuing certificate...
  Normal  CertObtained    25m   cert-manager  Obtained certificate from ACME server
  Normal  CertIssued      25m   cert-manager  Certificate issued successfully
</code></pre></div></div>
<p><br /></p>

<p>Again, if everything worked under the <code class="language-plaintext highlighter-rouge">Events</code> section you should be able to see that the certificate was issued successfully.</p>

<p>The last obovious verification is to load the nginx web page and verify with the a browser that a LetsEncypt certificate has
been issued from the staging API.  Since I chose to issue certs from the staging API the browser will still generate a certificate
error however, you can see from the certificate details that is has been Issued By <code class="language-plaintext highlighter-rouge">Fake LE Intermediate X1</code>.</p>

<p><img src="/assets/images/20190524/verify_browser.png" alt="alt text" title="Verify Browser" /></p>

<h2 id="additional-notes">Additional Notes</h2>
<p>I’m sure if this process works for you you’ll want to proceed to issuing certs from LetsEncrypt’s production API.  I have
test this exact same procedure using the <code class="language-plaintext highlighter-rouge">letsencrypt-prod</code> cluster issuer on a clean cluster.  I have not attempted to run
more than one cluster issue on the same Rancher cluster.  If you’re ready to issue valid certificates I recommend you delete
the cert-manager app you deployed and start over.  You should be able to follow all of the steps in this post replacing all
instances have <code class="language-plaintext highlighter-rouge">letsencrypt-staging</code> with <code class="language-plaintext highlighter-rouge">letsencrypt-prod</code>.</p>

<h3 id="dns">DNS</h3>

<p>You may or may not have noticed that may cluster nodes have private IP addresses.  I’ll share a little about my setup in case
some of you are attempting to use cert-manager on a private lab network.  I assume that clusters deployed in public clouds
won’t have as many http01 verification issues but I could be wrong.</p>

<ul>
  <li>First, I have a wildcard dns record in AWS Route53 that points *.bsptn.xyz to a device performing NAT for my lab environment.</li>
  <li>That NAT boundary forwards all port 80 and 443 to an L4-7 loadbalancer that services my Kubernetes clusters.</li>
  <li>I have a private DNS server built with <a href="https://www.powerdns.com/">PowerDNS</a> for internal name resolution of private IPs.  I chose PowerDNS because
it provides an API that integrates with the Kubernetes add on service <a href="https://github.com/kubernetes-incubator/external-dns">external-dns</a></li>
  <li>Inside my Kubernetes cluster’s I deploy the Bitnami version of the external-dns application.</li>
</ul>

<p>Any Ingress I create in my clusters is automatically registered in PowerDNS via the external-dns application.  This make the
process of performing HTTP01 verification much easier in my environment.</p>

<p>If you’re interested in more details of how I set up my lab environment feel free to contact me.  I have posted a lot of the work
I’ve done to GitHub <a href="https://github.com/2stacks">@2stacks</a> and I mostly use Terraform so that my deployments are repeatable.</p>]]></content><author><name>2stacks</name></author><category term="Blog" /><category term="Kubernetes" /><category term="Rancher" /><category term="Letsencrypt" /><summary type="html"><![CDATA[How to automate Letsencrypt certificates with Rancher 2 Ingress]]></summary></entry><entry><title type="html">Freeradius-Django</title><link href="/github/freeradius-django/" rel="alternate" type="text/html" title="Freeradius-Django" /><published>2019-03-21T00:00:00+00:00</published><updated>2019-03-21T00:00:00+00:00</updated><id>/github/freeradius-django</id><content type="html" xml:base="/github/freeradius-django/"><![CDATA[<p>Build a Freeradius server with Docker, customized for use with <a href="https://github.com/openwisp/django-freeradius">django-freeradius</a>, a
Freeradius frontend built with Django.</p>]]></content><author><name>2stacks</name></author><category term="Github" /><category term="link" /><category term="Freeradius" /><category term="Docker" /><category term="Django" /><category term="Openwisp" /><summary type="html"><![CDATA[Build a Freeradius server with Docker, customized for use with django-freeradius, a Freeradius frontend built with Django.]]></summary></entry><entry><title type="html">Micropython and ESP8266</title><link href="/hackster.io/Micropython-and-ESP8266/" rel="alternate" type="text/html" title="Micropython and ESP8266" /><published>2019-01-16T00:00:00+00:00</published><updated>2019-01-16T00:00:00+00:00</updated><id>/hackster.io/Micropython-and-ESP8266</id><content type="html" xml:base="/hackster.io/Micropython-and-ESP8266/"><![CDATA[<p>I had a few Arduino projects that I wanted to port over to Micropython since python is my language of choice these days.
Most of my dev boards are some version of ESP8266 and following the instructions <a href="https://docs.micropython.org/en/latest/esp8266/quickref.html">@micropython.org</a> was pretty easy.</p>

<p>Check out the work I’ve done so far on <a href="https://github.com/2stacks/alexa-esp8266">GitHub</a> or on <a href="https://www.hackster.io/2stacks/alexa-trigger-esp8266-181f0d">Hackster.io</a></p>

<p><img src="https://hackster.imgix.net/uploads/attachments/660551/20181119_113805_4I5xasSWWW.jpg?auto=compress%2Cformat&amp;w=900&amp;h=675&amp;fit=min" alt="alt text" title="Prototype" /></p>]]></content><author><name>2stacks</name></author><category term="hackster.io" /><category term="Alexa" /><category term="IoT" /><category term="esp8266" /><summary type="html"><![CDATA[I had a few Arduino projects that I wanted to port over to Micropython since python is my language of choice these days. Most of my dev boards are some version of ESP8266 and following the instructions @micropython.org was pretty easy.]]></summary></entry><entry><title type="html">Rapid Deployment of Web Apps with Docker</title><link href="/github/rapid-deployment-of-web-apps-with-docker/" rel="alternate" type="text/html" title="Rapid Deployment of Web Apps with Docker" /><published>2017-04-08T00:00:00+00:00</published><updated>2017-04-08T00:00:00+00:00</updated><id>/github/rapid-deployment-of-web-apps-with-docker</id><content type="html" xml:base="/github/rapid-deployment-of-web-apps-with-docker/"><![CDATA[<p>I recently started learning Docker and was immediately amazed by the <a href="https://docs.docker.com/compose/overview/">docker-compose</a> feature.  I’ve created a sample web application using <a href="http://nginx.org/en/">NGINX</a> that can be rapidly deployed using <a href="https://docs.docker.com/compose/overview/">docker-compose</a>.  My GitHub repo page walks you through running and testing the application.  It is composed of an <a href="https://www.nginx.com/resources/admin-guide/tcp-load-balancing/">NGINX loadbalancer</a> fronting two <a href="https://hub.docker.com/_/nginx/">NGINX web servers</a>.  The demo can be downloaded at <a href="https://github.com/2stacks/docker-nginx-lb">2stacks/docker-nginx-lb</a>.</p>]]></content><author><name>2stacks</name></author><category term="Github" /><category term="Docker" /><category term="NGINX" /><summary type="html"><![CDATA[I recently started learning Docker and was immediately amazed by the docker-compose feature. I’ve created a sample web application using NGINX that can be rapidly deployed using docker-compose. My GitHub repo page walks you through running and testing the application. It is composed of an NGINX loadbalancer fronting two NGINX web servers. The demo can be downloaded at 2stacks/docker-nginx-lb.]]></summary></entry><entry><title type="html">Alexa Trigger ESP8266</title><link href="/hackster.io/alexa-trigger-esp8266/" rel="alternate" type="text/html" title="Alexa Trigger ESP8266" /><published>2017-03-14T00:00:00+00:00</published><updated>2017-03-14T00:00:00+00:00</updated><id>/hackster.io/alexa-trigger-esp8266</id><content type="html" xml:base="/hackster.io/alexa-trigger-esp8266/"><![CDATA[<p>Use Alexa to voice control a relay connected to the Internet via ESP8266.</p>

<p><img src="https://hackster.imgix.net/uploads/attachments/660551/20181119_113805_4I5xasSWWW.jpg?auto=compress%2Cformat&amp;w=900&amp;h=675&amp;fit=min" alt="alt text" title="Prototype" /></p>

<h3 id="story">Story</h3>

<p>I wanted to build a voice controlled smart outlet that was cheap and that didn’t require an existing smart home infrastructure (i.e. SmartThings or HomeKit). There are probably cheaper ways of doing this but I had an unused Power Tail and an ESP8266. This project can be adapted to drive any digital IO pin using Alexa and covers the integration of Alexa, IFTTT and Adafruit.IO.</p>

<p>See project details <a href="https://www.hackster.io/2stacks/alexa-trigger-esp8266-181f0d">@hackster.io</a> or download the code on <a href="https://github.com/2stacks/alexa-esp8266">GitHub</a></p>]]></content><author><name>2stacks</name></author><category term="hackster.io" /><category term="Alexa" /><category term="IoT" /><category term="esp8266" /><summary type="html"><![CDATA[Use Alexa to voice control a relay connected to the Internet via ESP8266.]]></summary></entry></feed>