How To Automate  Creation Of  A Cross-Platform GIS Micro-Environment 

Using Vagrant And VirtualBox

bootstrap responsive templates

This tutorial assumes the reader has some familiarity with the major operating systems: Windows, macOS, Ubuntu and also has some basic knowledge of Linux and DOS sintax.

  1. First of all, what is Vagrant? - according to the official page www.vagrantup.com: "Vagrant is a tool for building and managing virtual machine environments in a single workflow. In other words, instead of manually creating your VirtualBox Machine and run all kind of commands to install your apps, Vagrant helps you put all of this in a file or two and all you have to do is wait for everything to be install and also have a coffee while doing that, it might take a while.
  2. How did the idea of this tutorial came? - Because I am using virtual machines very often for testing purposes before actually running things on the Cloud and because I work both with Windows 10 OS and a Mac OS, I had often have problems replicating what I build on one of the OS to the other. Building a VirtualBox machine is tedus, especially on Windows, if you have to do it multiple times. 
  3. Who is this tutorial for? - If you have tried to install and configure VirtualBox on different OS and had issues enabeling the guest additions and copy/paste from your host machine to the guest virtual machine and struggled to share a folder from your host machine or an external HDD to the guest machine and vice versa, then this tutorial is for you. Vagrant makes installing things and configuring VirtualBox a lot easier for different OS.   

1. Next I will explain how to install Vagrant  for each of the three OS 

But first let's download Vagrant by going to https://www.vagrantup.com, choose the installer that matches your OS and install it.  After this, some Vagrant plugins must be installed, plugins that will make our life a lot easier when we want to run the VirtualBox machine. 

Windows

Click the Start button type cmd and open the windows command prompt and run:
vagrant --version
to check that vagrant has been successfully installed.
To automaticaly set the size of the display for the virtual machine and enable bidirectional copy/paste and drag and drop from host to guest and vice versa, run:
vagrant plugin install vagrant-vbguest
To be able to increase the disk size of our virtual machine, which by default has 10GB of disk space, run:
vagrant plugin install vagrant-disksize

macOS

Open terminal and run the same command lines as in Windows.
Check if vagrant has been successfully installed:
vagrant --version
To automaticaly set the size of the display for the virtual machine and enable bidirectional copy/paste and drag and drop from host to guest run:
vagrant plugin install vagrant-vbguest
To be able to increase the disk size of our virtual machine, which by default has 10GB of disk space, run:
vagrant plugin install vagrant-disksize

Ubuntu

Open terminal and run the same commands.
Check if vagrant has been successfully installed:
vagrant --version
To automaticaly set the size of the display for the virtual machine and enable bidirectional copy/paste and drag and drop from host to guest run:
vagrant plugin install vagrant-vbguest
To be able to increase the disk size of our virtual machine, which by default has 10GB of disk space, run:
vagrant plugin install vagrant-disksize

2. Next, we will install VirtualBox

We do this by going to this link and choose the installer that matches our OS.  

A bit more about Vagrant

  • Vagrant provides easy to configure, reproducible, and portable work environments;
  • Isolates dependencies and their configuration within a single disposable, consistent environment;
  • It gives you a disposable environment and consistent workflow for developing and testing infrastructure management scripts;
  • It will automatically set everything up that is required for that web app in order for you to focus on doing what you do best;
  • It is designed for everyone as the easiest and fastest way to create a virtualized environment!
There are at least seven important commands in Vagrant:

  • vagrant init  - to create the Vagrantfile, if we don't have it already
  • vagrant up - to create and configure guest machines according to your Vagrantfile.
  • vagrant halt - to shut down the running machine Vagrant is managing.
  • vagrant reload - to restart the machine
  • vagrant provision - runs any configured provisioners against the running Vagrant managed machine, or in other words installs anything we want from a file like bootstratp.sh in our case.
  • vagrant package - to packages a currently running VirtualBox or Hyper-V environment into a re-usable box.                                                                                                                               
  • vagrant destroy - if we want to delete the machine.                                                                   All this commands will be explained later in this tutorial.                                                                 The list with all commands can be seen here.

If you want to learn more about Vagrant, check the official website .


3. How does Vagrant work?

The first step to run a Vagrant project is to create a Vagrantfile ! 

  • To do this, first create a new folder named Project where you’ll create your Vagrant machine.
  • Open the command prompt/terminal and run cd:\path_to_your_Project_folder, for Windows. Or for Ubuntu and Mac cd path_to_your_Project_folder
  • Run vagrant init. If you check the content of the folder, you should be able to see the newly created Vagrantfile. So run ls for macOS and Ubuntu or dir for Windows to see the content of the folder.
  • To see the content of the Vargrantfile, open it with a text editor like Notepad++ (for Windows), Notepadqq (for Ubuntu) Coda (for Mac) or Sublime Text. The content is actually the setting up requiremets of the VirtualBox, how much memory, how much disk space, how many cpu's to use etc. Many of the commands are commented out, giving us the option to decide what we want to install. 

So, let’s edit the Vagrantfile we have just created so that we can set up a VirtualBox machine with a GIS micro-environment. 

What we'll install:

  1. Lubuntu Desktop as GUI, to make things a bit more friendly;
  2. QGIS 2.18; Although the latest version of QGIS is now 3.0, it is not very straitforward to install it on Ubuntu and I don't want to make things more complecated then they already are. Feel free to modify the Vagrant file and install the 3.0 version if you want. :);
  3. Latest GDAL version 2.2.2 (QGIS also installs GDAL1.11, but is better to use the latest version, so we will delete version 1.11) and make QGIS use this version (add GDAL environment variable to .bashrc file));
  4. Postgresql 2.10;
  5. PostGIS 2.4.4 and add postgis and postgis topology extentions to postgres database
  6. pgAdmin4, a graphical user interface to PostgreSQL database.

Copy/Paste the content below (between the hashtag lines) to the Vagrantfile inside the folder Project or download Vagrantfile from github.

Please also read the commented lines (the ones that have # at the begining of the line) in this file!

###############################################################

# -*- mode: ruby -*-

# vi: set ft=ruby :

#We will add the 16.04 version of ubuntu

#And change the disk space from 10 GB to 30 GB

Vagrant.configure("2") do |config|

config.vm.box = "ubuntu/xenial64"

config.disksize.size = "30GB"

#We will share the Downloads folder from the Windows host machine with Ubuntu guest OS #from our virtual machine

#Make sure to change this path with the one coresponding to your computer !!!!!!!!!!!!!!!!!!   

#This will help us share the Downloads folder from the Windows machine with the virtual     #machine. But on the guest machine the Downloads folder will be found as vagrant_data

#for Windows OS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Comment next line if you are using macOS or Ubuntu!!!!!!!!!!

config.vm.synced_folder "C:/Users/your_user_name/Downloads", "/vagrant_data"

#For macOS and Ubuntu!!!!!!!!!!!!!!!!!!!!! Uncomment next line if you are using macOS or Ubuntu!!!!!!!!!

#config.vm.synced_folder "/Users/your_user_name/Downloads", "/vagrant_data"

#This is where we pass the bootstrap.sh file so that Vagrant can run install all we need in the #machine.

config.vm.provision :shell, path: "bootstrap.sh"

# Add the VirtualBox machine:

config.vm.provider "virtualbox" do |vb|

# # Display the VirtualBox GUI when booting the machine

vb.gui = true

# Customize the amount of memory on the VM:

vb.memory = "2048"

# set cpu’s  

vb.cpus = 1

#Enable biderectional “copy/paste” and “drag and drop”

vb.customize ['modifyvm', :id, '--clipboard', 'bidirectional']    

vb.customize ['modifyvm', :id, '--draganddrop', 'bidirectional']

end

end

###############################################################

4.  Lets create the bootstrap.sh file!

In the bootstrap.sh file we will write all the linux commands that normaly we run one by one in terminal, which can be very tedious if we have to do it everytime we create a new machine. Here is where Vagrant is very helpfull, it does the job for us. 

The bootstrap.sh file has to be in the same folder with the Vagrantfile!

Copy/Paste the content below (between the hashtag) in a txt file and save it as bootstrap.sh or download the bootstrap.sh file from github.


Please also read the commented lines inside the bootstrapt.sh file!

###############################################################

#!/usr/bin/env bash

sudo apt-get update

#Install Graphical User Interface Lubuntu Desktop
sudo apt install --no-install-recommends lubuntu-desktop -y
sudo apt install virtualbox-guest-x11 -y

#Install QGIS 2.18
sudo sh -c 'echo "deb http://qgis.org/debian xenial main" >> /etc/apt/sources.list'
sudo sh -c 'echo "deb-src http://qgis.org/debian xenial main " >> /etc/apt/sources.list'
wget -O - http://qgis.org/downloads/qgis-2017.gpg.key | gpg --import
gpg --fingerprint CAEB3DC3BDF7FB45
gpg --export --armor CAEB3DC3BDF7FB45 | sudo apt-key add -
sudo apt-get update && sudo apt-get install qgis python-qgis -y

#uninstall GDAL 1.11
cd /usr/share/gdal
sudo rm -rf 1.11
cd

#install GDAL 2.2.2
sudo add-apt-repository -y ppa:ubuntugis/ubuntugis-unstable
sudo apt update  
sudo apt-get install gdal-bin 
sudo apt install python-gdal -y
sudo apt install python3-gdal -y
sudo su

#add GDAL environment variable to .bashrc
sudo echo 'export GDAL_DATA=/usr/share/gdal/2.2' >> /home/vagrant/.bashrc
source .bashrc

#Install PostgreSQL 10
echo 'deb http://apt.postgresql.org/pub/repos/apt/ xenial-pgdg main' >> /etc/apt/sources.list.d/pgdg.list -y
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get install postgresql-10 -y

#Install PostGIS extension
sudo apt install postgis -y

#add postgis and postgis topology extention to postgres database
sudo -u postgres psql -c "CREATE EXTENSION postgis; CREATE EXTENSION postgis_topology;" postgres


#Install pgAdmin4
sudo apt install pgadmin4 -y

###############################################################

5. Lets run vagrant up!

So now you should have 2 files in the folder Project: Vagrantfile and bootstrap.sh

Make sure that in your shell/terminal you are inside the folder Project !

You are now ready to run vagrant up from the command prompt/terminal !

The installation is going to take some time to complete, depending also on the resources of the host machine.

If everything went all, you should see a black screen inside the VirtualBox machine. No worries, everything is ok, you just have to run vagrant reload on the shell/terminal. It will be necessary to run this command twice, first time virtualbox guest addition will be installed and second time to be able to login the vagrant machine.

The credentials to login into the machine are: 

username: vagrant

password: vagrant 

6. Create our own box package.

But what if we have developed things in our virtual machine, we installed GeoServer, for example, and we don't want to lose the work, the files inside the machine and we want to have everything on an external hard disk?

The answer is createing our box package!

If we go back to our Vagrantfile we see that we are using this box config.vm.box = "ubuntu/xenial64", but instead of ubuntu/xenial64 we want our own box package developed on top on ubuntu/xenial64 with GeoServer and all our files.

What we need to do is enter in the folder Project and run vagrant package . This will create a file named package.box that will contain everything we have installed and developed inside the virtual machine, including all we have installed using the bootstrap.sh file. So we will not have to run everything that is in that file with the new box we create.

What we need to do next is copy the folder Project with only this files: package.box, Vagrantfileubuntu-xenial-16.04-cloudimg-console.log, bootstrat.sh to the new location, an external HDD for example. 

After this, in the shell/terminal run vagrant box add my-box name-of-the-box.box, where my-box is the name of ower new box, lets say gisbox and name-of-the-box.box is the name of the package box we just created, in our case is package box. So this command could in the end look like: vagrant box add gisbox package.box . This will add gisbox among the boxes list in vagrant. To check if gisbox was really added in vagrant list boxes, run  vagrant box list and you should be able to see it there.

Next go to the new location where we just copied the files (package.box, Vagrantfile, ubuntu-xenial-16.04-cloudimg-console.log, bootstrat.sh) and open Vagrantfile. Replace the line "config.vm.box = "ubuntu/xenial64" with "config.vm.box = "gisbox" .

Your new Vagrantfile should look like this:

1. Copy/Paste the content below to the the Vagrantfile inside the folder Project!

###############################################################

#-*- mode: ruby -*-

# vi: set ft=ruby :

#We will add the 16.04 version of ubuntu

#And change the disk space from 10 GB to 30 GB

Vagrant.configure("2") do |config|

config.vm.box = "gisbox"

config.disksize.size = "30GB"

#We will share the Downloads folder from the Windows host machine with Ubuntu guest OS #from our virtual machine

#Make sure to change this path with the one coresponding to your computer !!!!!!!!!!!!!!!!!!   

#This will help us share the Downloads folder from the Windows machine with the virtual #machine. But on the guest machine the Downloads folder will be found as vagrant_data

#for Windows OS!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Comment next line if you are using macOS or Ubuntu!!!!!!!!!!

config.vm.synced_folder "C:/Users/your_user_name/Downloads", "/vagrant_data"

#For macOS and Ubuntu!!!!!!!!!!!!!!!!!!!!! Uncomment next line if you are using macOS or Ubuntu!!!!!!!!!

#config.vm.synced_folder "/Users/your_user_name/Downloads", "/vagrant_data"

#This is where we pass the bootstrap.sh file so that Vagrant can run install all we need in the #machine.

config.vm.provision :shell, path: "bootstrap.sh"

# Add the VirtualBox machine:

config.vm.provider "virtualbox" do |vb|

# # Display the VirtualBox GUI when booting the machine

vb.gui = true

# Customize the amount of memory on the VM:

vb.memory = "2048"

# set cpu’s  

vb.cpus = 1

#Enable biderectional “copy/paste” and “drag and drop”

vb.customize ['modifyvm', :id, '--clipboard', 'bidirectional']    

vb.customize ['modifyvm', :id, '--draganddrop', 'bidirectional']

end

end

Bootstrap.sh

Next we modify the bootstrap.sh file. Because our new box will install almost everything we have in this file, we will only keep the lines that configures the virtualbox guest additions.

The file should look like this:

###############################################################

#!/usr/bin/env bash

sudo apt-get update

sudo apt install virtualbox-guest-x11 -y

###############################################################

Now, in windows command prompt/terminal run vagrant up .

Attention!!! You will most probably see this error message if you create the new box on a Windows machine and unpack it in a macOS or Ubuntu OS: 

"There was an error while executing `VBoxManage`, a CLI used by Vagrant for controlling VirtualBox. The command and stderr is shown below. Command: ["startvm", "93344143-9326-4423-81c1-715484930dde", "--type", "headless"] Stderr: VBoxManage: error: RawFile#0 failed to create the raw output file C:/gis/ubuntu-xenial-16.04-cloudimg-console.log (VERR_FILE_NOT_FOUND) VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole"

What you'll have to do is open VirtualBox stop the new machine we are trying to create and right click on the new machine, then go to Settings -> Ports and in Path/Address change the path to ubuntu-xenial-16.04-cloudimg-console.log to point to the folder of the new machine. Remember that previously we also copied this file to the location of the new machine. Then run vagrant up again. Everything should work like a charm and you should also be able to see all of your work there.

7. SSH into virtual machine and skip installing a GUI

Of course we could skip the installation of the GUI and the configuration VirtualBox and ssh into virtual machine.

The command is: vagrant ssh

The small problem with this approach, is the difficulty of running software that uses a GUI, like QGIS, for example. For this we should enable X11 on our computer and install a X11 display server, so things might get even more complicated. For each OS there is a different X11 display server, for Windows is Xming, for macOS is XQuartz and for Ubuntu it's Xorg.

8. Conclusions

So if you are still asking yourself why use Vagrant, I will give you a few reasons: 

  • because most of us are using virtual machines in our day to day work to test different solutions;
  • because we don’t want to install things directly on our host machines and mess things up;
  • because most of us are using all three big OS: macOS, Ubuntu and Windows;
  • because installing or moving a VirtualBox machine from one OS to another and configure it was always a challenge;
  • because setting the VirtualBox guest addition was always difficult and enabling copy/paste from the host to the guest machine and vice versa was tedious too;
  • because sharing a folder from your host machine or an external HDD to the guest machine was often a strugle;
  • because we are doing all this manually.

If you had the same issues as the ones listed above, then use Vagrant!

Although Vagrant can do a lot more than what the tutorial shows, respectively provision it with Puppet, Chef, Ansible, Salt, or Docker, this tutorial is about how to use Vagrant with VirtualBox provider and boot it with a GUI because I usually also use QGIS and other software that need a GUI and many times having a graphical user interface makes things a lot easier.

For questions, remarks and recommendation send e-mail at corina@mapwizard.eu .