Deploy Azure Bastion centrally in Hub & Spoke

Azure Bastion Host is used for remote access of virtual machines without need exposing thoose virtual machines with public IPs. When launched Azure Bastion Host came with a serious drawback, as it did not support VNET peering, hence, we needed to deploy per Virtual Network, making the solution expensive as you needed one per spoke.

Luckily, Azure Bastion Host now has support for VNET peering. Hence the requirement to have one Azure Bastion host per virtual network is not required anymore to get it to work. The next logical placement of the Azure Bastion Host is now to put it more centrally, to keep one Azure Bastion host per region deployment, thus saving cost. A common place can be the connection hub vnet, since it is a well connected with the spokes. If that is the case, you do not want to give more access then needed, since there is other important resources in that subscription. All needed permissions for the Azure Bastion Host can be read at the FAQ for BastionHost for VNET Peering.

Implementation
First part is to establish a role defination as there is no role for Bastion Host as for this date. Hence we need to create a custom role defination. Here is deployment template for BastionReader.
Important to note, in case you do not have access to the root management group, you will need to modify it to a scope where you have the permissions to add Role definitions.

Second part is to deploy the actual Azure Bastion host resource and allocate an group to that role defination. Template for that can be found here.

The input parameters are

  • Basename, prefix for the resources
  • VirtualnetworkID, this is the ID to the virtual network with the AzureBastionSubnet
  • Role defination ID, the ID of the Bastion Reader
  • AAD group, the object ID of the group that will have access to the Bastion resource

After the template is deployed, we are now ready to use the Azure Bastion Host. When using it, make sure that subscription filter is not filtering out the subscription where you are Azure Bastion Host is located, since it utilize this for lookup.

Conclussion
From an architectual this is not an optimal solution, since you do not want to give permissions in the your hub subscription to “everyone” if not needed. Hopefully there is some clever solutions from Microsoft coming to change this.

Enable Hybrid benefit on all already deployed Windows VMs in your subscription

Hybrid benefit in Azure means that you are purchasing the Windows license elsewhere, instead of “pay-as-you-go” with the VM consumption. The reason to do this is either your organization already owns licenses or purchased licensese to save cost (there is a brake even when it makes sense to buy the license outside of Azure).

To retrofit this to existing subscription, it is quite straight forward with Powershell. It is just a metadata for the virtual machine, so there is no down time required (reference https://docs.microsoft.com/en-us/azure/virtual-machines/windows/hybrid-use-benefit-licensing#convert-an-existing-vm-using-azure-hybrid-benefit-for-windows-server) .

The following Powershell code takes your current logged in Azure subscription context, get all Windows machines that does not have Hybrid benefit enabled and loops thru them, enabling Hybrid benefit.

[Cmdletbinding()]
Param
(
)


$nonehbvm = Get-AzVM -Status | where{$_.StorageProfile.OsDisk.OsType -eq "Windows" -and (!($_.LicenseType))} | Select-Object Name,ResourceGroupName,Licensetype
foreach($i in $nonehbvm)
{
    $vm = Get-AzVM -ResourceGroup $($i.ResourceGroupName) -Name $($i.Name) 
    Write-Verbose "Setting hybrid benefit on VM $($i.Name) "
    $vm.LicenseType = "Windows_Server"
    Update-AzVM -ResourceGroupName $($i.ResourceGroupName) -VM $vm
} 

SQL Backup with Azure Recovery Services

Microsoft announced a bit back that they now include SQL Backup of SQL Server installed on Azure VMs. https://azure.microsoft.com/en-us/blog/azure-backup-for-sql-server-on-azure-vm-public-preview/. This enables us to gather most (if not yet all) backup services in one tool.

Setup
First of, you will need a Recovery service vault in the same subscription as your SQL Server resides. When recovery service is in place, choose backup and there is an option “SQL Server in Azure VM”

This will prompt to run a discovery, to search for Azure virtual machines with SQL Server on it. After it finds it, it will install a small windows service AzureBackupWindowsWorkload that utilize a user (NT Service\AzureWLBackupPluginSvc) for the backup. This process takes in my experience a few minutes. If your VM was not provisioned thru Azure SQL Server image, the SQLIaaSExtension will need to be installed and add NT Service\AzureWLBackupPluginSvc as Sysadmin user on the SQL Server instance.

After discovery is completed, there will be a list with SQL Server Instances and what databases resides on the particular instance. From here, either check the top to get all database, or choose what databases that are going to be backed up.

Next step is configure the backup policy. There is a default created that will do a full/diff backup each day, and logbackup each hour. Or create a policy that match your company RPO needs. (During preview the default policy cannot be changed)

After configuration, recommended is to intial a backup, this will run an initial full backup of the databases. The progress can be tracked in Recovery service under backup jobs

Restore
A backup is only as good as its ability to be restored. Restores are intiated from Recovery service. Simply choose the database that are going to be recovered and choose Restore DB

Restores option are to overwrite current database or to restore on Alternate Location (on same SQL Server with different name or other registrered SQL Servers). Makes it handy to restore one offs when you want to test something on a database or do restore tests

If the database is in full recovery model, the possibility to restore in a specific time and date will be presented, or from latest full/diff backup. This experience is similiar what is presented in SQL Management Studio. Easy and to the point.

Lastly, a few moreoption to set in NO Recovery and the physical name and location of the .mdf and .ldf files are given. After that, review and press Restore. The restore job can be tracked under Backup jobs in Recovery services.

Summary
This functionality adds a bit in the puzzle to make the backup and recovery service more complete. The easiness of configuration and restore makes it available for a broad audience.

Force a reboot with DSC in ARM-template

When deploying an ARM-tempalate, the Azure Virtual Network got updated with the new DNS server. The Virtual machines deployed before this update, still got the old DNS client settings, before the VM gets rebooted. This becomes a bit of a challange, when trying to domain join the computer with Desired State Configuration (DSC), since it will not find the domain controller (if not the old DNS settings where already pointed to a current domain controller).

To force a reboot thru DSC can be achieved by the module xPendingReboot and DSC Script. xPendingReboot checks if there is a pending reboot of the machine and works in conjunction with the Local Configuration Manager (LCM). If LCM is set to RebootNodeIfNeeded=$true, it will reboot. However, a new machine got no pending reboot. To achieve this we will need to set DSCMachineStatus to 1 and write a small registry item. This will indicate there is a PendingReboot.

First we need to load the module (including the domainJoin as well)

Import-DscResource -ModuleName xPendingReboot, xDSCDomainjoin

Set the LCM RebootNodeIfNeeded to true

LocalConfigurationManager 
{
   RebootNodeIfNeeded = $true
}

Create a xPendingReboot resource

xPendingReboot Reboot
{
   Name = "Reboot"
}

Add our “fake” reboot.

Script Reboot
{
    TestScript = {
    return (Test-Path HKLM:\SOFTWARE\MyMainKey\RebootKey)
    }
    SetScript = {
			New-Item -Path HKLM:\SOFTWARE\MyMainKey\RebootKey -Force
			$global:DSCMachineStatus = 1 
        }
    GetScript = { return @{result = 'result'}}
}

and finally the domainjoin with a DependsOn so it will happend before domainjoin.

xDSCDomainjoin JoinDomain
{
  Domain = $DomainName
  Credential = $DomainCreds
  DependsOn = "[Script]Reboot"
}

Updating the Microsoft Linux Agent – waagent – on Oracle Linux

I stumpled on a problem that the Azure backups were not going thru correctly and this was because the Azure Recovery services could not get proper contact with the Azure agent on the virtual machine. The Virtual machine in this case is a Oracle Linux deployed from the Azure gallery.

Investigate
SSH to the machine and make sure you got an account with sudo permissions.
First of, checked if the waagent was running

sudo systemctl status waagent

In my case, it was not running. Next step was to consult the logs that is located in /var/log/waagent.log. For now, this was empty and the old logs were compress .gz files. To unpack the compressed files

gunzip waagent.log-20180201.gz

After some searching around. The general tip was to update to latest version. To check current version, waagent has a switch for it. I also include the yum version that is the on the offical MS documentation sie

waagent -version
sudo yum list WALinuxAgent

My version was a low 2.0 version. So to check for updates, issue another yum command.

sudo yum check-update waagent

This will tell you if there is updates to your package.

Updating

To run the update, you issue yet another yum command

sudo yum install WALinuxAgent

This will update your waagent to latest version. This does not require a reboot.

Verifying

After we are done, we need to verify that the agent is running

sudo systemctl status waagent

In my case, it was running, but I got the following message

Warning: waagent.service changed on disk. Run 'systemctl daemon-reload' to reload units.

To quote a excellent stackexchange answer,

taking changed configurations from filesystem and regenerating dependency trees

What does “systemctl daemon-reload” do?

So proceeding, we need to wrap up and issue the daemon reload command to reload defination files.

 sudo systemctl daemon-reload

Running the

sudo systemctl status waagent

will now not generate any errors – and the backups are running yet again.