About…

Creating a Poor Man’s Express Route

Lately we’ve been running into issues with client’s connecting to Office 365 and having high latency with this connection. this results in issues such as “Outlook is not Responding” & “Getting data from server outlook.Office365.com”. Both of these issues disappear when running in Cached mode due to less data-traffic being required for normal operations such as opening e-mails and retrieving folder lists.

After researching we’ve found that this is mostly related to enormous detours that the ISP was taking to reach the Office 365 back-end. Mostly due to “cheaper” connections to the DUB(EU) or AMS(EU) data-centers of Microsoft. A good example of this can be seen with the Dutch ISP RoutIT, where you have 28 hops on average to the Office 365 back-end(of which 90% is still in their network), but only 5 hops to the Azure back-end strangely enough. After contacting Microsoft Support we’ve got the dreaded answer: Please invest into an Express-Route solution to circumvent these issues or have the provider resolve the connection problems.

We have multiple small sites with 5+ users that simply cannot afford this, and the clients can’t accept this latency. So we decided to ignore Microsoft’s advice and go on our own path of discovery 😉 After reading some of the service descriptions we’ve found that Microsoft assures better connections from Azure to Office 365 by using extensive QoS, and of course simply because the virtual machines are in the same network as the Office 365 services.

I’ve created the step-by-step below to research, test, and implement a solution.

Research

first we needed to discover if the client actually suffered from this issue:

  • Open outlook on a user with connection issues.
  • Right-click the tray icon, while holding CTRL and click on “Connection Status”
  • In online mode you’d want the AVG. Proc time to be lower than 25(Cached mode this can be anything from 25-300)
  • In online mode you’d want the AVG RTT Time to be lower than 200(Cached mode this can be anything from 0-1000)
  • If you have results higher than above, you are possibly having issues connecting to outlook.office365.com and a poor-man’s ExpressRoute might help you run your outlook in Online mode.

After these tests, we run a Traceroute to tripple-check if the route is the issue, or if the connection latency itself is the issue. You expect a traceroute to Outlook.office365.com to always be less than 20 hops, preferably in the 10-15 range, such as my results;

tracert outlook.office365.com

Tracing route to outlook-emeacenter.office365.com [132.245.229.114]
over a maximum of 30 hops:

  1     1 ms    <1 ms     1 ms  xxx.xxx.xxx.xxx
  2     9 ms    9 ms    10 ms  xxx.xxx.xxx.xxx
  3     7 ms     8 ms     7 ms  ae0-33.crt02-gsw.redhosting.nl [178.238.96.45]
  4     4 ms    4 ms    12 ms  ae0-31.crt01-nkf.redhosting.nl [178.238.96.37]
  5     4 ms    4 ms    13 ms  ams-ix-2.microsoft.com [80.249.209.21]
  6     6 ms    7 ms    14 ms  be-62-0.ibr01.amb.ntwk.msn.net [104.44.9.134]
  7     13 ms    12 ms    15 ms  be-7-0.ibr01.ams.ntwk.msn.net [104.44.5.33]
  8     14 ms    16 ms    16 ms  ae63-0.am3-96c-1a.ntwk.msn.net [104.44.5.1]
  9     6 ms     5 ms     5 ms  132.245.229.114

Trace complete.

If you have results as above, this solution might not be the one for you, Of course you can continue testing below but YMMV.

Create Virtual Machine Azure for performance tests & RRAS

Create a virtual machine in the Azure portal with the following specifications and configuration, If you have never created a virtual machine please use the  following guide on the Microsoft website. The interfaces changes every couple of months so its best to use the resources from Microsoft themselves.

Configuration:

  • at least 1 Core
  • at least 1.75GB of RAM
  • Static Internal IP for the server
  • Allow traffic: SSL to server(443)
  • Install the following roles: Remote Access, DHCP Server.
  • Setup DHCP and Remote Access.

After configuring the Azure side of the requirements make sure you configure SSTP VPN connections(Not required – For testing only.) and allow Router advertisement and IP forwarding. If you require a SSL Certificate you can use a free certificate from LetsEncrypt for this test.

After performing all configurations connect to the SSTP VPN from a client that experienced issues. Remember to set the “Use default gateway on remote network” to force all traffic over this tunnel for the sake of the test. Restart outlook and perform the testing steps once more. You should see a lower AVG. Proc Time and lower AVG. RTT Time. Have the user work on the SSTP connection for a day to see if they actually see a noticeable difference. If this is the case, its time for a more permanent solution.

Create IPSec tunnel

If the tests above have been performed, and you are ready for a permanent solution you can use Azure to create an ipsec tunnel, for that I advise the following guide.

After configuration of your ipsec tunnel, remember to tunnel all Office 365 traffic using the Microsoft IP address list found on this link. You can choose what traffic you route over the tunnel and what traffic you’d prefer using your local internet connection. When using the IPSEC tunnel remember that you must use your RRAS server as the gateway or else the routing will end on the side of the the Azure S2S VPN.

an ipsec VPN Tunnel and RRAS for NAT/Routing costs about 90% less than ExpressRoute, and might resolve most issue you have with sluggish outlooks, or slow responding Office 365 services.

Happy O365ing.

Using PowerShell to monitor Backups

We are using a RMM that has integrated BackupExec monitoring. I’ve found that this integrated monitoring was somewhat lacking. It gave us the current job status and that’s about it, Meaning there was not really a way to resolve issues pre-preemptively.

To resolve this we’ve created the following monitoring PowerShell script and integrated it into our RMM solution. For your convenience we’ll dissect the script so you can re-use this in your own solution or set it as scheduled job 🙂

We’ll start by importing the BEMCLI module which is included in BackupExec 2012 and up.

#Setting default CLI
import-module bemcli 

After importing the BEMCLI We’ll be able to use the cmdlets that get us the info we want, in this case “get-BEAlert” which gives us the current Alerts that have not been acknowledged within BackupExec

#Getting Alerts
$Alerts = Get-BEAlert

Now that we have the Alerts stored in a variable we wil loop through the contents of the alerts to find what we require;

#Looping through the alerts and writing the message to the console
foreach($Alert in $Alerts){
write-host $Alert.message
}

Now that we see that the messages are posted in the console we’d of course like a better overview.

#Looping through the alerts and setting them.
foreach($Alert in $Alerts){

switch ($Alert.Category)
{
JobWarning{$JobWarning = "TRUE - $($Alert.Message)" }
JobFailure{$JobFailure = "TRUE - $($Alert.Message)"}
JobCancellation{$JobCancellation = "TRUE - $($Alert.Message)" }
CatalogError{$CatalogError = "TRUE - $($Alert.Message)"}
SoftwareUpdateWarning{$SoftwareUpdateWarning = "TRUE - $($Alert.Message)"}
SoftwareUpdateError{$SoftwareUpdateError = "TRUE - $($Alert.Message)" }
DatabaseMaintenanceFailure{$DatabaseMaintenanceFailure = "TRUE - $($Alert.Message)"}
IdrCopyFailed{$IdrCopyFailed = "TRUE - $($Alert.Message)"}
BackupJobContainsNoData{$BackupJobContainsNoData = "TRUE - $($Alert.Message)" }
JobCompletedWithExceptions{$JobCompletedWithExceptions = "TRUE - $($Alert.Message)"}
JobStart{$JobStart = "TRUE - $($Alert.Message)"}
ServiceStart{$ServiceStart = "TRUE - $($Alert.Message)"}
ServiceStop{$ServiceStop = "TRUE - $($Alert.Message)"}
DeviceError{$DeviceError = "TRUE - $($Alert.Message)"}
DeviceWarning{$DeviceWarning = "TRUE - $($Alert.Message)"}
DeviceIntervention{$DeviceIntervention = "TRUE - $($Alert.Message)"}
MediaError{$MediaError = "TRUE - $($Alert.Message)"}
MediaWarning{$MediaWarning = "TRUE - $($Alert.Message)" }
MediaIntervention{$MediaIntervention = "TRUE - $($Alert.Message)"}
MediaInsert{$MediaInsert = "TRUE - $($Alert.Message)"}
MediaOverwrite{$MediaOverwrite = "TRUE - $($Alert.Message)"}
MediaRemove{$MediaRemove = "TRUE - $($Alert.Message)"}
LibraryInsert{$LibraryInsert = "TRUE - $($Alert.Message)"}
TapeAlertWarning{$TapeAlertWarning = "TRUE - $($Alert.Message)"}
TapeAlertError{$TapeAlertError = "TRUE - $($Alert.Message)" }
IdrFullBackupSuccessWarning{$IdrFullBackupSuccessWarning = "TRUE - $($Alert.Message)"}
LicenseAndMaintenanceWarning{$LicenseAndMaintenanceWarning = "TRUE - $($Alert.Message)"}
default{$OtherErr = "TRUE - $($Alert.Message)" }
}
}

Now if we put this all together the result would be;

&lt;# .SYNOPSIS Gets BackupExec information and reports on Running Alerts - Only works on BackupExec 2012 and higher. .DESCRIPTION Using BEMCLI we retrieve data from BACKUPEXEC, including multiple types of alerts, LastRunTime, etc. Currently Alerts are generated for the following catagories: JobWarning JobFailure JobCancellation CatalogError SoftwareUpdateInformation SoftwareUpdateWarning SoftwareUpdateError DatabaseMaintenanceFailure IdrCopyFailed IdrFullBackupSuccess BackupJobContainsNoData JobCompletedWithExceptions JobStart ServiceStart ServiceStop DeviceError DeviceWarning DeviceIntervention MediaError MediaWarning MediaIntervention MediaInsert MediaOverwrite MediaRemove LibraryInsert TapeAlertWarning TapeAlertError IdrFullBackupSuccessWarning LicenseAndMaintenanceWarning .LINK http://www.cyberdrain.com #&gt;

#Setting default CLI
i#Setting default CLI
import-module bemcli 
#Getting Alerts
$Alerts = Get-BEAlert

#Looping through the alerts and setting them.
foreach($Alert in $Alerts){

switch ($Alert.Category)
{
JobWarning{$JobWarning = "TRUE - $($Alert.Message)" }
JobFailure{$JobFailure = "TRUE - $($Alert.Message)"}
JobCancellation{$JobCancellation = "TRUE - $($Alert.Message)" }
CatalogError{$CatalogError = "TRUE - $($Alert.Message)"}
SoftwareUpdateWarning{$SoftwareUpdateWarning = "TRUE - $($Alert.Message)"}
SoftwareUpdateError{$SoftwareUpdateError = "TRUE - $($Alert.Message)" }
DatabaseMaintenanceFailure{$DatabaseMaintenanceFailure = "TRUE - $($Alert.Message)"}
IdrCopyFailed{$IdrCopyFailed = "TRUE - $($Alert.Message)"}
BackupJobContainsNoData{$BackupJobContainsNoData = "TRUE - $($Alert.Message)" }
JobCompletedWithExceptions{$JobCompletedWithExceptions = "TRUE - $($Alert.Message)"}
JobStart{$JobStart = "TRUE - $($Alert.Message)"}
ServiceStart{$ServiceStart = "TRUE - $($Alert.Message)"}
ServiceStop{$ServiceStop = "TRUE - $($Alert.Message)"}
DeviceError{$DeviceError = "TRUE - $($Alert.Message)"}
DeviceWarning{$DeviceWarning = "TRUE - $($Alert.Message)"}
DeviceIntervention{$DeviceIntervention = "TRUE - $($Alert.Message)"}
MediaError{$MediaError = "TRUE - $($Alert.Message)"}
MediaWarning{$MediaWarning = "TRUE - $($Alert.Message)" }
MediaIntervention{$MediaIntervention = "TRUE - $($Alert.Message)"}
MediaInsert{$MediaInsert = "TRUE - $($Alert.Message)"}
MediaOverwrite{$MediaOverwrite = "TRUE - $($Alert.Message)"}
MediaRemove{$MediaRemove = "TRUE - $($Alert.Message)"}
LibraryInsert{$LibraryInsert = "TRUE - $($Alert.Message)"}
TapeAlertWarning{$TapeAlertWarning = "TRUE - $($Alert.Message)"}
TapeAlertError{$TapeAlertError = "TRUE - $($Alert.Message)" }
IdrFullBackupSuccessWarning{$IdrFullBackupSuccessWarning = "TRUE - $($Alert.Message)"}
LicenseAndMaintenanceWarning{$LicenseAndMaintenanceWarning = "TRUE - $($Alert.Message)"}
default{$OtherErr = "TRUE - $($Alert.Message)" }
}
}

Now you can schedulde this script in your own RMM or sent e-mails based on the result.:) Happy Scripting!