Blog Series: Monitoring using PowerShell: Part Seven – Monitoring back-ups with PowerShell

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.


Requirements:

  • (Optional) ShadowProtect SPX
  • (Optional) IASO Backup
  • (Optional) BackupExec 12 or higher
  • PowerShell v3 or higher

Creating the monitoring sets:

This blog will contain multiple monitoring sets for multiple backup sets. For some of these we’re using event-log viewing to check when the last backup happened and what the state was, for others we’re using the included module the backup supplier has given.

Monitoring ShadowProtect Jobs
Unfortunately ShadowProtect is one of those products that do not truly believe in the power of automation, but I’ve found that you can monitor the jobs simply by filtering for the eventID related to the ShadowProtect jobs. The message always contains “Warning”, “Error” or “failed”

try{
$ShadowProtectLog = Get-EventLog -logname Application | Where-Object {(Get-Date $_.TimeWritten) -gt ((Get-Date).AddHours(-24))}
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($logentry in $ShadowProtectLog){
if($logentry.Message -match "Warning"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has a warning"}
if($logentry.Message -match "Error"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has an error"}
if($logentry.Message -match "Failed"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has failed"}
}
if (!$ShadowProtectStatus) { $ShadowProtectStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Monitoring IASO/MAX-Backup/MSP-Backup

IASO/MAX/MSP-Backup suffer from pretty much the same fault as ShadowProtect, but they do give us much better presentable logs. We only have to look for 2 states in the logs they create to check the backup status “[E]” which stands for backup errors and warning, and the string “NotStarted” which is logged when VSS errors are found.

try{
$TodayFormatted = Get-Date -format yyyy_M_d
$IASOLog = get-content "C:\ProgramData\MXB\Backup Manager\logs\BackupFP\BackupFP_$($TodayFormatted).log"
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($line in $IASOLog){
if($line -match "NotStarted") {$IASOStatus += "'n$($line)"}
if($line -match "[e]") {$IASOStatus +=  "'n$($line)"}
}
if (!$IASOStatus) { $IASOStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Monitoring BackupExec Backups

We rarely use backupexec anymore, but I know some MSP’s stuggle with monitoring it. The following script uses the “BEMCLI” which is the PowerShell suite given to us by Symantec. The upside about this monitoring method is that the alerts only get cleared after you acknowledge them from the BackupExec console. To do this you open the BackupExec console and click in the bottom left of the status bar to view/clear the alerts.

#Setting default CLI
import-module bemcli
#Getting Alerts
$Alerts = Get-BEAlert
#Looping through the alerts and setting them.
foreach($Alert in $Alerts){
switch ($Alert.Category)
{
JobWarning{$JobWarning = "TRUE - $($Alert.Message)" }
JobFailure{$JobFailure = "TRUE - $($Alert.Message)"}
JobCancellation{$JobCancellation = "TRUE - $($Alert.Message)" }
CatalogError{$CatalogError = "TRUE - $($Alert.Message)"}
SoftwareUpdateWarning{$SoftwareUpdateWarning = "TRUE - $($Alert.Message)"}
SoftwareUpdateError{$SoftwareUpdateError = "TRUE - $($Alert.Message)" }
DatabaseMaintenanceFailure{$DatabaseMaintenanceFailure = "TRUE - $($Alert.Message)"}
IdrCopyFailed{$IdrCopyFailed = "TRUE - $($Alert.Message)"}
BackupJobContainsNoData{$BackupJobContainsNoData = "TRUE - $($Alert.Message)" }
JobCompletedWithExceptions{$JobCompletedWithExceptions = "TRUE - $($Alert.Message)"}
JobStart{$JobStart = "TRUE - $($Alert.Message)"}
ServiceStart{$ServiceStart = "TRUE - $($Alert.Message)"}
ServiceStop{$ServiceStop = "TRUE - $($Alert.Message)"}
DeviceError{$DeviceError = "TRUE - $($Alert.Message)"}
DeviceWarning{$DeviceWarning = "TRUE - $($Alert.Message)"}
DeviceIntervention{$DeviceIntervention = "TRUE - $($Alert.Message)"}
MediaError{$MediaError = "TRUE - $($Alert.Message)"}
MediaWarning{$MediaWarning = "TRUE - $($Alert.Message)" }
MediaIntervention{$MediaIntervention = "TRUE - $($Alert.Message)"}
MediaInsert{$MediaInsert = "TRUE - $($Alert.Message)"}
MediaOverwrite{$MediaOverwrite = "TRUE - $($Alert.Message)"}
MediaRemove{$MediaRemove = "TRUE - $($Alert.Message)"}
LibraryInsert{$LibraryInsert = "TRUE - $($Alert.Message)"}
TapeAlertWarning{$TapeAlertWarning = "TRUE - $($Alert.Message)"}
TapeAlertError{$TapeAlertError = "TRUE - $($Alert.Message)" }
IdrFullBackupSuccessWarning{$IdrFullBackupSuccessWarning = "TRUE - $($Alert.Message)"}
LicenseAndMaintenanceWarning{$LicenseAndMaintenanceWarning = "TRUE - $($Alert.Message)"}
default{$OtherErr = "TRUE - $($Alert.Message)" }
}
}

Monitoring other backup software & VSS Errors
Most backup application these days use VSS to create snapshots and we can monitor these to check if the snapshots have succes status or failures. The same issue as before occurs – Microsoft does not have a PowerShell VSS module or log that we can capture so we’ll have to use the eventlog that registers VSS snapshots

try{
$VSSLog = Get-EventLog -logname Application -Source VSS | Where-Object {(Get-Date $_.TimeWritten) -gt ((Get-Date).AddHours(-24))}
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($logentry in $VSSLog){
if($logentry.EntryType -eq "Warning"){ $VSSStatus += "`nVSS Snapshot at at $($logentry.TimeGenerated) has a warning"}
if($logentry.EntryType -eq "Error"){ $VSSStatus += "`nVSS Snapshot at at $($logentry.TimeGenerated) has an error"}
}
if (!$VSSStatus) { $VSSStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Hope you’ve enjoyed this blog, the next one will be about monitoring windows performance and unexpected shutdown states. 🙂

Downloads for RMM packages:

N-Central 11.0+ – BackupExec Monitoring


					

10 thoughts on “Blog Series: Monitoring using PowerShell: Part Seven – Monitoring back-ups with PowerShell

  1. Aaron

    I tested this script on a server (2012) using Solarwinds Managed Online Backup, but it failed. When checking, the folder location is slightly different, so I changed that path and retried. The script executes, but doesn’t display a message! I’m not sure where to look at next!

    Folder: C:\ProgramData\Managed Online Backup\Backup Manager\logs\BackupFP\
    Latest File: BackupFP_2019_11_04

    Reply
  2. Hendrik Huibers

    Hello,

    I am using the script Monitoring IASO/MAX-Backup/MSP-Backup
    I have some Back-up Satus “[E]” errors in de logs, but the IASO dashboard give a success result.
    I have spoken with Solarwinds support and they say that the “[E]” errors in the log file are not important and i can trust the succes result from the IASO dashboard.

    So i’m looking for another way te create a reliable result via powershell, do you have any suggestions for me?

    The error from the log file looks like below:
    [2020-10-19 10:29:28.281946] [E] [05460] [Cleaning] Node removal with subtree skipped. NodeId: 731130, ParentId: 384571, SessionId: 7. Reason: Attempt to delete node with children. Error code: 19:1811 (constraint failed)

    Reply
      1. Hendrik Huibers

        Hello Kelvin,

        I have tested the script and if i understand the script correctly, the script checks if there were any errors for the last 48 hours right?

        But when in the same 48 hours a newer back-up is successed for exampe after a applied fix, the script still saying that the back-up is failed until the failed state is longer ago then 48 hours.

        So is there a way or do you have any suggestions that the failed state go back to: “Healthy. No Failed backups” when the newest or the very last back-up is succeeded?

        Thanks for helping so far,

        Kind regards,
        H. Huibers

        Reply
  3. Hendrik Huibers

    Hello Kelvin,

    I repeat this message againg because mayne you missed it 🙂

    I have tested the script and if i understand the script correctly, the script checks if there were any errors for the last 48 hours right?

    But when in the same 48 hours a newer back-up is successed for exampe after a applied fix, the script still saying that the back-up is failed until the failed state is longer ago then 48 hours.

    So is there a way or do you have any suggestions that the failed state go back to: “Healthy. No Failed backups” when the newest or the very last back-up is succeeded?

    Thanks for helping so far,

    Kind regards,
    H. Huibers

    Reply
    1. Marcel B

      Hi Kelvin,

      I ran into the same problem as H. Huibers. Last 2 back-ups where reported fine within the IASO console, but in Datto RMM the status still is failed.

      Reply
  4. Hendrik Huibers

    Hello,

    For your information, i have customized a new script wich is working well for so far!

    See below for more information (Sorry, some things are in dutch)

    ######################################################################
    #Script: Monitor Solarwinds MSP Backup
    #Author: Hendrik Huibers
    #Created: 13-10-2020
    #Changed Date: 24-11-2020
    #Description: Monitor the back-up status of ISAO Online Back-up.
    ######################################################################
    Set-ExecutionPolicy “Unrestricted” -Confirm:$false -force
    # functions for writing results
    function write-DRMMDiag ($messages) {
    write-host ”
    foreach ($Message in $Messages) { $Message }
    write-host ”
    }
    function write-DRRMAlert ($message) {
    write-host ”
    write-host “Alert=$message”
    write-host ”
    }

    # check if right powershell version is installed
    $version = (Get-Host | Select-Object Version)
    if($Version.Version.Major -lt “3”) {
    write-DRRMAlert “Niet ondersteunde powershell versie. Alleen powershell versie 4.0 of hoger word ondersteund.”
    exit 1
    }

    # PowerShell Checks If a File Exists
    $todayFormatted = Get-Date -Format yyyy_M_dd
    $yesterdayFormatted = (Get-Date).AddDays(-1).ToString(“yyyy_M_dd”)
    $WantFiletoday = “C:\ProgramData\MXB\Backup Manager\logs\BackupFP\BackupFP_$($todayFormatted).log”
    $WantFileyesterday = “C:\ProgramData\MXB\Backup Manager\logs\BackupFP\BackupFP_$($yesterdayFormatted).log”
    $FileExiststoday = Test-Path $WantFiletoday
    $FileExistsyesterday = Test-Path $WantFileyesterday
    If ($FileExiststoday -ne $True -and $FileExistsyesterday -ne $true ) {
    write-DRRMAlert “log file niet gedetecteerd of back-up heeft niet gedraaid”
    exit 1
    }

    #get date minus 1 hours
    $Date = (get-date).AddHours((-48))

    #read back-up status
    $SessionsList = & “C:\Program Files\Backup Manager\ClientTool.exe” -machine-readable control.session.list -delimiter “,” | convertfrom-csv -Delimiter “,” | where-object { [datetime]$_.start -gt $date } | Sort-Object {[datetime]$_.start -gt $date} | Sort-Object dsrc -Unique
    $FailedBackups = foreach ($session in $SessionsList) {
    if ($Session.state -eq ‘InProcess’ -and $session.START -lt (get-date).AddHours(-23)) { “Backup draait langer dan 23 uur. Backup gestart op $($session.START)” }
    if ($Session.state -eq ‘CompletedwithErrors’) { “Backup is afgerond met een error. Backup gestart op $($session.START)” }
    if ($Session.state -eq ‘Failed’) { “Backup is mislukt met een error. Backup gestart op $($session.START)” }
    if ($Session.state -eq ‘Skipped’) { “Backup is overgeslagen omdat er nog een andere job actief is. Backup gestart op $($session.START)” }
    if ($Session.state -eq ‘NotStarted’) { “Backup is niet gestart, meer info zie back-up log. Backup gestart op $($session.START)” }
    }

    #error message when backup is failed
    if ($FailedBackups) {
    write-DRRMAlert “Probleem met de back-up gevonden. Backup $($session.STATE), Check de logs voor meer info.”
    write-DRMMDiag ($FailedBackups, $SessionsList | Out-String)
    exit 1
    }

    if ($SessionsList -eq $null) {
    write-DRRMAlert “Geen 1 back-up log gevonden in afgelopen 48 uur”
    write-DRMMDiag ($FailedBackups, $SessionsList | Out-String)
    exit 1
    }

    #success message when back-up is succeeded
    else {
    write-DRRMAlert “Gezond, geen mislukte back-ups gevonden”
    write-DRMMDiag ($FailedBackups, $SessionsList | Out-String)
    exit 0
    }

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.