About…

Blog Series: Monitoring using PowerShell: Part Eight – Monitoring health with PowerShell

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • PowerShell v3 or higher

Creating the monitoring sets:

This blog will contain multiple monitoring sets for checking the health state of the OS. The first sets will leverage the eventviewer as we’ve done in previous blogs but only to check the state of an unclean shutdown or BSOD. The second part we will check the actual performance of the Disk IOPS and CPU by using the cooked value that is presented by the windows performances countes. As always I’ll supply some RMM packages at the bottom of the blog. For performance monitoring I do want you to understand that most RMM packages already have this intergrated and this is just an alternative method.

Monitoring BSOD’s and unclean Shutdowns

BSODs and unclean shutdowns aren’t really presented to the OS in any other way than event log entries. So to get this information we try getting the related logs:

$EventShutdown = get-eventlog -LogName System | Where-Object {$_.EventID -eq 6008 -AND $_.timegenerated -gt (get-date).adddays(-7)}| select message
$EventBSOD = get-eventlog -LogName System | Where-Object {$_.EventID -eq 1005 -AND $_.timegenerated -gt (get-date).adddays(-7)}| select message

		
if($EventShutdown.count -ge 1){
$CleanShutdown = "Unclean shutdown has occured in the past 7 days."
}

if($EventBSOD.count -ge 1){
$CleanBSOD = "BSOD shutdown has occured in the past 7 days. "
}

Pretty simple script, but effective 🙂

Performance monitoring with PowerShell:
For this script we’re grabbing windows counters using get-counter, get-counter presents raw values by default, Raw values aren’t giving us alot of use so we’re requesting the subset that contains the raw values, mixed with the second values. These 2 factors combined give us the cooked value, the cooked value is human readable instead of just a random string of numbers.

When using this type of monitoring I advise to have a very low polling ratio, e.g. have the script run every 1 minute to make sure you capture spikey traffic that is sometimes associated with performance issues.

$DiskQueueLength = (get-counter -counter "\PhysicalDisk(*)\Current Disk Queue Length").countersamples.cookedvalue

foreach($Disk in $DiskQueueLength){
if($Disk -le "5"){
$DiskQueueHealth = "Healthy"
} else {
$DiskQueueHealth +="The Disk Queue Length was $Disk for 30 minutes"
}

}

Monitoring CPU Health

So to monitor CPU health we have a couple of options; of course monitoring the precentage used is very important, but we’re also going to monitor the CPU-queue, which are commands queued by the CPU. When this raises you’ll notice those terrible slowdowns and even the dreaded “Not responding” pop-ups.

$CPUQueueLenth = (get-counter -counter "\System\Processor Queue Length").countersamples.cookedvalue 
$CPUUserTime = (get-counter -counter "\Processor(*)\% User Time").countersamples.cookedvalue
$CPUPrivTime = (get-counter -counter "\Processor(*)\% Privileged Time").countersamples.cookedvalue
if($CPUQueueLenth -le "20"){
$CPUQueueHealth = "Healthy"
} else {
$CPUQueueHealth ="The CPU Queue Length was $CPUQueueLenth for 30 minutes"
}
if($CPUUSerTime-le "98"){
$CPUUserTimeHealth = "Healthy"
} else {
$CPUUserTimeHealth ="The CPU User Time was $CPUUserTime for 30 minutes"
}
if($CPUPrivTime-le "98"){
$CPUPrivTimeHealth = "Healthy"
} else {
$CPUPrivTimeHealth ="The CPU User Time was $CPUUserTime for 30 minutes"
}

So, that’s the end of this series. I’ve actually decided to make this more of a regular thing and join me again next month for a new PowerShell monitoring series where I will be touching on reporting, auto-recovery of issues without human intervention.

 

Downloads for RMM packages:

N-Central 11.0+ – BSOD Monitoring

N-Central 11.0+ – DiskIO Monitoring

Blog Series: Monitoring using PowerShell: Part Seven – Monitoring back-ups with PowerShell

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • (Optional) ShadowProtect SPX
  • (Optional) IASO Backup
  • (Optional) BackupExec 12 or higher
  • PowerShell v3 or higher

Creating the monitoring sets:

This blog will contain multiple monitoring sets for multiple backup sets. For some of these we’re using event-log viewing to check when the last backup happened and what the state was, for others we’re using the included module the backup supplier has given.

Monitoring ShadowProtect Jobs
Unfortunately ShadowProtect is one of those products that do not truly believe in the power of automation, but I’ve found that you can monitor the jobs simply by filtering for the eventID related to the ShadowProtect jobs. The message always contains “Warning”, “Error” or “failed”

try{
$ShadowProtectLog = Get-EventLog -logname Application | Where-Object {(Get-Date $_.TimeWritten) -gt ((Get-Date).AddHours(-24))}
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($logentry in $ShadowProtectLog){
if($logentry.Message -match "Warning"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has a warning"}
if($logentry.Message -match "Error"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has an error"}
if($logentry.Message -match "Failed"){ $ShadowProtectStatus += "Backup at $($logentry.TimeGenerated) has failed"}
}
if (!$ShadowProtectStatus) { $ShadowProtectStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Monitoring IASO/MAX-Backup/MSP-Backup

IASO/MAX/MSP-Backup suffer from pretty much the same fault as ShadowProtect, but they do give us much better presentable logs. We only have to look for 2 states in the logs they create to check the backup status “[E]” which stands for backup errors and warning, and the string “NotStarted” which is logged when VSS errors are found.

try{
$TodayFormatted = Get-Date -format yyyy_M_d
$IASOLog = get-content "C:\ProgramData\MXB\Backup Manager\logs\BackupFP\BackupFP_$($TodayFormatted).log"
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($line in $IASOLog){
if($line -match "NotStarted") {$IASOStatus += "'n$($line)"}
if($line -match "[e]") {$IASOStatus +=  "'n$($line)"}
}
if (!$IASOStatus) { $IASOStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Monitoring BackupExec Backups

We rarely use backupexec anymore, but I know some MSP’s stuggle with monitoring it. The following script uses the “BEMCLI” which is the PowerShell suite given to us by Symantec. The upside about this monitoring method is that the alerts only get cleared after you acknowledge them from the BackupExec console. To do this you open the BackupExec console and click in the bottom left of the status bar to view/clear the alerts.

#Setting default CLI
import-module bemcli
#Getting Alerts
$Alerts = Get-BEAlert
#Looping through the alerts and setting them.
foreach($Alert in $Alerts){
switch ($Alert.Category)
{
JobWarning{$JobWarning = "TRUE - $($Alert.Message)" }
JobFailure{$JobFailure = "TRUE - $($Alert.Message)"}
JobCancellation{$JobCancellation = "TRUE - $($Alert.Message)" }
CatalogError{$CatalogError = "TRUE - $($Alert.Message)"}
SoftwareUpdateWarning{$SoftwareUpdateWarning = "TRUE - $($Alert.Message)"}
SoftwareUpdateError{$SoftwareUpdateError = "TRUE - $($Alert.Message)" }
DatabaseMaintenanceFailure{$DatabaseMaintenanceFailure = "TRUE - $($Alert.Message)"}
IdrCopyFailed{$IdrCopyFailed = "TRUE - $($Alert.Message)"}
BackupJobContainsNoData{$BackupJobContainsNoData = "TRUE - $($Alert.Message)" }
JobCompletedWithExceptions{$JobCompletedWithExceptions = "TRUE - $($Alert.Message)"}
JobStart{$JobStart = "TRUE - $($Alert.Message)"}
ServiceStart{$ServiceStart = "TRUE - $($Alert.Message)"}
ServiceStop{$ServiceStop = "TRUE - $($Alert.Message)"}
DeviceError{$DeviceError = "TRUE - $($Alert.Message)"}
DeviceWarning{$DeviceWarning = "TRUE - $($Alert.Message)"}
DeviceIntervention{$DeviceIntervention = "TRUE - $($Alert.Message)"}
MediaError{$MediaError = "TRUE - $($Alert.Message)"}
MediaWarning{$MediaWarning = "TRUE - $($Alert.Message)" }
MediaIntervention{$MediaIntervention = "TRUE - $($Alert.Message)"}
MediaInsert{$MediaInsert = "TRUE - $($Alert.Message)"}
MediaOverwrite{$MediaOverwrite = "TRUE - $($Alert.Message)"}
MediaRemove{$MediaRemove = "TRUE - $($Alert.Message)"}
LibraryInsert{$LibraryInsert = "TRUE - $($Alert.Message)"}
TapeAlertWarning{$TapeAlertWarning = "TRUE - $($Alert.Message)"}
TapeAlertError{$TapeAlertError = "TRUE - $($Alert.Message)" }
IdrFullBackupSuccessWarning{$IdrFullBackupSuccessWarning = "TRUE - $($Alert.Message)"}
LicenseAndMaintenanceWarning{$LicenseAndMaintenanceWarning = "TRUE - $($Alert.Message)"}
default{$OtherErr = "TRUE - $($Alert.Message)" }
}
}

Monitoring other backup software & VSS Errors
Most backup application these days use VSS to create snapshots and we can monitor these to check if the snapshots have succes status or failures. The same issue as before occurs – Microsoft does not have a PowerShell VSS module or log that we can capture so we’ll have to use the eventlog that registers VSS snapshots

try{
$VSSLog = Get-EventLog -logname Application -Source VSS | Where-Object {(Get-Date $_.TimeWritten) -gt ((Get-Date).AddHours(-24))}
} catch {
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($logentry in $VSSLog){
if($logentry.EntryType -eq "Warning"){ $VSSStatus += "`nVSS Snapshot at at $($logentry.TimeGenerated) has a warning"}
if($logentry.EntryType -eq "Error"){ $VSSStatus += "`nVSS Snapshot at at $($logentry.TimeGenerated) has an error"}
}
if (!$VSSStatus) { $VSSStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Hope you’ve enjoyed this blog, the next one will be about monitoring windows performance and unexpected shutdown states. 🙂

Downloads for RMM packages:

N-Central 11.0+ – BackupExec Monitoring


	

Blog Series: Monitoring using PowerShell: Part Six – Monitoring CSV volumes for space and status

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • Cluster Shared Volume
  • PowerShell v3 or higher

Creating the monitoring sets:

We have a few clients that have shared storage, this shared storage needs to be monitored for size and usage but as Microsoft decided to make the mount points on C:\ClusterStorage its not always easy to do this from your RMM or without logging into the server. To make sure we can monitor the CSV state regarding storage used we use the following script:

Size Monioring:

param(
[string]$MonitorPercent = 5
)
try {
$CSV = Get-ClusterSharedVolume
}catch{
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($volume in $CSV.SharedVolumeInfo){
$Path        = $volume.FriendlyVolumeName
$Size        = [math]::truncate($volume.Partition.Size / 1024MB)
$FreeSpace   = [math]::truncate($volume.Partition.FreeSpace/ 1024MB)
$UsedSpace   = [math]::truncate($volume.Partition.UsedSpace/ 1024MB)
$PercentFree = [math]::truncate($volume.Partition.PercentFree)
if($PercentFree -le $MonitorPercent){
$Status +=  "$name at path $path is at $percentfree% free`n$UsedSpace GB used of $Size GB"
}
}
if (!$Status) { $Status = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Status Monitoring:

try {
$CSV = Get-ClusterSharedVolume
}catch{
$ScriptError = "Query Failed: $($_.Exception.Message)"
}
foreach($volumes in $CSV){
$Name   = $volumes.name
if($volumes.State -ne "Online"){
$Status += "$name is $($volumes.State)"
}
if (!$Status) { $Status = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

And thats how you can monitor the CSV volume state when you’re using shared storage. 🙂

Downloads for RMM packages:

N-Central 11.0+ – CSV Monitoring

Blog Series: Monitoring using PowerShell: Part five – Monitoring the Windows Search Database, iSCSI Connections, and Bitlocker state.

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • (Optional): Windows Search Service Installed
  • (Optional): TPM/Bitlocker
  • (Optional): a iSCSI connected disk
  • PowerShell v3 or higher

Creating the monitoring sets:

In this blog we’re going a bit more diverse and I will explain how to monitor very specific Windows Components.  This is just a large combination of stuff I like to monitor and see people struggling with sometimes. I hope these sets help in creating your own. 🙂

Monitor the Windows Search Database

If you’re using RDS2012 or 2016 with the Windows Search Service you know the Windows.edb database can sometimes grow explosively. A part of the solution for this can be found in the CoreCount Registery key found in my blog here. This script is to monitor the Windows search database and report if its growing out of control.

param(
[string]$MaxSizeInGB = '50'
)
$getservice = Get-service "wsearch" -ErrorAction SilentlyContinue
if($getservice.Status -eq "running"){
$CurrentLoc = Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows Search\" -name DataDirectory
$File =  Get-item -path "$($CurrentLoc.DataDirectory)\Applications\Windows\windows.edb"
$FileSize =   [math]::truncate($file.length / 1GB)
if($FileSize -gt $MaxSizeInGB){
$searchHealth = "SearchDB is $($filesize)GB - Please investigate"
}
if (!$SearchHealth) { $SearchHealth = “Healthy” }
}

Just knowing its getting large is of course only half the battle. I’ll also include the script we have to automatically rebuild the search database when this happens. Just pay mind that you do not run this while users are using the servers and schedulde this only in maintenance windows

Rebuild the searchdb:

Stop-Service Wsearch
$CurrentLoc = Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\Windows Search\" -name DataDirectory
remove-item $CurrentLoc.DataDirectory -force -Recurse
Start-Service Wsearch

Monitoring iSCSI connections and restoring them.

For our clients we often use iSCSI SANs, or iSCSI NAS devices for backups. Sometimes these devices get disconnected or lose one of the iSCSI connections. We can monitor this using get-iscsiconnection on any server 2012+ by using the following script.

try{
$Sessions = Get-iScsisession
}Catch {
$ScriptError = "Get-IscsiSession failed. : $($_.Exception.Message)"
exit
}
foreach($session in $Sessions){
if($session.isConnected -eq $false -and $session.NumberOfConnections -eq 0){
$iSCSIStatus += "`n$($Session.TargetNodeAddress) is disconnected"
}
}
if (!$iSCSIStatus) { $iSCSIStatus = “Healthy” }
if (!$ScriptEror) { $ScriptError = “Healthy” }

Now restoring them is quite simple; You can run the following command to reconnect all disconnected sessions:

Get-IscsiTarget | Connect-IscsiTarget

Or to only connect the target that is disconnected specifically:

Get-IscsiTarget | where-object IsConnected -eq $False | Connect-IscsiTarget

Monitor Bitlocker status:

We also have clients that want us to monitor the bitlocker state for them. So we’ve created a monitoring set for this too, monitoring the bitlocker state is done by checking for the string “Protection on”.

$Key =  (Get-BitLockerVolume -MountPoint C).KeyProtector
Try {
Get-WmiObject -Namespace "root\CIMV2\Security\MicrosoftVolumeEncryption" -Class Win32_EncryptableVolume |
ForEach-Object {$ID = $_.DriveLetter ;
Switch($_.GetProtectionStatus().ProtectionStatus)
{
0 {$State = "PROTECTION OFF"}
1 {$State = "PROTECTION ON - $key"}
2 {$State = "PROTECTION UNKNOWN"}
}
$ProtectionStatus =  "$ID $State"
}
} catch {
$ScriptError = "Get Bitlocker State Failed : $($_.Exception.Message)"
exit
}
if (!$ScriptEror) { $ScriptError = “Healthy” }

And that’s it!

Downloads for RMM packages:

N-Central 11.0+ – iSCSI Monitoring

N-Central 11.0+ – SearchDB Monitoring

N-Central 11.0+ – Bitlocker Monitoring

Blog Series: Monitoring using PowerShell: Part four – Using Powershell to update and maintain unifi devices

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • Ubiquity Unifi Controller running 5.x.
  • Read-only user on all sites.
  • PowerShell v3 or higher

Creating the monitoring sets:

Before we start, credit where credit is due: Most of this post is based on the code found on this reddit post. Connecting to the Unifi controller via the API is completely based on that post and I’ve only modified a few small parts to make connecting easier, I also want to make sure you understand what we’re doing here; We will be using the REST API to query the controller for information, not the access points, switches, or routers themselves. Now that we’re clear lets dive straight into it and start maintaining our unifi devices by using PowerShell.

This blog post relates to the previous blog found here. There will be no monitoring sets included as this blog is more about maintenance jobs than device management and monitoring. You can run this script from the any RMM package using schedulded tasks, of by using an Azure Function as described here.

Lets get into the scripting part of maintaining our devices, First we’ll have to connect to the API as before:


param(
[string]$URL = 'yourcontroller.controller.tld',
[string]$port = '8443',
[string]$User = 'APIUSER',
[string]$Pass = 'SomeReallyLongPassword',
[string]$SiteCode = 'default' #you can enter each site here. This way when you assign the monitoring to a client you edit this to match the correct siteID.
)
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[string]$controller = "https://$($URL):$($port)"
[string]$credential = "`{`"username`":`"$User`",`"password`":`"$Pass`"`}"

try {
$null = Invoke-Restmethod -Uri "$controller/api/login" -method post -body $credential -ContentType "application/json; charset=utf-8"  -SessionVariable myWebSession
}catch{
$APIerror = "Api Connection Error: $($_.Exception.Message)"
}

After connecting to the API we’re going to use a loop to check all the devices that require and upgrade, and send the upgrade command to them. Before moving on, make sure you are connected to the API by querying $APIError.

Upgrade all devices in the selected site($SiteID) that require upgrade:

try {
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$SiteCode/stat/device/" -WebSession $myWebSession
}catch{
$APIerror = "Query Failed: $($_.Exception.Message)"
}
foreach($Device in $APIResult.data | where-object { $_.upgradable -eq "true"}){
try{
$cmd = @"
{
"cmd":"upgrade",
"mac":"$($device.MAC)"
}
"@
$upgrade = Invoke-RestMethod -Uri "$controller/api/s/$SiteCode/cmd/devmgr" -Method post -Body $cmd -WebSession $myWebSession -ErrorAction SilentlyContinue
} catch {
$UpgradeError += "Upgrade Failed for device $($device.name) with MAC $($Device.mac) : $($_.Exception.Message)"
}
}

If you’re having alot of sites and agreed on a single maintenance cycle for all your client base, you can use the script below instead. This upgrades all sites and all devices to the latest version available on the controller.

Upgrade all devices in all sites that require upgrade:


$AllSites = Invoke-RestMethod -Uri "$controller/api/self/sites" -WebSession $myWebSession
foreach($site in $AllSites.data){
try {
$siteIDFromList = ($site.name)
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$siteIDFromList/stat/device/" -WebSession $myWebSession
}catch{
$APIerror += "`nGet Devices for $siteIDFromList Failed: $($_.Exception.Message)"
}

foreach($Device in $APIResult.data | where-object { $_.upgradable -eq "true"}){
try{
$cmd = @"
{
"cmd":"upgrade",
"mac":"$($device.MAC)"
}
"@
$upgrade = Invoke-RestMethod -Uri "$controller/api/s/$siteIDFromList/cmd/devmgr" -Method post -Body $cmd -WebSession $myWebSession -ErrorAction SilentlyContinue
} catch {
$UpgradeError = "Upgrade Failed for device $($device.name) with MAC $($Device.mac) : $($_.Exception.Message)"
}
}

And that’s it! using these scripts you can automate any firmware upgrades to only occur when you want them too and without interrupting the user experience.

Blog Series: Monitoring using PowerShell: Part three – Using Powershell to monitor Unifi Controllers

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquiti, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • Ubiquiti Unifi Controller running 5.x.
  • Read-only user on all sites.
  • PowerShell v3 or higher

Creating the monitoring sets:

Before we start, credit where credit is due: Most of this post is based on the code found on this reddit post. Connecting to the Unifi controller via the API is completely based on that post and I’ve only modified a few small parts to make connecting easier, I also want to make sure you understand what we’re doing here; We will be using the REST API to query the controller for information, not the access points, switches, or routers themselves. You can also use the examples in this post to connect to several other APIs.

Now that we’re clear lets dive straight into the connecting and querying the portal for the information.

param(
[string]$URL = 'yourcontroller.controller.tld',
[string]$port = '8443',
[string]$User = 'APIUSER',
[string]$Pass = 'SomeReallyLongPassword',
[string]$SiteCode = 'default' #you can enter each site here. This way when you assign the monitoring to a client you edit this to match the correct siteID.
)
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
[System.Net.ServicePointManager]::ServerCertificateValidationCallback = {$true}
[string]$controller = "https://$($URL):$($port)"
[string]$credential = "`{`"username`":`"$User`",`"password`":`"$Pass`"`}"
try {
$null = Invoke-Restmethod -Uri "$controller/api/login" -method post -body $credential -ContentType "application/json; charset=utf-8"  -SessionVariable myWebSession
}catch{
$APIerror = "Api Connection Error: $($_.Exception.Message)"
}

After executing the commands above we’ve established a connection to the unifi controller, we can check if the connection was succesfull by calling the $APIError variable. The Unifi API sometimes give somewhat strange errors which account to a credentials issue 99% of the time.

Based on the connection we can try to connect to the Unifi portal with different queries. I’ve generated a small list of possible queries and of course you can change these to suit your needs:

Check if devices need an update:

try {
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$SiteCode/stat/device/" -WebSession $myWebSession
}catch{
$APIerror = "Query Failed: $($_.Exception.Message)"
}
Foreach ($entry in ($APIResult.data | where-object { $_.upgradable -eq $true})){
$DeviceUpgrade += "Upgrades Available on $($Entry.Name)"
}
if(!$APIError){ $APIError = "Healthy"}
if(!$DeviceUpgrade){ $DeviceUpgrade = "Healthy"}

Check if all devices are online:

try {
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$SiteCode/stat/device/" -WebSession $myWebSession
}catch{
$APIerror = "Query Failed: $($_.Exception.Message)"
}
Foreach ($entry in ($APIResult.data | where-object { $_."state" -ne "1"})){
$DeviceState += "Device not connected $($Entry.Name)“
}
if(!$APIError){ $APIError = "Healthy"}
if(!$DeviceState){ $DeviceState = "Healthy"}

Check if device health is OK:

try {
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$SiteCode/stat/device/" -WebSession $myWebSession
}catch{
$APIerror = "Query Failed: $($_.Exception.Message)"
}
Foreach ($entry in ($APIResult.data)){
if( [math]::Round($entry.'system-stats'.cpu) -gt "80.0") { $DeviceHealthCPU += " `n $($entry.name) has CPU usage of $($entry.'system-stats'.cpu)%" }
if( [math]::Round($entry.'system-stats'.mem) -gt "80.0") { $DeviceHealthMEM += " `n $($entry.name) has a memory usage of $($entry.'system-stats'.mem)%" }
if( [math]::Round($entry.'system-stats'.uptime) -lt "300") { $DeviceHealthUptime += " `n $($entry.name) has an uptime of $($entry.'uptime') seconds" }
}
if(!$APIError){ $APIError = "Healthy"}
if(!$DeviceHealthCPU){ $DeviceHealthCPU = "Healthy"}
if(!$DeviceHealthMEM){ $DeviceHealthMEM = "Healthy"}
if(!$DeviceHealthUptime){ $DeviceHealthUptime = "Healthy"}

Get WAN status from Unifi Gateway:

try {
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$SiteCode/stat/device/" -WebSession $myWebSession
}catch{
$APIerror = "Query Failed: $($_.Exception.Message)"
}
Foreach ($entry in ($APIResult.data.wan1 | where-object { $_.enable -eq "true"})){
if($entry.up -eq $false) { $WAN1Health = " `n $($entry.name) is down" }
}
Foreach ($entry in ($APIResult.data.wan2 | where-object { $_.enable -eq "true"})){
if($entry.up -eq $false) { $WAN2Health = " `n $($entry.name) is down" }
}
if(!$APIError){ $APIError = "Healthy"}
if(!$WAN1Health){ $WAN1Health = "Healthy"}
if(!$WAN2Health){ $WAN2Health = "Healthy"}

Get blocked STP Ports

try {
$APIResult = Invoke-Restmethod -Uri "$controller/api/s/$SiteCode/stat/device/" -WebSession $myWebSession
}catch{
$APIerror = "Query Failed: $($_.Exception.Message)"
}
Foreach ($entry in ($APIResult.data.port_table | where-object { $_.stp_state -match "discard"})){
$STPError += "`n$($entry.name) has been blocked due to STP issues."
}
if(!$APIError){ $APIError = "Healthy"}
if(!$STPError){ $STPError = "Healthy"}

And thats the scripts for this session! The next session I’ll be dedicating to automated maintenance on the unifi controller and devices.

Downloads for RMM packages:

N-Central 11.0+ – Unifi Controller package

Blog Series: Monitoring using PowerShell: Part two – Using Powershell to monitor Dell systems

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • Dell OpenManage Managed Node installation
  • PowerShell v2 or higher

Creating the monitoring sets:

This is the second monitoring script is and still pretty straight forward. Instead of using the data retrieved from StorCli we’re going to use the OMReport functionality, which is a part of the Dell OpenManage Managed Node suite to extract the data to a usable array. OMReport has some cavets that aren’t present in the StorCLI files and it does not give very nicely structured data, So we will have to do a bit of string maniplulation to get good results.

If you’re only hear for ready made packages you can scroll down to the downloads, where the packages for PRTG and N-Central can be found.

Getting information from OMReport.exe

Getting information from the OpenManage suite is pretty easy – When you install OpenManage OMReport.exe is added to your path variable, meaning to get the chassis status you can simply open a command prompt on your server and type the command “omreport chassis”.  This reports a visually nice table of information. A very quick and dirty script could be:

$OmReport = omreport chassis | out-string
if($OmReport -Match "Critical"){
$ChassisStatus = $OmReport
} else {
$ChassisStatus = "Healthy"
}

This script does not contain any troubleshooting, error handling, or even a decent way to filter on how the data is displayed. It will simply match the word “Critical” and if its found dump the entire omreport output as a single string, so lets change this up to make sure it’ll give the results like we want it.

To make sure we get cleaner results, we’re going to tell OMReport we don’t want simple data to be returned, but XML data.

For this, we’ll add some commands to the OpenReport CLI tool and tell powershell that the expected output is an XML string. To do that you can do the following:

[xml]$omreport = omreport chassis -fmt xml

After running the command above we can run $omreport to get a list of XML data, pretty useless in its current state, but its actual quite simple to extract usable data out of this. OpenManage OMReport creates an item for each chassis part with its “Computed Object State”, You can query $omreport.oma.Parent for all chassis items. The great thing about this is that you can get advanced metrics by querying the entire object, or just the health statistics by querying the computedobjstatus as follows;

try {
[xml]$OmReport = omreport chassis  -fmt xml | out-string 
}catch{
$ScriptError = "omreport Command has Failed: $($_.Exception.Message)"
exit
}
$intrusionState = $omreport.oma.Parent.intrusion.computedobjstatus.strval
$voltagesState  = $omreport.oma.Parent.voltages.computedobjstatus.strval
$temperaturesState  = $omreport.oma.Parent.temperatures.computedobjstatus.strval
$fansState = $omreport.oma.Parent.fans.computedobjstatus.strval
$currentsState  = $omreport.oma.Parent.currents.computedobjstatus.strval
$powersupplyState = $omreport.oma.Parent.powersupply.computedobjstatus.strval
$powermonitoringState = $omreport.oma.Parent.powermonitoring.computedobjstatus.strval
$processorState  = $omreport.oma.Parent.processor.computedobjstatus.strval
$memoryState = $omreport.oma.Parent.memory.computedobjstatus.strval
$esmlogState = $omreport.oma.Parent.esmlog.computedobjstatus.strval
$batteriesState = $omreport.oma.Parent.batteries.computedobjstatus.strval
$sdcardState = $omreport.oma.Parent.sdcard.computedobjstatus.strval

if(!$ScriptError){ $ScriptError = "Healthy"}

So, after getting this list of information, we can alert on the chassis status simply by calling one of the variables above.

Grabbing the RAID Status via OMReport is slightly trickier. When we use the XML export funtionality we’ve used before we get alot of garbage data and non-descriptive information. To resolve this we’re using another trick out of the monitoring play: converting the data we’ve received to a CSV formatted array.

try {
omconfig preferences cdvformat delimiter=comma
$OmReport = omreport storage vdisk -fmt cdv |  select-string -SimpleMatch "ID,Status," -Context 0,5000
}catch{
$ScriptError = "omreport Command has Failed: $($_.Exception.Message)"
exit
}

$VDarray = convertfrom-csv $OmReport -Delimiter ","

foreach($VirtualDisk in $VDarray){
if($($virtualdisk.State) -eq "Ready" -or $($virtualdisk.Status) -eq "Ok"){
}else{
$RAIDStatus = "$($VirtualDisk.Name) / $($VirtualDisk.'Device Name') Has Status $($VirtualDisk.Status) / $($VirtualDisk.State)"
}
}

if(!$RAIDStatus){ $RAIDStatus = "Healthy"}
if(!$ScriptError){ $ScriptError = "Healthy"}

By running “omconfig preferences cdvformat delimiter=comma” we’re setting the export file to be a comma delimited file, next we set the export format to CDV which Dell believes is a CSV. 😉 By looping this array we get the correct results for the virtual disks. The physical disks work in exactly the same manner;

try {
omconfig preferences cdvformat delimiter=comma
$OmReport = omreport storage pdisk controller=0 -fmt cdv | select-string -SimpleMatch "ID,Status" -Context 0,5000
} catch {
$ScriptError = "omreport Command has Failed: $($_.Exception.Message)"
exit
}

$Parray = convertfrom-csv $OmReport -Delimiter ","

foreach($PhysicalDisk in $Parray){
if($($PhysicalDisk.State) -ne "Online" -or $($PhysicalDisk.Status) -ne "Ok") {
$DiskStatus += "$($PhysicalDisk.Name) / $($PhysicalDisk.'Serial No.') Has Status $($PhysicalDisk.Status) / $($PhysicalDisk.State)`n"
}

if($($PhysicalDisk.'Failure Predicted') -eq "Yes"){
$DiskStatus += "$($PhysicalDisk.Name) / $($PhysicalDisk.'Serial No.') Has a predicted failure error `n"
}
}

if(!$DiskStatus){ $DiskStatus = "Healthy"}
if(!$ScriptError){ $ScriptError = "Healthy"}

For ease of use I’ve included N-Able monitoring script for the Chassis, RAID, and physical disk monitoring

Downloads for RMM packages:

N-Central 11.0+ – Chassis Monitoring

N-Central 11.0+ – RAID Monitoring

N-Central 11.0+ – Physical Disk Monitoring

PRTG – Chassis, VD, RAID monitoring. – COMING SOON.

App-V screenshot

Blog Series: Monitoring using PowerShell: Part one – Using PowerShell to monitor MegaRaid

Preface:

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every day that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

Requirements:

  • StorCLI.exe or Storecli64.exe extracted to any path you’d like. (Download can be found here)
  • PowerShell v2 or higher

Creating the monitoring sets:

The first monitoring script is a good start but still a fairly simple script. We’re going to use the StorCli executable to get the status of the RAID array, and the physical disk data. The StorCLI executable has multiple options for outputting its data. One of the options is outputting it as a JSON string. JSON is a way to present structured data as a string.

If you’re only hear for ready made packages you can scroll down to the downloads, where the packages for PRTG and N-Central can be found.

Getting information from StorCLI:

To get the information required from StorCLI we’ll have to execute it within PowerShell and tell it to output the data to the JSON format we’re going to use.

The command to get the RAID configuration for Controller 0(Most likely the only controller in your system) is:

StorCli64.exe "/c0 /vall show j"

The command to get all physical disks status on Controller 0 is:

StorCli64.exe "/c0 /eall /sall show"

To load the configuration into a usable array we will use PowerShell to run the executable, grab the string output and then convert it to an array:

$StoreCliLocation = “C:\Executables\StorCli64.exe”
$ExecuteStoreCLI = & $StoreCliLocation "/c0 /eall /sall show j" | out-string
$ArrayStorCLI= ConvertFrom-Json $ExecuteStoreCLI

Now we can run $ArrayStorCli and we will get the entire array back with the information about the RAID array. The next step is straightforward – We check each Virtual Drive(Which is a RAID array) that does not have the state “Optl” and set the current health variable based on that.

foreach($VirtualDrive in $ArrayStorCLI.Controllers.'response data'.'Virtual Drives'){
if($($VirtualDrive.state) -ne "Optl"){
$RAIDStatus += "Virtual Drive $($Controller.'DG/VD') With Size $($Controller.'Size') is $($Controller.State)"
}

If we now check $RAIDStatus it will be empty if our RAID array is healthy, and it will contain “Virtual drive X with Size X has state X” if it has failed in any way. Of course this script needs some more steps to ensure that there is sufficient error handeling in case the StorCli send us incorrect data.

For this, See the full script below. In the full script we’ve added the parameters so we can set the correct path for the StorCli executable and a bit or error handeling in case the executable is not found. For error handeling we’re using a try and catch block.

RAID Status Monitoring:

param(
[string]$StorCLILocation = 'C:\Executables\StorCli64.exe',
[string]$StorCliCommand = "/c0 /vall show j"
)
try {
$ExecuteStoreCLI = & $StorCliLocation $StorCliCommand | out-string
$ArrayStorCLI= ConvertFrom-Json $ExecuteStoreCLI
}catch{
$ScriptError = "StorCli Command has Failed: $($_.Exception.Message)"
exit
}

foreach($VirtualDrive in $ArrayStorCLI.Controllers.'response data'.'Virtual Drives'){
if($($VirtualDrive.state) -ne "Optl"){
$RAIDStatus += "Virtual Drive $($VirtualDrive .'DG/VD') With Size $($VirtualDrive .'Size') is $($VirtualDrive.State)"
}
}
#If the variables are not set, We’re setting them to a “Healthy” state as our final action.
if (!$RAIDStatus) { $RAIDStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

Physical Disk Status Monitoring:

param(
[string]$StorCLILocation = 'C:\Executables\StorCli64.exe',
[string]$StorCliCommand = "/c0 /eall /sall show j"
)
try {
$ExecuteStoreCLI = & $StorCliLocation $StorCliCommand | out-string
$ArrayStorCLI= ConvertFrom-Json $ExecuteStoreCLI
}catch{
$ScriptError = "StorCli Command has Failed: $($_.Exception.Message)"
exit
}
foreach($Controller in $ArrayStorCLI. Controllers.'Response data'.'Drive Information'){
if($($Controller.state) -ne "Onln"){
$PhysicalStatus += “$($Controller.Model) With Disk ID $($Controller.DID) is $($Controller.State)"
}
}
#If the variables are not set, We’re setting them to a “Healthy” state as our final action.
if (!$PhysicalStatus) { $RAIDStatus = “Healthy” }
if (!$ScriptError) { $ScriptError = “Healthy” }

And that concludes todays simple MegaRAID/Lenovo RAID/LSI Raid monitoring script in PowerShell. If you want to implement this in your own monitoring system you can download the respective files below.

Any questions? Feel free to comment below! 🙂

Downloads for RMM packages:

N-Central 11.0+ – RAID Monitoring

N-Central 11.0+ – Physical Disk Monitoring

PRTG – RAID and Physical Disk monitoring (Coming soon!)

Blog Series: Monitoring using PowerShell

Hi All,

My next couple of blogs will be a series of blogs where I will be explaining on how to use PowerShell for the monitoring of critical infrastructure. I will be releasing a blog every few days that will touch on how to monitor specific software components, but also network devices from Ubiquity, third-party API’s and Office365. I will also be showing how you can integrate this monitoring in current RMM packages such as Solarwinds N-Central, Solarwinds RMM MSP and even include the required files to import the  monitoring set directly into your system.

What is RMM?

RMM stands for “Remote Management and Monitoring” – Most modern IT suppliers(Managed Services Providers) use RMM suites to assist users and monitor within complex network envirnoments. There are alot of different RMM suites and due to this its sometimes difficult to see exactly what the functionality is, This is where our PowerShell scripts will come in. Most RMM suites include a method to extend functionality by using PowerShell.

Why not use SNMP, or product X instead of PowerShell?

Good question! Why not use PowerShell for everything?! 😉 Of course I understand some people are purist and want to use SNMP for monitoring network devices and servers, and frown upon using (external) applications running on the OS , but I’ve often found that when using SNMP I cannot get the clarity or depth I want out of my monitoring sets, Also for some RMM systems using SNMP makes the system prone to more maintenance tasks and inreases the attack surface. In these cases the PowerShell version of the monitoring prevents added maintenance and keeps everything contained in the agent based monitoring system.

So, I hope you’ll enjoy the series of blogs, any questions or comments can be dropped below.

Mini-Blog: How to use Azure Functions to run PowerShell scripts

 

Lately I’ve had a couple of scripts that needed to run on a daily basis, in the past I used the task scheduler on a server for this but that would mean I had to mess around when passwords that expired, and all the other misery that is related to standard scheduled tasks.

To work around all the known limitatons I’ve created a new Azure Function App. Azure Functions allow you to run scripts at timed intervals which you set. To learn more about Azure Functions you can find information here

To create a new Azure Function App log into your Azure Portal and look for “Function App” and fill in the requested information:

Remember to choose the hosting plan “Consumption” if you are planning on only using it for minor and short running scripts, if your scripts run longer than 5 minutes you will need a paid plan. After the deployment is complete you can open the function menu and choose “Functions”, click the button Add and select PowerShell as the language:

Here you can choose the trigger, in my case I want to use a time based trigger. When selecting this it asks us to name the function, and give the format in Cron format. To get the schedule you want I suggest using this wiki to found out how to format the schedule. After creating it its a matter of  copying or entering your script and hitting the save button.

 

Happy scripting! 🙂