Monitoring with PowerShell: Monitoring Active Directory replication

I’ve often deployed domain controllers in environments that weren’t the most stable due to connectivity issues. To make sure that the domain controllers keep replicating correctly and we detect issues early we use the Active Directory cmdlets in combination with our RMM system. This makes it so we can monitor the current status of the replication and alert if it does not work for a longer period of time.

The script is suitable for server 2012R2 and up. You can use this in your RMM system to detect issues early. I like monitoring when the replication has not worked for 6 hours, but you can always change this to your own preference.

Active Directory Replication Monitoring

$AlertTime = (get-date).AddHours(-6)
$FailedArr = @()
$RepStatus = Get-ADReplicationPartnerMetadata -Target * -Partition * | Select-Object Server, Partition, Partner, ConsecutiveReplicationFailures, LastReplicationSuccess, LastRepicationResult
foreach ($Report in $RepStatus) {
    $Partner = $Report.partner -split "CN="
    if ($report.LastReplicationSuccess -lt $AlertTime) {
        $FailedArr += "$($Report.Server) could not replicate with partner $($Partner[2]) for 6 hours. please investigate"
    }
}
if (!$FailedArr) { $FailedArr = "Healthy" } 

And that’s it for today! as always, Happy PowerShelling.

Converting group policy registry preferences to PowerShell scripts

Most of the clients at my firm are moving to cloud only solutions in which we have less management options available. We can use Intune for Administrative Templates, or as we do use our RMM system as the management platform. To make sure we can use our RMM system we have several scripts that deploy registry keys in the same way as the GPO does. If you want to find what keys a GPO sets you can use this website.

Now the issue with this is that you do not directly have the option to deploy Group Policy Preferences instead of Group Policy Administrative Templates. A lot of our clients have these for applications that do not support ADMX files. To convert these I’ve created the following script.

The conversion script

 function convert-gpo($GPOFilePath) {
    [xml]$XMLFile = get-content $GPOFilePath
    $RegKeys = ($xmlfile | Select-Xml -Xpath "//Registry").Node.properties
    foreach ($regkey in $RegKeys) {
        #Converting Hkey to actual path
        switch ($regkey.hive) {
            "HKEY_CURRENT_USER" { $regkey.hive = "HKCU:\" ; break }
            "HKEY_LOCAL_MACHINE" { $regkey.hive = "HKLM:\" ; break }
        }
        $RegPath = Join-Path $regkey.hive $regkey.key
        write-host "####### $($RegPath) with $($regkey.name) #######"
        write-host "New-ItemProperty -Path $($RegPath) -name $($regkey.name) -Value $($regkey.value) -Force"
        write-host "####### NEXT KEY #######"
    }
    
}

Using this script you can run the Convert-GPO function with the file path of the export of your GPO preference file as the only parameter.

The script will then write a new script using write-host. You can copy and paste this to your RMM system, or save it as a file by running "Convert-GPO $Filename | out-file NewScript.ps1" and that’s it! as always, Happy PowerShelling.

Monitoring with PowerShell: Monitoring RDS UPD size

So our clients have RDS deployment, WVD deployments, and just in general VDI-like environments. To make sure their profile can be loaded on each machine without having to set everything up again we use UPDs.

Of course these UDP’s have a maximum size defined and need to be monitored, you can monitor the location where you host your UDP’s but that is not enough. The disk could reach a maximum size without running out of disk space on the shared location.

You can use the following script to monitor the RDS UPD size, measured against the disks own maximum size. This script only works for disks that are currently mounted – So the user has be logged in to monitor the disk size.

UPD Monitor script

$DisksInWarning = @()
$VHDS = get-disk | Where-Object {$_.Location -match "VHD"}
foreach($VHD in $VHDS){
$Volume = $VHD | Get-Partition | Get-Volume
if($Volume.SizeRemaining -lt $volume.Size * 0.10 ){ $DisksInWarning += "$($Volume.friendlyname) Less than 10% remaining"}
}

This script will alert when we have less than 10% available. Now the downside of using this script is that it only states the friendly name of the disk that is in warning. In the case of UPDs this is often a long SID or a generic name. So to make sure this is actually useful we’re also going to retrieve the SID of the users, and translate these to the username.

$DisksInWarning = @()
$VHDS = get-disk | Where-Object {$_.Location -match "VHD"}
foreach($VHD in $VHDS){
$FilePath = [io.path]::GetFileNameWithoutExtension("$($VHD.Location)")
$SIDObject = New-Object System.Security.Principal.SecurityIdentifier ($FilePath) 
$Username = $SIDObject.Translate([System.Security.Principal.NTAccount])
$Volume = $VHD | Get-Partition | Get-Volume
if($Volume.SizeRemaining -lt $volume.Size * 0.10 ){ $DisksInWarning += "$($Username.Value) UPD Less than 10% remaining. Path: $($VHD.Location)"}
}

And that’s it! as always, Happy PowerShelling!

Functional PowerShell for MSPs webinar

Hi all,

I hope you’ve enjoyed the webinar. The recording can be found here. I have to admit I was a little nervous due to over 600 attendees! The scripts used during the presentation can be found attached here.

As I said; I am available for code reviews, personal PowerShell classes, or any other automation assistance you need, just let me know! I love to help.

In any case; I hope I’ll be able to do another webinar around the new year. Thank you all for attending and asking all your questions. Questions that I did not answer will get an e-mail from me directly.

Monitoring with PowerShell: Monitoring RRAS status.

So for my clients I’ve always relied completely on the Microsoft stack – I do not like most VPN appliances but still want to offer a stable SSL VPN for all clients. Enter SSTP, I’ve blogged about SSTP before when looking at DirectAccess or even Always-on VPN.

As with all products, appliances or server I always want to know the current state of the availability. In the case of SSTP VPN on anything higher than Windows Server 2012 we have a lot of PowerShell options available but we’ll only need one – Microsoft’s get-RemoteAccessHealth. When we execute Get-RemoteAccessHealth we get a nice display of the current state of the RRAS services.

Using the script below you can directly monitor the health of the current RRAS server, it’s a simple and short one.

$RRASHealth = Get-RemoteAccessHealth -Refresh | where-object { $_.HealthState -ne "OK"}
if(!$RRASHealth) { $RRASHealth = "Healthy"}

Hope it helps, and as always, Happy PowerShelling!

Monitoring with PowerShell: Monitoring Office365 admin password changes

So when I was at Dattocon I was approached by an MSP that was using his RMM system to alert on changes of the local admin password, as he wanted to be updated every time a local admin got a new password. He did this by using an older script of mine below.

Monitoring Local Admin Password changes

$LastDay = (Get-Date).addhours(-24)
$AdminGroup = Get-LocalGroupMember -SID "S-1-5-32-544"
foreach($Admin in $AdminGroup){
$ChangedAdmins = get-localuser -sid $admin.sid | Where-Object {$_.PasswordLastSet -gt $LastDay}
}

But he came to me telling me that recently he had a need to start using this to alert on that a password needed to be updated in his documentation system to complete a process, but he was missing this for Office365 environments. I figured I would give him a hand and made the following script

Monitoring Office365 Global Admin Password changes – All tenants

$LastDay = (Get-Date).addhours(-24)
$credential = Get-Credential
Connect-MsolService -Credential $credential
$customers = Get-msolpartnercontract -All
$ChangedUsers = @()
foreach($customer in $customers){
write-host "getting users for $($Customer.Name)" -ForegroundColorGreen
$adminemails = Get-MsolRoleMember -TenantId $customer.tenantid -RoleObjectId(Get-MsolRole-RoleName"CompanyAdministrator").ObjectId
$Users = $adminemails | get-msoluser-TenantId$customer.TenantId
foreach($User in $Users){
if($User.LastPasswordChangeTimestamp -gt $LastDay){$ChangedUsers += "$($User.UserPrincipalName)has changed his password in the last 24 hours.Please update documentation to reflect.`n"}
}
}

 

Monitoring Office365 Global Admin Password Changes – Single tenant

$TenantName = "YourTenantName.onmicrosoft.com"
$LastDay = (Get-Date).addhours(-24)
$credential = Get-Credential
Connect-MsolService -Credential $credential
$Customer=Get-msolpartnercontract -All | Where-Object{$_.DefaultDomainName -eq $TenantName}
$ChangedUsers=@()
write-host"getting users for $($Customer.Name)" -ForegroundColorGreen
$adminemails = Get-MsolRoleMember -TenantId$customer.tenantid -RoleObjectId (Get-MsolRole -RoleName "CompanyAdministrator").ObjectId
$Users= $adminemails | get-msoluser-TenantId $customer.TenantId
foreach($User in $Users){
if($User.LastPasswordChangeTimestamp -gt $LastDay){$ChangedUsers +="$($User.UserPrincipalName) has changed his password in the last 24 hours.Please update documentation to reflect.`n"}
}

 

This script checks if a password has been changed in the last day, and if so alerts on it, notifying you that a global admin password has been updated and needs to be changed in the documentation. You can also use this as a warning system if you do not have anyone that should be changing these passwords.

Anyway, hope it helps, and as always. Happy PowerShelling!

Dattocon!

Hi all,

I’ll be presenting at Dattocon next week, so I will not be able to release the new blogs about monitoring with PowerShell. If you’re coming to Dattocon feel free to join my session. you can find information about the session here. The description is a little bit off as I will be talking mostly about PowerShell and automation at MSPs.

I’ll upload all resources to this blog after, including some documentation, examples to use during scripting, and the slide deck & recording.

The normal blogging schedule will resume directly after the PowerShell for MSP’s webinar. To get tickets for that, click here.

Function: Write-DRRMAlert

New-DattoRMMAlert was shared to me by Stan Lee at Datto, Stan also loves the DRY principle of coding and as such I’m also sharing it with you. You can only alert single line items, and not arrays or multiline contents as Datto does not support this.

function write-DRRMAlert ($message) {
    write-host '<-Start Result->'
    write-host "Alert=$message"
    write-host '<-End Result->'
    }

Function: Write-DRMMDiag

Similar to the top one, but generated by myself is the diagnostics printing module. We can feed this anything from objects, to arrays, to single string items 🙂

function&nbsp;write-DRMMDiag&nbsp;($messages)&nbsp;{
write-host&nbsp; '&lt;-Start&nbsp;Diagnostic-&gt;'
foreach($Message&nbsp;in&nbsp;$Messages){&nbsp;$Message}
write-host&nbsp;'&lt;-End&nbsp;&nbsp;Diagnostic-&gt;'
}

Monitoring with PowerShell: Monitoring failed logins for Office365

So this was another request by a reader; he has MFA configured for all his users, but still wants to know when the failed logon count increases. Mostly so he can warn his users that a possible spear-phising attempt might also be imminent. We know that when brute force does not work, focussed bad actors will often try the next avenue of attack.

At his request i’ve made the following scripts, one will monitor all possible locations using your partner credentials. The other will monitor only one tenant. I personally like the latter better as I’ve integrated this into my RMM so it can run and alert per client, Also it’s a little faster.

The Script

The script is designed to run at least every 4 hours, but can be run even on a 5-10 minute basis. It will get all info for the previous 4 hours, If you want to decrease on increase this you can edit line 13. Getting the logs is based on Elliot’s script to get the unified logs here. To connect with MFA, use my other blog here to generate your Secure App Model credentials.

Get Failed Logon information for all tenants

$credential = Get-Credential
Connect-MsolService -Credential $credential
$customers = Get-msolpartnercontract -All
$FilteredLogs = @()
$FailedLogonCount = 0
foreach ($customer in $customers) {
    $InitDomain = $customer.DefaultDomainName
    $DelegatedOrgURL = "https://outlook.office365.com/powershell-liveid?DelegatedOrg=" + $InitDomain
    write-host "Connecting to $($customer.Name) Security Center"
    $s = New-PSSession -ConnectionUri $DelegatedOrgURL -Credential $credential -Authentication Basic -ConfigurationName Microsoft.Exchange -AllowRedirection
    Import-PSSession $s -CommandName Search-UnifiedAuditLog -AllowClobber
    write-host "Getting last 30 minutes of logs for $($customer.Name)"
    $startDate = (Get-Date).addhours(-4)
    $endDate = (Get-Date)
    $Logs = @()
    Write-Host "Retrieving logs for $($customer.name)" -ForegroundColor Blue
    do {
        $logs += Search-unifiedAuditLog -SessionCommand ReturnLargeSet -SessionId $customer.name -ResultSize 5000 -StartDate $startDate -EndDate $endDate -Operations userloginfailed #-SessionId "$($customer.name)"
        Write-Host "Retrieved $($logs.count) logs" -ForegroundColor Yellow
    }while ($Logs.count % 5000 -eq 0 -and $logs.count -ne 0)
   $FilteredLogs +=  $logs.auditdata | convertfrom-json -ErrorAction SilentlyContinue | Select-Object UserID, ClientIP | Group-Object -Property UserID
   Foreach($item in $FilteredLogs){
       if($item.name -ne $null){ $FailedLogonCount += $item.count; $FailedLogon += "$($Item.name) has $($item.count) failed logons from the following IPs: $($Item.group.ClientIP) `n" }
   }
}

if(!$FailedLogonCount){ $FailedLogon = "Healthy"}

Get failed logins for only one tenant

$TenantName = "TenantDomain.onmicrosoft.com"

$credential = Get-Credential
Connect-MsolService -Credential $credential
$FilteredLogs = @()
$FailedLogonCount = 0
$customer = Get-msolpartnercontract | Where-Object {$_.DefaultDomainName -eq $TenantName}

$InitDomain = $customer.DefaultDomainName
    $DelegatedOrgURL = "https://outlook.office365.com/powershell-liveid?DelegatedOrg=" + $InitDomain
    write-host "Connecting to $($customer.Name) Security Center"
    $s = New-PSSession -ConnectionUri $DelegatedOrgURL -Credential $credential -Authentication Basic -ConfigurationName Microsoft.Exchange -AllowRedirection
    Import-PSSession $s -CommandName Search-UnifiedAuditLog -AllowClobber
    write-host "Getting last 30 minutes of logs for $($customer.Name)"
    $startDate = (Get-Date).addhours(-4)
    $endDate = (Get-Date)
    $Logs = @()
    Write-Host "Retrieving logs for $($customer.name)" -ForegroundColor Blue
    do {
        $logs += Search-unifiedAuditLog -SessionCommand ReturnLargeSet -SessionId $customer.name -ResultSize 5000 -StartDate $startDate -EndDate $endDate -Operations userloginfailed #-SessionId "$($customer.name)"
        Write-Host "Retrieved $($logs.count) logs" -ForegroundColor Yellow
    }while ($Logs.count % 5000 -eq 0 -and $logs.count -ne 0)
   $FilteredLogs +=  $logs.auditdata | convertfrom-json -ErrorAction SilentlyContinue | Select-Object UserID, ClientIP | Group-Object -Property UserID
   Foreach($item in $FilteredLogs){
       if($item.name -ne $null){ $FailedLogonCount += $item.count; $FailedLogon += "$($Item.name) has $($item.count) failed logons from the following IPs: $($Item.group.ClientIP) `n" }
   }


if(!$FailedLogonCount){ $FailedLogon = "Healthy"}

You can choose to alert just on the FailedLogon variable, or alert based on the actual count via FailedLogonCount. As always, Happy Powershelling.

Monitoring with PowerShell: Monitoring Office365 Azure AD Sync

We deploy Azure AD Sync for all of our clients that have hybrid environments. Sometimes the Office365 Azure AD Sync might break down, due to the Accidental Deletion Threshold or no longer perform passwords syncs due to other problems. The Azure AD sync client does tend to break from time to time.

To make sure you are alerted when this happens and can jump in on it early, there are a couple of solutions. In the Office365 portal you can easily set up Office365 to send you an email when this happens. I just don’t like receiving emails for critical infrastructure, and our RMM system has the ability to monitor cloud systems.

The scripts below can be used to monitor the Office365 Azure Active Directory Sync for one tenant, or all tenants in one go.

Single tenant script

$TenantName = "ClientDomain.onmicrosoft.com"
$AlertingTime = (Get-Date).AddHours(-24)
$credential = Get-Credential
Connect-MsolService -Credential $credential
$customer = Get-msolpartnercontract | Where-Object {$_.DefaultDomainName -eq $TenantName}

    $DirectorySynchronizationEnabled = (Get-MsolCompanyInformation -TenantId $customer.TenantId).DirectorySynchronizationEnabled
    if ($DirectorySynchronizationEnabled -eq $true) {
        $LastDirSyncTime = (Get-MsolCompanyInformation -TenantId $customer.TenantId).LastDirSyncTime
        $LastPasswordSyncTime = (Get-MsolCompanyInformation -TenantId $customer.TenantId).LastPasswordSyncTime
        If ($LastDirSyncTime -lt $AlertingTime) { $LastDirSync += "Dirsync Failed for $($Customer.DefaultDomainName) - Last Sync time was at $LastDirSyncTime `n" }
        If ($LastPasswordSyncTime -lt $AlertingTime) { $LastPasswordSync += "Password Sync Failed for $($Customer.DefaultDomainName) - Last Sync time was at $LastPasswordSyncTime `n" }
    }

if(!$LastDirSync){ $LastDirSync = "Healthy"}
if(!$LastPasswordSync){ $LastPasswordSync = "Healthy"}

Multiple tenants script

$AlertingTime = (Get-Date).AddHours(-24)
$credential = Get-Credential
Connect-MsolService -Credential $credential
$customers = Get-msolpartnercontract -All
foreach ($customer in $customers) {
    $DirectorySynchronizationEnabled = (Get-MsolCompanyInformation -TenantId $customer.TenantId).DirectorySynchronizationEnabled
    if ($DirectorySynchronizationEnabled -eq $true) {
        $LastDirSyncTime = (Get-MsolCompanyInformation -TenantId $customer.TenantId).LastDirSyncTime
        $LastPasswordSyncTime = (Get-MsolCompanyInformation -TenantId $customer.TenantId).LastPasswordSyncTime
        If ($LastDirSyncTime -lt $AlertingTime) { $LastDirSync += "Dirsync Failed for $($Customer.DefaultDomainName) - Last Sync time was at $LastDirSyncTime `n" }
        If ($LastPasswordSyncTime -lt $AlertingTime) { $LastPasswordSync += "Password Sync Failed for $($Customer.DefaultDomainName) - Last Sync time was at $LastPasswordSyncTime `n" }
    }
}

if(!$LastDirSync){ $LastDirSync = "Healthy"}
if(!$LastPasswordSync){ $LastPasswordSync = "Healthy"} 

And that’s it! With this monitoring set you’ve created a cloud-sided monitoring set that can show you exactly where your Office365 Azure AD Sync fails. As always, Happy PowerShelling.

Documenting with PowerShell: Bulk edit configurations in IT-Glue

I know last week I said I’d take a break from the monitoring blogs, but a MSP recently requested if I knew a way to mass-edit specific configuration items in IT-Glue. In his case, he was going to change the network configuration of devices and wanted a quicker way than to just click on 20 devices. It would be getting annoying fast to do that via the interface.

To make these edits easier for him, I’ve decided to quickly script the following for him:

    #####################################################################
    $APIKEy =  "APIKEYHERE"
    $APIEndpoint = "https://api.eu.itglue.com"
    $orgID = "ORGIDHERE"
    $NewGateway = "192.1.1.254"
    #####################################################################
    If(Get-Module -ListAvailable -Name "ITGlueAPI") {Import-module ITGlueAPI} Else { install-module ITGlueAPI -Force; import-module ITGlueAPI}
    #Settings IT-Glue logon information
    Add-ITGlueBaseURI -base_uri $APIEndpoint
    Add-ITGlueAPIKey $APIKEy
    $ConfigList = (Get-ITGlueConfigurations -page_size 1000 -organization_id $OrgID).data.attributes | Out-GridView -PassThru
    foreach($Config in $ConfigList){
    $ConfigID = ($config.'resource-url' -split "/")[-1]
    $UpdatedConfig = 
    @{
        type = 'Configurations'
        attributes = @{
                    "default-gateway" = $NewGateway
        }
    }
    Set-ITGlueConfigurations -id $ConfigID -data $UpdatedConfig
    }

This grabs all configurations for the specific organisation ID you’ve filled in, it then gives you a grid with all the current configurations. Using this grid you can select the configurations you’d want to make a change and apply the new gateway. Its very easy to modify other fields in bulk too, for this, check the API documentation here.

Anyway, I hope it helps some people struggling with bulk edits, and as always, happy PowerShelling!