Category Archives: Series: PowerShell Monitoring

Monitoring with PowerShell: Alerting on large Office 365 mailboxes

This script is one we’ve used in the past as a sales tool – Some companies tend to use their mailbox as a storage location more than just a mailbox. They save large attachments, use it as a personal CRM system or even just really like sending eachother large PDFs ūüėČ

When mailboxes get too large your users will start experiencing performance or caching issues. It’s also just not a good practice to have huge mailboxes, just imagine you’ll want to work on a Remote Desktop or Windows Virtual Desktop server with a 60GB mailbox cached…

Anyway; to make sure that when users experience large growth in mailboxes I’ve been using the following monitoring set in our N-central RMM system. This monitoring script alerts whenever a user has a mailbox larger than 60GB. As always I’ve included two scripts: one for a single tenant, one for multiple tenants. As always, my scripts are using the Secure Application Model.

Multiple tenant script

$ApplicationId         = 'xxxx-xxxx-xxx-xxxx-xxxx'
$ApplicationSecret     = 'TheSecretTheSecret' | Convertto-SecureString -AsPlainText -Force
$TenantID              = 'YourTenantID'
$RefreshToken          = 'RefreshToken'
$ExchangeRefreshToken  = 'ExchangeRefreshToken'
$upn                   = 'UPN-Used-To-Generate-Tokens'
$SizeToMonitor         = 60

$credential = New-Object System.Management.Automation.PSCredential($ApplicationId, $ApplicationSecret)
$aadGraphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.windows.net/.default' -ServicePrincipal -Tenant $tenantID 
$graphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.microsoft.com/.default' -ServicePrincipal -Tenant $tenantID 
Connect-MsolService -AdGraphAccessToken $aadGraphToken.AccessToken -MsGraphAccessToken $graphToken.AccessToken
$customers = Get-MsolPartnerContract -All
$LargeMailboxes = @()
foreach ($customer in $customers) {
    write-host "Getting started for $($Customer.name)" -foregroundcolor green
    $token = New-PartnerAccessToken -ApplicationId 'a0c73c16-a7e3-4564-9a95-2bdf47383716'-RefreshToken $ExchangeRefreshToken -Scopes 'https://outlook.office365.com/.default' -Tenant $customer.TenantId
    $tokenValue = ConvertTo-SecureString "Bearer $($token.AccessToken)" -AsPlainText -Force
    $credential = New-Object System.Management.Automation.PSCredential($upn, $tokenValue)
    $customerId = $customer.DefaultDomainName
    $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://ps.outlook.com/powershell-liveid?DelegatedOrg=$($customerId)&BasicAuthToOAuthConversion=true" -Credential $credential -Authentication Basic -AllowRedirection
    Import-PSSession $session -allowclobber -Disablenamechecking
    $Mailboxes = Get-Mailbox | Get-MailboxStatistics | Select-Object DisplayName, @{name = "TotalItemSize (GB)"; expression = { [math]::Round((($_.TotalItemSize.Value.ToString()).Split("(")[1].Split(" ")[0].Replace(",", "") / 1GB), 2) } }, ItemCount | Sort "TotalItemSize (GB)" -Descending
    foreach ($Mailbox in $Mailboxes) { if ($Mailbox.'TotalItemSize (GB)' -gt  $SizeToMonitor) { $LargeMailboxes += $Mailbox } }
    Remove-PSSession $session
}

if (!$LargeMailboxes) { "No Large mailboxes found" }

Single Tenant Script

$ApplicationId         = 'xxxx-xxxx-xxx-xxxx-xxxx'
$ApplicationSecret     = 'TheSecretTheSecrey' | Convertto-SecureString -AsPlainText -Force
$TenantID              = 'YourTenantID'
$RefreshToken          = 'RefreshToken'
$ExchangeRefreshToken  = 'ExchangeRefreshToken'
$upn                   = 'UPN-Used-To-Generate-Tokens'
$customertenant        = 'CustomerTenant.onmicrosoft.com'
$SizeToMonitor         = 60 

$credential = New-Object System.Management.Automation.PSCredential($ApplicationId, $ApplicationSecret)
$aadGraphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.windows.net/.default' -ServicePrincipal -Tenant $tenantID 
$graphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.microsoft.com/.default' -ServicePrincipal -Tenant $tenantID 
Connect-MsolService -AdGraphAccessToken $aadGraphToken.AccessToken -MsGraphAccessToken $graphToken.AccessToken
$LargeMailboxes = @()

    write-host "Getting Large mailboxes" -ForegroundColor green
    $token = New-PartnerAccessToken -ApplicationId 'a0c73c16-a7e3-4564-9a95-2bdf47383716'-RefreshToken $ExchangeRefreshToken -Scopes 'https://outlook.office365.com/.default' -Tenant $customertenant
    $tokenValue = ConvertTo-SecureString "Bearer $($token.AccessToken)" -AsPlainText -Force
    $credential = New-Object System.Management.Automation.PSCredential($upn, $tokenValue)
    $customerId = $customertenant
    $session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri "https://ps.outlook.com/powershell-liveid?DelegatedOrg=$($customerId)&BasicAuthToOAuthConversion=true" -Credential $credential -Authentication Basic -AllowRedirection
    Import-PSSession $session -allowclobber -Disablenamechecking
    $Mailboxes = Get-Mailbox | Get-MailboxStatistics | Select-Object DisplayName, @{name = "TotalItemSize (GB)"; expression = { [math]::Round((($_.TotalItemSize.Value.ToString()).Split("(")[1].Split(" ")[0].Replace(",", "") / 1GB), 2) } }, ItemCount | Sort "TotalItemSize (GB)" -Descending
    foreach ($Mailbox in $Mailboxes) { if ($Mailbox.'TotalItemSize (GB)' -gt  $SizeToMonitor  { $LargeMailboxes += $Mailbox } }
    Remove-PSSession $session

if (!$LargeMailboxes) { "No Large mailboxes found" } 

And that’s it! as always, Happy PowerShelling.

Ps: I’m giving a new PowerShell webinar soon. Join me by clicking this link.

Monitoring with PowerShell: Alerting on Shodan results

This is a bit of a short script again – but that’s just because sometimes life can made be real simple. Shodan is a tool that scans the entire internet and documents which open ports are available, if it is vulnerable for specific CVE’s, and lots of cool other stuff explained here.

We’ve seen some MSP’s offer a simple Shodan query and selling it as a “Dark Web Scan” – Please note that this is absolutely not a comprehensive scan and finding online exposed services is not always such a big deal, for example in controlled environments.

The script I’ve made is one we run at our clients on IP addresses where we know nothing should be listed in Shodan, networks that should not have exposed services, or just IP addresses where we want to alert on changes. Simply change the list of IPs to the list you would like to monitor.

$APIKEY = "YourShodanAPIKey"
$CurrentIP = (Invoke-WebRequest -uri "http://ifconfig.me/ip" -UseBasicParsing ).Content
$ListIPs = @("1.1.1.1","2.2.2.2",$CurrentIP)
foreach($ip in $ListIPs){
   $Shodan = Invoke-RestMethod -uri "https://api.shodan.io/shodan/host/$($ip)?key=$APIKEY"
}
if(!$Shodan) { $HealthState = "Healthy"} else { $HealthState = "Alert - $($Shodan.ip_str) is found in Shodan."} 

We also like running these scripts at our prospects as a part of a security survey, because if Shodan has found external services such as RDP on a different port it often shows bad security practices as a whole.

Getting a Shodan subscription is absolutely worth it because it gives you that little bit more of visibility on how exposed you actually. Anyway, as always happy PowerShelling!

Monitoring with PowerShell: Monitoring OneDrive status for current logged on user!

Since the release of Onedrive and Onedrive for business, a lot of system administrators have been trying to figure out how to monitor the onedrive status. Rodney Viana at Microsoft made a pretty awesome module to be able to get the current OneDrive Sync status, you can find that module here.

The issue with this module is that it has to run under the current logged on user, You don’t always have the ability to do that, especially when using RMM systems that always use the NT AUTHORITY\SYSTEM account. Now PowerShell has the ability to load .NET components as code and execute them, this gave me the idea to use impersonation of the current user in my PowerShell script to monitor OneDrive.

After messing around trying to build my own, a friend of mine pointed me to Roger Zanders post here. Combining these two scripts was pretty simple and resolved the entire onedrive monitoring issue for me.

So without any more ado; I introduce the CyberDrain OneDrive Status monitoring script for RMM systems. The script downloads the latest version of the OneDriveLib.dll, runs Get-ODStatus cmdlet under the current user, and returns the state of the OneDrive sync in $ODErrors.

The script

$Source = @"
using System;  
using System.Runtime.InteropServices;

namespace murrayju.ProcessExtensions  
{
    public static class ProcessExtensions
    {
        #region Win32 Constants

        private const int CREATE_UNICODE_ENVIRONMENT = 0x00000400;
        private const int CREATE_NO_WINDOW = 0x08000000;

        private const int CREATE_NEW_CONSOLE = 0x00000010;

        private const uint INVALID_SESSION_ID = 0xFFFFFFFF;
        private static readonly IntPtr WTS_CURRENT_SERVER_HANDLE = IntPtr.Zero;

        #endregion

        #region DllImports

        [DllImport("advapi32.dll", EntryPoint = "CreateProcessAsUser", SetLastError = true, CharSet = CharSet.Ansi, CallingConvention = CallingConvention.StdCall)]
        private static extern bool CreateProcessAsUser(
            IntPtr hToken,
            String lpApplicationName,
            String lpCommandLine,
            IntPtr lpProcessAttributes,
            IntPtr lpThreadAttributes,
            bool bInheritHandle,
            uint dwCreationFlags,
            IntPtr lpEnvironment,
            String lpCurrentDirectory,
            ref STARTUPINFO lpStartupInfo,
            out PROCESS_INFORMATION lpProcessInformation);

        [DllImport("advapi32.dll", EntryPoint = "DuplicateTokenEx")]
        private static extern bool DuplicateTokenEx(
            IntPtr ExistingTokenHandle,
            uint dwDesiredAccess,
            IntPtr lpThreadAttributes,
            int TokenType,
            int ImpersonationLevel,
            ref IntPtr DuplicateTokenHandle);

        [DllImport("userenv.dll", SetLastError = true)]
        private static extern bool CreateEnvironmentBlock(ref IntPtr lpEnvironment, IntPtr hToken, bool bInherit);

        [DllImport("userenv.dll", SetLastError = true)]
        [return: MarshalAs(UnmanagedType.Bool)]
        private static extern bool DestroyEnvironmentBlock(IntPtr lpEnvironment);

        [DllImport("kernel32.dll", SetLastError = true)]
        private static extern bool CloseHandle(IntPtr hSnapshot);

        [DllImport("kernel32.dll")]
        private static extern uint WTSGetActiveConsoleSessionId();

        [DllImport("Wtsapi32.dll")]
        private static extern uint WTSQueryUserToken(uint SessionId, ref IntPtr phToken);

        [DllImport("wtsapi32.dll", SetLastError = true)]
        private static extern int WTSEnumerateSessions(
            IntPtr hServer,
            int Reserved,
            int Version,
            ref IntPtr ppSessionInfo,
            ref int pCount);

        #endregion

        #region Win32 Structs

        private enum SW
        {
            SW_HIDE = 0,
            SW_SHOWNORMAL = 1,
            SW_NORMAL = 1,
            SW_SHOWMINIMIZED = 2,
            SW_SHOWMAXIMIZED = 3,
            SW_MAXIMIZE = 3,
            SW_SHOWNOACTIVATE = 4,
            SW_SHOW = 5,
            SW_MINIMIZE = 6,
            SW_SHOWMINNOACTIVE = 7,
            SW_SHOWNA = 8,
            SW_RESTORE = 9,
            SW_SHOWDEFAULT = 10,
            SW_MAX = 10
        }

        private enum WTS_CONNECTSTATE_CLASS
        {
            WTSActive,
            WTSConnected,
            WTSConnectQuery,
            WTSShadow,
            WTSDisconnected,
            WTSIdle,
            WTSListen,
            WTSReset,
            WTSDown,
            WTSInit
        }

        [StructLayout(LayoutKind.Sequential)]
        private struct PROCESS_INFORMATION
        {
            public IntPtr hProcess;
            public IntPtr hThread;
            public uint dwProcessId;
            public uint dwThreadId;
        }

        private enum SECURITY_IMPERSONATION_LEVEL
        {
            SecurityAnonymous = 0,
            SecurityIdentification = 1,
            SecurityImpersonation = 2,
            SecurityDelegation = 3,
        }

        [StructLayout(LayoutKind.Sequential)]
        private struct STARTUPINFO
        {
            public int cb;
            public String lpReserved;
            public String lpDesktop;
            public String lpTitle;
            public uint dwX;
            public uint dwY;
            public uint dwXSize;
            public uint dwYSize;
            public uint dwXCountChars;
            public uint dwYCountChars;
            public uint dwFillAttribute;
            public uint dwFlags;
            public short wShowWindow;
            public short cbReserved2;
            public IntPtr lpReserved2;
            public IntPtr hStdInput;
            public IntPtr hStdOutput;
            public IntPtr hStdError;
        }

        private enum TOKEN_TYPE
        {
            TokenPrimary = 1,
            TokenImpersonation = 2
        }

        [StructLayout(LayoutKind.Sequential)]
        private struct WTS_SESSION_INFO
        {
            public readonly UInt32 SessionID;

            [MarshalAs(UnmanagedType.LPStr)]
            public readonly String pWinStationName;

            public readonly WTS_CONNECTSTATE_CLASS State;
        }

        #endregion

        // Gets the user token from the currently active session
        private static bool GetSessionUserToken(ref IntPtr phUserToken)
        {
            var bResult = false;
            var hImpersonationToken = IntPtr.Zero;
            var activeSessionId = INVALID_SESSION_ID;
            var pSessionInfo = IntPtr.Zero;
            var sessionCount = 0;

            // Get a handle to the user access token for the current active session.
            if (WTSEnumerateSessions(WTS_CURRENT_SERVER_HANDLE, 0, 1, ref pSessionInfo, ref sessionCount) != 0)
            {
                var arrayElementSize = Marshal.SizeOf(typeof(WTS_SESSION_INFO));
                var current = pSessionInfo;

                for (var i = 0; i < sessionCount; i++)
                {
                    var si = (WTS_SESSION_INFO)Marshal.PtrToStructure((IntPtr)current, typeof(WTS_SESSION_INFO));
                    current += arrayElementSize;

                    if (si.State == WTS_CONNECTSTATE_CLASS.WTSActive)
                    {
                        activeSessionId = si.SessionID;
                    }
                }
            }

            // If enumerating did not work, fall back to the old method
            if (activeSessionId == INVALID_SESSION_ID)
            {
                activeSessionId = WTSGetActiveConsoleSessionId();
            }

            if (WTSQueryUserToken(activeSessionId, ref hImpersonationToken) != 0)
            {
                // Convert the impersonation token to a primary token
                bResult = DuplicateTokenEx(hImpersonationToken, 0, IntPtr.Zero,
                    (int)SECURITY_IMPERSONATION_LEVEL.SecurityImpersonation, (int)TOKEN_TYPE.TokenPrimary,
                    ref phUserToken);

                CloseHandle(hImpersonationToken);
            }

            return bResult;
        }

        public static bool StartProcessAsCurrentUser(string appPath, string cmdLine = null, string workDir = null, bool visible = true)
        {
            var hUserToken = IntPtr.Zero;
            var startInfo = new STARTUPINFO();
            var procInfo = new PROCESS_INFORMATION();
            var pEnv = IntPtr.Zero;
            int iResultOfCreateProcessAsUser;

            startInfo.cb = Marshal.SizeOf(typeof(STARTUPINFO));

            try
            {
                if (!GetSessionUserToken(ref hUserToken))
                {
                    throw new Exception("StartProcessAsCurrentUser: GetSessionUserToken failed.");
                }

                uint dwCreationFlags = CREATE_UNICODE_ENVIRONMENT | (uint)(visible ? CREATE_NEW_CONSOLE : CREATE_NO_WINDOW);
                startInfo.wShowWindow = (short)(visible ? SW.SW_SHOW : SW.SW_HIDE);
                startInfo.lpDesktop = "winsta0\\default";

                if (!CreateEnvironmentBlock(ref pEnv, hUserToken, false))
                {
                    throw new Exception("StartProcessAsCurrentUser: CreateEnvironmentBlock failed.");
                }

                if (!CreateProcessAsUser(hUserToken,
                    appPath, // Application Name
                    cmdLine, // Command Line
                    IntPtr.Zero,
                    IntPtr.Zero,
                    false,
                    dwCreationFlags,
                    pEnv,
                    workDir, // Working directory
                    ref startInfo,
                    out procInfo))
                {
                    throw new Exception("StartProcessAsCurrentUser: CreateProcessAsUser failed.\n");
                }

                iResultOfCreateProcessAsUser = Marshal.GetLastWin32Error();
            }
            finally
            {
                CloseHandle(hUserToken);
                if (pEnv != IntPtr.Zero)
                {
                    DestroyEnvironmentBlock(pEnv);
                }
                CloseHandle(procInfo.hThread);
                CloseHandle(procInfo.hProcess);
            }
            return true;
        }
    }
}


"@
New-Item 'C:\programdata\Microsoft OneDrive' -ItemType directory -Force -ErrorAction SilentlyContinue
Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/rodneyviana/ODSyncService/master/Binaries/PowerShell/OneDriveLib.dll' -OutFile 'C:\programdata\Microsoft OneDrive\OneDriveLib.dll'

Add-Type -ReferencedAssemblies 'System', 'System.Runtime.InteropServices' -TypeDefinition $Source -Language CSharp 
$scriptblock = {
    Unblock-File 'C:\programdata\Microsoft OneDrive\OneDriveLib.dll'
    import-module 'C:\programdata\Microsoft OneDrive\OneDriveLib.dll'
    $ODStatus = Get-ODStatus | convertto-json | out-file 'C:\programdata\Microsoft OneDrive\OneDriveLogging.txt'
}


[murrayju.ProcessExtensions.ProcessExtensions]::StartProcessAsCurrentUser("C:\Windows\System32\WindowsPowershell\v1.0\Powershell.exe", "-command $($scriptblock)","C:\Windows\System32\WindowsPowershell\v1.0\",$false)
start-sleep 5
$ErrorList = @("NotInstalled", "ReadOnly", "Error", "OndemandOrUnknown")
$ODStatus = (get-content "C:\programdata\Microsoft OneDrive\OneDriveLogging.txt" | convertfrom-json).value
foreach ($ODStat in $ODStatus) {
    if ($ODStat.StatusString -in $ErrorList) { $ODerrors = "$($ODStat.LocalPath) is in state $($ODStat.StatusString)" }
}
if (!$ODerrors) {
    $ODerrors = "Healthy"
}

 

And that’s it! I’m quite proud of this one as I have seen a lot of people struggle with it so I hope it helps. As always, Happy PowerShelling!

Monitoring with PowerShell: Monitoring the creation of new teams.

Since my last blog about the Secure App Model I’ve been playing with using it in different cases such as collecting automatic documentation, but also using it as an early warning tool for events our helpdesk might want to monitor.

One of these tasks is monitoring the creation of new Teams. We like keeping our clients environments in check but of course we also love the teams structure that anyone can create a team at any moment to jump right into their work. The issue with this is often one of compliance and security – We need to be aware of creation to apply the correct processes to the Team and maybe assist the end-user in further setup.

To do this I’ve made the following script; The script uses the Graph API to find all current teams. It then alerts on any team that has been created in the last day. You can of course modify this to your own needs by changing the $MonitorDate variable.

All Tenants script:

##############################
$MonitorDate = (get-date).AddDays(-1)
$ApplicationId = 'XXXX-XXXX-XXXX-XXX-XXX'
$ApplicationSecret = 'YourApplicationSecret' | Convertto-SecureString -AsPlainText -Force
$TenantID = 'YourTenantID.Onmicrosoft.com'
$RefreshToken = 'VeryLongRefreshToken'
##############################
$credential = New-Object System.Management.Automation.PSCredential($ApplicationId, $ApplicationSecret)
write-host "Generating access tokens" -ForegroundColor Green
$graphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.microsoft.com/.default' -ServicePrincipal -Tenant $tenantID 
write-host "Connecting to MSOLService" -ForegroundColor Green
Connect-MsolService -AdGraphAccessToken $aadGraphToken.AccessToken -MsGraphAccessToken $graphToken.AccessToken
write-host "Grabbing client list" -ForegroundColor Green
$customers = Get-MsolPartnerContract -All
write-host "Connecting to clients" -ForegroundColor Green

foreach ($customer in $customers) {
    write-host "Generating token for $($Customer.name)" -ForegroundColor Green
    $graphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.microsoft.com/.default' -ServicePrincipal -Tenant $customer.TenantID
    $Header = @{
        Authorization = "Bearer $($graphToken.AccessToken)"
    }
    write-host "Grabbing Teams for $($Customer.name)" -ForegroundColor Green
    $GroupUri = "https://graph.microsoft.com/v1.0/Groups?`$top=999"
    $Groups = (Invoke-RestMethod -Uri $GroupUri -Headers $Header -Method Get -ContentType "application/json").value | Where-Object { $_.resourceProvisioningOptions -eq "Team" }
    $NewGroups = foreach ($group in $Groups | Where-Object { [datetime]$_.CreatedDateTime -gt $MonitorDate }) { 
        "$($Group.displayName) has been created on $($group.createdDateTime)"
    
    }
}
if(!$NewGroups){ $NewGroups = "Healthy. No New groups have been created."} 

this script grabs the new-teams for each of the tenants under your administration and alerts into $NewGroups is a group is newer than the date you’ve given.

Single Tenant script:

##############################
$MonitorDate = (get-date).AddDays(-1)
$ApplicationId = 'XXXX-XXXX-XXXX-XXX-XXX'
$ApplicationSecret = 'YourApplicationSecret' | Convertto-SecureString -AsPlainText -Force
$TenantID = 'TenantYouWantTOLogInto.Onmicrost.com'
$RefreshToken = 'VeryLongRefreshToken'
##############################
$credential = New-Object System.Management.Automation.PSCredential($ApplicationId, $ApplicationSecret)
write-host "Generating access tokens" -ForegroundColor Green
$graphToken = New-PartnerAccessToken -ApplicationId $ApplicationId -Credential $credential -RefreshToken $refreshToken -Scopes 'https://graph.microsoft.com/.default' -ServicePrincipal -Tenant $tenantID 
write-host "Connecting to MSOLService" -ForegroundColor Green
Connect-MsolService -AdGraphAccessToken $aadGraphToken.AccessToken -MsGraphAccessToken $graphToken.AccessToken
    $Header = @{
        Authorization = "Bearer $($graphToken.AccessToken)"
    }
    write-host "Grabbing Teams" -ForegroundColor Green
    $GroupUri = "https://graph.microsoft.com/v1.0/Groups?`$top=999"
    $Groups = (Invoke-RestMethod -Uri $GroupUri -Headers $Header -Method Get -ContentType "application/json").value | Where-Object { $_.resourceProvisioningOptions -eq "Team" }
    $NewGroups = foreach ($group in $Groups | Where-Object { &#91;datetime]$_.CreatedDateTime -gt $MonitorDate }) { 
        "$($Group.displayName) has been created on $($group.createdDateTime)"
    
    }
if(!$NewGroups){ $NewGroups = "Healthy. No New groups have been created."}

And that’s it! hope it helps, and as always, Happy PowerShelling.

Monitoring with PowerShell: Monitoring disk speed

Sometimes we get complaints from clients reporting “my machine is slow” and not really get any leads to work with. The client just experiences slowness. In most cases this is due to disk speeds – the client bought some cheap computer with a 5400RPM spinner in it and is expecting it to perform just as good as any machine we supplied with an SSD.

To prevent this we could look at things such as the disk queue explained in an earlier blog here. But the problem with this type of monitoring is that its quite intermittent; You often only find the issue after a user has already complained. Because of this I’ve created a monitoring set that runs once or twice a (work) day in our RMM system. This script simply does a quick test on how fast it can actually create and read files and is used for reporting on it.

The script

So the script uses Diskspd.exe by Microsoft. You can download diskspd.exe here. You have to download the file and host it somewhere yourself. The script then downloads diskspd.exe from this location and executes two commands; a read of a 50mb file for 30 seconds, and writing a 50mb file for 30 seconds, totaling to 1 minute of performance testing.

$DownloadURL = "https://example.com/diskspd.exe"
Invoke-WebRequest -Uri $DownloadURL -OutFile "C:\Windows\Temp\diskspd.exe"
$ReadTest =  & "C:\Windows\Temp\diskspd.exe" -b128K -d30 -o32 -t1 -W0 -S -w0 -c50M test.dat
$Writetest = & "C:\Windows\Temp\diskspd.exe" -b128K -d30 -o32 -t1 -W0 -S -w100 -Z128K -c50M test.dat
$ReadResults = $readtest[-8] | convertfrom-csv -Delimiter "|" -Header Bytes,IO,Mib,IOPS,File | Select-Object IO,MIB,IOPs
$writeResults = $writetest[-1] | convertfrom-csv -Delimiter "|" -Header Bytes,IO,Mib,IOPS,File | Select-Object IO,MIB,IOPS

$ReadResults and $WriteResults will contain the IO, Mib/ps and IOPS. You can alert on each of these values. My personal preference is alering on them when the Mib/ps drops below 500mbps, because then you can be fairly sure its either a spinning disk, performance issue, or ancient SSD that needs to be replaced.

And that’s all for today! as always, happy PowerShelling.

Monitoring with PowerShell: Monitoring Event log size

Lately I’ve been getting some questions about how to handle event logs when you do not have a SIEM or log collector in place. I like thinking about these situations as I know a lot of MSPs struggle with log analysis and collection.

As a test I’ve set our event logs to never overwrite as a lot of attackers these days simply try to spam the event log so hide their traces and make sure you can’t do forensics using the Windows Event logs. To make sure that we are able to view the past event logs we can use the following monitoring set. This set alerts when there is less than 10% space available in the event log.

The script checks the Application, System, and Security log.

$Logs = get-ciminstance -ClassName Win32_NTEventlogFile | Where-Object { $_.LogfileName -eq "Application" -or $_.LogfileName -eq "System" -or $_.LogfileName -eq "Security" }
$FullLogs = @()
foreach ($Log in $Logs) {
    if ($log.MaxFileSize / $log.FileSize -gt '1.1') { $FullLogs += "$($log.LogFileName) has less than 10% available." }
}
if (!$FullLogs) {
    $LogStatus = "Healthy"
}
else {
    $Logstatus = $FullLogs
}

Now that we have reporting and alerting available, the next step is to make sure we backup the logs to a secure location when we notice the logs are filling. To do this we use another script that is triggered as a self-healing protocol by our RMM system.

$RightNow = Get-Date -Format FileDateTime
$Logs = get-ciminstance -ClassName Win32_NTEventlogFile | Where-Object { $_.LogfileName -eq "Application" -or $_.LogfileName -eq "System" -or $_.LogfileName -eq "Security" }
foreach ($log in $logs) {
    $BackupPath = Join-Path "C:\SecureSecureLocation\$RightNow" "$($log.FileName).evtx"
    New-Item -ItemType File -Path $BackupPath -Force
    Copy-Item -path $($Log.Name) -Destination $BackupPath -Force
    If ($env:ClearLogs -eq "Clear") { Clear-EventLog $log.FileName }
}

There are a couple of cool things here – First is that we use the Get-date formatting FileDateTime, which makes a nice looking format for the folder where we want to store our backups, next is that we’re using New-item before copy-item, this is to make sure that the folder structure is created before we just copy the item over.

And that’s it! This way you can monitor the size of your event logs, and back them up automatically. As always, Happy PowerShelling!

Monitoring with PowerShell: Monitoring Active Directory replication

I’ve often deployed domain controllers in environments that weren’t the most stable due to connectivity issues. To make sure that the domain controllers keep replicating correctly and we detect issues early we use the Active Directory cmdlets in combination with our RMM system. This makes it so we can monitor the current status of the replication and alert if it does not work for a longer period of time.

The script is suitable for server 2012R2 and up. You can use this in your RMM system to detect issues early. I like monitoring when the replication has not worked for 6 hours, but you can always change this to your own preference.

Active Directory Replication Monitoring

$AlertTime = (get-date).AddHours(-6)
$FailedArr = @()
$RepStatus = Get-ADReplicationPartnerMetadata -Target * -Partition * | Select-Object Server, Partition, Partner, ConsecutiveReplicationFailures, LastReplicationSuccess, LastRepicationResult
foreach ($Report in $RepStatus) {
    $Partner = $Report.partner -split "CN="
    if ($report.LastReplicationSuccess -lt $AlertTime) {
        $FailedArr += "$($Report.Server) could not replicate with partner $($Partner[2]) for 6 hours. please investigate"
    }
}
if (!$FailedArr) { $FailedArr = "Healthy" } 

And that’s it for today! as always, Happy PowerShelling.

Monitoring with PowerShell: Monitoring RDS UPD size

So our clients have RDS deployment, WVD deployments, and just in general VDI-like environments. To make sure their profile can be loaded on each machine without having to set everything up again we use UPDs.

Of course these UDP’s have a maximum size defined and need to be monitored, you can monitor the location where you host your UDP’s but that is not enough. The disk could reach a maximum size without running out of disk space on the shared location.

You can use the following script to monitor the RDS UPD size, measured against the disks own maximum size. This script only works for disks that are currently mounted – So the user has be logged in to monitor the disk size.

UPD Monitor script

$DisksInWarning = @()
$VHDS = get-disk | Where-Object {$_.Location -match "VHD"}
foreach($VHD in $VHDS){
$Volume = $VHD | Get-Partition | Get-Volume
if($Volume.SizeRemaining -lt $volume.Size * 0.10 ){ $DisksInWarning += "$($Volume.friendlyname) Less than 10% remaining"}
}

This script will alert when we have less than 10% available. Now the downside of using this script is that it only states the friendly name of the disk that is in warning. In the case of UPDs this is often a long SID or a generic name. So to make sure this is actually useful we’re also going to retrieve the SID of the users, and translate these to the username.

$DisksInWarning = @()
$VHDS = get-disk | Where-Object {$_.Location -match "VHD"}
foreach($VHD in $VHDS){
$FilePath = [io.path]::GetFileNameWithoutExtension("$($VHD.Location)")
$SIDObject = New-Object System.Security.Principal.SecurityIdentifier ($FilePath) 
$Username = $SIDObject.Translate([System.Security.Principal.NTAccount])
$Volume = $VHD | Get-Partition | Get-Volume
if($Volume.SizeRemaining -lt $volume.Size * 0.10 ){ $DisksInWarning += "$($Username.Value) UPD Less than 10% remaining. Path: $($VHD.Location)"}
}

And that’s it! as always, Happy PowerShelling!

Functional PowerShell for MSPs webinar

Hi all,

I hope you’ve enjoyed the webinar. The recording can be found here. I have to admit I was a little nervous due to over 600 attendees! The scripts used during the presentation can be found attached here.

As I said; I am available for code reviews, personal PowerShell classes, or any other automation assistance you need, just let me know! I love to help.

In any case; I hope I’ll be able to do another webinar around the new year. Thank you all for attending and asking all your questions. Questions that I did not answer will get an e-mail from me directly.

Monitoring with PowerShell: Monitoring RRAS status.

So for my clients I’ve always relied completely on the Microsoft stack – I do not like most VPN appliances but still want to offer a stable SSL VPN for all clients. Enter SSTP, I’ve blogged about SSTP before when looking at DirectAccess or even Always-on VPN.

As with all products, appliances or server I always want to know the current state of the availability. In the case of SSTP VPN on anything higher than Windows Server 2012 we have a lot of PowerShell options available but we’ll only need one – Microsoft’s get-RemoteAccessHealth. When we execute Get-RemoteAccessHealth we get a nice display of the current state of the RRAS services.

Using the script below you can directly monitor the health of the current RRAS server, it’s a simple and short one.

$RRASHealth = Get-RemoteAccessHealth -Refresh | where-object { $_.HealthState -ne "OK"}
if(!$RRASHealth) { $RRASHealth = "Healthy"}

Hope it helps, and as always, Happy PowerShelling!