Identity is a crucial component of any application. Whether you’re authenticating users on a web app or trying to query data from a back-end server, chances are you’ll need to integrate with an identity provider. Containerized applications are no exception, which is why we’ve included support for Active Directory identities in Windows Containers from the beginning. Now that Windows Server 2019 has released, we’d like to show you what we’ve been working on the last 3 years to make Windows container identity easier and more reliable.

If you’d like to jump straight into the documentation on container identity, head on over to https://aka.ms/contianers/identity

Improved Reliability

When we launched support for containers in Windows Server 2016, we set off on an adventure to redefine how people manage their apps. One of those innovations was the use of a group managed service account (gMSA) to replace the computer identity in containers. Before containers were a thing, you would typically domain-join your computer and use its implicit identity or a service account to run the app. With containers, we wanted to avoid the complexity of domain join since it would quickly become difficult to manage short-lived computer objects in Active Directory. But we knew apps would still need to use AD identities, so we came up with a solution to assign a gMSA to the container computer account at runtime. This gave the container a similar experience to being domain joined, but let multiple containers use the same identity and avoided having to store sensitive credentials in the container image.

As more customers started using gMSA with a wide variety of applications, we identified two issues that affected the reliability of gMSA with containers:

1. If the hostname of the container did not match the gMSA name, certain functionality like inbound NTLM authentication and ASP.NET Membership role lookups would fail. This was an easy doc fix, but led to a new problem…
2. When multiple containers used the same hostname to talk to the same domain controller, the last container would supersede the others and terminate their connections, resulting in random authentication failures.

To address these issues, we changed how the container identifies itself on the network to ensure it uses its gMSA name for authentication regardless of its hostname and made sure multiple connections with the same identity are properly supported. All you need to do to take advantage of this new behavior is upgrade your container host and images to Windows Server 2019 or Windows 10 version 1809.

Additionally, if you were unable to use gMSA identities with Hyper-V isolated containers in Windows versions 1703, 1709, and 1803, you’ll be glad to know that we’ve fixed the underlying issue in Windows Server 2019 and Windows 10 version 1809. If you can’t upgrade to the latest version of Windows, you can also use gMSAs with Hyper-V isolation on Windows Server 2016 and Windows 10 version 1607.

Better docs and tooling

We’ve invested in improving our documentation to make it easier for you to get started using gMSAs with your Windows containers. From creating your first gMSA account, updating your Dockerfile to help your app use the gMSA, and troubleshooting tips for when things go wrong, you’ll find it all at https://aka.ms/containers/identity.

As part of the documentation upgrade, we’ve also made it easier to get the Credential Spec PowerShell module. The source code still lives on GitHub, but you can now easily download it from the PowerShell Gallery by running Install-Module CredentialSpec. There are also a few improvements under the hood, including better support for child domains and improved validation of the account information.

Kubernetes Support

Finally, we’re excited to announce that alpha support for gMSA with Windows containers is shipping with Kubernetes version 1.14! Kubernetes takes care of copying credential specs automatically to worker nodes and adds role-based access controls to limit which gMSAs can be scheduled by users. While gMSA support is not yet ready for production use, you can try it by enabling alpha features as described in the Kubernetes gMSA docs.

Hello readers.

The AskDS blog, as with all the other TechNet and MSDN blogs, will be moving to a new home on TechCommunity.

The migration is currently in progress.

This post will be updated with the new URL when the migration is complete.

Thanks!

___________________________________________________________________________________________________________________________

IMPORTANT ANNOUNCEMENT FOR OUR READERS!

AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!

We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!

Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.

If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.

NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!

As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!

__________________________________________________________________________________________________________________________

NOTE: This blog is going through a specific issue in order to help show the steps involved in troubleshooting this type of issue. The process name(s) referenced in this content, except for System, can be any process, and not just the process used as an example in this post, and in no way are there expectations, no is it implied, that this particular process will cause you any problems!

Hey everyone, Konstantin Chernyi here. I’m a Premier Field Engineer at Microsoft Russia and today I’m gonna tell you a real-world story that happened recently. Long story short, a customer asked me: “How do I understand why the System process is consuming 100% of a single CPU core on my machine?”

Whenever I see description or request like this, my first step is to collect ETW trace. In the past, I would send a long instruction how to install Windows Performance Toolkit
https://docs.microsoft.com/en-us/windows-hardware/test/wpt/ and how to use xperf with appropriate kernel flag to collect data, but these days, thanks to PG, I don’t need to do that anymore. Since very first release of Windows 10/Window server 2016 – WPR (Windows Performance Recorder) with a lot of predefined profiles shipped with the OS. So, all we need is – collect short trace at the very exact moment when problem exist. In this case we used CPU profile:

Wpr -start CPU

<wait 10-15 seconds, so we have enough information>

Wpr -stop C:temptrace.etl

As soon as customer provided the trace, I opened it in WPA (Windows Performance Analyzer).

The CPU is busy indeed:

Top CPU consumer – System:

The System process has multiple threads, but only one TID (#76) is very active and consuming CPU time:

With public symbols https://docs.microsoft.com/en-us/windows/desktop/dxtecharts/debugging-with-symbols we can go deeper and review function called in this thread:

ntoskrnl.exe!KeBalanceSetManager huh, time to remember what I’ve read in Windows Internals back in a days. On page 188 of the second part of Windows Internals 6^th version, you can find explanation of this function:

The balance set manager (KeBalanceSetManager, priority 16). It calls an inner routine, the

working set manager (MmWorkingSetManager), once per second as well as when free

memory falls below a certain threshold. The working set manager drives the overall memory

management policies, such as working set trimming, aging, and modified page writing.

Hmm, it looks like they are facing memory related problems, but they didn’t mention it, and the initial request was about high CPU consumption by System process. Let’s look at the memory info that we have in the trace, which isn’t much since we used the CPU profile, but at least let’s give it a try.

According to info in the trace, this box has 4.9GB in Zero and Free lists, and 2.7GB in Standby Lists, which gives us 7.6GB available memory…confusing, isn’t it? It looks like this box has plenty of available memory, but system calls KeBalanceSetManager routine every second:

Also, if you take a closer look at memory utilization, you can see that the Page Pool Commit is around 11.9GB, which is a lot:

Here is are some good articles to read on this area:

https://blogs.technet.microsoft.com/markrussinovich/2009/03/10/pushing-the-limits-of-windows-paged-and-nonpaged-pool/

https://docs.microsoft.com/en-us/windows/desktop/memory/memory-pools

Total installed RAM on this box – 32GB:

It looks like we need a better look at what’s happening in memory…the best way to do that – memory dump. In this case we decide to try generating mirror memory dump via livekd.exe
https://docs.microsoft.com/en-us/sysinternals/downloads/livekd

So with that in mind, I asked the customer to grab a dump by executing livekd -ml -o C:tempm.dmp

Next I’ll use WinDbgX aka WindDbg preview https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-using-windbg-preview and the Mex extension https://www.microsoft.com/en-us/download/details.aspx?id=53304

Let’s start by reviewing virtual memory states:

From here we can see that the PagedPool Commit = 11.95GB, but the PagedPool usage is zero, I assume it’s because we used mirror dump. Anyway, from this data we clearly see that we definitely have a memory issue since Available pages = 2.87MB and there are a lot pool allocation failures. First let’s see first memory usage by process:

Heh, SynTPEnh.exe consumed 9.48GB of RAM, very well. Now let’s see what’s in Paged Pool:

Hmm, bunch of Token objects, let’s shed some light and dump them all:

Again SynTPEnh.exe, let’s calculate all token handles:

Keeping in mind that every handle gives us about 8 rows, so we need to divide it and get in total about 160k handles, which is a lot, and almost all of them belong to SynTPEnh.exe.

On the customer machine, the application event log was full of events like this one:

So the next step is to check if there are any “zombie processes” https://randomascii.wordpress.com/2018/02/11/zombie-processes-are-eating-your-memory/, and we can see a lot of them:

The majority of the zombie processes are in session 1, which is not in used in this case since the customer is using RDP to connect to this machine:

The customer said that on this tower, there weren’t any pointing devices except a mouse, so it was safe to uninstall it and check. After uninstallation memory consumption immediately went down, and there was nothing to do for the system process, so it goes almost Idle.

So what was our root cause?

For some reason the process SynTPEnh.exe was being created every 4 seconds, do some work for about 1 sec, and then crash. The token handle from parent services that started this process is not released, which leads us to memory leak and high CPU consumption. Here is an example from the trace that process SynTPEnh.exe come and go all the time:

Wbr, Konstantin.

Enable-ExperimentalFeature -Name PSCommandNotFoundSuggestion

PS> Get-Commnd
Get-Commnd : The term 'Get-Commnd' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At line:1 char:1
+ Get-Commnd
+ ~~~~~~~~~~
+ CategoryInfo          : ObjectNotFound: (Get-Commnd:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException


Suggestion [4,General]: The most similar commands are: Get-Command, Get-Content, Get-Job, Get-Module, Get-Event, Get-Host, Get-Member, Get-Item, Set-Content.

Enable-ExperimentalFeature -Name PSImplicitRemotingBatching

Enable-ExperimentalFeature -Name PSTempDrive

PS> "Hello World!" > Temp:/hello.txt
PS> Get-Content Temp:/hello.txt
Hello World!

Enable-ExperimentalFeature -Name PSUseAbbreviationExpansion

PS> i-arsavsf

PS> Import-AzRecoveryServicesAsrVaultSettingsFile

The post General Availability of PowerShell Core 6.2 appeared first on PowerShell.

Hi there! Stanislav Belov here again to bring you the next issue of the Infrastructure + Security: Noteworthy News series! 

As a reminder, the Noteworthy News series covers various areas, to include interesting news, announcements, links, tips and tricks from Windows, Azure, and Security worlds on a monthly basis.

Cloud-based services have significantly increased productivity for today’s workforce, prompting users to adopt new cloud apps and services and making it a challenge for you to keep up. Microsoft Cloud App Security (MCAS), a cloud access security broker (CASB), helps you gain control over shadow IT with tools that give you visibility into the cloud apps and services used in your organization, asses them for risk, and provide sophisticated analytics. You can then make an informed decision about whether you want to sanction the apps you discover or block them from being accessed.

Read the full blog here.

Cloud-based services have significantly increased productivity for today’s workforce, prompting users to adopt new cloud apps and services and making it a challenge for you to keep up. Microsoft Cloud App Security (MCAS), a cloud access security broker (CASB), helps you gain control over shadow IT with tools that give you visibility into the cloud apps and services used in your organization, asses them for risk, and provide sophisticated analytics. You can then make an informed decision about whether you want to sanction the apps you discover or block them from being accessed.

Read the full blog here.

In our last blog, Step 5. Set up mobile device management, we introduced ContosoCars to illustrate the journey of implementing Intune as part of your UEM strategy. We continue their story to demonstrate how you can enhance endpoint security by managing mobile apps and tracking the deployment.

Read the full blog here.

The post LiveFyre commenting will no longer be available on the PowerShell Gallery appeared first on PowerShell.

Over the past few months, the team has been working hard to make the PowerShell Gallery as accessible as possible. This blog details why it matters and what work has been done.

Why making the PowerShell Gallery more accessible was a priority

Accessible products change lives and allow everyone to be included in our product. Accessibility is also a major component of striving toward Microsoft’s mission to “Empower every person and every organization on the planet to achieve more.” Improvements in accessibility mean improvements in usability which makes the experience better for everyone. In doing accessibility testing for the Gallery, for example, we found that it was confusing for users to distinguish between “deleting” and “unlisting” packages. By clearly naming this action in the UI, it makes the process of unlisting a package more clear for all package owners.

The steps taken to make the PowerShell Gallery more accessible

The first part of the process focused on bug generation and resolution. We used scanning technology to ensure that the Gallery alerts and helper texts were configured properly, and were compatible with screen reading technology. We use Keros scanning which is Microsoft’s premier accessibility tool to identify accessibility issues and worked to triage and fix the detected issues.

For the second part of the process, we undertook a scenario-focused accessibility study. For the study, blind or visually impaired IT professionals went through core scenarios for using the Gallery. These scenarios included: finding packages, publishing packages, managing packages, and getting support. The majority of the scenarios focused on searching for packages as we believe this is the primary way customers interact with the Gallery. After the study concluded we reviewed the results and watched recordings of the participants navigating through our scenarios. This process allowed us to focus on improving our lowest performing scenarios by addressing specific usability improvements. After making these improvements we underwent a review by accessibility experts to assure we had high usability and accessibility.

Usability Improvements

• Screen Reader Compatibility: Screen reader technologies make consuming web content accessible so we underwent thorough review, and improvement, to ensure that the Gallery was providing accurate, consistent, and helpful information to screen readers. Some examples of areas we improved:
  • Accurate Headers
  • Clearly labeled tables
  • Helpful tool tips
  • Labeled graph node points
• Improved Aria Tags: Accessible Rich Internet Applications (Aria) is a specification that makes web content more accessible by passing helpful information to assistive technologies such as screen readers. We underwent a thorough review, and enhancement, of our Aria tags to make sure they were as helpful as possible. One improvement we made, for example, was an ARIA description explaining how to use tags in the Gallery search bar.
• Renamed UI elements to be more descriptive: Through our review we noticed we were generating some confusion by labeling the unlist button as “delete” and we worked to fix these types of issues.
• Filters: We added filters for the operating system to make it easier to find compatible packages.
• Results description: we made searching for packages more straightforward by displaying the total number of results and pages.
• Page Scrolling: we made searching for packages easier by adding multi-page scrolling.

Reporting Issues

Our goal is to make the Gallery completely user friendly. If you encounter any issues in the PowerShell Gallery that make it less accessible/usable we would love to hear about it on our GitHub page. Please file an issue letting us know what we can do to make the Gallery even more accessible.

The post The PowerShell Gallery is now more Accessible appeared first on PowerShell.

(This post is co-authored by Adrian Moore, Senior Program Manager, and Mayunk Jain, Product Manager, Microsoft 365 Security)

Whether you have an official BYOD (bring your own device) policy or not, chances are you caught up on some work email this weekend on your mobile phone. If so, you’re not alone; more than 80% of employees admit using non-approved SaaS apps for work purposes, including mobile email. What is worth noting, is that 63% of confirmed data breaches involve weak, default, or stolen passwords. According to Verizon's 2018 Breach Investigations report, 92 percent of malware is still delivered by email. 

As an IT leader investing in Microsoft 365 modern workplace to meet cyber-security challenges head-on, secure email access is likely to be a key part of your strategy. In this article, we take a technical deep dive into the integrated approach of Microsoft Enterprise Mobility + Security (EMS) and Microsoft Outlook for iOS and Android devices, that we consider the gold standard of secure mobile email access.

How it works

Let us dig deeper and explore the configuration settings to deliver the rich experience of Microsoft secure mobile email. Check out the Sway below, or download the PDF

___________________________________________________________________________________________________________________________

IMPORTANT ANNOUNCEMENT FOR OUR READERS!

AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!

We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!

Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.

If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.

NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!

As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!

__________________________________________________________________________________________________________________________

Hi everyone, Ingmar Oosterhoff, Johannes Freundorfer, and Matthias Herfurth here continuing from our previous blog, which can be found at https://techcommunity.microsoft.com/t5/Core-Infrastructure-and-Security/MSIX-Package-Support-Framework-Part-One-The-Blueprint/ba-p/363594. In this post, we will now proceed with preparing our machine to make use of the PSF later on.

NOTE: To save on resources and creating additional Virtual Machines I’m going to be using my MSIX packaging machine for this (makes sense to me anyway).

Modifying an existing MSIX package is nice and easy using MakeAppx.exe. As MakeAppx.exe is part of the Microsoft Windows 10 SDK, the first step in our process will be to download it from: https://developer.microsoft.com/en-US/windows/downloads/windows-10-sdk

Once downloaded, install it, and select 2 components, as shown below:

Next, create some folders to use as working directories…

I’ve created a Resources folder in C: with a MakeAppx (for later use) and Nuget folder as subfolders as shown in the image below:

The next step in the process will be downloading the Package Support Framework using Nuget as follows.

• Download Nuget from https://www.nuget.org/downloads
• Save the nuget.exe in the c:resourcesNuget folder
• Start cmd.exe
• CD to the c:resourcesnuget folder
• Run the following command: nuget install Microsoft.PackageSupportFramework

This will create a subfolder in c:resourcesNuget folder containing the framework

Snapshot your VM to allow you to revert to this state.

Now we have the tools in our next post, where we’ll download and compile a simple “made to break” ready to repair application, stay tuned!

Thanks for reading!

# To list all modules that tagged as DSCResourceKit
Find-Module -Tag DSCResourceKit 
# To list all DSC resources from all sources 
Find-DscResource

Install-Module -Name < module name >

Install-Module -Name xWebAdministration

Update-Module

Get-DscResource

The post DSC Resource Kit Release April 2019 appeared first on PowerShell.

Over the last couple months I’ve been writing about Intune’s journey to become a globally scaled cloud service running on Azure.  I’m treating this as Part 3 (here’s Part 1 and Part 2) of a 4-part series.

Today, I’ll explain how we were able to make such dramatic improvements to our SLA’s, scale, performance, and engineering agility.

I think the things we learned while doing this can apply to any engineering team building a cloud service.

Last time, I noted the three major things we learned during the development process:

1. Every data move that copies or moves data from one location to another must have data integrity checks to make sure that the copied data is consistent with the source data.  We discovered that there are a variety of efficient/intelligent ways to achieve this without requiring an excessive amount of time or memory. 
2. It is a very bad idea to try building your own database for these purposes (No-SQL or SQL, etc), unless you are already in the database business.
3. It’s far better to over-provision than over-optimize.  In our case, because we set our orange line thresholds low, we had sufficient time to react and re-architect.

After we rolled out our new architecture, we focused on evolving and optimizing our services/resources and improving agility.  We came up with 4 groups of goals to evolve quickly and at high quality:

• Availability/SLAs
• Scale
• Performance
• Engineering agility

Here’s how we did it:

#1: Availability/SLAs

The overarching goal we defined for availability/SLA (strictly speaking, SLO) was to achieve 4+ 9’s for all our Intune services.

Before we started the entire process describe by this blog series, less than 25% of our services were running at 4+ 9’s, and 90% were running at 3+ 9’s.

Clearly something needed to change.

First, a carefully selected group of engineers began a systematic review of where we needed to drive SLA improvements across the 150+ services.  Based on what we learned here, we saw, over the next six months, dramatic improvements.  This review uncovered a variety of hidden issues and the fixes we rolled out made a huge difference.  Here are a few of the big ones:

• Retries:
  Our infrastructure supported a rudimentary form of retries and it needed some additional technical sophistication, specifically in terms of customized request timeouts. Initially, there was no way to cancel a request if it took more than a specified set time for a specific service. This meant that a request could never really be retried, because if a timeout happened, it most likely exceeded the threshold for the end-end operation.  To address this, we added a request timeout feature that enabled services to specify custom limits on the maximum time a request can take before being canceled. This allowed services to specify appropriate time limits and give them several other retry semantics (such as backoffs, etc.) within the bounds of the overall end-end operation. This was a huge improvement and it reduced our end-end timeouts by more than half.
• Circuit breakers:
  It didn’t take long for us to realize that retries can cause a retry storm and result in timeouts becoming much worse. We added a circuit breaker pattern to handle this.
• Caching:
  We started caching responses for repeated requests that matched the same criteria without breaking security boundaries.
• Threading:
  During cold starts and request spikes, we noticed that the underlying framework (.NET) took time to spin off threads. To address this, we adjusted the minimum worker threads a service needs to maintain to account for these behaviors and made it configurable on a per-service basis.  This almost eliminated all the timeouts that happened during these spikes and/or cold starts.
• Intelligent routing:
  This was a learning algorithm that determined the target service that had the best chance to succeed the request. This kind of routing avoided a hung or slow node, a deadlocked process, a slow network VM, and any other random issues experienced by the services. In a distributed cloud service operating at scale, these kinds of underlying issues must be expected and are more of a norm than an exception. This ended up being a critical feature for us to design and implement, and it made a huge difference across the board, especially when it came to reducing tail latencies.
• Customized configurations:
  Each of our services had slightly different requirement or behavior, and it was important for us to provide knobs to customize certain settings for optimal behavior. Examples of such customized settings included: http server queue lengths, service point count, max pending accepts, etc.

The result of all the above efforts was phenomenal.  The chart below demonstrates this dramatic improvement after the changes were rolled out.

You’ll notice that we started with less than 25% of services at 4+ 9’s, and by the time we rolled out all the changes, 95% or more of our services were running at 4+ 9’s!  Today, Intune maintains 4+ 9’s for over 95% of our services across all our clusters around the world.

#2: Scale

The re-architecture process enabled us to primarily use scale out of the cluster to handle our growth. It was clear that the growth we were experiencing required us to additionally optimize in scale-up improvements.  The biggest workload for Intune is triggered when a device checks-in to the service in order to receive policies, settings, apps, etc. – and we chose this workload as our first target.

The scale target goal we set was 50k devices checking in within a short period (approximately 10 minutes) for a given cluster.  For reference, at the time we set this goal, our scale was at 3k devices in a 10-minute window for an individual cluster – in other words our scale had to increase by about 17x.  As with the SLA work we did, a group of engineers pursued this effort and acted as a single unit to tackle the problem.  Some of the issues they identified and improved included:

• Batching:
  Some of the calls were made in a sequential manner and we identified a way for these calls to be batched together and sent in one request. This avoided multiple round trips and serialization/deserialization costs.
• Service Instance Count:
  Some of the critical services in our cluster were running with an instance count. We realized that these were the first bottlenecks that prevented us from scaling up.  By simply increasing the instance count of the services without changing the node or cluster sizes we completely eliminated these bottlenecks.
• Caching:
  Some of the properties in an account/tenant or user were frequently accessed. These properties were accessed by various different calls to the service(s) which held this data. We realized that we can cache these properties in the token that a request carried.  This eliminated the need for many calls to other services and the latencies or resource consumptions associated with them.
• Reduce Calls:
  We developed several ways to reduce calls from one service to another. For example, we used a Bloom Filter to determine if a change happened, and then we used that information to reduce a load of about 1 million calls to approximately 10k
• Leverage SLA improvements:
  We leveraged many of the improvements called out in the SLA section above, even though both efforts were operating (more or less) in parallel at the time. We also leveraged the customized configurations to experiment, learn, and test.

By the end of this exercise, we were successfully able to increase the scale from 3k devices checking-in to 70k+ device check-ins – an increase of more than 23x -- and we did this without scaling out the cluster!

#3: Performance

Our goal for performance had a very specific target:  Culture change.

We wanted to ensure that our performance was continuously evaluated in production and we wanted to be able to catch performance regressions before releasing to production.

To do this, we first used Azure profiler and associated flame graphs to perform continuous profiling in production.  This process showed our engineers how to drive several key improvements, and subsequently, it became a powerful daily tool for the engineers to determine bottlenecks in code, inefficiencies, high CPU usage, etc.  Some of the improvements identified by the engineering team as a result of this continuous profiling include:

• Blocking Calls:
  Some of the calls made from one service to another were incorrectly blocking instead of following async patterns.  We fixed this by removing the blocking calls and making it asynchronous. They resulted in reduced timeouts and thread pool exhaustions.
• Locking:
  Another pattern we noticed using the profiler was lock contention between threads.  We were clearly able to examine these via code that used the profiler’s call stacks to fix the bugs and remove the associated latencies.
• High CPU:
  There were numerous instances where we were easily able to catch high CPU situations using the profiles and quickly determine root causes and fixes.
• Tail latency:
  While investigating certain latencies associated with devices checking in or our portal flows, we noticed that some of the search requests were being sent across to all the partitions of a service. In many cases, there is just one partition that holds this data and the search can be performed against that single partition instead of fanning out across all of them.  We successfully made optimizations to do a search directly against the partition that held the data – and the result was a drop in latency from 200 msec to less than 15 msec (see chart below).  The end result was improved response times in devices checking in and faster data retrievals in our ITPro portal.

Our next action was to start a benchmark service that consistently and constantly ran high-traffic in our pre-production environments.  Our goal here was to catch performance regressions.  We also began running a consistent traffic load (that is equivalent to production loads) across all services in our pre-production environments.  We made a practice of considering a drop in our pre-production environment as a major blocker for production releases.

Together, both of these actions become a norm in the engineering organization, and we are proud of this positive culture change in meeting the performance goal.

#4: Engineering Agility

As called out in the first post in this series, Intune is composed of many independent and decoupled Service Fabric services. The development and deployment of these services, however, are genuinely monolithic in nature.  They deploy as a single unit, and all services are developed in a single large repo – essentially, a monolith.  This setup was an intentional decision when we started our modern service journey because a large portion of the team was focusing on the re-architecture effort and our cloud engineering maturity was not yet fully realized.  For these reasons we chose simplicity over agility.  As we dramatically developed the feature investments we were making (both in terms of the number of features and the number of engineers working them), we started experiencing agility issues.  The solution was decoupling the services in the monolith from development, deployment, and maintenance perspectives.  To do this we set three primary goals for improving agility:

• Building a service should complete within minutes (this was down from 7+ hrs)
• Pull requests should complete in minutes (down from 1+ day)
• Deployments to our pre-prod environments should occur several times per day (down from once or twice per week)

As indicated above, our agility was initially hurting us when it came to rapidly delivering features.  Pull requests (PR) would sometimes take days to complete due to the aforementioned monolithic nature of the build environments – this meant that any change anywhere by anyone in Intune would impact everyone’s PR.  On any given day, the churn was so high that it was extremely hard to get stable builds and fast builds or PRs.  This, in turn, impacted our ability to deploy this massive build to our internal dogfood environments.  In the best case, we were able to deploy once or twice per week.  This, obviously, was not something we wanted to sustain.

We made an investment in developing and decoupling the monolithic services and improve our agility.  Over a period of 2+ years, we invested two major improvements:

•  Move to individual GIT repos:
  Services moved from a proprietary source depot monolith branch to their own individual GIT repos. This decoupled development, PRs, unit and component tests, and builds.  The change resulted in build times getting completed in around 30 minutes – a huge difference from the previous 7-8 hours or more.
• Carve out of Micro Services from Monolith:
  Services were carved out of the Service Fabric application and packaged into their own application, and they were turned into their own independent deployable unit. We referred to such an application as a micro service.

As this investment progressed and evolved, we started seeing huge benefits. The following demonstrate some of these:

• Build/PR Times:
  For microservices, we reduced the time that a service typically completes a build to within 30 minutes from the previous 7+ hours. Similarly, the monolith saw an improvement to 2-3 hours from the 7 hours. A similar improvement happened in PR times as well, to a few minutes for micro services (from 1+ day).
• Deployments to Pre-prod Dogfood Environments:
  With the monolith, successful deployments to pre-production dogfood environments would take us minimum of 1 day and, in some extreme cases, up to a week. With the investments above, we are now able to complete several deployments per day across the monolith and micro services.  This is primarily because of faster deployment times (due to the parallel deployments of micro services) and the number/volume of services that have been removed from the monolith into their own micro services.

The chart below demonstrates one such an example.  The black line shows that we went from single digits to 1000’s of deployments per month in production environments.  In pre-production dogfood environments, this was even higher – typically reaching 10’s of deployments per day across all the services in a single cluster.

Challenges:

Today, Intune is part monolith and part micro services.  Eventually, we expect to compose 40-50 micro services from the existing monolith.  There are challenges in managing micro services due to the way they are independently created and managed and we are developing tooling to address some of the micro service management issues.  For example, binary dependencies between micro services is an issue because of versioning issues.  To address this, we developed a dependency tool to identify conflicting or missing binary dependencies between micro services.  Automation is also important if a critical fix needs to be rolled out across all micro services in order to mitigate a common library issue.  Without proper tooling, it can also be very hard and time consuming to propagate the fix to all micro services.  Similarly, we are developing tooling to determine all the resources required by a micro service, as well as all the resource management aspects, such as key rotation, expiration, etc.

Learnings

There were 3 learnings from this experience that are applicable to any large-scale cloud service:

1. It is critically important to set realistic and achievable goals for SLA and scale – and then be persistent and diligent in driving towards achieving these goals. The best outcomes happen when a set of engineers from across the org work together as a unit towards a common goal.  Once you have this in place, make incremental changes; the cumulative effect of all the small changes pays significant dividends over time.
2. Continuous profiling is a critical element of cloud service performance. It helps in reducing resource consumption, tail latencies, and it indirectly benefits all runtime aspects of a service.
3. Micro services help in improving agility. Proper tooling to handle patches, deployments, dependencies, and resource management are critical to deploy and operate micro services in a high-scale distributed cloud service.

Conclusion

The improvements that came about from this stage of our cloud journey have been incredibly encouraging, and we are proud of operating our services with high SLA and performance while also rapidly increasing the scale of our traffic and services.

The next stage of our evolution will be covered in Part-4 of this series:  A look at our efforts to make the Intune service even more reliable and efficient by ensuring that the rollout of new features produce minimal-to-no impact to existing feature usage by customers – all while continuing to improve our engineering agility.

The post The Next Release of PowerShell – PowerShell 7 appeared first on PowerShell.

The post PowerShell Core Release Improvements appeared first on PowerShell.

___________________________________________________________________________________________________________________________

IMPORTANT ANNOUNCEMENT FOR OUR READERS!

AskPFEPlat is in the process of a transformation to the new Core Infrastructure and Security TechCommunity, and will be moving by the end of March 2019 to our new home at https://aka.ms/CISTechComm (hosted at https://techcommunity.microsoft.com). Please bear with us while we are still under construction!

We will continue bringing you the same great content, from the same great contributors, on our new platform. Until then, you can access our new content on either https://aka.ms/askpfeplat as you do today, or at our new site https://aka.ms/CISTechComm. Please feel free to update your bookmarks accordingly!

Why are we doing this? Simple really; we are looking to expand our team internally in order to provide you even more great content, as well as take on a more proactive role in the future with our readers (more to come on that later)! Since our team encompasses many more roles than Premier Field Engineers these days, we felt it was also time we reflected that initial expansion.

If you have never visited the TechCommunity site, it can be found at https://techcommunity.microsoft.com. On the TechCommunity site, you will find numerous technical communities across many topics, which include discussion areas, along with blog content.

NOTE: In addition to the AskPFEPlat-to-Core Infrastructure and Security transformation, Premier Field Engineers from all technology areas will be working together to expand the TechCommunity site even further, joining together in the technology agnostic Premier Field Engineering TechCommunity (along with Core Infrastructure and Security), which can be found at https://aka.ms/PFETechComm!

As always, thank you for continuing to read the Core Infrastructure and Security (AskPFEPlat) blog, and we look forward to providing you more great content well into the future!

__________________________________________________________________________________________________________________________

Hi all! Johannes Freundorfer, Ingmar Oosterhoff, and Matthias Herfurth back again for part 3 of our series!

Using the tools downloaded to our Virtual Machine in the previous blog (https://techcommunity.microsoft.com/t5/Core-Infrastructure-and-Security/MSIX-Package-Support-Framework-Part-2-Preparation/ba-p/393864), we’re now going to fix a “made to break” application provided by Microsoft.

This application can be downloaded from Github and needs to be compiled with Visual Studio.

The sources of this application can be found here: https://github.com/Microsoft/MSIX-PackageSupportFramework

Choosing “Clone or Download” will allow you to download the whole set of files as a ZIP-container.

1. Expand the. zip to the folder structure created in the previous blog. Once expanded the folder structure should look like this:

2. The next step is to review and compile the broken Sample application. To do this we need Microsoft Visual Studio. You could install that yourself or take the easy route and use the Quick create template in Hyper-V manager on your Windows 10 machine.

3. Select the Windows 10 dev environment VM, and a new VM with Visual Studio is up and running in minutes.

4. We can then copy our Resources folder over to this VM. Oh, and did you notice there is also a VM available with the MSIX Packaging Tool Environment?

5. Once Visual Studio is up and running, open the following file:

“C:resourcesNugetMSIX-PackageSupportFramework-mastersamplesPSFSamplePSFSample.sln”

6. As this is a project from an external source, some warnings will pop up. To be able to proceed you’ll need to accept those. If you missed some features during Visual Studio installation those will be installed afterwards. You’ll know you’ve succeeded as soon as you’re able to see a similar window in the screenshot below. One Solution called ‘PSFSample’ containing 4 Projects.

7. Right click on the PSFSamplePackage to show the options available. Select Build.

8. “Open Folder in File Explorer” will finally open a window showing you the resulting files.

Preparation is now complete for fixing the application in our next posts…

References:

MSIX – The MSIX Packaging Tool – Using the First Package (Part 1)

https://techcommunity.microsoft.com/t5/Core-Infrastructure-and-Security/MSIX-The-MSIX-Packaging-Tool-Using-the-first-package/ba-p/363553

MSIX Package Support Framework Part 2 – Preparation

https://techcommunity.microsoft.com/t5/Core-Infrastructure-and-Security/MSIX-Package-Support-Framework-Part-2-Preparation/ba-p/393864

Hello Everyone! Allen Sudbring here, Premier Field Engineer at Microsoft. Today I’m putting a post out to get some critical information to everyone who supports Windows Server and Active Directory Domain Services.

netdom.exe trust fabrikam.com /domain:contoso.com /EnableTGTDelegation:No

When an attacker manages to break into an on-premises domain environment, one of the first steps they normally take is to gather information and perform domain reconnaissance. Reconnaissance involves identifying the users, resources and computers in the domain and then building an understanding of how those resources are used to form your domain environment.  

While an attacker can gather data without credentials, research has revealed that most of the time, attackers make use of normal, non-privileged, domain user rights to make their moves. 

Figure 1 - Bloodhound generated graph used to find a Domain Admin (source: https://wald0.com/?p=68)

How do LDAP-based attacks succeed if security is in place?  

In most environments, every account in the domain has the permissions needed to perform reconnaissance using the LDAP protocol, and LDAP is deployed as a default part of domain controller services. With the default configuration in place, any domain user can retrieve domain configurations, such as where exchange servers are installed, or get account related details, such as Domain Admin group membership lists, as well as details about which account can delegate authentication, what users have a Kerberos principal name, and more. 

Aside from user accounts, most on-premises domain services use LDAP as a key element for their basic functionality, and group policies are sent to every domain computer over LDAP.   

Attackers are known to use LDAP queries to visually map the domain environment using publicly available tools, such as PowerView and BloodHound to implement queries. These tools help get all users, groups, computer accounts and account access control lists (ACL) in the environment. Once the data collected is parsed, it is stored in a graph database and used to build a visual graph that displays the edges between the different accounts, helping the attackers determine and plan their moves laterally in the domain. 

Adding standard user account risk to LDAP group policy exposure, you can quickly start to see where LDAP is a potential attack gold mine. By exploiting your LDAP exposure and risk points, attackers find sensitive groups memberships, vulnerable services and map domain account relationships by exploiting any user permissions they can breach or find in your domain. 

A single point of failure on a standard user account can be the start of a large-scale breach.

There are also other types of attacks that can be initiated with an LDAP query. Attackers can initiate an internal phishing campaign by enumerating users in Finance or IT groups, harvest private phone numbers that allow them to send phishing links by text message, and find local administrators on end-points computers by retrieving and parsing group polices.

With so many methods and possible attack surfaces, can your domain be protected from LDAP risks?

YES!

To protect your domain, your organization must be able to:

1. Define and differentiate between legitimate and malicious activity
2. Identify and investigate activity sources and intentions
3. Correlate related activities from the same sources
4. Discover and remediate compromised accounts

Unprotected LDAP risks leave your entire organization at risk.

Backed by deep data learning modules, Azure Advanced Threat Protection now provides comprehensive LDAP alerts that learn and surface abnormal activities, identify and aid investigation of attack sources, provides correlation of events and suggest remediation steps for compromised accounts.

Figure 2 - Azure Advanced Threat Protection Security principal reconnaissance (LDAP) alert

As our security research team continues to develop and refine our threat protection modules and alerts, we welcome your feedback about our work and the security threats and attacks you encounter. We’re excited to hear from you and learn how we can help. 

Get Started Today

If you are just starting your journey, begin trials of the Microsoft Threat Protection services today to experience the benefits of the most comprehensive, integrated, and secure threat protection solution for the modern workplace:

• Windows Defender ATP trial
• Office 365 E5 trial
• Enterprise Mobility Suite (EMS) E5 trial
• Azure Security Center trial

In a typical Kerberoasting attack, attackers exploit LDAP vulnerabilities to generate a list of all user accounts with a Kerberos Service Principal Name (SPN) available. Once successful at listing these accounts, attackers grant Kerberos Service Tickets for each user account with an SPN and later perform offline Brute Force on the encrypted part of the Kerberos tickets. This action helps attackers locate a password that belongs to a domain account. Domain account passwords enable attackers to freely move laterally in your domain.

Environments where the Kerberos Ticket Granting Service (TGS) is encrypted with a weak cipher, and the cipher is generated from a well-known password (not randomly generated) are prime targets for successful brute force attacks of this type.  

The following attack logic is often used to find an organization's weakest link and perform LDAP based Kerberoast attacks.

Figure 1-Typical Kerberoasting attack flow

Typical LDAP based Kerberoasting attack flow and result: 

Step 1: Identify

In this attack phase, attackers are using LDAP to query and locate all user accounts with a Service Principal Name (SPN). Running this LDAP query is possible for all user accounts in a domain.

Figure 2- LDAP query that looks for all user accounts with a SPN set

Step 2: Enumerate

In this phase of the attack, a request is made for Kerberos TGS to the SPN using a valid TGT.

Figure 3- TGS request to ExampleService of user1 by user2

Figure 4 - TGS response with ticket to ExampleService of user1

Step 3: Brute force

In the brute force phase of the attack, by using commonly available password cracking tools on accounts with commonly used passwords, attackers easily succeed at obtaining the password.

In the following example, a commonly used password cracking tool, JohnTheRipper, performs a successful brute force using a rainbow table.  

Figure 5 - Cracked password using a rainbow table

Step 4: Attack  

In cases where the attempted brute force attack (shown previously) is successful, attackers use the newly obtained clear-text password to login to remote machines or access cloud resources and files.

Figure 6 - Interactive clear-text logon

How can you detect and prevent Kerberoast attacks from succeeding? 

Azure Advanced Threat Protection (Azure ATP) has risen to the Kerberoasting challenge and developed new methods to detect when malicious actors are attempting to perform LDAP based reconnaissance on your domain. While this type of attack is difficult to detect, and LDAP’s extensive query language presented additional challenges, our security research work involved differentiating legitimate workflows from malicious behavior and surfacing all related activities and entities.

Our newest security alert involves smart behavioral detection backed by extensive machine learning, designed to raise an alert when any type of abnormal enumeration (including SPN enumeration), or queries on sensitive security groups are detected.  

Starting from v2.72, Azure ATP issues a Security principal reconnaissance (LDAP) alert when the first stage of a Kerberoasting attack attempt is detected on the domains we monitor.  

Each alert includes vital information for use in your investigation and remediation:

1. Identification of malicious activity

2. Attempted enumeration details and specifics

3. Historical comparisons and activity correlation

4. Suggestion remediation steps 

The following workflow explains how to use Azure ATP alerts to detect and remediate Kerberoasting attempts on your domain.

Step 1: Review the alert to identify the actors and entities involved.

Figure 7 - Azure ATP alert on suspicious enumerations 

Step 2: Filter activities to review resource access on the entity involved

Figure 8 - Filter for resource access activities on Client1's profile

Step 3: Use the filter results to investigate the resource access activities

Figure 9 - Investigate the resource access activity (generated by Kerberos Ticket Granting Service) for ExampleService/User1

Step 4: Filter Interactive logon and Credential validation for the accessed entity

Figure 10 - Filter Interactive logon and Credential validation on User1’s profile

Step 5: Review logon and access attempts

Figure 11 - User1's clear text password was used to logon on interactively on Client2

Step 6: Remediate possible risks

1. Force a password reset on the compromised account
2. Require use of long and complex passwords for users with service principal accounts https://docs.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/minimum-password-length
3. Replace the user account by Group Managed Service Account (gMSA) https://docs.microsoft.com/en-us/windows-server/security/group-managed-service-accounts/group-managed-service-accounts-overview

Kerberoasting remains a popular attack method and heavily discussed security issue, but the effects of a successful Kerberoasting attack are real. Make sure your security team is aware of common Kerberoasting risks and strategies, along with the tools and alerts Azure ATP offers to help protect your domain.

As always, we welcome your feedback about our work, and are interested in learning more about the security threats and risks you encounter. For more information about features and threat protection, or to learn how we can help, contact us. 

Get Started Today

If you are just starting your journey, begin trials of the Microsoft Threat Protection services today to experience the benefits of the most comprehensive, integrated, and secure threat protection solution for the modern workplace:

• Windows Defender ATP trial
• Office 365 E5 trial
• Enterprise Mobility Suite (EMS) E5 trial
• Azure Security Center trial