Written by Nico Shyne & Josh Prager

Just as in the political landscape of Westeros, defenders face a dynamic adversarial relationship…except instead of fighting rival families, defenders are locked into a struggle with innovators on the offensive side of cybersecurity. This innovation manifests in the form of evolving tradecraft and tooling, and defenders must stand ready to reclaim and fortify their domain’s sovereignty. An organization’s users must have trust in both the domain and the fidelity of its architecture. This concept is important in maintaining your organization’s operational efficiency and making sure that client and proprietary data are secure.

Adversaries constantly seek ways to access and maintain presence in your domain. Assuming they’ve achieved persistence and there’s evidence of their activity, defenders typically concentrate on the methods used for access, and overlook the post-breach domain environment.

Typically, that post-breach recovery relies on surface level fixes: “rotating the KRBTGT password twice”, “increasing the available RID pool”, etc. What we are interested in exploring is what defenders can do beyond those steps. How do defenders truly reclaim control over their domain? How do they recover operational efficiency without compromising trust or security through the recovery process? What portions of the organization’s recovery process meet a minimum expectation to cut back the adversary’s stolen access?

This post will explore techniques adversaries use to gain and sustain access within a domain, including credential dumping on domain controllers, Active Directory configuration syncing, Kerberos protocol manipulation, and certificate abuse. Additionally, we’ll discuss effective recovery strategies, aiming for a more satisfying resolution than the Game of Thrones finale.

This blog covers multiple Kerberos abuse based attack techniques, detection guidance and remediation of a compromised domain. Due to the extensive length of these topics, we think it is important to provide the attack techniques in a concise and predictable format. Each attack technique discussed will have the following sections:

Overview:

• We will cover a brief description of the target technique.
• We will cover the technology that is required of the target technique.
• We will cover the motivation and use of the target technique.

Execution:

• We will cover how the attack is performed.
• We will mention any related blogs, tools, or variations of the attack performance.

Detection:

• We will cover the Sysmon and Windows Security event IDs related to discovering the target technique.

TLDR; If you’re familiar with the techniques listed below and want to review the remediation guidance, jump to the “Reclaiming the Iron Throne” section.

(But who would want to start on the last season of a show?)

Nation state affiliated threat actors such as FIN6, NICKEL, and Emissary Panda targeted critical Active Directory assets, notably the (Windows NT Directory Services) NTDS.dit file, the KRBTGT service account, and Active Directory certificates. They often gained initial access through phishing or exploiting vulnerabilities, then used a combination of native and custom tools to persist within the domain with elevated credentials. Emissary Panda, for instance, exploited a widespread SharePoint vulnerability, indicating a broader, more widespread approach rather than a focused target selection. Once defenders confirm a breach and an IR or triage team has eliminated the immediate threat, the focus should shift to blocking adversary access and rotating domain secrets. In order to avoid a compromise, detection engineers should prioritize identifying signs of domain persistence in the environment.

In the following section, this post will explore techniques adversaries use to maintain elevated privileges and the detection methods that help identify the “burn down the domain” scenario.

Overview

Adversaries with elevated access to a domain controller (such as domain administrator, enterprise admin, or local administrator) can execute credential theft by utilizing applications to obtain a handle to the LSASS.exe process.

The Local Security Authority Server Service (LSASS) handles the authentication of users within a domain. LSASS is critical in servicing the Kerberos Distribution Center (KDC) and the Keberos authentication protocol by generating tokens for requested resources. Whenever a user interactively authenticates to a domain controller, the accounts credential material is cached into the memory of the LSASS.exe process. Active Directory clients can use preventive mechanisms like Credential Guard to isolate the copy of the LSASS.exe process that contains cached credentials. When adversaries attempt to obtain a handle to LSASS.exe with read access rights, the version of LSASS.exe available to them will not contain those cached credentials and instead the credentials are located in the isolated version of LSASS.exe.. However, domain controllers don’t enable Credential Guard because of application compatibility issues.

Adversaries with sufficient access rights can obtain a handle to the LSASS.exe process on domain controllers and read the virtual memory of the LSASS.exe process, which contains cached credentials. Using these stolen credentials, adversaries can change their execution context to access high-valued resources and perform actions that could significantly impact an organization’s business continuity.

Execution

This technique can be conducted via many publicly available tools and native “living-off-the-land” Windows binaries (e.g., Task Manager) that can obtain a handle to LSASS.exe with the access rights needed to read the virtual memory of this process.

Using the Sysinternals suite a user can obtain a handle to the LSASS.exe process with read access rights to the virtual memory of the process. PsExec can remotely call a local copy of ProcDump to create a process dump of the LSASS.exe process.

./PsExec.exe -s accepteula \\dc1.asgard.corp -u asgard\da -p <PASSWORD> C:\Sysinternals\procdump.exe -accepteula lsass.exe C:\temp\totally_not_lsass.dmp

Detection

When an application requests an open handle to the LSASS.exe process using access rights that enable the application to read the virtual memory of the process, Sysmon will generate event ID 10 — “ProcessAccess.” Detection engineers will need to pair this process access attempt with additional process context specific to the target process of LSASS.exe or the application that obtained the handle to the LSASS.exe process. Natively, many applications will open a handle to the LSASS.exe process; which forces detection engineers to identify additonal telemetry to correlate the behavior.

Additionally, in his blog, Olaf Hartong suggests monitoring the process access attempt in conjunction with Windows Security event ID 4703 — “A token right was adjusted.” This event indicates when a user elevates the process integrity to high integrity, enabling SeDebugPrivilege, and obtains a new access token. The monitoring of process integrity elevation uses the assumption that the adversary will request all access rights to obtain a handle to the LSASS.exe process and read its virtual memory. While variances in technique execution exist to avoid requesting all access rights with SeDebugPrivilege, a dedicated detection engineering can build upon these and other detection recommendations.

Overview

Criminal actors target the NTDS.dit file of organizations by accessing or copying the database file in an attempt to harvest credentials from the organization.

The NTDS.dit file is the main database for Microsoft’s Active Directory data. The NTDS.dit file is used by Active Directory to manage and organize users, groups, security descriptors and password hashes. This file is stored on domain controllers typically under the location of %SystemRoot%\NTDS. Active Directory constantly uses the NTDS.dit file for Active Directory processes such as the KDC, and the domain controller deadlocks the file to avoid system errors (Figure 1). If a user attempts to manipulate the NTDS.dit file, this action will cause a SHARING_VIOLATION flag due to the file lock.

Adversaries must use techniques that avoid the deadlock and generate a copy of the file to exfiltrate the Active Directory data. Once an adversary has exfiltrated the NTDS.dit and a copy of the SYSTEM registry hive, adversaries can extract secrets and and the entire directory with tools such as ntdsdotsqlite or DSInternals.

Execution

Several native Windows binaries exist for generating backups of the Active Directory database and copying the deadlocked NTDS.dit file. Most of these tools make use of the (Volume Shadow Service) VSS.

To generate a backup of the Active Directory database, the following tools can be utilized in conjunction with the below commands:

Ntdsutil.exe “ac i ntds” “ifm” “create full C:\programdata\cache”
Vssadmin.exe create shadow /for=C
Copy \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\windows\ntds\ntds.dit C:\ntds.dit
Copy \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy1\windows\system32\SYSTEM c:\SYSTEM
esentutl.exe /y /vss c:\windows\ntds\ntds.dit /d c:\folder\ntds.dit

Detection

Most open-source detection strategies that exist for detecting the access of the NTDS.dit file do so with process execution telemetry that focuses on the the word “ntds” or the name of the suspected process (e.g., “vssadmin.exe”) within the command line arguments. While these brittle detections have their place within the detection and response program, detection strategies surrounding the access of the NTDS.dit should be the emphasis in broader detections.

Regardless, if the adversary utilizes ntdsutil.exe, vssadmin.exe, or esentutl.exe to create a backup or volume shadow copy; several Volume Snapshot Service based Windows Security event IDs are generated specific to the creation of the copy. As all of the above methods of execution use Volume Shadow technology; defenders can use events specific to the use of the VSS as a common denominator for detections.

Once the backup applications begin, a sequential process creation for VSSVC.exe (the Volume Snapshot Service) is executed by services.exe.

Additionally, a Windows Security event ID 8222 — “Shadow copy has been created” will be generated upon the use of any of these tools when a backup of the NTDS is created (Figure 2).

Finally, defenders can leverage anomaly detections for the access of the Security Account Manager (SAM). To generate a shadow copy of a C:\ drive, a READ_CONTROL handle is requested for the access of the SAM with Windows Security event ID 4661 in mass over a short period of time (Figure 3). The mass requests of object handles for the SAM is often a very easy anomaly detection to implement and can be correlated to multiple other techniques as well.

Overview

A DCSync attack exploits the trust relationship within the underlying Windows environment in an attempt to gain unauthorized access to user credentials by syncing a copy of Active Directory data. Directory Replication Service (DRS) is the method by which the Windows Active Directory Domain Service ensures that a change made on one domain controller replicates to all domain controllers. Active Directory replication is facilitated by connection objects that represent replication pathways between source and destination domain controllers. Each of these objects have attributes related to access and permissions. When a change to these objects occurs on one domain controller, the other domain controllers will sync these value changes throughout the Active Directory schema. Additionally, this replication ensures consistent object attributes in the event that a domain controller is offline or otherwise experiencing failure. The extended rights required to conduct a Active Directory replication are:

The MS-DSRS RPC protocol uses the RPC method GetNCChanges request and replies in conjunction with the RPC interface (Directory Replication Service Update API) DRSUAPI network protocol to sync active directory changes. Typically, this occurs from primary to backup domain controllers. Adversaries make use of this RPC method to replicate sensitive data (e.g., password hashes) from Active Directory.

Execution

A wide variety of public open-source tooling (e.g., Rubeus, Mimikatz, etc.) can use the the RPC method GetNCChanges to sync Active Directory data . Once the adversary has gained access to a compromised account with the sufficient privileges required, they can leverage the compromised account permissions to execute a DCSync and target specific accounts or all of the accounts.

To replicate accounts from Active Directory, adversaries can use the GetNCChanges RPC method to request sensitive data from the domain controller.

lsadump::dcsync /dc:$DomainController /domain:$DOMAIN /all /csv

Detection

When domain controllers replicate, they generate a corresponding event ID 4662, provided the domain controller pulling the sync has Directory Replication Service auditing enabled (Figure 4). Machine accounts usually handle Active Directory replication, so a user or service account appearing in the 4662 event would be noticeable. However, some custom replication services in client environments use a service account to replicate credentials. Still, administrators can often manage the allowlist for DCSync replication, making this alert easy to baseline.

In some situations, the adversary may already have gained local administrative access to the domain controller. Whenever DCSync is occurring locally on the domain controller, the network data will not traverse typical outbound/inbound interfaces, but instead use the local loopback (Figure 5). This method of DCSync will avoid most network-based detections because there wouldn’t be any system-to-system communication. The intercommunication occurs between two processes on the same system.

Overview

The Red Wedding in The Rains of Castamere, S3:E9, is memorable for its unexpected brutality. (#robbshouldhavelived). Adversaries who have established a foothold in an environment use a similar surprise element by imitating legitimate ticket requests. Adversaries can utilize the golden ticket technique to help them evade detection and maintain persistence.

The Kerberos authentication protocol makes use of ticket requests and grants to authenticate users to remote resources. The KRBTGT is a dedicated service account whose password is used to generate a key. This key is the one-way hashed transformation of the KRBTGT account’s password. The KDC uses this key to sign and encrypt (ticket-granting tickets) TGTs which are presented to the remote resources. Kerberos inherently trusts any TGT encrypted with the KRBTGT password hash. If an adversary compromises the KRBTGT’s password hash, the adversary can then generate their own TGTs and bypass the KDC completely. Adversaries can then use this illegitimate TGT, signed and encrypted with the KRBTGT’s password hash to authenticate to remote resources where Kerberos authentication is required. For more in depth information about the Kerberos authentication protocol, please visit the many available resources that cover this protocol in depth.

Execution

Adversaries can craft a new TGT with the following ticket parameters:

• (Fully Qualified Domain Name) FQDN of the domain
• (Security Identifier) SID of the domain
• Account to impersonate
• KRBTGT password hash

Once an adversary passes this information to a crafted ticket, the adversary can then save the crafted ticket in the current session’s ticket cache.

mimikatz.exe “kerberos::golden /domain:dc.asgard.corp /sid:S-1–5–21–4198639423–1025486511–2226459690 /rc4:a8bbd83cc1ded03f7db3b07d78e95036 /user:da /id:500 /ptt”

Detection

Golden ticket detection methods can focus on the theft of the KRBTGT password hash or the anomalous use of the KRBTGT account itself. From the perspective of the ticket request, the Windows Security events generated look the same whether its a legitimate TGT request or a request related to the golden ticket attack (4768 and 4769). This forces detection engineers to focus their detection efforts on the theft of the KRBTGT password hash, rather than on the use of that credential material to carry out a golden ticket attack (credential theft on the domain controller, NTDS.dit access, and DCSync).

Defenders can track group membership changes of users from source hosts identified in earlier detection strategies. Windows Security event ID 4627 -” Group Membership Information” displays the logged-on account’s group memberships. We showcased the difference in group memberships between a legitimate logon (timestamp 00:37) and a Golden Ticket attack (timestamp 00:35) for user bouj33boy (Figure 6).

In the screenshot, the SID S-1–5–21…-512 corresponds to the Domain Admins group. To reduce noise in 4627 events, use a Security group for Domain Admins instead of assigning them individually. This way, 4627 events from individual users become inherently suspicious and merit investigation.

An additional detection method relies upon tracking (ticket-granting-service ticket request) TGS-REQs (Windows Security event ID: 4769) that have no corresponding (Authentication Service Request) AS-REQ (Windows Security event ID: 4768). Additionally, some resources explain that forged tickets may not properly display FQDNs when the TGT is forged.

To validate a suspected Golden Ticket event, use the klist command to list the Kerberos tickets for that user. When an adversary imports the stolen KRBTGT ticket, it triggers a sequential event ID 4624 for a successful logon. This event provides the user’s logon ID, and klist can then show the user’s ticket cache (Figure 7).

Overview

Adversaries can avoid crafting their own TGTs and instead opt to modify a legitimately issued TGT from the domain controller. To obtain a legitimate TGT, adversaries can use the (Generic Security Service) GSS-API and obtain a TGT for a user without requiring the user’s password or NTLM\AES hash. The GSS-API provides a standardized way for applications to use the Kerberos security provider. The adversary can use the AcquireCredentialsHandle and InitializeSecurityContext function calls with the ISC_REQ_DELEGATE flag to coerce the domain controller to provide a (Application Request) AP-REQ in the GSS-API output containing the KRB_CRED in the authenticator checksum.

As stated in the Golden Ticket section, the KDC uses the password hash of the KRBTGT account to sign and encrypt legitimate TGTs. This same password hash can be used to decrypt the legitimate TGTs provided by the domain controller. Upon successful decryption, adversaries can then modify the legitimate TGT and re-encrypt the ticket. The specific portion of the legitimate TGT that is modified is the Kerberos (Privilege Attribute Certificate) PAC. Within the PAC there are various attributes stored like user identification, group memberships, and specific privileges or permissions granted to the user. Adversaries that modify the PAC with the desired elevated permissions and re-encrypt the TGT can then carry out domain persistence without snaring their activity by golden ticket detections (TGS-REQs that have no corresponding AS-REQ).

Execution

To modify a legitimate TGT, adversaries can obtain a usable TGT via the GSS-API, decrypt the TGT via the KRBTGT password hash, modify the PAC, and re-sign/re-encrypt the TGT.

Rubeus.exe diamond /domain:asgard.corp /user:bouj33boy /password:<PASSWORD> /dc:dc.asgard.corp /enctype:AES256 /krbkey:HASH /ticketuser:da /ticketuserid:USER_ID /groups:GROUP_IDS

Detection

Similar to the Golden Ticket detection strategy, defenders should monitor KRBTGT password hash theft using strategies like lsadump, NTDS.dit theft, and DCSync. They should also watch for unexpected group membership changes, especially when low-privilege users suddenly have significant SIDs like 512 for Domain Admins without a corresponding Windows Security event ID 4627. Unlike a Golden Ticket, a Diamond Ticket triggers a 4768 event when requested. The initial Diamond Ticketing phase produces a Windows Security event ID: 4648, as the ticket’s PAC must be modified. Additionally, network based detections may be able to track and identify anomalies within the AS-REQs where the PA-PAC-REQUEST is set to false and the service has its NA bit set to false.

Given that these methodologies might not capture all pertinent events of interest, it remains advisable to employ additional validation techniques for suspicious ticket operations, particularly for specific users. This can be achieved using tools such as klist or ACE: Get-KerberosTicketCache. Although the scalability of tools like klist for enterprise-wide threat hunting remains limited, they prove invaluable for targeted remediation efforts within the infrastructure.

Overview

Adversaries can extract (Certificate Authority) CA private keys and create forged certificates signed by the stolen key. The Microsoft (Active Directory Certificate Services) AD CS server role allows administrators to produce digital certificates for authentication and encryption. The root CA, at the top of the (Public Key Infrastructure) PKI hierarchy is trusted by all intermediate CAs. Its certificate identifies the root CA and the public key that verifies signatures of all other certificates in the PKI.

CAs, often underestimated in importance, can be remotely managed by lower-level privileged roles such as the server operators group. If adversaries compromise these roles or gain local administrative access to the CA server, they can export the CA private key to create forged certificates. Many organizations set long validity periods for these keys, allowing adversaries to misuse them for extended periods.

Windows stores certificate private keys via the (Data Protection Application Programming Interface) DPAPI. A plaintext DPAPI masterkey is needed to decrypt and successfully exfiltrate the private key. Adversaries can retrieve the plaintext DPAPI masterkey by obtaining the domain’s DPAPI backup which can be used to decrypt any domain user’s master key file.

Execution

To obtain a CA’s private key, adversaries can use a decrypted DPAPI masterkey to decrypt the private key and associated certificates.

SharpDPAPI.exe certificates /machine

Detection

Will Schroder and Lee Christensen provide a couple different detection methods within their AD CS focused white-paper, Certified-Preowned. This white paper details cryptographic event IDs, including Windows Security event ID 5058 for operations on files with a Microsoft Software Key Storage Provider (KSP). It also mentions Windows Security event ID 5061 for open key or key creation operations against KSP files. The export of cryptographic keys is detected through Windows Security event ID 5059, while backup operations for these keys are tracked with Windows Security event IDs 4876 and 4877, indicating backup start and completion.

To monitor the access of the DPAPI-encrypted private keys, we can set a (System Access Control List) SACL on the private keys that normally only NT AUTHORITY\SYSTEM accesses. Access to these private keys once the SACL is set will generate Windows Security event ID 4663 (Figures 10, 11).

There are some blind spots to this detection method, as stated by Will Schroder and Lee Christensen, such as the Mimikatz (CryptoAPI) CAPI/(Cryptography Next Generation) CNG patching methods of reading files.

If the alert team identifies the mentioned techniques in use, the next logical step is to look for signs of data exfiltration. However, both incident responders should operate under the assumption that domain credential material is already compromised regardless of detected exfiltration. This stance is rooted in the principle of “unknown unknowns,” which means defenders are unaware of what they might be missing. If the adversary triggers the organization’s alerts for the aforementioned techniques, it indicates their ability to achieve total domain control. Defenders can’t predict all adversary exfiltration methods. Adversaries often send fragmented, encrypted data in small segments, making detection difficult as it blends with regular traffic.

Post-incident activity requires efficient recovery operations to reduce adversary dwell time after detecting a compromise. Once the incident’s scope is determined, hosts are isolated, reprovisioned, user accounts rotated, and system owners are alerted to close attack paths. If any of the mentioned techniques are identified, full domain recovery is essential. Incident responders should prioritize systems that elevated users could access. Ideally, organizations should rebuild each domain controller related to the adversary’s access. Additionally, if the adversary had domain administrative access and the domain is within a forest where parent/child or tree/root trust relationships are established, defenders must determine the trust type direction and consider these additional domains within the scope of recovery. If telemetry does not indicate forest trust lateral movement, organizations can choose to accept the risk that the adversary may have had access but did not attempt to use it. However, if telemetry is not available (and in many cases, it is not), the additional domains should stay in the scope of recovery as a precaution.

At this point, some heavy critical-decision making must occur at the organizational level to recover these potentially compromised domains or modify the scope and accept some level of risk.

Assuming the adversary has been contained and access has been remediated, the organization would begin the recovery operation. The below sections discuss high-level recommendations that an organization can take to recover from an incident at this level of compromise. Keep in mind there are nuanced caveats to any recovery plan that requires the unique perspective of the organization implementing the recovery.

In this section, this post will explore two options available to organizations that are ready to begin recovery operations. The first of these options is to utilize the domain controllers within the scope of the compromised environment and replace them with completely new domain controllers. The reason for replacing domain controllers completely stems from the inherent risk of an adversary having stolen the domain DPAPI backup key. The domain DPAPI backup key is found within the NTDS.dit database file and contains the master key to decrypt the private/public keypair associated with DPAPI. Domain controllers have a domain-wide public/private keypair and clients request the public key when joined to a domain as a member. This DPAPI public key is transmitted via authenticated and private RPC call to the client. This domain-wide backup key cannot be rotated as per Microsoft documentation (Figure 12).

The obvious recommendation is to go beyond simply accepting the risk and rotating everything except the domain-wide DPAPI backup key. Instead, organizations should identify the domain controllers where the credential dumping occurred with privilege login and destroy them in order to rebuild these domain controllers from scratch. Rebuilding can occur by re-installing a new Windows Server and promoting the server to domain controller via Windows Server Manager or by promoting the disk of another unaffected Windows Server to the domain controller role (Figure 13).

Domain persistence techniques often focus on abusing default authentication protocols and their specific components within the Active Directory domain. These techniques (and future gemstone-named variations) will continue to evolve in approach and strategy. Defenders need to keep an eye on both emerging trends on the offensive side and on new tools and techniques that help find evidence of adversary behavior within their domain.

We’ve also walked you through some of the most popular domain persistence techniques and potential detection strategies that defenders can implement in their environments.

‘But what happens if we find evidence of adversarial behavior?’, you say. Good question, hypothetical reader. Unlike waiting for the works of a certain popular author, you won’t have to sit for a few decades to see how the story ends. We are going to continue to provide remediation recommendations in the upcoming second half of this blog: Domain of Thrones Part II.

• https://www.jpcert.or.jp/english/pub/sr/20170612ac-ir_research_en.pdf
• https://learn.microsoft.com/en-us/windows-server/storage/file-server/volume-shadow-copy-service
• https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/understand-security-identifiers
• https://learn.microsoft.com/en-us/azure/active-directory/fundamentals/how-to-manage-groups
• https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/replication/active-directory-replication-concepts
• https://learn.microsoft.com/en-us/azure/active-directory-domain-services/concepts-forest-trust
• https://learn.microsoft.com/en-us/windows/win32/seccng/cng-dpapi-backup-keys-on-ad-domain-controllers
• https://learn.microsoft.com/en-us/windows/security/threat-protection/security-policy-settings/domain-member-maximum-machine-account-password-age
• https://specterops.io/wp-content/uploads/sites/3/2022/06/Certified_Pre-Owned.pdf
• https://atomicredteam.io/
• https://media.defense.gov/2023/May/24/2003229517/-1/-1/0/CSA_Living_off_the_Land.PDF