Piercing the Shadow Root Using CDP

Part of my day job is orchestrating browsers using the Chrome DevTools Protocol, also known as CDP. CDP is a powerful protocol that exposes many ways to control and inspect a running Chrome instance. People usually interact with a wrapper around CDP like Puppeteer or Playwright but sometimes to get the full power of the protocol, you need to use it directly. Sadly it seems that the protocol was not properly designed and everything was bolted on as the protocol expanded to expose new functionality.

Since the protocol is a bit hard to understand and even remember, I usually document for myself how to do some non-obvious things in it. This post is hopefully one of many short snippets on how to wield the protocol in the best way possible. We will start with a post on how to pierce1 closed shadow roots.

The Best Way?

CDP is split into multiple domains like DOM, Network, Page etc. To interact with the DOM we will first try to use the DOM domain of course. We will start by sending the DOM.enable method and look at what ways we can query the page. The most straightforward way to query the page is with the DOM.querySelector and DOM.querySelectorAll methods. Let’s try to query the following page:

<!DOCTYPE html>
<html>
	<head>
		<meta charset="UTF-8" />
		<title>Closed Shadow Root</title>
	</head>
	<body>
		<p id="outside">OUTSIDE CLOSED SHADOW ROOT</p>
		<template shadowrootmode="closed">
			<p id="inside">INSIDE CLOSED SHADOW ROOT</p>
		</template>
	</body>
</html>

Both of the query selector methods already require us to have a node ID to query on. We can get the root node ID by calling the method DOM.getDocument. After we retrieve the root node ID, we can try to query for the inside element. Using the chrome-remote-interface library, it would look like this:

import CDP from 'chrome-remote-interface';

const client = await CDP();
const getDocumentResponse = await client.DOM.getDocument();
const querySelectorResponse = await client.DOM.querySelector({ nodeId: getDocumentResponse.root.nodeId, selector: '#inside' });
console.log(querySelectorResponse.nodeId);

But if you will run it with the example page above, the node ID you will get will be zero. That’s because the method actually failed to find the desired element. We need to somehow pierce the closed shadow root. If we look at the DOM.getDocument method again, we will see that the method receives a pierce argument that the DOM.querySelector method does not. According to the documentation, the argument will let us traverse iframes and shadow roots in the tree2. Let’s use it alongside the depth argument to get the whole DOM tree together with the closed shadow root tree.

import CDP from 'chrome-remote-interface';

const client = await CDP();
const getDocumentResponse = await client.DOM.getDocument({
	depth: -1,
	pierce: true
});
console.log(JSON.stringify(getDocumentResponse, null, 2));

Notice that we’ve got the children property for our regular elements and the shadowRoots property for our shadow root elements. Great, we’ve managed to pierce the shadow root and get the desired element. But this approach is not really scalable with a real web page, we will get too much data just for trying to retrieve a single element. There is another method, similar to DOM.getDocument, that can give us the tree under a specific node using its ID. In our example web page, the shadow root is just below the body element so we will query and describe it like so:

import CDP from 'chrome-remote-interface';

const client = await CDP();
const getDocumentResponse = await client.DOM.getDocument();
const querySelectorResponse = await client.DOM.querySelector({ nodeId: getDocumentResponse.root.nodeId, selector: 'body' });
const describeNodeResponse = await client.DOM.describeNode({ nodeId: querySelectorResponse.nodeId, depth: 1, pierce: true });
console.log(JSON.stringify(describeNodeResponse.node.shadowRoots, null, 2));

Now that we’ve got the top element of the shadow root, we can query it using DOM.querySelector and find our desired element. Our complete code would be like so:

import CDP from 'chrome-remote-interface';

const client = await CDP();
const getDocumentResponse = await client.DOM.getDocument();
const querySelectorResponse = await client.DOM.querySelector({ nodeId: getDocumentResponse.root.nodeId, selector: 'body' });
const describeNodeResponse = await client.DOM.describeNode({ nodeId: querySelectorResponse.nodeId, depth: 0, pierce: true });
const shadowRoots = describeNodeResponse.node.shadowRoots ?? [];
const shadowRootQuerySelectorResponse = await client.DOM.querySelector({ nodeId: shadowRoots[0].nodeId, selector: '#inside' });
console.log(shadowRootQuerySelectorResponse); // Should log { nodeId: 13 }

If you executed the script against our example page, you should have gotten the desired node ID. You can now use the DOM domain to interact with the node using that ID.

Alternative paths

As I’ve said at the start, the protocol usually has more than one way of doing things. While looking at the best way of doing it, I’ve stumbled upon several other ways to pierce the shadow root and/or iframes.

Execution Contexts

Each iframe has its own execution context so we can use it to run code inside the iframe. This approach is kind of ugly for two reasons:

  1. The execution context of an iframe is obtained by listening to the Runtime.executionContextCreated and the Runtime.executionContextDestroyed events. You can’t simply fetch the desired execution context for the desired frame3.
  2. Usually you can convert objects from a script (that are accessed using object IDs in CDP) to a node ID if the object is pointing to a node. I haven’t found a way to do it in the case of an iframe but I might have missed it.

DOM.requestChildNodes

Another way of descending into a closed shadow root is by calling the method DOM.requestChildNodes. This method sounds exactly like what we need and it seems that the inspector in Chrome is actually using this method. I’m not sure why this method exists alongside the DOM.describeNode method which almost does the same thing but DOM.requestChildNodes is a lot harder to use. Instead of returning the children in the response of the method, you must also listen to a separate event called DOM.setChildNodes with the actual nodes. Not a big deal but I’m not sure why they implemented it that way.

Exploring The Protocol

I’ve mentioned in the previous section that the inspector in Chrome is using the DOM.requestChildNodes method. There are two ways of learning about the protocol usage in the devtools. One is to read the source code, which can take a while to orientate yourself. The second option, which I highly recommend, is to open the protocol monitor in the devtools and see for yourself how each interaction in the devtools send and receive data through the protocol.

  1. This is the verb used by the protocol as well. ↩︎

  2. It’s important to note that the pierce argument does not modify the shadow root in the DOM and won’t open it. ↩︎

  3. You can create another execution context for the iframe using Page.createIsolatedWorld. ↩︎