Is scraping shadow DOM an option?

[jtlimo]

Hi, I'm trying to web scrapping YouTube charts, unsuccessfully because they use polymer / shadow DOM. With Geziyor, could I do that? I'm using colly, and they don't have support for that.

[musabgultekin]

Hi! Thanks for reporting that!

Its not possible rn. Since actions are hardcoded. Im working on adding a way to add custom actions.

[musabgultekin]

Ive added custom actions in request.
So you'll add custom chromedp.Action to the geziyor.Request object. like req.Actions = []chromedp.Actions{XXX}

See TestGetRenderedCustomActions test

738852f

And you'll access to shadow doms like this action:

chromedp.Click(document.querySelector('#container').shadowRoot.querySelector('#foo'), chromedp.ByJSPath),

See this issue: chromedp/chromedp#376

Let me know if this fixes your problem!

[jtlimo]

Ive added custom actions in request. So you'll add custom chromedp.Action to the geziyor.Request object. like req.Actions = []chromedp.Actions{XXX}

See TestGetRenderedCustomActions test

738852f

And you'll access to shadow doms like this action:

chromedp.Click(document.querySelector('#container').shadowRoot.querySelector('#foo'), chromedp.ByJSPath),

See this issue: chromedp/chromedp#376

Let me know if this fixes your problem!

I'll try this between today and tomorrow… after testing, I say if it is work for my case.

[jtlimo]

Hey @musabgultekin I'm trying to scrape yt charts, but unfortunately unsuccessfully. I push my code to this repo if you have time to take a look, I appreciate.

The new error is when try to start scraping, the context was cancelled.

[musabgultekin]

Yeah I can look into it tomorrow, busy for today. Will comment here when i can find time.

[jtlimo]

Yeah I can look into it tomorrow, busy for today. Will comment here when i can find time.

Hi @musabgultekin some news in this issue? I can help you providing some more information or anything else?