seleniumbase (undetected Chrome driver): how to set request header? – Seleniumbase

The Problem:

While using seleniumbase with undetected Chrome driver (uc=True), how can I set the request header for specific web requests?

The Solutions:

Solution 1: Use seleniumwire for undetected Chrome driver and set header_overrides

Here’s a revised version of the solution:

You can utilize SeleniumWire to accomplish this. SeleniumWire is a wrapper around Selenium that extends it with additional features, including the ability to modify request headers. To set the request header using SeleniumWire with an undetected Chrome driver, follow these steps:

1. Import the necessary modules:

from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from seleniumwire.undetected_chromedriver.v2 import Chrome, ChromeOptions

2. Create an instance of ChromeOptions:

chrome_options = ChromeOptions()

3. Start the Chrome driver with SeleniumWire:

driver = Chrome(seleniumwire_options={"options": chrome_options})

4. Set the request header using the header_overrides attribute:

driver.header_overrides = {
    "Accept": "application/json",
}

That’s it! Now, when you make requests using the driver, the specified header will be automatically included in the request.

Solution 2: Via Javascript

A straightforward and effective solution for setting the request header in seleniumbase with Driver(uc=True) is to utilize Javascript. Here’s how you can achieve it:

1. Import the necessary modules:

from seleniumbase import Driver
import io

2. Initialize the seleniumbase driver with the uc=True option to enable undetected Chrome:

driver = Driver(uc=True)

3. Perform any necessary login procedures to access the required website.

4. Construct the JavaScript code to set the request header and make the HTTP request:

var callback = arguments[arguments.length - 1];
fetch('https://example.com', {method: 'GET', headers: {'Accept' : 'application/json'}}).then((response) => response.text().then((text) => callback({'status': response.status, 'text': text})))

5. Execute the JavaScript code using the driver’s execute_async_script method:

response = driver.execute_async_script(javascript_code)

6. Extract the response status and text from the returned JSON object:

status_code = response['status']
response_text = response['text']

7. Handle the response based on the status code. For example, if the status code is 200 (indicating success), you can save the JSON content to a file:

if status_code == 200:
    with io.open(outfile, 'w', encoding='utf8', newline='\n') as f:
        f.write(response_text)

8. Return the response status code.

This approach allows you to set custom request headers and make HTTP requests with the undetected Chrome driver provided by seleniumbase.

Video Explanation: