Links

Selenium WebDriver

Introduction:

modheader/modheader_selenium is our repo for using ModHeader in WebDriver. We created special builds for ModHeader that expose some URLs as APIs to manipulate ModHeader. These APIs are not available in the regular ModHeader. With these builds, you can setup WebDriver to configure ModHeader programmatically.

Installation:

To use this in NodeJS for Chrome, install the chrome-modheader package:
npm install chrome-modheader
To use this in NodeJS for Firefox, install the firefox-modheader package:
npm install firefox-modheader
For other programming languages, you can download the prepackaged extensions below and load them into WebDriver as needed.

Usage:

Take a look at the examples directory for more detailed examples
Selenium Webdriver with Chrome:
const { getExtension, getAddHeaderUrl } = require('chrome-modheader');
const options = new chrome.Options().addExtensions(getExtension());
const driver = await new Builder().forBrowser('chrome').setChromeOptions(options).build();
await driver.get(getAddHeaderUrl('HeaderName', 'HeaderValue'));
await driver.wait(until.titleIs('Done'), 5000);
Selenium Webdriver with Firefox:
const { getExtension, getAddHeaderUrl } = require('firefox-modheader');
const options = new firefox.Options();
options.addExtensions(getExtension());
const driver = await new Builder().forBrowser('firefox').setFirefoxOptions(options).build();
await driver.get(getAddHeaderUrl('HeaderName', 'HeaderValue'));
await driver.wait(until.titleIs('Done'), 5000);
Webdriver.io:
Modify wdio.conf.js file
const chromeModheader = require('chrome-modheader');
const firefoxModheader = require('firefox-modheader');
exports.config = {
capabilities: [
{
browserName: 'chrome',
'goog:chromeOptions': {
extensions: [chromeModheader.getEncodedExtension()]
}
},
{
browserName: 'firefox'
}
],
services: [
['selenium-standalone', { logPath: 'logs', installArgs: { drivers }, args: { drivers } }],
[
'firefox-profile',
{
extensions: [firefoxModheader.getExtension()]
}
]
],
before: function (capabilities, specs) {
browser.url(chromeModheader.getAddHeaderUrl('accept-encoding', ''));
}
};
Java Selenium
For Chrome
Path currentRelativePath = Paths.get("chrome-modheader/modheader.crx");
ChromeOptions options = new ChromeOptions();
options.addExtensions(new File(currentRelativePath.toAbsolutePath().toString()));
ChromeDriver driver = new ChromeDriver(options);
driver.get("https://webdriver.modheader.com/add?test=ModHeader%20Test");
new WebDriverWait(driver, Duration.ofSeconds(500)).until(ExpectedConditions.titleIs("Done"));
For Firefox
FirefoxDriver driver = new FirefoxDriver();
driver.installExtension(Paths.get("firefox-modheader/modheader.xpi"));
driver.get("https://webdriver.modheader.com/add?test=ModHeader%20Test");
new WebDriverWait(driver, Duration.ofSeconds(500)).until(ExpectedConditions.titleIs("Done"));
Python Selenium
For Chrome
options = webdriver.ChromeOptions()
options.add_extension('chrome-modheader/modheader.crx')
driver = webdriver.Chrome(options=options, service=Service(ChromeDriverManager().install()))
driver.get("https://webdriver.modheader.com/add?test=ModHeader%20Test")
WebDriverWait(driver, 5).until(expected_conditions.title_is("Done"))
For Firefox
driver = webdriver.Firefox(service=Service(GeckoDriverManager().install()))
driver.install_addon('firefox-modheader/modheader.xpi')
driver.get("https://webdriver.modheader.com/add?test=ModHeader%20Test")
WebDriverWait(driver, 5).until(expected_conditions.title_is("Done"))

API:

All APIs are URL-based. Please make sure to URL encode your name and value properly.
Note that the webdriver.modheader.com URLs only work when the extensions are properly loaded. Older versions of the extensions use webdriver.bewisse.com and bewisse.com. These will continue to work in the newer version.
If you are using npm, you can also use the getAddHeaderUrl(), getAddHeadersUrl() and getClearHeadersUrl() functions to craft these URLs. Be sure to do driver.get(), and be mindful that these will change the URL of the WebDriver.
Add request header:
https://webdriver.modheader.com/add?{name1}={value1}&{name2}={value2}&...
e.g., https://webdriver.modheader.com/add?Test=1
Node API equivalent:
function getAddHeaderUrl(name, value) {}
function getAddHeadersUrl({ name: value }) {}
Construct the URL above using getAddHeaderUrl('Test', '1') or getAddHeadersUrl({ Test: '1' })
Clear all modified request headers:
https://webdriver.modheader.com/clear
Node API equivalent:
function getClearHeadersUrl() {}
Load custom profile:
https://webdriver.modheader.com/load?profile={exported_profile_in_json}
exported_profile_in_json can be obtained from the regular ModHeader extension using ... -> Export Profile.
Node API equivalent:
function getLoadProfileUrl(exported_profile) {}

FAQ:

  • Does this work in headless mode?
    • Yes, but be sure to use --headless=new, not --headless.
    • NOTE: As of Chrome 109, you will need to use --headless=new. Older versions of Chrome should use --headless=chrome.
  • I am stuck on the webdriver.modheader.com page. What is wrong?
    • Most likely, this is because the ModHeader selenium extension is not installed correctly. Please make sure to download the latest ModHeader selenium extension at the top of this page (Do NOT use the ModHeader extension from Chrome Web Store).
    • To debug further, please pause the WebDriver execution, and go to chrome://extensions in the WebDriver Chrome instance. Make sure ModHeader selenium extension is installed. If not, that means that is something wrong with the WebDriver setup. If it is installed, enable "Developer mode" (top right corner), then click on ModHeader's "background page" link. This should open up the "Console". Send the errors to [email protected] This will help us identify the root cause.
    • Note: The webdriver.modheader.com subdomain maps to a static page hosted on S3. It is used as a hint for ModHeader selenium extension to redirect to the extension internal page. We do not log any requests to this page.
  • ModHeader sometimes fail to modify request headers in WebDriver, why?
    • When you navigate to webdriver.modheader.com, Please add wait until the page's title is "Done" before proceeding. This will make sure that the modification has been correctly stored in ModHeader.