Pass-through mozilla tracking protection requests

[JanikkinaJ]

Describe the bug

When scanning our site with zap automation framework inside docker using the Logmessages.js script I discovered that a third of all requests made were made to https://tracking-protection.cdn.mozilla.net. The Responses were sometimes up to several mb. I believe this should be disabled by default for privacy and efficiency reasons. Mozilla will probably appreciate it too.

Steps to reproduce the behavior

Create A dockerfile which inherits softwaresecurityproject/zap-stable
Create a yaml scrip which scans a site and logs the requests made using LogMessages.js
Add a entrypoint which starts zap like so: zap.sh -cmd -autorund -script.yaml
Look through the logfile when the scan completes.

Expected behavior

Zap having tracking-protection preference set to be off by default. turning off tracking-protection for firefox using selenium - Stackoverflow link I believe that this change would need to be implemented here: ExtensionSelenium.java But may need more information to implement this myself.

Software versions

softwaresecurityproject/zap-stable latest 9899971d6689

Screenshots

No response

Errors from the zap.log file

No response

Additional context

No response

Would you like to help fix this issue?

• Yes

[thc202]

It's not ZAP but Firefox that's sending those requests.

[psiinon]

Having worked at Mozilla I'm pretty sure they wont mind either way 😉
But I'm all for reducing the noise from browser calls home..

[JanikkinaJ]

It's not ZAP but Firefox that's sending those requests.

I'm aware that firefox is initiating those requests, but the way that zap initiates firefox (as in not setting this setting to off) is causing firefox to initiate those requests.

[thc202]

What is the plan doing?

[JanikkinaJ]

It is basically a full authenticated scan, as in the type that is created when clicking "new plan..." in the automation framework tab in the gui and then selecting the Full scan profile

[JanikkinaJ]

It has some additions such as excluded paths and alertfilters that have been added put apart from that it is the same template

[Lonzak]

Having worked at Mozilla I'm pretty sure they wont mind either way 😉 But I'm all for reducing the noise from browser calls home..

I totally agree. We should reduce this "noise" since a ZAP scan doesn't need tracking-protection...
I think we should disable tracking protection when using the firefox...

It seems the option is called something like: SetPreference("pref.privacy.disable_button.tracking_protection_exceptions", $true)

In the ExtensionSelenium.java, maybe like this?

// Ensure tracking protection is disabled
firefoxOptions.addPreference("pref.privacy.disable_button.tracking_protection_exceptions", true);

The only thing where I am not sure is the option name. It contains button so maybe it is a different one like: pref.privacy.trackingprotection.enabled

[thc202]

ZAP is also used by humans.

[Lonzak]

ZAP is also used by humans.

Sure but either way - I don't see any benefit to report such requests to mozilla. It is a pentest of a site - automated or by a human. Firefox sometimes sends several megabyte of data (per request!) to this URL when using ZAP. And I doubt that this data is in anyway helpful on the mozilla side (e.g. checking whether some tracking was involved or not).

Update:
After thinking about this a bit longer, I think these requests can not only affect the efficiency of the scan by unnecessarily increasing the volume of data, but also raise privacy concerns as they send potentially sensitive information to Mozilla.
Although this feature is useful in a normal browsing context to protect users from trackers, I think it is even counterproductive in a pentesting context where an automated tool like ZAP is used.
Disabling this feature by default when ZAP controls Firefox instances for pentesting makes sense. This would not only improve the efficiency of the scanning process by avoiding unnecessary data transfers, but would also help to better maintain the privacy and security of the tested systems. (Plus it would be a kind gesture to Mozilla as it reduces the unnecessary load on their servers.)

[Lonzak]

@psiinon Do you have an idea how to disable this?

[thc202]

In that case it's something to be set by geckodriverthen.

[Lonzak]

So it is possible to pass preferences along when starting it?

[JanikkinaJ]

Is there a plan to change this? If not I could work on creating a pull request, but may need a few pointers on where to look as it seems like @thc202 thinks it should be set by the geckodriver and not by firefox preferences as I had originally assumed.

[thc202]

Feel free to open a pull request but only set the preference for "tools" not manual usage, or if you really want to do for all make sure to document that so users know that it's being disabled.

The geckodriver is what sets the preferences when starting Firefox, what I was saying (based on your arguments) is that it should be a default preference but that would have to be raised in geckodriver repo not here.

[Lonzak]

Do you mean here? GeckoDriver project at mozilla.
This option should be default when started from ZAP - not a default for the "whole world" per se. Gecko driver is used by lots of other projects with different intentions and goals. For other projects it might make sense to have it activated.
I don't know the inner workings of ZAP too well but there must be a place where the gecko driver is configured and started. Like it is described this issue:

FirefoxOptions options = new FirefoxOptions();
options.SetPreference("network.proxy.http", "localhost");
options.SetPreference("network.proxy.http_port", "8080");
FirefoxDriverService service = FirefoxDriverService.CreateDefaultService(Settings.DriverDirectory, "geckodriver.exe");
service.FirefoxBinaryPath = @"...";
WebDriver = new FirefoxDriver(service, options, time);

In the options I would add (which of the two I do not know):

options.SetPreference("pref.privacy.trackingprotection.enabled", "false");
options.SetPreference("pref.privacy.disable_button.tracking_protection_exceptions", "true");

[JanikkinaJ]

Feel free to open a pull request but only set the preference for "tools" not manual usage, or if you really want to do for all make sure to document that so users know that it's being disabled.

How would I set this to only work for tools? My idea would have been to add it like so in the selenium extension :

case FIREFOX_HEADLESS:
                FirefoxOptions firefoxOptions = new FirefoxOptions();
                setCommonOptions(firefoxOptions, proxyAddress, proxyPort);

                String binaryPath =
                        System.getProperty(SeleniumOptions.FIREFOX_BINARY_SYSTEM_PROPERTY);
                if (binaryPath != null && !binaryPath.isEmpty()) {
                    firefoxOptions.setBinary(binaryPath);
                }

                // Keep proxying localhost on Firefox >= 67
                firefoxOptions.addPreference("network.proxy.allow_hijacking_localhost", true);

                // Ensure ServiceWorkers are enabled for the HUD.
                firefoxOptions.addPreference("dom.serviceWorkers.enabled", true);

                // The new option
                firefoxOptions.addPreference("privacy.trackingprotection.enabled", false);

But this would be a global change.

[JanikkinaJ]

I haven't been able to create a working solution. Neither with the above code or just a plain policies file which works on the normal firefox browser when put in /etc/firefox/policies/policies.json

{
  "policies": {
    "EnableTrackingProtection": {
      "Value": false,
      "Locked": true,
      "Cryptomining": false,
      "Fingerprinting": false,
      "EmailTracking": false,
      "Exceptions": []
    }
  }
}

For some reason firefox when started by zap ignores this policy even though it is meant to globally effect firefox instances. @psiinon @thc202 do you have any idea why this might be?

[kingthorin]

It probably has to do with selenium/webdriver launching the browser and using an empty profile. it'd have to be set programmatically.

[JanikkinaJ]

okay thanks, do you have any further hints on how to do this? Or any pointer to the class where this geckodriver is instantiated with this empty profile

[kingthorin]

It'd be back in this code you quoted earlier: #8374 (comment)

I'm not sure how you'd limit it to specific invocations/tools though.

@thc202 / @psiinon is there a reason this shouldn't be added as a default Global Exclude or TLS passthru.

• https://www.zaproxy.org/docs/desktop/addons/network/options/globalexclusions/
• https://github.com/zaproxy/zap-extensions/blob/45adde52612b64a0352ed3e166b6fc8caf5539dc/addOns/network/src/main/java/org/zaproxy/addon/network/GlobalExclusionsOptions.java#L53
• https://www.zaproxy.org/docs/desktop/addons/network/options/localservers/#pass-through

[JanikkinaJ]

I have attempted to fix this, this is what I have tried so far:
To check requests made I just used mitmproxy as it was the easiest way to directly see calls made.
I started mitmproxy like so mitmdump -v --listen-port 7070
To use mitmproxy with zap I created a conf file for zap to run with with these settings:

connection.proxyChain.enabled=true
connection.proxyChain.hostName=0.0.0.0
connection.proxyChain.port=7070

and started zap with
./gradlew run --args="-configfile conf"

You can see many calls made to tracking protection by just using the zap to manually go to google.com

This does not happen by default on my normal firefox browser when proxying this.
And the calls don't show up in the zap site tree:

I have added a firefox policy which should disable this to /etc/firefox/policies/.

{
  "policies": {
    "EnableTrackingProtection": {
      "Value": false,
      "Locked": true,
      "Cryptomining": false,
      "Fingerprinting": false,
      "EmailTracking": false,
      "Exceptions": []
    }
  }
}

Which is enabled in the firefox browser started by zap:

But i still get trackingprotection requests initiated.
I also made some changes in ExtensionSelenium.java and deployed it with gradle:

firefoxOptions.addPreference("privacy.disable_button.tracking_protection_exceptions", true);
firefoxOptions.addPreference("network.cookie.cookieBehavior", 0);
firefoxOptions.addPreference("privacy.socialtracking.block_cookies.enabled", false);
firefoxOptions.addPreference("privacy.trackingprotection.enabled", false);
firefoxOptions.addPreference("privacy.trackingprotection.pbmode.enabled", false);
firefoxOptions.addPreference("privacy.trackingprotection.cryptomining.enabled", false);
firefoxOptions.addPreference("privacy.trackingprotection.fingerprinting.enabled", false);

and yet the tracking protection requests are still made.
Does anyone know what is causing them to be made and how to stop them?

[Lonzak]

Ok that is quite strange and should have worked imho but it seems that both ways (code change in ExtensionSelenium.java or via policy file) don't work. Any other thoughts how to stop the tracking protection calls?

[kingthorin]

To the original issue: Is this really about pass-thru or are you after blocking/prevention?

Because you should be able to use Global Excludes or TLS Pass-thru if that fits the requirement.