tests(smoke): check for passing robots-txt

[connorjclark]

would have prevented #12936

[connorjclark]

took awhile to figure out why the smoke test wasn't passing, it's because seo tester html overrides URL! so, luckily we get a driveby fix here for naughty pages overriding globals.

[patrickhulce]

hm, _cachedNativesPreamble should be solving this, is something broken there? that's more concerning than the smoke itself :/

lighthouse/lighthouse-core/gather/driver/execution-context.js

Lines 226 to 235 in ce87060

	/**
	* Prefix every script evaluation with a shadowing of common globals that tend to be ponyfilled
	* incorrectly by many sites. This allows functions to still refer to `Promise` instead of
	* Lighthouse-specific backups like `__nativePromise` (injected by `cacheNativesOnNewDocument` above).
	*/
	static _cachedNativesPreamble = [
	'const Promise = globalThis.__nativePromise || globalThis.Promise',
	'const URL = globalThis.__nativeURL || globalThis.URL',
	'const performance = globalThis.__nativePerformance || globalThis.performance',
	].join(';\n');

[connorjclark]

ok so this seems fine and i can't get it to fail again, weird

[connorjclark]

@patrickhulce / @adamraine any idea why FR is failing seo-passing / to fetch robots.txt?

[adamraine]

Could it be flaky? I'm not seeing a failure locally on the smoke test or a direct FR navigation of the test page.