Unable to log in to Hypothesis from USA Today pages in Via

[mattdricker]

Update 2022-04-19: The problem with usatoday.com and the Cross-Origin-Opener-Policy header does not affect Via at present. It only affects the extension and bookmarklet. See #333 (comment) for explanation.

Bug report form

Steps to reproduce

1. Make sure you are logged out Hypothesis
2. Open any page at USA Today e.g. https://www.usatoday.com/story/news/politics/2022/03/07/ukraine-russia-invasion-live-updates/9404740002/
3. Activate Hypothesis extension or bookmarklet, and click the sidebar link to Log In

Expected behaviour

The Hypothesis login pop-up window should appear with appropriate username and password fields available.

Actual behaviour

An entirely blank pop-up window displays.

Browser/system information

Tested on:

• Chrome 99
• Firefox 97
• Brave 1.36.109

Note: The issue does not seem to affect Safari 15.3

Additional details

If you are already logged in to Hypothesis all functionality works as expected.

[robertknight]

Interestingly when I visit that URL I get redirected to https://eu.usatoday.com/story/news/politics/2022/03/07/ukraine-russia-invasion-live-updates/9404740002/ and I can't reproduce there in Chrome 101.

[chrisshaw]

To make this more annoying, when I try to go to eu. I get redirected to www where I can reproduce it 🙃

The following is actually a WEBM file (my OS's screen recorder outputs WEBM, but GitHub doesn't support WEBM file uploads). It won't play in browsers without WEBM support.

Chrome Version 99.0.4844.51 (Official Build) (64-bit)

Screencast.from.03-07-2022.01.49.12.PM.webm.mp4

[robertknight]

I was able to reproduce in Chrome using Sauce Labs to load the page from a US-based server, using the browser extension. The window.open call that the OAuthClient class in the client makes is returning null.

This might be something to do with the headers the top-level frame is returning:

content-type: text/html; charset=utf-8
etag: W/"36e87-xQxIdQhDQBw7nNrIxHR+WGsIvwA"
x-content-type-options: nosniff
x-frame-options: deny
x-xss-protection: 1; mode=block
feature-policy: camera 'none';display-capture 'none';geolocation 'none';microphone 'none';payment 'none';usb 'none';xr-spatial-tracking 'none'
permissions-policy: camera=(),display-capture=(),geolocation=(),microphone=(),payment=(),usb=(),xr-spatial-tracking=()
referrer-policy: strict-origin-when-cross-origin
content-security-policy-report-only: script-src https: blob: 'unsafe-inline' 'unsafe-eval' 'self';base-uri 'self';report-uri https://reporting-api.gannettinnovation.com;report-to default
content-security-policy: upgrade-insecure-requests;frame-ancestors 'none';object-src 'none'
cross-origin-opener-policy: same-origin
cross-origin-resource-policy: same-origin
origin-agent-cluster: ?1
accept-ranges: bytes
date: Wed, 16 Mar 2022 14:39:52 GMT
age: 5
set-cookie: gup_anonid=c4d0b197-c729-4a26-8d66-d91134674253; Domain=.usatoday.com; Max-Age=31536000; Path=/; SameSite=Lax; Secure
set-cookie: gup_clientid=3216a3a7-781d-4a99-b099-5d5d66f39e1c; Domain=.usatoday.com; Max-Age=31536000; Path=/; SameSite=Lax; Secure
cache-control: no-store
set-cookie: gnt_ub=1; domain=.usatoday.com; path=/; secure; samesite=lax; max-age=31536000;
set-cookie: gnt_sb=1; domain=.usatoday.com; path=/; secure; samesite=lax; max-age=31536000;
set-cookie: gnt_eid=control:1; domain=.usatoday.com; path=/; secure; samesite=lax; max-age=5184000;
gannett-cam-experience-id: control:1
nel: {"report_to":"default","max_age":31557600,"include_subdomains":true,"success_fraction":0.005}
report-to: {"max_age":31557600,"include_subdomains":true,"endpoints":[{"url":"https://reporting-api.gannettinnovation.com"}]}
link: <https://www.usatoday.com/tangstatic/svg/weather/7-q1a2z3371d08dc.svg>;rel=preload;as=image;nopush
set-cookie: gnt_d=%7B%22w%22%3A%7B%22t%22%3A%2248%22%2C%22f%22%3A%227-q1a2z3371d08dc%22%2C%22c%22%3A%22Cloudy%22%7D%2C%22z%22%3A%2295123%22%2C%22c%22%3A%22san%20jose%22%2C%22s%22%3A%22CA%22%7D; domain=.usatoday.com; path=/; samesite=lax; secure; priority=high;
x-cache: MISS, HIT
x-timer: S1647441593.799128,VS0,VE1
vary: X-AbVariant,X-AbVCfg,X-AltUrl,Accept-Encoding,User-Agent
strict-transport-security: max-age=63072000
content-length: 199138

[robertknight]

The cross-origin-opener-policy header in particular is a new thing that may be relevant. From this related Chrome issue: https://bugs.chromium.org/p/chromium/issues/detail?id=1221127:

Sites that wish to continue using SharedArrayBuffer must opt-into cross-origin isolation. Among other things, cross-origin isolation will prevent cross-origin popups from having access to their opener. This behavior ships today in Firefox, and Chrome aims to ship it as well in Chrome 92.

As part of crossOriginIsolation, websites must send a Cross-Origin-Opener-Policy: same-origin header. COOP same-origin prevents pages with different top-level origins from being able to communicate with each other. This breaks many OAuth or payment flows that rely on opening a cross-origin popup that will communicate back with the page through window.postMessage for example.

I'm not clear how this interacts with iframes yet.

This feature request in this issue is written with the assumption that the target audience is a web developer who can change the headers that web pages are served with. We can change these headers in Via, but not in the bookmarklet or browser extension.

[chrisshaw]

Based on @robertknight 's investigation. We really have 2 issues here:

• Log in on USA Today when loading in Via (We'll treat this issue as a solution for Via.)
• Log in on USA Today when loading in extension: [Extension] Unable to log in to Hypothesis from USA Today pages in Extension #353

[robertknight]

This issue only affects new login attempts, existing logins can still work. A workaround is to first login when visiting another page, and then visit USA Today. When using the Chrome extension, this will work in browsers that allow Hypothesis to access its previous login state across different sites.

[robertknight]

For the case of the embedded client, an alternative same-origin messaging system may work. I got a proof of concept working in Chrome and Firefox using BroadcastChannel. Unfortunately this API is not supported in Safari. In Firefox I encountered an issue in this test where the popup window, after successfully relaying the access token back to the client, was unable to close itself, displaying this console error:

Scripts may not close windows that were not opened by script. [post-auth.bundle.js:51]

A simple but ugly solution for this would be to add a manual "Close" button to the window that is displayed if the window is unable to automatically close itself for any reason.

In the browser extension or custom clients we can't use BroadcastChannel since the login page is not same-origin with the client. What we could do is have the login page redirect the popup window to some client-provided page, using the standard redirect-based method for returning the access token in OAuth, and then having the client page in the popup window use BroadcastChannel to communicate with the sidebar.

---

BroadcastChannel proof of concept:

Changes in h:

diff --git a/h/static/scripts/post-auth.js b/h/static/scripts/post-auth.js
index fb5330700..c2d8677cd 100644
--- a/h/static/scripts/post-auth.js
+++ b/h/static/scripts/post-auth.js
@@ -11,50 +11,19 @@ import { settings } from './base/settings';
 const appSettings = settings(document);
 
 function sendAuthResponse() {
-  if (!window.opener) {
-    console.error('The client window was closed');
-    return;
-  }
-
   const msg = {
     type: 'authorization_response',
     code: appSettings.code,
     state: appSettings.state,
   };
 
-  // `document.documentMode` is a non-standard IE-only property.
-  const isIE = 'documentMode' in document;
-  if (isIE) {
-    try {
-      // IE 11 does not support `window.postMessage` between top-level windows
-      // [1]. For the specific use case of the Hypothesis client, the sidebar
-      // HTML page is on the same domain as the h service, so we can dispatch
-      // the "message" event manually. Third-party clients will need to use
-      // redirects to receive the auth code if they want to support IE 11.
-      //
-      // [1] https://blogs.msdn.microsoft.com/thebeebs/2011/12/21/postmessage-popups-and-ie/
-
-      // Create an event in the target window.
-      const clientWindow = window.opener;
-      const event = clientWindow.document.createEvent('HTMLEvents');
-      event.initEvent('message', true, true);
-
-      // Clone the `msg` object into an object belonging to the target window.
-      event.data = clientWindow.JSON.parse(JSON.stringify(msg));
-
-      // Trigger "message" event listener in the target window.
-      clientWindow.dispatchEvent(event);
-      window.close();
-    } catch (err) {
-      console.error(
-        'The "web_message" response mode is not supported in IE',
-        err
-      );
-    }
-    return;
+  if (window.opener) {
+    window.opener.postMessage(msg, appSettings.origin);
+  } else {
+    const channel = new BroadcastChannel('client-login');
+    channel.postMessage(msg);
   }
 
-  window.opener.postMessage(msg, appSettings.origin);
   window.close();
 }

Changes in the client:

diff --git a/src/sidebar/services/auth.js b/src/sidebar/services/auth.js
index f00e7b1ec..77cbbadc9 100644
--- a/src/sidebar/services/auth.js
+++ b/src/sidebar/services/auth.js
@@ -270,9 +270,8 @@ export class AuthService extends TinyEmitter {
      * then exchange for access and refresh tokens.
      */
     async function login() {
-      const authWindow = OAuthClient.openAuthPopupWindow($window);
       const client = await oauthClient();
-      const code = await client.authorize($window, authWindow);
+      const code = await client.authorize($window);
 
       // Save the auth code. It will be exchanged for an access token when the
       // next API request is made.
diff --git a/src/sidebar/util/oauth-client.js b/src/sidebar/util/oauth-client.js
index eeb821d32..a8b50f96d 100644
--- a/src/sidebar/util/oauth-client.js
+++ b/src/sidebar/util/oauth-client.js
@@ -116,11 +116,9 @@ export class OAuthClient {
    * Returns an authorization code which can be passed to `exchangeAuthCode`.
    *
    * @param {Window} $window - Window which will receive the auth response.
-   * @param {Window} authWindow - Popup window where the login prompt will be shown.
-   *   This should be created using `openAuthPopupWindow`.
    * @return {Promise<string>}
    */
-  authorize($window, authWindow) {
+  authorize($window) {
     // Random state string used to check that auth messages came from the popup
     // window that we opened.
     //
@@ -149,7 +147,8 @@ export class OAuthClient {
         }
         $window.removeEventListener('message', authRespListener);
       }
-      $window.addEventListener('message', authRespListener);
+      const channel = new BroadcastChannel('client-login');
+      channel.addEventListener('message', authRespListener);
     });
 
     // Authorize user and retrieve grant token
@@ -160,10 +159,21 @@ export class OAuthClient {
     authURL.searchParams.set('response_type', 'code');
     authURL.searchParams.set('state', state);
 
-    // @ts-ignore - TS doesn't support `location = <string>`. We have to
-    // use this method to set the URL rather than `location.href = <string>`
-    // because `authWindow` is cross-origin.
-    authWindow.location = authURL.toString();
+    // In Chrome & Firefox the sizes passed to `window.open` are used for the
+    // viewport size. In Safari the size is used for the window size including
+    // title bar etc. There is enough vertical space at the bottom to allow for
+    // this.
+    //
+    // See https://bugs.webkit.org/show_bug.cgi?id=143678
+    const width = 475;
+    const height = 430;
+    const left = $window.screen.width / 2 - width / 2;
+    const top = $window.screen.height / 2 - height / 2;
+
+    // Generate settings for `window.open` in the required comma-separated
+    // key=value format.
+    const authWindowSettings = `left=${left},top=${top},width=${width},height=${height}`;
+    $window.open(authURL, 'Log in to Hypothesis', authWindowSettings);
 
     return authResponse.then(rsp => rsp.code);
   }

[robertknight]

If we are unable to rely on window.opener access in the popup window, then a general-purpose solution for the login process might look something like:

1. Client opens popup window at h authorization URL
2. After authorization completes, h redirects the popup window to a pre-registered client-specific redirect URL, following the standard OAuth protocol for returning an access code to the client
3. Client page in the popup window communicates the access token back to the sidebar using a same-origin communication mechanism, such as BroadcastChannel.
4. Sidebar completes exchange of access code for access token

A downside of this change is that it is going to be slightly slower due to the added redirect. An upside is it will remove the client's reliance on the non-standard window.postMessage-based method of receiving the access code.

Something I'm not sure about, and will need to check, is whether HTTP URLs are allowed to redirect to extension URLs.

[robertknight]

BroadcastChannel is supported in Safari 15.4 but I encountered an issue with it being unable to be used for popup => opener communication, whereas in Chrome and Firefox this works. This might relate to origin partitioning that is implemented in Safari and being planned for Chrome. See whatwg/html#5803.

[robertknight]

Some related discussion about the Cross-Origin-Opener-Policy specs and popups here: whatwg/html#7713. From the linked explainer:

The solution we propose is to add (yet) another COOP value: Popups (name likely to change). It says: "Pages that set a different COOP policy or have different top level origin will be placed in a different BrowsingContext group. However the opener relationship will not be severed, but the only windowProxy properties that remain accessible through it are postMessage and closed." Served together with a suitable COEP policy it would enable crossOriginIsolated.

That unfortunately might not help us since we can't control the COOP policy on the top-level document in all cases, though we can control it in Via.

whatwg/html#6364 is also relevant. In particular the second bullet point in the issue description (under "possible alternatives") is what we are looking at implementing here with BroadcastChannel.

[robertknight]

The situation with standards is fluid, but whatwg/html#6364 has the most up-to-date discussion. My understanding of where this is likely to end up is:

1. Browsers will eventually provide a way for a popup window to communicate with a cross-origin frame that opens it via window.opener, even if that iframe or its parent document are using Cross-Origin-Opener-Policy. The window.opener reference would be limited to just use of the postMessage and closed properties, which would be adequate for us. Enabling this might require the parent document or popup to set a custom Cross-Origin-Opener-Policy headers. If headers have to be changed on the site where the client was embedded, as opposed to the URL opened in the popup window, that's more complicated and would depend on the method of embedding the client:
   • In Via we can override headers on any document.
   • In the bookmarklet we can't. This would add another category of site (in addition to those that use Content-Security-Policy) where the bookmarklet simply can't work.
   • In the Chrome extension we'd have to ask for additional permissions, which is undesirable, so I think we'd use some other extension-specific mechanism to handle the login
2. In today's browsers, BroadcastChannel can be used as a workaround for pages that use COOP in Firefox and Chrome but not Safari. The same applies to other mechanisms of same-origin communication such as localStorage. BroadcastChannel is liable to stop working in future though as Firefox/Chrome are planning to partition it similar to Safari. Hopefully by then they will have implemented (1) however
3. It might be possible to use document.requestStorageAccess() together with document.cookie to allow the popup to communicate with the iframe in Safari/Firefox. I need to prototype this.

So the status quo seems to be that there is no solution that works in all browsers, all methods of delivering the client and does not affect the page's ability to use web platform features that are depend on COOP.

Some options solutions we can implement today:

1. Strip the Cross-Origin-Opener-Policy header in Via, and rely on sites not actively using the web platform features that are gated behind it. When the solution described in (1) is available, we could change the header to one that allows popups rather than strip it entirely.
2. Use BroadcastChannel or document.cookie as popup <-> iframe communication mechanisms, with the understanding that these may become unavailable in future.
3. Use an alternative method of communicating the access token from the popup to iframes in the extension. This will involve extension-specific APIs.

[chrisshaw]

@robertknight Focusing only on Via for this ticket (Extension covered in #353):

• @mattdricker do you still see this issue on USA Today? Am I crazy, or did they change their COOP? (I also am unable to reproduce the archlinux.org example ignore me, can't read the right headers)
  • With that said, could you test this on Via to make sure I'm not crazy? Because:
  • I think Via may be a solved problem. I don't know if we just tested incorrectly or whether there was work done on this already, but it looks like Via is already stripping this header. I am able to get the log-in pop-up and successfully log in.
• @robertknight Let's assume for a moment this is a problem on other sites: if we have uesrs complete a workaround where they go log in on a site that does not use this COOP (maybe even one we host), can they then come back to this Via page, would that work to "log them in?" In my testing, it seems like no, but you've mentioned this affects only new logins, which make me think I'm just doing it wrong or you were referring specifically to the extension.

I'm starting to think the best answer here is likely to point people to a workaround and let the standards solidify, rather than stripping out headers that sites likely don't want stripped out. Especially if neither report is still an issue. Once again, I can't read headers. Plus it seems like we are stripping out headers, if I'm testing correctly. So the best answer for Via is to do nothing.

[robertknight]

In general Via does not strip the Cross-Origin-Opener-Policy header. If test a local instance of Via with the client's COOP test page at http://localhost:3000/document/coop-test then I see that viahtml preserves the Cross-Origin-Opener-Policy: same-origin header. Searching through pywb's source I didn't find any relevant mentions of COOP either.

With the original test URL (https://www.usatoday.com/story/news/politics/2022/03/07/ukraine-russia-invasion-live-updates/9404740002/) however I do see a difference. To avoid the redirect mentioned in #333 (comment) I ran these tests from one of our AWS servers:

curl -i 'https://www.usatoday.com/story/news/politics/2022/03/07/ukraine-russia-invasion-live-updates/9404740002/' 

Returns a response with both cross-origin-resource-policy: same-origin and cross-origin-opener-policy: same-origin headers

curl -H 'Referer: https://via.hypothes.is' -i 'https://viahtml.hypothes.is/https://www.usatoday.com/story/news/politics/2022/03/07/ukraine-russia-invasion-live-updates/9404740002/'

Returns a response with a cross-origin-resource-policy: same-origin header but no cross-origin-opener-policy: same-origin header.

So there might be something about the way that ViaHTML makes the request which affects on USA Today's end whether a Cross-Origin-Opener-Policy header gets set. I'll look for some other, simpler, test page to confirm.

[robertknight]

It turns out that one of the engineers working on these specifications has created a test page which can be used for testing this with Via. See https://via.hypothes.is/https://arturjanc.com/cgi-bin/coop/index-with-coop.py?coop=same-origin

Inspecting the requests made when accessing this page through Via, I can see that Via has preserved the cross-origin-opener-policy: same-origin header from the original site. So Via is not, in general, stripping this header. However, the user can successfully login! How comes? I believe this is because of the way Via is structured: Cross-Origin-Opener-Policy only takes effect if set on the top-level frame, but in the public Via the HTML content is currently always displayed inside a child frame. As a result the COOP header is ignored and login still works. I tested this in Safari and Firefox with the same results. In the LMS's Via the structure is different, Via redirects to ViaHTML rather than embedding it in an iframe. In this context the COOP header from the top-level frame, which is the LMS, would apply. That is outside of our control so perhaps this could become an issue in future.

The lack of COOP header on the top-level frame may eventually cause problems for some sites if they rely on using features which are only available when these headers are set. Today however this is not an issue.

If instead of using Via you visit https://arturjanc.com/cgi-bin/coop/index-with-coop.py?coop=same-origin directly and activate the extension, you'll find that login fails. What you can do for the extension is visit some other website, login there and then reload the page where COOP is set. This will continue to work in Chrome until third-party cookies are phased out. In Safari however third-party cookies are blocked and so this doesn't work.

So to summarize: contrary to what I wrote earlier, it looks like the Cross-Origin-Opener-Policy header is not a problem for public Via today. This is because the header must be set on the top-level frame to take effect. In Via however, the proxied headers from the upstream site apply to the inner iframe where ViaHTML document is being loaded. In the LMS's Via we might eventually run into the problem again if the LMS itself enables the COOP header. I don't expect that will happen soon.

[robertknight]

We've agreed to close this issue for now since the problem doesn't currently affect Via. For the Chrome extension there is a workaround which is to log in using a different site. I will keep an eye on the relevant standards discussions and create / update issues as needed.