Storage emulator returns 403 Forbidden when trying to access files uploaded with firebaseStorageDownloadTokens set in metadata

[miikebar]

[REQUIRED] Environment info

firebase-tools: 9.11.0

Platform: Windows 10

[REQUIRED] Test case

Cloud Function responsible for image optimization (based on official extension). Downloads stored file to tmp directory, performs optimization and overwrites the orignal file, copying over the firebaseStorageDownloadTokens

functions/optimizeImage.ts

import * as path from 'path';
import * as os from 'os';
import * as functions from 'firebase-functions';
import * as fs from 'fs';
// Admin SDK initialization
import { admin } from '@/lib/firebase/admin';

const storage = admin.storage();
const mkdirp = require('mkdirp');
const sharp = require('sharp');

export const optimizeImage = functions.storage
  .object()
  .onFinalize(async (object) => {
    if (!object.name || !object.contentType?.startsWith('image/')) {
      return null;
    }

    const parts = object.name.split('/');
    const fileName = parts[parts.length - 1];

    const bucket = storage.bucket(object.bucket);
    const originalFile = bucket.file(object.name);
    const tmpFilePath = path.join(os.tmpdir(), object.name);
    const tmpFolderPath = path.dirname(tmpFilePath);
    const optimizedFilePath = path.join(tmpFolderPath, `optimized-${fileName}`);

    try {
      const metaResponse = await originalFile.getMetadata();
      const metadata = metaResponse?.[0]?.metadata;

     // Print the token for debug purposes
     console.log('Firebase Storage download token:', metadata.firebaseStorageDownloadTokens)

      if (metadata.optimized) {
        return;
      }

      await mkdirp(tmpFolderPath);
      await originalFile.download({
        destination: tmpFilePath,
        validation: false,
      });

      await sharp(tmpFilePath)
        .resize({
          width: 1920,
          withoutEnlargement: true,
        })
        .toFormat('jpg')
        .jpeg({
          quality: 70,
          chromaSubsampling: '4:4:4',
          force: true,
        })
        .toFile(optimizedFilePath);

      // Merge original metadata containing firebaseStorageDownloadTokens
      // with custom metadata
      const optimizedFileMetadata = {
        metadata: {
          ...metadata,
          optimized: true,
        },
      };

      await bucket.upload(optimizedFilePath, {
        destination: originalFile,
        metadata: optimizedFileMetadata,
      });

      await fs.unlinkSync(tmpFilePath);
      await fs.unlinkSync(optimizedFilePath);

      return { success: true };
    } catch (error) {
      return Promise.reject(error);
    }
  });

[REQUIRED] Steps to reproduce

1. Run the emulators: firebase emulators:start
2. Upload an image to the Storage (for example using the Storage Emulator UI)
3. Open the file from Storage Emulator and copy file URL from 403 error in console (related to Unexpected Firebase Cloud Storage Emulator behaviour when objects uploaded using the Admin SDK and the Web SDK #3393) and append
   ?alt=media&token=<token> to the link, replacing token with the token you get from optimizeImage function console.log

Alternatively build the URL yourself:
http://localhost:9199/v0/b/<bucket>/o/<file-path>?alt=media&token=<token>

Attempting to obtain the download url using fileRef.getDownloadURL() results in the following error:

The above function works fine and returns the URL when running using production Firebase.

[REQUIRED] Expected behavior

Setting firebaseStorageDownloadTokens should allow you to access the file by providing the ?token=<token> in the resource url. This is used by the official image resize extension and works fine in production (even though it is not officialy documented):

[REQUIRED] Actual behavior

After overwriting the file or even creating a new file with firebaseStorageDownloadTokens set in the metadata, the file is inaccessible and returns 403 Forbidden

[2021-05-22T20:44:15.631Z] [runtime-status] [8668] Trigger "optimizeImage" has been found, beginning invocation! {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] Trigger \"optimizeImage\" has been found, beginning invocation!"}}
i  functions: Beginning execution of "optimizeImage" {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"Beginning execution of \"optimizeImage\""}}
[2021-05-22T20:44:15.632Z] [runtime-status] [8668] triggerDefinition {"regions":["europe-central2"],"eventTrigger":{"resource":"projects/_/buckets/<bucket>.appspot.com","eventType":"google.storage.object.finalize","service":"storage.googleapis.com"},"name":"optimizeImage","entryPoint":"optimizeImage"} {"met
adata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] triggerDefinition {\"regions\":[\"europe-central2\"],\"eventTrigger\":{\"resource\":\"projects/_/buckets/<bucket>.appspot.com\",\"eventType\":\"google.storage.object.finalize\",\"service\":\"storag
e.googleapis.com\"},\"name\":\"optimizeImage\",\"entryPoint\":\"optimizeImage\"}"}}
[2021-05-22T20:44:15.632Z] [runtime-status] [8668] Running optimizeImage in mode BACKGROUND {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] Running optimizeImage in mode BACKGROUND"}}
[2021-05-22T20:44:15.633Z] [runtime-status] [8668] ProcessBackground {"eventId":"1621716254902","timestamp":"2021-05-22T22:44:14.902Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66t
tZM-unsplash.jpg","type":"storage#object"},"data":{"kind":"#storage#object","name":"danielle-cerullo-CQfNt66ttZM-unsplash.jpg","bucket":"<bucket>.appspot.com","generation":"1621716254898","metageneration":"1","contentType":"image/jpeg","timeCreated":"2021-05-22T22:44:14.897Z","updated":"2021-05-22T22:44:14.
900Z","storageClass":"STANDARD","size":"264694","md5Hash":"HWnJ+ZgmyNHPj9HLQqCXwQ==","etag":"someETag","metadata":{"firebaseStorageDownloadTokens":"c2887239-8b0c-4b47-93e1-974415480b6e","optimized":true},"crc32c":"----6A==","timeStorageClassUpdated":"2021-05-22T22:44:14.897Z","id":"<bucket>.appspot.com/dani
elle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898","selfLink":"http://localhost:9199/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg","mediaLink":"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg?generation=16
21716254898&alt=media"}} {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] ProcessBackground {\"eventId\":\"1621716254902\",\"timestamp\":\"2021-05-22T22:44:14.902Z\",\"eventType\":\"google.storage.object.finalize\",\"resource\":{\"service\":\"stora
ge.googleapis.com\",\"name\":\"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"type\":\"storage#object\"},\"data\":{\"kind\":\"#storage#object\",\"name\":\"danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"bucket\":\"<bucket>.appspot.com\",\"generation\":\"16
21716254898\",\"metageneration\":\"1\",\"contentType\":\"image/jpeg\",\"timeCreated\":\"2021-05-22T22:44:14.897Z\",\"updated\":\"2021-05-22T22:44:14.900Z\",\"storageClass\":\"STANDARD\",\"size\":\"264694\",\"md5Hash\":\"HWnJ+ZgmyNHPj9HLQqCXwQ==\",\"etag\":\"someETag\",\"metadata\":{\"firebaseStorageDownloadTokens\
":\"c2887239-8b0c-4b47-93e1-974415480b6e\",\"optimized\":true},\"crc32c\":\"----6A==\",\"timeStorageClassUpdated\":\"2021-05-22T22:44:14.897Z\",\"id\":\"<bucket>.appspot.com/danielle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898\",\"selfLink\":\"http://localhost:9199/storage/v1/b/<bucket>.appspot.co
m/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"mediaLink\":\"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg?generation=1621716254898&alt=media\"}}"}}
[2021-05-22T20:44:15.634Z] [runtime-status] [8668] RunBackground {"data":{"kind":"#storage#object","name":"danielle-cerullo-CQfNt66ttZM-unsplash.jpg","bucket":"<bucket>.appspot.com","generation":"1621716254898","metageneration":"1","contentType":"image/jpeg","timeCreated":"2021-05-22T22:44:14.897Z","updated
":"2021-05-22T22:44:14.900Z","storageClass":"STANDARD","size":"264694","md5Hash":"HWnJ+ZgmyNHPj9HLQqCXwQ==","etag":"someETag","metadata":{"firebaseStorageDownloadTokens":"c2887239-8b0c-4b47-93e1-974415480b6e","optimized":true},"crc32c":"----6A==","timeStorageClassUpdated":"2021-05-22T22:44:14.897Z","id":"<bucket>
/danielle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898","selfLink":"http://localhost:9199/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg","mediaLink":"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-uns
plash.jpg?generation=1621716254898&alt=media"},"context":{"eventId":"1621716254902","timestamp":"2021-05-22T22:44:14.902Z","eventType":"google.storage.object.finalize","resource":{"service":"storage.googleapis.com","name":"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66ttZM-unsplash
.jpg","type":"storage#object"}}} {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] RunBackground {\"data\":{\"kind\":\"#storage#object\",\"name\":\"danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"bucket\":\"<bucket>.appspot.com\",\"generation\"
:\"1621716254898\",\"metageneration\":\"1\",\"contentType\":\"image/jpeg\",\"timeCreated\":\"2021-05-22T22:44:14.897Z\",\"updated\":\"2021-05-22T22:44:14.900Z\",\"storageClass\":\"STANDARD\",\"size\":\"264694\",\"md5Hash\":\"HWnJ+ZgmyNHPj9HLQqCXwQ==\",\"etag\":\"someETag\",\"metadata\":{\"firebaseStorageDownloadTo
kens\":\"c2887239-8b0c-4b47-93e1-974415480b6e\",\"optimized\":true},\"crc32c\":\"----6A==\",\"timeStorageClassUpdated\":\"2021-05-22T22:44:14.897Z\",\"id\":\"<bucket>.appspot.com/danielle-cerullo-CQfNt66ttZM-unsplash.jpg/1621716254898\",\"selfLink\":\"http://localhost:9199/storage/v1/b/<bucket>.appsp
ot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"mediaLink\":\"http://localhost:9199/download/storage/v1/b/<bucket>.appspot.com/o/danielle-cerullo-CQfNt66ttZM-unsplash.jpg?generation=1621716254898&alt=media\"},\"context\":{\"eventId\":\"1621716254902\",\"timestamp\":\"2021-05-22T22:44:14.902Z\",\"even
tType\":\"google.storage.object.finalize\",\"resource\":{\"service\":\"storage.googleapis.com\",\"name\":\"projects/_/buckets/<bucket>.appspot.com/objects/danielle-cerullo-CQfNt66ttZM-unsplash.jpg\",\"type\":\"storage#object\"}}}"}}
>  Firebase Storage download token: c2887239-8b0c-4b47-93e1-974415480b6e {"user":"Firebase Storage download token: c2887239-8b0c-4b47-93e1-974415480b6e","metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"\u001b[90m> \u001b[39m Firebase Storage download token: c2887239-8b0c-4
b47-93e1-974415480b6e"}}
[2021-05-22T20:44:15.672Z] [runtime-status] [8668] Ephemeral server survived. {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"[runtime-status] [8668] Ephemeral server survived."}}
i  functions: Finished "optimizeImage" in ~1s {"metadata":{"emulator":{"name":"functions"},"function":{"name":"optimizeImage"},"message":"Finished \"optimizeImage\" in ~1s"}}
[2021-05-22T20:44:15.675Z] [worker-optimizeImage-70ee1f4d-bb78-4c7e-9017-c7d8108f8742]: IDLE {"metadata":{"emulator":{"name":"functions"},"message":"[worker-optimizeImage-70ee1f4d-bb78-4c7e-9017-c7d8108f8742]: IDLE"}}
[2021-05-22T20:44:15.676Z] [work-queue] {"queueLength":0,"workRunningCount":0}

[abeisgoat]

Interesting, thanks for the report, we probably drop the tokens because it's a special field in the metadata, we probably just missed implementing a setter. Will get it fixed for for a release this week.