Transfer ArrayBuffers between AndroidWebview and JavaScript
Author: linyhe@microsoft.com
Summary: Current API cannot efficiently pass binary data between Android Webview and JavaScript. Existing transferable objects support move (zero copy) objects between threads. But due to Chromium’s multi-process architecture, transferring ArrayBuffer between BrowserProcess and RendererProcess is challenging. This doc has 3 options for that goal, and discusses related AndroidX Webkit API. This doc also discusses other transferable object types that may be implemented.
Transfer ArrayBuffer on MessagePorts
Discussion #1: How to transfer ArrayBuffer between BrowserProcess and RendererProcess
#1: Use MessagePort, copy data over interprocess shared memory (BigBuffer)
#2: components/js_injection: copy data over interprocess shared memory (BigBuffer)
Discussion #2: AndroidX public API and other data types
#1 Pass JavaScript data object with structuredClone
#2 AndroidX public API and maybe other data types
#2.1 AndroidX - Webview boundary interface
Background
HTML5 spec defines a mechanism to pass ownership of specific objects (a.k.a. transferable objects) among windows, iframes and workers. This is useful to transfer objects without copying internal data (we call this move semantics).
This works well between different threads, the BackingStore owned by existing ArrayBuffer can be switched to newly created. However, if message is transferred between process, 2 times of copy is needed.
Current implementation
Transfer ArrayBuffer on MessagePorts
Summary: within process: move (zero copy). Different process: two times of copy.
Same process:
Each ArrayBuffer holds a BackingStore, when transferring ArrayBuffer, it will transfer the BackingStore to the newly created ArrayBuffer.
Cross process:
Existing implementation in Chromium uses Mojom TransferableMessage, ArrayBuffer is one of the supported types.
For one way data transfer, a new BigBuffer with SharedMemory (If data is larger than 64KiB) will be created. Data in BackingStore is copied onto SHM, and passes the handle to the receiver process. Receiver process will map the SHM and create a new SharedBuffer, then copy data from SHM into the new SharedBuffer.
Discussion #1: How to transfer ArrayBuffer between BrowserProcess and RendererProcess
This will focus on discussing internal implementation in Android Webview. Public API (AndroidX) please see Discussion#2.
#1: Use MessagePort, copy data over interprocess shared memory (BigBuffer)
Existing AndroidX Webkit API: WebMessagePortCompat (Also available in AOSP).
For now Android Webview already uses this mechanism to pass String (WebMessageCompat#String data). We can reuse MessagePort in Android Webview to add support of ArrayBuffer.
+ Reuse existing logic in MessagePort.
- For 1 way data transfer, 2 times of copy is required.
- App developer need to handle with MessagePort.
#2: components/js_injection: copy data over interprocess shared memory (BigBuffer)
Existing AndroidX Webkit API: WebMessageListener.
The existing API supports to inject a JavaScript object, and receive messages from it. Unlike WebMessagePort, this API does not require app developers to understand the complexity of MessagePort. This is a new API only designed for Android Webview.
We can extend this API to add support for ArrayBuffer, and reuse existing SerializedArrayBuffer mojom type.
+ New API for AWebview, app developers do not need to handle with MessagePort.
+ Reuse existing logic in js_injection, and ArrayBuffer mojom type.
- For 1 way data transfer, 2 times of copy is required.
#3: components/js_injection: Alloc shared memory from browser process, pass handle to renderer process
Similar to #2, this is to have an optimized way to reduce times of copy needed.
For the current implementation of ArrayBuffer, the data is not allocated on SHM, and it’s not possible to do that due to the limitation of SHM size.
We can create an ArrayBuffer(Not same type in V8) and alloc SHM for it. App developers will copy data into that ArrayBuffer. Then we will “transfer” the ArrayBuffer (SHM) to the renderer process. In the renderer process, we can create a new JS ArrayBuffer that is directly backed by the SHM created in BrowserProcess. After JavaScript read/write data, it can “transfer” the ArrayBuffer back to BrowserProcess.
With this change, we can reduce the copy times needed (For a roundtrip, 2 times of copy at most), the overall flow will be:
+ New API for AWebview, app developers do not need to handle MessagePort.
+ Reuse existing logic in js_injection, and ArrayBuffer mojom type.
+ For 2 way data transfer (roundtrip), 2 times of copy at most.
- Additional code logic required to handle the special type of ArrayBuffer.
Discussion #2: AndroidX public API and other data types
For now both MessagePort and js_injection only support String type, this doc is primarily focused on ArrayBuffer. But we may add other transferable types in the future. This section will discuss the public API change, and possibly other types.
#1 Pass JavaScript data object with structuredClone
Current WebMessage#PostMessage API supports to post any JS object, and provide an array of transferable objects at the same time. This gives web developers a flexibility to handle JS objects. StructureClone and serialization is implemented in blink/V8.
But the existing serialization code requires execution of V8, and to implement the same serialization/deserialization logic in Java is complex. App developers can still use other ways like pass JSON String or use JavaScriptInterface to pass complex data structures.
We can focus on improve data transfer performance by reusing transferable objects, supporting any data type and StructureClone is a bit out of scope.
#2 AndroidX public API and maybe other data types
The existing AndroidX Webkit API of post/receive messages is similar. Both of them only support String type, and register a listener to receive messages from JS.
For now we have 1 type of WebMessage type in AndroidX: WebMessageCompat(String data, WebMessagePortsCompat ports[]). And it’s corresponding listener type.
So the choose of MessagePort or WebMessageListener will be similar:
#2.1 Questions
- Created a new data type (JavaScriptWebMessage), which replaces WebMessageCompat. It has all capacity of WebMessageCompat (MessagePort + String), and add support of ArrayBuffer.
- Do we need to create a new one or modify previous WebMessageCompat?
- I suggest we create a new one. The WebMessageCompat is a compat version of AOSP WebMessage. Hard to change.
- What if add other JS data types?
- Add new type inside JavaScriptWebMessage, add corresponding methods in Boundary interface and AndroidX.
- Should we deprecate old API (MessagePortCompat#PostMessage, MessagePortCompat#setWebMessageCallback)?
- Will keep backward compatibility. Deprecate?
#2.1 AndroidX - Webview boundary interface
Data types:
public interface JavaScriptWebMessageBoundaryInterface extends FeatureFlagHolderBoundaryInterface {
@MessageType int getType();
/* WebMessagePort */ InvocationHandler[] getPorts();
ByteBuffer getDataAsArrayBuffer();
String getDataAsString();
@IntDef(flag = true, value = {MessageType.TYPE_NONE, MessageType.TYPE_STRING,
MessageType.TYPE_ARRAY_BUFFER})
@Retention(RetentionPolicy.SOURCE)
@interface MessageType {
int TYPE_NONE = 0;
int TYPE_STRING = 1;
int TYPE_ARRAY_BUFFER = 2;
}
}
public interface JavaScriptWebMessageCallbackBoundaryInterface extends FeatureFlagHolderBoundaryInterface{
void onMessage(/* JavaScriptWebMessage */ InvocationHandler message);
}
Create, post and receive:
public interface WebMessagePortBoundaryInterface {
void postMessage(/* WebMessage */ InvocationHandler message);
void postJavaScriptWebMessage(/* JavaScriptWebMessage */ InvocationHandler message);
void close();
void setWebMessageCallback(/* WebMessageCallback */ InvocationHandler callback);
void setWebMessageCallback(
/* WebMessageCallback */ InvocationHandler callback, Handler handler);
void setJavaScriptWebMessageCallback(/* JavaScriptWebMessageCallback
*/ InvocationHandler callback, Handler handler);
}
#2.2 Example usage of UserApp
Create, post and receive:
final WebMessagePortCompat[] channels =
mOnUiThread.createWebMessageChannelCompat();
WebkitUtils.onMainThread(() -> {
channels[1].setJavaScriptWebMessageCallback(messageHandler,
new WebMessagePortCompat.JavaScriptWebMessageCallback() {
@Override
public void onJavaScriptWebMessage(
@NonNull final JavaScriptWebMessage message) {
Assert.assertEquals(message.getType(),
JavaScriptWebMessageBoundaryInterface.MessageType.TYPE_STRING);
latch.countDown();
}
});
final JavaScriptWebMessage message = new JavaScriptWebMessage().setDataAsString(testString);
channels[0].postJavaScriptWebMessage(message);
});
#2.3 AndroidX data types
public class JavaScriptWebMessage {
@MessageType
private int mType;
@Nullable
private final WebMessagePortCompat[] mPorts;
@Nullable
private ByteBuffer mArrayBuffer;
@Nullable
private String mString;
public JavaScriptWebMessage() {
this(null);
}
public JavaScriptWebMessage(@Nullable WebMessagePortCompat[] ports) {
this.mType = MessageType.TYPE_NONE;
this.mPorts = ports;
}
@MessageType
public int getType() {
return mType;
}
@Nullable
public WebMessagePortCompat[] getPorts() {
return mPorts;
}
@NonNull
public String getDataAsString() {
if (getType() == MessageType.TYPE_STRING) {
return mString;
}
throw new RuntimeException("Wrong type!");
}
@NonNull
public JavaScriptWebMessage setDataAsString(@NonNull String string) {
mType = MessageType.TYPE_STRING;
mString = string;
return this;
}
@NonNull
public ByteBuffer getDataAsArrayBuffer() {
if (getType() == MessageType.TYPE_ARRAY_BUFFER) {
return mArrayBuffer;
}
throw new RuntimeException("Wrong type!");
}
@NonNull
public JavaScriptWebMessage setDataAsArrayBuffer(@NonNull ByteBuffer arrayBuffer) {
mType = MessageType.TYPE_ARRAY_BUFFER;
mArrayBuffer = arrayBuffer;
return this;
}
}
Discussion #3: Buffer pool between host app and WebView (Content/) when receiving ArrayBuffer from renderer process
When receiving ArrayBuffer from the renderer process, a corresponding Java byte of Array (or ByteBuffer) is created, which is managed by Java VM, and will bring a lot of GC pressure. For example, when we receive the 1MiB ArrayBuffer at a very high frequency, like 30 times/second, this is the memory debug result. (Very high GC frequency).
While normal host apps may not be needed, we can still let the host app have the ability to keep a pool between WebView, it’s optional and safe to ignore.
#3.1 AndroidX boundary
public interface WebMessagePayloadClientInterface extends FeatureFlagHolderBoundaryInterface {
// The size of provided byte array should be greater or equal to the capacity.
byte[] onPrepareArrayBuffer(int capacity);
}
public interface WebViewProviderBoundaryInterface {
// ……
void setWebMessagePayloadClientInterface(
/* WebMessagePayloadClientInterface */ InvocationHandler webMessagePayloadClientInterface);
}
public interface WebMessagePayloadBoundaryInterface extends FeatureFlagHolderBoundaryInterface {
@WebMessagePayloadType
int getType();
@Nullable
String getAsString();
@NonNull
byte[] getAsArrayBuffer();
// Should be equal to the ArrayBuffer array length, unless host app provided a reused byte
// array, so this will return the acture ArrayBuffer length (Less or equal to ArrayBuffer
// length).
int getArrayBufferSize();
@Retention(RetentionPolicy.SOURCE)
@IntDef(flag = true, value = {WebMessagePayloadType.TYPE_STRING, WebMessagePayloadType.TYPE_ARRAY_BUFFER})
@interface WebMessagePayloadType {
int TYPE_STRING = 0;
int TYPE_ARRAY_BUFFER = 1;
}
}
#3.2 Example usage in HostApp
class MyWebMessagePayloadClient implements WebMessagePayloadClient {
private byte[] mCachedBytes;
@Override
public byte[] onPrepareArrayBuffer(int capacity) {
// cached bytes can be reused if it's larger than required capacity.
if (mCachedBytes != null && mCachedBytes.length >= capacity)
return mCachedBytes;
mCachedBytes = new byte[capacity];
return mCachedBytes;
}
}
#3.3 Changes in Chromium
This part is not finalized, I’m planning to add a static callback in MessagePayload.java, which can be set from WebView. Logic should be like:
When a new ArrayBuffer received:
If ArrayBuffer allocator callback is null:
Create byte array, and copy data onto it.
ArrayBuffer allocator callback.acquire ArrayBuffer with size:
Reuse the byte array, copy data onto it.
Wrap the byte array into MessagePayload, and pass to WebView
Related
Transferable ArrayBuffers on MessagePort - Google Docs - This is a bit out-dated, current implementation is to copy data onto SHM.
Related Bug: 1023334 - Android WebView: WebMessage API has large overhead for binary data - chromium