I’m building a file upload application to familiarize myself with the concept of streams. I’m trying to turn a file list or blobs into a stream then upload it to the backend and store it on the file system.
I had no issue with frontend and backend implementations respectively but I’m having a hard time connecting the two. My main problem is that I don’t understand the difference between the Web Streams API and the Node.js Streams API. I managed to turn the blobs of selected input files in the browser into a Web ReadableStream
but the packages I tried (axios
for requests, socket.io
and socket.io-stream
for WebSocket) only accept the Node.js version Stream as arguments. I also could not pipe a Web ReadableStream into a Node.js Writeable or Duplex Stream. The method names are also different (e.g.: pipeTo
or pipeThrough
in Web API and pipe
in Node.js API).
I know there are implementation differences between Node.js and browsers but naively, I thought the APIs would be similar. Can I somehow trivially convert between Web streams and browserified Node.js streams and I’m missing something? Does it worth using the Web Stream API over stream-browserify
?
Advertisement
Answer
It’s not too difficult to convert a web stream to a Node.js stream manually, but you should really try to find a library that accepts native web streams instead of shoehorning a Node.js shim for the stream
built-in into the browser with Browserify.
However, if it proves necessary to use a Node.js stream shim in the browser, you need to install stream-browserify
and use like this:
import { Readable, Writable } from 'stream-browserify;' // window.ReadableStream to Node.js Readable const webRSToNodeRS = rs => { const reader = rs.getReader(); const out = new Readable(); reader.read().then(async ({ value, done }) => { while (!done) { out.push(value); ({ done, value } = await reader.read()); } out.push(null); }); return out; } // window.WritableStream to Node.js Writable const webWSToNodeWS = ws => { const writer = ws.getWriter(); const out = new Writable(); out._write = (chunk, encoding, callback) => { writer.write(chunk); callback(); }; out._final = callback => { writer.close(); callback(); }; return out; }
These methods should be enough to have full interop between web and Node streams. For example, if you want to pipe a web ReadableStream to a Node.js Writable/Duplex:
const pipeWebRSToWritable = (rs, writable) => { // After converting you can use the normal pipe methods webRSToNodeRS(rs).pipe(writable); }
However I’d like to mention that you don’t need a library to stream data from the client to the server. The fetch
API natively supports web streams and is probably the way you should go.
// POST a ReadableStream (perhaps of a file) to the server // Way easier and more performant than using a 3rd party library... const postRSToServer = rs => fetch('/your/server/endpoint', { method: 'POST', body: rs });
Last note: make sure you’re directly using the Blob.prototype.stream
method (call this on a File
object, e.g. file.stream()
, since File
extends Blob
). There are some ways to get a ReadableStream
from a file in JS that actually end up loading all of the file into memory in the browser, which you don’t want.