Node.js / general

Caching streaming HTML?

Caching streaming HTML?

Node.js / general · February 2, 2018 at 2:11pm(Edited 8 months ago)
I'm working on moving Spectrum to streaming server-side rendering to help with backpressure on our SSR server. Unfortunately, as I was building it I realized that it completely broke our current Redis caching mechanism, which relies on monkey-patching res.send—but res.send isn't called when streaming a response.
This is what our streaming code looks like (very simplified pseudo code):
// Write the <head> and root <div> res.write('<html><head>' + metaTags + '</head><body><div id="root>') // Render the frontend React app const stream = renderToNodeStream(<Frontend />) // Pipe that HTML to the response stream.pipe(res, { end: false }); // When React is finished, clean up the dangling HTML tags stream.on('end', () => res.end('</div></body></html>'))
Now the question is, how do I cache the generated HTML from that?

February 2, 2018 at 2:12pm

Currently starting to read https://github.com/substack/stream-handbook, maybe I just need a deeper understanding of streams

  • reply
  • like

According to @xander76 on Twitter (https://twitter.com/xander76/status/959429678372188160) the code would have to use a Transform stream, which looks something like this:

  • reply
  • like
let cacheEntry = "";

renderToNodeStream(<Frontend/>)
  .pipe(new Transform({
    transform(chunk, enc, callback) {
      cacheEntry += chunk; callback(chunk);
    },
    flush(callback) {
      redis.set(req.path, cacheEntry);
    }
  })
  .pipe(res);
like-fill
1
  • reply
  • like

Going to try that right away!

like-fill
1
  • reply
  • like

Apparently that specific code will have UTF-8 issues according to resident Node wizard @mafintosh, awaiting further information before proceeding

like-fill
1
  • reply
  • like

February 7, 2018 at 7:30pm

Thanks for the blog post Max!!

  • reply
  • like

Very welcome, glad you liked it!

  • reply
  • like
  • reply
  • like
Your message here...

*bold*_italic_`code````codeblock```