Failed to pipe response after upgrade next.js to v15.1.2 (from v14.x.x)

[studentIvan]

Summary

Site looks working well, nothing fall from quick look, but in the AWS console there are a lot of spammy errors with no any information:

15:43:02 ⨯ [Error: failed to pipe response] {
15:43:02 [cause]: [TypeError: readable.pipeTo is not a function]
15:43:02 }
15:43:06 ⨯ [Error: failed to pipe response] {
15:43:06 [cause]: [TypeError: readable.pipeTo is not a function]
15:43:06 }

Anybody can help me with how to debug it or at least what is the direction I should seek this problem?

Additional information

node v18.18.0
next v15.1.2
running on aws (flightcontrol)
using redis custom cache handler
NO api using
custom middleware is using

using run command: yarn next start --keepAliveTimeout 70000 -p 8080

Example

No response

[vitaliemiron]

same question

[justinadkins]

@studentIvan Do you happen to be using a patched version of https://github.com/caching-tools/next-shared-cache? I was seeing this same issue and I believe it was resolved with this change.

[studentIvan]

nope it doesn't solve the issue :( see the next15 pr topic in next-shared-cache
the reason is in the data weight

[erwanriou]

Right, but there is no release of the new version yet right? Do you know when this PR will be merged?

[justinadkins]

I do not, I've been running a patched version and addressing issues as they come up in my use case.

[erwanriou]

Hmm then i will wait. I will migrate to 15 once we have this PR done. I guess there is still a lot of backward compatibility to cover.

[AyronK]

This is related to string - buffer breaking change in Next15. We have made a workaround available to the community https://github.com/fortedigital/nextjs-cache-handler/tree/1.1.4?tab=readme-ov-file#3-buffer-string-decorator

@neshca/cache-handler does not officially support Next 15 yet, there is a Pull Request in progress.

[omavi]

There are a few more properties that are of type Buffer: https://github.com/vercel/next.js/blob/f5444a16ec2ef7b82d30048890b613aa3865c1f1/packages/next/src/server/response-cache/types.ts

I updated @AyronK's approach to handle the extra cases: https://gist.github.com/omavi/5527110cc6b664eefccd33fbb3f4aa20

That function solved my issues. Thanks for the inspiration @AyronK!

[AyronK]

FYI our package supports it already https://github.com/fortedigital/nextjs-cache-handler/releases/tag/1.2.0.
Thank you @omavi!

[omavi]

Oh wow, that was fast. Of course and I appreciate the attribution 🙂

[erwanriou]

@reviskar I tested it and so far, i didn't had a single pipeTo error (i let it run one more week in staging with the e2e testing and let see).
This is the configuration i used based on the work from @AyronK

// @neshca/cache-handler DEPENDENCIES
const { CacheHandler } = require("@neshca/cache-handler")
const createLruHandler = require("@neshca/cache-handler/local-lru").default

// Next/Redis DEPENDENCIES
const { createClient } = require("redis")
const { PHASE_PRODUCTION_BUILD } = require("next/constants")

// @fortedigital/nextjs-cache-handler DEPENDENCIES
const createCompositeHandler = require("@fortedigital/nextjs-cache-handler/composite").default
const createRedisHandler = require("@fortedigital/nextjs-cache-handler/redis-strings").default
const createBufferStringHandler = require("@fortedigital/nextjs-cache-handler/buffer-string-decorator").default
const { Next15CacheHandler } = require("@fortedigital/nextjs-cache-handler/next-15-cache-handler")

CacheHandler.onCreation(() => {
  // CONDITIONALS
  if (global.cacheHandlerConfig) return global.cacheHandlerConfig
  if (global.cacheHandlerConfigPromise) return global.cacheHandlerConfigPromise
  if (process.env.NODE_ENV === "development") {
    const lruCache = createLruHandler()
    return { handlers: [lruCache] }
  }

  // MAIN PROMISE
  global.cacheHandlerConfigPromise = (async () => {
    let redisClient = null
    if (PHASE_PRODUCTION_BUILD !== process.env.NEXT_PHASE) {
      try {
        redisClient = createClient({ url: process.env.REDIS_URL, pingInterval: 10000 })
        redisClient.on("error", e => {
          console.warn("Redis error", e)
          global.cacheHandlerConfig = null
          global.cacheHandlerConfigPromise = null
        })
      } catch (error) {
        console.warn("Failed to create Redis client:", error)
      }
    }

    if (redisClient) {
      try {
        console.info("Connecting Redis client...")
        await redisClient.connect()
        console.info("Redis client connected.")
      } catch (error) {
        console.warn("Failed to connect Redis client:", error)
        await redisClient.disconnect().catch(() => console.warn("Failed to quit the Redis client after failing to connect."))
      }
    }
    const lruCache = createLruHandler()

    if (!redisClient?.isReady) {
      console.error("Failed to initialize caching layer.")
      global.cacheHandlerConfigPromise = null
      global.cacheHandlerConfig = { handlers: [lruCache] }
      return global.cacheHandlerConfig
    }

    const redisCacheHandler = createRedisHandler({
      client: redisClient,
      keyPrefix: `ARCHSPLACE_MAIN_${process.env.GIT_HASH}` ?? process.env.NODE_ENV,
      timeoutMs: 1000,
      revalidateTagQuerySize: 100
    })

    global.cacheHandlerConfigPromise = null
    global.cacheHandlerConfig = {
      handlers: [
        createCompositeHandler({
          handlers: [lruCache, createBufferStringHandler(redisCacheHandler)],
          setStrategy: ctx => (ctx?.tags.includes("memory-cache") ? 0 : 1)
        })
      ]
    }

    return global.cacheHandlerConfig
  })()

  return global.cacheHandlerConfigPromise
})

module.exports = new Next15CacheHandler()

I literraly copied the config from the doc, changed a bit on the keyPrefix and make sure to --force the neshca package when switching to 15.3.0

Logs are cleans and website seems performant.

[AyronK]

FYI we've been using our package on production with Next15+ on quite busy ecommerce since January.

[ttaccp]

get(key) {
......

// Redis stores strings, so we need to parse the JSON.
const cacheValue = JSON.parse(result)

// This is the most important
if (cacheValue.value?.rscData?.type === 'Buffer') {
  cacheValue.value.rscData = Buffer.from(cacheValue.value.rscData.data)
}

......
return cacheValue
}

[ttaccp]

rscData is a Buffer, so we need to convert it to a Buffer objects