How to Stream Large Files in Node.js Safely

A file-processing service can run out of memory if it reads a large file into one buffer before doing any work. Node.js streams let you process data in chunks instead, while pipeline connects the streams, applies backpressure, and reports failures. This tutorial shows a safer file-copy pattern that removes incomplete output when something goes wrong.

Key Takeaways

• readFile buffers the whole file; streams let a pipeline process it in chunks.
• Use pipeline from node:stream/promises to connect streams and receive errors as promise rejections.
• Backpressure slows a readable source when a downstream stream cannot keep up, helping control buffering.
• Write to a temporary file and rename it only after the pipeline succeeds; remove the temporary file on failure.
• Streaming reduces whole-file buffering, but it does not impose a strict memory limit if transforms or application code buffer data.

Why stream a large file?

readFile is convenient when a file is small enough to fit comfortably in memory and the application needs all its contents at once. For example:

import { readFile, writeFile } from 'node:fs/promises';

const data = await readFile('input.bin');
await writeFile('output.bin', data);

This holds the file contents in a Buffer while the operation runs. Concurrent requests can multiply that memory use. With a stream pipeline, data moves through the process in chunks rather than requiring one buffer for the entire file.

Streaming is a good fit for copying files, parsing large logs, hashing data, or passing data through a transform. If your task genuinely requires the complete contents at once, streaming alone cannot remove that requirement.

Copy a file with pipeline

The example below uses Node.js built-in modules and targets Node.js 20 or later. Save it as copy-large-file.mjs; run it with node copy-large-file.mjs input.bin output.bin. The output directory must already exist, and use a distinct output path rather than the input path.

import { createReadStream, createWriteStream } from 'node:fs';
import { rename, unlink } from 'node:fs/promises';
import { basename, dirname, resolve } from 'node:path';
import { randomUUID } from 'node:crypto';
import { pipeline } from 'node:stream/promises';

async function copyLargeFile(inputPath, outputPath) {
  const source = resolve(inputPath);
  const destination = resolve(outputPath);

  if (source === destination) {
    throw new Error('Input and output paths must be different');
  }

  // Keep the temporary file beside the destination so rename stays on the
  // same filesystem in normal directory setups.
  const temporaryPath = resolve(
    dirname(destination),
    `.${basename(destination)}.${randomUUID()}.tmp`,
  );

  try {
    await pipeline(
      createReadStream(source),
      createWriteStream(temporaryPath, { flags: 'wx' }),
    );

    // Publish the output only after the stream pipeline finishes successfully.
    await rename(temporaryPath, destination);
  } catch (error) {
    // pipeline destroys its connected streams on failure. Remove any partial
    // temporary output, but preserve the original error if cleanup also fails.
    try {
      await unlink(temporaryPath);
    } catch (cleanupError) {
      if (cleanupError.code !== 'ENOENT') {
        console.error('Could not remove temporary file:', cleanupError);
      }
    }

    throw error;
  }
}

const [inputPath, outputPath] = process.argv.slice(2);

if (!inputPath || !outputPath) {
  console.error('Usage: node copy-large-file.mjs <input> <output>');
  process.exitCode = 2;
} else {
  try {
    await copyLargeFile(inputPath, outputPath);
    console.log('Copy completed');
  } catch (error) {
    console.error('Copy failed:', error);
    process.exitCode = 1;
  }
}

pipeline connects the readable and writable streams, waits for them to finish, and rejects if a stream errors. That means the caller can handle failures with try/catch rather than relying on separate error handlers for each stream. On failure, pipeline also destroys the streams it connected, allowing their file resources to close.

The destination is written under a unique temporary name first. A failed copy therefore does not leave a partial file at the final destination path. The catch block attempts to remove the temporary file and rethrows the original failure. A process killed abruptly may not run that cleanup, so services handling such interruptions may also need a policy for finding and removing stale temporary files.

How backpressure helps

A readable stream can produce data faster than a destination or transform can handle it. Backpressure communicates that the downstream stream needs the source to slow down. pipeline manages this flow between the connected streams, so the producer does not have to keep writing unbounded data into a slower consumer.

This is not a promise that memory usage is fixed or independent of all inputs. Streams buffer chunks, and transforms may keep additional data in memory. A transform that collects every chunk into an array, for example, defeats the benefit. The highWaterMark option influences a stream's buffering threshold; it is not a strict cap on total process memory.

To add processing, place a transform between the source and destination. For example, compression can be connected with Node.js's createGzip from node:zlib:

import { createReadStream, createWriteStream } from 'node:fs';
import { createGzip } from 'node:zlib';
import { pipeline } from 'node:stream/promises';

await pipeline(
  createReadStream('archive.log'),
  createGzip(),
  createWriteStream('archive.log.gz'),
);

This short example demonstrates stream composition; for a service that must preserve an existing output on failure, apply the temporary-file and cleanup pattern from the full example to the compressed destination too.

Handling failures and output files

A stream can fail for ordinary reasons: the input may not exist, permissions may deny access, the disk may fill up, or a downstream transform may throw an error. Awaiting pipeline makes those failures visible to the surrounding try/catch.

The temporary-file pattern helps avoid publishing an incomplete result, but a few operational details still matter:

• Destination behavior: rename behavior when the destination already exists can vary by operating system. Decide whether replacement is acceptable for your service and test that policy on the deployment platform.
• Temporary files: The example cleans up after caught failures. Abrupt termination can leave a temporary file behind; use a safe age-based cleanup policy if that matters to your service.
• Path safety: Resolving paths and checking whether the two path strings match is a basic guard, not a complete defense against aliases, symlinks, or concurrent filesystem changes. For untrusted paths, enforce an application-specific directory policy and appropriate filesystem permissions.
• Resource limits: Streaming does not replace request-size limits, timeouts, concurrency controls, or disk-space monitoring in a server.

Common Mistakes

• Using readFile for files that can exceed the service's memory budget. Prefer streaming when the operation can work incrementally.
• Manually wiring data events and ignoring backpressure. A consumer that writes faster than its destination can accept data may accumulate buffers. Use pipeline for connected streams.
• Assuming highWaterMark is a hard memory limit. It affects stream buffering, not every buffer a transform or application might allocate.
• Writing directly to the final destination. A failure can leave a partial output there. Write to a temporary file and publish only after success when that distinction matters.
• Ignoring cleanup failures. Cleanup can fail too, for example because of filesystem permissions. Log or otherwise report that failure without hiding the original processing error.
• Treating a stream as a complete resource-control strategy. Bound concurrent jobs and account for disk use and transform behavior as well as memory.

Best Practices

• Prefer pipeline when connecting readable, transform, and writable streams; await it so errors reach the caller.
• Keep each processing step incremental. Avoid collecting all chunks unless the data size is explicitly bounded.
• Use a unique temporary output and clean it up when processing fails.
• Define what should happen when the final destination already exists, and verify that behavior in your target environment.
• Test failure paths, including a missing input and an unwritable or full destination filesystem, in an isolated environment.
• Monitor memory, disk space, and concurrent work in the service; streaming reduces whole-file buffering but cannot prevent every resource-exhaustion problem.

Conclusion

For large files, a Node.js stream pipeline avoids loading the entire file into a single buffer and lets backpressure regulate data flow. Pair pipeline with explicit failure handling and temporary-output cleanup to make file processing more resilient.

References

• Node.js Streams API — stream concepts, pipeline, and stream behavior.
• Node.js File system API — file streams such as createReadStream and createWriteStream.
• Node.js File system promises API — promise-based filesystem operations, including rename and unlink.