Node.js Streams: A Practical Guide

Streams handle large data efficiently. Here's how to use them without the headache.

Why Streams

Without streams:

const data = fs.readFileSync('huge-file.csv'); // Loads entire file into memory
process(data); // Memory spike!

With streams:

fs.createReadStream('huge-file.csv')
  .pipe(transform)
  .pipe(destination); // Processes chunks, constant memory

Stream Types

• **Readable** - Data source (files, HTTP requests)
• **Writable** - Data destination (files, HTTP responses)
• **Transform** - Modify data as it flows
• **Duplex** - Both readable and writable

Reading Files

const fs = require('fs');

const readStream = fs.createReadStream('large-file.txt', {
  encoding: 'utf8',
  highWaterMark: 64 * 1024 // 64KB chunks
});

readStream.on('data', (chunk) => {
  console.log(`Received ${chunk.length} bytes`);
});

readStream.on('end', () => {
  console.log('Done reading');
});

readStream.on('error', (err) => {
  console.error('Error:', err);
});

Writing Files

const writeStream = fs.createWriteStream('output.txt');

writeStream.write('Hello ');
writeStream.write('World');
writeStream.end('!');

writeStream.on('finish', () => {
  console.log('Done writing');
});

Piping

Connect streams together:

const fs = require('fs');
const zlib = require('zlib');

// Compress a file
fs.createReadStream('input.txt')
  .pipe(zlib.createGzip())
  .pipe(fs.createWriteStream('input.txt.gz'));

Transform Streams

Modify data in flight:

const { Transform } = require('stream');

const upperCase = new Transform({
  transform(chunk, encoding, callback) {
    this.push(chunk.toString().toUpperCase());
    callback();
  }
});

process.stdin
  .pipe(upperCase)
  .pipe(process.stdout);

HTTP with Streams

const http = require('http');
const fs = require('fs');

http.createServer((req, res) => {
  // Stream file to response
  const stream = fs.createReadStream('video.mp4');
  stream.pipe(res);
}).listen(3000);

No memory issues even with huge files.

Processing CSV

const fs = require('fs');
const readline = require('readline');

const rl = readline.createInterface({
  input: fs.createReadStream('data.csv'),
  crlfDelay: Infinity
});

rl.on('line', (line) => {
  const [name, email] = line.split(',');
  // Process each line
});

Error Handling

Always handle errors:

readStream
  .on('error', handleError)
  .pipe(transform)
  .on('error', handleError)
  .pipe(writeStream)
  .on('error', handleError);

Or use `pipeline` (handles errors and cleanup):

const { pipeline } = require('stream/promises');

await pipeline(
  fs.createReadStream('input.txt'),
  transform,
  fs.createWriteStream('output.txt')
);

When to Use Streams

• Files larger than available memory
• Real-time data processing
• Network I/O
• Video/audio processing

For small data, regular methods are fine. Streams shine at scale.

Back to Journal