Streaming — katta fayllarni stream qilish
Muammodan boshlaylik
Serverda 2 GB li video fayl bor. Foydalanuvchi uni yuklab olmoqchi. Shoshib qolsangiz shunday yozasiz:
app.get('/video', (req, res) => {
const data = fs.readFileSync('video.mp4'); // ← 2 GB xotiraga yuklanadi
res.send(data);
});
Nima bo'ladi? Server 2 GB RAM sarflaydi. 10 ta parallel so'rov — 20 GB. Server crash bo'ladi. Kichik fayllarda ham muammo: foydalanuvchi birinchi baytni faqat butun fayl yuklangandan keyin oladi.
Stream bilan fayl bo'laklarga bo'linib, birin-ketin yuboriladi — xotira muammosi yo'q, foydalanuvchi darhol ko'ra boshlaydi.
Stream nima?
Stream — ma'lumotni to'liq kutmasdan, bo'laklap (chunk) qayta ishlash usuli.
Hayotiy o'xshatma
Suv quvuri vs. chelak. Vannani to'ldirmoqchisiz:
- Chelak (buffer): manba suvni chelakka to'ldiradi, to'lsa — olib boradi. Manba uzoq bo'lsa — uzoq kutasiz, chelak og'ir bo'ladi.
- Quvur (stream): suv manbadan to'g'ridan-to'g'RI vannaga oqadi. Darhol boshlanadi, tinimsiz oqadi.
Buffer: [===========fayl to'liq yuklanadi===========] → yuboriladi
Stream: [chunk1]→ [chunk2]→ [chunk3]→ [chunk4]→ ...
Oddiy fayl stream
const express = require('express');
const fs = require('fs');
const path = require('path');
const app = express();
// ✅ Stream bilan — xotira samarali
app.get('/fayl/:nom', (req, res) => {
const filePath = path.join(__dirname, 'fayllar', req.params.nom);
// Fayl mavjudligini tekshirish
if (!fs.existsSync(filePath)) {
return res.status(404).json({ error: 'Fayl topilmadi' });
}
const stream = fs.createReadStream(filePath); // ← stream yaratish
stream.pipe(res); // ← response ga ulash
});
pipe nima qiladi? Read stream dan kelgan har bir chunk ni response ga yozadi. Fayl tugaganda response yopiladi. Xotiraga hech narsa yuklanmaydi.
Video streaming — Range so'rovlari
Brauzer video o'ynatganda faqat kerakli qismini so'raydi. Masalan, foydalanuvchi 10 daqiqali videoda 7 daqiqaga o'tib ketsa — brauzer boshidan yuklaydi deb o'ylash xato. U Range: bytes=85000000- sarlavhasi bilan faqat shu qismini so'raydi.
app.get('/video/:nom', (req, res) => {
const filePath = path.join(__dirname, 'videolar', req.params.nom);
if (!fs.existsSync(filePath)) {
return res.status(404).send('Topilmadi');
}
const stat = fs.statSync(filePath);
const fileSize = stat.size; // ← fayl hajmi (bayt)
const range = req.headers.range; // ← "bytes=0-999999"
if (range) {
// Range so'rovi — faqat bir qismini yuborish
const [startStr, endStr] = range.replace('bytes=', '').split('-');
const start = parseInt(startStr, 10);
const end = endStr ? parseInt(endStr, 10) : fileSize - 1;
const chunkSize = end - start + 1;
res.writeHead(206, { // ← 206 Partial Content
'Content-Range': `bytes ${start}-${end}/${fileSize}`,
'Accept-Ranges': 'bytes',
'Content-Length': chunkSize,
'Content-Type': 'video/mp4',
});
fs.createReadStream(filePath, { start, end }).pipe(res); // ← faqat shu qism
} else {
// Range yo'q — to'liq fayl
res.writeHead(200, {
'Content-Length': fileSize,
'Content-Type': 'video/mp4',
});
fs.createReadStream(filePath).pipe(res);
}
});
Nima uchun 206 Partial Content? Bu HTTP standarti — brauzer "qisman javob" ekanligini tushunadi va video o'ynatgich to'g'ri ishlaydi. 200 qaytarilsa — brauzer butun faylni kutishi mumkin.
Frontend da:
<!-- Brauzer Range so'rovlarini o'zi boshqaradi -->
<video controls>
<source src="/video/dars-01.mp4" type="video/mp4">
</video>
S3 dan stream qilish
S3 dagi faylni serverga yuklamasdan to'g'ridan-to'g'RI foydalanuvchiga stream qilish:
const { GetObjectCommand } = require('@aws-sdk/client-s3');
const s3 = require('./config/s3');
app.get('/s3-fayl/:key', async (req, res) => {
const command = new GetObjectCommand({
Bucket: process.env.AWS_S3_BUCKET,
Key: req.params.key,
});
const response = await s3.send(command);
// S3 dan kelgan stream ni response ga ulash
res.setHeader('Content-Type', response.ContentType);
res.setHeader('Content-Length', response.ContentLength);
response.Body.pipe(res); // ← S3 stream → response
});
Nima uchun bu muhim? Server S3 dan faylni xotiraga yuklamasdan, to'g'ridan-to'g'RI foydalanuvchiga uzatadi. 1 GB fayl uchun server faqat bir necha MB RAM sarflaydi.
Transform stream — o'qish jarayonida o'zgartirish
Ba'zan faylni yuborish davomida qayta ishlash kerak. Masalan: katta CSV faylni o'qib, JSON ga o'tkazib yuborish:
const { Transform } = require('stream');
app.get('/csv-json', (req, res) => {
res.setHeader('Content-Type', 'application/json');
let isFirst = true;
let headers = [];
res.write('['); // ← JSON massiv boshlanishi
const csvTransform = new Transform({
objectMode: true,
transform(line, encoding, callback) {
const row = line.toString().trim();
if (!row) return callback();
const cols = row.split(',');
if (isFirst) {
headers = cols; // ← birinchi qator — sarlavhalar
isFirst = false;
return callback();
}
const obj = {};
headers.forEach((h, i) => { obj[h] = cols[i]; });
this.push(JSON.stringify(obj));
callback();
}
});
let lineCount = 0;
fs.createReadStream('malumotlar.csv')
.pipe(require('readline').createInterface({ input: process.stdin }))
// Oddiyroq yondashuv — line by line:
.on('line', (line) => {
// Har qatorni qayta ishlash
});
// Oddiyroq misol — gzip bilan siqib yuborish:
const zlib = require('zlib');
fs.createReadStream('katta-fayl.txt')
.pipe(zlib.createGzip()) // ← o'qish + siqish
.pipe(res); // ← siqilgan holda yuborish
res.setHeader('Content-Encoding', 'gzip');
});
Amaliyroq misol — gzip bilan:
const zlib = require('zlib');
app.get('/compressed-log', (req, res) => {
res.setHeader('Content-Encoding', 'gzip');
res.setHeader('Content-Type', 'text/plain');
fs.createReadStream('server.log')
.pipe(zlib.createGzip()) // ← stream da siqish
.pipe(res); // ← siqilgan holda yuborish
// Xotiraga hech narsa yuklanmaydi
});
Stream xatolarini ushlash
app.get('/fayl/:nom', (req, res) => {
const filePath = path.join(__dirname, 'fayllar', req.params.nom);
const stream = fs.createReadStream(filePath);
// ✅ Stream xatolarini ushlash
stream.on('error', (err) => {
if (err.code === 'ENOENT') {
res.status(404).json({ error: 'Fayl topilmadi' });
} else {
res.status(500).json({ error: 'Serverda xato' });
}
});
stream.pipe(res);
});
Nima uchun on('error') kerak? pipe xatolarni avtomatik ushlaydi deb o'ylash xato. Stream xatosi ushlanmasa — Node.js jarayoni crash bo'lishi mumkin (UnhandledStreamError).
Tez-tez uchraydigan xatolar
1. readFileSync bilan katta fayl o'qish
// ❌ — butun fayl xotiraga yuklanadi
const data = fs.readFileSync('video.mp4');
res.send(data);
// ✅ — stream bilan
fs.createReadStream('video.mp4').pipe(res);
Sababi: readFileSync sinxron — fayl to'liq xotiraga yuklanguncha boshqa hech narsa ishlamaydi. Katta fayllarda server butunlay muzlab qoladi.
2. Response tugagandan keyin yozishga urinish
// ❌
stream.pipe(res);
res.json({ message: 'Yuborildi' }); // ← stream allaqachon res ni boshqarmoqda
Sababi: pipe(res) response ni stream ga topshiradi. Keyin res.json() chaqirilsa — "Cannot set headers after they are sent" xatosi.
3. Range sarlavhasini tekshirmaslik
// ❌ — range bo'lmasa ham range kodi ishlaydi
const [start, end] = req.headers.range.replace('bytes=', '').split('-');
// ↑ req.headers.range undefined bo'lsa — crash
// ✅ — avval tekshirish
const range = req.headers.range;
if (range) {
// range bilan ishlash
} else {
// to'liq fayl
}
4. Path traversal — xavfsizlik zaiflik
// ❌ — foydalanuvchi ../../etc/passwd yuborishi mumkin
app.get('/fayl/:nom', (req, res) => {
const filePath = path.join(__dirname, 'fayllar', req.params.nom);
fs.createReadStream(filePath).pipe(res);
});
// ✅ — path resolve qilib tekshirish
app.get('/fayl/:nom', (req, res) => {
const safeBase = path.resolve(__dirname, 'fayllar');
const filePath = path.resolve(safeBase, req.params.nom);
// path safeBase ichida ekanligini tekshirish
if (!filePath.startsWith(safeBase)) {
return res.status(403).json({ error: 'Ruxsat yo\'q' });
}
fs.createReadStream(filePath).pipe(res);
});
Sababi: ../../etc/passwd kabi yo'l bilan server tizimdagi istalgan faylni o'qish mumkin. path.resolve + startsWith tekshiruvi buni oldini oladi.
Xulosa
- Stream — faylni to'liq xotiraga yuklamasdan bo'laklap yuboradi
fs.createReadStream().pipe(res)— eng oddiy fayl stream- Video uchun Range so'rovlari —
206 Partial Contentbilan - S3 dan stream — server xotirasi sarflanmaydi
- Har doim
stream.on('error')qo'shing — ushlanmagan xato serverni crash qiladi - Path traversal — foydalanuvchi kiritgan fayl nomini
path.resolve+startsWithbilan tekshiring
Keyingi dars: 36-dars: REST API — qoidalar, best practices, versiyalash