Cover Image for Big file upload

Big file upload

nodejsreact

Big file upload

File upload has two features in our implementation.

  1. File upload is done in chunks. Size of the chunk is determined by frontend.
  2. If file upload is cancelled by some reason from frontend, next time the file is uploaded it will be resumed from the same byte.

Execution

uniqueFileId:

We declared a global object named uploads in the upload file. This object will keep track of the uploaded bytes that have been uploaded for a particular file. We identify every file with a unique file name (passed by the frontend) i.e. <file-size>-<user-id>-<original-file-name> . The uploads object has the following interface.

interface IUploads {
  uniqueFileId: {
    bytesReceived: number,
  };
}

If uniqueFileId is not found in the uploads object then it means either the user is uploading a new file or file has already been uploaded.

already uploaded file

We store the uploaded files in the uploads/ folder. Before user starts uploading the file, we check this folder if the file is already been uploaded.

Three api routes are made for the upload process(used sequencially):

1. /api/upload/status

Headers passed x-file-name, file-size

Check the status of file before upload. We check for already uploaded file, resumed upload and new file.

1. If file exists in the uploads/ directory that means the file has already been completely uploaded.
2. If uniqueFileId property exists in uploads object then that means the file is being resumed and in that case we return the number of bytes that have been uploaded in response to frontend.
3. If above both cases don't pass then that means we are uploading a new file and we let frontend know in response

2. /api/upload/files

Headers passed x-file-name, content-range, file-size

In this route we upload the chunk sent from frontend to backend. The upload is done by using a fileStream. If user is uploading a new file then we create a writable file stream and if it's not then we create a appendable file stream in the tmp/ directory. content-range header is used to let the backend know from which byte it should start writing or appending. After successful write or append we update the number of bytesReceived in the uploads object.

3. /api/upload/complete

Headers passed x-file-name, x-orig-file-name

This api lets backend do post-upload actions.

1. We check the format of the original file name. Original file name format is {vehicle name}_{timestamp:HHMMSS}_{date:YYMMDD}.dat // 1*115431_210224.dat
2. If file name is correct then we create a drive record and attach the respective vehicle with it. Otherwise we simply create a drive record.
3. We remove the uniqueFileId reference from the uploads object to mark the file upload complete.
4. At the end we move the uploaded file from tmp/ directory to uploads/ directory



About the author

Ameer Hamza
Ameer Hamza
Software Developer

Software engineer at Nerdyfactory. Node.js, Golang, Postgres, Javascript, React