Skip to content

99 data product uploads

ryanlong requested to merge 99-data-product-uploads into main

Created by: ri-pandey

Description

Please provide a brief description of the changes made in this PR.

Related Issue(s)

Closes #99 (closed)

If applicable, please reference the issue(s) that this PR addresses. If the PR does not address any specific issue, you can remove this section.

Changes Made

List the main changes made in this PR. Be as specific as possible.

  • Added upload functionality, with a form for metadata selection. Uploads are done in 2MB chunks.
  • Added two levels of checksum validation (per chunk, and post-file-creation)
  • Added worker to merge uploaded chunks into files, and initiate the Integrated workflow.
  • Added script that monitors pending uploads, and retries to process them. Uploads which have not been in COMPLETE status for more than 24 hours

Checklist

Before submitting this PR, please make sure that:

  • Your code passes linting and coding style checks.
  • Documentation has been updated to reflect the changes.
  • You have reviewed your own code and resolved any merge conflicts.
  • You have requested a review from at least one team member.
  • Any relevant issue(s) have been linked to this PR.

Additional Information

Add any other information or context that may be relevant to this PR. This can include potential impacts, known issues, or future work related to this change.

Merge request reports