Skip to content

Rewrite bundle population and download logic

ryanlong requested to merge 201-rewrite-bundle-population into main

Created by: ri-pandey

Description

Rewrote the logic that populates bundle metadata and prepares bundles for download.

Related Issue(s)

Closes #201 (closed)

Changes Made

List the main changes made in this PR. Be as specific as possible.

  • Feature added
  • Bug fixed
  • Code refactored
  • Documentation updated
  • Other changes: [describe]

Checklist

Before submitting this PR, please make sure that:

  • Your code passes linting and coding style checks.
  • Documentation has been updated to reflect the changes.
  • You have reviewed your own code and resolved any merge conflicts.
  • You have requested a review from at least one team member.
  • Any relevant issue(s) have been linked to this PR.

Additional Information

The old logic downloaded bundles from the SDA to populate it's metadata. The new logic populates bundle metadata by reading it from the dataset.bundle_size column, and downloading bundles from SDA is no longer needed. This way the bundles can be prepared for download much faster.

After the bundles are prepared for download in the CPA release, I will be doing a second release immediately after that removes the bundle_size column.

Merge request reports