173 - duplicate datasets

Created by: ri-pandey

Description

Allow duplicate datasets to be ingested in Bioloop, and for authorized users to be able to accept/reject them.

Related Issue(s)

Closes #173

Changes Made

List the main changes made in this PR. Be as specific as possible.

• Feature added
• Bug fixed
• Code refactored
• Documentation updated

Checklist

Before submitting this PR, please make sure that:

• Your code passes linting and coding style checks.
• Documentation has been updated to reflect the changes.
• You have reviewed your own code and resolved any merge conflicts.
• You have requested a review from at least one team member.
• Any relevant issue(s) have been linked to this PR.

Additional Information Documentation of the process - https://github.com/IUSCA/bioloop/blob/173-duplicates/docs/dataset_duplication.md.

Summary of features:

1. Bioloop can now register duplicate datasets for a given dataset.
2. Multiple duplicate datasets can coexist in the system, with versions being assigned to concurrent duplicates (this feature is disabled at the moment).
3. Alerts are shown regarding state of a dataset in case it has a duplicate, or is a duplicate. This is done in 3 places:
   1. project dataset modal
      1. project dataset table
      2. dataset page
   2. file browser
4. Operators and admins are shown notifications to make them aware of the duplication, and buttons to accept/reject datasets in alerts. Users only see alerts.
5. Both the API and the worker layers are involved in accepting or rejecting a duplicate. For detailed explanation of steps, see the linked documentation.
6. During acceptance/rejection, state locks are placed on the original and/or duplicate datasets before they are accepted/rejected, and the locks are released once the acceptance/rejection steps are successful.
7. State locks ensure that
   1. Workflows can’t be kicked off
      1. pending workflows will start failing due to API locks
   2. Worker scripts that try to write to dataset will fail
   3. Dataset information can’t be written to via API
   4. Dataset's state can’t be modified via API when acceptance/rejection is in progress
   5. Acceptance and Rejection can’t be initiated on a dataset at the same time
   6. UI controls are disabled when a dataset is being accepted/rejected.
8. Checks have been added to the API layer so that certain operations are forbidden for duplicate datasets (like adding them to a project). The API/UI code has also been updated to omit duplicate datasets from the existing UI controls that show datasets (like the Project-dataset search).
9. At the end of the acceptance workflow, the original dataset's filesystem resources are purged, and the duplicate dataset replaces the original dataset at the database level and filesystem level. The original dataset is (soft-) deleted in the database.
10. At the end of the rejection workflow, the duplicate dataset is (soft-) deleted in the database, and its filesystem resources are purged.
11. Acceptance/rejection are irreversible operations. Users see a modal to confirm before they accept/reject a dataset.
12. After a file downloaded is initiated, the dataset is re-fetched (without page refresh), so user can be made aware of the fact that a duplicate of the dataset that they are downloading is currently being integrated by the system.
13. Duplicate datasets can be seen in the /duplicateDatasets view