Make workflow steps involding bundles agnostic of worker node
Created by: ri-pandey
In tickets #102 (closed) (and follow-up ticket #205) we introduced the concept of bundles and bundle downloads, with the bundle's metadata being persisted in the added bundle
table.
The archive, stage and download steps were rewritten to facilitate this change. The rewrite introduced a dedicated path for storing bundles, which is where the archive steps creates the bundle, and the stage/download steps try to read the bundle from.
However, on Bioloop instances with multiple worker instances, it's possible for the archive and stage steps to run on different machines, which would result in the archive step generating the bundle on a different machine than the one that the stage step reads the bundle directory from.
The process should be rewritten, so that the archive step doesn't persist a path in the bundle
table. The stage/download steps should know the expected path where the bundle is to be found through env-dependent config.