Implement Scheduled Jobs for Workflows and Tasks Cleanup
Created by: deepakduggirala
We need to implement cron jobs that will clean up our MongoDB instance. Specifically, the scheduled jobs (using pm2) should perform the following tasks:
-
Remove Unreferenced Workflows: The cron job should remove any workflows that are no longer referenced in the application database. This will help to prevent our database from becoming cluttered with unused data.
-
Clean Up Orphaned Tasks: The cron job should also clean up any tasks that are orphaned, i.e., tasks whose
kwargs.workflow_id
no longer exists in the database. This will ensure that our task data remains consistent and relevant.
This cleanup process should be automated and run at regular intervals to ensure our database remains clean and efficient.
These scripts would be placed in workers/workers/scripts and run with pm2. The second script may belong with rhythm_api as it has no connection with a particular app.