In case the database crashes you'd have to follow these steps.
- First check if there is a recent backup under /home/marco/postgres.
- Use /home/marco/postgres/bin/restore_database_pg11.sh to bring back the data. (will take up to an hour)
- disable cronjob to restart platform in cluster-head>crontab -e
- Manually ingest missing SWPC data
- use /home/marco/git/sandbox/sam_sandbox/parallelscheduler/ to add missing features
- modify params.json (is a symbolic link, you may want to adjust it. 3hour cadence may be fine, set proper time range)
- open a "screen" to run in background.
- python batch_run.py (will take a couple of hours to days)
- close screen (ctrl-a d, use screen -r to reattach)
- make sure to disable autorestart in crontab on cluster-head.
- use /home/marco/bin/all_ml_batchjob.py to ingest missing predictions
- adjust /home/marco/bin/cfg/*
- python all_ml_batchjob.py cfg/
- reenable cronjob in cluster-head>crontab -e