...
Follow the instructions on Access to infrastructure on clusterFLARECAST cluster (sshuttle) to make sure you have access to the host "cluster-r730-1".
...
| Code Block | ||
|---|---|---|
| ||
# download oneall file files of an entry def download_filefiles(urlurls): for url in urls: r = requests.get(url, stream=True) dispo = r.headers['content-disposition'] file_name = re.findall("filename=(.+)", dispo)[0] print("downloading %s ..." % file_name) if r.status_code == 200: with open(file_name, 'wb') as f: for chunk in r.iter_content(1024): f.write(chunk) return file_name |
With this method the download of all SHARP files is now a one-liner:
| Code Block | ||
|---|---|---|
| ||
# download everyall filefiles sharp_files = map(lambda s: download_filefiles(s['urlurls']), sharp_list) |
The sharp_files variable now contains the name to each of the files.
...