This tutorial will show how to download HMI SHARP data from France to use it in the algorithms.
Samples are written in Python but can be adapted to IDL by following Access to REST-Services in IDL
Prepare
Open sshuttle connection to host "cluster-r730-1" (see Access FLARECAST cluster (sshuttle)).
Open http://cluster-r730-1:8001/ui in a browser.
To download sharp_720s images over a period of time you have to set the series parameter and the start and end date. After trying it out you should see the request url which you can use in your python code:
http://cluster-r730-1:8001/HMI/sharp_720s?start=2015-12-11T00%3A00%3A00Z&end=2015-12-12T00%3A00%3A00Z
The result of this query will be a list of SHARP files with a link to them to download it:
[
{
"date__obs": "2016-01-19T01:10:08.400000Z",
"download_time": null,
"exptime": null,
"mask_cadence": null,
"nbr_update": 0,
"quality": 1024,
"recnum": 1958485,
"recnum_init": null,
"series_name": "hmi.sharp_cea_720s_nrt",
"t_rec_index": 1010166,
"time": "2016-01-20T01:35:28.643542Z",
"type": null,
"urls": [
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_conf_disambig.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Bt_err.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Br.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Br_err.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_magnetogram.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Bp_err.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_bitmap.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Bp.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Dopplergram.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_Bt.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958485_20160119011008_continuum.fits"
],
"wavelnth": 6173
},
{
"date__obs": "2016-01-19T01:10:08.400000Z",
"download_time": null,
"exptime": null,
"mask_cadence": null,
"nbr_update": 0,
"quality": 1024,
"recnum": 1958492,
"recnum_init": null,
"series_name": "hmi.sharp_cea_720s_nrt",
"t_rec_index": 1010166,
"time": "2016-01-20T01:35:30.847712Z",
"type": null,
"urls": [
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_conf_disambig.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Bt_err.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Br.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Br_err.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_magnetogram.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Bp_err.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_bitmap.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Bp.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Dopplergram.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_Bt.fits",
"http://cluster-r730-1:8001/HMI/sharp_cea_720s_nrt/1958492_20160119011008_continuum.fits"
],
"wavelnth": 6173
}
]
Loading Metadata
We recommend the python library requests to interact with the rest interfaces from python.
To get a list of SHARP images in this time range you have to create a GET request to the url we tried out before. For a better overview we recommend to separate the parameters of the query into variables.
# define the query parameter
start_date = '2015-12-11T00:00:00Z'
end_date = '2015-12-12T00:00:00Z'
# retrieving list of SHARP images
print('loading SHARP list...')
sharp_list = requests.get("http://cluster-r730-1:8001/HMI/sharp_720s?start=%s&end=%s" % (start_date, end_date)).json()
This list contains the url and the date of the specific SHARP image.
Downloading Images
With the metadata it is now possible to download the actual images. To do this we have written a tiny function which needs a url as parameter and downloads the file. The image name will be obtained from the header of the http response. The files are written in to the same folder where the script is.
# download all files of an entry
def download_files(urls):
for url in urls:
r = requests.get(url, stream=True)
dispo = r.headers['content-disposition']
file_name = re.findall("filename=(.+)", dispo)[0]
print("downloading %s ..." % file_name)
if r.status_code == 200:
with open(file_name, 'wb') as f:
for chunk in r.iter_content(1024):
f.write(chunk)
return file_name
With this method the download of all SHARP files is now a one-liner:
# download all files sharp_files = map(lambda s: download_files(s['urls']), sharp_list)
The sharp_files variable now contains the name to each of the files.
Script
Here you can download the complete script.