WP4: Infrastructure
Space shortcuts
Page tree
Skip to end of metadata
Go to start of metadata

This tutorial will show how to download HMI SHARP data from France to use it in the algorithms.

Samples are written in Python but can be adapted to IDL by following Access to REST-Services in IDL

Prepare

Open sshuttle connection to host "cluster-r730-1" (see Access FLARECAST cluster (sshuttle)).

Open http://cluster-r730-1:8001/ui in a browser.

To download sharp_720s images over a period of time you have to set the series parameter and the start and end date. Optionally, you can request the available hmi-meta data by the corresponding meta parameter, which may requires some time. After trying it out you should see the request url which you can use in your python code:

http://cluster-r730-1:8001/HMI/hmi.sharp_720s?start=2015-12-11T00%3A00%3A00Z&end=2015-12-12T00%3A00%3A00Z&meta=false

The result of this query will be a list of SHARP files with a link to them to download it:

[
  {
    "date__obs": "2015-12-11T23:34:08.799999Z",
    "download_time": null,
    "exptime": null,
    "harpnum": 6178,
    "mask_cadence": null,
    "nbr_update": 0,
    "quality": 0,
    "recnum": 5605571,
    "recnum_init": null,
    "series_name": "hmi.sharp_720s",
    "t_rec_index": 1005598,
    "time": "2016-01-20T05:41:33.750935Z",
    "type": null,
    "urls": [
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/magnetogram.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/bitmap.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/Dopplergram.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/continuum.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/inclination.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/azimuth.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/field.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/vlos_mag.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/dop_width.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/eta_0.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/damping.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/src_continuum.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/src_grad.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/alpha_mag.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/chisq.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/conv_flag.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/info_map.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/confid_map.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/inclination_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/azimuth_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/field_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/vlos_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/alpha_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/field_inclination_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/field_az_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/inclin_azimuth_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/field_alpha_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/inclination_alpha_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/azimuth_alpha_err.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/disambig.fits",
      "http://cluster-r730-1:8001/HMI/hmi.sharp_720s/5605571/conf_disambig.fits"
    ],
    "wavelnth": 6173
  },
  ...
]

Loading Metadata

We recommend the python library requests to interact with the rest interfaces from python.

To get a list of SHARP images in this time range you have to create a GET request to the url we tried out before. For a better overview we recommend to separate the parameters of the query into variables.

# define the query parameter
start_date = '2015-12-11T00:00:00Z'
end_date = '2015-12-12T00:00:00Z'
incl_meta = 'false'

# retrieving list of SHARP images
print('loading SHARP list...')
response = requests.get("http://cluster-r730-1:8001/HMI/hmi.sharp_720s?start=%s&end=%s&meta=%s" % (start_date, end_date, incl_meta)).json()
sharp_list = response['data']

# extract specific properties of the first SHARP image
sharp_image = 0
first_sharp_date = sharp_list[sharp_image]['date__obs']
first_sharp_urls = sharp_list[sharp_image]['urls']

 

Given the above request url we could download all related hmi-meta data which is, however, not recommended. Resolving hmi-meta data requires some time whereas we may not be interessted in all meta data or all SHARP images. For this reason, we can reload specific meta data for a given set of images.

As an example, the above request reloads the area, cmask and distcoef meta data from the SHARP image 5601499. The corresponding url request looks like this:

http://localhost:8001/HMI/hmi.sharp_720s/meta/5601499?fields=area,cmask,distcoef

The result of this query is a list of header informations and meta data for each requested SHARP image:

{
  "5601499": {
    "date__obs": "2015-12-11T23:34:08.800000Z",
    "download_time": null,
    "exptime": null,
    "harpnum": 6167,
    "mask_cadence": null,
    "meta": {
      "area": 3507.3125,
      "cmask": 6423,
      "distcoef": "/home/jsoc/cvs/Development/JSOC/proj/lev1.5_hmi/apps//../libs/lev15/"
    },
    "nbr_update": 0,
    "quality": 0,
    "recnum": 5601499,
    "recnum_init": null,
    "series_name": "hmi.sharp_720s",
    "t_rec_index": 1005598,
    "time": "2016-05-26T15:39:51Z",
    "type": null,
    "urls": [],
    "wavelnth": 6173
  }
}

The same way as given in the first code example of this section we can now access those meta data using python and requests:

# define the query parameter
sharp_image = 5601499
fields = ['area', 'cmask', 'distcoef']

# retrieving list of meta data
print('loading meta data...')
response = requests.get("http://cluster-r730-1:8001/HMI/hmi.sharp_720s/meta/%s?fields=%s" % (str(sharp_image), ','.join(fields))).json()
meta_data = response['data']

Downloading Images

With the metadata it is now possible to download the actual images. To do this we have written a tiny function which needs a url as parameter and downloads the file. The image name will be obtained from the header of the http response. The files are written in to the same folder where the script is.

# download all files of an entry
def download_files(urls):
    for url in urls:
        r = requests.get(url, stream=True)
        dispo = r.headers['content-disposition']
        file_name = re.findall("filename=(.+)", dispo)[0]

        print("downloading %s ..." % file_name)

        if r.status_code == 200:
            with open(file_name, 'wb') as f:
                for chunk in r.iter_content(1024):
                    f.write(chunk)

    return file_name

With this method the download of all SHARP files is now a one-liner:

# download all files
sharp_files = map(lambda s: download_files(s['urls']), sharp_list)

The sharp_files variable now contains the name to each of the files. 

Script

Here you can download the complete script.

load_sharp.py

  • No labels

3 Comments

  1. Jordan Guerra

    Marco Soldati, samuelvonstachelski, Florian Bruggisser

    We noticed (Sung-Hong Park and I) few things regarding the HMI service website:

    1. When requesting one file for an specific date, the service did return one entry but this file does not specify to which SHARP region it corresponds.

    2. There is no specification of the specific measurement (segments) that such file contains - that is, the component of the magnetic field. For NRT CEA data it could be Br, Bp, Bt, magnetogram, bitmap, etc, for NRT Bx, By, Bz, magnetogram, bitmap, etc...

    3. Inside the FITS file (e.g. sharp_cea_720s_nrt_20160118_105808.fits) only the following information is contained in the headers:

     

    ** Structure <165a9ff8>, 13 tags, length=424, data length=416, refs=1:

       SIMPLE          INT              1

       BITPIX          LONG                32

       NAXIS           LONG                 2

       NAXIS1          LONG               262

       NAXIS2          LONG               153

       EXTEND          INT              1

       BLANK           LONG        -214748368

       BZERO           FLOAT           0.00000

       BSCALE          FLOAT           1.00000

       CHECKSUM        STRING    '7CWG99VE7CVE79VE'

       DATASUM         STRING    '1653067982'

       COMMENT         STRING    Array[15]

       HISTORY         STRING    Array[7]

    For many purposes regarding the property extraction, the missing information in the headers is needed. 

  2. Jordan Guerra

    I made this small piece of code following some of the instructions Marco Soldati gave during the consortium meeting presentation.

    Loading SHARP data from France - IDL version

     OBTAINING FILES FROM U-PSUD USING THE HMI SERVICE

      ; Create url path
      url_p = 'HMI/sharp_cea_720s_nrt?start='+ini_date+'&end='+fin_date
      ; Initial and final dates must be in the format: YYYY-MM-DDThh%3Amm%3AssZ
      urlObj = OBJ_NEW('IDLnetUrl')
      urlObj->SetProperty, URL_SCHEME = 'http'
      urlObj->SetProperty, URL_HOST = '127.0.0.1'
      urlObj->SetProperty, URL_PORT = '8001'
      urlObj->SetProperty, URL_PATH = url_p
      urlObj->SetProperty, URL_QUERY = ''
      res = urlObj->Get(/string_array, /BUFFER)
      resJson = JSON_PARSE(strjoin(res))

    FOR CHECKING, PRINT ALL URLS RESULTING FROM THE REQUEST

    for i=0, n_elements(resJson)-1 do print, resJson[i,'url']

    DOWNLOADING THE FIRST FILE FOR PROCESSING

    fname = strmid(resJson[0,'url'],45,34)+'.fits' ; CREATE NAME OF THE FILE
    file = wget(resJson[0,'url'], filename=fname) ; DOWNLOAD THE FILE

     

    Please feel free to provide any comments about it. I'm not particularly sure about the usage of wget in this context.

  3. Sung-Hong Park

    Based on the metadata in the hmi-service, I made an IDL program (test_load_sharp_from_upsub.pro) to search urls and to download SHARP Br fits files for same active regions (HARPNUM) during a given time period. Any comments/suggestions about this program are welcome.