Ingest property data in database (REST API)

This short tutorial shows how to store data into the property service with python and requests (see also Access to REST-Services in Python).

For IDL you can take a similar rout by adapting these instructions to Access to REST-Services in IDL

The Property Service

The property service represents a web interface which allows to request, insert and modify property data from the database. Operations are performed by sending URL requests, whereas each operation is well defined as a so called route. The property service comes along with a graphical user interface at http://localhost:8002/ui/ which provides visual access to all available routes. Hereby, all routes involving the insertion or modification of data are enlisted under the Edit section.

For this tutorial we are using only two routes. One to add a new datasets and one to add a new region with properties.

• /dataset/bulk
• /region/{dataset}

Each route can hold up to three different parameter types which we discuss in detail on the basis of the route /region/{dataset}.

Firstly, the definition of a generic URL and its fragments:

https://localhost:8002/region/dataset1?algorithm_run_id=1
\___/   \_______/ \__/\______________/ \________________/
  |         |      |         |                 |
Schema     Host   Port      Path             Query

When we look at the definition of the route /region/{dataset} we can find three parameters of three different types (path, body, query):

Following, a definition of the types:

For a more technical view, one can push the “Try it out!” button. Hereby, the HTTP request for CURL is generated, e.g.:

Implementation

Prepare

In our example we are going to store some machine learning results into the dataset ml-algorithms.

# define data to store
ml_datasets = [
  	{
   	 	"name": "ml-algorithms",
    	"responsible": "John Doe",
    	"description": "no comment!"
  	}
]
ml_result = {
    'time_start': '2016-01-22T17:27:59.001Z',
    'lat_hg': 12,
    'long_hg': 4.2,
    'algorithm_name': 'LassoCV',
    'algorithm_parameters': {
        'store': 'what you want',
        'also here': [1, 2, 3, 4]
    }
}

Unfortunately, the above given structure ml_result does not yet fit the model for the property_group_data definition as given by the route /region/{dataset}:

A restructuration is required to accomplish the following definition:

• Every region needs at least a time_start, lat_hg and long_hg attribute.
• Every specific property has to be added to the data attribute and has to be a key-value pair.

This would then look like this:

# define data to store
post_data = {
    'time_start': '2016-01-21T17:27:59.001Z',
    'lat_hg': 12,
    'long_hg': 4.2,
    'data': {
	    "LassoCV": {
			'store': 'what you want',
	        'also here': [1, 2, 3, 4]
		}
    }
}

Ingest

Finally, the ingest of this post_data is very simple. We first have to add the new dataset to the property service and then add a new region, using the routes as enlisted above.

# add dataset
print('creating dataset...')
requests.post("http://localhost:8002/dataset/bulk", json=ml_datasets)

# add region
print('storing data...')
requests.post("http://localhost:8002/region/%s" % ml_datasets[0]['name'], json=post_data)

Source Code

Here you can download the full python source code.

ingest_property_data.py