Data Blending

In this use case we are going to process a CSV file send to a DimML application. Furthermore we will blend the data with an existing lookup table.

Use the following code:

concept upload {
  match '*'

    flow
        => select[`name != null`@groovy]
        => buffer:aggregate['1s', 'name', totalscore = 'sum score'] 
        => debug

     plugin debug

     plugin input:csv['name','score']
}

This will create the possibility to send files to the O2mc platform in CSV format. The file do not require a header, but expect 2 columns: name and score.

Also note that since the match rule is set as it is any interaction for the domain will go through the file. As a result opening a file in the browser will also trigger the flow. To avoid errors the select element is added to stop the application from continuing.

1. Create a text.csv file with comma separated values (e.g. “peter,10”)

2. Execute the following cURL statement

curl -H "Content-Type: text/csv" -i -X POST --data-binary @test.csv --referer http://www.domain.nl/?dimml=sandbox/d994df195bd2129815029892f0df11f4439c14ee http://dynamic.dimml.io/input

The domain dynamic.dimml.io is used specifically for file processing.

3. Open a page of the domain in your browser. In the console you should see the result of your application and the CSV file

4. Remove the select element and upload the file again. Look at your browser. Why do you see these results?

5. Extend the input with more columns and perform additional aggregations functions

6. The previous used demo database contains a table ‘lookup’ with columns ‘name’ and ‘gender’. This table contains combinations of names and genders. Extend the application such that for each item in the file, the gender of the person (name) is collected from the lookup table.

Note that the table contains the following rows: ‘sheldon-male’, ‘leonard-male’, ‘penny-female’

7. Write the result to a new CSV file on one of the existing FTP locations.