Configurable Importer

[milafrerichs]

Hi,

most of you probably know that I’m working on a new Importer with React.

We’re currently working on trying to make it easy configurable, and I came up with a few ideas how this could look like.

I would like your input on my current idea.
Thanks.

Idea

Have a config object, that is loaded at runtime and can be edited for each instance. There is a default which only handles name changes.

The default config would look like this:

config =  {
  steps: {
    1: {
      title: 'Layer Name'
      fields: [
        { title: '', api_name: 'layer_name', type: 'text'}
      ]
    }
  }
};

The basic idea is that each step in the create layer window is represented by it’s position number.
Each Step has a title and fields. The Layername step only has one item. But for example the Configure time item would have multiple.

Here is a more detailed example

var config = {
  steps: {
    1: {
      title: 'Layer Name',
      fields: [
        { title: 'Layer', api_name: 'layer_name', type: 'text'}
      ]
    },
    2: { 
      title: 'Dates', 
      fields: [
        {title: 'Start Date', api_name: 'start_date', type: 'fields'},
        {title: 'End Date', api_name: 'end_date', type: 'fields'},
        {title: '', api_name: 'configureTime', type: 'hidden', value: true, parent: 'start_date‘}
        {title: '', api_name: 'convert_to_date', type: 'hidden', value: (d) => { return [d.start_date]}, parent: 'start_date'}
    ]},
    3: {
      title: 'Version Control',
      fields: [
        {title: '', api_name: 'editable', type: 'switch', values: [true, false]},
        {title: '', api_name: 'geoserver_store', type: 'hidden', value: {'type': 'geogig'}, parent: 'editable'}
    ]}
  }
}

As you can see there is even a idea about adding functions to it. They would be called with the results.
Hidden fields provide a way to add values to the configuration without displaying them.
The parent item ensures that this field only get’s added when the parent is filled.

What do you think? Feedback very much welcome and appreciated.

[jj0hns0n]

@Coop56 @davisc @JivanAmara can you guys take a look at this?

@milafrerichs can you add a permissions step in the 2nd example.

[milafrerichs]

The next step is to include the API to fill out certain fields.
I imagine something like:

var config = {
  steps: ...,
  endpoints: {
    users: {
      url: 'account/ajax_lookup',
      type: 'POST',
      query: 'query'
    }
  }

[jj0hns0n]

Also, @afabiani can you take a look here and lets think through what would be needed for the ImageMosaic functionality to get ported over.

[bitner]

I think having an option to tie the config to either the index or explicitly to the file name of the particular file (or file + layer name in the case of gpkg) would be good as the index order can potentially be compromised and this would make sure that you could write a portable config that could potentially be just thrown in a zip or directory as a text file to make a sort of GeoNode package that could be used to add data into different instances.

…

On Tue, Apr 11, 2017 at 10:38 AM, Mila Frerichs ***@***.***> wrote: The next step is to include the API to fill out certain fields. I imagine something like: var config = { steps: ..., endpoints: { users: { url: 'account/ajax_lookup', type: 'POST', query: 'query' } } — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKD3M-6XhTqUZTagRjMS1A8a6jv23CUks5ru56RgaJpZM4M6Pnb> .

-- ************************************ David William Bitner dbSpatial LLC 612-424-9932

[milafrerichs]

the idea was that this is a more general config for creating layers.
but we could allow for it to be included in the upload and be used.
but not sure how this could work.

[bitner]

Even then it can be nice to refer to the layer by a name rather than worrying about what the order is. More pointing out a potential advantage of this approach could be to extend to actually saving that config in a file.

…

On Tue, Apr 11, 2017 at 10:52 AM, Mila Frerichs ***@***.***> wrote: the idea was that this is a more general config for creating layers. but we could allow for it to be included in the upload and be used. but not sure how this could work. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKD3KrSWaT0-I4uP2OoYPOqXCtrXbfZks5ru6G9gaJpZM4M6Pnb> .

-- ************************************ David William Bitner dbSpatial LLC 612-424-9932

[jj0hns0n]

If I am not mistaken, the order applies to how these fields will be shown in the UI for each particular layer, not the order of the files in the upload. @milafrerichs ?

[milafrerichs]

@bitner not sure I completely understand. could you elaborate more?

@jj0hns0n yes, the order is for the UI. here is an example where the order is displayed.

[bitner]

Never mind me, reading into this without fully reading!

…

On Tue, Apr 11, 2017 at 11:01 AM, Mila Frerichs ***@***.***> wrote: not sure I completely understand. could you elaborate more? @jj0hns0n <https://github.com/jj0hns0n> yes, the order is for the UI. here is an example where the order is displayed. [image: bildschirmfoto 2017-04-11 um 18 01 14] <https://cloud.githubusercontent.com/assets/688980/24918486/dfafc3ce-1ee0-11e7-91ea-e67e70811eda.png> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#146 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKD3JkN-v0tUh9DjwwCj-J1CDcZhz9cks5ru6PXgaJpZM4M6Pnb> .

-- ************************************ David William Bitner dbSpatial LLC 612-424-9932

[JivanAmara]

It looks to me like it's over-generalized. Something with each of the steps already coded in the importer & named would result in configs like: config = ['layerName'] or config = ['layerName', 'dates', 'versionControl'] The code that actually implements the steps would likely be easier to understand & maintain than a single piece of highly-generalized code to cover all the steps. My 2 cents, Jivan

[milafrerichs]

Hello,

thanks for your feedback. After getting through your feedback and discussing it internally I came up with a new solution which integrates a lot of what you were talking about.
It has (for now) four defaults: layerName, dateTime, permissions, versionControl (geogig).

You can specify the config in an array as @JivanAmara suggested:

config = ['layerName', 'dateTime']

but you can also specify your own methods on how a step would look like.

They follow along a pattern and must have specific methods which will be called.
Here is an example:

var layerName = function() {
  var layer, options,valuePromise,
  title = 'Layer Name';
  var steps = function(layer, options) {
    valuePromise = Promise.resolve(layer.layerName); // change this value for the layer field
  };
  steps.title = function() {
    return title;
  };
  steps.fields = function() {
    return [{
      name: 'layerName', // change this value for the api field
      type: 'text',
      value: valuePromise
    }];
  }
  steps.convert = function(values) {
    return values;
  }
  steps.valid = function(values) {
    return values;
  }
  return steps;
}

It is inspired by the work of Mike Bostock (creator of d3) for reusable charts.

The full documentation can be found here and will be updated as we go along. And includes a more complex but not complete example for permissions

https://github.com/terranodo/importer-client/blob/reusable-importer-steps/Customize.md

[JivanAmara]

That looks a lot easier to maintain, especially for adding steps without disturbing previously created ones. Good work :-)

A link to where these are being put to use would be helpful.