Feature python enkf by nilsvanvelzen · Pull Request #76 · OpenDA-Association/OpenDA

nilsvanvelzen · 2019-03-13T14:39:21Z

No description provided.

nilsvanvelzen · 2020-01-28T19:04:40Z

Here is an overview of what got changed by this pull request:

Issues
======
- Added 29
           

Complexity increasing per file
==============================
- py_openda/py_openda/costFunctions/CsvStochObserver.py  2
- py_openda/py_openda/algorithms/ensemble_kalman.py  3
- py_openda/py_openda/costFunctions/LorenzStochModelFactory.py  8
- py_openda/py_openda/costFunctions/GenericEnsembleKalmanFilter.py  4
- py_openda/py_openda/costFunctions/JObjects.py  6
- py_openda/py_openda/costFunctions/LorenzStochModelInstance.py  8
- py_openda/py_openda/utils/py4j_utils.py  5
- py_openda/examples/py_kalman/enkf_swan.py  4
         

Clones added
============
- py_openda/py_openda/costFunctions/JObjects.py  1
- py_openda/py_openda/costFunctions/LorenzStochModelInstance.py  1

See the complete overview on Codacy

nilsvanvelzen · 2020-01-28T19:04:40Z

py_openda/README_ENKF.md

+
+TODO: java observation description maken!
+
+* `get_observation_descriptions(self)` returns the observation descriptions that the model instance can use to determine which elements of the state vector corresponds to quantities that are observed. Keep in mind that there is no utility routine for creating a java observation description from python, so for now you cannot combine a python observer with a java model.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:42Z

py_openda/README_ENKF.md

+
+* `get_observations(self, descriptions)` returns values of the state vector that correspond to the given observation descriptions. These descriptions are given by the stoch observer, so pay special attention when trying to use them.
+
+* `update_state(self, state_array, main_or_ens)` changes the state vector based on the input `state_array`. Note that the input is a little different depending on whether the model to be updated is an ensemble member or the main model. For an ensemblemember `state_array` should be added to the state vector, while for the main model it is supposed to replace the state vector. This is controlled by the `main_or_ens` argument, which is a string equal to either "main" or "ens".


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:43Z

py_openda/README_ENKF.md

+
+TODO: Misschien leuk om init ook met alleen een xml file als input te doen?
+
+* `__init__(self, config=None, scriptdir=None)` initializes the object. Here `config` is the stochObserver element of the main `.oda` configuration file as a dictionary. `scriptdir` is the directory containing the `.oda` file.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:43Z

py_openda/README_ENKF.md

+    """
+```
+
+* `__init__(self, config, scriptdir)` initializes the object. Here `config` is the stochModelFactory element of the main `.oda` configuration file as a dictionary. `scriptdir` is the directory containing the `.oda` file.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:44Z

py_openda/py_openda/costFunctions/JObjects.py

+            observer_config_xml = config.get('configFile')
+            self.observer = None
+            observer_object = 'gateway.jvm.'+config.get('@className')
+            exec('self.observer =%s()' % observer_object)


Issue found: Use of exec detected.

nilsvanvelzen · 2020-01-28T19:04:46Z

py_openda/py_openda/costFunctions/LorenzStochModelInstance.py

+        """
+        (self.param, self.param_uncertainty, self.state, self.state_uncertainty, self.sys_mean,
+         self.sys_std, self.span) = model_attributes
+


Issue found: Trailing whitespace

nilsvanvelzen · 2020-01-28T19:04:47Z

py_openda/py_openda/costFunctions/JObjects.py

+        model_config_xml = config.get('configFile')
+        self.model_factory = None
+        model_object = 'gateway.jvm.'+config.get('@className')
+        exec('self.model_factory =%s()' % model_object)


Issue found: Use of exec

nilsvanvelzen · 2020-01-28T19:04:47Z

py_openda/README_ENKF.md

+    """
+```
+
+* `get_time_horizon(self)` should return a time object spanning the start and end of the simulation.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:49Z

py_openda/README_ENKF.md

+
+* `get_observation_descriptions(self)` returns the observation descriptions that the model instance can use to determine which elements of the state vector corresponds to quantities that are observed. Keep in mind that there is no utility routine for creating a java observation description from python, so for now you cannot combine a python observer with a java model.
+
+* `get_sqrt_covariance(self)` generates an np array representing the squared covariance matrix. Note that you first have to convert the output yourself. For this you can use the utility routine `j_vector_array_to_np_array(j_sqrt)` if you have a java vector array.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:50Z

py_openda/README_ENKF.md

+
+* `compute(self, time)` runs the model until the given time.
+
+* `get_observations(self, descriptions)` returns values of the state vector that correspond to the given observation descriptions. These descriptions are given by the stoch observer, so pay special attention when trying to use them.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:52Z

py_openda/README_ENKF.md

+
+* `get_time_horizon(self)` should return a time object spanning the start and end of the simulation.
+
+* `get_current_time(self)` returns the time up to which the state vector has been calculated.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:53Z

py_openda/py_openda/costFunctions/JObjects.py

+                noise_config = {'@stochParameter':False, '@stochForcing':False, '@stochInit':False}
+            elif main_or_ens == "ens":
+                noise_config = {'@stochParameter':False, '@stochForcing':True, '@stochInit':True}
+


Issue found: Trailing whitespace

nilsvanvelzen · 2020-01-28T19:04:54Z

py_openda/py_openda/costFunctions/LorenzStochModelInstance.py

+        """
+        return self.current_time
+
+    def announce_observed_values(self, descriptions):


Issue found: Method could be a function

nilsvanvelzen · 2020-01-28T19:04:55Z

py_openda/py_openda/costFunctions/JObjects.py

+            observer_config_xml = config.get('configFile')
+            self.observer = None
+            observer_object = 'gateway.jvm.'+config.get('@className')
+            exec('self.observer =%s()' % observer_object)


Issue found: Use of exec

nilsvanvelzen · 2020-01-28T19:04:56Z

py_openda/README_ENKF.md

@@ -0,0 +1,182 @@
+## Using the main program


Issue found: [no-file-name-irregular-characters] Do not use _ in a file name

nilsvanvelzen · 2020-01-28T19:04:57Z

py_openda/py_openda/utils/py4j_utils.py

+    elif isinstance(obj, np.ndarray):
+        obj = obj.squeeze().tolist()
+    elif isinstance(obj, list):
+        None


Issue found: Statement seems to have no effect

nilsvanvelzen · 2020-01-28T19:04:58Z

py_openda/README_ENKF.md

+
+* `get_sqrt_covariance(self)` generates an np array representing the squared covariance matrix. Note that you first have to convert the output yourself. For this you can use the utility routine `j_vector_array_to_np_array(j_sqrt)` if you have a java vector array.
+
+* `get_realizations(self)` creates realizations of the observed values using the mean and the standarddeviation of the measurement. The output will be converted automatically.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:04:59Z

py_openda/examples/py_kalman/setup.py

+from setuptools import setup, find_packages
+with open("README.txt", 'r') as f:
+    long_description = f.read()
+


Issue found: Trailing whitespace

nilsvanvelzen · 2020-01-28T19:05:00Z

py_openda/examples/py_kalman/enkf_swan.py

+# ensembleModel@stochParameter=false
+# ensembleModel@stochForcing=true
+# ensembleModel@stochInit=true
+


Issue found: Trailing whitespace

nilsvanvelzen · 2020-01-28T19:05:02Z

py_openda/README_ENKF.md

+
+* `get_current_time(self)` returns the time up to which the state vector has been calculated.
+
+* `announce_observed_values(self, descriptions)` function which can be used to make sure that the model will save the observations described by the descriptions. This function does get called by the main algorithm, but it does not have to do anything if your implementation does not need it to.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:05:03Z

py_openda/README_ENKF.md

+
+* `get_times(self)` lists all the times where there are observations available. The output will automatically be converted to a python list of time objects.
+
+* `get_count(self)` simply returns an integer indicating the number of measurements available.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:05:05Z

py_openda/README_ENKF.md

+
+* `__init__(self, config=None, scriptdir=None)` initializes the object. Here `config` is the stochObserver element of the main `.oda` configuration file as a dictionary. `scriptdir` is the directory containing the `.oda` file.
+
+* `create_selection(self, model_span)` creates a new stoch observer, limited to the span given by the time object `model_span`. The class of the output should be the same as the class of `self`.


Issue found: [list-item-indent] Incorrect list-item indent: add 2 spaces

nilsvanvelzen · 2020-01-28T19:05:06Z

py_openda/py_openda/costFunctions/JObjects.py

+        model_config_xml = config.get('configFile')
+        self.model_factory = None
+        model_object = 'gateway.jvm.'+config.get('@className')
+        exec('self.model_factory =%s()' % model_object)


Issue found: Use of exec detected.

nilsvanvelzen · 2020-01-28T19:05:07Z

py_openda/py_openda/costFunctions/LorenzStochModelInstance.py

+        :param descriptions: an ObservationDescriptions object with meta data for the observations.
+        :return:
+        """
+        None


Issue found: Statement seems to have no effect

nilsvanvelzen · 2020-01-28T19:05:07Z

py_openda/README_ENKF.md

+* `get_observation_descriptions(self)` returns the observation descriptions that the model instance can use to determine which elements of the state vector corresponds to quantities that are observed. Keep in mind that there is no utility routine for creating a java observation description from python, so for now you cannot combine a python observer with a java model.
+
+* `get_sqrt_covariance(self)` generates an np array representing the squared covariance matrix. Note that you first have to convert the output yourself. For this you can use the utility routine `j_vector_array_to_np_array(j_sqrt)` if you have a java vector array.
+


Issue found: [list-item-spacing] Extraneous new line after list item

RickHegeman added 3 commits November 29, 2018 15:54

Python enkf which can use observers and models from Java and Python

6cd6b53

Updated min todo list

2c6e568

Moved Python library and added TODO's

a58c77d

nilsvanvelzen self-assigned this Mar 13, 2019

nilsvanvelzen commented Jan 28, 2020

View reviewed changes


		TODO: java observation description maken!

		* `get_observation_descriptions(self)` returns the observation descriptions that the model instance can use to determine which elements of the state vector corresponds to quantities that are observed. Keep in mind that there is no utility routine for creating a java observation description from python, so for now you cannot combine a python observer with a java model.


		* `get_observations(self, descriptions)` returns values of the state vector that correspond to the given observation descriptions. These descriptions are given by the stoch observer, so pay special attention when trying to use them.

		* `update_state(self, state_array, main_or_ens)` changes the state vector based on the input `state_array`. Note that the input is a little different depending on whether the model to be updated is an ensemble member or the main model. For an ensemblemember `state_array` should be added to the state vector, while for the main model it is supposed to replace the state vector. This is controlled by the `main_or_ens` argument, which is a string equal to either "main" or "ens".


		TODO: Misschien leuk om init ook met alleen een xml file als input te doen?

		* `__init__(self, config=None, scriptdir=None)` initializes the object. Here `config` is the stochObserver element of the main `.oda` configuration file as a dictionary. `scriptdir` is the directory containing the `.oda` file.


		* `get_observation_descriptions(self)` returns the observation descriptions that the model instance can use to determine which elements of the state vector corresponds to quantities that are observed. Keep in mind that there is no utility routine for creating a java observation description from python, so for now you cannot combine a python observer with a java model.

		* `get_sqrt_covariance(self)` generates an np array representing the squared covariance matrix. Note that you first have to convert the output yourself. For this you can use the utility routine `j_vector_array_to_np_array(j_sqrt)` if you have a java vector array.


		* `compute(self, time)` runs the model until the given time.

		* `get_observations(self, descriptions)` returns values of the state vector that correspond to the given observation descriptions. These descriptions are given by the stoch observer, so pay special attention when trying to use them.


		* `get_time_horizon(self)` should return a time object spanning the start and end of the simulation.

		* `get_current_time(self)` returns the time up to which the state vector has been calculated.


		* `get_sqrt_covariance(self)` generates an np array representing the squared covariance matrix. Note that you first have to convert the output yourself. For this you can use the utility routine `j_vector_array_to_np_array(j_sqrt)` if you have a java vector array.

		* `get_realizations(self)` creates realizations of the observed values using the mean and the standarddeviation of the measurement. The output will be converted automatically.


		* `get_current_time(self)` returns the time up to which the state vector has been calculated.

		* `announce_observed_values(self, descriptions)` function which can be used to make sure that the model will save the observations described by the descriptions. This function does get called by the main algorithm, but it does not have to do anything if your implementation does not need it to.


		* `get_times(self)` lists all the times where there are observations available. The output will automatically be converted to a python list of time objects.

		* `get_count(self)` simply returns an integer indicating the number of measurements available.


		* `__init__(self, config=None, scriptdir=None)` initializes the object. Here `config` is the stochObserver element of the main `.oda` configuration file as a dictionary. `scriptdir` is the directory containing the `.oda` file.

		* `create_selection(self, model_span)` creates a new stoch observer, limited to the span given by the time object `model_span`. The class of the output should be the same as the class of `self`.

Conversation

nilsvanvelzen commented Mar 13, 2019

Uh oh!

nilsvanvelzen commented Jan 28, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants