3 ================================================================================
4 Reference description of the ADAO commands and keywords
5 ================================================================================
7 This section presents the reference description of the ADAO commands and
8 keywords available through the GUI or through scripts.
10 Each command or keyword to be defined through the ADAO GUI has some properties.
11 The first property is to be *required*, *optional* or only factual, describing a
12 type of input. The second property is to be an "open" variable with a fixed type
13 but with any value allowed by the type, or a "restricted" variable, limited to
14 some specified values. The EFICAS editor GUI having build-in validating
15 capacities, the properties of the commands or keywords given through this GUI
16 are automatically correct.
18 The mathematical notations used afterward are explained in the section
19 :ref:`section_theory`.
21 Examples of using these commands are available in the section
22 :ref:`section_examples` and in example files installed with ADAO module.
24 List of possible input types
25 ----------------------------
27 .. index:: single: Dict
28 .. index:: single: Function
29 .. index:: single: Matrix
30 .. index:: single: ScalarSparseMatrix
31 .. index:: single: DiagonalSparseMatrix
32 .. index:: single: String
33 .. index:: single: Script
34 .. index:: single: Vector
36 Each ADAO variable has a pseudo-type to help filling it and validation. The
37 different pseudo-types are:
40 This indicates a variable that has to be filled by a dictionary, usually
41 given either as a string or as a script.
44 This indicates a variable that has to be filled by a function, usually given
45 as a script or a component method.
48 This indicates a variable that has to be filled by a matrix, usually given
49 either as a string or as a script.
51 **ScalarSparseMatrix**
52 This indicates a variable that has to be filled by a unique number, which
53 will be used to multiply an identity matrix, usually given either as a
54 string or as a script.
56 **DiagonalSparseMatrix**
57 This indicates a variable that has to be filled by a vector, which will be
58 over the diagonal of an identity matrix, usually given either as a string or
62 This indicates a script given as an external file. It can be described by a
63 full absolute path name or only by the file name without path. If the file
64 is given only by a file name without path, and if a study directory is also
65 indicated, the file is searched in the given directory.
68 This indicates a string giving a literal representation of a matrix, a
69 vector or a vector serie, such as "1 2 ; 3 4" for a square 2x2 matrix.
72 This indicates a variable that has to be filled by a vector, usually given
73 either as a string or as a script.
75 **VectorSerie** This indicates a variable that has to be filled by a list of
76 vectors, usually given either as a string or as a script.
78 When a command or keyword can be filled by a script file name, the script has to
79 contain a variable or a method that has the same name as the one to be filled.
80 In other words, when importing the script in a YACS Python node, it must create
81 a variable of the good name in the current namespace.
83 Reference description for ADAO calculation cases
84 ------------------------------------------------
86 List of commands and keywords for an ADAO calculation case
87 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
89 .. index:: single: ASSIMILATION_STUDY
90 .. index:: single: Algorithm
91 .. index:: single: AlgorithmParameters
92 .. index:: single: Background
93 .. index:: single: BackgroundError
94 .. index:: single: ControlInput
95 .. index:: single: Debug
96 .. index:: single: EvolutionError
97 .. index:: single: EvolutionModel
98 .. index:: single: InputVariables
99 .. index:: single: Observation
100 .. index:: single: ObservationError
101 .. index:: single: ObservationOperator
102 .. index:: single: Observers
103 .. index:: single: OutputVariables
104 .. index:: single: Study_name
105 .. index:: single: Study_repertory
106 .. index:: single: UserDataInit
107 .. index:: single: UserPostAnalysis
109 The first set of commands is related to the description of a calculation case,
110 that is a *Data Assimilation* procedure or an *Optimization* procedure. The
111 terms are ordered in alphabetical order, except the first, which describes
112 choice between calculation or checking. The different commands are the
115 **ASSIMILATION_STUDY**
116 *Required command*. This is the general command describing the data
117 assimilation or optimization case. It hierarchically contains all the other
121 *Required command*. This is a string to indicate the data assimilation or
122 optimization algorithm chosen. The choices are limited and available through
123 the GUI. There exists for example "3DVAR", "Blue"... See below the list of
124 algorithms and associated parameters in the following subsection `Options
125 and required commands for calculation algorithms`_.
127 **AlgorithmParameters**
128 *Optional command*. This command allows to add some optional parameters to
129 control the data assimilation or optimization algorithm. It is defined as a
130 "*Dict*" type object, that is, given as a script. See below the list of
131 algorithms and associated parameters in the following subsection `Options
132 and required commands for calculation algorithms`_.
135 *Required command*. This indicates the background or initial vector used,
136 previously noted as :math:`\mathbf{x}^b`. It is defined as a "*Vector*" type
137 object, that is, given either as a string or as a script.
140 *Required command*. This indicates the background error covariance matrix,
141 previously noted as :math:`\mathbf{B}`. It is defined as a "*Matrix*" type
142 object, a "*ScalarSparseMatrix*" type object, or a "*DiagonalSparseMatrix*"
143 type object, that is, given either as a string or as a script.
146 *Optional command*. This indicates the control vector used to force the
147 evolution model at each step, usually noted as :math:`\mathbf{U}`. It is
148 defined as a "*Vector*" or a *VectorSerie* type object, that is, given
149 either as a string or as a script. When there is no control, it has to be a
153 *Required command*. This define the level of trace and intermediary debug
154 information. The choices are limited between 0 (for False) and 1 (for
158 *Optional command*. This indicates the evolution error covariance matrix,
159 usually noted as :math:`\mathbf{Q}`. It is defined as a "*Matrix*" type
160 object, a "*ScalarSparseMatrix*" type object, or a "*DiagonalSparseMatrix*"
161 type object, that is, given either as a string or as a script.
164 *Optional command*. This indicates the evolution model operator, usually
165 noted :math:`M`, which describes a step of evolution. It is defined as a
166 "*Function*" type object, that is, given as a script. Different functional
167 forms can be used, as described in the following subsection `Requirements
168 for functions describing an operator`_. If there is some control :math:`U`
169 included in the evolution model, the operator has to be applied to a pair
173 *Optional command*. This command allows to indicates the name and size of
174 physical variables that are bundled together in the control vector. This
175 information is dedicated to data processed inside an algorithm.
178 *Required command*. This indicates the observation vector used for data
179 assimilation or optimization, previously noted as :math:`\mathbf{y}^o`. It
180 is defined as a "*Vector*" or a *VectorSerie* type object, that is, given
181 either as a string or as a script.
184 *Required command*. This indicates the observation error covariance matrix,
185 previously noted as :math:`\mathbf{R}`. It is defined as a "*Matrix*" type
186 object, a "*ScalarSparseMatrix*" type object, or a "*DiagonalSparseMatrix*"
187 type object, that is, given either as a string or as a script.
189 **ObservationOperator**
190 *Required command*. This indicates the observation operator, previously
191 noted :math:`H`, which transforms the input parameters :math:`\mathbf{x}` to
192 results :math:`\mathbf{y}` to be compared to observations
193 :math:`\mathbf{y}^o`. It is defined as a "*Function*" type object, that is,
194 given as a script. Different functional forms can be used, as described in
195 the following subsection `Requirements for functions describing an
196 operator`_. If there is some control :math:`U` included in the observation,
197 the operator has to be applied to a pair :math:`(X,U)`.
200 *Optional command*. This command allows to set internal observers, that are
201 functions linked with a particular variable, which will be executed each
202 time this variable is modified. It is a convenient way to monitor variables
203 of interest during the data assimilation or optimization process, by
204 printing or plotting it, etc. Common templates are provided to help the user
205 to start or to quickly make his case.
208 *Optional command*. This command allows to indicates the name and size of
209 physical variables that are bundled together in the output observation
210 vector. This information is dedicated to data processed inside an algorithm.
213 *Required command*. This is an open string to describe the study by a name
217 *Optional command*. If available, this directory is used as base name for
218 calculation, and used to find all the script files, given by name without
219 path, that can be used to define some other commands by scripts.
222 *Optional command*. This commands allows to initialize some parameters or
223 data automatically before data assimilation algorithm processing.
226 *Optional command*. This commands allows to process some parameters or data
227 automatically after data assimilation algorithm processing. It is defined as
228 a script or a string, allowing to put post-processing code directly inside
229 the ADAO case. Common templates are provided to help the user to start or
230 to quickly make his case.
232 Options and required commands for calculation algorithms
233 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
235 .. index:: single: 3DVAR
236 .. index:: single: Blue
237 .. index:: single: ExtendedBlue
238 .. index:: single: EnsembleBlue
239 .. index:: single: KalmanFilter
240 .. index:: single: ExtendedKalmanFilter
241 .. index:: single: LinearLeastSquares
242 .. index:: single: NonLinearLeastSquares
243 .. index:: single: ParticleSwarmOptimization
244 .. index:: single: QuantileRegression
246 .. index:: single: AlgorithmParameters
247 .. index:: single: Bounds
248 .. index:: single: CostDecrementTolerance
249 .. index:: single: GradientNormTolerance
250 .. index:: single: GroupRecallRate
251 .. index:: single: MaximumNumberOfSteps
252 .. index:: single: Minimizer
253 .. index:: single: NumberOfInsects
254 .. index:: single: ProjectedGradientTolerance
255 .. index:: single: QualityCriterion
256 .. index:: single: Quantile
257 .. index:: single: SetSeed
258 .. index:: single: StoreInternalVariables
259 .. index:: single: StoreSupplementaryCalculations
260 .. index:: single: SwarmVelocity
262 Each algorithm can be controlled using some generic or specific options given
263 through the "*AlgorithmParameters*" optional command, as follows for example::
265 AlgorithmParameters = {
266 "Minimizer" : "LBFGSB",
267 "MaximumNumberOfSteps" : 25,
268 "StoreSupplementaryCalculations" : ["APosterioriCovariance","OMA"],
271 This section describes the available options algorithm by algorithm. If an
272 option is specified for an algorithm that doesn't support it, the option is
273 simply left unused. The meaning of the acronyms or particular names can be found
274 in the :ref:`genindex` or the :ref:`section_glossary`. In addition, for each
275 algorithm, the required commands/keywords are given, being described in `List of
276 commands and keywords for an ADAO calculation case`_.
281 *"Background", "BackgroundError",
282 "Observation", "ObservationError",
283 "ObservationOperator"*
285 StoreInternalVariables
286 This boolean key allows to store default internal variables, mainly the
287 current state during iterative optimization process. Be careful, this can be
288 a numerically costly choice in certain calculation cases. The default is
291 StoreSupplementaryCalculations
292 This list indicates the names of the supplementary variables that can be
293 available at the end of the algorithm. It involves potentially costly
294 calculations. The default is a void list, none of these variables being
295 calculated and stored by default. The possible names are in the following
296 list: ["APosterioriCovariance", "BMA", "OMA", "OMB", "Innovation",
297 "SigmaBck2", "SigmaObs2", "MahalanobisConsistency"].
302 *"Background", "BackgroundError",
303 "Observation", "ObservationError",
304 "ObservationOperator"*
306 StoreInternalVariables
307 This boolean key allows to store default internal variables, mainly the
308 current state during iterative optimization process. Be careful, this can be
309 a numerically costly choice in certain calculation cases. The default is
312 StoreSupplementaryCalculations
313 This list indicates the names of the supplementary variables that can be
314 available at the end of the algorithm. It involves potentially costly
315 calculations. The default is a void list, none of these variables being
316 calculated and stored by default. The possible names are in the following
317 list: ["APosterioriCovariance", "BMA", "OMA", "OMB", "Innovation",
318 "SigmaBck2", "SigmaObs2", "MahalanobisConsistency"].
320 **"LinearLeastSquares"**
323 *"Observation", "ObservationError",
324 "ObservationOperator"*
326 StoreInternalVariables
327 This boolean key allows to store default internal variables, mainly the
328 current state during iterative optimization process. Be careful, this can be
329 a numerically costly choice in certain calculation cases. The default is
332 StoreSupplementaryCalculations
333 This list indicates the names of the supplementary variables that can be
334 available at the end of the algorithm. It involves potentially costly
335 calculations. The default is a void list, none of these variables being
336 calculated and stored by default. The possible names are in the following
342 *"Background", "BackgroundError",
343 "Observation", "ObservationError",
344 "ObservationOperator"*
347 This key allows to choose the optimization minimizer. The default choice
348 is "LBFGSB", and the possible ones are "LBFGSB" (nonlinear constrained
349 minimizer, see [Byrd95]_ and [Zhu97]_), "TNC" (nonlinear constrained
350 minimizer), "CG" (nonlinear unconstrained minimizer), "BFGS" (nonlinear
351 unconstrained minimizer), "NCG" (Newton CG minimizer).
354 This key allows to define upper and lower bounds for every control
355 variable being optimized. Bounds can be given by a list of list of pairs
356 of lower/upper bounds for each variable, with possibly ``None`` every time
357 there is no bound. The bounds can always be specified, but they are taken
358 into account only by the constrained minimizers.
361 This key indicates the maximum number of iterations allowed for iterative
362 optimization. The default is 15000, which is very similar to no limit on
363 iterations. It is then recommended to adapt this parameter to the needs on
364 real problems. For some minimizers, the effective stopping step can be
365 slightly different due to algorithm internal control requirements.
367 CostDecrementTolerance
368 This key indicates a limit value, leading to stop successfully the
369 iterative optimization process when the cost function decreases less than
370 this tolerance at the last step. The default is 1.e-7, and it is
371 recommended to adapt it to the needs on real problems.
373 ProjectedGradientTolerance
374 This key indicates a limit value, leading to stop successfully the iterative
375 optimization process when all the components of the projected gradient are
376 under this limit. It is only used for constrained minimizers. The default is
377 -1, that is the internal default of each minimizer (generally 1.e-5), and it
378 is not recommended to change it.
380 GradientNormTolerance
381 This key indicates a limit value, leading to stop successfully the
382 iterative optimization process when the norm of the gradient is under this
383 limit. It is only used for non-constrained minimizers. The default is
384 1.e-5 and it is not recommended to change it.
386 StoreInternalVariables
387 This boolean key allows to store default internal variables, mainly the
388 current state during iterative optimization process. Be careful, this can be
389 a numerically costly choice in certain calculation cases. The default is
392 StoreSupplementaryCalculations
393 This list indicates the names of the supplementary variables that can be
394 available at the end of the algorithm. It involves potentially costly
395 calculations. The default is a void list, none of these variables being
396 calculated and stored by default. The possible names are in the following
397 list: ["APosterioriCovariance", "BMA", "OMA", "OMB", "Innovation",
398 "SigmaObs2", "MahalanobisConsistency"].
400 **"NonLinearLeastSquares"**
404 "Observation", "ObservationError",
405 "ObservationOperator"*
408 This key allows to choose the optimization minimizer. The default choice
409 is "LBFGSB", and the possible ones are "LBFGSB" (nonlinear constrained
410 minimizer, see [Byrd95]_ and [Zhu97]_), "TNC" (nonlinear constrained
411 minimizer), "CG" (nonlinear unconstrained minimizer), "BFGS" (nonlinear
412 unconstrained minimizer), "NCG" (Newton CG minimizer).
415 This key allows to define upper and lower bounds for every control
416 variable being optimized. Bounds can be given by a list of list of pairs
417 of lower/upper bounds for each variable, with possibly ``None`` every time
418 there is no bound. The bounds can always be specified, but they are taken
419 into account only by the constrained minimizers.
422 This key indicates the maximum number of iterations allowed for iterative
423 optimization. The default is 15000, which is very similar to no limit on
424 iterations. It is then recommended to adapt this parameter to the needs on
425 real problems. For some minimizers, the effective stopping step can be
426 slightly different due to algorithm internal control requirements.
428 CostDecrementTolerance
429 This key indicates a limit value, leading to stop successfully the
430 iterative optimization process when the cost function decreases less than
431 this tolerance at the last step. The default is 1.e-7, and it is
432 recommended to adapt it to the needs on real problems.
434 ProjectedGradientTolerance
435 This key indicates a limit value, leading to stop successfully the iterative
436 optimization process when all the components of the projected gradient are
437 under this limit. It is only used for constrained minimizers. The default is
438 -1, that is the internal default of each minimizer (generally 1.e-5), and it
439 is not recommended to change it.
441 GradientNormTolerance
442 This key indicates a limit value, leading to stop successfully the
443 iterative optimization process when the norm of the gradient is under this
444 limit. It is only used for non-constrained minimizers. The default is
445 1.e-5 and it is not recommended to change it.
447 StoreInternalVariables
448 This boolean key allows to store default internal variables, mainly the
449 current state during iterative optimization process. Be careful, this can be
450 a numerically costly choice in certain calculation cases. The default is
453 StoreSupplementaryCalculations
454 This list indicates the names of the supplementary variables that can be
455 available at the end of the algorithm. It involves potentially costly
456 calculations. The default is a void list, none of these variables being
457 calculated and stored by default. The possible names are in the following
458 list: ["BMA", "OMA", "OMB", "Innovation"].
463 *"Background", "BackgroundError",
464 "Observation", "ObservationError",
465 "ObservationOperator"*
468 This key allow to give an integer in order to fix the seed of the random
469 generator used to generate the ensemble. A convenient value is for example
470 1000. By default, the seed is left uninitialized, and so use the default
471 initialization from the computer.
476 *"Background", "BackgroundError",
477 "Observation", "ObservationError",
478 "ObservationOperator",
479 "EvolutionModel", "EvolutionError",
483 This key allows to choose the type of estimation to be performed. It can be
484 either state-estimation, named "State", or parameter-estimation, named
485 "Parameters". The default choice is "State".
487 StoreSupplementaryCalculations
488 This list indicates the names of the supplementary variables that can be
489 available at the end of the algorithm. It involves potentially costly
490 calculations. The default is a void list, none of these variables being
491 calculated and stored by default. The possible names are in the following
492 list: ["APosterioriCovariance", "BMA", "Innovation"].
494 **"ExtendedKalmanFilter"**
497 *"Background", "BackgroundError",
498 "Observation", "ObservationError",
499 "ObservationOperator",
500 "EvolutionModel", "EvolutionError",
504 This key allows to define upper and lower bounds for every control variable
505 being optimized. Bounds can be given by a list of list of pairs of
506 lower/upper bounds for each variable, with extreme values every time there
507 is no bound. The bounds can always be specified, but they are taken into
508 account only by the constrained minimizers.
511 This key allows to define the method to take bounds into account. The
512 possible methods are in the following list: ["EstimateProjection"].
515 This key allows to choose the type of estimation to be performed. It can be
516 either state-estimation, named "State", or parameter-estimation, named
517 "Parameters". The default choice is "State".
519 StoreSupplementaryCalculations
520 This list indicates the names of the supplementary variables that can be
521 available at the end of the algorithm. It involves potentially costly
522 calculations. The default is a void list, none of these variables being
523 calculated and stored by default. The possible names are in the following
524 list: ["APosterioriCovariance", "BMA", "Innovation"].
526 **"ParticleSwarmOptimization"**
529 *"Background", "BackgroundError",
530 "Observation", "ObservationError",
531 "ObservationOperator"*
534 This key indicates the maximum number of iterations allowed for iterative
535 optimization. The default is 50, which is an arbitrary limit. It is then
536 recommended to adapt this parameter to the needs on real problems.
539 This key indicates the number of insects or particles in the swarm. The
540 default is 100, which is a usual default for this algorithm.
543 This key indicates the part of the insect velocity which is imposed by the
544 swarm. It is a positive floating point value. The default value is 1.
547 This key indicates the recall rate at the best swarm insect. It is a
548 floating point value between 0 and 1. The default value is 0.5.
551 This key indicates the quality criterion, minimized to find the optimal
552 state estimate. The default is the usual data assimilation criterion named
553 "DA", the augmented ponderated least squares. The possible criteria has to
554 be in the following list, where the equivalent names are indicated by "=":
555 ["AugmentedPonderatedLeastSquares"="APLS"="DA",
556 "PonderatedLeastSquares"="PLS", "LeastSquares"="LS"="L2",
557 "AbsoluteValue"="L1", "MaximumError"="ME"]
560 This key allow to give an integer in order to fix the seed of the random
561 generator used to generate the ensemble. A convenient value is for example
562 1000. By default, the seed is left uninitialized, and so use the default
563 initialization from the computer.
565 StoreInternalVariables
566 This boolean key allows to store default internal variables, mainly the
567 current state during iterative optimization process. Be careful, this can be
568 a numerically costly choice in certain calculation cases. The default is
571 StoreSupplementaryCalculations
572 This list indicates the names of the supplementary variables that can be
573 available at the end of the algorithm. It involves potentially costly
574 calculations. The default is a void list, none of these variables being
575 calculated and stored by default. The possible names are in the following
576 list: ["BMA", "OMA", "OMB", "Innovation"].
578 **"QuantileRegression"**
583 "ObservationOperator"*
586 This key allows to define the real value of the desired quantile, between
587 0 and 1. The default is 0.5, corresponding to the median.
590 This key allows to choose the optimization minimizer. The default choice
591 and only available choice is "MMQR" (Majorize-Minimize for Quantile
595 This key indicates the maximum number of iterations allowed for iterative
596 optimization. The default is 15000, which is very similar to no limit on
597 iterations. It is then recommended to adapt this parameter to the needs on
600 CostDecrementTolerance
601 This key indicates a limit value, leading to stop successfully the
602 iterative optimization process when the cost function or the surrogate
603 decreases less than this tolerance at the last step. The default is 1.e-6,
604 and it is recommended to adapt it to the needs on real problems.
606 StoreInternalVariables
607 This boolean key allows to store default internal variables, mainly the
608 current state during iterative optimization process. Be careful, this can be
609 a numerically costly choice in certain calculation cases. The default is
612 StoreSupplementaryCalculations
613 This list indicates the names of the supplementary variables that can be
614 available at the end of the algorithm. It involves potentially costly
615 calculations. The default is a void list, none of these variables being
616 calculated and stored by default. The possible names are in the following
617 list: ["BMA", "OMA", "OMB", "Innovation"].
619 Reference description for ADAO checking cases
620 ---------------------------------------------
622 List of commands and keywords for an ADAO checking case
623 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
625 .. index:: single: CHECKING_STUDY
626 .. index:: single: Algorithm
627 .. index:: single: AlgorithmParameters
628 .. index:: single: CheckingPoint
629 .. index:: single: Debug
630 .. index:: single: ObservationOperator
631 .. index:: single: Study_name
632 .. index:: single: Study_repertory
633 .. index:: single: UserDataInit
635 The second set of commands is related to the description of a checking case,
636 that is a procedure to check required properties on information somewhere else
637 by a calculation case. The terms are ordered in alphabetical order, except the
638 first, which describes choice between calculation or checking. The different
639 commands are the following:
642 *Required command*. This is the general command describing the checking
643 case. It hierarchically contains all the other commands.
646 *Required command*. This is a string to indicate the data assimilation or
647 optimization algorithm chosen. The choices are limited and available through
648 the GUI. There exists for example "FunctionTest", "AdjointTest"... See below
649 the list of algorithms and associated parameters in the following subsection
650 `Options and required commands for checking algorithms`_.
652 **AlgorithmParameters**
653 *Optional command*. This command allows to add some optional parameters to
654 control the data assimilation or optimization algorithm. It is defined as a
655 "*Dict*" type object, that is, given as a script. See below the list of
656 algorithms and associated parameters in the following subsection `Options
657 and required commands for checking algorithms`_.
660 *Required command*. This indicates the vector used,
661 previously noted as :math:`\mathbf{x}^b`. It is defined as a "*Vector*" type
662 object, that is, given either as a string or as a script.
665 *Required command*. This define the level of trace and intermediary debug
666 information. The choices are limited between 0 (for False) and 1 (for
669 **ObservationOperator**
670 *Required command*. This indicates the observation operator, previously
671 noted :math:`H`, which transforms the input parameters :math:`\mathbf{x}` to
672 results :math:`\mathbf{y}` to be compared to observations
673 :math:`\mathbf{y}^o`. It is defined as a "*Function*" type object, that is,
674 given as a script. Different functional forms can be used, as described in
675 the following subsection `Requirements for functions describing an
679 *Required command*. This is an open string to describe the study by a name
683 *Optional command*. If available, this repertory is used to find all the
684 script files that can be used to define some other commands by scripts.
687 *Optional command*. This commands allows to initialize some parameters or
688 data automatically before data assimilation algorithm processing.
690 Options and required commands for checking algorithms
691 +++++++++++++++++++++++++++++++++++++++++++++++++++++
693 .. index:: single: AdjointTest
694 .. index:: single: FunctionTest
695 .. index:: single: GradientTest
696 .. index:: single: LinearityTest
698 .. index:: single: AlgorithmParameters
699 .. index:: single: AmplitudeOfInitialDirection
700 .. index:: single: EpsilonMinimumExponent
701 .. index:: single: InitialDirection
702 .. index:: single: ResiduFormula
703 .. index:: single: SetSeed
705 We recall that each algorithm can be controlled using some generic or specific
706 options given through the "*AlgorithmParameters*" optional command, as follows
709 AlgorithmParameters = {
710 "AmplitudeOfInitialDirection" : 1,
711 "EpsilonMinimumExponent" : -8,
714 If an option is specified for an algorithm that doesn't support it, the option
715 is simply left unused. The meaning of the acronyms or particular names can be
716 found in the :ref:`genindex` or the :ref:`section_glossary`. In addition, for
717 each algorithm, the required commands/keywords are given, being described in
718 `List of commands and keywords for an ADAO checking case`_.
724 "ObservationOperator"*
726 AmplitudeOfInitialDirection
727 This key indicates the scaling of the initial perturbation build as a vector
728 used for the directional derivative around the nominal checking point. The
729 default is 1, that means no scaling.
731 EpsilonMinimumExponent
732 This key indicates the minimal exponent value of the power of 10 coefficient
733 to be used to decrease the increment multiplier. The default is -8, and it
734 has to be between 0 and -20. For example, its default value leads to
735 calculate the residue of the scalar product formula with a fixed increment
736 multiplied from 1.e0 to 1.e-8.
739 This key indicates the vector direction used for the directional derivative
740 around the nominal checking point. It has to be a vector. If not specified,
741 this direction defaults to a random perturbation around zero of the same
742 vector size than the checking point.
745 This key allow to give an integer in order to fix the seed of the random
746 generator used to generate the ensemble. A convenient value is for example
747 1000. By default, the seed is left uninitialized, and so use the default
748 initialization from the computer.
754 "ObservationOperator"*
756 NumberOfPrintedDigits
757 This key indicates the number of digits of precision for floating point
758 printed output. The default is 5, with a minimum of 0.
761 This key indicates the number of time to repeat the function evaluation. The
765 This key requires the activation, or not, of the debug mode during the
766 function evaluation. The default is True, the choices are True of False.
772 "ObservationOperator"*
774 AmplitudeOfInitialDirection
775 This key indicates the scaling of the initial perturbation build as a vector
776 used for the directional derivative around the nominal checking point. The
777 default is 1, that means no scaling.
779 EpsilonMinimumExponent
780 This key indicates the minimal exponent value of the power of 10 coefficient
781 to be used to decrease the increment multiplier. The default is -8, and it
782 has to be between 0 and -20. For example, its default value leads to
783 calculate the residue of the scalar product formula with a fixed increment
784 multiplied from 1.e0 to 1.e-8.
787 This key indicates the vector direction used for the directional derivative
788 around the nominal checking point. It has to be a vector. If not specified,
789 this direction defaults to a random perturbation around zero of the same
790 vector size than the checking point.
793 This key indicates the residue formula that has to be used for the test. The
794 default choice is "Taylor", and the possible ones are "Taylor" (residue of
795 the Taylor development of the operator, which has to decrease with the power
796 of 2 in perturbation) and "Norm" (residue obtained by taking the norm of the
797 Taylor development at zero order approximation, which approximate the
798 gradient, and which has to remain constant).
801 This key allow to give an integer in order to fix the seed of the random
802 generator used to generate the ensemble. A convenient value is for example
803 1000. By default, the seed is left uninitialized, and so use the default
804 initialization from the computer.
810 "ObservationOperator"*
812 AmplitudeOfInitialDirection
813 This key indicates the scaling of the initial perturbation build as a vector
814 used for the directional derivative around the nominal checking point. The
815 default is 1, that means no scaling.
817 EpsilonMinimumExponent
818 This key indicates the minimal exponent value of the power of 10 coefficient
819 to be used to decrease the increment multiplier. The default is -8, and it
820 has to be between 0 and -20. For example, its default value leads to
821 calculate the residue of the scalar product formula with a fixed increment
822 multiplied from 1.e0 to 1.e-8.
825 This key indicates the vector direction used for the directional derivative
826 around the nominal checking point. It has to be a vector. If not specified,
827 this direction defaults to a random perturbation around zero of the same
828 vector size than the checking point.
831 This key indicates the residue formula that has to be used for the test. The
832 default choice is "CenteredDL", and the possible ones are "CenteredDL"
833 (residue of the difference between the function at nominal point and the
834 values with positive and negative increments, which has to stay very small),
835 "Taylor" (residue of the Taylor development of the operator normalized by
836 the nominal value, which has to stay very small), "NominalTaylor" (residue
837 of the order 1 approximations of the operator, normalized to the nominal
838 point, which has to stay close to 1), and "NominalTaylorRMS" (residue of the
839 order 1 approximations of the operator, normalized by RMS to the nominal
840 point, which has to stay close to 0).
843 This key allow to give an integer in order to fix the seed of the random
844 generator used to generate the ensemble. A convenient value is for example
845 1000. By default, the seed is left uninitialized, and so use the default
846 initialization from the computer.
848 Requirements for functions describing an operator
849 -------------------------------------------------
851 The operators for observation and evolution are required to implement the data
852 assimilation or optimization procedures. They include the physical simulation
853 numerical simulations, but also the filtering and restriction to compare the
854 simulation to observation. The evolution operator is considered here in its
855 incremental form, representing the transition between two successive states, and
856 is then similar to the observation operator.
858 Schematically, an operator has to give a output solution given the input
859 parameters. Part of the input parameters can be modified during the optimization
860 procedure. So the mathematical representation of such a process is a function.
861 It was briefly described in the section :ref:`section_theory` and is generalized
862 here by the relation:
864 .. math:: \mathbf{y} = O( \mathbf{x} )
866 between the pseudo-observations :math:`\mathbf{y}` and the parameters
867 :math:`\mathbf{x}` using the observation or evolution operator :math:`O`. The
868 same functional representation can be used for the linear tangent model
869 :math:`\mathbf{O}` of :math:`O` and its adjoint :math:`\mathbf{O}^*`, also
870 required by some data assimilation or optimization algorithms.
872 Then, **to describe completely an operator, the user has only to provide a
873 function that fully and only realize the functional operation**.
875 This function is usually given as a script that can be executed in a YACS node.
876 This script can without difference launch external codes or use internal SALOME
877 calls and methods. If the algorithm requires the 3 aspects of the operator
878 (direct form, tangent form and adjoint form), the user has to give the 3
879 functions or to approximate them.
881 There are 3 practical methods for the user to provide an operator functional
882 representation. These methods are chosen in the "*FROM*" field of each operator
883 having a "*Function*" value as "*INPUT_TYPE*", as shown by the following figure:
885 .. eficas_operator_function:
886 .. image:: images/eficas_operator_function.png
890 **Choosing an operator functional representation**
892 First functional form: using "*ScriptWithOneFunction*"
893 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
895 .. index:: single: ScriptWithOneFunction
896 .. index:: single: DirectOperator
897 .. index:: single: DifferentialIncrement
898 .. index:: single: CenteredFiniteDifference
900 The first one consist in providing only one potentially non-linear function, and
901 to approximate the tangent and the adjoint operators. This is done by using the
902 keyword "*ScriptWithOneFunction*" for the description of the chosen operator in
903 the ADAO GUI. The user have to provide the function in a script, with a
904 mandatory name "*DirectOperator*". For example, the script can follow the
907 def DirectOperator( X ):
908 """ Direct non-linear simulation operator """
914 In this case, the user can also provide a value for the differential increment,
915 using through the GUI the keyword "*DifferentialIncrement*", which has a default
916 value of 1%. This coefficient will be used in the finite difference
917 approximation to build the tangent and adjoint operators. The finite difference
918 approximation order can also be chosen through the GUI, using the keyword
919 "*CenteredFiniteDifference*", with 0 for an uncentered schema of first order,
920 and with 1 for a centered schema of second order (of twice the first order
921 computational cost). The keyword has a default value of 0.
923 This first operator definition allow easily to test the functional form before
924 its use in an ADAO case, greatly reducing the complexity of implementation.
926 **Important warning:** the name "*DirectOperator*" is mandatory, and the type of
927 the X argument can be either a python list, a numpy array or a numpy 1D-matrix.
928 The user has to treat these cases in his script.
930 Second functional form: using "*ScriptWithFunctions*"
931 +++++++++++++++++++++++++++++++++++++++++++++++++++++
933 .. index:: single: ScriptWithFunctions
934 .. index:: single: DirectOperator
935 .. index:: single: TangentOperator
936 .. index:: single: AdjointOperator
938 The second one consist in providing directly the three associated operators
939 :math:`O`, :math:`\mathbf{O}` and :math:`\mathbf{O}^*`. This is done by using
940 the keyword "*ScriptWithFunctions*" for the description of the chosen operator
941 in the ADAO GUI. The user have to provide three functions in one script, with
942 three mandatory names "*DirectOperator*", "*TangentOperator*" and
943 "*AdjointOperator*". For example, the script can follow the template::
945 def DirectOperator( X ):
946 """ Direct non-linear simulation operator """
950 return something like Y
952 def TangentOperator( (X, dX) ):
953 """ Tangent linear operator, around X, applied to dX """
957 return something like Y
959 def AdjointOperator( (X, Y) ):
960 """ Adjoint operator, around X, applied to Y """
964 return something like X
966 Another time, this second operator definition allow easily to test the
967 functional forms before their use in an ADAO case, reducing the complexity of
970 **Important warning:** the names "*DirectOperator*", "*TangentOperator*" and
971 "*AdjointOperator*" are mandatory, and the type of the X, Y, dX arguments can be
972 either a python list, a numpy array or a numpy 1D-matrix. The user has to treat
973 these cases in his script.
975 Third functional form: using "*ScriptWithSwitch*"
976 +++++++++++++++++++++++++++++++++++++++++++++++++
978 .. index:: single: ScriptWithSwitch
979 .. index:: single: DirectOperator
980 .. index:: single: TangentOperator
981 .. index:: single: AdjointOperator
983 This third form give more possibilities to control the execution of the three
984 functions representing the operator, allowing advanced usage and control over
985 each execution of the simulation code. This is done by using the keyword
986 "*ScriptWithSwitch*" for the description of the chosen operator in the ADAO GUI.
987 The user have to provide a switch in one script to control the execution of the
988 direct, tangent and adjoint forms of its simulation code. The user can then, for
989 example, use other approximations for the tangent and adjoint codes, or
990 introduce more complexity in the argument treatment of the functions. But it
991 will be far more complicated to implement and debug.
993 **It is recommended not to use this third functional form without a solid
994 numerical or physical reason.**
996 If, however, you want to use this third form, we recommend using the following
997 template for the switch. It requires an external script or code named
998 "*Physical_simulation_functions.py*", containing three functions named
999 "*DirectOperator*", "*TangentOperator*" and "*AdjointOperator*" as previously.
1000 Here is the switch template::
1002 import Physical_simulation_functions
1003 import numpy, logging
1006 for param in computation["specificParameters"]:
1007 if param["name"] == "method":
1008 method = param["value"]
1009 if method not in ["Direct", "Tangent", "Adjoint"]:
1010 raise ValueError("No valid computation method is given")
1011 logging.info("Found method is \'%s\'"%method)
1013 logging.info("Loading operator functions")
1014 Function = Physical_simulation_functions.DirectOperator
1015 Tangent = Physical_simulation_functions.TangentOperator
1016 Adjoint = Physical_simulation_functions.AdjointOperator
1018 logging.info("Executing the possible computations")
1020 if method == "Direct":
1021 logging.info("Direct computation")
1022 Xcurrent = computation["inputValues"][0][0][0]
1023 data = Function(numpy.matrix( Xcurrent ).T)
1024 if method == "Tangent":
1025 logging.info("Tangent computation")
1026 Xcurrent = computation["inputValues"][0][0][0]
1027 dXcurrent = computation["inputValues"][0][0][1]
1028 data = Tangent(numpy.matrix(Xcurrent).T, numpy.matrix(dXcurrent).T)
1029 if method == "Adjoint":
1030 logging.info("Adjoint computation")
1031 Xcurrent = computation["inputValues"][0][0][0]
1032 Ycurrent = computation["inputValues"][0][0][1]
1033 data = Adjoint((numpy.matrix(Xcurrent).T, numpy.matrix(Ycurrent).T))
1035 logging.info("Formatting the output")
1036 it = numpy.ravel(data)
1037 outputValues = [[[[]]]]
1039 outputValues[0][0][0].append(val)
1042 result["outputValues"] = outputValues
1043 result["specificOutputInfos"] = []
1044 result["returnCode"] = 0
1045 result["errorMessage"] = ""
1047 All various modifications could be done from this template hypothesis.
1049 Special case of controled evolution operator
1050 ++++++++++++++++++++++++++++++++++++++++++++
1052 In some cases, the evolution or the observation operators are required to be
1053 controled by an external input control, given a priori. In this case, the
1054 generic form of the incremental evolution model is slightly modified as follows:
1056 .. math:: \mathbf{y} = O( \mathbf{x}, \mathbf{u})
1058 where :math:`\mathbf{u}` is the control over one state increment. In this case,
1059 the direct operator has to be applied to a pair of variables :math:`(X,U)`.
1060 Schematically, the operator has to be set as::
1062 def DirectOperator( (X, U) ):
1063 """ Direct non-linear simulation operator """
1067 return something like X(n+1) or Y(n+1)
1069 The tangent and adjoint operators have the same signature as previously, noting
1070 that the derivatives has to be done only partially against :math:`\mathbf{x}`.
1071 In such a case with explicit control, only the second functional form (using
1072 "*ScriptWithFunctions*") and third functional form (using "*ScriptWithSwitch*")
1075 Requirements to describe covariance matrices
1076 --------------------------------------------
1078 Multiple covariance matrices are required to implement the data assimilation or
1079 optimization procedures. The main ones are the background error covariance
1080 matrix, noted as :math:`\mathbf{B}` and the observation error covariance matrix,
1081 noted as :math:`\mathbf{R}`. Such a matrix is required to be a squared symetric
1082 semi-definite positive matrix.
1084 There are 3 practical methods for the user to provide a covariance matrix. These
1085 methods are chosen as the "*INPUT_TYPE*" of each defined covariance matrix, as
1086 shown by the following figure:
1088 .. eficas_covariance_matrix:
1089 .. image:: images/eficas_covariance_matrix.png
1093 **Choosing covariance matrix representation**
1095 First matrix form: using "*Matrix*" representation
1096 ++++++++++++++++++++++++++++++++++++++++++++++++++
1098 .. index:: single: Matrix
1099 .. index:: single: BackgroundError
1100 .. index:: single: EvolutionError
1101 .. index:: single: ObservationError
1103 This first form is the default and more general one. The covariance matrix
1104 :math:`\mathbf{M}` has to be fully specified. Even if the matrix is symetric by
1105 nature, the entire :math:`\mathbf{M}` matrix has to be given.
1107 .. math:: \mathbf{M} = \begin{pmatrix}
1108 m_{11} & m_{12} & \cdots & m_{1n} \\
1109 m_{21} & m_{22} & \cdots & m_{2n} \\
1110 \vdots & \vdots & \vdots & \vdots \\
1111 m_{n1} & \cdots & m_{nn-1} & m_{nn}
1114 It can be either a Python Numpy array or a matrix, or a list of lists of values
1115 (that is, a list of rows). For example, a simple diagonal unitary background error
1116 covariance matrix :math:`\mathbf{B}` can be described in a python script as::
1118 BackgroundError = numpy.eye(...)
1122 BackgroundError = [[1, 0 ... 0], [0, 1 ... 0] ... [0, 0 ... 1]]
1124 Second matrix form: using "*ScalarSparseMatrix*" representation
1125 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1127 .. index:: single: ScalarSparseMatrix
1128 .. index:: single: BackgroundError
1129 .. index:: single: EvolutionError
1130 .. index:: single: ObservationError
1132 On the opposite, this second form is a very simplified method to provide a
1133 matrix. The covariance matrix :math:`\mathbf{M}` is supposed to be a positive
1134 multiple of the identity matrix. The matrix can then be specified only by this
1137 .. math:: \mathbf{M} = m \times \begin{pmatrix}
1138 1 & 0 & \cdots & 0 \\
1139 0 & 1 & \cdots & 0 \\
1140 \vdots & \vdots & \vdots & \vdots \\
1144 The multiplier :math:`m` has to be a floating point or integer positive value
1145 (if it is negative, which is impossible, converted to positive value). For example, a simple diagonal unitary background error
1146 covariance matrix :math:`\mathbf{B}` can be described in a python script as::
1148 BackgroundError = 1.
1150 or, better, by a "*String*" directly in the ADAO case.
1152 Third matrix form: using "*DiagonalSparseMatrix*" representation
1153 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1155 .. index:: single: DiagonalSparseMatrix
1156 .. index:: single: BackgroundError
1157 .. index:: single: EvolutionError
1158 .. index:: single: ObservationError
1160 This third form is also a simplified method to provide a matrix, but a little
1161 more powerful. The covariance matrix :math:`\mathbf{M}` is already supposed to
1162 be diagonal, but the user has to specify all the positive diagonal values. The
1163 matrix can then be specified only by a vector :math:`\mathbf{V}` which will be
1164 set on a diagonal matrix:
1166 .. math:: \mathbf{M} = \begin{pmatrix}
1167 v_{1} & 0 & \cdots & 0 \\
1168 0 & v_{2} & \cdots & 0 \\
1169 \vdots & \vdots & \vdots & \vdots \\
1170 0 & \cdots & 0 & v_{n}
1173 It can be either a Python Numpy array or a matrix, or a list or a list of list
1174 of positive values (if some are negative, which is impossible, converted to
1175 positive values). For example, a simple diagonal unitary background error
1176 covariance matrix :math:`\mathbf{B}` can be described in a python script as::
1178 BackgroundError = [1, 1 ... 1]
1182 BackgroundError = numpy.ones(...)