2 Copyright (C) 2008-2024 EDF R&D
4 This file is part of SALOME ADAO module.
6 This library is free software; you can redistribute it and/or
7 modify it under the terms of the GNU Lesser General Public
8 License as published by the Free Software Foundation; either
9 version 2.1 of the License, or (at your option) any later version.
11 This library is distributed in the hope that it will be useful,
12 but WITHOUT ANY WARRANTY; without even the implied warranty of
13 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
14 Lesser General Public License for more details.
16 You should have received a copy of the GNU Lesser General Public
17 License along with this library; if not, write to the Free Software
18 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
20 See http://www.salome-platform.org/ or email : webmaster.salome@opencascade.com
22 Author: Jean-Philippe Argaud, jean-philippe.argaud@edf.fr, EDF R&D
26 ================================================================================
27 **[DocU]** Advanced usage of the ADAO module and interoperability
28 ================================================================================
30 This section presents advanced methods to use the ADAO module, how to get more
31 information during calculation, or how to use it without the graphical user
32 interface (GUI). It requires to know how to find files or commands included
33 inside the whole SALOME installation. All the names to be replaced by user are
34 indicated by the syntax ``<...>``.
36 .. _section_advanced_convert_JDC:
38 Converting and executing an ADAO command file (JDC) using a Shell script
39 ------------------------------------------------------------------------
41 It is possible to convert and execute an ADAO command file (JDC, or ".comm/.py"
42 files pair, which resides in ``<ADAO JDC file directory>``) automatically by
43 using a template Shell script containing all the required steps. If the SALOME
44 main launching command , named ``salome``, is not available in a classical
45 terminal, the user has to know where are the main SALOME launching files, and
46 in particular the ``salome`` one. The directory in which this script resides is
47 symbolically named ``<SALOME main installation dir>`` and has to be replaced by
48 the good one in the Shell file template.
50 When an ADAO command file is build by the ADAO graphical editor and saved, if
51 it is named for example "AdaoStudy1.comm", then a companion file named
52 "AdaoStudy1.py" is automatically created in the same directory. It is named
53 ``<ADAO Python file>`` in the template, and it is converted to YACS as an
54 ``<ADAO YACS xml scheme>`` as a ".xml" file named "AdaoStudy1.xml". After that,
55 this last one can be executed in console mode using the standard YACS console
56 command (see YACS documentation for more information).
58 In all launching command Shell files examples, we choose to start and stop the
59 SALOME application server in the same script. It is not mandatory, but it is
60 useful to avoid stalling SALOME sessions.
62 The simplest example consist in only launching the given YACS scheme, which was
63 previously generated by the user in the graphical interface. In this case,
64 after having replaced the strings between ``<...>`` symbols, one needs only to
65 save the following Shell script::
68 USERDIR="<ADAO JDC file directory>"
69 SALOMEDIR="<SALOME main installation directory>"
70 $SALOMEDIR/salome start -k -t
71 $SALOMEDIR/salome shell -- "driver $USERDIR/<ADAO YACS xml scheme>"
72 $SALOMEDIR/salome shell killSalome.py
74 It is then required to change it to be in executable mode.
76 A more complete example consist in launching execution of a YACS scheme given
77 by the user, having previously verified its availability. For that, replacing
78 the text ``<SALOME main installation directory>``, one needs only to save the
79 following Shell script::
84 echo -e "\nUsage: $0 <ADAO YACS xml scheme>\n"
89 if (test ! -e $USERFILE)
91 echo -e "\nError : the XML file named $USERFILE does not exist.\n"
94 SALOMEDIR="<SALOME main installation directory>"
95 $SALOMEDIR/salome start -k -t
96 $SALOMEDIR/salome shell -- "driver $USERFILE"
97 $SALOMEDIR/salome shell killSalome.py
100 An another example consist in adding the conversion of the ADAO command file
101 (JDC, or ".comm/.py" files pair) in an associated YACS scheme (".xml" file). At
102 the end of the script, one choose also to remove the ``<ADAO YACS xml scheme>``
103 because it is a generated file. For that, after having carefully replaced the
104 text ``<SALOME main installation directory>``, one needs only to save the
105 following Shell script::
110 echo -e "\nUsage: $0 <ADAO .comm/.py case>\n"
114 F=`basename -s .comm $1`
115 F=`basename -s .py $F`
118 if (test ! -e $USERFILE.py)
120 echo -e "\nError : the PY file named $USERFILE.py does not exist.\n"
123 SALOMEDIR="<SALOME main installation directory>"
124 $SALOMEDIR/salome start -k -t
125 $SALOMEDIR/salome shell -- "python $SALOMEDIR/bin/AdaoYacsSchemaCreator.py $USERFILE.py $USERFILE.xml"
126 $SALOMEDIR/salome shell -- "driver $USERFILE.xml"
127 $SALOMEDIR/salome shell killSalome.py
131 In all cases, the standard output and errors come in the launching terminal.
133 .. _section_advanced_YACS_tui:
135 Running an ADAO calculation scheme in YACS using the text user mode (YACS TUI)
136 ------------------------------------------------------------------------------
138 This section describes how to execute in TUI (Text User Interface) YACS mode a
139 YACS calculation scheme, obtained in the graphical interface by using the ADAO
140 "Export to YACS" function. It uses the standard YACS TUI mode, which is briefly
141 recalled here (see YACS documentation for more information) through a simple
142 example. As described in documentation, a XML scheme can be loaded in a Python.
143 We give here a whole sequence of command lines to test the validity of the
144 scheme before executing it, adding some initial supplementary ones to
145 explicitly load the types catalog to avoid weird difficulties::
150 SALOMERuntime.RuntimeSALOME_setRuntime()
152 r = pilot.getRuntime()
153 xmlLoader = loader.YACSLoader()
154 xmlLoader.registerProcCataLoader()
156 catalogAd = r.loadCatalog("proc", "<ADAO YACS xml scheme>")
157 r.addCatalog(catalogAd)
162 p = xmlLoader.load("<ADAO YACS xml scheme>")
164 print("IO exception:",ex)
166 logger = p.getLogger("parser")
167 if not logger.isEmpty():
168 print("The imported file has errors :")
169 print(logger.getStr())
172 print("The schema is not valid and can not be executed")
173 print(p.getErrorReport())
175 info=pilot.LinkInfo(pilot.LinkInfo.ALL_DONT_STOP)
176 p.checkConsistency(info)
177 if info.areWarningsOrErrors():
178 print("The schema is not consistent and can not be executed")
179 print(info.getGlobalRepr())
181 e = pilot.ExecutorSwig()
183 if p.getEffectiveState() != pilot.DONE:
184 print(p.getErrorReport())
186 This method allows for example to edit the YACS XML scheme in TUI, or to gather
187 results for further use.
189 .. _section_advanced_R:
191 Running an ADAO calculation in R environment using the TUI ADAO interface
192 -------------------------------------------------------------------------
195 .. index:: single: rPython
196 .. index:: single: reticulate
198 To extend the analysis and treatment capacities, it is possible to use ADAO
199 calculations in **R** environment (see [R]_ for more details). It is available
200 in SALOME by launching the R interpreter in the shell "``salome shell``".
201 Moreover, the package "*rPython*" (or the more recent "*reticulate*" one) has
202 to be available, it can be installed by the user if required by the following R
206 # IMPORTANT: to be run in R interpreter
207 # -------------------------------------
208 install.packages("rPython")
210 One will refer to the [GilBellosta15]_ documentation for more information on
213 The ADAO calculations defined in text interface (API/TUI, see
214 :ref:`section_tui`) can be interpreted from the R environment, using some data
215 and information from R. The approach is illustrated in the example
216 :ref:`subsection_tui_example`, suggested in the API/TUI interface description.
217 In the R interpreter, one can run the following commands, directly coming from
221 # IMPORTANT: to be run in R interpreter
222 # -------------------------------------
225 from numpy import array
226 from adao import adaoBuilder
227 case = adaoBuilder.New()
228 case.set( 'AlgorithmParameters', Algorithm='3DVAR' )
229 case.set( 'Background', Vector=[0, 1, 2] )
230 case.set( 'BackgroundError', ScalarSparseMatrix=1.0 )
231 case.set( 'Observation', Vector=array([0.5, 1.5, 2.5]) )
232 case.set( 'ObservationError', DiagonalSparseMatrix='1 1 1' )
233 case.set( 'ObservationOperator', Matrix='1 0 0;0 2 0;0 0 3' )
234 case.set( 'Observer', Variable='Analysis', Template='ValuePrinter' )
240 Analysis [ 0.25000264 0.79999797 0.94999939]
242 In writing the ADAO calculations run from R, one must take close attention to
243 the good use of single and double quotes, that should not collide between the
246 The data can come from the R environment and should be stored in properly
247 assigned variables to be used later in Python for ADAO. One will refer to the
248 [GilBellosta15]_ documentation for the implementation work. We can transform the
249 above example to use data from R to feed the three variables of background,
250 observation and observation operator. We get in the end the optimal state also
251 in a R variable. The other lines are identical. The example thus becomes::
254 # IMPORTANT: to be run in R interpreter
255 # -------------------------------------
260 yo <- c(0.5, 1.5, 2.5)
261 h <- '1 0 0;0 2 0;0 0 3'
266 python.assign( "xb", xb )
267 python.assign( "yo", yo )
268 python.assign( "h", h )
270 from numpy import array
271 from adao import adaoBuilder
272 case = adaoBuilder.New()
273 case.set( 'AlgorithmParameters', Algorithm='3DVAR' )
274 case.set( 'Background', Vector=xb )
275 case.set( 'BackgroundError', ScalarSparseMatrix=1.0 )
276 case.set( 'Observation', Vector=array(yo) )
277 case.set( 'ObservationError', DiagonalSparseMatrix='1 1 1' )
278 case.set( 'ObservationOperator', Matrix=str(h) )
279 case.set( 'Observer', Variable='Analysis', Template='ValuePrinter' )
281 xa = list(case.get('Analysis')[-1])
286 xa <- python.get("xa")
288 One notices the explicit ``str`` and ``list`` type conversions to ensure that
289 the data are transmitted as known standard types from "*rPython*" package.
290 Moreover, it is the data that can be transferred between the two languages, not
291 functions or methods. It is therefore necessary to prepare generically in
292 Python the functions to execute required by ADAO, and to forward them correctly
293 the data available in R.
295 The most comprehensive cases, proposed in :ref:`subsection_tui_advanced`, can
296 be executed in the same way, and they give the same result as in the standard
299 .. _section_advanced_eficas_gui:
301 Using the ADAO EFICAS graphical interface as an ADAO TUI command
302 ----------------------------------------------------------------
304 To make it easier to quickly edit an ADAO command file with ADAO EFICAS (JDC,
305 or pair of files ".comm/.py", that are together in a user study directory), you
306 can launch the graphical user interface from the Python interpreter. To do
307 this, in a Python interpreter obtained from the "SALOME shell", the following
310 from adao import adaoBuilder
313 As a reminder, the easiest way to get a Python interpreter included in a
314 "SALOME shell" session is to run the following command in a terminal::
316 $SALOMEDIR/salome shell -- python
318 with ``SALOMEDIR`` the ``<SALOME main installation directory>``.
320 If necessary, explicit messages can be used to identify the required
321 environment variables that are missing. However, **this command should not be
322 run in the SALOME Python console** (because in this case it is enough to
323 activate the module since we already are in the graphical environment...) or in
324 an independent Python install, but it can be run in a "SALOME shell" session
325 obtained from the "Tools/Extensions" menu of SALOME.
327 .. _section_advanced_execution_mode:
329 Change the default execution mode of nodes in YACS
330 --------------------------------------------------
332 .. index:: single: YACS
333 .. index:: single: ExecuteInContainer
335 Various reasons may lead to want to change the default mode of node execution
336 in YACS (see [#]_ for the correct use of these possibilities). This may be for
337 performance reasons, or for example for reasons of resource conflicts.
339 One may want to use this change in execution mode to extend the use of local
340 computing resources or to set remote calculations for a node that requires it.
341 This is particularly the case for a node that should use a simulation resource
342 available on a cluster, for example.
344 In addition, the various calculations that are carried out (user-provided
345 operators, results retrieval functions, etc.) may also present conflicts if
346 they are performed in a single process, and in particular in the main process
347 of SALOME. This is the default YACS operating mode for performance and
348 simplicity reasons. However, it is recommended to change this functioning when
349 encountering execution instabilities or error messages in the graphical
352 In any case, in the YACS schema being edited, it is sufficient to change the
353 execution mode of the node(s) that require it. They have to be executed in a
354 new container created for the occasion (it is not enough to use the default
355 container, it is explicitly necessary to create a new one) and whose properties
356 are adapted to the intended use. The procedure is therefore as follows:
358 #. Create a new YACS container, using the context menu in the tree view of the YACS schema (usually on the left),
359 #. Adapt the characteristics of the container, for example by selecting a "*type*" property with the value "*multi*" for a truly parallel execution, or by choosing a remote computing resource defined by the "*Resource*" property, or by using advanced parameters,
360 #. Graphically select in the central view the node whose execution mode you want to change,
361 #. In the panel to the right of the node entries, unfold the execution choices (named "*Execution Mode*"), check the "*Container*" box instead of the "*YACS*" default, and choose the newly created container (it is usually named "*container0*"),
362 #. Save the modified schema
364 This can be repeated for each node that requires it, by reusing the same new
365 container for all nodes, or by creating a new container for each node.
367 A more generic way to impose a global execution in a separate container is to
368 use a variable named "*ExecuteInContainer*". This variable is available for
369 ADAO cases through graphical user interface (GUI) or the textual one (it is for
370 example available by default in the :ref:`section_ref_assimilation_keywords`).
374 This change in execution mode is extremely powerful and flexible. It is
375 therefore recommended that the user both use it, and at the same time be
376 attentive to the interaction of the different choices he makes, to avoid, for
377 example, an unintended performance deterioration, or computer conflicts that
378 are complicated to diagnose.
380 .. _section_advanced_observer:
382 Getting information on special variables during the ADAO calculation
383 --------------------------------------------------------------------
385 .. index:: single: Observer
386 .. index:: single: Observer Template
388 Some special internal optimization variables, used during calculations, can be
389 monitored during the ADAO calculation. These variables can be printed, plotted,
390 saved, etc. This can be done using "*observer*", that are commands gathered in
391 scripts, each associated with one variable.
393 Some templates are available when editing the ADAO case in graphical editor.
394 These simple scripts can be customized by the user, either at the embedded
395 edition stage, or at the edition stage before execution, to improve the tuning
396 of the ADAO calculation.
398 To implement these "*observer*" efficiently, one can look to the
399 :ref:`section_ref_observers_requirements`.
401 .. _section_advanced_logging:
403 Getting more information when running a calculation
404 ---------------------------------------------------
406 .. index:: single: Logging
407 .. index:: single: Debug
408 .. index:: single: setDebug
410 When running a calculation, useful data and messages are logged. There are two
411 ways to obtain theses information.
413 The first one, and the preferred way, is to use the built-in variable "*Debug*"
414 available in every ADAO case. It can be reached in the module's graphical user
415 interface (GUI) as well as in the textual interface (TUI). Setting it to "*1*"
416 will send messages in the log window of the YACS scheme execution.
418 The second one consist in using the "*logging*" native module of Python (see
419 the Python documentation http://docs.python.org/library/logging.html for more
420 information on this module). Everywhere in the YACS scheme, mainly through the
421 scripts entries, the user can set the logging level in accordance to the needs
422 of detailed information. The different logging levels are: "*DEBUG*", "*INFO*",
423 "*WARNING*", "*ERROR*", "*CRITICAL*". All the information flagged with a
424 certain level will be printed for whatever activated level above this
425 particular one (included). The easiest way is to change the log level by using
426 the following Python lines::
429 logging.getLogger().setLevel(logging.DEBUG)
431 The standard logging module default level is "*WARNING*", the default level in
432 the ADAO module is "*INFO*".
434 It is also recommended to include logging monitoring or debugging mechanisms in
435 the user's physical simulation code, and to exploit them in conjunction with
436 the previous two methods. But be careful not to store "too big" variables
437 because it cost time or memory, whatever logging level is chosen (that is, even
438 if these variables are not printed).
440 .. _subsection_ref_parallel_df:
442 Accelerating numerical derivatives calculations by using a parallel mode
443 ------------------------------------------------------------------------
445 .. index:: single: EnableWiseParallelism
446 .. index:: single: NumberOfProcesses
448 When setting an operator, as described in
449 :ref:`section_ref_operator_requirements`, the user can choose a functional form
450 "*ScriptWithOneFunction*". This form explicitly leads to approximate the
451 tangent and adjoint operators (if they are required) by a finite differences
452 calculation. It requires several calls to the direct operator (which is the
453 user defined function), at least as many times as the dimension of the state
454 vector. This are these calls that can potentially be executed in parallel.
456 Under some conditions (described right after), it is then possible to
457 accelerate the numerical derivatives calculations by using a parallel mode for
458 the finite differences approximation. When setting up an ADAO case, it is done
459 by adding the optional keyword "*EnableWiseParallelism*", set to "1" or
460 "*True*". This keyword is included in the "*SCRIPTWITHONEFUNCTION*" command in
461 the operator definition by graphical interface, or in the "*Parameters*"
462 accompanying the command "*OneFunction*" by textual interaface. By default,
463 this parallel mode is disabled ("*EnableWiseParallelism=0*"). The parallel mode
464 will only use local resources (both multi-cores or multi-processors) of the
465 computer on which execution is running, requiring by default as many resources
466 as available. If necessary, one can reduce the available resources by limiting
467 the possible number of parallel processes using the keyword
468 "*NumberOfProcesses*", set to desired maximum number (or to "0" for automatic
469 control, which is the default value).
471 The main conditions to perform parallel calculations come from the user defined
472 function, that represents the direct operator. This function has at least to be
473 "thread safe" to be executed in Python parallel environment (notions out of
474 scope of this paragraph). It is not obvious to give general rules, so it's
475 recommended, for the user who enable this internal parallelism, to carefully
476 verify his function and the obtained results.
478 From a user point of view, some conditions, that have to be met to set up
479 parallel calculations for tangent and the adjoint operators approximations, are
482 #. The dimension of the state vector is more than 2 or 3.
483 #. Unitary calculation of user defined direct function "last for long time", that is, more than few minutes.
484 #. The user defined direct function does not already use parallelism (or parallel execution is disabled in the user calculation).
485 #. The user defined direct function avoids read/write access to common resources, mainly stored data, output files or memory capacities.
486 #. The "*observer*" added by the user avoid read/write access to common resources, such as files or memory.
488 If these conditions are satisfied, the user can choose to enable the internal
489 parallelism for the numerical derivative calculations. Despite the simplicity of
490 activating, by setting one variable only, the user is urged to verify the
491 results of its calculations. One must at least doing them one time with
492 parallelism enabled, and an another time with parallelism disabled, to compare
493 the results. If it does fail somewhere, you have to know that this parallel
494 scheme is working for complex codes, like *Code_Aster* in *SalomeMeca*
495 [SalomeMeca]_ for example. So, if it does not work in your case, check your
496 operator function before and during enabling parallelism...
500 In case of doubt, it is recommended NOT TO ACTIVATE this parallelism.
502 It is also recalled that one have to choose the type "*multi*" for the default
503 container in order to launch the scheme, to allow a really parallel execution.
505 .. _subsection_iterative_convergence_control:
507 Convergence control for calculation cases and iterative algorithms
508 ------------------------------------------------------------------
510 .. index:: single: Convergence
511 .. index:: single: Iterative convergence
513 There are many reasons to want to control the convergence of available
514 calculation cases or algorithms in ADAO. For example, one may want
515 *repeatability* of optimal solutions, certified *quality*, *stability* of
516 optimal search conditions in studies, *saving of global computation time*, etc.
517 Moreover, we notice that the methods used in ADAO are frequently iterative,
518 reinforcing the interest of this convergence control.
520 By default, **the available calculation cases or algorithms in ADAO give access
521 to multiple ways to control their convergence, specially adapted to each
522 method**. These controls are derived from classical optimization theory and
523 from the possibilities of each algorithm. The default values of the controls
524 are chosen to ensure an optimal search for high quality simulation functions
525 with "*standard*" behavior (regularity, physical and numerical quality...),
526 which is not necessarily the main property of real simulations due to various
527 constraints. It is therefore quite normal to adapt the convergence criteria to
528 the study cases encountered, but it is an expert approach to establish the
531 There are fairly generic ways to control the optimal search and the convergence
532 of algorithms. We indicate here the most useful ones, in a non-exhaustive way,
533 and with the significant restriction that there are many exceptions to the
534 recommendations made. To go further, this generic information must be completed
535 by the information specific to each algorithm or calculation case, indicated in
536 the documentation of the different :ref:`section_reference_assimilation`.
538 **A first way is to limit the default number of iterations in the iterative
539 search processes**. Even if this is not the best theoretical way to control the
540 algorithm, it is very effective in a real study process. For this purpose, the
541 keyword "*MaximumNumberOfIterations*" exists in all cases of calculations that
542 support it, and its default value is usually set to an equivalent of infinity
543 so that it is not the stopping criterion. This is the case for calculations
544 based on variational methods such as :ref:`section_ref_algorithm_3DVAR`,
545 :ref:`section_ref_algorithm_4DVAR` and
546 :ref:`section_ref_algorithm_NonLinearLeastSquares`, but this is also the case
547 for other ones like the :ref:`section_ref_algorithm_DerivativeFreeOptimization`
548 or :ref:`section_ref_algorithm_QuantileRegression`. In practice, a value
549 between 10 and 30 is recommended to make this control parameter effective and
550 still obtain an optimal search of good quality. For an optimal search of
551 sufficient quality, this restriction should not be set too strictly, i.e. a 30
552 iteration limit should be more favorable than a 10 iteration limit.
554 **A second way to control convergence is to adapt the relative decrement
555 tolerance in the minimization of the cost functional considered**. This
556 tolerance is controlled by the keyword "*CostDecrementTolerance*" in the
557 algorithms that support it. The default value is rather strict, it is chosen
558 for a theoretical convergence control when the numerical simulations are of
559 high numerical quality. In practice, it can be adapted without hesitation to be
560 between :math:`10^{-5}` and :math:`10^{-2}`. This adaptation allows in
561 particular to reduce or avoid the difficulties of optimal search which are
562 manifested by many successive iterations on almost identical states.
564 **A third way to improve convergence is to adapt the default setting of the
565 finite difference approximation, primarily for the observation operator and a
566 single-operator representation**. The control of this property is done with the
567 keyword "*DifferentialIncrement*" which sets the definition using the
568 :ref:`section_ref_operator_one`. Its default value is :math:`10^{-2}` (or 1%),
569 and it can usually be adjusted between :math:`10^{-5}` and :math:`10^{-3}`
570 (although it is wise to check carefully the relevance of its value, it is easy
571 in ADAO to change this parameter). The convergence criterion must then be
572 adjusted so that it does not exceed the order of magnitude of this
573 approximation. In practice, it is sufficient to set the
574 "*CostDecrementTolerance*" criterion to approximately the same precision (i.e.
575 with an order of magnitude more or less) as the "*DifferentialIncrement*"
576 criterion. This way of improvement is also to be completed with analyses using
577 the :ref:`section_ref_algorithm_LinearityTest` and
578 :ref:`section_ref_algorithm_GradientTest`.
580 From experience, it is *a priori* not recommended to use other means to control
581 convergence, even if they exist. These parameter adjustments are simple to
582 implement, and it is favorable to try them (in twin experiments or not) because
583 they solve many problems encountered in practice.
585 .. [#] For more information on YACS, see the *YACS module* and its integrated help available from the main menu *Help* of the SALOME platform.