2 Copyright (C) 2008-2020 EDF R&D
4 This file is part of SALOME ADAO module.
6 This library is free software; you can redistribute it and/or
7 modify it under the terms of the GNU Lesser General Public
8 License as published by the Free Software Foundation; either
9 version 2.1 of the License, or (at your option) any later version.
11 This library is distributed in the hope that it will be useful,
12 but WITHOUT ANY WARRANTY; without even the implied warranty of
13 MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
14 Lesser General Public License for more details.
16 You should have received a copy of the GNU Lesser General Public
17 License along with this library; if not, write to the Free Software
18 Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
20 See http://www.salome-platform.org/ or email : webmaster.salome@opencascade.com
22 Author: Jean-Philippe Argaud, jean-philippe.argaud@edf.fr, EDF R&D
26 ================================================================================
27 **[DocU]** Advanced usage of the ADAO module
28 ================================================================================
30 This section presents advanced methods to use the ADAO module, how to get more
31 information during calculation, or how to use it without the graphical user
32 interface (GUI). It requires to know how to find files or commands included
33 inside the whole SALOME installation. All the names to be replaced by user are
34 indicated by the syntax ``<...>``.
36 Converting and executing an ADAO command file (JDC) using a Shell script
37 ------------------------------------------------------------------------
39 It is possible to convert and execute an ADAO command file (JDC, or ".comm/.py"
40 files pair, which resides in ``<ADAO JDC file directory>``) automatically by
41 using a template Shell script containing all the required steps. If the SALOME
42 main launching command , named ``salome``, is not available in a classical
43 terminal, the user has to know where are the main SALOME launching files, and in
44 particular the ``salome`` one. The directory in which this script resides is
45 symbolically named ``<SALOME main installation dir>`` and has to be replaced by
46 the good one in the Shell file template.
48 When an ADAO command file is build by the ADAO graphical editor and saved, if it
49 is named for example "AdaoStudy1.comm", then a companion file named
50 "AdaoStudy1.py" is automatically created in the same directory. It is named
51 ``<ADAO Python file>`` in the template, and it is converted to YACS as an
52 ``<ADAO YACS xml scheme>`` as a ".xml" file named "AdaoStudy1.xml". After that,
53 this last one can be executed in console mode using the standard YACS console
54 command (see YACS documentation for more information).
56 In all launching command Shell files examples, we choose to start and stop the
57 SALOME application server in the same script. It is not mandatory, but it is
58 useful to avoid stalling SALOME sessions.
60 The simplest example consist in only launching the given YACS sheme, which was
61 previously generated by the user in the graphical interface. In this case, after
62 having replaced the strings between ``<...>`` symbols, one needs only to save
63 the following Shell script::
66 USERDIR="<ADAO JDC file directory>"
67 SALOMEDIR="<SALOME main installation directory>"
68 $SALOMEDIR/salome start -k -t
69 $SALOMEDIR/salome shell -- "driver $USERDIR/<ADAO YACS xml scheme>"
70 $SALOMEDIR/salome shell killSalome.py
72 It is then required to change it to be in executable mode.
74 A more complete example consist in launching execution of a YACS scheme given by
75 the user, having previously verified its availability. For that, replacing the
76 text ``<SALOME main installation directory>``, one needs only to save the
77 following Shell script::
82 echo -e "\nUsage: $0 <ADAO YACS xml scheme>\n"
87 if (test ! -e $USERFILE)
89 echo -e "\nError : the XML file named $USERFILE does not exist.\n"
92 SALOMEDIR="<SALOME main installation directory>"
93 $SALOMEDIR/salome start -k -t
94 $SALOMEDIR/salome shell -- "driver $USERFILE"
95 $SALOMEDIR/salome shell killSalome.py
98 An another example consist in adding the conversion of the ADAO command file
99 (JDC, or ".comm/.py" files pair) in an associated YACS scheme (".xml" file). At
100 the end of the script, one choose also to remove the ``<ADAO YACS xml scheme>``
101 because it is a generated file. For that, after having carefully replaced the
102 text ``<SALOME main installation directory>``, one needs only to save the
103 following Shell script::
108 echo -e "\nUsage: $0 <ADAO .comm/.py case>\n"
112 F=`basename -s .comm $1`
113 F=`basename -s .py $F`
116 if (test ! -e $USERFILE.py)
118 echo -e "\nError : the PY file named $USERFILE.py does not exist.\n"
121 SALOMEDIR="<SALOME main installation directory>"
122 $SALOMEDIR/salome start -k -t
123 $SALOMEDIR/salome shell -- "python $SALOMEDIR/bin/AdaoYacsSchemaCreator.py $USERFILE.py $USERFILE.xml"
124 $SALOMEDIR/salome shell -- "driver $USERFILE.xml"
125 $SALOMEDIR/salome shell killSalome.py
129 In all cases, the standard output and errors come in the launching terminal.
131 Running an ADAO calculation scheme in YACS using the text user mode (YACS TUI)
132 ------------------------------------------------------------------------------
134 This section describes how to execute in TUI (Text User Interface) YACS mode a
135 YACS calculation scheme, obtained in the graphical interface by using the ADAO
136 "Export to YACS" function. It uses the standard YACS TUI mode, which is briefly
137 recalled here (see YACS documentation for more information) through a simple
138 example. As described in documentation, a XML scheme can be loaded in a Python.
139 We give here a whole sequence of command lines to test the validity of the
140 scheme before executing it, adding some initial supplementary ones to explicitly
141 load the types catalog to avoid weird difficulties::
146 SALOMERuntime.RuntimeSALOME_setRuntime()
148 r = pilot.getRuntime()
149 xmlLoader = loader.YACSLoader()
150 xmlLoader.registerProcCataLoader()
152 catalogAd = r.loadCatalog("proc", "<ADAO YACS xml scheme>")
153 r.addCatalog(catalogAd)
158 p = xmlLoader.load("<ADAO YACS xml scheme>")
160 print("IO exception:",ex)
162 logger = p.getLogger("parser")
163 if not logger.isEmpty():
164 print("The imported file has errors :")
165 print(logger.getStr())
168 print("The schema is not valid and can not be executed")
169 print(p.getErrorReport())
171 info=pilot.LinkInfo(pilot.LinkInfo.ALL_DONT_STOP)
172 p.checkConsistency(info)
173 if info.areWarningsOrErrors():
174 print("The schema is not consistent and can not be executed")
175 print(info.getGlobalRepr())
177 e = pilot.ExecutorSwig()
179 if p.getEffectiveState() != pilot.DONE:
180 print(p.getErrorReport())
182 This method allows for example to edit the YACS XML scheme in TUI, or to gather
183 results for further use.
185 .. _section_advanced_R:
187 Running an ADAO calculation in R environment using the TUI ADAO interface
188 -------------------------------------------------------------------------
191 .. index:: single: rPython
193 To extend the analysis and treatment capacities, it is possible to use ADAO
194 calculations in **R** environment (see [R]_ for more details). It is available
195 in SALOME by launching the R interpreter in the shell "``salome shell``".
196 Moreover, the package "*rPython*" has to be available, it can be installed by
197 the user if required by the following R command::
200 # IMPORTANT: to be run in R interpreter
201 # -------------------------------------
202 install.packages("rPython")
204 One will refer to the [GilBellosta15]_ documentation for more information on
207 The ADAO calculations defined in text interface (API/TUI, see
208 :ref:`section_tui`) can be interpreted from the R environment, using some data
209 and information from R. The approach is illustrated in the example
210 :ref:`subsection_tui_example`, suggested in the API/TUI interface description.
211 In the R interpreter, one can run the following commands, directly coming from
215 # IMPORTANT: to be run in R interpreter
216 # -------------------------------------
219 from numpy import array
220 from adao import adaoBuilder
221 case = adaoBuilder.New()
222 case.set( 'AlgorithmParameters', Algorithm='3DVAR' )
223 case.set( 'Background', Vector=[0, 1, 2] )
224 case.set( 'BackgroundError', ScalarSparseMatrix=1.0 )
225 case.set( 'Observation', Vector=array([0.5, 1.5, 2.5]) )
226 case.set( 'ObservationError', DiagonalSparseMatrix='1 1 1' )
227 case.set( 'ObservationOperator', Matrix='1 0 0;0 2 0;0 0 3' )
228 case.set( 'Observer', Variable='Analysis', Template='ValuePrinter' )
234 Analysis [ 0.25000264 0.79999797 0.94999939]
236 In writing the ADAO calculations run from R, one must take close attention to
237 the good use of single and double quotes, that should not collide between the
240 The data can come from the R environment and should be stored in properly
241 assigned variables to be used later in Python for ADAO. One will refer to the
242 [GilBellosta15]_ documentation for the implementation work. We can transform the
243 above example to use data from R to feed the three variables of background,
244 observation and observation operator. We get in the end the optimal state also
245 in a R variable. The other lines are identical. The example thus becomes::
248 # IMPORTANT: to be run in R interpreter
249 # -------------------------------------
254 yo <- c(0.5, 1.5, 2.5)
255 h <- '1 0 0;0 2 0;0 0 3'
260 python.assign( "xb", xb )
261 python.assign( "yo", yo )
262 python.assign( "h", h )
264 from numpy import array
265 from adao import adaoBuilder
266 case = adaoBuilder.New()
267 case.set( 'AlgorithmParameters', Algorithm='3DVAR' )
268 case.set( 'Background', Vector=xb )
269 case.set( 'BackgroundError', ScalarSparseMatrix=1.0 )
270 case.set( 'Observation', Vector=array(yo) )
271 case.set( 'ObservationError', DiagonalSparseMatrix='1 1 1' )
272 case.set( 'ObservationOperator', Matrix=str(h) )
273 case.set( 'Observer', Variable='Analysis', Template='ValuePrinter' )
275 xa = list(case.get('Analysis')[-1])
280 xa <- python.get("xa")
282 One notices the explicit ``str`` and ``list`` type conversions to ensure that
283 the data are transmitted as known standard types from "*rPython*" package.
284 Moreover, it is the data that can be transferred between the two languages, not
285 functions or methods. It is therefore necessary to prepare generically in Python
286 the functions to execute required by ADAO, and to forward them correctly the
289 The most comprehensive cases, proposed in :ref:`subsection_tui_advanced`, can be
290 executed in the same way, and they give the same result as in the standard
293 .. _section_advanced_eficas_gui:
295 Using the ADAO EFICAS graphical interface as an ADAO TUI command
296 ----------------------------------------------------------------
298 To make it easier to quickly edit an ADAO command file with ADAO EFICAS (JDC,
299 or pair of files ".comm/.py", that are together in a directory), you can launch
300 the graphical user interface from the Python interpreter. To do this, in a
301 Python interpreter obtained from the "SALOME shell", the following commands are
304 from adao import adaoBuilder
307 If necessary, explicit messages can be used to identify the required
308 environment variables that are missing. However, this command should not be run
309 in the SALOME Python console (because in this case it is enough to activate the
310 module...) but it can be done in a "SALOME shell" session obtained from the
311 "Tools/Extensions" menu of SALOME. As a reminder, the easiest way to get a
312 Python interpreter included in a "SALOME shell" session is to run the following
313 command in a terminal::
315 $SALOMEDIR/salome shell -- python
317 with ``SALOMEDIR`` the ``<SALOME main installation directory>``.
319 .. _section_advanced_execution_mode:
321 Change the default execution mode of nodes in YACS
322 --------------------------------------------------
324 .. index:: single: YACS
326 Various reasons may lead to want to change the default mode of node execution
327 in YACS (see [#]_ for the correct use of these possibilities). This may be for
328 performance reasons, or for example for reasons of resource conflicts.
330 One may want to use this change in execution mode to extend the use of local
331 computing resources or to set remote calculations for a node that requires it.
332 This is particularly the case for a node that should use a simulation resource
333 available on a cluster, for example.
335 In addition, the various calculations that are carried out (user-provided
336 operators, results retrieval functions, etc.) may also present conflicts if
337 they are performed in a single process, and in particular in the main process
338 of SALOME. This is the default YACS operating mode for performance and
339 simplicity reasons. However, it is recommended to change this functioning when
340 encountering execution instabilities or error messages in the graphical
343 In any case, in the YACS schema being edited, it is sufficient to change the
344 execution mode of the node(s) that require it. They have to be executed in a
345 new container created for the occasion (it is not enough to use the default
346 container, it is explicitly necessary to create a new one) and whose properties
347 are adapted to the intended use. The procedure is therefore as follows:
349 #. Create a new YACS container, using the context menu in the tree view of the YACS schema (usually on the left),
350 #. Adapt the characteristics of the container, for example by selecting a "*type*" property with the value "*multi*" for a truly parallel execution, or by choosing a remote computing resource defined by the "*Resource*" property, or by using advanced parameters,
351 #. Graphically select in the central view the node whose execution mode you want to change,
352 #. In the panel to the right of the node entries, unfold the execution choices (named "*Execution Mode*"), check the "*Container*" box instead of the "*YACS*" default, and choose the newly created container (it is usually named "*container0*"),
353 #. Save the modified schema
355 This can be repeated for each node that requires it, by reusing the same new
356 container for all nodes, or by creating a new container for each node.
360 this change in execution mode is extremely powerful and flexible. It is
361 therefore recommended that the user both use it, and at the same time be
362 attentive to the interaction of the different choices he makes, to avoid, for
363 example, an unintended performance deterioration, or computer conflicts that
364 are complicated to diagnose.
366 .. _section_advanced_observer:
368 Getting information on special variables during the ADAO calculation
369 --------------------------------------------------------------------
371 .. index:: single: Observer
372 .. index:: single: Observer Template
374 Some special internal optimization variables, used during calculations, can be
375 monitored during the ADAO calculation. These variables can be printed, plotted,
376 saved, etc. This can be done using "*observer*", that are commands gathered in
377 scripts, each associated with one variable.
379 Some templates are available when editing the ADAO case in graphical editor.
380 These simple scripts can be customized by the user, either at the embedded
381 edition stage, or at the edition stage before execution, to improve the tuning
382 of the ADAO calculation.
384 To implement these "*observer*" efficiently, one can look to the
385 :ref:`section_ref_observers_requirements`.
387 .. _section_advanced_logging:
389 Getting more information when running a calculation
390 ---------------------------------------------------
392 .. index:: single: Logging
394 When running a calculation, useful data and messages are logged. There are two
395 ways to obtain theses information.
397 The first one, and the preferred way, is to use the built-in variable "*Debug*"
398 available in every ADAO case. It is available through the edition GUI of the
399 module. Setting it to "*1*" will send messages in the log window of the YACS
402 The second one consist in using the "*logging*" native module of Python (see the
403 Python documentation http://docs.python.org/library/logging.html for more
404 information on this module). Everywhere in the YACS scheme, mainly through the
405 scripts entries, the user can set the logging level in accordance to the needs
406 of detailed information. The different logging levels are: "*DEBUG*", "*INFO*",
407 "*WARNING*", "*ERROR*", "*CRITICAL*". All the information flagged with a
408 certain level will be printed for whatever activated level above this particular
409 one (included). The easiest way is to change the log level by using the
410 following Python lines::
413 logging.getLogger().setLevel(logging.DEBUG)
415 The standard logging module default level is "*WARNING*", the default level in
416 the ADAO module is "*INFO*".
418 It is also recommended to include some logging or debug mechanisms in the
419 simulation code, and use them in conjunction with the two previous methods. But
420 be careful not to store too big variables because it cost time, whatever logging
421 level is chosen (that is, even if these variables are not printed).
423 .. _subsection_ref_parallel_df:
425 Accelerating numerical derivatives calculations by using a parallel mode
426 ------------------------------------------------------------------------
428 .. index:: single: EnableMultiProcessing
429 .. index:: single: NumberOfProcesses
431 When setting an operator, as described in
432 :ref:`section_ref_operator_requirements`, the user can choose a functional form
433 "*ScriptWithOneFunction*". This form explicitly leads to approximate the tangent
434 and adjoint operators by a finite differences calculation. It requires several
435 calls to the direct operator (user defined function), at least as many times as
436 the dimension of the state vector. This are these calls that can potentially be
437 executed in parallel.
439 Under some conditions, it is then possible to accelerate the numerical
440 derivatives calculations by using a parallel mode for the finite differences
441 approximation. When setting up an ADAO case, it is done by adding the optional
442 keyword "*EnableMultiProcessing*", set to "1", for the "*SCRIPTWITHONEFUNCTION*"
443 command in the operator definition. The parallel mode will only use local
444 resources (both multi-cores or multi-processors) of the computer on which SALOME
445 is running, requiring as many resources as available. If necessary, one can
446 reduce the available ressources by limiting the possible number of parallel
447 processes using the keyword "*NumberOfProcesses*", set to desired maximum (or to
448 "0" for automatic control, which is the default value). By default, this
449 parallel mode is disabled ("*EnableMultiProcessing=0*").
451 The main conditions to perform parallel calculations come from the user defined
452 function, that represents the direct operator. This function has at least to be
453 "thread safe" to be executed in Python parallel environment (notions out of
454 scope of this paragraph). It is not obvious to give general rules, so it's
455 recommended, for the user who enable this internal parallelism, to carefully
456 verify his function and the obtained results.
458 From a user point of view, some conditions, that have to be met to set up
459 parallel calculations for tangent and the adjoint operators approximations, are
462 #. The dimension of the state vector is more than 2 or 3.
463 #. Unitary calculation of user defined direct function "last for long time", that is, more than few minutes.
464 #. The user defined direct function does not already use parallelism (or parallel execution is disabled in the user calculation).
465 #. The user defined direct function avoids read/write access to common resources, mainly stored data, output files or memory capacities.
466 #. The "*observer*" added by the user avoid read/write access to common resources, such as files or memory.
468 If these conditions are satisfied, the user can choose to enable the internal
469 parallelism for the numerical derivative calculations. Despite the simplicity of
470 activating, by setting one variable only, the user is urged to verify the
471 results of its calculations. One must at least doing them one time with
472 parallelism enabled, and an another time with parallelism disabled, to compare
473 the results. If it does fail somewhere, you have to know that this parallel
474 scheme is working for complex codes, like *Code_Aster* in *SalomeMeca*
475 [SalomeMeca]_ for example. So, if it does not work in your case, check your
476 operator function before and during enabling parallelism...
480 in case of doubt, it is recommended NOT TO ACTIVATE this parallelism.
482 It is also recalled that one have to choose the type "*multi*" for the default
483 container in order to launch the scheme, to allow a really parallel execution.
485 .. _subsection_new_adao_version:
487 Switching from a version of ADAO to a newer one
488 -----------------------------------------------
490 .. index:: single: Version
492 The ADAO module and its ".comm" case files are identified by versions, with
493 "Major", "Minor" and "Revision" characteristics. A particular version is
494 numbered as "Major.Minor.Revision", with strong link with the numbering of the
497 Each version "Major.Minor.Revision" of the ADAO module can read ADAO case files
498 of the previous minor version "Major.Minor-1.*". In general, it can also read
499 ADAO case files of all the previous minor versions for one major branch, but it
500 is not guaranteed for all the commands or keywords. In general also, an ADAO
501 case file for one version can not be read by a previous minor or major version
504 Switching from 9.x to 9.y with y > x
505 ++++++++++++++++++++++++++++++++++++
507 There is no known incompatibility for the ADAO case files. The upgrade procedure
508 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
511 Switching from 8.5 to 9.2
512 +++++++++++++++++++++++++
514 There is no known incompatibility for the ADAO case files. The upgrade procedure
515 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
518 However, there may be incompatibilities from user script files that would not
519 have a syntax compatible with Python 3. The most immediate error is the use of
520 printing "*print*" with "*command*" syntax instead of functional syntax
521 "*print(...)*". In this case, it is suggested to correct the syntax of user
522 files in environment 8 before switching to environment 9.
524 Switching from 8.x to 8.y with y > x
525 ++++++++++++++++++++++++++++++++++++
527 There is no known incompatibility for the ADAO case files. The upgrade procedure
528 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
531 To make future developments easier, it is strongly recommended to ensure that
532 your user scripts files use a Python 2 and a Python 3 compatible syntax. In
533 particular, it is recommended to use the functional syntax for "*print*" and not
534 the "*command*" syntax, for example::
538 print( "x = %s %s"%(str(x),str(unit)) )
544 print( "x = {0} {1}".format(str(x),str(unit)) )
552 Switching from 7.8 to 8.1
553 +++++++++++++++++++++++++
555 There is no known incompatibility for the ADAO case files. The upgrade procedure
556 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
559 Switching from 7.x to 7.y with y > x
560 ++++++++++++++++++++++++++++++++++++
562 There is no known incompatibility for the ADAO case files. The upgrade procedure
563 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
566 Switching from 6.6 to 7.2
567 +++++++++++++++++++++++++
569 There is no known incompatibility for the ADAO case files. The upgrade procedure
570 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
573 There is one incompatibility introduced for the post-processing or observer
574 script files. The old syntax to call a result object, such as the "*Analysis*"
575 one (in a script provided through the "*UserPostAnalysis*" keyword), was for
578 Analysis = ADD.get("Analysis").valueserie(-1)
579 Analysis = ADD.get("Analysis").valueserie()
581 The new syntax is entirely similar to the (classical) one of a list or tuple
584 Analysis = ADD.get("Analysis")[-1]
585 Analysis = ADD.get("Analysis")[:]
587 The post-processing scripts has to be modified.
589 Switching from 6.x to 6.y with y > x
590 ++++++++++++++++++++++++++++++++++++
592 There is no known incompatibility for the ADAO case file. The upgrade procedure
593 is to read the old ADAO case file with the new SALOME/ADAO module, and save it
596 There is one incompatibility introduced for the operator script files, in the
597 naming of operators used to for the observation operator. The new mandatory
598 names are "*DirectOperator*", "*TangentOperator*" and "*AdjointOperator*", as
599 described in the last subsection of the chapter :ref:`section_reference`. The
600 operator script files has to be modified.
602 .. [#] For more information on YACS, see the *YACS module* and its integrated help available from the main menu *Help* of the SALOME platform.