STRING(TOUPPER ${PROJECT_NAME} PROJECT_NAME_UC)
SET(${PROJECT_NAME_UC}_MAJOR_VERSION 8)
-SET(${PROJECT_NAME_UC}_MINOR_VERSION 4)
+SET(${PROJECT_NAME_UC}_MINOR_VERSION 5)
SET(${PROJECT_NAME_UC}_PATCH_VERSION 0)
SET(${PROJECT_NAME_UC}_VERSION
${${PROJECT_NAME_UC}_MAJOR_VERSION}.${${PROJECT_NAME_UC}_MINOR_VERSION}.${${PROJECT_NAME_UC}_PATCH_VERSION})
#extensions = ['rst2pdf.pdfbuilder']
# Add any paths that contain templates here, relative to this directory.
-templates_path = [os.path.join('@CMAKE_CURRENT_SOURCE_DIR@','_templates')]
+templates_path = [os.path.join('@CMAKE_CURRENT_SOURCE_DIR@','templates')]
# The suffix of source filenames.
source_suffix = '.rst'
.. _resource:
-*******************************************************
+**************************************************
Concurrent branches and multiple machine execution
-*******************************************************
+**************************************************
.. _concurrent:
Execution of concurrent branches
-===================================
+================================
YACS can execute calculation nodes of a scheme simultaneously.
However, simultaneous execution of a large number of nodes can saturate the system.
The maximum number of simultaneous executions can be controlled by fixing the maximum number of threads used with the
.. _multi:
Execution on multiple machines
-===================================
+==============================
YACS can execute the nodes of a scheme on several machines where SALOME is
already installed.
Each machine is a resource which has to be declared in the resources catalog
Containers use a set of constraints and rules for choosing the resource where
the node will be executed (see :ref:`containers`).
+
+.. _multi_HP:
+
+Execution on multiple machines with mutliples HP container types
+================================================================
+
+For a complex graph using multiples HP containers it can be really difficult to determine the best values
+of pool sizes for each HP container type and the numbers of branchs of the ForEach loops. In order to let Yacs resolves
+this problem by itself, one can use the Playground feature: ::
+
+ pilot.PlayGround()
+ machineCatalog=[('is2541',8),] # [('machine name', 'nb cores on machine'), ...]
+ pg.setData(machineCatalog)
+ p.fitToPlayGround(pg)
+
+Principle
+---------
+The graph is analyzed and recursively, for all fork, the current domain is equitably divided into sub-domains
+(one for each sub-branch). A domain of a branch corresponds to the cores from the machine catalog which will be used
+for the execution of the nodes of this branch. This division in sub-domains assures that one core will be used by at
+most one container at once, even if it can be used by different HP containers type during the whole execution of a graph.
+
+Once all the domains have been set, the numbers of branchs of the ForEach loops are calculated in order to use the whole domain associated in runtime.
+
+Note: If a branch does not contain any ForEach loop, the size of its domain is reduced to the minimal size.
+
+Graph with multiples ForEach Loop executed in parallel
+------------------------------------------------------
+In some cases it can be better to dispatch unfairly the computing power between the ForEach Loops. In order to do so it is possible to allocate
+a weight to the nodes of the graph: ::
+
+ elem_node.setWeight(30.) # means execution time of the node is 30s
+ for_each_loop.setWeight(500.) # means execution time of the whole for each is 500s (exemple: 10 iterations and 50s each)
+ ...
+ p.fitToPlayGround(pg)
+
+Assuming that the weight corresponds to the execution time of each node during a sequential execution, the algorithm automatically determines, at all fork,
+the best way to divide the domain into sub-domains in order to minimize the execution time of whole graph.
+
+Explanation: When the computing power allocated to a branch is multiplied by a factor ‘a’, thus weights of all ForEach nodes of this branch
+are divided by the same factor. Using this the algorithm searchs the best sharing between all branchs at a fork in order the minimize the following
+parameter: max(for all branchs at the fork: weight of the branch).
+
+Note:
+
+- If at least one node of a branch does not have a weight, the weight of the branch cannot be calculated and thus its domain receives 1/n of the computing power (n being the number of branchs containing at least one ForEach Loop at this fork point).
+- The weight does not have a unit. Any can be used.
+- The calculation of the weight of a whole branch is not recursive. It is the sum of the weight of all its elementary and ForEach nodes (but as seen previously the sharing is done recursively).
+
+
+
+
n.setExecutionMode("remote")
n.setContainer(cont1)
+The default option for the execution mode is **local** where the node will run
+on the same container as the scheme executor.
+
.. _pyfunc:
Python function node
n2.setExecutionMode("remote")
n2.setContainer(cont1)
+The default option for the execution mode is **local** where the node will run
+on the same container as the scheme executor.
+
.. _pyservice:
SALOME service node
Definition of containers
''''''''''''''''''''''''''''
-A container is defined using the runtime createContainer method and it is then given a name using its setName method.
-The next step is to assign constraints to it by adding properties.
-The following is an example creation of a container named “A”::
- c1=r.createContainer()
- c1.setName("A")
+This example shows how to add a container to a scheme::
+
+ c1=p.createContainer("MyContainer")
A property is added to a container using its setProperty method that uses 2 arguments (character strings).
The first is the property name. The second is its value.
-The following is an example of this container “A” with constraints::
+The following is an example of how to set constraints on the container::
- c1=r.createContainer()
- c1.setName("A")
c1.setProperty("container_name","FactoryServer")
c1.setProperty("hostname","localhost")
c1.setProperty("mem_mb","1000")
of a SALOME service node is to obtain the component instance of this service node using the getComponent method for this node.
The previously defined container is then assigned to this component instance using the setContainer method of the component instance.
-If it is required to place the SALOME service defined above (node “node3”) on container “A”, we will write::
+If it is required to place the SALOME service defined above (node “node3”) on
+“MyContainer”, we will write::
n3.getComponent().setContainer(c1)
+It is also possible to place python nodes on containers, but the code is a
+little different (see :ref:`pyscript`)::
+
+ n1.setExecutionMode("remote")
+ n1.setContainer(c1)
+
Since SALOME v7.5, there is a new type of container:
*Homogeneous Pool of SALOME containers* (HP container).
It is possible to create this type of container this way::
--- /dev/null
+{% extends "!layout.html" %}
+
+{%- block sidebarlogo %}
+{{ super() }}
+{%-
+include "searchbox.html"
+%}
+<p/>
+{%- endblock %}
+{%- block sidebarsearch %}
+{%- endblock %}
\ No newline at end of file
class BlocPoint;
class ComposedNode;
class ForkBlocPoint;
+ class PointVisitor;
class LinkedBlocPoint;
class YACSLIBENGINE_EXPORT AbstractPoint
virtual void getWeightRegardingDPL(ComplexWeight *weight) = 0;
virtual void partitionRegardingDPL(const PartDefinition *pd, std::map<ComposedNode *, YACS::BASES::AutoRefCnt<PartDefinition> >& zeMap) const = 0;
virtual std::string getRepr() const = 0;
+ virtual void accept(PointVisitor *pv) = 0;
virtual ~AbstractPoint();
public:
static bool IsGatherB4Ext(Node *node);
return ret;
}
+void BagPoint::accept(PointVisitor *pv)
+{
+ if(_nodes.size()!=1)
+ throw YACS::Exception("BagPoint::accept : simplification has failed !");
+ AbstractPoint *ret(*_nodes.begin());
+ if(!ret)
+ throw YACS::Exception("BagPoint::accept : Ooops !");
+ ret->accept(pv);
+}
+
Node *BagPoint::getFirstNode()
{
return getUnique()->getFirstNode();
void partitionRegardingDPL(const PartDefinition *pd, std::map<ComposedNode *, YACS::BASES::AutoRefCnt<PartDefinition> >& zeMap) const;
std::string getRepr() const;
AbstractPoint *getUniqueAndReleaseIt();
+ void accept(PointVisitor *pv) override;
public:
int size() const { return (int)_nodes.size(); }
void replaceInMe(BlocPoint *aSet);
for(std::list<ForEachLoop *>::const_iterator it=vis._fes.begin();it!=vis._fes.end();it++)
(*it)->edGetNbOfBranchesPort()->edInit(1);
this->removeRecursivelyRedundantCL();
+ if (this->getMaxLevelOfParallelism() > pg->getNumberOfCoresAvailable())
+ throw YACS::Exception("Bloc::fitToPlayGround : Not enough cores available to run the calculation !");
this->partitionRegardingDPL(pd,zeMap);
this->accept(&vis);
for(std::list<ForEachLoop *>::const_iterator it=vis._fes.begin();it!=vis._fes.end();it++)
ComplexWeight& addWeight(const ComplexWeight *other);
protected:
bool _bootWeight;
- // _loopWeight: vect<pair(weight,nbcorePerIteration)>, for first element of vector: nbcorePerIteration<0 -> unset, nbcorePerIteration==0 -> no loopweight
+ // _loopWeight: vect<pair(weight,nbcorePerIteration)>, for first element of vector: nbcorePerIteration<0 -> unset, nbcorePerIteration==0 -> no loopweight (means no loop inside)
std::vector<std::pair<double,int> > _loopWeight;
double _elementaryWeight;
private:
//
#include "ElementaryPoint.hxx"
+#include "PointVisitor.hxx"
#include "Node.hxx"
using namespace YACS::ENGINE;
{
return _node->getName();
}
+
+void ElementaryPoint::accept(PointVisitor *pv)
+{
+ pv->beginElementaryPoint(this);
+ pv->endElementaryPoint(this);
+}
void getWeightRegardingDPL(ComplexWeight *weight);
void partitionRegardingDPL(const PartDefinition *pd, std::map<ComposedNode *, YACS::BASES::AutoRefCnt<PartDefinition> >& zeMap) const;
std::string getRepr() const;
+ void accept(PointVisitor *pv) override;
virtual ~ElementaryPoint();
};
}
//
#include "ForkBlocPoint.hxx"
+#include "PointVisitor.hxx"
#include "Exception.hxx"
#include <algorithm>
ret+="]";
return ret;
}
+
+void ForkBlocPoint::accept(PointVisitor *pv)
+{
+ pv->beginForkBlocPoint(this);
+ for(auto it:_nodes)
+ it->accept(pv);
+ pv->endForkBlocPoint(this);
+}
void getWeightRegardingDPL(ComplexWeight *weight);
void partitionRegardingDPL(const PartDefinition *pd, std::map<ComposedNode *, YACS::BASES::AutoRefCnt<PartDefinition> >& zeMap) const;
std::string getRepr() const;
+ void accept(PointVisitor *pv) override;
virtual ~ForkBlocPoint();
};
}
//
#include "LinkedBlocPoint.hxx"
+#include "PointVisitor.hxx"
#include "Exception.hxx"
using namespace YACS::ENGINE;
return ret;
}
+void LinkedBlocPoint::accept(PointVisitor *pv)
+{
+ pv->beginLinkedBlocPoint(this);
+ for(auto it:_nodes)
+ it->accept(pv);
+ pv->endLinkedBlocPoint(this);
+}
+
LinkedBlocPoint::~LinkedBlocPoint()
{
}
void getWeightRegardingDPL(ComplexWeight *weight);
void partitionRegardingDPL(const PartDefinition *pd, std::map<ComposedNode *, YACS::BASES::AutoRefCnt<PartDefinition> >& zeMap) const;
std::string getRepr() const;
+ void accept(PointVisitor *pv) override;
virtual ~LinkedBlocPoint();
};
}
}
else if (!(*it).first->hasValidLoopWeight())
{
- nbOfCoresAllocated[i]=(int)((double)totalSpace/(double)(sz)); // branch with undefined weight
+ nbOfCoresAllocated[i]=std::max((int)((double)totalSpace/(double)(sz)), nbCoresPerShot[i]); // branch with undefined weight, takes his part proportionnally to the number of branchs
}
else
{
--- /dev/null
+// Copyright (C) 2015-2016 CEA/DEN, EDF R&D
+//
+// This library is free software; you can redistribute it and/or
+// modify it under the terms of the GNU Lesser General Public
+// License as published by the Free Software Foundation; either
+// version 2.1 of the License, or (at your option) any later version.
+//
+// This library is distributed in the hope that it will be useful,
+// but WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+// Lesser General Public License for more details.
+//
+// You should have received a copy of the GNU Lesser General Public
+// License along with this library; if not, write to the Free Software
+// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+//
+// See http://www.salome-platform.org/ or email : webmaster.salome@opencascade.com
+//
+
+#pragma once
+
+#include "YACSlibEngineExport.hxx"
+
+namespace YACS
+{
+ namespace ENGINE
+ {
+ class ElementaryPoint;
+ class LinkedBlocPoint;
+ class ForkBlocPoint;
+
+ class YACSLIBENGINE_EXPORT PointVisitor
+ {
+ public:
+ virtual void beginForkBlocPoint(ForkBlocPoint *pt) = 0;
+ virtual void endForkBlocPoint(ForkBlocPoint *pt) = 0;
+ virtual void beginLinkedBlocPoint(LinkedBlocPoint *pt) = 0;
+ virtual void endLinkedBlocPoint(LinkedBlocPoint *pt) = 0;
+ virtual void beginElementaryPoint(ElementaryPoint *pt) = 0;
+ virtual void endElementaryPoint(ElementaryPoint *pt) = 0;
+ };
+ }
+}
{
return _bp->getUniqueAndReleaseIt();
}
+
+void SetOfPoints::accept(PointVisitor *pv)
+{
+ _bp->accept(pv);
+}
#include "YACSlibEngineExport.hxx"
#include "PlayGround.hxx"
+#include "PointVisitor.hxx"
#include "AutoRefCnt.hxx"
#include <map>
void getWeightRegardingDPL(ComplexWeight *weight);
void partitionRegardingDPL(const PartDefinition *pd, std::map<ComposedNode *, YACS::BASES::AutoRefCnt<PartDefinition> >& zeMap) const;
AbstractPoint *getUniqueAndReleaseIt() const;
+ void accept(PointVisitor *pv);
private:
BagPoint *_bp;
};
}
}
+%typemap(in) const std::list<YACS::ENGINE::Node *>&
+{
+ if(!PyList_Check($input))
+ {
+ PyErr_SetString(PyExc_TypeError,"not a list");
+ return NULL;
+ }
+ $1=new std::list<YACS::ENGINE::Node *>;
+ int size(PyList_Size($input));
+ for(int i=0;i<size;i++)
+ {
+ PyObject *o=PyList_GetItem($input,i);
+ YACS::ENGINE::Node *temp(nullptr);
+ if((SWIG_ConvertPtr(o,(void **)&temp, $descriptor(YACS::ENGINE::Node*),0)) == -1)
+ {
+ PyErr_SetString(PyExc_TypeError,"not a YACS::ENGINE::TypeCode*");
+ return nullptr;
+ }
+ $1->push_back(temp);
+ }
+}
+
%typemap(out) YACS::ENGINE::Node*
{
$result=convertNode($1,$owner);
}
}
+%typemap(out) std::vector< std::list<YACS::ENGINE::Node *> >
+{
+ std::vector< std::list<YACS::ENGINE::Node *> >::const_iterator it;
+ $result = PyList_New($1.size());
+ int i(0);
+ for (it = $1.begin(); it != $1.end(); ++it, ++i)
+ {
+ const std::list<YACS::ENGINE::Node *>& elt(*it);
+ PyObject *tmp(PyList_New(elt.size()));
+ int j(0);
+ for (auto it2=elt.begin() ; it2!= elt.end() ; ++it2, ++j)
+ PyList_SetItem(tmp,j,convertNode(*it2));
+ PyList_SetItem($result,i,tmp);
+ }
+}
+
#endif
// ----------------------------------------------------------------------------
#include "ComponentInstance.hxx"
#include "DataNode.hxx"
#include "PlayGround.hxx"
+#include "SetOfPoints.hxx"
+#include "PointVisitor.hxx"
+#include "ForkBlocPoint.hxx"
+#include "LinkedBlocPoint.hxx"
+#include "ElementaryPoint.hxx"
using namespace YACS::ENGINE;
self->assignPassedResults(passedIds,passedOutputsCpp,nameOfOutputs);
}
}
+
+namespace YACS
+{
+ namespace ENGINE
+ {
+ class AbstractPoint
+ {
+ protected:
+ virtual ~AbstractPoint();
+ AbstractPoint();
+ AbstractPoint(const AbstractPoint&);
+ };
+
+ class ElementaryPoint : public AbstractPoint
+ {
+ public:
+ Node *getFirstNode();
+ private:
+ ~ElementaryPoint();
+ ElementaryPoint();
+ ElementaryPoint(const ElementaryPoint&);
+ };
+
+ class BlocPoint : public AbstractPoint
+ {
+ protected:
+ ~BlocPoint();
+ BlocPoint();
+ BlocPoint(const BlocPoint&);
+ };
+
+ class LinkedBlocPoint : public BlocPoint
+ {
+ private:
+ ~LinkedBlocPoint();
+ LinkedBlocPoint();
+ LinkedBlocPoint(const LinkedBlocPoint&);
+ };
+
+ class ForkBlocPoint : public BlocPoint
+ {
+ private:
+ ~ForkBlocPoint();
+ ForkBlocPoint();
+ ForkBlocPoint(const ForkBlocPoint&);
+ };
+
+ class SetOfPoints
+ {
+ public:
+ SetOfPoints(const std::list<Node *>& nodes);
+ ~SetOfPoints();
+ void simplify();
+ std::string getRepr() const;
+ %extend
+ {
+ void accept(PyObject *pv)
+ {
+ class MyPointVisitor : public YACS::ENGINE::PointVisitor
+ {
+ public:
+ MyPointVisitor(PyObject *py):_py(py) { }
+ void beginForkBlocPoint(ForkBlocPoint *pt)
+ {
+ PyObject *ptPy(SWIG_NewPointerObj((void*)pt,SWIGTYPE_p_YACS__ENGINE__ForkBlocPoint,0));
+ PyObject *meth(PyString_FromString("beginForkBlocPoint"));
+ PyObject *ret(PyObject_CallMethodObjArgs(_py,meth,ptPy,nullptr));
+ Py_XDECREF(ret); Py_XDECREF(meth); Py_XDECREF(ptPy);
+ }
+ void endForkBlocPoint(ForkBlocPoint *pt)
+ {
+ PyObject *ptPy(SWIG_NewPointerObj((void*)pt,SWIGTYPE_p_YACS__ENGINE__ForkBlocPoint,0));
+ PyObject *meth(PyString_FromString("endForkBlocPoint"));
+ PyObject *ret(PyObject_CallMethodObjArgs(_py,meth,ptPy,nullptr));
+ Py_XDECREF(ret); Py_XDECREF(meth); Py_XDECREF(ptPy);
+ }
+ void beginLinkedBlocPoint(LinkedBlocPoint *pt)
+ {
+ PyObject *ptPy(SWIG_NewPointerObj((void*)pt,SWIGTYPE_p_YACS__ENGINE__LinkedBlocPoint,0));
+ PyObject *meth(PyString_FromString("beginLinkedBlocPoint"));
+ PyObject *ret(PyObject_CallMethodObjArgs(_py,meth,ptPy,nullptr));
+ Py_XDECREF(ret); Py_XDECREF(meth); Py_XDECREF(ptPy);
+ }
+ void endLinkedBlocPoint(LinkedBlocPoint *pt)
+ {
+ PyObject *ptPy(SWIG_NewPointerObj((void*)pt,SWIGTYPE_p_YACS__ENGINE__LinkedBlocPoint,0));
+ PyObject *meth(PyString_FromString("endLinkedBlocPoint"));
+ PyObject *ret(PyObject_CallMethodObjArgs(_py,meth,ptPy,nullptr));
+ Py_XDECREF(ret); Py_XDECREF(meth); Py_XDECREF(ptPy);
+ }
+ void beginElementaryPoint(ElementaryPoint *pt)
+ {
+ PyObject *ptPy(SWIG_NewPointerObj((void*)pt,SWIGTYPE_p_YACS__ENGINE__ElementaryPoint,0));//$descriptor(YACS::ENGINE::ElementaryPoint *)
+ PyObject *meth(PyString_FromString("beginElementaryPoint"));
+ PyObject *ret(PyObject_CallMethodObjArgs(_py,meth,ptPy,nullptr));
+ Py_XDECREF(ret); Py_XDECREF(meth); Py_XDECREF(ptPy);
+ }
+ void endElementaryPoint(ElementaryPoint *pt)
+ {
+ PyObject *ptPy(SWIG_NewPointerObj((void*)pt,SWIGTYPE_p_YACS__ENGINE__ElementaryPoint,0));
+ PyObject *meth(PyString_FromString("endElementaryPoint"));
+ PyObject *ret(PyObject_CallMethodObjArgs(_py,meth,ptPy,nullptr));
+ Py_XDECREF(ret); Py_XDECREF(meth); Py_XDECREF(ptPy);
+ }
+ private:
+ PyObject *_py;
+ };
+ MyPointVisitor mpv(pv);
+ self->accept(&mpv);
+ }
+ }
+ };
+ }
+}
+
void addOutputVar(const std::string& name) { n2Script<< name << ", "; }
void assignOutput(YACS::ENGINE::InlineNode *node) {
n2Script << "]\nwith open(\"" << _jobName << "\",\"w\") as f:" << std::endl;
- n2Script << " f.write(str(zeRes))" << std::endl;
+ n2Script << " f.write(repr(zeRes))" << std::endl;
node->setScript(n2Script.str());
}
private:
YACSEvalSession();
~YACSEvalSession();
void launch();
+ void launchUsingCurrentSession();
bool isLaunched() const;
bool isAttached() const;
bool isAlreadyPyThreadSaved() const;
rss=efx.giveResources()
rss[0][0].setWantedMachine("athos")
cp=rss.getAddParamsForCluster() ; cp.setRemoteWorkingDir(os.path.join("/scratch",getpass.getuser(),"TMP3")) ; cp.setLocalWorkingDir(os.path.join(os.path.expanduser("~"),"TMP52"))
-cp.setWCKey("P11U50:CARBONES") ; cp.setNbProcs(5) ; cp.setMaxDuration("00:05")
+cp.setWCKey("P11U5:CARBONES") ; cp.setNbProcs(5) ; cp.setMaxDuration("00:05")
assert(not rss.isInteractive())
a,b=efx.run(session)
print(("************",a,b))
<parameter name="YACS" value="%YACS_ROOT_DIR%/share/salome/resources/yacs"/>
</section>
<section name="yacs_help">
- <parameter name="sub_menu" value="%1 module"/>
- <parameter name="User's Guide" value="%YACS_ROOT_DIR%/share/doc/salome/gui/YACS/index.html"/>
- <parameter name="Developer's Guide" value="%YACS_ROOT_DIR%/share/doc/salome/tui/YACS/index.html"/>
+ <parameter name="User's Guide/YACS module" value="%YACS_ROOT_DIR%/share/doc/salome/gui/YACS/index.html"/>
+ <parameter name="Developer's Guide/YACS module" value="%YACS_ROOT_DIR%/share/doc/salome/tui/YACS/index.html"/>
</section>
<section name="YACS" >
<!-- Module preferences -->
_nbProcs(1),
_hours(0),
_minutes(5),
- _wcKey("P11U50:CARBONES"),
+ _wcKey("P11U5:CARBONES"),
_localDirectory("/tmp"),
_remoteDirectory("/tmp"),
_parallelizeStatus(true),
