<Chapter Label="Using ZeroMQ with the zgap script"><Heading>Using ZeroMQ with the zgap script</Heading>

The <C>zgap</C> script provides facilities to start a number of child processes controlled by a single master process
and to allow for easy coordination between them.

<Section Label="Running zgap">
  <Heading>Running zgap</Heading>

From the shell, run zgap via:

<Example><![CDATA[
bin/zgap -N <nodes> <gap_options> <gap_files>
]]></Example>

Here, <C>nodes</C> should be a positive integer that describes the number of workers one wishes to start. The rest of
the command line, consisting of gap options and gap files, will be passed to the master and the worker processes
verbatim. This allows, for example, the initialization of functions that need to be known by all workers.

The first line of output will be prefixed with <C>[zgap]</C> and will list the directory where <C>zgap</C> will store
the files and sockets it uses to communicate. In particular, the <C>logXX.txt</C> files within that directory will
contain the output generated by the workers; this is useful for debugging, as the workers do not have a working break
loop.

Example:

<Example><![CDATA[
bin/zgap -N 4 -P 8 -m 1G common.g
]]></Example>

On NUMA architectures that support the <C>numactl</C> command, it is possible to further specify which node each worker
should be running on. This can take one of two forms:

<Example><![CDATA[
bin/zgap -N <count>:<start>-<end>
bin/zgap -N <count>:+<start>-<end>
]]></Example>

Each will distribute <C>count</C> worker processes on the physical nodes ranging from <C>start</C> to <C>end</C>
in a round-robin fashion, reusing nodes if there are more workers than nodes. The first mode (without a <C>+</C> sign)
will use absolute node numbers, the second will be relative to the master process. See the <C>numactl</C> manual page
for further details.

Example:

<Example><![CDATA[
bin/zgap -N 4:+0-3 -P 8 -m 1G common.g
]]></Example>

Note: Currently, <C>zgap</C> can only be run from the GAP root directory. This is an implementation restriction that is
to be removed at a later date.

  </Section>
  <Section Label="Zgap API">
    <Heading>Zgap API</Heading>

Most of the following API functions take a <C>dest</C> argument, which is used to specify the destination of the
operation. To specify a worker thread, <C>dest</C> would have to be an integer in the range from 1 to the number of
worker processes; 0 specifies the master process. Multiple processes can be specified by a range or list of integers.
The variable <C>ZAll</C> contains a range encompassing the worker processes; <C>ZSelf</C> contains the index of the
current worker or 0 for the master.

<ManSection>
<Func Name="ZExec" Arg="dest, cmd"/>
<Description>
This function sends <C>cmd</C> to the given destination and executes it there. The command must be a valid GAP statement
ending in a semicolon. If <C>dest</C> specifies multiple processes, the command will be executed on all of them.
</Description>
</ManSection>

<ManSection>
<Func Name="ZBind" Arg="dest, var, expr"/>
<Description>
This function binds the global variable described by the string <C>var</C> to the value <C>expr</C> in all processes
listed in <C>dest</C>. Note that <C>expr</C> must evaluate to a serializable value.

<Example><![CDATA[
gap> ZBind(ZAll, "counter", 0);
]]></Example>
</Description>
</ManSection>

<ManSection>
<Func Name="ZUnbind" Arg="dest, var"/>
<Description>
This function is the counterpart to <C>ZBind</C>. It will unbind <C>var</C> in all specified processes.

<Example><![CDATA[
gap> ZUnbind(ZAll, "status");
]]></Example>
</Description>
</ManSection>

<ManSection>
<Func Name="ZCall" Arg="dest, func, args"/>
<Description>
This function will execute the function specified by the string <C>func</C> in the specified processes. The string
<C>func</C> must be the name of a global variable referring to the function to be executed. This function should be
created at startup by adding a file to the commandline that defines it in all workers or by <C>ZExec</C>.

<Example><![CDATA[
gap> ZBind(ZAll, "counter", 0);
gap> ZExec(Zall, "add := function(n) counter := counter + n; end;");
gap> ZCall(1, "add", [1]);
]]></Example>
</Description>
</ManSection>

<ManSection>
<Func Name="ZQuery" Arg="dest, func, args, callback"/>
<Description>
This function works like <C>ZCall</C>, except that any return value will be passed to the callback function.

<Example><![CDATA[
gap> res := false;
false
gap> ZQuery(1, "ReturnTrue", [], function(x) res := x; end);
gap> res;
true
]]></Example>
</Description>
</ManSection>

<ManSection>
<Func Name="ZResponse" Arg=""/>
<Description>
<C>ZResponse</C> is a convenience function to construct blocking callbacks for <C>ZCall</C> and <C>ZTask</C>. It returns
a record containing a <C>put</C>, a <C>get</C>, and a <C>test</C> function. Here, <C>put</C> is passed as the callback;
<C>get</C> can be used to read the returned value; and <C>test</C> can be used to test for the presence of a value.

<Example><![CDATA[
gap> resp := ZResponse();;
gap> ZQuery(1, "Z", [4], resp.put);
gap> resp.get();
Z(2^2)
gap> resp.test();
true
]]></Example>
</Description>
</ManSection>

<ManSection>
<Func Name="ZTask" Arg="dest, func, args, callback"/>
<Description>
This function works like <C>ZQuery</C>, except that the function will be executed via a task and <C>callback</C> will be
called after the task finishes and returns a result.
</Description>
</ManSection>

<ManSection>
<Func Name="ZAsync" Arg="dest, func, args"/>
<Description>
This function works like <C>ZCall</C>, except that the function will be executed via a task.
</Description>
</ManSection>

<ManSection>
<Func Name="ZRead" Arg="dest, file"/>
<Description>
This function does a <C>Read(file)</C> for all specified processes.
</Description>
</ManSection>

<ManSection>
<Func Name="ZReadGapRoot" Arg="dest, file"/>
<Description>
This function does a <C>ReadGapRoot(file)</C> for all specified processes.
</Description>
</ManSection>

  </Section>
</Chapter>
