<!-- Generator apps

     $Id: sect_generator.docbook 1164 2010-01-06 10:03:51Z ewald-arnold $
-->

<chapter id="devel-generator">
<title>Automatically Create Server and Client Sources</title>

<sect1 id="gen-from-source">
<title>Sources From your C++ Headers</title>

<para>The process for the creation of the source files from C++ headers is
split into two parts:

<itemizedlist spacing="compact">
  <listitem><para>The first application <filename>ulxr2xml</filename> parses the
  C++ header file and writes the constructors and methods you need to an intermediate
  XML file.</para></listitem>

  <listitem><para>In the second step <filename>xml2ulxr</filename> reads the
  XML file and creates a C++ header and the according implementation files.
  An extra file is created where user defined information is located.</para></listitem>
</itemizedlist>
</para>

<para>Splitting the process into two parts enables postprocessing the XML file with
simple methods. Otherwise a rather complicated C++ parser based on less known
tools like <filename>yacc</filename> and <filename>lex</filename> would have to
be changed.</para>

<para>Additionally one could create a different application for the second step.
Based on the XML file it could create totally different user
defined C++ sources.</para>

<para>If you integrate the two applications into your build process, your client
and server files are automatically kept up to date with the production sources. You
simply make the files depend on the according header file.
Your specialized constuctors and method names are placed in the third file. This file is never
updated automatically. Instead a template file is written from where you can copy-and-paste
what you need and adjust the rest manually.</para>

<!-- ===================================== -->

<sect2 id="devel-gen-ulxr2xml">
<title>Parsing The Header File</title>

<para>In the first step <filename>ulxr2xml</filename> parses the header file
and outputs the
according XML data stream to standard output. If you want to have an XML file you
must redirect standard outout to the desired file. The only possible parameter is the name
of the C++ header file. All the classes in the header file are written to the
XML stream.</para>

<para>The parser needs markers whch methods and constructors shall be written to
the XML stream. This is done by appending a pseudo keyword after one of the access
modifiers <token>public</token>, <token>protected</token> or <token>private</token>
before the definition.</para>

<para>The parser distinguishes between constructors and methods. Each of them
uses a different keyword:

<variablelist>

  <varlistentry>
    <term><token>ulxr_constructor</token></term>
    <listitem><para>Exists only for historical reasons and has no effect.
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><token>ulxr_methods</token></term>
    <listitem><para>All the methods after this keyword are written to the XML
    stream. The inclusion stops at the next access modifier unless it contains
    this keyword again.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><token>ulxr_function</token></term>
    <listitem><para>Free function need markers as well. Unlike methods within a
    class each function needs its own marker.</para></listitem>
  </varlistentry>

</variablelist>

Destructors are never included even if they have a starting marker. To make
these keywords invisible for the compiler you define these two with empty content.</para>

<para>A sample class could then look like this:

<programlisting>

#define ulxr_methods

class ulxrIdlTest : public IdlInterface
{
    ulxrIdlTest(int i, const std::string &amp;s)
      : IdlInterface("name")
    {
    }

  private ulxr_methods:
    virtual std::string * url4() = 0;
};

</programlisting>

</para>

<para>The following line shows how to invoke the C++ parser and write the XML stream
to a file:

<screen>

  ulxr2xml   header.h  >header-data.xml

</screen>

</para>

<para>If you are familiar with with development around
<ulink url="http://www.kde.org">KDE</ulink> and
<ulink url="http://www.troll.no">QT</ulink> you are probably familiar with this
approach.</para>

<para>It might also be interesting to know that the basis for this parser was
taken from the
<ulink url="http://www.kde.org">KDE Project</ulink>. For that reason it is able
to parse (in the meaning of to ignore) the special keywords related to DCOP and
QOBJECT like <token>k_dcop</token> or <token>signals</token>.
</para>

<note>
<title>The Difficulty of Parsing C++</title>
<para>Parsing even valid C++ source code is is a difficult task.
<filename>xml2ulxr</filename> handles most of the common files. But there may
be problems with some code, for example nested templates. In the worst case
you may have to remove or
comment out the problematic parts while parsing. Probably it is possible to use
<token>#define</token>'s with the preprocessor to do this automatically.</para>

<para>There also an inherent limitation in the data types you can use. The generator
is only able to handle data types used in &xmlrpc; that can be mapped to native
C++ data types like <token>int</token>, <token>std::string</token>, <token>float</token>
or <token>const char*</token>. No arrays or structures from &xmlrpc;
are supported. On the other side, the generator is able to resolve pointers and
references in parameters and return values. </para>

<para>The test files <filename>ulxr/stubber/ulxr2xml/dcopidl_test.h</filename>
and <filename>ulxr/stubber/ulxr2xml/ulxridl_test.h</filename> give an
overview of what the parser is able to process.</para>

<para>In case you have a file which is not parsed correctly but you think this
should be the case since it looks trivial: please post
<ulink url="http://sourceforge.net/tracker/?group_id=49635&amp;atid=456926">a
bug report.</ulink></para>

</note>

</sect2>

<!-- ===================================== -->

<sect2 id="devel-gen-xml2ulxr">
<title>Generating The Files</title>

<para>In the second step the XML file is processed and the C++ files are created.
Each invocation of <filename>xml2ulxr</filename> creates a set consisting of
three or more files depending on the given options and the XML file. The
filenames are composed of the according C++ class name and a file specific
extension depending on the intended type of code either for a server or a client.

<itemizedlist spacing="compact">
  <listitem><para>A C++ header file with the filename extension "_ulxr_client.h".</para></listitem>
  <listitem><para>A C++ implementation file with the filename extension "_ulxr_client.cpp".</para></listitem>
  <listitem><para>A C++ implementation file containing user defined information
  with the filename ending in "_ulxr_client_user.cpp". If this file already exists
  it is never overwritten but a forth file ending in "_ulxr_client_user.cpp.new"
  is created. From there you must manually copy-and-paste the new parts.</para></listitem>

  <listitem><para>A C++ header file with the filename extension "_ulxr_server.h".</para></listitem>
  <listitem><para>A C++ implementation file with the filename extension "_ulxr_server.cpp".</para></listitem>
  <listitem><para>A C++ implementation file containing user defined information
  with the filename ending in "_ulxr_server_user.cpp". This file is also never
  overwritten.</para></listitem>
  <listitem><para>A C++ header file with the filename extension "_names.h" that
  contains <token>#define</token>s for the method names which is used for both
  the server and the client files. This file is also never overwritten.</para></listitem>
</itemizedlist>

When you export free functions there is another set of up to 6 more files containing
code for server, client and names. These filenames are built with the prefix you passed
via <command>--gen-funcs</command> plus the literal <constant>"Funcs"</constant>
plus the aforementioned suffixes to distinguish between server, client and names.</para>

<para>There are several options that influence the file creation process:

<variablelist>

  <varlistentry>
    <term><command>--gen-server</command></term>
    <listitem><para>Generate files that are used on the server side to receive
    the requests and forward them to the actual worker methods.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--gen-client</command></term>
    <listitem><para>Generate files used for the client code to transform a regular
    C++ invocation into an &xmlrpc; method call sent to the network.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--gen-class=classname</command></term>
    <listitem><para>Without additional options each of the classes in the XML file
    result in a set of header and source files. If you want to create files for
    only a selected set of classes you have to pass the class name using this
    option. When you need more than one class you have to pass this option
    multiple times.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--gen-funcs=prefix</command></term>
    <listitem><para>Generate code for free functions. The prefix is prepended
    to filenames and member names similar to the code for classes to prevent
    name clashes. The method names of the client stubs itself remaing the same
    and are placed into the according namespaces.
    <note>
    <title>Also Select Client or Server Code</title>
    <para>This option additionally needs <command>--gen-client</command> and
    <command>--gen-server</command> to select the desired source type.</para>
    </note>
    </para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--file-name=filename.xml</command></term>
    <listitem><para>The name of the XML file with the class data.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--dest-dir=dest-path</command></term>
    <listitem><para>Without further options the resulting files are written to
    the current directory. If you want a different destination you provide
    the desired path with this parameter.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--disable-timestamp</command></term>
    <listitem><para>When generating the code a timestamp is added into the comment header.
    Mainly for testing purposes this feature can be disabled.</para></listitem>
  </varlistentry>

</variablelist>

</para>

<para>The following line creates the C++ files
<filename>ulxrIdlTest_ulxr_server.h</filename>,
<filename>ulxrIdlTest_ulxr_server.cpp</filename>
and <filename>ulxrIdlTest_ulxr_server_user.cpp</filename> from the XML file
<filename>header-data.xml</filename>. These files are based on
 and they are written
to the subdirectory <filename>forwarders</filename>. For better readability the long
command line is broken into two lines.

<screen>

  xml2ulxr  --gen-server --file-name=header-data.xml \
              --dest-dir=forwarders  --gen-class=UlxrIdlTest

</screen>

</para>

<para>Once the files are generated there are just a few changes needed. On the
server side you must register the existing production code with the &xmlrpc;
method dispatcher:

<calloutlist>
  <callout id="devel-srv-cls-1" arearefs="devel-srv-cls-1-co">
    <para>Instantiate the class for the actual worker code.</para>
  </callout>

  <callout id="devel-srv-cls-2" arearefs="devel-srv-cls-2-co">
   <para>Prepare a multithreaded method dispatcher or something similar as you need
   it.</para>
  </callout>

  <callout id="devel-srv-cls-3" arearefs="devel-srv-cls-3-co">
   <para>Instantiate the object which connects the &xmlrpc; methods
   with the worker code in the class. In case there are several worker objects
   you have to instantiate a connector for each of them. The method registration
   is done implicitly in the constructor.</para>
  </callout>

  <callout id="devel-srv-cls-4" arearefs="devel-srv-cls-4-co">
   <para>Register the free functions. In case you have several files with free
   functions you need to call each of the according register functions.</para>
  </callout>

  <callout id="devel-srv-cls-5" arearefs="devel-srv-cls-5-co">
   <para>Once your server has finished you should clean up and remove all
   methods pointing to free functions. The deregistration of methods in
   objects is implicitly done in their destructors when the object leaves its
   scope.</para>
  </callout>

</calloutlist>

<programlisting>

  ulxr::TcpIpConnection server_conn (true, host, port);
  ulxr::HttpProtocol server_prot(&amp;server_conn);

  UlxrIdlTest worker;                                         <co  id="devel-srv-cls-1-co" linkends="devel-srv-cls-1" />
  ulxr::MultiThreadRpcServer handler(&amp;server_prot, 2, false); <co  id="devel-srv-cls-2-co" linkends="devel-srv-cls-2" />
  UlxrIdlTestServer server(handler, worker);                  <co  id="devel-srv-cls-3-co" linkends="devel-srv-cls-3" />
  UlxrIdlTestFuncs_setupServerMethods(handler);               <co  id="devel-srv-cls-4-co" linkends="devel-srv-cls-4" />

  ...

  UlxrIdlTestFuncs_removeServerMethods(handler);              <co  id="devel-srv-cls-5-co" linkends="devel-srv-cls-5" />

</programlisting>

</para>

<para>Preparing the client code requires also some additional steps:

<calloutlist>
  <callout id="devel-cln-cls-1" arearefs="devel-cln-cls-1-co">
    <para>Store the credentials for remote access and the name of the remote resource
   in variables since you might need them several times for different objects.</para>
  </callout>

  <callout id="devel-cln-cls-2" arearefs="devel-cln-cls-2-co">
   <para>Instantiate the proxy object for the remote methods.</para>
  </callout>

  <callout id="devel-cln-cls-3" arearefs="devel-cln-cls-3-co">
   <para>Free functions need additional setup for each file with exported
   functions.</para>
  </callout>

  <callout id="devel-cln-cls-4" arearefs="devel-cln-cls-4-co">
   <para>Since the proxy object supports
   all the "exported" methods from the object on the server side the code looks
   exactly the same as it would on the remote machine.</para>
  </callout>

</calloutlist>

<programlisting>

  ulxr::TcpIpConnection client_conn (false, "localhost", 32000);
  ulxr::HttpProtocol client_prot(&amp;client_conn);
  ulxr::Requester requester(&amp;client_prot);

  ulxr::CppString user = "ali-baba";                                     <co  id="devel-cln-cls-1-co" linkends="devel-cln-cls-1" />
  ulxr::CppString pass = "open-sesame";
  ulxr::CppString realm = "/RPC2";

  UlxrIdlTestClient client(requester, realm, user, pass);                <co  id="devel-cln-cls-2-co" linkends="devel-cln-cls-2" />
  UlxrIdlTestFuncs_setClientCredentials(requester, realm, user, pass);   <co  id="devel-cln-cls-3-co" linkends="devel-cln-cls-3" />

  client.remoteCall("abc");                                              <co  id="devel-cln-cls-4-co" linkends="devel-cln-cls-4" />
  remoteFunction(1,2);

</programlisting>

</para>

<para>And you should not forget to give the &xmlrpc; methods meaningful names.
The generator prepares macros which contain the method names. Both the macro and
method names are based on the class and methods to achieve unique naming. Most of
time you will want to choose better method names by editing the according header
<filename>*_names.h</filename>. Don't forget that this file is never overwritten. So if the number of
exported methods increases you have to copy-and-paste the new methods manually from
<filename>*_names.h.new</filename>. For better orientation a comment with the complete
method signature is prepended to each <token>#define</token>:

<programlisting>

// mapped to: std::wstring * UlxrIdlTest::nextView1(int i, long int * l, std::string &amp; s, const bool * b, char c) const
#define ULXR_CALLTO_UlxrIdlTest_nextView1 \
  ULXR_PCHAR("UlxrIdlTest_nextView1")

</programlisting>

</para>

<para>The last step is to complete the method description in the user files named
<filename>*_server_user.cpp</filename>. Just change the last line of each block
to a meaningful comment about the remote method:

<programlisting>

  // mapped to: void first_url4();   (there are overloaded methods)
  method_adder.addMethod(ulxr::make_method(*this, &amp;UlxrIdlTestServer::first_url4_ovr4),
                         ulxr::Void::getValueName(),
                         ULXR_CALLTO_UlxrIdlTest_first_url4_ovr4,
                         ulxr::Signature(),
                         ulxr_i18n(ULXR_PCHAR("Some descriptive comment about 'void UlxrIdlTest::first_url4()'."))); // TODO adjust comment

</programlisting>

</para>

</sect2>

</sect1>

<sect1 id="devel-introspector">
<title>Sources From Introspection</title>

<para>Another possibility to create source code is to "ask" the &xmlrpc; server
about the available methods and generate minimal source code fragments.
The fragments also contain all available information like method signature and
description.</para>

<para>To gather this information you invoke application <filename>introspect</filename>.
Most of the &xmlrpc; servers support some standard methods which return information
about the available methods:

<screen>

  introspect --host=localhost:32000  --cpp-impl=dump.cpp --cpp-decl=dump.h

</screen>

There are several options to control this application:

<variablelist>

  <varlistentry>
    <term><command>--host=hostname/resource:port</command></term>
    <listitem><para>The most important option is the desired &xmlrpc; host.
    Optionally you can add a resource name and a port number in standard notation.
    The resource defaults
    to <constant>RPC2</constant> an the standard port number is 80. Therefor
    <userinput>--host=localhost</userinput> is the same as
    <userinput>--host=localhost:80/RPC2</userinput>.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--cpp-decl=filename</command></term>
    <listitem><para>This enables the gneration of the source code fragment for the
    declaration in a C++ class header. You can either <token>#include</token>
    this file or copy-and-paste it.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--cpp-impl=filename</command></term>
    <listitem><para>The second code fragment concerns the implementation
    of the member variables and is intended to be placed in a constructor
    list.</para></listitem>
  </varlistentry>

  <varlistentry>
    <term><command>--use-proxy=hostname:port</command></term>
    <listitem><para>In case you are located behind a HTTP proxy or firewall you will need
    to pass the according proxy to connect to the remote server.</para></listitem>
  </varlistentry>

</variablelist>

</para>

<para>The generated header fragment will contain one or more blocks similar
to the following:

<programlisting>

  // signature: struct system.getCapabilities(void)                 <co  id="devel-intro1-co" linkends="devel-intro1" />
  // purpose:   Returns Structs describing available capabilities.  <co  id="devel-intro2-co" linkends="devel-intro2" />
  ulxr::MethodCall system_getCapabilities;                          <co  id="devel-intro3-co" linkends="devel-intro3" />

</programlisting>

<calloutlist>
  <callout id="devel-intro1" arearefs="devel-intro1-co">
    <para>A simplified signature of the method.</para>
  </callout>

  <callout id="devel-intro2" arearefs="devel-intro2-co">
   <para>A short description.</para>
  </callout>

  <callout id="devel-intro3" arearefs="devel-intro3-co">
   <para>A variable declation of this method call.</para>
  </callout>

</calloutlist>

</para>

<para>The generated implementation lists the member contructors and
basically only contains the according method names:

<programlisting>

  , system_getCapabilities (ULXR_PCHAR("system.getCapabilities"))
  , system_listMethods (ULXR_PCHAR("system.listMethods"))
  , system_methodHelp (ULXR_PCHAR("system.methodHelp"))
  , system_methodSignature (ULXR_PCHAR("system.methodSignature"))

</programlisting>

</para>

</sect1>

</chapter>


