{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input",
     "active-ipynb",
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    from openmdao.utils.notebook_utils import notebook_mode\n",
    "except ImportError:\n",
    "    !python -m pip install openmdao[notebooks]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ExternalCodeComp\n",
    "\n",
    "ExternalCodeComp is a component that runs an external program in a subprocess on your operating system.\n",
    "\n",
    "If external programs do not have Python APIs, it is necessary to \"file wrap\" them.\n",
    "`ExternalCodeComp` is a utility component that makes file wrapping easier by\n",
    "taking care of the mundane tasks associated with executing the external application.\n",
    "These include:\n",
    "\n",
    "- Making the system call using the Subprocess module\n",
    "- Redirecting `stdin, stdout,` and `stderr` to the user's specification\n",
    "- Capturing error codes\n",
    "- Defining environment variables\n",
    "- Handling timeout and polling\n",
    "- Running the code on a remote server if required\n",
    "\n",
    "## ExternalCodeComp Options\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input"
    ]
   },
   "outputs": [],
   "source": [
    "import openmdao.api as om\n",
    "om.show_options_table(\"openmdao.components.external_code_comp.ExternalCodeComp\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## ExternalCodeComp Constructor\n",
    "\n",
    "The call signature for the `ExternalCodeComp` constructor is:\n",
    "\n",
    "```{eval-rst}\n",
    "    .. automethod:: openmdao.components.external_code_comp.ExternalCodeComp.__init__\n",
    "        :noindex:\n",
    "```\n",
    "\n",
    "## ExternalCodeComp Example\n",
    "\n",
    "In this example we will give an example based on a common scenario of a code that takes\n",
    "its inputs from an input file, performs some computations, and then writes the results\n",
    "to an output file. `ExternalCodeComp` supports multiple input and output files but\n",
    "for simplicity, this example only uses one of each.  Also, for the purposes of this\n",
    "example we have kept the input and output files as simple as possible. In practice,\n",
    "the data will likely be organized in some defined way and thus some care must be taken\n",
    "to read and write the data as dictated by the file format. OpenMDAO provides a set\n",
    "of [File Wrapping](../../../other_useful_docs/file_wrap.ipynb) tools to help with this.\n",
    "\n",
    "```{Note}\n",
    "To make it easy for you to run our example external code in any operating system or environment,\n",
    "we built it as a Python script that evaluates the paraboloid\n",
    "equation. We'll just call this script like any other executable, even though it is a Python script,\n",
    "and could be turned directly an OpenMDAO `Component`. Just keep in mind that any external code will\n",
    "work here, not just python scripts!\n",
    "```\n",
    "Here is the script for this external code. It simply reads its inputs, `x` and `y`, from an external file,\n",
    "does the same computation as the [Paraboloid Tutorial](../../../basic_user_guide/single_disciplinary_optimization/first_analysis) and writes the output, `f_xy`, to an output file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "#!/usr/bin/env python\n",
    "#\n",
    "# usage: extcode_paraboloid.py input_filename output_filename\n",
    "#\n",
    "# Evaluates the equation f(x,y) = (x-3)^2 + xy + (y+4)^2 - 3.\n",
    "#\n",
    "# Read the values of `x` and `y` from input file\n",
    "# and write the value of `f_xy` to output file.\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    import sys\n",
    "\n",
    "    input_filename = sys.argv[1]\n",
    "    output_filename = sys.argv[2]\n",
    "\n",
    "    with open(input_filename, 'r') as input_file:\n",
    "        file_contents = input_file.readlines()\n",
    "\n",
    "    x, y = [float(f) for f in file_contents]\n",
    "\n",
    "    f_xy = (x-3.0)**2 + x*y + (y+4.0)**2 - 3.0\n",
    "\n",
    "    with open(output_filename, 'w') as output_file:\n",
    "        output_file.write('%.16f\\n' % f_xy)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following example demonstrates how to build an OpenMDAO component that makes use of this external code.\n",
    "\n",
    "```{Note}\n",
    "If you pass a string as a command, OpenMDAO sets `shell=True` which can add [overhead](https://docs.python.org/3/library/subprocess.html#frequently-used-arguments) leading to\n",
    "a decrease in performance and a security [loophole](https://docs.python.org/3/library/subprocess.html#security-considerations). Use list when possible.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import openmdao.api as om\n",
    "\n",
    "\n",
    "class ParaboloidExternalCodeComp(om.ExternalCodeComp):\n",
    "    def setup(self):\n",
    "        self.add_input('x', val=0.0)\n",
    "        self.add_input('y', val=0.0)\n",
    "\n",
    "        self.add_output('f_xy', val=0.0)\n",
    "\n",
    "        self.input_file = 'paraboloid_input.dat'\n",
    "        self.output_file = 'paraboloid_output.dat'\n",
    "\n",
    "        # providing these is optional; the component will verify that any input\n",
    "        # files exist before execution and that the output files exist after.\n",
    "        self.options['external_input_files'] = [self.input_file]\n",
    "        self.options['external_output_files'] = [self.output_file]\n",
    "\n",
    "        # If you want to write your command as a list, the code below will also work.\n",
    "        # self.options['command'] = [\n",
    "        #     sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file\n",
    "        # ]\n",
    "\n",
    "        self.options['command'] = ('python extcode_paraboloid.py {} {}').format(self.input_file, self.output_file)\n",
    "\n",
    "    def compute(self, inputs, outputs):\n",
    "        x = inputs['x']\n",
    "        y = inputs['y']\n",
    "\n",
    "        # generate the input file for the paraboloid external code\n",
    "        with open(self.input_file, 'w') as input_file:\n",
    "            input_file.write('%.16f\\n%.16f\\n' % (x, y))\n",
    "\n",
    "        # the parent compute function actually runs the external code\n",
    "        super().compute(inputs, outputs)\n",
    "\n",
    "        # parse the output file from the external code and set the value of f_xy\n",
    "        with open(self.output_file, 'r') as output_file:\n",
    "            f_xy = float(output_file.read())\n",
    "\n",
    "        outputs['f_xy'] = f_xy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will go through each section and explain how this code works.\n",
    "\n",
    "OpenMDAO provides a base class, `ExternalCodeComp`, which you should inherit from to\n",
    "build your wrapper components. Just like any other component, you will define the\n",
    "necessary inputs and outputs in the `setup` method.\n",
    "If you want the component to check to make sure any files exist before/after you run,\n",
    "then you can set the `external_input_files` and `external_output_files`, respectively.\n",
    "You'll also define the command that should be called by the external code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def setup(self):\n",
    "    self.add_input('x', val=0.0)\n",
    "    self.add_input('y', val=0.0)\n",
    "\n",
    "    self.add_output('f_xy', val=0.0)\n",
    "\n",
    "    self.input_file = 'paraboloid_input.dat'\n",
    "    self.output_file = 'paraboloid_output.dat'\n",
    "\n",
    "    # providing these is optional; the component will verify that any input\n",
    "    # files exist before execution and that the output files exist after.\n",
    "    self.options['external_input_files'] = [self.input_file]\n",
    "    self.options['external_output_files'] = [self.output_file]\n",
    "\n",
    "    # If you want to write your command as a list, the code below will also work.\n",
    "    # self.options['command'] = [\n",
    "    #     sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file\n",
    "    # ]\n",
    "\n",
    "    self.options['command'] = ('python extcode_paraboloid.py {} {}').format(self.input_file, self.output_file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `compute` method is responsible for calculating outputs for a given set of inputs. When running an external code, this means you have to take the parameter values and push them down into files, run your code, then pull the output values back up. So there is some Python code needed to do all that file writing, reading, and parsing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute(self, inputs, outputs):\n",
    "    x = inputs['x']\n",
    "    y = inputs['y']\n",
    "\n",
    "    # generate the input file for the paraboloid external code\n",
    "    with open(self.input_file, 'w') as input_file:\n",
    "        input_file.write('%.16f\\n%.16f\\n' % (x, y))\n",
    "\n",
    "    # the parent compute function actually runs the external code\n",
    "    super().compute(inputs, outputs)\n",
    "\n",
    "    # parse the output file from the external code and set the value of f_xy\n",
    "    with open(self.output_file, 'r') as output_file:\n",
    "        f_xy = float(output_file.read())\n",
    "\n",
    "    outputs['f_xy'] = f_xy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ParaboloidExternalCodeComp` is now complete. All that is left is to actually use it in a model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "prob = om.Problem()\n",
    "model = prob.model\n",
    "\n",
    "model.add_subsystem('p', ParaboloidExternalCodeComp(), promotes_inputs=['x', 'y'])\n",
    "\n",
    "# run the ExternalCodeComp Component\n",
    "prob.setup()\n",
    "\n",
    "# Set input values\n",
    "prob.set_val('p.x', 3.0)\n",
    "prob.set_val('p.y', -4.0)\n",
    "\n",
    "prob.run_model()\n",
    "\n",
    "# print the output\n",
    "print(prob.get_val('p.f_xy'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input",
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "assert(prob.get_val('p.f_xy') == -15.0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using ExternalCodeComp in an Optimization\n",
    "\n",
    "If you are going to use an ExternalCodeComp component in a gradient based optimization, you'll need to\n",
    "get its [partial derivatives](../../../advanced_user_guide/analytic_derivatives/partial_derivs_explicit) somehow.\n",
    "One way would be just to use [finite-difference approximations](../../core_features/working_with_derivatives/approximating_partial_derivatives) for the partials.\n",
    "\n",
    "In the following example, the `ParaboloidExternalCodeComp` component has been modified to specify\n",
    "that partial derivatives are approximiated via finite difference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ParaboloidExternalCodeCompFD(om.ExternalCodeComp):\n",
    "    def setup(self):\n",
    "        self.add_input('x', val=0.0)\n",
    "        self.add_input('y', val=0.0)\n",
    "\n",
    "        self.add_output('f_xy', val=0.0)\n",
    "\n",
    "        self.input_file = 'paraboloid_input.dat'\n",
    "        self.output_file = 'paraboloid_output.dat'\n",
    "\n",
    "        # providing these is optional; the component will verify that any input\n",
    "        # files exist before execution and that the output files exist after.\n",
    "        self.options['external_input_files'] = [self.input_file]\n",
    "        self.options['external_output_files'] = [self.output_file]\n",
    "\n",
    "        self.options['command'] = [\n",
    "            sys.executable, 'extcode_paraboloid.py', self.input_file, self.output_file\n",
    "        ]\n",
    "\n",
    "    def setup_partials(self):\n",
    "        # this external code does not provide derivatives, use finite difference\n",
    "        self.declare_partials(of='*', wrt='*', method='fd')\n",
    "\n",
    "    def compute(self, inputs, outputs):\n",
    "        x = inputs['x']\n",
    "        y = inputs['y']\n",
    "\n",
    "        # generate the input file for the paraboloid external code\n",
    "        with open(self.input_file, 'w') as input_file:\n",
    "            input_file.write('%.16f\\n%.16f\\n' % (x, y))\n",
    "\n",
    "        # the parent compute function actually runs the external code\n",
    "        super().compute(inputs, outputs)\n",
    "\n",
    "        # parse the output file from the external code and set the value of f_xy\n",
    "        with open(self.output_file, 'r') as output_file:\n",
    "            f_xy = float(output_file.read())\n",
    "\n",
    "        outputs['f_xy'] = f_xy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can perform an optimization using the external code, as shown here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = om.Problem()\n",
    "model = prob.model\n",
    "\n",
    "model.add_subsystem('p', ParaboloidExternalCodeCompFD())\n",
    "\n",
    "# find optimal solution with SciPy optimize\n",
    "# solution (minimum): x = 6.6667; y = -7.3333\n",
    "prob.driver = om.ScipyOptimizeDriver()\n",
    "prob.driver.options['optimizer'] = 'SLSQP'\n",
    "\n",
    "prob.model.add_design_var('p.x', lower=-50, upper=50)\n",
    "prob.model.add_design_var('p.y', lower=-50, upper=50)\n",
    "\n",
    "prob.model.add_objective('p.f_xy')\n",
    "\n",
    "prob.driver.options['tol'] = 1e-9\n",
    "prob.driver.options['disp'] = True\n",
    "\n",
    "prob.setup()\n",
    "\n",
    "# Set input values\n",
    "prob.set_val('p.x', 3.0)\n",
    "prob.set_val('p.y', -4.0)\n",
    "\n",
    "prob.run_driver()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(prob.get_val('p.x'))\n",
    "print(prob.get_val('p.y'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input",
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "from openmdao.utils.assert_utils import assert_near_equal\n",
    "\n",
    "assert_near_equal(prob.get_val('p.x'), 6.66666667, 1e-6)\n",
    "assert_near_equal(prob.get_val('p.y'), -7.3333333, 1e-6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, if the code you are wrapping happens to provide analytic derivatives you could\n",
    "have those written out to a file and then parse that file in the\n",
    "[compute_partials](../../core_features/working_with_components/explicit_component) method.\n",
    "\n",
    "Here is a version of our external script that writes its derivatives to a second output file:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```python\n",
    "#!/usr/bin/env python\n",
    "#\n",
    "# usage: extcode_paraboloid_derivs.py input_filename output_filename derivs_filename\n",
    "#\n",
    "# Evaluates the equation f(x,y) = (x-3)^2 + xy + (y+4)^2 - 3.\n",
    "#\n",
    "# Read the values of `x` and `y` from input file\n",
    "# and write the value of `f_xy` to output file.\n",
    "#\n",
    "# Also write derivatives to another output file.\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    import sys\n",
    "\n",
    "    input_filename = sys.argv[1]\n",
    "    output_filename = sys.argv[2]\n",
    "    derivs_filename = sys.argv[3]\n",
    "\n",
    "    with open(input_filename, 'r') as input_file:\n",
    "        file_contents = input_file.readlines()\n",
    "\n",
    "    x, y = [float(f) for f in file_contents]\n",
    "\n",
    "    f_xy = (x-3.0)**2 + x*y + (y+4.0)**2 - 3.0\n",
    "\n",
    "    with open(output_filename, 'w') as output_file:\n",
    "        output_file.write('%.16f\\n' % f_xy)\n",
    "\n",
    "    with open(derivs_filename, 'w') as derivs_file:\n",
    "        # partials['f_xy', 'x']\n",
    "        derivs_file.write('%.16f\\n' % (2.0*x - 6.0 + y))\n",
    "        # partials['f_xy', 'y']\n",
    "        derivs_file.write('%.16f\\n' % (2.0*y + 8.0 + x))\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the corresponding `ParaboloidExternalCodeCompDerivs` component:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class ParaboloidExternalCodeCompDerivs(om.ExternalCodeComp):\n",
    "    def setup(self):\n",
    "        self.add_input('x', val=0.0)\n",
    "        self.add_input('y', val=0.0)\n",
    "\n",
    "        self.add_output('f_xy', val=0.0)\n",
    "\n",
    "        self.input_file = 'paraboloid_input.dat'\n",
    "        self.output_file = 'paraboloid_output.dat'\n",
    "        self.derivs_file = 'paraboloid_derivs.dat'\n",
    "\n",
    "        # providing these is optional; the component will verify that any input\n",
    "        # files exist before execution and that the output files exist after.\n",
    "        self.options['external_input_files'] = [self.input_file]\n",
    "        self.options['external_output_files'] = [self.output_file, self.derivs_file]\n",
    "\n",
    "        self.options['command'] = [\n",
    "            sys.executable, 'extcode_paraboloid_derivs.py',\n",
    "            self.input_file, self.output_file, self.derivs_file\n",
    "        ]\n",
    "\n",
    "    def setup_partials(self):\n",
    "        # this external code does provide derivatives\n",
    "        self.declare_partials(of='*', wrt='*')\n",
    "\n",
    "    def compute(self, inputs, outputs):\n",
    "        x = inputs['x']\n",
    "        y = inputs['y']\n",
    "\n",
    "        # generate the input file for the paraboloid external code\n",
    "        with open(self.input_file, 'w') as input_file:\n",
    "            input_file.write('%.16f\\n%.16f\\n' % (x, y))\n",
    "\n",
    "        # the parent compute function actually runs the external code\n",
    "        super().compute(inputs, outputs)\n",
    "\n",
    "        # parse the output file from the external code and set the value of f_xy\n",
    "        with open(self.output_file, 'r') as output_file:\n",
    "            f_xy = float(output_file.read())\n",
    "\n",
    "        outputs['f_xy'] = f_xy\n",
    "\n",
    "    def compute_partials(self, inputs, partials):\n",
    "        outputs = {}\n",
    "\n",
    "        # the parent compute function actually runs the external code\n",
    "        super().compute(inputs, outputs)\n",
    "\n",
    "        # parse the derivs file from the external code and set partials\n",
    "        with open(self.derivs_file, 'r') as derivs_file:\n",
    "            partials['f_xy', 'x'] = float(derivs_file.readline())\n",
    "            partials['f_xy', 'y'] = float(derivs_file.readline())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we can perform an optimization using the external code with derivatives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prob = om.Problem()\n",
    "model = prob.model\n",
    "\n",
    "model.add_subsystem('p', ParaboloidExternalCodeCompDerivs())\n",
    "\n",
    "# find optimal solution with SciPy optimize\n",
    "# solution (minimum): x = 6.6667; y = -7.3333\n",
    "prob.driver = om.ScipyOptimizeDriver()\n",
    "prob.driver.options['optimizer'] = 'SLSQP'\n",
    "\n",
    "prob.model.add_design_var('p.x', lower=-50, upper=50)\n",
    "prob.model.add_design_var('p.y', lower=-50, upper=50)\n",
    "\n",
    "prob.model.add_objective('p.f_xy')\n",
    "\n",
    "prob.driver.options['tol'] = 1e-9\n",
    "prob.driver.options['disp'] = True\n",
    "\n",
    "prob.setup()\n",
    "\n",
    "# Set input values\n",
    "prob.set_val('p.x', 3.0)\n",
    "prob.set_val('p.y', -4.0)\n",
    "\n",
    "prob.run_driver();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(prob.get_val('p.x'))\n",
    "print(prob.get_val('p.y'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "remove-input",
     "remove-output"
    ]
   },
   "outputs": [],
   "source": [
    "assert_near_equal(prob.get_val('p.x'), 6.66666667, 1e-6)\n",
    "assert_near_equal(prob.get_val('p.y'), -7.3333333, 1e-6)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.1"
  },
  "orphan": true
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
