.. _api_mmalobj:

=============
API - mmalobj
=============

.. module:: picamera.mmalobj

.. currentmodule:: picamera.mmalobj

This module provides an object-oriented interface to ``libmmal`` which is the
library underlying picamera, ``raspistill``, and ``raspivid``.  It is provided
to ease the usage of ``libmmal`` to Python coders unfamiliar with C and also
works around some of the idiosyncrasies in ``libmmal``.

.. warning::

    This part of the API is still experimental and subject to change in future
    versions. Backwards compatibility is not (yet) guaranteed.


The MMAL Tour
=============

MMAL operates on the principle of pipelines:

* A pipeline consists of one or more MMAL components
  (:class:`MMALBaseComponent` and derivatives) connected together in series.

* A :class:`MMALBaseComponent` has one or more ports.

* A port (:class:`MMALControlPort` and derivatives) is either a control port,
  an input port or an output port (there are also clock ports but you generally
  don't need to deal with these as MMAL sets them up automatically):

  - Control ports are used to accept and receive commands, configuration
    parameters, and error messages. All MMAL components have a control port,
    but in picamera they're only used for component configuration.

  - Input ports receive data from upstream components.

  - Output ports send data onto downstream components (if they're connected),
    or to callback routines in the user's program (if they're not connected).

  - Input and output ports can be audio, video or sub-picture (subtitle) ports,
    but picamera only deals with video ports.

  - Ports have a :attr:`~MMALPort.format` which (in the case of video ports)
    dictates the format of image/frame accepted or generated by the port (YUV,
    RGB, JPEG, H.264, etc.)

  - Video ports have a :attr:`~MMALVideoPort.framerate` which specifies the
    number of images expected to be received or sent per second.

  - Video ports also have a :attr:`~MMALVideoPort.framesize` which specifies
    the resolution of images/frames accepted or generated by the port.

  - Finally, all ports (control, input and output) have
    :attr:`~MMALControlPort.params` which affect their operation.

* An output port can have a :class:`MMALConnection` to an input port.
  Connections ensure the two ports use compatible formats, and handle
  transferring data from output ports to input ports in an orderly fashion. A
  port cannot have more than one connection from/to it.

* Data is written to / read from ports via instances of :class:`MMALBuffer`.

  - Buffers belong to a port and can't be passed arbitrarily between ports.

  - The size of a buffer is dictated by the format and frame-size of the port
    that owns the buffer. The memory allocation of a buffer (readable from
    :attr:`~MMALBuffer.size`) cannot be altered once the port is enabled, but
    the buffer can contain any amount of data up its allocation size. The
    actual length of data in a buffer is stored in :attr:`~MMALBuffer.length`.

  - Likewise, the number of buffers belonging to a port is fixed and cannot be
    altered without disabling the port, reconfiguring it and re-enabling it.
    The more buffers a port has, the less likely it is that the pipeline will
    have to drop frames because a component has overrun, but the more GPU
    memory is required.

  - Buffers also have :attr:`~MMALBuffer.flags` which specify information about
    the data they contain (e.g. start of frame, end of frame, key frame, etc.)

  - When a connection exists between two ports, the connection continually
    requests a buffer from the output port, requests another buffer from the
    input port, copies the output buffer's data to the input buffer's data,
    then returns the buffers to their respective ports (this is a
    simplification; various tricks are pulled under the covers to minimize the
    amount of data copying that *actually* occurs, but as a mental model of
    what's going on it's reasonable).

  - Components take buffers from their input port(s), process them, and write
    the result into a buffer from the output port(s).


Components
----------

Now we've got a mental model of what an MMAL pipeline consists of, let's build
one. For the rest of the tour I strongly recommend using a Pi with a screen (so
you can see preview output) but controlling it via an SSH session (so the
preview doesn't cover your command line). Follow along, typing the examples
into your remote Python session. And feel free to deviate from the examples if
you're curious about things!

We'll start by importing the :mod:`~picamera.mmalobj` module with a convenient
alias, then construct a :class:`MMALCamera` component, and a
:class:`MMALRenderer` component.

.. code-block:: pycon

    >>> from picamera import mmal, mmalobj as mo
    >>> camera = mo.MMALCamera()
    >>> preview = mo.MMALRenderer()


Ports
-----

Before going any further, let's have a look at the various ports on these
components.

.. code-block:: pycon

    >>> len(camera.inputs)
    0
    >>> len(camera.outputs)
    3
    >>> len(preview.inputs)
    1
    >>> len(preview.outputs)
    0

The fact the camera has three outputs should come as little surprise to those
who have read the :ref:`camera_hardware` chapter (if you haven't already, you
might want to skim it now). Let's examine the first output port of the camera
and the input port of the renderer:

.. code-block:: pycon

    >>> camera.outputs[0]
    <MMALVideoPort "vc.ril.camera:out:0": format=MMAL_FOURCC('I420')
    buffers=1x7680 frames=320x240@0fps>
    >>> preview.inputs[0]
    <MMALVideoPort "vc.ril.video_render:in:0" format=MMAL_FOURCC('I420')
    buffers=2x15360 frames=160x64@0fps>

Several things to note here:

* We can tell from the port name what sort of component it belongs to, what
  its index is, and whether it's an input or an output port

* Both ports are currently configured for the I420 format; this is MMAL's
  name for `YUV420`_ (full resolution Y, quarter resolution UV).

* The ports have different frame-sizes (320x240 and 160x64 respectively),
  buffer counts (1 and 2 respectively) and buffer sizes (7680 and 15360
  respectively).

* The buffer sizes look unrealistic. For example, 7680 bytes is nowhere near
  enough to hold 320 * 240 * 1.5 bytes (YUV420 requires 1.5 bytes per pixel).

Now we'll configure the camera's output port with a slightly higher resolution,
and give it a frame-rate:

.. code-block:: pycon

    >>> camera.outputs[0].framesize = (640, 480)
    >>> camera.outputs[0].framerate = 30
    >>> camera.outputs[0].commit()
    >>> camera.outputs[0]
    <MMALVideoPort "vc.ril.camera:out:0(I420)": format=MMAL_FOURCC('I420')
    buffers=1x460800 frames=640x480@30fps>

Note that the changes to the configuration won't actually take effect until
the :meth:`~MMALPort.commit` call. After the port is committed, note that the
buffer size now looks reasonable: 640 * 480 * 1.5 = 460800.


Connections
-----------

Now we'll try connecting the renderer's input to the camera's output. Don't
worry about the fact that the port configurations are different. One of the
nice things about MMAL (and the ``mmalobj`` layer) is that connections try very
hard to auto-configure things so that they "just work". Usually,
auto-configuration is based upon the *output* port being connected so it's
important to get that configuration right, but you don't generally need to
worry about the input port.

The renderer is what ``mmalobj`` terms a "downstream component". This is a
component with a single input that typically sits downstream from some feeder
component (like a camera). All such components have the
:meth:`~MMALComponent.connect` method which can be used to connect the
sole input to a specified output:

.. code-block:: pycon

    >>> preview.connect(camera)
    <MMALConnection "vc.ril.camera:out:0/vc.ril.video_render:in:0">
    >>> preview.connection.enable()

Note that we've been quite lazy in the snippet above by simply calling
:meth:`~MMALComponent.connect` with the ``camera`` component. In this case, a
connection will be attempted between the first input port of the owner
(``preview``) and the *first unconnected* output of the parameter (``camera``).
However, this is not always what's wanted so you can specify the exact ports
you wish to connect. In this case the example was equivalent to calling:

.. code-block:: pycon

    >>> preview.inputs[0].connect(camera.outputs[0])
    <MMALConnection "vc.ril.camera:out:0/vc.ril.video_render:in:0">
    >>> preview.inputs[0].connection.enable()

Note that the :meth:`~MMALComponent.connect` method returns the connection
that was constructed but you can also retrieve this by querying the port's
:attr:`~MMALPort.connection` attribute later.

As soon as the connection is enabled you should see the camera preview appear
on the Pi's screen. Let's query the port configurations now:

.. code-block:: pycon

    >>> camera.outputs[0]
    <MMALVideoPort "vc.ril.camera:out:0(OPQV)": format=MMAL_FOURCC('OPQV')
    buffers=10x128 frames=640x480@30fps>
    >>> preview.inputs[0]
    <MMALVideoPort "vc.ril.video_render:in:0(OPQV)": format=MMAL_FOURCC('OPQV')
    buffers=10x128 frames=640x480@30fps>

Note that the connection has implicitly reconfigured the camera's output port
to use the OPAQUE ("OPQV") format. This is a special format used internally by
the camera firmware which avoids passing complete frame data around, instead
passing pointers to frame data around (this explains the tiny buffer size of
128 bytes as very little data is actually being shuttled between the
components). Further, note that the connection has automatically copied the
port format, frame size and frame-rate to the preview's input port.

.. image:: images/preview_pipeline.*
    :align: center


Opaque Format
-------------

At this point it is worth exploring the differences between the camera's
three output ports:

* Output 0 is the "preview" output. On this port, the OPAQUE format contains a
  pointer to a complete frame of data.

* Output 1 is the "video recording" output. On this port, the OPAQUE format
  contains a pointer to *two* complete frames of data. The dual-frame format
  enables the H.264 video encoder to calculate motion estimation without the
  encoder having to keep copies of prior frames itself (it can do this when
  something other than OPAQUE format is used, but dual-image OPAQUE is *much*
  more efficient).

* Output 2 is the "still image" output. On this port, the OPAQUE format
  contains a pointer to a strip of an image. The "strips" format is used by the
  JPEG encoder (not to be confused with the MJPEG encoder) to deal with high
  resolution images efficiently.

Generally, you don't need to worry about these differences. The ``mmalobj``
layer knows about them and negotiates the most efficient format it can for
connections. However, they're worth bearing in mind if you're aiming to get the
most out of the firmware or if you're confused about why a particular format
has been selected for a connection.


Component Configuration
-----------------------

So far we've seen how to construct components, configure their ports, and
connect them together in rudimentary pipelines. Now, let's see how to configure
components via control port parameters:

.. code-block:: pycon

    >>> camera.control.params[mmal.MMAL_PARAMETER_SYSTEM_TIME]
    177572014208
    >>> camera.control.params[mmal.MMAL_PARAMETER_SYSTEM_TIME]
    177574350658
    >>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS]
    Fraction(1, 2)
    >>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS] = 0.75
    >>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS]
    Fraction(3, 4)
    >>> fx = camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT]
    >>> fx
    <picamera.mmal.MMAL_PARAMETER_IMAGEFX_T object at 0x765b8440>
    >>> dir(fx)
    ['__class__', '__ctypes_from_outparam__', '__delattr__', '__dict__',
    '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
    '__gt__', '__hash__', '__init__', '__le__', '__lt__', '__module__',
    '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__',
    '__setattr__', '__setstate__', '__sizeof__', '__str__', '__subclasshook__',
    '__weakref__', '_b_base_', '_b_needsfree_', '_fields_', '_objects', 'hdr',
    'value']
    >>> fx.value
    0
    >>> mmal.MMAL_PARAM_IMAGEFX_NONE
    0
    >>> fx.value = mmal.MMAL_PARAM_IMAGEFX_EMBOSS
    >>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = fx
    >>> camera.control.params[mmal.MMAL_PARAMETER_BRIGHTNESS] = 1/2
    >>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = mmal.MMAL_PARAM_IMAGEFX_NONE
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/home/pi/picamera/picamera/mmalobj.py", line 1109, in __setitem__
        assert mp.hdr.id == key
    AttributeError: 'int' object has no attribute 'hdr'
    >>> fx.value = mmal.MMAL_PARAM_IMAGEFX_NONE
    >>> camera.control.params[mmal.MMAL_PARAMETER_IMAGE_EFFECT] = fx
    >>> preview.disconnect()

Things to note:

* The parameter dictates the type of the value returned (and accepted, if the
  parameter is read-write).

* Many parameters accept a multitude of simple types like :class:`int`,
  :class:`float`, :class:`~fractions.Fraction`, :class:`str`, etc. However,
  some parameters use :mod:`ctypes` structures and such parameters only accept
  the relevant structure.

* The easiest way to use such "structured" parameters is to query them first,
  modify the resulting structure, then write it back to the parameter.

To find out what parameters are available for use with the camera component,
have a look at the source for the :class:`~picamera.PiCamera` class, especially
property getters and setters.


File Output (RGB capture)
-------------------------

Let's see how we can produce some file output from the camera. First we'll
perform a straight unencoded RGB capture from the still port (2). As this is
unencoded output we don't need to construct anything else. All we need to do is
configure the port for RGB encoding, select an appropriate resolution, then
activate the output port:

.. code-block:: pycon

    >>> camera.outputs[2].format = mmal.MMAL_ENCODING_RGB24
    >>> camera.outputs[2].framesize = (640, 480)
    >>> camera.outputs[2].commit()
    >>> camera.outputs[2]
    <MMALVideoPort "vc.ril.camera:out:2(RGB3)": format=MMAL_FOURCC('RGB3')
    buffers=1x921600 frames=640x480@0fps>
    >>> camera.outputs[2].enable()

Unfortunately, that didn't seem to do much! An output port that is
participating in a connection needs nothing more: it knows where its data is
going. However, an output port *without* a connection requires a callback
function to be assigned so that something can be done with the buffers of data
it produces.

The callback will be given two parameters: the :class:`MMALPort` responsible
for producing the data, and the :class:`MMALBuffer` containing the data. It is
expected to return a :class:`bool` which will be ``False`` if further data is
expected and ``True`` if no further data is expected. If ``True`` is returned,
the callback will not be executed again. In our case we're going to write data
out to a file we'll open before-hand, and we should return ``True`` when we see
a buffer with the "frame end" flag set:

.. code-block:: pycon

    >>> camera.outputs[2].disable()
    >>> import io
    >>> output = io.open('image.data', 'wb')
    >>> def image_callback(port, buf):
    ...     output.write(buf.data)
    ...     return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
    ...
    >>> camera.outputs[2].enable(image_callback)
    >>> output.tell()
    0

At this stage you may note that while the file exists, nothing's been written
to it. This is because output ports 1 and 2 (the video and still ports) won't
produce any buffers until their "capture" parameter is enabled:

.. code-block:: pycon

    >>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
    >>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
    >>> output.tell()
    921600
    >>> camera.outputs[2].disable()
    >>> output.close()

Congratulations! You've just captured your first image with the MMAL layer.
Given we disconnected the preview above, the current state of the system looks
something like this:

.. image:: images/rgb_capture_pipeline.*
    :align: center


File Output (JPEG capture)
--------------------------

Whilst RGB is a useful format for processing we'd generally prefer something
like JPEG for output. So, next we'll construct an MMAL JPEG encoder and use it
to compress our RGB capture. Note that we're not going to connect the JPEG
encoder to the camera yet; we're just going to construct it standalone and feed
it data from our capture file, writing the output to another file:

.. code-block:: pycon

    >>> encoder = mo.MMALImageEncoder()
    >>> encoder.inputs
    (<MMALVideoPort "vc.ril.image_encode:in:0": format=MMAL_FOURCC('RGB2')
    buffers=1x15360 frames=96x80@0fps>,)
    >>> encoder.outputs
    (<MMALVideoPort "vc.ril.image_encode:out:0": format=MMAL_FOURCC('GIF ')
    buffers=1x81920 frames=0x0@0fps>,)
    >>> encoder.inputs[0].format = mmal.MMAL_ENCODING_RGB24
    >>> encoder.inputs[0].framesize = (640, 480)
    >>> encoder.inputs[0].commit()
    >>> encoder.outputs[0].copy_from(encoder.inputs[0])
    >>> encoder.outputs[0]
    <MMALVideoPort "vc.ril.image_encode:out:0": format=MMAL_FOURCC('RGB3')
    buffers=1x81920 frames=640x480@0fps>
    >>> encoder.outputs[0].format = mmal.MMAL_ENCODING_JPEG
    >>> encoder.outputs[0].commit()
    >>> encoder.outputs[0]
    <MMALVideoPort "vc.ril.image_encode:out:0(JPEG)": format=MMAL_FOURCC('JPEG')
    buffers=1x307200 frames=0x0@0fps>
    >>> encoder.outputs[0].params[mmal.MMAL_PARAMETER_JPEG_Q_FACTOR] = 90

Just pausing for a moment, let's re-cap what we've got: an image encoder
constructed, configured for 640x480 RGB input, and JPEG output with a quality
factor of "90" (i.e. "very good" - don't try to read much more than this into
JPEG quality settings!). Note that MMAL has set the buffer size at a size it
thinks will be typical for the output. As JPEG is a lossy format this won't be
precise and it's entirely possible that we may receive multiple callbacks for
a single frame (if the compression overruns the expected buffer size).

Let's continue:

.. code-block:: pycon

    >>> rgb_data = io.open('image.data', 'rb')
    >>> jpg_data = io.open('image.jpg', 'wb')
    >>> def image_callback(port, buf):
    ...     jpg_data.write(buf.data)
    ...     return bool(buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END)
    ...
    >>> encoder.outputs[0].enable(image_callback)


File Input (JPEG encoding)
--------------------------

How do we feed data to a component without a connection? We enable its input
port with a dummy callback (we don't need to "do" anything on data input). Then
we request buffers from its input port, fill them with data and send them back
to the input port:

.. code-block:: pycon

    >>> encoder.inputs[0].enable(lamdba port, buf: True)
    >>> buf = encoder.inputs[0].get_buffer()
    >>> buf.data = rgb_data.read()
    >>> encoder.inputs[0].send_buffer(buf)
    >>> jpg_data.tell()
    87830
    >>> encoder.outputs[0].disable()
    >>> encoder.inputs[0].disable()
    >>> jpg_data.close()
    >>> rgb_data.close()

Congratulations again! You've just produced a hardware-accelerated JPEG
encoding. The following illustrates the state of the system at the moment (note
the camera and renderer still exist; they're just not connected to anything at
the moment):

.. image:: images/jpeg_encode_pipeline.*
    :align: center

Now let's repeat the process but with the encoder attached to the
still port on the camera directly. We can re-use our ``image_callback`` routine
from earlier and just assign a different output file to ``jpg_data``:

.. code-block:: pycon

    >>> encoder.connect(camera.outputs[2])
    <MMALConnection "vc.ril.camera:out:2/vc.ril.image_encode:in:0">
    >>> encoder.connection.enable()
    >>> encoder.inputs[0]
    <MMALVideoPort "vc.ril.image_encode:in:0(OPQV)": format=MMAL_FOURCC('OPQV')
    buffers=10x128 frames=640x480@0fps>
    >>> jpg_data = io.open('direct.jpg', 'wb')
    >>> encoder.outputs[0].enable(image_callback)
    >>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
    >>> camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
    >>> jpg_data.tell()
    99328
    >>> encoder.connection.disable()
    >>> jpg_data.close()

Now the state of our system looks like this:

.. image:: images/jpeg_capture_pipeline.*
    :align: center


Threads & Synchronization
-------------------------

The one issue you may have noted is that ``image_callback`` is running in a
background thread. If we were running our capture extremely fast our main
thread might disable the capture before our callback had run. Ideally we want
to activate capture, wait on some signal indicating that the callback has
completed a single frame successfully, then disable capture. We can do this
with the communications primitives from the standard :mod:`threading` module:

.. code-block:: pycon

    >>> from threading import Event
    >>> finished = Event()
    >>> def image_callback(port, buf):
    ...     jpg_data.write(buf.data)
    ...     if buf.flags & mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END:
    ...         finished.set()
    ...         return True
    ...     return False
    ...
    >>> def do_capture(filename='direct.jpg'):
    ...     global jpg_data
    ...     jpg_data = io.open(filename, 'wb')
    ...     finished.clear()
    ...     encoder.outputs[0].enable(image_callback)
    ...     camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = True
    ...     if not finished.wait(10):
    ...         raise Exception('capture timed out')
    ...     camera.outputs[2].params[mmal.MMAL_PARAMETER_CAPTURE] = False
    ...     encoder.outputs[0].disable()
    ...     jpg_data.close()
    ...
    >>> do_capture()

The above example has several rough edges: globals, no proper clean-up in the
case of an exception, etc. but by now you should be getting a pretty good idea
of how picamera operates under the hood.

The major difference between picamera and a "typical" MMAL setup is that upon
construction, the :class:`~picamera.PiCamera` class constructs both a
:class:`MMALCamera` (accessible as ``PiCamera._camera``) *and* a
:class:`MMALSplitter` (accessible as ``PiCamera._splitter``). The splitter
remains permanently attached to the camera's video port (output port 1).
Furthermore, there's *always* something connected to the camera's preview port;
by default it's a :class:`MMALNullSink` component which is switched with a
:class:`MMALRenderer` when the preview is started.

Encoders are constructed and destroyed as required by calls to
:meth:`~picamera.PiCamera.capture`, :meth:`~picamera.PiCamera.start_recording`,
etc. The following illustrates a typical picamera pipeline whilst video
recording without a preview:

.. image:: images/picamera_pipeline.*
    :align: center


Debugging Facilities
--------------------

Before we move onto the pure Python components it's worth mentioning the
debugging capabilities built into ``mmalobj``. Firstly, most objects have
useful :func:`repr` outputs (in particular, it can be useful to simply evaluate
a :class:`MMALBuffer` to see what flags it's got and how much data is stored in
it). Also, there's the :func:`print_pipeline` function. Give this a port and
it'll dump a human-readable version of your pipeline leading up to that port:

.. code-block:: pycon

    >>> preview.inputs[0].enable(lambda port, buf: True)
    >>> buf = preview.inputs[0].get_buffer()
    >>> buf
    <MMALBuffer object: flags=_____ length=0>
    >>> buf.flags = mmal.MMAL_BUFFER_HEADER_FLAG_FRAME_END
    >>> buf
    <MMALBuffer object: flags=E____ length=0>
    >>> buf.release()
    >>> preview.inputs[0].disable()
    >>> mo.print_pipeline(encoder.outputs[0])
     vc.ril.camera [2]                           [0] vc.ril.image_encode [0]
       encoding    OPQV-strips    -->    OPQV-strips      encoding       JPEG
          buf      10x128                     10x128         buf         1x307200
        bitrate    0bps                         0bps       bitrate       0bps
         frame     640x480@0fps         640x480@0fps        frame        0x0@0fps


Python Components
-----------------

So far all the components we've looked at have been "real" MMAL components
which is to say that they're implemented in C, and all talk to bits of the
firmware running on the GPU. However, a frequent request has been to be able to
modify frames from the camera before they reach the image or video encoder.
The Python components are an attempt to make this request relatively simple to
achieve from within Python.

The means by which this is achieved are inefficient (to say the least) so don't
expect this to work with high resolutions or framerates. The ``mmalobj`` layer
in picamera includes the concept of a "Python MMAL" component. To the user
these components look a lot like the MMAL components you've been playing with
above (:class:`MMALCamera`, :class:`MMALImageEncoder`, etc). They are
instantiated in a similar manner, they have the same sort of ports, and they're
connected using the same means as ordinary MMAL components.

Let's try this out by placing a transformation between the camera and a preview
which will draw a cross over the frames going to the preview. For this we'll
subclass :class:`picamera.array.PiArrayTransform`. This derives from
:class:`MMALPythonComponent` and provides the useful capability of
providing the source and target buffers as numpy arrays containing RGB data:

.. code-block:: pycon

    >>> from picamera import array
    >>> class Crosshair(array.PiArrayTransform):
    ...     def transform(self, source, target):
    ...         with source as sdata, target as tdata:
    ...             tdata[...] = sdata
    ...             tdata[240, :, :] = 0xff
    ...             tdata[:, 320, :] = 0xff
    ...         return False
    ...
    >>> transform = Crosshair()

That's all there is to constructing a transform! This one is a bit crude in
as much as the coordinates are hard-coded, and it's very simplistic, but it
should illustrate the principle nicely. Let's connect it up between the camera
and the renderer:

.. code-block:: pycon

    >>> transform.connect(camera)
    <MMALPythonConnection "vc.ril.camera.out:0(RGB3)/py.component:in:0">
    >>> preview.connect(transform)
    <MMALPythonConnection "py.component:out:0/vc.ril.video_render:in:0(RGB3)">
    >>> transform.connection.enable()
    >>> preview.connection.enable()
    >>> transform.enable()

At this point we should take a look at the pipeline to see what's been
configured automatically:

.. code-block:: pycon

    >>> mo.print_pipeline(preview.inputs[0])
     vc.ril.camera [0]                             [0] py.transform [0]                             [0] vc.ril.video_render
       encoding    RGB3            -->            RGB3   encoding   RGB3            -->            RGB3      encoding
          buf      1x921600                   2x921600     buf      2x921600                   2x921600         buf
         frame     640x480@30fps         640x480@30fps    frame     640x480@30fps         640x480@30fps        frame

Apparently the MMAL camera component is outputting RGB data (which is extremely
large) to a "py.transform" component, which draws our cross-hair on the buffer
and passes it onto the renderer again as RGB. This is part of the inefficiency
alluded to above: RGB is a very large format (compared to I420 which is half
its size, or OPQV which is tiny) so we're shuttling a *lot* of data around
here. Expect this to drop frames at higher resolutions or framerates.

The other source of inefficiency isn't obvious from the debug output above
which gives the impression that the "py.transform" component is actually part
of the MMAL pipeline. In fact, this is a lie. Under the covers ``mmalobj``
installs an output callback on the camera's output port to feed data to the
"py.transform" input port, uses a background thread to run the transform, then
copies the results into buffers obtained from the preview's input port. In
other words there's really *two* (very short) MMAL pipelines with a hunk of
Python running in between them. If ``mmalobj`` does its job properly you
shouldn't need to worry about this implementation detail but it's worth bearing
in mind from the perspective of performance.


Performance Hints
-----------------

Generally you want to your frame handlers to be *fast*. To avoid dropping
frames they've got to run in less than a frame's time (e.g.  33ms at 30fps).
Bear in mind that a significant amount of time is going to be spent shuttling
the huge RGB frames around so you've actually got much less than 33ms available
to you (how much will depend on the speed of your Pi, what resolution you're
using, the framerate, etc).

Sometimes, performance can mean making unintuitive choices. For example, the
`Pillow library`_ (the main imaging library in Python these days) can construct
images which share buffer memory (see ``Image.frombuffer``), but only for the
indexed (grayscale) and RGBA formats, not RGB. Hence, it can make sense to use
RGBA (a format even larger than RGB) if only because it allows you to avoid
copying any data when performing a composite.

Another trick is to realize that although YUV420 has different sized planes,
it's often enough to manipulate the Y plane only. In that case you can treat
the front of the buffer as an indexed image (remember that Pillow can share
buffer memory with such images) and manipulate that directly. With tricks like
these it's possible to perform multiple composites in realtime at 720p30 on a
Pi3.

Here's a (heavily commented) variant of the cross-hair example above that uses
the lower level :class:`MMALPythonComponent` class instead, and the `Pillow
library`_ to perform compositing on YUV420 in the manner just described:

.. literalinclude:: examples/mmal_crosshair.py

It's a sensible idea to perform any overlay rendering you want to do in a
separate thread and then just handle compositing your overlay onto the frame in
the :meth:`MMALPythonComponent._handle_frame` method. Anything you can do to
avoid buffer copying is a bonus here.

Here's a final (rather large) demonstration that puts all these things together
to construct a :class:`MMALPythonComponent` derivative with two purposes:

1. Render a partially transparent analogue clock in the top left of the frame.

2. Produces two equivalent I420 outputs; one for feeding to a preview renderer,
   and another to an encoder (we could use a proper MMAL splitter for this but
   this is a demonstration of how Python components can have multiple outputs
   too).

.. literalinclude:: examples/mmal_clock_splitter.py


.. _YUV420: https://en.wikipedia.org/wiki/YUV#Y.E2.80.B2UV420p_.28and_Y.E2.80.B2V12_or_YV12.29_to_RGB888_conversion
.. _Pillow library: https://pillow.readthedocs.io/


IO Classes
----------

The Python MMAL components include a couple of useful IO classes:
:class:`MMALSource` and :class:`MMALTarget`. We could have used these instead
of messing around with output callbacks in the sections above but it was worth
exploring how those callbacks operated first (in order to comprehend how
Python transforms operated).


Components
==========

.. autoclass:: MMALBaseComponent

.. autoclass:: MMALCamera
    :show-inheritance:

.. autoclass:: MMALCameraInfo
    :show-inheritance:

.. autoclass:: MMALComponent
    :show-inheritance:

.. autoclass:: MMALSplitter
    :show-inheritance:

.. autoclass:: MMALResizer
    :show-inheritance:

.. autoclass:: MMALISPResizer
    :show-inheritance:

.. autoclass:: MMALEncoder
    :show-inheritance:

.. autoclass:: MMALVideoEncoder
    :show-inheritance:

.. autoclass:: MMALImageEncoder
    :show-inheritance:

.. autoclass:: MMALDecoder
    :show-inheritance:

.. autoclass:: MMALVideoDecoder
    :show-inheritance:

.. autoclass:: MMALImageDecoder
    :show-inheritance:

.. autoclass:: MMALRenderer
    :show-inheritance:

.. autoclass:: MMALNullSink
    :show-inheritance:


Ports
=====

.. autoclass:: MMALControlPort

.. autoclass:: MMALPort
    :show-inheritance:

.. autoclass:: MMALVideoPort
    :show-inheritance:

.. autoclass:: MMALSubPicturePort
    :show-inheritance:

.. autoclass:: MMALAudioPort
    :show-inheritance:

.. autoclass:: MMALPortParams


Connections
===========

.. autoclass:: MMALBaseConnection

.. autoclass:: MMALConnection(source, target, formats=default_formats, callback=None)
    :show-inheritance:


Buffers
=======

.. autoclass:: MMALBuffer

.. autoclass:: MMALQueue

.. autoclass:: MMALPool

.. autoclass:: MMALPortPool
    :show-inheritance:


Python Extensions
=================

.. autoclass:: MMALPythonPort

.. autoclass:: MMALPythonBaseComponent
    :private-members:

.. autoclass:: MMALPythonComponent
    :show-inheritance:
    :private-members:

.. autoclass:: MMALPythonConnection(source, target, formats=default_formats, callback=None)
    :show-inheritance:

.. autoclass:: MMALPythonSource
    :show-inheritance:

.. autoclass:: MMALPythonTarget
    :show-inheritance:


Debugging
=========

The following functions are useful for quickly dumping the state of a given
MMAL pipeline:

.. autofunction:: debug_pipeline

.. autofunction:: print_pipeline

.. note::

    It is also worth noting that most classes, in particular
    :class:`MMALVideoPort` and :class:`MMALBuffer` have useful :func:`repr`
    outputs which can be extremely useful with simple :func:`print` calls for
    debugging.


Utility Functions
=================

The following functions are provided to ease certain common operations in the
picamera library. Users of ``mmalobj`` may find them handy in various
situations:

.. autofunction:: open_stream

.. autofunction:: close_stream

.. autofunction:: to_resolution

.. autofunction:: to_rational

.. autofunction:: buffer_bytes
