499 lines
17 KiB
ReStructuredText
499 lines
17 KiB
ReStructuredText
.. _module-pw_protobuf_compiler:
|
|
|
|
====================
|
|
pw_protobuf_compiler
|
|
====================
|
|
The Protobuf compiler module provides build system integration and wrapper
|
|
scripts for generating source code for Protobuf definitions.
|
|
|
|
--------------------
|
|
Protobuf compilation
|
|
--------------------
|
|
|
|
Generator support
|
|
=================
|
|
Protobuf code generation is currently supported for the following generators:
|
|
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Generator | Code | Notes |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| pw_protobuf | ``pwpb`` | Compiles using ``pw_protobuf``. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| pw_protobuf | ``pwpb_rpc`` | Compiles pw_rpc service and client code for |
|
|
| RPC | | ``pw_protobuf``. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Nanopb | ``nanopb`` | Compiles using Nanopb. The build argument |
|
|
| | | ``dir_pw_third_party_nanopb`` must be set to |
|
|
| | | point to a local nanopb installation. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Nanopb RPC | ``nanopb_rpc`` | Compiles pw_rpc service and client code for |
|
|
| | | nanopb. Requires a nanopb installation. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Raw RPC | ``raw_rpc`` | Compiles raw binary pw_rpc service code. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Go | ``go`` | Compiles using the standard Go protobuf |
|
|
| | | plugin with gRPC service support. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Python | ``python`` | Compiles using the standard Python protobuf |
|
|
| | | plugin, creating a ``pw_python_package``. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
| Typescript | ``typescript`` | Compilation is supported in Bazel via |
|
|
| | | @rules_proto_grpc. ProtoCollection provides |
|
|
| | | convience methods for proto descriptors. |
|
|
+-------------+----------------+-----------------------------------------------+
|
|
|
|
GN template
|
|
===========
|
|
This module provides a ``pw_proto_library`` GN template that defines a
|
|
collection of protobuf files that should be compiled together. The template
|
|
creates a sub-target for each supported generator, named
|
|
``<target_name>.<generator>``. These sub-targets generate their respective
|
|
protobuf code, and expose it to the build system appropriately (e.g. a
|
|
``pw_source_set`` for C/C++).
|
|
|
|
For example, given the following target:
|
|
|
|
.. code-block::
|
|
|
|
pw_proto_library("test_protos") {
|
|
sources = [ "my_test_protos/test.proto" ]
|
|
}
|
|
|
|
``test_protos.pwpb`` compiles code for pw_protobuf, and ``test_protos.nanopb``
|
|
compiles using Nanopb (if it's installed).
|
|
|
|
Protobuf code is only generated when a generator sub-target is listed as a
|
|
dependency of another GN target.
|
|
|
|
GN permits using abbreviated labels when the target name matches the directory
|
|
name (e.g. ``//foo`` for ``//foo:foo``). For consistency with this, the
|
|
sub-targets for each generator are aliased to the directory when the target name
|
|
is the same. For example, these two labels are equivalent:
|
|
|
|
.. code-block::
|
|
|
|
//path/to/my_protos:my_protos.pwpb
|
|
//path/to/my_protos:pwpb
|
|
|
|
``pw_python_package`` subtargets are also available on the ``python`` subtarget:
|
|
|
|
.. code-block::
|
|
|
|
//path/to/my_protos:my_protos.python.lint
|
|
//path/to/my_protos:python.lint
|
|
|
|
**Supported Codegen**
|
|
|
|
GN supports the following compiled proto libraries via the specified
|
|
sub-targets generated by a ``pw_proto_library``.
|
|
|
|
* ``${target_name}.pwpb`` - Generated C++ pw_protobuf code
|
|
* ``${target_name}.pwpb_rpc`` - Generated C++ pw_protobuf pw_rpc code
|
|
* ``${target_name}.nanopb`` - Generated C++ nanopb code (requires Nanopb)
|
|
* ``${target_name}.nanopb_rpc`` - Generated C++ Nanopb pw_rpc code (requires
|
|
Nanopb)
|
|
* ``${target_name}.raw_rpc`` - Generated C++ raw pw_rpc code (no protobuf
|
|
library)
|
|
* ``${target_name}.go`` - Generated GO protobuf libraries
|
|
* ``${target_name}.python`` - Generated Python protobuf libraries
|
|
|
|
**Arguments**
|
|
|
|
* ``sources``: List of input .proto files.
|
|
* ``deps``: List of other pw_proto_library dependencies.
|
|
* ``other_deps``: List of other non-proto dependencies.
|
|
* ``inputs``: Other files on which the protos depend (e.g. nanopb ``.options``
|
|
files).
|
|
* ``prefix``: A prefix to add to the source protos prior to compilation. For
|
|
example, a source called ``"foo.proto"`` with ``prefix = "nested"`` will be
|
|
compiled with protoc as ``"nested/foo.proto"``.
|
|
* ``strip_prefix``: Remove this prefix from the source protos. All source and
|
|
input files must be nested under this path.
|
|
* ``python_package``: Label of Python package to which to add the proto modules.
|
|
The .python subtarget will redirect to this package.
|
|
* ``enabled_targets``: List of sub-targets to enable (see Supported Codegen),
|
|
e.g. ``["pwpb", "raw_rpc"]``. By default, all sub-targets are enabled. The
|
|
enabled sub-targets are built only as requested by the build system, but it
|
|
may be necessary to explicitly disable an unused sub-target if it conflicts
|
|
with another target in the same package. (For example, ``nanopb`` codegen can
|
|
conflict with the default C++ codegen provided by ``protoc``.)
|
|
TODO(b/235132083): Remove this argument once we've removed the file-name
|
|
conflict between nanopb and protoc code generators.
|
|
|
|
**Example**
|
|
|
|
.. code-block::
|
|
|
|
import("$dir_pw_protobuf_compiler/proto.gni")
|
|
|
|
pw_proto_library("my_protos") {
|
|
sources = [
|
|
"my_protos/foo.proto",
|
|
"my_protos/bar.proto",
|
|
]
|
|
}
|
|
|
|
pw_proto_library("my_other_protos") {
|
|
sources = [ "some/other/path/baz.proto" ] # imports foo.proto
|
|
|
|
# This removes the "some/other/path" prefix from the proto files.
|
|
strip_prefix = "some/other/path"
|
|
|
|
# This adds the "my_other_protos/" prefix to the proto files.
|
|
prefix = "my_other_protos"
|
|
|
|
# Proto libraries depend on other proto libraries directly.
|
|
deps = [ ":my_protos" ]
|
|
}
|
|
|
|
source_set("my_cc_code") {
|
|
sources = [
|
|
"foo.cc",
|
|
"bar.cc",
|
|
"baz.cc",
|
|
]
|
|
|
|
# When depending on protos in a source_set, specify the generator suffix.
|
|
deps = [ ":my_other_protos.pwpb" ]
|
|
}
|
|
|
|
From C++, ``baz.proto`` included as follows:
|
|
|
|
.. code-block:: cpp
|
|
|
|
#include "my_other_protos/baz.pwpb.h"
|
|
|
|
From Python, ``baz.proto`` is imported as follows:
|
|
|
|
.. code-block:: python
|
|
|
|
from my_other_protos import baz_pb2
|
|
|
|
Proto file structure
|
|
--------------------
|
|
Protobuf source files must be nested under another directory when they are
|
|
compiled. This ensures that they can be packaged properly in Python.
|
|
|
|
Using ``prefix`` and ``strip_prefix`` together allows remapping proto files to
|
|
a completely different path. This can be useful when working with protos defined
|
|
in external libraries. For example, consider this proto library:
|
|
|
|
.. code-block::
|
|
|
|
pw_proto_library("external_protos") {
|
|
sources = [
|
|
"//other/external/some_library/src/protos/alpha.proto",
|
|
"//other/external/some_library/src/protos/beta.proto,
|
|
"//other/external/some_library/src/protos/internal/gamma.proto",
|
|
]
|
|
strip_prefix = "//other/external/some_library/src/protos"
|
|
prefix = "some_library"
|
|
}
|
|
|
|
These protos will be compiled by protoc as if they were in this file structure:
|
|
|
|
.. code-block::
|
|
|
|
some_library/
|
|
├── alpha.proto
|
|
├── beta.proto
|
|
└── internal
|
|
└── gamma.proto
|
|
|
|
.. _module-pw_protobuf_compiler-add-to-python-package:
|
|
|
|
Adding Python proto modules to an existing package
|
|
--------------------------------------------------
|
|
By default, generated Python proto modules are organized into their own Python
|
|
package. These proto modules can instead be added to an existing Python package
|
|
declared with ``pw_python_package``. This is done by setting the
|
|
``python_package`` argument on the ``pw_proto_library`` and the
|
|
``proto_library`` argument on the ``pw_python_package``.
|
|
|
|
For example, the protos declared in ``my_protos`` will be nested in the Python
|
|
package declared by ``my_package``.
|
|
|
|
.. code-block::
|
|
|
|
pw_proto_library("my_protos") {
|
|
sources = [ "hello.proto ]
|
|
prefix = "foo"
|
|
python_package = ":my_package"
|
|
}
|
|
|
|
pw_python_pacakge("my_package") {
|
|
generate_setup = {
|
|
metadata = {
|
|
name = "foo"
|
|
version = "1.0"
|
|
}
|
|
}
|
|
|
|
sources = [ "foo/cool_module.py" ]
|
|
proto_library = ":my_protos"
|
|
}
|
|
|
|
The ``hello_pb2.py`` proto module can be used alongside other files in the
|
|
``foo`` package.
|
|
|
|
.. code-block:: python
|
|
|
|
from foo import cool_module, hello_pb2
|
|
|
|
Working with externally defined protos
|
|
--------------------------------------
|
|
``pw_proto_library`` targets may be used to build ``.proto`` sources from
|
|
existing projects. In these cases, it may be necessary to supply the
|
|
``strip_prefix`` argument, which specifies the protobuf include path to use for
|
|
``protoc``. If only a single external protobuf is being compiled, the
|
|
``python_module_as_package`` option can be used to override the requirement that
|
|
the protobuf be nested under a directory. This option generates a Python package
|
|
with the same name as the proto file, so that the generated proto can be
|
|
imported as if it were a standalone Python module.
|
|
|
|
For example, the ``pw_proto_library`` target for Nanopb sets
|
|
``python_module_as_package`` to ``nanopb_pb2``.
|
|
|
|
.. code-block::
|
|
|
|
pw_proto_library("proto") {
|
|
strip_prefix = "$dir_pw_third_party_nanopb/generator/proto"
|
|
sources = [ "$dir_pw_third_party_nanopb/generator/proto/nanopb.proto" ]
|
|
python_module_as_package = "nanopb_pb2"
|
|
}
|
|
|
|
In Python, this makes ``nanopb.proto`` available as ``import nanopb_pb2`` via
|
|
the ``nanopb_pb2`` Python package. In C++, ``nanopb.proto`` is accessed as
|
|
``#include "nanopb.pwpb.h"``.
|
|
|
|
The ``python_module_as_package`` feature should only be used when absolutely
|
|
necessary --- for example, to support proto files that include
|
|
``import "nanopb.proto"``.
|
|
|
|
CMake
|
|
=====
|
|
CMake provides a ``pw_proto_library`` function with similar features as the
|
|
GN template. The CMake build only supports building firmware code, so
|
|
``pw_proto_library`` does not generate a Python package.
|
|
|
|
**Arguments**
|
|
|
|
* ``NAME``: the base name of the libraries to create
|
|
* ``SOURCES``: .proto source files
|
|
* ``DEPS``: dependencies on other ``pw_proto_library`` targets
|
|
* ``PREFIX``: prefix add to the proto files
|
|
* ``STRIP_PREFIX``: prefix to remove from the proto files
|
|
* ``INPUTS``: files to include along with the .proto files (such as Nanopb
|
|
.options files)
|
|
|
|
**Example**
|
|
|
|
.. code-block:: cmake
|
|
|
|
include($ENV{PW_ROOT}/pw_build/pigweed.cmake)
|
|
include($ENV{PW_ROOT}/pw_protobuf_compiler/proto.cmake)
|
|
|
|
pw_proto_library(my_module.my_protos
|
|
SOURCES
|
|
my_protos/foo.proto
|
|
my_protos/bar.proto
|
|
)
|
|
|
|
pw_proto_library(my_module.my_protos
|
|
SOURCES
|
|
my_protos/foo.proto
|
|
my_protos/bar.proto
|
|
)
|
|
|
|
pw_proto_library(my_module.my_other_protos
|
|
SOURCES
|
|
some/other/path/baz.proto # imports foo.proto
|
|
|
|
# This removes the "some/other/path" prefix from the proto files.
|
|
STRIP_PREFIX
|
|
some/other/path
|
|
|
|
# This adds the "my_other_protos/" prefix to the proto files.
|
|
PREFIX
|
|
my_other_protos
|
|
|
|
# Proto libraries depend on other proto libraries directly.
|
|
DEPS
|
|
my_module.my_protos
|
|
)
|
|
|
|
add_library(my_module.my_cc_code
|
|
foo.cc
|
|
bar.cc
|
|
baz.cc
|
|
)
|
|
|
|
# When depending on protos in a source_set, specify the generator suffix.
|
|
target_link_libraries(my_module.my_cc_code PUBLIC
|
|
my_module.my_other_protos.pwpb
|
|
)
|
|
|
|
These proto files are accessed in C++ the same as in the GN build:
|
|
|
|
.. code-block:: cpp
|
|
|
|
#include "my_other_protos/baz.pwpb.h"
|
|
|
|
**Supported Codegen**
|
|
|
|
CMake supports the following compiled proto libraries via the specified
|
|
sub-targets generated by a ``pw_proto_library``.
|
|
|
|
* ``${NAME}.pwpb`` - Generated C++ pw_protobuf code
|
|
* ``${NAME}.pwpb_rpc`` - Generated C++ pw_protobuf pw_rpc code
|
|
* ``${NAME}.nanopb`` - Generated C++ nanopb code (requires Nanopb)
|
|
* ``${NAME}.nanopb_rpc`` - Generated C++ Nanopb pw_rpc code (requires Nanopb)
|
|
* ``${NAME}.raw_rpc`` - Generated C++ raw pw_rpc code (no protobuf library)
|
|
|
|
Bazel
|
|
=====
|
|
In Bazel we provide a set rules with similar features to the GN templates:
|
|
|
|
* ``pwpb_proto_library`` - Generated C++ pw_protobuf code
|
|
* ``pwpb_rpc_proto_library`` - Generated C++ pw_protobuf pw_rpc code
|
|
* ``raw_rpc_proto_library`` - Generated C++ raw pw_rpc code (no protobuf library)
|
|
* ``nanopb_proto_library`` - Generated C++ nanopb code
|
|
* ``nanopb_rpc_proto_library`` - Generated C++ Nanopb pw_rpc code
|
|
|
|
These rules build the corresponding firmware code; there are no rules for
|
|
generating Python libraries. The Bazel rules differ slightly compared to the GN
|
|
build to be more in line with what would be considered idiomatic in Bazel.
|
|
|
|
To use Pigweeds Protobuf rules you must first pull in the required dependencies
|
|
into your Bazel WORKSPACE file. e.g.
|
|
|
|
.. code-block:: python
|
|
|
|
# WORKSPACE ...
|
|
load("@pigweed//pw_protobuf_compiler:deps.bzl", "pw_protobuf_dependencies")
|
|
pw_protobuf_dependencies()
|
|
|
|
Bazel uses a different set of rules to manage proto files than it does to
|
|
compile them. e.g.
|
|
|
|
.. code-block:: python
|
|
|
|
# BUILD ...
|
|
load("@rules_proto//proto:defs.bzl", "proto_library")
|
|
load("@pigweed//pw_protobuf_compiler:proto.bzl",
|
|
"nanopb_proto_library",
|
|
"nanopb_rpc_proto_library",
|
|
"pwpb_proto_library",
|
|
"raw_rpc_proto_library",
|
|
)
|
|
|
|
# Manages proto sources and dependencies.
|
|
proto_library(
|
|
name = "my_proto",
|
|
srcs = [
|
|
"my_protos/foo.proto",
|
|
"my_protos/bar.proto",
|
|
]
|
|
)
|
|
|
|
# Compiles dependent protos to C++.
|
|
pwpb_proto_library(
|
|
name = "my_proto_pwpb",
|
|
deps = [":my_proto"],
|
|
)
|
|
|
|
nanopb_proto_library(
|
|
name = "my_proto_nanopb",
|
|
deps = [":my_proto"],
|
|
)
|
|
|
|
raw_rpc_proto_library(
|
|
name = "my_proto_raw_rpc",
|
|
deps = [":my_proto"],
|
|
)
|
|
|
|
nanopb_rpc_proto_library(
|
|
name = "my_proto_nanopb_rpc",
|
|
nanopb_proto_library_deps = [":my_proto_nanopb"],
|
|
deps = [":my_proto"],
|
|
)
|
|
|
|
# Library that depends on only pw_protobuf generated proto targets.
|
|
pw_cc_library(
|
|
name = "my_proto_only_lib",
|
|
srcs = ["my/proto_only.cc"],
|
|
deps = [":my_proto_pwpb"],
|
|
)
|
|
|
|
# Library that depends on only Nanopb generated proto targets.
|
|
pw_cc_library(
|
|
name = "my_nanopb_only_lib",
|
|
srcs = ["my/nanopb_only.cc"],
|
|
deps = [":my_proto_nanopb"],
|
|
)
|
|
|
|
# Library that depends on pw_protobuf and pw_rpc/raw.
|
|
pw_cc_library(
|
|
name = "my_raw_rpc_lib",
|
|
srcs = ["my/raw_rpc.cc"],
|
|
deps = [
|
|
":my_proto_pwpb",
|
|
":my_proto_raw_rpc",
|
|
],
|
|
)
|
|
pw_cc_library(
|
|
name = "my_nanopb_rpc_lib",
|
|
srcs = ["my/proto_only.cc"],
|
|
deps = [
|
|
":my_proto_nanopb_rpc",
|
|
],
|
|
)
|
|
|
|
|
|
From ``my/lib.cc`` you can now include the generated headers.
|
|
e.g.
|
|
|
|
.. code:: cpp
|
|
|
|
#include "my_protos/bar.pwpb.h"
|
|
// and/or RPC headers
|
|
#include "my_protos/bar.raw_rpc.pb.h
|
|
// or
|
|
#include "my_protos/bar.nanopb_rpc.pb.h"
|
|
|
|
|
|
Why isn't there one rule to generate all the code?
|
|
--------------------------------------------------
|
|
There is! Like in GN, it's called ``pw_proto_library``, and has subtargets
|
|
corresponding to the different codegen flavors. However, *we recommend against
|
|
using this target*. It is deprecated, and will be removed in the future.
|
|
|
|
The ``pw_proto_library`` target has a number of disadvantages:
|
|
|
|
#. As a general bazel style rule, macros should produce exactly one target for
|
|
external use, named according to the invocation's name argument. ``BUILD``
|
|
files are easier to follow when the name specified in the macro call
|
|
actually matches the name of the generated target. This is not possible if a
|
|
single macro is generating multiple targets, as ``pw_proto_library`` does.
|
|
#. If you depend directly on the ``pw_proto_library``, rather than the
|
|
appropriate subtargets, you will build code you don't actually use. You may
|
|
even fetch dependencies you don't need, like nanopb.
|
|
#. The subtargets you don't depend on are still added to your BUILD files by
|
|
the ``pw_proto_library`` macro, and bazel will attempt to build them when
|
|
you run ``bazel build //...``. This may cause build breakages, and has
|
|
forced us to implement `awkward workarounds
|
|
<https://pigweed-review.git.corp.google.com/c/pigweed/pigweed/+/96980>`_.
|
|
|
|
----------------------
|
|
Python proto libraries
|
|
----------------------
|
|
``pw_protobuf_compiler`` includes utilties for working with protocol buffers
|
|
in Python. The tools facilitate using protos from their package names
|
|
(``my.pkg.Message()``) rather than their generated module names
|
|
(``proto_source_file_pb2.Message()``).
|
|
|
|
``python_protos`` module
|
|
========================
|
|
.. automodule:: pw_protobuf_compiler.python_protos
|
|
:members: proto_repr, Library
|