Understanding protocol buffer version compatibility issues in Python

‹ Testing Ansible roles with test-kitchen and Docker using the Ansible Role Tester | Generating grpc stubs for Python from the command line ›

(Updated December 12, 2017 for Python 3)

There are multiple protocol buffer and protobuf-related software packages available in multiple versions, as well as two major protocol buffer syntax releases. Forward compatibility does not exist, so the error messages that appear require research to understand:

Conclusions

  • proto2 files should always declare syntax = "proto2" instead of leaving it implicit. protoc>=2.5.0 generates identical output with and without this declaration and protoc>=3.0.0 issues warnings if it is missing.
  • Use the latest protobuf Python package.
    • They are backward compatible with older versions of _pb2.py files.
    • GRPC.io (if you want to use it) also requires protobuf>=3.0.0
    • Python 3 also requires protobuf>=3.0.0

.proto file versions

There are three major ways to declare the version of a .proto file

1. No declaration

service MyService {
  rpc DoStuff(MyRequest) returns (MyResponse);
}
message MyRequest {
  required int32 myfield = 1;
}
message MyResponse {
  required int32 myresult = 1;
}

2. Explicit proto2 declaration

syntax = "proto2";
service MyService2 {
  rpc DoStuff(MyRequest2) returns (MyResponse2);
}
message MyRequest2 {
  required int32 myfield = 1;
}
message MyResponse2 {
  required int32 myresult = 1;
}

3. Explicit proto3 declaration

syntax = "proto3";
service MyService3 {
  rpc DoStuff(MyRequest3) returns (MyResponse3);
}
message MyRequest3 {
  int32 myfield = 1;
}
message MyResponse3 {
  int32 myresult = 1;
}

protoc versions

The protoc compiler is a native application written in C++

Every current combination of compiler and .proto file syntax will generate python output named *_pb2.py even when using proto3 syntax.


libprotoc 2.5.0 libprotoc 2.6.1 libprotoc 3.0.0
No declaration Identical output to explicit proto2 Identical output to explicit proto2
Identical output to explicit proto2. Also produces this warning:
[libprotobuf WARNING google/protobuf/compiler/parser.cc:547] No syntax specified for the proto file: nosyntax.proto. Please use 'syntax = "proto2";' or 'syntax = "proto3";' to specify a syntax version. (Defaulted to proto2 syntax.)
Explicit proto2 DESCRIPTOR = _descriptor.FileDescriptor(
name='proto2.proto',
package='',
serialized_pb='...
...
class MyRequest2(_message.Message):
...
DESCRIPTOR = _descriptor.FileDescriptor(
name='proto2.proto',
package='',
serialized_pb=_b('...
...
MyRequest2 = _reflection.GeneratedProtocolMessageType(...
...
DESCRIPTOR = _descriptor.FileDescriptor(
name='proto2.proto',
package='',
syntax='proto2',
serialized_pb=_b(...
...
MyRequest2 = _reflection.GeneratedProtocolMessageType(...
...
Explicit proto3 Unrecognized syntax identifier "proto3".  This parser only recognizes "proto2". Unrecognized syntax identifier "proto3".  This parser only recognizes "proto2". DESCRIPTOR = _descriptor.FileDescriptor(
name='proto3.proto',
package='',
syntax='proto3',
serialized_pb=_b(...
...
MyRequest3 = _reflection.GeneratedProtocolMessageType(...
...

protobuf python package versions

The python protobuf package available on PyPI is necessary to import the generated _pb2.py files. New python packages have been kept backward compatible with older _pb2.py files

Python 2


protobuf==2.5.0 protobuf==2.6.1 protobuf==3.0.0
protoc 2.5.0
protoc 2.6.1
from google.protobuf import symbol_database as _symbol_database
ImportError: cannot import name symbol_database
protoc 3.0.0
from google.protobuf import symbol_database as _symbol_database
ImportError: cannot import name symbol_database
TypeError: __init__() got an unexpected keyword argument 'syntax'

Python 3


protobuf==2.5.0 protobuf==2.6.1 protobuf==3.0.0
protoc 2.5.0
n/a, python package installation fails

SyntaxError: Missing parentheses in call to 'print'
File ".../google/protobuf/internal/python_message.py", line 848
except struct.error, e:
                   ^
SyntaxError: invalid syntax
TypeError: expected bytes, str found
protoc 2.6.1
n/a, python package installation fails

SyntaxError: Missing parentheses in call to 'print'
File ".../google/protobuf/internal/python_message.py", line 848
except struct.error, e:
                   ^
SyntaxError: invalid syntax
protoc 3.0.0
n/a, python package installation fails

SyntaxError: Missing parentheses in call to 'print'
File ".../google/protobuf/internal/python_message.py", line 848
except struct.error, e:
                   ^
SyntaxError: invalid syntax

Links

Source code is available at https://github.com/wtanaka/proto-versions

Comments

# ./protobuf-2.5.0/src/protoc --version

libprotoc 2.5.0

# pip show python3-protobuf

Name: python3-protobuf

Version: 2.5.0

# ./protobuf-2.5.0/src/protoc --python_out=. meta.proto

# python meta_pb2.py 

Traceback (most recent call last):

  File "meta_pb2.py", line 318, in <module>

    has_default_value=False, default_value=unicode("", "utf-8"),

 

NameError: name 'unicode' is not defined

Subscribe to All Posts - Wesley Tanaka