Add msgpack-jackson3 module by komamitsu · Pull Request #983 · msgpack/msgpack-java

komamitsu · 2026-05-16T16:11:30Z

Summary

Adds a new msgpack-jackson3 submodule (jackson-dataformat-msgpack3) that ports the existing Jackson 2.x integration to Jackson 3.x (tools.jackson namespace)
Requires Java 17+ and Jackson 3.1.2; built separately so it does not affect users on Java 8
Uses JUnit 4 via com.github.sbt:junit-interface — no extra sbt plugins needed

Key differences from `msgpack-jackson`

Package namespace: com.fasterxml.jackson → tools.jackson
TokenStreamLocation replaces JsonLocation; byte offset passed as columnNr (truncates for inputs > 2 GB, documented with comment)
JavaInfo / char-array optimization removed — always false on Java 17+ since String.value is byte[]
OSGi importPackage excludes android.os and sun.* matching msgpack-core
javacOptions set to --source 17 --target 17

Test plan

./sbt "msgpack-jackson3/test" — 77 tests pass
./sbt "msgpack-jackson3/test:compile" — no checkstyle errors
Existing msgpack-core and msgpack-jackson tests unaffected

Copilot

Pull request overview

This PR introduces a new msgpack-jackson3 submodule intended to provide MessagePack <-> Jackson integration for Jackson 3.x (tools.jackson.*) while keeping the existing Jackson 2.x module intact.

Changes:

Adds a new msgpack-jackson3 sbt subproject targeting Java 17 and Jackson tools.jackson.core:jackson-databind:3.1.2.
Ports the MessagePack TokenStreamFactory, JsonParser, JsonGenerator, and mapper/key-serialization glue to the Jackson 3 API surface.
Adds a full test suite (plus benchmark tests) for parser/generator/factory/mapper behavior under the new module.

Reviewed changes

Copilot reviewed 25 out of 26 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
build.sbt	Adds `msgpack-jackson3` subproject definition and aggregates it under root.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/ExtensionTypeCustomDeserializers.java	Extension-type custom deserializer registry for parser embedded objects.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/JsonArrayFormat.java	Jackson annotation introspector to serialize POJOs as arrays (schema-less).
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackExtensionType.java	POJO + serializer for MessagePack extension types.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackFactory.java	Jackson `TokenStreamFactory` implementation for MessagePack.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackGenerator.java	Jackson `JsonGenerator` implementation that writes MessagePack.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackKeySerializer.java	Key serializer that routes non-String map keys through MessagePack key handling.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackMapper.java	`ObjectMapper` wrapper + builder helpers for BigInteger/BigDecimal formatting.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackParser.java	Jackson `JsonParser` implementation that reads MessagePack.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackReadContext.java	Read context implementation for tracking nesting/name/dup detection.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackSerializedString.java	`SerializableString` wrapper used for MessagePack key serialization.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackSerializerFactory.java	Serializer factory override to enforce MessagePack key serialization.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/TimestampExtensionModule.java	Jackson module for MessagePack timestamp extension <-> `Instant`.
msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/Tuple.java	Simple tuple used for cached unpacker bookkeeping.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/ExampleOfTypeInformationSerDe.java	Example test for embedding type metadata for polymorphic objects.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackDataformatForFieldIdTest.java	Tests for integer-key support and mixed key handling.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackDataformatForPojoTest.java	POJO round-trip tests (normal, nested, schema-less array format, etc.).
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackDataformatTestBase.java	Shared test fixtures and helper POJOs.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackFactoryTest.java	Factory copy/creation tests.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackGeneratorTest.java	Generator behavioral tests (objects/arrays/primitives/keys/big numbers/etc.).
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackMapperTest.java	Mapper builder behavior tests for BigInteger/BigDecimal-as-string options.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/MessagePackParserTest.java	Parser behavioral tests for objects/arrays/primitives/extensions/errors.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/TimestampExtensionModuleTest.java	Tests for MessagePack timestamp extension serialization/deserialization.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/benchmark/Benchmarker.java	Benchmark harness used by benchmark tests.
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/benchmark/MessagePackDataformatHugeDataBenchmarkTest.java	Benchmark test for huge data (JSON vs MessagePack).
msgpack-jackson3/src/test/java/org/msgpack/jackson/dataformat/benchmark/MessagePackDataformatPojoBenchmarkTest.java	Benchmark test for POJO ser/de (JSON vs MessagePack).

Comments suppressed due to low confidence (2)

msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackGenerator.java:501

getBytesIfAscii writes into bytes[i] while iterating i from offset to offset+len. When offset != 0 this will write past the end of the newly allocated bytes array (length len) and can throw ArrayIndexOutOfBoundsException. Index into the destination array relative to offset (or use a separate destination index).

    private byte[] getBytesIfAscii(char[] chars, int offset, int len)
    {
        byte[] bytes = new byte[len];
        for (int i = offset; i < offset + len; i++) {
            char c = chars[i];
            if (c >= 0x80) {
                return null;
            }
            bytes[i] = (byte) c;
        }
        return bytes;

msgpack-jackson3/src/main/java/org/msgpack/jackson/dataformat/MessagePackGenerator.java:531

writeByteArrayTextValue ignores the provided offset/len when the bytes are ASCII: it wraps the entire text array in AsciiCharString. This can serialize extra bytes outside the intended slice. Consider copying the [offset, offset+len) range (or store the slice metadata) when creating AsciiCharString.

    private void writeByteArrayTextValue(byte[] text, int offset, int len) throws IOException
    {
        if (areAllAsciiBytes(text, offset, len)) {
            addValueNode(new AsciiCharString(text));
            return;
        }
        addValueNode(new String(text, offset, len, DEFAULT_CHARSET));
    }

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        else if (v instanceof ByteBuffer) {
+            ByteBuffer bb = (ByteBuffer) v;
+            int len = bb.remaining();
+            if (bb.hasArray()) {
+                messagePacker.packBinaryHeader(len);
+                messagePacker.writePayload(bb.array(), bb.arrayOffset(), len);
+            }
+            else {
+                byte[] data = new byte[len];
+                bb.get(data);
+                messagePacker.packBinaryHeader(len);
+                messagePacker.addPayload(data);
+            }


+    private String unpackString(MessageUnpacker messageUnpacker) throws IOException
+    {
+        int strLen = messageUnpacker.unpackRawStringHeader();
+        if (strLen <= tempBytes.length) {
+            messageUnpacker.readPayload(tempBytes, 0, strLen);
+            return new String(tempBytes, 0, strLen);
+        }
+        else {
+            byte[] bytes = messageUnpacker.readPayload(strLen);
+            return new String(bytes, 0, strLen, StandardCharsets.UTF_8);
+        }


+import com.fasterxml.jackson.annotation.JsonFormat;
+
+import tools.jackson.core.Version;
+import tools.jackson.databind.ObjectMapper;
+import tools.jackson.databind.cfg.MapperBuilder;
+import tools.jackson.databind.cfg.MapperBuilderState;
+


+
+import com.fasterxml.jackson.annotation.JsonFormat;
+
+import static com.fasterxml.jackson.annotation.JsonFormat.Shape.ARRAY;
+


+        return 0;
+    }
+
+    @Override
+    public int appendQuoted(char[] chars, int i)
+    {
+        return 0;
+    }
+
+    @Override
+    public int appendUnquotedUTF8(byte[] bytes, int i)
+    {
+        return 0;
+    }
+
+    @Override
+    public int appendUnquoted(char[] chars, int i)
+    {
+        return 0;
+    }
+
+    @Override
+    public int writeQuotedUTF8(OutputStream outputStream)
+    {
+        return 0;
+    }
+
+    @Override
+    public int writeUnquotedUTF8(OutputStream outputStream)
+    {
+        return 0;
+    }
+
+    @Override
+    public int putQuotedUTF8(ByteBuffer byteBuffer)
+    {
+        return 0;
+    }
+
+    @Override
+    public int putUnquotedUTF8(ByteBuffer byteBuffer)
+    {
+        return 0;


+                MessagePackExtensionType ext = p.readValueAs(MessagePackExtensionType.class);
+                if (ext.getType() != EXT_TYPE) {
+                    throw new RuntimeException(
+                            String.format("Unexpected extension type (0x%X) for Instant object", ext.getType()));


+import com.fasterxml.jackson.annotation.JsonIgnore;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
+import com.fasterxml.jackson.annotation.JsonProperty;


Adds a new msgpack-jackson3 module providing Jackson 3.x (tools.jackson.*) integration for MessagePack serialization, requiring Java 17+. New module structure: - MessagePackFactory / MessagePackMapper / MessagePackFactoryBuilder - MessagePackGenerator: deferred-flush design (buffers a node tree, emits MessagePack headers with correct sizes on flush/close) - MessagePackParser: streaming MessagePack deserialization - MessagePackSerializedString: SerializableString for non-string map keys - MessagePackExtensionType: extension type support - TimestampExtensionModule: Instant serialization via msgpack-core timestamps - JMH benchmarks (src/jmh/java) replacing hand-rolled System.nanoTime tests Key correctness fixes vs. initial draft: - Generator contract: streamWriteContext() returns live SimpleStreamWriteContext; addValueNode calls writeContext.writeValue(); writeName/writePropertyId check writeContext.writeName() return; writeStartArray/Object call _verifyValueWrite - writePropertyId (integer keys): calls writeContext.writeName() to keep Jackson context state in sync alongside the internal node state - writeString(Reader, int): chunked reading (8192-char buffer) for len>=0 path to avoid unbounded upfront allocation - MessagePackSerializedString: asQuotedUTF8 returns unquoted bytes ("quoted" means escaped, not surrounded by quotes); all append/write/put methods implemented correctly - TimestampExtensionModule: IOException wrapped as UncheckedIOException - MessagePackFactory.snapshot(): delegates to copy() instead of returning this - isClosed(): fixed by delegating close() lifecycle to super.close() - ByteBuffer serialization: uses bb.arrayOffset() + bb.position() to honour non-zero position - writeString(char[], offset, len) / writeUTF8String(byte[], offset, len): offset bugs fixed - Stale currentState after container close: reset to IN_ROOT in endCurrentContainer Build/CI: - msgpack-jackson3 excluded from root aggregate on Java < 17; CI matrix skips the module for older JDK entries - sbt-jmh plugin added for JMH benchmark support - OSGi manifest cleaned up; dead code removed

- MessagePackParser: make inner constructor package-private so MessagePackFactory can use it directly - MessagePackFactory._createParser(byte[],int,int): replace Arrays.copyOfRange with ArrayBufferInput(data, offset, len) to avoid an unnecessary copy when parsing a byte-array slice - MessagePackSerializedString: add capacity checks to all append/put methods (return -1 instead of throwing on overflow) and propagate IOException from writeQuotedUTF8/writeUnquotedUTF8 - TimestampExtensionModule.InstantDeserializer: replace bare RuntimeException with ctxt.reportInputMismatch for proper Jackson error reporting - plans/preexisting-issues.md: document node-buffering architectural constraint and nested-POJO generator inefficiency as known FYI items

…ublic

…ove unused byte[] constructor

komamitsu closed this May 16, 2026

komamitsu reopened this May 16, 2026

komamitsu requested a review from Copilot May 16, 2026 16:17

Copilot started reviewing on behalf of komamitsu May 16, 2026 16:17 View session

Copilot AI reviewed May 16, 2026

View reviewed changes

github-actions Bot added the doc label May 17, 2026

komamitsu marked this pull request as draft May 17, 2026 15:20

komamitsu force-pushed the support-jackson3 branch from 5e3ef27 to 504e02b Compare May 23, 2026 12:31

komamitsu added 2 commits May 24, 2026 00:08

Make MessagePackParser(MessageBufferInput, Object, ...) constructor p…

4164438

…ublic

komamitsu force-pushed the support-jackson3 branch from 1b49f14 to 4164438 Compare May 23, 2026 16:08

Revert MessagePackParser inner constructor to package-private and rem…

caad184

…ove unused byte[] constructor

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add msgpack-jackson3 module#983

Add msgpack-jackson3 module#983
komamitsu wants to merge 4 commits into
msgpack:mainfrom
komamitsu:support-jackson3

komamitsu commented May 16, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		import com.fasterxml.jackson.annotation.JsonFormat;

		import static com.fasterxml.jackson.annotation.JsonFormat.Shape.ARRAY;

Conversation

komamitsu commented May 16, 2026

Summary

Key differences from msgpack-jackson

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Key differences from `msgpack-jackson`