Seeking New Maintainer
For more details, see the discussion.
If no new maintainer is found, this project will be archived on October 22, 2023 (the 10 year anniversary of this project's first public commit).
Overview
This is a Gradle plugin to allow easily performing Java code generation for Apache Avro. It supports JSON schema declaration files, JSON protocol declaration files, and Avro IDL files.
Compatibility
NOTE: Pre-1.0 versions used a different publishing process/namespace. It is strongly recommended to upgrade to a newer version. Further details can be found in the change log.
- Currently tested against Java 8-18
- Though not supported yet, tests are also run against Java 19 to provide early notification of potential incompatibilities.
- Java 18 support requires Gradle 7.5 or higher (as per Gradle's release notes)
- Java 17 support requires Gradle 7.3 or higher (as per Gradle's release notes)
- Java 16 support requires Gradle 7.0 or higher (as per Gradle's release notes)
- Java 15 support requires Gradle 6.7 or higher (as per Gradle's release notes)
- Java 14 support requires Gradle 6.3 or higher (as per Gradle's release notes)
- Java 13 support requires Gradle 6.0 or higher
- Java 8-12 support requires Gradle 5.1 or higher (versions lower than 5.1 are no longer supported)
- Currently built against Gradle 7.5.1
- Currently tested against Gradle 5.1-5.6.4 and 6.0-7.5.1
- Currently built against Avro 1.11.1
- Currently tested against Avro 1.11.0-1.11.1
- Avro 1.9.0-1.10.2 were last supported in version 1.2.1
- Support for Kotlin
- Dropped integration with the Kotlin plugin in plugin version 1.4.0, as Kotlin 1.7.x would require compile-time dependency on a specific Kotlin version
- Wiring between the tasks added by the plugin and the Kotlin compilation tasks can either be added by your build logic, or a derived plugin
- Plugin version 1.3.0 was the last version with tested support for Kotlin
- It is believed to work with Kotlin 1.6.x as well
- It was tested against Kotlin plugin versions 1.3.20-1.3.72 and 1.4.0-1.4.32 and 1.5.0-1.5.31 using the latest compatible version of Gradle
- It was tested against Kotlin plugin versions 1.2.20-1.2.71 and 1.3.0-1.3.11 using Gradle 5.1
- Kotlin plugin versions 1.4.20-1.4.32 require special settings to work with Java 17+; see KT-43704
- Kotlin plugin version 1.3.30 is not compatible with Gradle 7.0+
- Kotlin plugin versions starting with 1.4.0 require Gradle 5.3+
- Kotlin plugin versions prior to 1.3.20 do not support Gradle 6.0+
- Kotlin plugin versions prior to 1.2.30 do not support Java 10+
- Version of the Kotlin plugin prior to 1.2.20 are unlikely to work
- Dropped integration with the Kotlin plugin in plugin version 1.4.0, as Kotlin 1.7.x would require compile-time dependency on a specific Kotlin version
- Support for Gradle Kotlin DSL
Usage
Add the following to your build files. Substitute the desired version based on CHANGES.md.
settings.gradle
:
pluginManagement {
repositories {
gradlePluginPortal()
mavenCentral()
}
}
build.gradle
:
plugins {
id "com.github.davidmc24.gradle.plugin.avro" version "VERSION"
}
Additionally, ensure that you have an implementation dependency on Avro, such as:
repositories {
mavenCentral()
}
dependencies {
implementation "org.apache.avro:avro:1.11.0"
}
If you now run gradle build
, Java classes will be compiled from Avro files in src/main/avro
.
Actually, it will attempt to process an "avro" directory in every SourceSet
(main, test, etc.)
Configuration
There are a number of configuration options supported in the avro
block.
option | default | description |
---|---|---|
createSetters | true |
createSetters passed to Avro compiler |
createOptionalGetters | false |
createOptionalGetters passed to Avro compiler |
gettersReturnOptional | false |
gettersReturnOptional passed to Avro compiler |
optionalGettersForNullableFieldsOnly | false |
optionalGettersForNullableFieldsOnly passed to Avro compiler |
fieldVisibility | "PRIVATE" |
fieldVisibility passed to Avro compiler |
outputCharacterEncoding | see below | outputCharacterEncoding passed to Avro compiler |
stringType | "String" |
stringType passed to Avro compiler |
templateDirectory | see below | templateDir passed to Avro compiler |
additionalVelocityToolClasses | see below | additionalVelocityTools passed to Avro compiler |
enableDecimalLogicalType | true |
enableDecimalLogicalType passed to Avro compiler |
Additionally, the avro
extension exposes the following methods:
logicalTypeFactory(String typeName, Class typeFactoryClass)
: register an additional logical type factorycustomConversion(Class conversionClass)
: register a custom conversion
createSetters
Valid values: true
(default), false
; supports equivalent String
values
Set to false
to not create setter methods in the generated classes.
Example:
avro {
createSetters = false
}
createOptionalGetters
Valid values: false
(default), true
; supports equivalent String
values
Set to true
to create additional getter methods that return their fields wrapped in an
Optional. For a field with
name abc
and type string
, this setting will create a method
Optional<java.lang.String> getOptionalAbc()
.
Example:
avro {
createOptionalGetters = false
}
gettersReturnOptional
Valid values: false
(default), true
; supports equivalent String
values
Set to true
to cause getter methods to return
Optional wrappers of the
underlying type. Where createOptionalGetters
generates an additional
method, this one replaces the existing getter.
Example:
avro {
gettersReturnOptional = false
}
optionalGettersForNullableFieldsOnly
Valid values: false
(default), true
; supports equivalent String
values
Set to true
in conjuction with gettersReturnOptional
to true
to return
Optional wrappers of the
underlying type. Where gettersReturnOptional
alone changes all getters to
return Optional
, this one only returns Optional for nullable (null union) field definitions.
Setting this to true
without setting gettersReturnOptional
to true
will result in this flag having no effect.
Example:
avro {
gettersReturnOptional = true
optionalGettersForNullableFieldsOnly = true
}
fieldVisibility
Valid values: any FieldVisibility or equivalent String
name (matched case-insensitively); default "PRIVATE"
(default)
By default, the fields in generated Java files will have private visibility.
Set to "PRIVATE"
to explicitly specify private visibility of the fields, or "PUBLIC"
to specify public visibility of the fields.
Example:
avro {
fieldVisibility = "PUBLIC"
}
outputCharacterEncoding
Valid values: any Charset or equivalent String
name
Controls the character encoding of generated Java files.
If using the plugin's conventions (i.e., not just the base plugin), the associated JavaCompile
task's encoding will be used automatically.
Otherwise, it will use the value configured in the avro
block, defaulting to "UTF-8"
.
Examples:
// Option 1: configure compilation task (avro plugin will automatically match)
tasks.withType(JavaCompile).configureEach {
options.encoding = 'UTF-8'
}
// Option 2: just configure avro plugin
avro {
outputCharacterEncoding = "UTF-8"
}
stringType
Valid values: any StringType or equivalent String
name (matched case-insensitively); default "String"
(default)
By default, the generated Java files will use java.lang.String
to represent string types.
Alternatively, you can set it to "Utf8"
to use org.apache.avro.util.Utf8
or "charSequence"
to use java.lang.CharSequence
.
avro {
stringType = "CharSequence"
}
templateDirectory
By default, files will be generated using Avro's default templates.
If desired, you can override the template set used by either setting this property or the "org.apache.avro.specific.templates"
System property.
avro {
templateDirectory = "/path/to/velocity/templates"
}
additionalVelocityToolClasses
When overriding the default set of Velocity templates provided with Avro, it is often desirable to provide additional tools to use during generation. The class names you provide will be made available for use in your Velocity templates. An instance of each class provided will be created using the default constructor (required). When registered, they will be available as $class.simpleName(). Given the example configuration below, two tools would be registered, and be available as escape and json.
avro {
additionalVelocityToolClasses = ['com.yourpackage.Escape', 'com.yourpackage.JSON']
}
enableDecimalLogicalType
Valid values: true
(default), false
; supports equivalent String
values
By default, generated Java files will use java.math.BigDecimal
for representing fixed
or bytes
fields annotated with "logicalType": "decimal"
.
Set to false
to use java.nio.ByteBuffer
in generated classes.
Example:
avro {
enableDecimalLogicalType = false
}
IntelliJ Integration
The plugin attempts to make IntelliJ play more smoothly with generated sources when using Gradle-generated project files.
However, there are still some rough edges. It will work best if you first run gradle build
, and after that run gradle idea
.
If you do it in the other order, IntelliJ may not properly exclude some directories within your build
directory.
Alternate Usage
If the defaults used by the plugin don't work for you, you can still use the tasks by themselves.
In this case, use the com.github.davidmc24.gradle.plugin.avro-base
plugin instead, and create tasks of type GenerateAvroJavaTask
and/or GenerateAvroProtocolTask
.
Here's a short example of what this might look like:
import com.github.davidmc24.gradle.plugin.avro.GenerateAvroJavaTask
apply plugin: "java"
apply plugin: "com.github.davidmc24.gradle.plugin.avro-base"
dependencies {
implementation "org.apache.avro:avro:1.11.0"
}
def generateAvro = tasks.register("generateAvro", GenerateAvroJavaTask) {
source("src/avro")
outputDir = file("dest/avro")
}
tasks.named("compileJava").configure {
source(generateAvro)
}
File Processing
When using this plugin, it is recommended to define each record/enum/fixed type in its own file rather than using inline type definitions.
This approach allows defining any type of schema structure, and eliminates the potential for conflicting definitions of a type between multiple files.
The plugin will automatically recognize the dependency and compile the files in the correct order.
For example, instead of Cat.avsc
:
{
"name": "Cat",
"namespace": "example",
"type": "record",
"fields" : [
{
"name": "breed",
"type": {
"name": "Breed",
"type": "enum",
"symbols" : [
"ABYSSINIAN", "AMERICAN_SHORTHAIR", "BIRMAN", "MAINE_COON", "ORIENTAL", "PERSIAN", "RAGDOLL", "SIAMESE", "SPHYNX"
]
}
}
]
}
use Breed.avsc
:
{
"name": "Breed",
"namespace": "example",
"type": "enum",
"symbols" : ["ABYSSINIAN", "AMERICAN_SHORTHAIR", "BIRMAN", "MAINE_COON", "ORIENTAL", "PERSIAN", "RAGDOLL", "SIAMESE", "SPHYNX"]
}
and Cat.avsc
:
{
"name": "Cat",
"namespace": "example",
"type": "record",
"fields" : [
{"name": "breed", "type": "Breed"}
]
}
There may be cases where the schema files contain inline type definitions and it is undesirable to modify them. In this case, the plugin will automatically recognize any duplicate type definitions and check if they match. If any conflicts are identified, it will cause a build failure.
Kotlin Support
The Java classes generated from your Avro files should be automatically accessible in the classpath to Kotlin classes in the same sourceset, and transitively to any sourcesets that depend on that sourceset. This is accomplished by this plugin detecting that the Kotlin plugin has been applied, and informing the Kotlin compilation tasks of the presence of the generated sources directories for cross-compilation.
This support does not support producing the Avro generated classes as Kotlin classes, as that functionality is not currently provided by the upstream Avro library.
Kotlin DSL Support
Special notes relevant to using this plugin via the Gradle Kotlin DSL:
- Apply the plugin declaratively using the
plugins {}
block. Otherwise, various features may not work as intended. See Configuring Plugins in the Gradle Kotlin DSL for more details. - Configuration in the
avro {}
block must be applied differently than in the Groovy DSL. See the example below for details.
Example Kotlin DSL Setup:
In gradle.build.kts
add:
plugins {
// Find latest release here: https://github.com/davidmc24/gradle-avro-plugin/releases
id("com.github.davidmc24.gradle.plugin.avro") version "VERSION"
}
And then in your settings.gradle.kts
add:
pluginManagement {
repositories {
gradlePluginPortal()
mavenCentral()
}
}
The syntax for configuring the extension looks like this:
avro {
isCreateSetters.set(true)
isCreateOptionalGetters.set(false)
isGettersReturnOptional.set(false)
isOptionalGettersForNullableFieldsOnly(false)
fieldVisibility.set("PUBLIC_DEPRECATED")
outputCharacterEncoding.set("UTF-8")
stringType.set("String")
templateDirectory.set(null as String?)
isEnableDecimalLogicalType.set(true)
}
Resolving schema dependencies
If desired, you can generate JSON schema with dependencies resolved.
Example build:
import com.github.davidmc24.gradle.plugin.avro.ResolveAvroDependenciesTask
apply plugin: "com.github.davidmc24.gradle.plugin.avro-base"
tasks.register("resolveAvroDependencies", ResolveAvroDependenciesTask) {
source file("src/avro/normalized")
outputDir = file("build/avro/resolved")
}
Generating schema files from protocol/IDL
If desired, you can generate JSON schema files.
To do this, apply the plugin (either avro
or avro-base
), and define custom tasks as needed for the schema generation.
From JSON protocol files, all that's needed is the GenerateAvroSchemaTask
.
From IDL files, first use GenerateAvroProtocolTask
to convert the IDL files to JSON protocol files, then use GenerateAvroSchemaTask
.
Example using base plugin with support for both IDL and JSON protocol files in src/main/avro
:
import com.github.davidmc24.gradle.plugin.avro.GenerateAvroProtocolTask
import com.github.davidmc24.gradle.plugin.avro.GenerateAvroSchemaTask
apply plugin: "com.github.davidmc24.gradle.plugin.avro-base"
def generateProtocol = tasks.register("generateProtocol", GenerateAvroProtocolTask) {
source file("src/main/avro")
include("**/*.avdl")
outputDir = file("build/generated-avro-main-avpr")
}
tasks.register("generateSchema", GenerateAvroSchemaTask) {
dependsOn generateProtocol
source file("src/main/avro")
source file("build/generated-avro-main-avpr")
include("**/*.avpr")
outputDir = file("build/generated-main-avro-avsc")
}