Running 09-demo-aggregate.yaml using script results in Linux ownership issues

Question

Running 09-demo-aggregate.yaml using script results in Linux ownership issues

tonvanbart opened this issue 2 years ago · comments

Ton van Bart commented 2 years ago

Found this while trying to reproduce #35 on my private (Linux) laptop - fails with yet another exception:

Platform: Ubuntu 20.04 LTS
Steps

build ksml runner image: docker build -t axual/ksml -f Dockerfile-build-runner . --build-arg runner=ksml-runner
start local environment: docker-compose up -d
modify examples/ksml-runner.yml so that it points to 09-demo-aggregate.yaml
start the local run: ./examples/run.sh

Expected result: same exception as #35
Actual result: the runner fails with the following stacktrace:

2022-04-22T20:55:17,973Z [system] [main] INFO  i.a.k.g.TopologyGeneratorImpl -
                  owner_count: retention=PT3M, key=BINARY:String, value=BINARY:Long, caching=false
Exception in thread "main" org.apache.kafka.streams.errors.StreamsException: org.apache.kafka.streams.errors.ProcessorStateException: state directory [/ksml/io.ksml.example.processor] doesn't exist and couldn't be created
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:807)
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:782)
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:692)
	at io.axual.ksml.runner.backend.kafka.KafkaBackend.<init>(KafkaBackend.java:82)
	at io.axual.ksml.runner.backend.kafka.KafkaBackendProvider.create(KafkaBackendProvider.java:44)
	at io.axual.ksml.runner.config.KSMLRunnerConfig.getConfiguredBackend(KSMLRunnerConfig.java:80)
	at io.axual.ksml.runner.KSMLRunner.main(KSMLRunner.java:57)
Caused by: org.apache.kafka.streams.errors.ProcessorStateException: state directory [/ksml/io.ksml.example.processor] doesn't exist and couldn't be created
	at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:134)
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:804)
	... 6 more
.../ksml on  main [$!?]

Ton van Bart · Answer 1 · Sat Apr 23 2022 05:17:27 GMT+0800 (China Standard Time)

Tried to work around by trying to create this directory by hand in the examples/ subdir, gave it chmod 777 for good measure.
Result: I do get the expected exception now, but also some nio exceptions before that:

2022-04-22T21:20:25,221Z [system] [main] ERROR o.a.k.s.p.internals.StateDirectory -
                Error changing permissions for the directory /ksml 
java.nio.file.FileSystemException: /ksml: Operation not permitted
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setMode(UnixFileAttributeViews.java:254)
	at java.base/sun.nio.fs.UnixFileAttributeViews$Posix.setPermissions(UnixFileAttributeViews.java:276)
	at java.base/java.nio.file.Files.setPosixFilePermissions(Files.java:2080)
	at org.apache.kafka.streams.processor.internals.StateDirectory.configurePermissions(StateDirectory.java:154)
	at org.apache.kafka.streams.processor.internals.StateDirectory.<init>(StateDirectory.java:144)
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:804)
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:782)
	at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:692)
	at io.axual.ksml.runner.backend.kafka.KafkaBackend.<init>(KafkaBackend.java:82)
	at io.axual.ksml.runner.backend.kafka.KafkaBackendProvider.create(KafkaBackendProvider.java:44)
	at io.axual.ksml.runner.config.KSMLRunnerConfig.getConfiguredBackend(KSMLRunnerConfig.java:80)
	at io.axual.ksml.runner.KSMLRunner.main(KSMLRunner.java:57)

Maybe there should be some additional instructions?

Jeroen van Disseldorp · Answer 2 · Tue Apr 26 2022 03:09:06 GMT+0800 (China Standard Time)

You should probably point to a user directory somewhere, potentially relative to the working directory. If you leave out the initial '/' then Kafka Streams creates a cache directory underneath the directory where you run.

Ton van Bart · Answer 3 · Tue Apr 26 2022 21:40:43 GMT+0800 (China Standard Time)

I tried changing the value from /ksml to ksml in ksml-runner.yml (so, leaving out the initial /, however this does not work:

Exception in thread "main" io.axual.ksml.runner.exception.KSMLRunnerConfigurationException: The provided path does not exists or is not a directory
Configuration Key   : 'workingDirectory'
Configuration Value : 'ksml' 
        at io.axual.ksml.runner.config.KSMLConfig.validate(KSMLConfig.java:75)
        at io.axual.ksml.runner.config.KSMLRunnerConfig.validate(KSMLRunnerConfig.java:51)
        at io.axual.ksml.runner.KSMLRunner.main(KSMLRunner.java:55)

Is that the correct property to change?

Ton van Bart · Answer 4 · Wed Apr 27 2022 03:39:40 GMT+0800 (China Standard Time)

Looking in run.sh I see that the runner image is started like this;

docker run --name ksml-example --rm -ti -v "${BASEDIR}":/ksml -w /ksml --network ksml_example axual/ksml:latest

So it resolved the directory where run.sh resides as $BASEDIR and mounts this as /ksml in the container and sets it as the working directory. It's expectingksml-runner.yml in that location, and the runner config points to /ksml as well.
The process inside the image is running as USER 1024 but this is not present on the host; this is the cause of the ownership issue.

Tried to make a workaround with the following steps:

create a subdirectory work/ inside examples/
change the ownership of that chown 1024 work
copy the definition to run (in this case 09-demo-aggregate.yaml into work/ OR change the definitions: entry in ksml-runner.yml to point to ../09-demo-aggregate.yaml (so one directory up from work/)
change workingDirectory in ksml-runner.yml to point to /ksml/work so the process gets a directory that it owns

This seems to avoid the ownership issue, however still no joy:

2022-04-26T19:50:24,257Z [system] [main] INFO  i.a.k.g.TopologyGeneratorImpl -
                Reading KSML from source file(s): [../09-demo-aggregate.yaml]
2022-04-26T19:50:24,289Z [system] [main] WARN  io.axual.ksml.avro.AvroSchemaLoader -
                Could not find/load Avro schema: SensorData
2022-04-26T19:50:24,289Z [system] [main] WARN  io.axual.ksml.avro.AvroSchemaLoader -
                Could not find/load Avro schema: SensorData
Exception in thread "main" io.axual.ksml.exception.KSMLParseException: Could not load Avro schema: SensorData
	at io.axual.ksml.avro.AvroSchemaLoader.load(AvroSchemaLoader.java:54)
	at io.axual.ksml.schema.SchemaLibrary.getSchema(SchemaLibrary.java:46)
	at io.axual.ksml.parser.UserTypeParser.parseAvroType(UserTypeParser.java:137)
	at io.axual.ksml.parser.UserTypeParser.parseTypeAndNotation(UserTypeParser.java:121)
	at io.axual.ksml.parser.UserTypeParser.parseListOfTypesAndNotation(UserTypeParser.java:77)
(...continues...)

After this copied SensorData.avsc into work/ and restarted the script --> ownership issue is solved and we're back at issue #37 .