Consider reporting errors during encoding in ReaderInputStream
bjmi opened this issue · comments
API(s)
ReaderInputStream(reader, charset, bufferSize)How do you want it to be improved?
Use the default error action for malformed input and unmappable characters when creating the encoder by removing following lines:
.onMalformedInput(CodingErrorAction.REPLACE)
.onUnmappableCharacter(CodingErrorAction.REPLACE)
and update the corr. Javadoc.
Why do we need it to be improved?
Data is silently changed and conversion errors are just ignored.
This lead to corrupt data, it is particularly bad if it goes undetected for a long period of time.
Therefore the default behavior should make the user aware of the problem.
Newer introduced API (JDK 11) like java.nio.file.Files.readString(..) and java.nio.file.Files.writeString(..) also use the reporting error approach.
An alternative would be to make ReaderInputStream public and allow to pass a Reader and a CharsetEncoder.
The encoder can be created by the caller via charset.newEncoder() easily and configured according the intended use case.
This was requested in #5376.
Example
No code needs to be changed beforehand. It's just about reporting a problem.Current Behavior
Encoding errors are silently ignored and lead to corrupt text files.
Desired Behavior
Encoding errors should raise an exception and make errors visible, subsequently the code or data in question gets fixed.
Concrete Use Cases
We process texts from customers that use German and Cyrillic letters and it is crucial that the content remains intact when decoding / encoding is used.
Checklist
-
I agree to follow the code of conduct.
-
I have read and understood the contribution guidelines.
-
I have read and understood Guava's philosophy, and I strongly believe that this proposal aligns with it.
Further ideas
- Introduce
com.google.common.io.CharStreams.asInputStream(Reader, Charset): InputStreamandcom.google.common.io.CharStreams.asInputStream(Reader, CharsetEncoder): InputStream - Overload
com.google.common.io.CharSource.asByteSource(Supplier<CharsetEncoder>)
Passing a CharsetEncoder allows you to define the behavior on malformed input and on unmappable characters. The existing Charset is actually an encoder factory.
The relevant public API is CharSource.asByteSource(Charset).openStream(), right? I don't see ReaderInputStream being used from anywhere else.
You are right :)
I gave this a shot in #7218. I'd appreciate any input you might have. I only implemented the suggestions in #6944 (comment) . Please don't yell at me if I did something wrong because I only based my changes on the original comment and am quite new to this codebase. ;)
Should I try tackling the stuff mentioned in #6944 (comment) as well?
Should I try tackling the stuff mentioned in #6944 (comment) as well?
Thank you for your effort.
It would suffice to provide com.google.common.io.CharStreams.asInputStream(Reader, Charset): InputStream without changing CodingErrorActions of the encoder.