Error exporting models using csv_custom file format

[Domingos Freitas]

Hello.

After upgrading our site from v2 to v3.6 I noticed a problem when exporting to csv_custom file format (separator is “;”).

The error ("The stream filter API can not be used with a SplTempFileObject instance." on line 35 of D:\code\sitev3\vendor\league\csv\src\UnavailableFeature.php) happens in “modules\backend\models\exportmodel\EncodesCsv.php” at line 96 because the custom export format form always send a “encoding” option.

I was able to "correct the error”, since I am using UTF-8, by changing the line 96 from:

if ($options['encoding'] !== null) {

to:

if ($options['encoding'] !== null && $csv->supportsStreamFilterOnWrite()) {

The export work flawlessly if I use the default csv export format, but I am unable to open the csv file directly in Excel because it uses “,” as separator instead of “;”.

Of course this is not a permanent solution since any October update could rollback the changes.

With my fix, we are unable to change the output file encoding, so it is a not optimal solution. It work for me since I do not need to change the encoding from UTF-8.

Thank you in advance.

With best regards,
Domingos

[Marco Grüter]

I can confirm the issue, encountered this one myself today and didn't find a good solution yet either.

I tried using the formatter like mentioned in the phpleague docs:

        if ($options['encoding'] !== null) {
            $csv->addFormatter((new CharsetConverter())
                ->inputEncoding('utf-8')
                ->outputEncoding($options['encoding'])
            );
        }

but that lead to other problems and errors. I don't know the CSVWriter well enough, but maybe this helps a bit.

[Domingos Freitas]

@marcogrueter Hi Marco, I tried your code and it worked for some models but not for others. I ended up keeping my version until a official fix since I don't need character encoding conversion.
The only specific thing that I must do is making the exported csv file compatible with MS Office for Mac OS and Windows.

[Marco Grüter]

@uematsusoft Yeah I mostly got either a type conversion error or an error from the file reader (imho BOM related or something).

I just gave it another shot, and I think I got it working with this:

    /**
     * processExportDataAsCsv returns the export data as a CSV string
     */
    protected function processExportDataAsCsv($columns, $results, $options)
    {
        // Parse options
        $options = array_merge([
            'firstRowTitles' => true,
            'savePath' => null,
            'useOutput' => false,
            'fileName' => null,
            'delimiter' => null,
            'enclosure' => null,
            'escape' => null,
            'encoding' => null
        ], $options);

        // Prepare CSV
        $csv = CsvWriter::createFromStream(tmpfile());
        $csv->setOutputBOM(ByteSequence::BOM_UTF8);

        if ($options['delimiter'] !== null) {
            $csv->setDelimiter($options['delimiter']);
        }

        if ($options['enclosure'] !== null) {
            $csv->setEnclosure($options['enclosure']);
        }

        if ($options['escape'] !== null) {
            $csv->setEscape($options['escape']);
        }

        $encoder = (new CharsetConverter())->inputEncoding('UTF-8');

        if ($options['encoding'] !== null) {
            $encoder->outputEncoding($options['encoding']);
        }

        $csv->addFormatter($encoder);

        // Add headers
        if ($options['firstRowTitles']) {
            $headers = $this->getColumnHeaders($columns);
            $csv->insertOne($headers);
        }

        // Add records
        foreach ($results as $result) {
            $data = $this->matchDataToColumns($result, $columns);
            $csv->insertOne($data);
        }

        // Output
        if ($options['useOutput']) {
            $csv->output($options['fileName']);
            return;
        }

        // Save to file
        if ($path = $options['savePath']) {
            $resource = @fopen($path, 'w+');
            if (!is_resource($resource)) {
                throw new SystemException("{$path}: failed to open stream for export: No such file or directory.");
            }
            foreach ($csv->chunk(8192) as $chunk) {
                fwrite($resource, $chunk);
            }
            fclose($resource);
            return;
        }

        return $csv->toString();
    }

on top: use League\Csv\ByteSequence;

export config is just standard, with formatting defaults:

defaultFormatOptions:
    fileFormat: csv_custom
    delimiter: ';'
    enclosure: '"'
    escape: '\'
    encoding: 'utf-8'

I'm not sure about side effects, but I tried a few exports and everything seems fine.
Might help until a real / permanent solution is available.

OCMS version 3.6.8. Check that you got the same version, Looks like there was a change in a recent version, so this patch might not work before that.

[daft]

Thanks for the detailed investigation here.

We've got a fix coming for this in v3.6.10

We just created the stream from an empty string (as per docs):

$csv = CsvWriter::createFromString();

And then used the existing static call on CharsetConverter (docs):

if (
    $options['encoding'] !== null &&
    $csv->supportsStreamFilterOnWrite()
) {
    CharsetConverter::addTo($csv, 'UTF-8', $options['encoding']);
}

Along with the supportsStreamFilterOnWrite sanity check in case the internals change on us again. Confirming that supportsStreamFilterOnWrite returns true now.

Thanks!