help - how to efficiently send files from spiffs

[biancoblu79]

Hello,

I am using google-cloud-iot-arduino on an ESP8266 based board. I am trying to send files to Google Cloud and I am currently able to do it by splitting them in 256 byte chunks. I store these chunks in Firestore using a cloud function and then put them back together using another cloud function and I store the file in a GCS bucket. What is the best way for me to use publishTelemetry to send an entire file (avg file size is 40k)? thanks! Marco .. here below what I am currently doing

size_t size = 0;
File dataFile = SPIFFS.open(pic_file_camera, "r");
size = dataFile.size();

if (dataFile) {
byte data[256];

int i = 0;
int total_bytes = 0;
while (dataFile.available()) {
  if (i < 256) {
    // if we haven't read 256 bytes yet, then keep reading!
    data[i++] = dataFile.read();
    total_bytes ++;
  } else {
    delay(100);
    //  send the read 256 bytes and reset the counter
    publishTelemetry("/pictures", (char *)data, 256);
    i = 0;
  }
}

// send the remaining i bytes 
delay(100);
publishTelemetry((char *)data, i);

delay(20);
// Close the file
dataFile.close();

[Gal Zahavi]

Are you trying to send a picture or a text document ?

[biancoblu79]

pictures..

thanks, marco

[Gal Zahavi]

I would suggest taking a look at the camera example since it's accomplishing something similar to what you want to do, i would look at the transmitImage,publishTelemetryFromFile functions

Link : https://github.com/GoogleCloudPlatform/google-cloud-iot-arduino/blob/master/examples/complex/esp32/camera/camera.ino

[biancoblu79]

Thanks a lot for the suggestion! It kind of helped.. Kind of because, the free heap I get on the ESP8266 is pretty small (about 6k). But it should still help me to reduce the number of times I sent data to google cloud. So this is how I refactored my code:

Serial.println(String(" Free Heap Memory"));
Serial.println(ESP.getFreeHeap(),DEC);

if (dataFile) {
int mem_size = 4096;
char *data = (char *)malloc(mem_size);

int i = 0;
int total_bytes = 0;
while (dataFile.available()) {
  if (i < mem_size) {
    // if we haven't read mem_size bytes yet, then keep reading!
    data[i++] = dataFile.read();
    total_bytes ++;
  } else {
    delay(100);
    //  send the read mem_size bytes and reset the counter
    Serial.println(String("Sending Data..."));
    publishTelemetry("/pictures", (char *)data, mem_size);
    i = 0;
  }
}

// send the remaining i bytes in the buffer
delay(100);
publishTelemetry((char *)data, i);

// free heap memory
Serial.println(String("Freeing up memory..."));
free(data);

// output the # of total bytes read
Serial.println(total_bytes + String(" Total Bytes Read"));

delay(20);
// Close the file
dataFile.close();

---

The code above works, in the sense that I do not get any compiler errors, any errors in the serial output, no overflows, nothing, everything seems to be working fine. But then the weird that is happening is that if mem_size is above 256 nothing gets really sent and from if I go and look at the cloud iot core logs there is an error message that only says: disconnected

do you have any idea why this could be happening?

(by the way.. I guess I'll invest few bucks on the ESP32 and that will probably reduce a lot of headaches..)

thanks! marco

[Gal Zahavi]

The reason you can't publish anything above 256, is because publishTelemetry has a limit of 256 KB. Unfortunately i think you'll still need to break up the file if you want to send more then 256 KB.

Here's a link for a resource that shows the limits of IoT Core

[Gal Zahavi]

Also the ESP32 is nice, i haven't had much experience with ESP8266. You can also use the ESP32 with ESP-IDF which is different then Arduino and more stable, Google also has support for that IDF.

[biancoblu79]

I wish I was sending 256k :) I am able to send 256 bytes at the time. The files I am trying to send are also much smaller (30k on average) than that 256KB IoT limit. I did some tests with memsize = 512, 1024, 2408 and I also tried to increase the delay time in between each publishtelemetry call, in the thought that maybe I was not letting a transmission complete before calling the next one. But that didn't help either. I then tried to slightly increase memsize (starting from 256) in small increments to see if 256 was really an hard limit some how and was able to successfully send data up until memsize = 400. Then if I keep increasing I get again this error message.. on IoT Core..

status: {
code: 9
description: "FAILED_PRECONDITION"
message: "The connection broke or was closed by the client."
}

I was not aware of ESP-IDF.. I'll definitely give it a try and I also put an order for a couple of ESP32s

thanks

[Gal Zahavi]

Yea there's less documentation and examples for Google's ESP-IDF IoT Core Library, so if i understand correctly you have multiple publishTelemetry calls separated by delays ? if you do then that's why you're getting this error , i would suggest just calling one publishTelemetry if you have multiple

The error states that the Cloud is closing connection with your device , this can happen for multiple reasons , one of them is the multiple publish calls, could you maybe post a link to your repo on github so i can take a look at your code ?

[Raphael Smith]

Just in case, please have a look at #144. 1. You may be getting that error because you are running out of heap when you generate the large packet. So it would be worthwhile monitoring your heap usage. 2. You will get periodic failures when regenrating your JWT.

[biancoblu79]

Thanks Galz10! I actually already put the code in a prior post. I am calling publishTelemetry every time I read mem_size bytes from the file and was using delays to make sure there was enough time for the transmission to complete before calling publishTelemetry again..

thanks Raphael I will definitely take a look at 144 as, true, I also get these JWT failures as well.

[biancoblu79]

I got the ESP32 and refactored the code to use

char* data = (char*)heap_caps_malloc(size, MALLOC_CAP_8BIT);

as suggested by the example that was provided. This solved the issue of not having to break the file in multiple parts when sending it to GCP. But I still faced the issue below though:

status: {
code: 9
description: "FAILED_PRECONDITION"
message: "The connection broke or was closed by the client."
}

I solved this by increasing the buffer size when instantiating the MQTTClient: mqttClient = new MQTTClient(512).. 512 this was too little for what I was trying to do. I increased it and now everything works fine. I believe I will still have an issue when regenerating the JWT.. so I'll keep an eye on 144 for that...