Migration with Option TSMW : fatal : "from time is not less than until time"

[olref]

I have many wsp files and don't know precisely the date of the oldest data.
I just know it is around the month of February 2013 => I use the following command :
./migration -option=TSMW -wspPath=TEST-from="2013-02-01" -until="2016-01-01" -dbname=migrated -influxDataDir=/data/db -tagconfig=migration_config.json

When I launch this command migrator.go end with a fatal :
log.Fatal(err) into the function MapWSPToTSMByWhisperFile().

In MapWSPToTSMByShard the first treated shard has a from=2013-02-01 and an until=2013-02-04.
The FetchUntilTime function from whisper.go (into "github.com/uttamgandhi24/whisper-go/whisper") is called with these parameters.
The FetchUntil change this "from" value to 2013-02-21 because there are no data before in my wsp files. At this point we have a "from" at 2013-02-21 and a "until" at 2013-02-04 => whisper.go return an error and migrator stop the treatment.

With some debug I found that I have to use a "from" at 2013-02-21 but it took me some time ;-)

Maybe it is possible to ignore this error ("from time is not less than until time") or to inform the user about the "from" he should use ?

[Uttam Gandhi]

Hi,
Thanks for testing the utility and reporting the bug
I think you have an older version, I had pushed code on Feb 16.
Please get latest version and check it

[olref]

Hello,

I check with the last version, I still have the problem.

[Uttam Gandhi]

Thanks for your feedback

I have added -wspinfo option, with it you can check the oldest timestamp in file and total number of points in whisper file. I hope this helps you.

I have created a pull request, to handle this request
#8

If you are keen to have a look you take the feature branch

[olref]

Thanks for the wspinfo option.

I make my tests with 450 whisper files (If I choose to migrate to influxdb, I'll have more than 10,000 whisper files to convert). To have the oldest data in my files, I changed GethisperInfo() to only return informations about the oldest data:

func (migrationData *MigrationData) GetWhisperInfo() error {
    var oldestTime uint32 = uint32(time.Now().Unix())
    var oldestFile string

    for _, wspFile := range migrationData.wspFiles {
        w, err := whisper.Open(wspFile)
        if err != nil {
            return err
        }
        t, err := w.GetOldest()
        if err != nil {
            return err
        }
        if t < oldestTime {
            oldestTime = t
            oldestFile = wspFile
        }

        w.Close()
    }

    fmt.Printf("Oldest Whisper File : %s\n", oldestFile)
    fmt.Println("Oldest Data in this File : ", time.Unix(int64(oldestTime), 0))
    fmt.Println("-----------------------------------------------------------------------")

    return nil
}

[Dmitry]

I think that you need to check -until time for every WSP file to prevent reading of the undefined points from it. Because if you have many WSP files you can't be sure that they all have the same -until timestamp.
And this is correct for -from setting to.