Harvest metadata from SRDA
jggautier opened this issue · comments
After Dataverse v6.1 is applied to Harvard Dataverse, we'll need to re-harvest SRDA's metadata into the collection at https://dataverse.harvard.edu/dataverse/srda_harvested
OAI URL:
https://srda.sinica.edu.tw/oai_pmh/oai
Set:
None
This is following the work discussed in the GitHub issue at IQSS/dataverse#7624
I saw that it failed to harvest any records in prod. on Friday.
I believe we need to remove all the SRDA records harvested from Datacite first, before trying to re-harvest their metadata into the same collection. I.e., we will need to delete that client, then re-create it from scratch.
I unscheduled it for now, let's handle this after 6.1 is deployed (in the next couple of days, hopefully).
BTW, the 6.1 release patch that I'm deploying in prod. is already running on demo.
So you can see the redirects working for the records harvested from https://srda.sinica.edu.tw/ there:
https://demo.dataverse.org/dataverse/srda
Also, note that the old records harvested via Datacite in https://dataverse.harvard.edu/dataverse/srda_harvested are now redirecting properly.
Let's coordinate about re-harvesting their records directly, later this week maybe?
Nice! Yeah anytime this week would be fine I think. If you don't mind, could you delete the client, harvest directly from them later this week, and let me know if that went okay? Or I could do it when you think it's best this week. I'll have blocks of time this Thursday and Friday afternoon that would work well.
A quick update: it took a good portion of the day on Fri. to delete all the datacite-harvested records.
An attempt to re-harvest directly did not succeed, unfortunately. I did re-run that direct harvest on demo, just as a quick check, and it worked like a charm again there. I have no explanation of what's going on. Another production mystery that will need to be resolved.
https://dataverse.harvard.edu/dataverse/srda_harvested
I left the client scheduled, weekly, Sun. 10pm. Let's keep an eye on it and check what happens over the weekend.
Our contact at SRDA emailed us that he noticed that those 2926 records were harvested. That's one short of the 2,927 records in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc (which interestingly has no pagination, so it takes a while to load the metadata of all 2,927 records).
I let him know that you told Harvard Dataverse to try again this Sunday and that we'll see what happens then 🤞
2928 actually, 2 failed. I can give them the 2 identifiers - it looks like it may be an intermittent problem on their end.
I would maybe also ask them why they are formatting their identifiers as <dc:identifier>10.6141/TW-SRDA-AA000001-1</dc:identifier>; instead of <dc:identifier>https://doi.org/10.6141/TW-SRDA-AA000001-1</dc:identifier> or <dc:identifier>doi:10.6141/TW-SRDA-AA000001-1</dc:identifier>? - that was the part that was causing most of the problems with their archive so far.
Those may all be moot-er points. On a closer look, I am very apprehensive about having their archive harvested regularly on a schedule. Their server does not appear to understand the incremental harvesting parameter (from=). Sending ListIdentifiers with from=2024-02-06 returns the same list of 2928 records. (Our Harvester doesn't use ListRecords; it does ListIdentifiers, then GetRecord individually). The <datestamp> entries don't seem to represent the modification times of the records - it is simply the current date/time for all the records.
I am not ok with re-harvesting ~3000 records from scratch every week. It would be preferable if they could fix this.
No pagination is another thing, yes; but less of a big deal compared to this.
Okay, in the email thread I asked about the identifier format and about the datestamp.
OK, great, saw your message.
If they ask followup questions about the identifier format - that part is not a problem for us anymore; Dataverse was confused by it, but I worked around it. It's just something they may want to consider - since the identifiers in question are registered and resolvable DOIs, why not advertise them as such?
The "from=" parameter on the other hand is a fairly important thing.
@landreev, our contact, Chucky, wrote back that "regarding the date stamp issue, we have already instructed our IT personnel to update the date to reflect the latest data version release date in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc," and that they'd make corrections to the identifier format (dc: identifier).
I'm not sure if it matters that they're referring to a ListRecords request and not the ListIdentifiers request that you said Dataverse uses. Should I ask them to clarify?
I scheduled it again for Sun. night, it's running now, we'll check on it tomorrow and see what it does.
It's usually the same code underneath that drives the behavior of ListRecords and ListIdentifiers, when it comes to dates. I didn't think it was necessary to ask, unless we see something weird.
I'm a little worried about how they phrased it - "update the date to reflect the latest data version". With "the date" possibly implying that it's the same date for all their records; which would again mean having to reharvest all their records every time. But let's see what/how many records we get tonight.
It did re-harvest all their records last night, i.e., still not incremental. I'm assuming we can just harvest their stuff like this once in a while manually for the time being, until they get the from= parameter to work on their end.
One small improvement is that there were no broken records this time (there were 2 that failed to import the last time around).
Nice! Okay. I got the sense that they hadn't updated the dates, yet, and Chucky will email again when they have. I can reach out in a month, on Mar 26, to ask if they've made the change, too. If they have by then we can set the schedule to weekly. If they haven't yet, we can just run another manual harvest then.
How's that sound?
2024/03/12
- We are now receiving records from this repository (running the harvesting jobs "manually" instead of on a schedule)
- SRDA will need to eventually fix their system so that we can harvest incrementally and schedule weekly harvest
- Keeping this issue open until they are able to fix their repository and so that we will remember to reharvest periodically
I see different dates in the <datestamp> of each record in https://srda.sinica.edu.tw/oai_pmh/oai?verb=ListRecords&metadataPrefix=oai_dc, but I'm not sure if that means that the datestamp issue has been resolved so that Harvard Dataverse can harvest incrementally and we can schedule weekly harvest.
I've emailed Chucky to ask if it's been resolved and if not, to email us when it has been so that I don't forget. See the email thread in RT
@landreev, Chucky's replied and I'm concerned that I've cause some confusion, maybe because I don't understand enough about how incremental harvesting works.
I've CC'ed you on the email thread. Could you reply when you get the chance?