awkward / Ocarina

A library to get metadata and Open Graph information from URLs

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Can't fetch twitter metadata

jinthislife opened this issue · comments

Hello Ocarina.

I would like to fetch metadata from twitter url but it doesn't work as I expected.
I was able to only get faviconURL which seems to be kind of broken and most other information was not fetched.
I would like to know if I am missing something here or some fix should be done for this library.
I tried out the code below.

     let url = URL(string: "https://twitter.com/qldhealthnews/status/1326368544985059331?s=20")

     url?.oca.fetchInformation(completionHandler: { (information, error) in
     if let information = information {
         print("Title     : \(String(describing: information.title))")
         print("URL       : \(String(describing: information.url))")
         print("imageURL  : \(String(describing: information.imageURL))")
         print("imageSize : \(String(describing: information.imageSize))")
         print("faviconURL: \(String(describing: information.faviconURL))")
         print(information)
      } else if let error = error {
         print(String(describing: error))
       }
    })

Then, I got logs below

Title     : nil
URL       : https://twitter.com/qldhealthnews/status/1326368544985059331?s=20
imageURL  : nil
imageSize : nil
faviconURL: Optional(//abs.twimg.com/favicons/twitter.ico -- https://twitter.com/qldhealthnews/status/1326368544985059331?s=20)
Ocarina.URLInformation

It seems Twitter requiring JavaScript for all user-agents except a few.

One way to get around that is to use a user agent that includes Twitterbot like:

Ocarinabot/1.1.3 facebookexternalhit/1.1 Facebot Twitterbot/1.0

You can use the following line of code to achieve this. Place this code anywhere before you can .oca on a URL. For example, you can place this code in your AppDelegate's didFinishWithLaunching.

OcarinaManager.shared.userAgent = "Ocarinabot/\(Int(OcarinaVersionNumber)) facebookexternalhit/1.1 Facebot Twitterbot/1.0"

Thanks for taking your time into this @renssies.

Based on your reply, I did some research and I found out that it works when the user agent includes the word "bot" string even if when it doesn't include "Twitterbot"

I tried 3 examples below and first two work except the last one.

OcarinaManager.shared.userAgent = "Ocarina bot/\(Int(OcarinaVersionNumber))"     // OK
OcarinaManager.shared.userAgent = "Ocarina Bot/\(Int(OcarinaVersionNumber))"     // OK
OcarinaManager.shared.userAgent = "Ocarinabot/\(Int(OcarinaVersionNumber))".     // Not Ok