Roald87 / InfoSysCrawler

Crawler for Beckhoff's InfoSys

InfoSysCrawler

Crawler for Beckhoff's InfoSys.

How it works

The crawler goes through the InfoSys menu and finds all subfolders and pages. It is not necessary to start at a top folder, but you can also start at a lower folder. Optionally some folders/pages can be ignored.

Current application

This crawler can be used to walk through InfoSys pages and look for a TwinCAT version number. The version number is often found on the bottom of a page, as shown below.

The version numbers are used for the unofficial TwinCAT changelog.

How to use it

1. Build the project using dotnet build.
2. Open a FSharp interactive console.
3. Import the dll using #r "InfoSysCrawler/bin/Debug/net6.0/InfoSysCrawler.dll", where you should change the path of the dll according to where it is saved on your local system.
4. Then you can crawl a subfolder as follows:

   open InfoSysCrawler.SiteMap
   open InfoSysCrawler.Request
   
   let url = (Url "https://infosys.beckhoff.com/english/menu/menu.php?id=8644252870837316006") // TC1xxx - TwinCAT 3 base // 5 s
   // Add folder names you do not want to into
   let ignoreFolders = ["Foreword"; "Installation"; "Samples"]
   // Add pages which you do not want to look for a twincat version number
   let ignorePages = ["Overview"; "Search"]
   // Start crawling
   let menu = traverseMenu ignoreFolders ignorePages url
   
   // Output you should see
   TC1000 | TwinCAT 3 ADS
   TC1100 | TwinCAT 3 I/O
   TC1200 | TwinCAT 3 PLC
   TC1210 | TwinCAT 3 PLC/C++
   TC1220 | TwinCAT 3 PLC/C++/MATLAB®/Simulink®
   TC1250 | TwinCAT 3 PLC/NC PTP 10
   TC1260 | TwinCAT 3 PLC/NC PTP 10/NC I
   TC1270 | TwinCAT 3 PLC/NC PTP 10/NC I/CNC
   TC1275 | TwinCAT 3 PLC/NC PTP 10/NC I/CNC E
   TC1300 | TwinCAT 3 C++
   TC1320 | TwinCAT 3 C++/MATLAB®/Simulink®

5. You can save the crawled data with menu |> saveAsJson "menu.json"
6. You can later open the data again with

   #r "InfoSysCrawler/FSharp.Json.dll"
   
   open FSharp.Json
   
   let deserialized = Json.deserialize<Node list> (File.ReadAllText("menu.json"))

Current dumps

In data/ you will find a few folders which are crawled and saved as a json file. Currently the following folders were crawled where the folders "Foreword", "Installation" and "Samples" and the pages "Overview" and "Search" are ignored. On ignored pages/folders the crawler doesn't try to find a TwinCAT version number. Data was last updated on 8 October 2022.

• data/plc_libraries.json: TwinCAT 3/TE1000 XAE/PLC/PLC Libraries
• data/texxxx.json: TwinCAT 3/TExxxx | TwinCAT 3 Engineering
• data/tc1xxx.json: TwinCAT 3/TC1xxx | TwinCAT 3 Base
• data/tfxxxx.json: TwinCAT 3/TFxxxx | TwinCAT 3 Functions