YRG999 / babel_lib

Experiments with randomness, Wikipedia, and YouTube APIs

BABEL_LIB

This is called BABEL_LIB after the Library of Babel website.

It started as an experiment in randomness but now hosts a bunch of experimental programs.

I used chatGPT to help me write some of them.

Experiments

1. random_sentence.py - creates random "words" and sentences.
   • In the spirit of the library of babel, I wanted to create a program that would string together random words to create sentences.
   • Ideally I wanted to recreate the library of babel site so that I could create books and eventually find one that made sense.
   • But with infinity, there's infinite nothingness and chaos as well as everything that's ever existed and ever will be, so I abandoned this.
   • I also thought about adding a dictionary so that only words that made sense would be output, but abandoned that too.
   • Update: see gensen_worddict.py for further explorations
2. random_sentence_save.py - same, except saves to file.
   • this is the same as the first one, except it saves the randomness to a file instead of outputting it in the console.
3. dice_roles.py - rolls 2 6-sided dice.
   • Inspired by randomness and craps, I asked chatGPT to "write a python program that simulates the role of two six-sided dice and generates a histogram of results."
   • I probably could have written it myself, but it was easier to ask chatGPT to do it, and it worked!
4. wiki_articles.py - displays the 10 most recently-updated articles.
   • I asked chatGPT if wikipedia had an API, then to write a program that would display the 10 most recently-updated articles.
   • "Create a python program that appends the titles of the most recently edited Wikipedia articles to a text file."
   • I thought I wasn't getting any responses, but it was because I had the output file open in VSCode, and it doesn't update text files in real time, I have to close & open the file again to see changes.
5. your_youtube.py - displays the 10 most recently-added YouTube videos.
   • I decided to switch to the YouTube API, which necessitated signing up with the Google develper program to get an API key. I didn't really try this sample until after I got the search location working.
6. youtube_search_loc.py - displays the 10 most recently-added YouTube videos in a specific location.
   • I skipped to this as I wanted to see the 10 most recent YouTube videos in a particular location. I had some trouble getting this running. See the section below for details.
7. lat_long.py - Displays the latitude and longitude for an address. Uses the Google Maps API.
   • I needed the latitude and longitude for the YouTube search location so I asked chatGPT to help me with the maps API.
   • This API required a separate API Key so I got that and ran it successfully.
8. yt_loc2.py - User-friendly YouTube search by location.
   • This integrates the YouTube search app with the latitude and longitude app and adds input questions in the console for an easier experience.
9. gnews_scraper.py - Scrape Google News headlines.
   • Just grabs the top stories. Can't figure out how to get all the stories on the page yet.
   • Updated to pull latest 5 headlines based on h4 top stories.
10. gensen_worddict.py - Expanded random_sentence.py.
    • Pulls random words from the https://random-word-api.herokuapp.com/word API endpoint and strings them together to create a sentence, then verifies the sentence against the language_tool_python API.
    • TODO: Needs articles added to create a valid sentence.
11. gen_pass.py - Generate passphrase.
    • Uses the https://random-word-api.herokuapp.com/word API endpoint to generate a passphrase.
    • As it is, this creates very complex passphrases that may be easier to remember than random character strings, but they're very long and complex.
    • TODO: Needs to use simpler words that can be easily remembered, less capitalization (however this may make them less secure).
12. news_scraper - BBC and NY Times headline scraper.
    • Adapted Indently's BBC News Headline scraper to scrape BBC news and New York Times headlines.
    • Removed keyword highlighter.

Setup

For initial setup, create a virtual environment & run it:

$ python -m venv venv
$ . venv/bin/activate

Then install requirements:

pip install -r requirements.txt

In subsequent setups, just run the second venv line to activate the virtual environment.

YouTube and Geolocation Requirements

Required for apps 5-8

• Sign up to develop with the Google Cloud Platform, create a project, and activate the YouTube Data API v3 and the Geocoding API.
• Create an .env file with YOUTUBE_API_KEY=yourapikey and MAPS_API_KEY=yourmapsapikey.

YouTube search

• I had to educate myself about how to get an API key. The Google console is excessively complex, but I was able to add the YouTube endpoint and restrict the API key to that API. I did the same for the Google Maps API key later.
• chatGPT omitted the location radius value, which is required if you pass location. I figured this out by reading the YouTube endpoint doc (link to come).
• I was getting some errors. I discovered it had to do with the location parameter by doing some experimentation with the URL. Removing the location query parameter worked, so I figured it had something to do with that.

Latitude and longitude

• Was using the wrong longitude. I didn't realize I needed to use a negative number for the longitude until I realized I was getting responses from a country in the eastern part of the globe instead of New York.
• I found the correct longitude on Google Maps, so I thought I could use the API to make this simpler.
• I got errors and narrowed it down to the radius. I forgot that the radius needed a measurement value, like mi for miles. And my input request left it out, so I fixed this.

Programming notes & links

• I don't know why I get this error in the console when I run dice_roles.py, But I think it's something I can ignore. 1 2

+[CATransaction synchronize] called within transaction

• Pyplot tutorial - something to read later.
• Had some trouble setting the environment variable using the .env file. 3 4 5 6
  • I had this issue before and solved it with load_dotenv(). 7
    • But I still can't figure out why os.getenv doesn't just work.
• The twelve-factor app - mentioned in the python-dotenv doc. It's a methodology for building Software as a service (SaaS) apps.
• Format JSON in VSCode - I got some JSON errors back and wanted to read it better so I wanted to know how to format JSON in VSCode.
  • This ended up not working for the initial error as it wasn't properly formatted, so I had to do it manually, but subsequent JSON was properly formatted and this did help.
  • Basically, used the Command Palette (⇧⌘P) and chose Format Selection.
• requests - doc for the python HTTP requests library, which makes it easy to send requests in python.
• MediaWiki API help - didn't read it but here it is as a resource.
• youtube python api samples
• YouTube Analytics API - not the one I needed.
• YouTube Data API Overview - this is the API I used for search; v3.
• Google console API dashboard
• Obtaining authorization credentials
• Google console credentials page
• Address Geocoding in the Google Maps APIs
• maps api metrics
• billing change not allowed - I had an issue using PayPal. Don't know why.
• YouTube Data API v3
• the youtube search API - this didn't tell me what I needed to know about the location parameter.
• Videos: list - this didn't help either.
• Search:list location - this helped. It said I needed to add the locationRadius as well.
• Let users watch, find, and manage YouTube content (main API page) - I kept getting this page, which is not useful as I can't get anywhere from here.
• Request contains an invalid argument Location paramter [sic] Youtube API - This helped me troubleshoot errors I was having with sending an invalid location parameter. Prior to this, I didn't know how to specify the latitude and longitude. I had been sending it as (lat,long) with the parentheses, but this answer showed that I should not use that.
• Jump Start Solution: Log analysis pipeline - this is totally off track and had nothing to do with anything. I just wanted to see what a jump start solution looked like, but I couldn't make sense of this (just a way to log and analyse data). Something for another day.
• I included pip>=23.0.1 in my requirements.txt file as I kept getting a note that I had an earlier version.
  • Run pip list to list installed packages.
• Git keeps asking me for my ssh key passphrase - helpful tip.

Jul 2 notes

• Sun, Jul 2, 2023
• Random word & dictionary APIs
  • https://developer.wordnik.com/docs#!/words/getRandomWord
  • https://random-word-api.herokuapp.com/home
  • https://pipedream.com/apps/dictionary-api
    • https://dictionaryapi.dev/
    • https://pipedream.com/apps/dictionary-api/integrations/google
  • https://rapidapi.com/blog/dictionary-apis/
  • https://www.wordsapi.com/
    • https://www.wordsapi.com/docs/#random-words
  • https://www.reddit.com/r/learnprogramming/comments/uslfi2/looking_for_a_random_word_api_where_i_can_specify/
    • https://github.com/nltk/nltk
    • https://www.datamuse.com/api/
• https://pypi.org/project/language-tool-python/
  • https://languagetool.org/
• https://docs.python.org/3/library/unittest.html
• https://stackabuse.com/guide-to-parsing-html-with-beautifulsoup-in-python/
• https://www.geeksforgeeks.org/beautifulsoup-scraping-paragraphs-from-html/
• https://towardsdatascience.com/web-scraping-basics-82f8b5acd45c