runprism / prism

Prism is the easiest way to develop, orchestrate, and execute data pipelines in Python.

Home Page:https://runprism.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Initialize hooks in Jupyter notebooks

mtrivedi50 opened this issue · comments

Is your feature request related to a problem? Please describe.
When developing task code, users may want to experiment and test the code in a Jupyter notebook prior to running it in the project. This enables a faster development feedback loop (i.e., users can rapidly test iterations of code in a notebook rather than modifying and re-running tasks after every change).

Describe the solution you'd like
I'd like to add a function initialize_hooks() that works as follows:

  • When inside a Prism project directory, the initialize_hooks function parses the prism_project.py file for a profile. If a profile doesn't exist, then it throws an error. If a profile does exist, then it parses the profile
  • After parsing the profile, the function instantiates all the adapters contained therein.
  • The hooks function should return an instance of the prism.infra.hooks.PrismHooks class. Users can use the methods of this class in their Jupyter notebook.
  • Here's what the function definition should look like
def initialize_hooks(project_dir: Optional[Path] = None):
    """
    Initialize hooks within a Prism project

    args:
        project_dir: Prism project directory. If `None`, then assume the current working
            directory is inside a Prism project. Default is `None`
    returns:
        hooks: instance of prism.infra.hooks.PrismHooks class
    """
  • Here's what the output should look like in a notebook:
image

Describe alternatives you've considered
N/A - open to hearing suggestions!

Additional context
This shouldn't require a ton of net-new code. The assignee should be able to accomplish this with a lot of the existing classes and structures.