Problem with scrapegraphai/graphs/pdf_scraper_graph.py

Question

Problem with scrapegraphai/graphs/pdf_scraper_graph.py

tindo2003 opened this issue 2 months ago · comments

Hello,

output of FetchNode being fed directly into RAGNode won't work because RagNode is expecting the second argument as a list of str. However, FetchNode is outputing a list of langchain Document.

When I perform the run function on an instance of PDFScraperGraph, I get the following error

ValidationError                           Traceback (most recent call last)
Cell In[3], [line 13](vscode-notebook-cell:?execution_count=3&line=13)
      [1](vscode-notebook-cell:?execution_count=3&line=1) from scrapegraphai.graphs import PDFScraperGraph
      [3](vscode-notebook-cell:?execution_count=3&line=3) pdf_scraper = PDFScraperGraph(
      [4](vscode-notebook-cell:?execution_count=3&line=4)     prompt="Which company sponsored the research?",
      [5](vscode-notebook-cell:?execution_count=3&line=5)     source="/Users/tindo/Desktop/lang_graph/data/lorem_ipsum.pdf",
   (...)
     [11](vscode-notebook-cell:?execution_count=3&line=11)     },
     [12](vscode-notebook-cell:?execution_count=3&line=12) )
---> [13](vscode-notebook-cell:?execution_count=3&line=13) result = pdf_scraper.run()

File ~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:105, in PDFScraperGraph.run(self)
     [97](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:97) """
     [98](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:98) Executes the web scraping process and returns the answer to the prompt.
     [99](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:99) 
    [100](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:100) Returns:
    [101](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:101)     str: The answer to the prompt.
    [102](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:102) """
    [104](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:104) inputs = {"user_prompt": self.prompt, self.input_key: self.source}
--> [105](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:105) self.final_state, self.execution_info = self.graph.execute(inputs)
    [107](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/pdf_scraper_graph.py:107) return self.final_state.get("answer", "No answer found.")

File ~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:171, in BaseGraph.execute(self, initial_state)
    [169](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:169)     return (result["_state"], [])
    [170](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:170) else:
--> [171](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:171)     return self._execute_standard(initial_state)

File ~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:110, in BaseGraph._execute_standard(self, initial_state)
    [107](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:107) current_node = next(node for node in self.nodes if node.node_name == current_node_name)
    [109](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:109) with get_openai_callback() as cb:
--> [110](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:110)     result = current_node.execute(state)
    [111](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:111)     node_exec_time = time.time() - curr_time
    [112](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/graphs/base_graph.py:112)     total_exec_time += node_exec_time

File ~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:85, in RAGNode.execute(self, state)
     [82](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:82) chunked_docs = []
     [84](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:84) for i, chunk in enumerate(doc):
---> [85](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:85)     doc = Document(
     [86](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:86)         page_content=chunk,
     [87](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:87)         metadata={
     [88](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:88)             "chunk": i + 1,
     [89](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:89)         },
     [90](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:90)     )
     [91](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:91)     chunked_docs.append(doc)
     [93](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/scrapegraphai/nodes/rag_node.py:93) self.logger.info("--- (updated chunks metadata) ---")

File ~/Desktop/lang_graph/env/lib/python3.10/site-packages/langchain_core/documents/base.py:22, in Document.__init__(self, page_content, **kwargs)
     [20](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/langchain_core/documents/base.py:20) def __init__(self, page_content: str, **kwargs: Any) -> None:
     [21](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/langchain_core/documents/base.py:21)     """Pass page_content in as positional or named arg."""
---> [22](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/langchain_core/documents/base.py:22)     super().__init__(page_content=page_content, **kwargs)

File ~/Desktop/lang_graph/env/lib/python3.10/site-packages/pydantic/v1/main.py:341, in BaseModel.__init__(__pydantic_self__, **data)
    [339](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/pydantic/v1/main.py:339) values, fields_set, validation_error = validate_model(__pydantic_self__.__class__, data)
    [340](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/pydantic/v1/main.py:340) if validation_error:
--> [341](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/pydantic/v1/main.py:341)     raise validation_error
    [342](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/pydantic/v1/main.py:342) try:
    [343](https://file+.vscode-resource.vscode-cdn.net/Users/tindo/Desktop/lang_graph/~/Desktop/lang_graph/env/lib/python3.10/site-packages/pydantic/v1/main.py:343)     object_setattr(__pydantic_self__, '__dict__', values)

ValidationError: 1 validation error for Document
page_content
  str type expected (type=type_error.str)

I believe the programmer expects us to perform the pipeline FetchNode -> ParseNode -> RAGNode, instead. Although, this may not make sense in the pdf scraping scenario. This is because the text_splitter.split_text() in parse_node turns a list of Document into a list of str. Thanks!

Marco Vinciguerra · Answer 1 · Fri Jun 07 2024 17:14:23 GMT+0800 (China Standard Time)

can you give me the script you used?

Tin Do · Answer 2 · Fri Jun 07 2024 20:59:51 GMT+0800 (China Standard Time)

I was trying to run the following:

from scrapegraphai.graphs import PDFScraperGraph

pdf_scraper = PDFScraperGraph(
    prompt="Which company sponsored the research?",
    source="data/lorem_ipsum.pdf",
    config={
        "llm": {
            "model": "gpt-3.5-turbo",
            "api_key": "my_key",
        }
    },
)
result = pdf_scraper.run()

and I received the error above. Thanks.

Marco Vinciguerra · Answer 3 · Fri Jun 07 2024 22:24:33 GMT+0800 (China Standard Time)

please open it separately

Tin Do · Answer 4 · Fri Jun 07 2024 23:02:54 GMT+0800 (China Standard Time)

I am not sure what you mean. The code snippet I post on the very top of this issue is the source code in pdf_scraper_graph.py, not the code that I tried to run. Sorry for the confusion if that is the case.

Marco Vinciguerra · Answer 5 · Tue Jun 11 2024 03:54:18 GMT+0800 (China Standard Time)

Yes can I have the entire main?

Marco Vinciguerra · Answer 6 · Sun Jun 16 2024 19:33:13 GMT+0800 (China Standard Time)

hi, please update to the new beta, it should be fixed.
If it does not work please reopen the issue