Cannot configure fields of optional dataclasses using env-vars

[maticus]

🐛 Bug report

Prompted by the super-quick fix of #507 (thanks!) I'm doing some more testing of jsonargparse with nested optional dataclasses, and found out:

Cannot configure fields of optional dataclasses using env-vars

To reproduce

Following script

import jsonargparse
from typing import Optional
from dataclasses import dataclass

@dataclass
class B:
    c: int = 3

@dataclass
class A:
    b: B
    ob: Optional[B]

def fun(a: A):
    print(a)

jsonargparse.CLI(fun, default_env=True, env_prefix="FOO")

when ran:
FOO_A__OB__C=7 FOO_A__B__C=8 python python-jsonargparse-optional-dataclass.py

displays:
A(b=B(c=8), ob=None)

Note, I can still configure this filed using --a.ob.c:

$ python python-jsonargparse-optional-dataclass.py  --a.b.c 77 --a.ob.c 78
A(b=B(c=77), ob=B(c=78))

Expected behavior

FOO_A__OB__C=7 env-var should configure the appropriate field.

Additionally, the FOO_A__OB__C should be visible in generated help (running script with -h) - related to #509

Environment

• jsonargparse version: today's master :-) (git-describe: v4.28.0-10-gf44ba32)
• Python version: 3.10
• How jsonargparse was installed: pip install -e ".[dev,all]" from checked-out repo.
• OS: Ubuntu Jammy

[Mauricio Villegas]

This behavior is expected. Might seem inconsistent but not a bug. Let me explain. When there is b: B the only possibility is that b will be an instance of B. Each of the fields of B can be added to the parser since they are always supported. In contrast, when ob: Optional[B] then the fields of B are not always supported, and thus not added to the parser. Only --a.ob is available in the parser and its corresponding env var. This can be observed by using --help. Note that Optional[B] is equivalent to a union B | None, and could be a much more complex type, e.g. int | B | X | None. Allowing all possibilities to be set by environment variables would be rather complex to implement, and there wouldn't even be a way to show the corresponding var names in the help. And would create weird cases, e.g. env vars given for B field and X field.

Even though FOO_A__OB__C is not available, it is possible to set like:

$ FOO_A__OB='{"c": 4}' ./issue_510.py
A(b=B(c=3), ob=B(c=4))