[Bug] How do you handle graph breaks coming from Dynamo?

[tbaggu]

Hi

I am taking this repo as reference to implement code for our custom back-end. During the development process most of the cases when i use hugging-face models directly i see lot of graph breaks, in Fx graph,

My understanding on the inductor side is, each subgraph compiled and ran the inference sent the results to CPU, and the next subgraph will start executing which is time consuming?

So my question is ,
Have you seen such graph breaks if so how Hidet handles them?

similar to below case
https://discuss.pytorch.org/t/stitching-together-graph-breaks-for-large-compilation-units/194793/5

[Yaoyao Ding]

Hi @tbaggu ,

The dynamo custom backends can not control how the torch model is partitioned and converted to fx graph.

The torch dynamo dispatch the fx graph to custom backend and the backend compiles the fx graph to some executable and returns to dynamo. The compilation only happens once and the compiled executable will be used many times. As long as the compiled executable is efficient, the overhead will not be very large.

[tbaggu]

Hi Thanks for the quick response, Is it each sub graph compiled and executed then the second will compile ?? Is that flow is n sequence? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Yaoyao Ding ***@***.***> Sent: Friday, February 23, 2024 12:58:15 PM To: hidet-org/hidet ***@***.***> Cc: Tirupathi Rao Baggu ***@***.***>; Mention ***@***.***> Subject: Re: [hidet-org/hidet] [Bug] How do you handle graph breaks coming from Dynamo? (Issue #431) Caution: This email originated from outside of the organization. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department Hi @tbaggu<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_tbaggu&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=ZC4zaNjHruoICMj8o_Zo5qwbgveusL9RZskQW2w8A1KgxSL6oInP6y-Ex5HuY9gp&s=4_Z96Qo5vny3sSw6vm4IyQCZTRobTc3rD8tKRg0schw&e=> , The dynamo custom backends can not control how the torch model is partitioned and converted to fx graph. The torch dynamo dispatch the fx graph to custom backend and the backend compiles the fx graph to some executable and returns to dynamo. The compilation only happens once and the compiled executable will be used many times. As long as the compiled executable is efficient, the overhead will not be very large. — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hidet-2Dorg_hidet_issues_431-23issuecomment-2D1960847409&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=ZC4zaNjHruoICMj8o_Zo5qwbgveusL9RZskQW2w8A1KgxSL6oInP6y-Ex5HuY9gp&s=MqsJKff43jwbudwrspJOiSF1GuW41WH3S6wCLxG90gE&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_A4BNIQPD7MKQSZMQJKSLJALYVBAI7AVCNFSM6AAAAABDWFYL36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQHA2DONBQHE&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=ZC4zaNjHruoICMj8o_Zo5qwbgveusL9RZskQW2w8A1KgxSL6oInP6y-Ex5HuY9gp&s=KMT6SPIoBZHfgg4XIazK4WMrWj_Uke6Xciv6yteoHrk&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Yaoyao Ding]

Yes.

model_opt = torch.compile(model, backend='custom-backend')

model_opt(x) # sub graph will be compiled and executed
model_opt(x) # the cached compiled executable will be used, no compilation at all

[tbaggu]

So for each subgraph result should comeback to CPU and again it goes back to device right? Which is latency over head right? Get Outlook for Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Yaoyao Ding ***@***.***> Sent: Friday, February 23, 2024 1:19:48 PM To: hidet-org/hidet ***@***.***> Cc: Tirupathi Rao Baggu ***@***.***>; Mention ***@***.***> Subject: Re: [hidet-org/hidet] [Bug] How do you handle graph breaks coming from Dynamo? (Issue #431) Caution: This email originated from outside of the organization. Please take care when clicking links or opening attachments. When in doubt, contact your IT Department Yes. model_opt = torch.compile(model, backend='custom-backend') model_opt(x) # sub graph will be compiled and executed model_opt(x) # the cached compiled executable will be used, no compilation at all — Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_hidet-2Dorg_hidet_issues_431-23issuecomment-2D1960869256&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=NYlKO7jDMD63GsvK19tQ-JTXAGjnwoobE8ZiR5paV2kWfRyUvFYLswhWGrMy7c-M&s=yNyynTt-0EY6muA2axQ7QbOw334I8WQLlH6Ev7Smh-E&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_A4BNIQPJNZRMHG2BB4WMO2LYVBCZZAVCNFSM6AAAAABDWFYL36VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRQHA3DSMRVGY&d=DwMCaQ&c=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM&r=gq-inQZcrlLntwcvIjjLIH_HIhFbKD0Qm_YSVPXFWYE&m=NYlKO7jDMD63GsvK19tQ-JTXAGjnwoobE8ZiR5paV2kWfRyUvFYLswhWGrMy7c-M&s=odv07tt-4qJQryU3koLan4MauB9i2j2LYOdN-QIgZdM&e=>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Yaoyao Ding]

for each subgraph result should comeback to CPU

No, the result will stay in the same device (cpu or gpu) as the eager execution of the orginal model.

[Shang Wang]

Closing as this issue is not directly related to Hidet.