[@azure-rest/ai-inference] incorrect type when using streaming
sinedied opened this issue · comments
- Package Name: @azure-rest/ai-inference
- Package Version: 1.0.0-beta.5
- Operating system: MacOS
- nodejs
- version: v22.13.1
- browser
- name/version:
- typescript
- version: 5.8.2
- Is the bug related to documentation in
- README.md
- source code documentation
- SDK API docs on https://learn.microsoft.com
Describe the bug
Having to force-cast types like here:
const stream = responseStream.body as IncomingMessage;when using streaming feels off, it would be best to fix either .asNodeStream() or .createSseStream() method to make them compatible and not force a fix onto users (again, strict linters rules can forbid casting at all).
I'm not sure when the limitation comes from, as the object are compatible with each other so it means that the type definition in one of these two method is wrong.
Expected behavior
Not having to force-cast the type of the stream body, types should be compatible out of the box.
The reason createSseStream takes an IncomingMessage is that it has a cancel method that is called in cancellations scenarios. NodeJS.ReadableStream on the other hand doesn't have such API.
Is it possible to update the asNodeStream method to return IncomingMessage? stream cancellation is an important scenario and it is supported by the stream returned by the other method asBrowserStream. /cc @mpodwysocki @jeremymeng @timovv
Is it possible to update the
asNodeStreammethod to returnIncomingMessage?
The readableStreamBody on the body may be transformed already (for example, progress report). @joheredi do you know whether that is possible with RLC?
azure-sdk-for-js/sdk/core/core-rest-pipeline/src/nodeHttpClient.ts
Lines 165 to 175 in d18dd5e
@jeremymeng, Great callout! @azure-rest/core-client supports progress reports so yes, we should consider them supported. I think there are a couple candidates to be the asNodeStream's updated return type:
Readable, which supports the cancellation API and is parent of bothhttp.IncomingMessageandTransform. The downside is that we are exposing NodeJS class type in our APIs which we have been trying not to for some time now.- A new interface that extends
NodeJS.ReadableStreamwith adestroymethod.
createSseStream should also take either of these as input but that should be a straightforward overload addition.
As an alternative to updating the return type of asNodeStream would it be possible to add NodeJS.ReadableStream as an accepted input for createSseStream and feature-detect for destroy in core-sse?
@timovv I hate that type tbh and don't want it to spread. Generally, we should give the user the ability to cancel the stream which is useful in cases where the consumer is different from the one made the request.
I just feel that it's less risky than forcing through the other approach which has already caused build failures. Like it or not, the type is there and it's going to be hard to get rid of without a breaking change (we can do it for unbranded though!)
The change is not breaking. The build failure is because of multiple copies of core being imported by the test. Are we supporting scenarios that allow multiple different copies living alongside each other?
The change is not breaking. The build failure is because of multiple copies of core being imported by the test. Are we supporting scenarios that allow multiple different copies living alongside each other?
Technically it's breaking because in PipelinePolicy we return Promise<PipelineResponse> and we're adding a new required parameter to an output type that a consumer must implement. We ran into similar problems when we tried to extend the old FormData primitive in core not that long ago to handle multipart.
While our guidance is to avoid mix and matching versions of core, sometimes it happens, like say:
- We introduce a small breaking change to core and major version it
- We update SDK A to use the new core and release that version
- We update SDK B to use the new core, but haven't released a new version yet (maybe they're in the midst of a service API update)
- A customer depends on SDK A and B in their application, but now there are two different versions of core pulled in.
Really, the types for node are wrong and should reflect this API as streams have had support for destroy since 8.0: https://nodejs.org/docs/latest/api/stream.html#readabledestroyerror
It looks like someone tried to fix this back in 2017 but ran into issues (probably from other places that reference the old type): DefinitelyTyped/DefinitelyTyped#17977
As a stopgap solution, could we make it optional? It forces us to write ?.destroy() instead of .destroy() but at least there's no casting involved. @deyaaeldeen thoughts?
As a stopgap solution, could we make it optional? It forces us to write ?.destroy() instead of .destroy() but at least there's no casting involved. @deyaaeldeen thoughts?
FWIW this is what we had to do with addEvent in core-tracing
Etc.
Not sure if it's apples-to-apples but similar enough 😄
Really, the types for node are wrong and should reflect this API as streams have had support for destroy since 8.0
For context, this interface is meant to represent the old style streams that has a subset of the current stream APIs:
https://github.com/DefinitelyTyped/DefinitelyTyped/blob/3a17ff00600a50db103774cfb818685c5677e815/types/node/stream.d.ts#L449, hence it misses APIs such as destroy as well as others. Our usage of this interface is not ideal because we shouldn't want to restrict our customers to only that subset.
While our guidance is to avoid mix and matching versions of core, sometimes it happens
I think it is reasonable to assume that the package version resolution algorithm will do the reasonable thing and use a single version as long as they're simvar-compatible. Otherwise, we should forbid adding required properties on return types.
Technically it's breaking because in PipelinePolicy we return Promise and we're adding a new required parameter to an output type that a consumer must implement.
Dang, I think this could be the actual deal breaker. I wonder why I didn't bump into this, this use case may not be captured in our tests, is it? I can see it is hard to represent because we will need to capture the old type as the return type of a stream factory function to force the assignability error.
As a stopgap solution, could we make it optional?
Could this be confusing to our customers, to have something that always exist at runtime be typed as optional? I don't have strong feelings here.
I wonder why I didn't bump into this
Yeah we don't really have thorough testing of our types which would cause this to get picked up in the dev loop. But ultimately it did get picked up in the Storage build failure -- if the two Core versions' API surfaces were truly compatible, it would have built just fine despite there being two versions present (in fact, those tests have been using two versions of Core ever since we shipped Recorder v3 and bumped the copy in the repo to v4 as part of the ESM migration work).
At one point I was working on a prototype of a breaking change detector which likely would have flagged this, but I haven't worked on it for a while.
Personally I don't feel that customers gain all that much by adding destroy as optional and would lean toward just using the existing type in core-rest-pipeline, warts and all, with the intent of breaking and moving to the new type in the unbranded package. core-sse could then widen the type for input to include destroy as optional. But I understand I may be in the minority here. If we do proceed with making the change in core-rest-pipeline, are there any other methods on Readable that would be worth adding at the same time? I don't know if we've had any customers request support for cancellation in particular outside of this SSE case, so I wonder if there's anything else we should add in while we're thinking about this.