E2E Tests
sidmohanty11 opened this issue · comments
We need to add e2e tests and have a mock api setup so that we can much better track any regressions or bugs
Tasks
API: https://github.com/RocketChat/EmbeddedChat/blob/develop/packages/api/src/EmbeddedChatApi.ts
-
setAuth(auth: RocketChatAuth) -
getAuth() -
getHost() -
googleSSOLogin(signIn: Function, acsCode: string) -
login(userOrEmail: string, password: string, code: string) -
logout() -
connect() -
addMessageListener(callback: (message: any) => void) -
removeMessageListener(callback: (message: any) => void) -
addMessageDeleteListener(callback: (messageId: string) => void) -
removeMessageDeleteListener(callback: (messageId: string) => void) -
addTypingStatusListener(callback: (users: string[]) => void) -
removeTypingStatusListener(callback: (users: string[]) => void) -
addActionTriggeredListener(callback: (data: any) => void) -
removeActionTriggeredListener(callback: (data: any) => void) -
addUiInteractionListener(callback: (data: any) => void) -
removeUiInteractionListener(callback: (data: any) => void) -
updateUserNameThroughSuggestion(userid: string) -
updateUserUsername(userid: string, username: string) -
channelInfo() -
permissionInfo() -
close() -
getMessages(anonymousMode = false, options: { query?: object | undefined; field?: object | undefined; } = { query: undefined, field: undefined }, isChannelPrivate = false) -
getThreadMessages(tmid: string, isChannelPrivate = false) -
getChannelRoles(isChannelPrivate = false) -
sendTypingStatus(username: string, typing: boolean) -
sendMessage(message: any, threadId: string) -
deleteMessage(msgId: string) -
updateMessage(msgId: string, text: string) -
getAllFiles(isChannelPrivate = false) -
getAllImages() -
starMessage(mid: string) -
unstarMessage(mid: string) -
getStarredMessages() -
getPinnedMessages() -
getMentionedMessages() -
pinMessage(mid: string) -
unpinMessage(mid: string) -
reactToMessage(emoji: string, messageId: string, shouldReact: string) -
reportMessage(messageId: string, description: string) -
findOrCreateInvite() -
sendAttachment(file: File, fileName: string, fileDescription = "", threadId = undefined) -
me() -
getChannelMembers(isChannelPrivate = false) -
getSearchMessages(text: string) -
getMessageLimit() -
triggerBlockAction({ type, actionId, appId, rid, mid, viewId, container, ...rest }: any) -
getCommandsList() -
execCommand({ command, params }: { command: string; params: string }) -
getUserStatus(reqUserId: string)
@sidmohanty11 Any tools in mind? I'd recommend Cypress for e2e tests and json-server for mock APIs
We already have a base setup using Playwright here (very similar to Cypress): https://github.com/RocketChat/EmbeddedChat/tree/develop/packages/e2e-react, and I was thinking if we can improve it from there.
Well there is only one example test...we do need to add more tests and some test coverage requirements
This would be a large epic - we can divide it into a task list like this
Would like to work on this too. @sidmohanty11 @JeffreytheCoder maybe we can divide the work if that is okay.
Yes @JeffreytheCoder that would be awesome! We can also go API by API and their related feature set first and then target a bit harder features like realtime communications etc. What do you think?
@umangutkarsh that would be awesome! Lets divide it using a central issue and we can keep track from there only
Yes @JeffreytheCoder that would be awesome! We can also go API by API and their related feature set first and then target a bit harder features like realtime communications etc. What do you think?
Sounds good! Could you create a central issue with a task list?
I've listed all the functions here, so mostly we can just mock the API requests and test the feature with the help of that
@sidmohanty11 Do we need mock APIs for e2e tests? My understanding is that e2e tests should simulate as close to real-world usage as possible. Although the RC server is an external service to EC, it's basically our entire backend.
Check out this discussion on Reddit.
Can we use something like msw with playwright as mentioned in the discussion, (only for api testing) to avoid hitting actual endpoints?
Hey @sidmohanty11 , @JeffreytheCoder , @umangutkarsh ! 🌟 Since this issue is huge, and I personally want to learn writing e2e tests and learn more about automated workflows, I would love to contribute to this central issue if everyone is on board with it. 😊
Yes! @Spiral-Memory Let's do it together 🚀
I researched about mock service worker. Maybe we could use that with playwright tests? What do you all think?
Can we use something like msw with playwright as mentioned in the discussion, (only for api testing) to avoid hitting actual endpoints?
But why? I think it should be as close to real-world situations. Why do we want to do this? Any specific reason, @umangutkarsh ? We already have avitechlab as hosted server on which we can hit for testing. Please correct me if i am missing something
If the concern is "server unavailable" or something like that, then we can test on real server, but when it is down or unavailable then we may have msw
@JeffreytheCoder , what's your opinion?
Having mock APIs make our tests become integration tests, which also does the job since EC is just frontend. However, doing e2e tests means we don't have to create mock APIs and put EC into test by real user interaction scenarios. I would lean towards e2e tests
Can we use something like msw with playwright as mentioned in the discussion, (only for api testing) to avoid hitting actual endpoints?
But why? I think it should be as close to real-world situations. Why do we want to do this? Any specific reason, @umangutkarsh ? We already have avitechlab as hosted server on which we can hit for testing. Please correct me if i am missing something
If the concern is "server unavailable" or something like that, then we can test on real server, but when it is down or unavailable then we may have msw
@JeffreytheCoder , what's your opinion?
Avitechlab might not always be available I think. The certificate on the server might sometimes expire. And I think the tests should not be dependent on that. Please correct me if I am wrong.
Can we use something like msw with playwright as mentioned in the discussion, (only for api testing) to avoid hitting actual endpoints?
But why? I think it should be as close to real-world situations. Why do we want to do this? Any specific reason, @umangutkarsh ? We already have avitechlab as hosted server on which we can hit for testing. Please correct me if i am missing something
If the concern is "server unavailable" or something like that, then we can test on real server, but when it is down or unavailable then we may have msw
@JeffreytheCoder , what's your opinion?
Avitechlab might not always be available I think. The certificate on the server might sometimes expire. And I think the tests should not be dependent on that.
Yep, that's why i was thinking to have both approaches. For now, let's go with e2e test as suggested by @JeffreytheCoder
Later we can handle the unavailable service issue by mocking.
So for testing only the user interaction scenarios, what do you think about using codegen which comes with playwright?
So for testing only the user interaction scenarios, what do you think about using codegen which comes with playwright?
But for complex scenarios we might require manual coding for additional edge cases.
So for testing only the user interaction scenarios, what do you think about using codegen which comes with playwright?
Not sure, what exactly it does.
So for testing only the user interaction scenarios, what do you think about using codegen which comes with playwright?
Not sure, what exactly it does.
I mean it will generate the code for the user interactions which you can see in the Playwright Inspector window.
Ohh @umangutkarsh then if required, it can be added..
import { test, expect } from '@playwright/test';
test('Send Message', async ({ page }) => {
await page.goto('/');
await page.getByPlaceholder('example@example.com').click();
await page.getByPlaceholder('example@example.com').fill('tester123');
await page.locator('input[type="password"]').click();
await page.locator('input[type="password"]').fill('tester');
await page.getByRole('button', { name: 'Login' }).click();
await page.getByPlaceholder('Message').click();
await page.getByPlaceholder('Message').fill('send2');
await page.locator('div:nth-child(3) > div:nth-child(2) > div > .ec-box > .ec-button').click();
});
import { test, expect } from '@playwright/test';
test('Login', async ({ page }) => {
await page.goto('/');
await page.getByRole('button', { name: 'JOIN' }).click();
await page.getByPlaceholder('example@example.com').click();
await page.getByPlaceholder('example@example.com').fill('tester123');
await page.locator('input[type="password"]').click();
await page.locator('input[type="password"]').fill('tester');
await page.locator('input[type="password"]').press('Enter');
});
a basic test of sending a message and login .
@sidmohanty11 @Spiral-Memory @JeffreytheCoder . some basic user interaction tests can be generated with codegen i think. But for testing we would require a shared account on the avitechlab server.
Any suggestions?
@umangutkarsh once read our discussions in PR #570 regarding shared account. Also let's discuss more about this and work after we have a meeting and requirements have been clarified.
Currently @JeffreytheCoder have hardcoded the test user in playright, later Siddarth or Abhinav can add those credentials as env in Github automated workflows.
Oh, my bad. I didn't see this PR.
Sure, @Spiral-Memory .