How to setup OAuth2

Question

How to setup OAuth2

micol92 opened this issue 2 months ago · comments

Describe the bug

A clear and concise description of what the bug is.

I'm trying to set up Answer in my SAP BTP Kyma runtime(aka managed K8s), confirmed that basic functions(Q&A) works well, and would like to specify OAuth2 configuation to connect my Identity Provider.

It looks that configuration tasks are almost done, but it fails after logging and erros occurrs as below.

---error message in my K8s Pod.
2024-04-21 13:50:00.316 ERROR connector-basic/basic.go:155 fail to get user id from json path: email

--error screen in my web page.
see the attached file.

To Reproduce

It's possible to reproduce, but if you really need to, we'll pass along the information you need to configure it.

Expected behavior

Can I modify the original source and leave a log for debugging?
Here is the source for the step before the error occurs, and I would like to see the data and userinfo.

----connector-basic/basic.go
data, _ := io.ReadAll(response.Body)

userInfo = plugin.ExternalLoginUserInfo{
	MetaInfo: string(data),
}

A clear and concise description of what you expected to happen.

Screenshots

see the attached file for error screen.

Platform

Device: Desktop
OS: K8S. my desktop : Mac book M1
Browser and version: Chrome, Safari
Version: �2.0

LinkinStars · Answer 1 · Sun Apr 21 2024 17:51:32 GMT+0800 (China Standard Time)

would like to specify OAuth2 configuation to connect my Identity Provider

@micol92

Is the OAuth2 implemented by yourself? If it is self-implemented, you need to return the JSON data in the API response, and the JSON should include a unique identifier, like user_id or something else.

If you have configured another OAuth2 service, please tell us which one you have configured. Google?

Additionally, providing your OAuth configuration can help us pinpoint the issue.

Can I modify the original source and leave a log for debugging?

You can build docker image with your own plugins. FYI: https://answer.apache.org/docs/plugins#build-docker-image-with-plugin-from-answer-base-image

Here is the source for the step before the error occurs, and I would like to see the data and userinfo.

Let me make it clear.

OAuth2 servier response JSON will be like.

{
    "user_id": "xxx",
    "eamil": "exp@xxx.com"
}

The User ID JSON Path should config like user_id.

micol92 · Answer 2 · Sun Apr 21 2024 21:48:44 GMT+0800 (China Standard Time)

Hello.
Thank you for your reply.

SAP BTP(Cloud Platform) provides IAS(Identity Authentication Service), and I used it for OAuth2 service.
For the OAuth2 configuration, see the attached file.

And OAuth2 response JSON is as followings. Please note what value I should use for User ID JSON Path.

{
  "firstname": "Jungwoo",
  "lastname": "Han",
  "email": "jungwoo.han@sap.com",
  "name": "jungwoo.han@sap.com",
  "scopes": [
    "openid",
    "uaa.user"
  ],
  "displayName": "Jungwoo Han (jungwoo.han@sap.com)"
}

For debugging the second source, I tried the following.
Before the source code where the error occurs, I added some source code to get some information.
And then, I tried to build using the guide link for source debugging and failed, so please make sure my Dockerfile is correct.

=====modified source codes in my github=========================

	log.Errorf("[jw log] data :  %s", data)
	
	userInfo = plugin.ExternalLoginUserInfo{
		MetaInfo: string(data),
	}
	log.Errorf("[jw log] userInfo :  %s", uerInfo)
	
	if len(g.Config.UserIDJsonPath) > 0 {
		userInfo.ExternalID = gjson.GetBytes(data, g.Config.UserIDJsonPath).String()
	}

=======================================================
=============Dockerfile for building Plugin functions=============
=======================================================

FROM apache/answer as answer-builder
ENV COREPACK_ENABLE_STRICT 0
FROM golang:1.19-alpine AS golang-builder

COPY --from=answer-builder /usr/bin/answer /usr/bin/answer

RUN apk --no-cache add \
    build-base git bash nodejs npm go && \
    npm install -g pnpm


RUN answer build --with github.com/micol92/jw-incubator-answer-plugins/connector-basic  --with github.com/apache/incubator-answer-plugins/storage-s3 --output /usr/bin/new_answer


FROM alpine
LABEL maintainer="linkinstar@apache.org"

ARG TIMEZONE
ENV TIMEZONE=${TIMEZONE:-"Asia/Shanghai"}

RUN apk update \
    && apk --no-cache add \
        bash \
        ca-certificates \
        curl \
        dumb-init \
        gettext \
        openssh \
        sqlite \
        gnupg \
        tzdata \
    && ln -sf /usr/share/zoneinfo/${TIMEZONE} /etc/localtime \
    && echo "${TIMEZONE}" > /etc/timezone

COPY --from=golang-builder /usr/bin/new_answer /usr/bin/answer
COPY --from=answer-builder /data /data
COPY --from=answer-builder /entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh

VOLUME /data
EXPOSE 80
ENTRYPOINT ["/entrypoint.sh"]

=======================================================
====================error log occured during building===================
=======================================================

go: finding module for package github.com/apache/incubator-answer-plugins/storage-s3
go: downloading github.com/micol92/jw-incubator-answer-plugins v1.2.1
go: downloading github.com/apache/incubator-answer-plugins v1.2.1
go: downloading github.com/micol92/jw-incubator-answer-plugins/connector-basic v1.2.6
go: downloading github.com/apache/incubator-answer v1.3.0
go: downloading github.com/apache/incubator-answer-plugins/storage-s3 v1.2.6
go: found github.com/apache/incubator-answer-plugins/storage-s3 in github.com/apache/incubator-answer-plugins/storage-s3 v1.2.6
go: found github.com/apache/incubator-answer/cmd in github.com/apache/incubator-answer v1.3.0
go: found github.com/micol92/jw-incubator-answer-plugins/connector-basic in github.com/micol92/jw-incubator-answer-plugins/connector-basic v1.2.6
go: answer imports
github.com/micol92/jw-incubator-answer-plugins/connector-basic: github.com/micol92/jw-incubator-answer-plugins/connector-basic@v1.2.6: parsing go.mod:
module declares its path as: github.com/apache/incubator-answer-plugins/connector-basic
but was required as: github.com/micol92/jw-incubator-answer-plugins/connector-basic
build failed exit status 1

LinkinStars · Answer 3 · Sun Apr 21 2024 23:37:08 GMT+0800 (China Standard Time)

github.com/micol92/jw-incubator-answer-plugins/connector-basic: github.com/micol92/jw-incubator-answer-plugins/connector-basic@v1.2.6: parsing go.mod:
module declares its path as: github.com/apache/incubator-answer-plugins/connector-basic

@micol92

Maybe you haven't modified the module declare to github.com/micol92/jw-incubator-answer-plugins/connector-basic in your go.mod file.

micol92 · Answer 4 · Mon Apr 22 2024 06:34:37 GMT+0800 (China Standard Time)

Please check go.mod file modified by myself.
You can see the file in direct using this link.
https://github.com/micol92/jw-incubator-answer-plugins/blob/main/connector-basic/go.mod

================go.mod=============
module github.com/micol92/jw-incubator-answer-plugins/connector-basic

go 1.19

require (
	github.com/apache/incubator-answer v1.2.5
	github.com/segmentfault/pacman v1.0.5-0.20230822083413-c0075a2d401f
	github.com/tidwall/gjson v1.14.4
	golang.org/x/oauth2 v0.4.0
)

require (
	github.com/LinkinStars/go-i18n/v2 v2.2.2 // indirect
	github.com/Masterminds/semver/v3 v3.1.1 // indirect
	github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d // indirect
	github.com/aymerick/douceur v0.2.0 // indirect
	github.com/bytedance/sonic v1.9.1 // indirect
	github.com/chenzhuoyu/base64x v0.0.0-20221115062448-fe3a3abad311 // indirect
	github.com/dustin/go-humanize v1.0.1 // indirect
	github.com/gabriel-vasile/mimetype v1.4.2 // indirect
	github.com/gin-contrib/sse v0.1.0 // indirect
	github.com/gin-gonic/gin v1.9.1 // indirect
	github.com/go-playground/locales v0.14.1 // indirect
	github.com/go-playground/universal-translator v0.18.1 // indirect
	github.com/go-playground/validator/v10 v10.14.0 // indirect
	github.com/go-sql-driver/mysql v1.6.0 // indirect
	github.com/goccy/go-json v0.10.2 // indirect
	github.com/golang/protobuf v1.5.2 // indirect
	github.com/golang/snappy v0.0.4 // indirect
	github.com/google/uuid v1.3.0 // indirect
	github.com/google/wire v0.5.0 // indirect
	github.com/gorilla/css v1.0.0 // indirect
	github.com/json-iterator/go v1.1.12 // indirect
	github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
	github.com/klauspost/cpuid/v2 v2.2.4 // indirect
	github.com/leodido/go-urn v1.2.4 // indirect
	github.com/lib/pq v1.10.7 // indirect
	github.com/mattn/go-isatty v0.0.19 // indirect
	github.com/microcosm-cc/bluemonday v1.0.21 // indirect
	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
	github.com/modern-go/reflect2 v1.0.2 // indirect
	github.com/patrickmn/go-cache v2.1.0+incompatible // indirect
	github.com/pelletier/go-toml/v2 v2.0.8 // indirect
	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
	github.com/segmentfault/pacman/contrib/cache/memory v0.0.0-20230822083413-c0075a2d401f // indirect
	github.com/segmentfault/pacman/contrib/i18n v0.0.0-20230516093754-b76aef1c1150 // indirect
	github.com/syndtr/goleveldb v1.0.0 // indirect
	github.com/tidwall/match v1.1.1 // indirect
	github.com/tidwall/pretty v1.2.0 // indirect
	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
	github.com/ugorji/go/codec v1.2.11 // indirect
	github.com/yuin/goldmark v1.4.13 // indirect
	golang.org/x/arch v0.3.0 // indirect
	golang.org/x/crypto v0.13.0 // indirect
	golang.org/x/image v0.1.0 // indirect
	golang.org/x/mod v0.12.0 // indirect
	golang.org/x/net v0.15.0 // indirect
	golang.org/x/sys v0.12.0 // indirect
	golang.org/x/text v0.13.0 // indirect
	golang.org/x/tools v0.13.0 // indirect
	google.golang.org/appengine v1.6.7 // indirect
	google.golang.org/protobuf v1.30.0 // indirect
	gopkg.in/yaml.v2 v2.4.0 // indirect
	gopkg.in/yaml.v3 v3.0.1 // indirect
	lukechampine.com/uint128 v1.2.0 // indirect
	modernc.org/cc/v3 v3.40.0 // indirect
	modernc.org/ccgo/v3 v3.16.13 // indirect
	modernc.org/libc v1.22.5 // indirect
	modernc.org/mathutil v1.5.0 // indirect
	modernc.org/memory v1.5.0 // indirect
	modernc.org/opt v0.1.3 // indirect
	modernc.org/sqlite v1.24.0 // indirect
	modernc.org/strutil v1.1.3 // indirect
	modernc.org/token v1.0.1 // indirect
	sigs.k8s.io/yaml v1.3.0 // indirect
	xorm.io/builder v0.3.12 // indirect
	xorm.io/xorm v1.3.2 // indirect
)

LinkinStars · Answer 5 · Mon Apr 22 2024 10:20:25 GMT+0800 (China Standard Time)

@micol92 OK, maybe I found the problem.

https://github.com/micol92/jw-incubator-answer-plugins/blob/50b0e6972341648e271a6169c5cb2ebaa9f9d77e/connector-basic/basic.go#L32C3-L32C66

You should import "github.com/micol92/jw-incubator-answer-plugins/connector-basic/i18n" not "github.com/apache/incubator-answer-plugins/connector-basic/i18n"

LinkinStars · Answer 6 · Mon Apr 22 2024 12:03:33 GMT+0800 (China Standard Time)

RUN apk --no-cache add build-base git bash nodejs npm go && npm install -g pnpm

BTW, using npm install -g pnpm@8.9.2 to avoid some of other problems.

micol92 · Answer 7 · Mon Apr 22 2024 14:33:43 GMT+0800 (China Standard Time)

Thank you for your prompt update.

I've modified "basic.go" file you mentioned, but it failed. Please review the error log below.
And also ran the "npm install -g pnpm@8.9.2".

By any chance, could you share a docker image that reflects the source code I'm curious about?
This is probably the best option for quick debugging.

============error log occurred while running docker build.
go: finding module for package github.com/apache/incubator-answer/cmd
go: downloading github.com/micol92/jw-incubator-answer-plugins/connector-basic v1.2.6
go: downloading github.com/apache/incubator-answer-plugins/storage-s3 v1.2.6
go: downloading github.com/micol92/jw-incubator-answer-plugins v1.2.1
go: downloading github.com/apache/incubator-answer-plugins v1.2.1
go: downloading github.com/apache/incubator-answer v1.3.0
go: found github.com/apache/incubator-answer-plugins/storage-s3 in github.com/apache/incubator-answer-plugins/storage-s3 v1.2.6
go: found github.com/apache/incubator-answer/cmd in github.com/apache/incubator-answer v1.3.0
go: found github.com/micol92/jw-incubator-answer-plugins/connector-basic in github.com/micol92/jw-incubator-answer-plugins/connector-basic v1.2.6
go: answer imports
github.com/micol92/jw-incubator-answer-plugins/connector-basic: github.com/micol92/jw-incubator-answer-plugins/connector-basic@v1.2.6: parsing go.mod:
module declares its path as: github.com/apache/incubator-answer-plugins/connector-basic
but was required as: github.com/micol92/jw-incubator-answer-plugins/connector-basic
build failed exit status 1

LinkinStars · Answer 8 · Mon Apr 22 2024 15:05:23 GMT+0800 (China Standard Time)

@micol92 I make a PR to fix your compile error. micol92/jw-incubator-answer-plugins#1

After merge it, add new tag and build with new tag version.

$ git pull
$ git tag connector-basic/v1.2.7
$ git push origin connector-basic/v1.2.7

FROM apache/answer as answer-builder

FROM golang:1.19-alpine AS golang-builder

COPY --from=answer-builder /usr/bin/answer /usr/bin/answer

RUN apk --no-cache add \
    build-base git bash nodejs npm go && \
    npm install -g pnpm@8.9.2

RUN answer build \
    --with github.com/micol92/jw-incubator-answer-plugins/connector-basic@1.2.7 \
    --output /usr/bin/new_answer

FROM alpine
LABEL maintainer="linkinstar@apache.org"

ARG TIMEZONE
ENV TIMEZONE=${TIMEZONE:-"Asia/Shanghai"}

RUN apk update \
    && apk --no-cache add \
        bash \
        ca-certificates \
        curl \
        dumb-init \
        gettext \
        openssh \
        sqlite \
        gnupg \
        tzdata \
    && ln -sf /usr/share/zoneinfo/${TIMEZONE} /etc/localtime \
    && echo "${TIMEZONE}" > /etc/timezone

COPY --from=golang-builder /usr/bin/new_answer /usr/bin/answer
COPY --from=answer-builder /data /data
COPY --from=answer-builder /entrypoint.sh /entrypoint.sh
RUN chmod 755 /entrypoint.sh

VOLUME /data
EXPOSE 80
ENTRYPOINT ["/entrypoint.sh"]

micol92 · Answer 9 · Tue Apr 23 2024 22:09:13 GMT+0800 (China Standard Time)

I'm very appreciate with your help.
The added logs gave me a hint and I fixed the issue: the root cause was an invalid User JSON URL.
Next, I would like to change database setting from SQLite to PostgreSQL, and also apply AWS S3 and ElasticSearch plug-ins.
One last question is that Do I have to redeploy docker image to change database setting ? I could not find out the menu to change database setting after deployment.
Once I have your answer, let me close this issue. Thanks again.

micol92 · Answer 10 · Tue Apr 23 2024 22:12:16 GMT+0800 (China Standard Time)

As an additional question, if I build the AWS S3 plugin, will I be able to add attachments to my questions? And if I add the Elastic search plugin, will I be able to perform full-text search on attachments?

LinkinStars · Answer 11 · Tue Apr 23 2024 23:07:59 GMT+0800 (China Standard Time)

I'm very appreciate with your help. The added logs gave me a hint and I fixed the issue: the root cause was an invalid User JSON URL. Next, I would like to change database setting from SQLite to PostgreSQL, and also apply AWS S3 and ElasticSearch plug-ins. One last question is that Do I have to redeploy docker image to change database setting ? I could not find out the menu to change database setting after deployment. Once I have your answer, let me close this issue. Thanks again.

@micol92 You need redeploy again. Becasue we do not support data migration currently. FYI: https://answer.apache.org/docs/faq#what-are-the-currently-supported-database-types-is-migration-supported

LinkinStars · Answer 12 · Tue Apr 23 2024 23:11:17 GMT+0800 (China Standard Time)

As an additional question, if I build the AWS S3 plugin, will I be able to add attachments to my questions? And if I add the Elastic search plugin, will I be able to perform full-text search on attachments?

@micol92 S3 only save the images. FYI: #211

micol92 · Answer 13 · Wed Apr 24 2024 08:44:27 GMT+0800 (China Standard Time)

Thank you.