After some time mount disappears
Dmitry-Filippov-Rival opened this issue · comments
Hi, I am using this repository as a reference in my project. I have a pod with an s3 mount running an FFMPEG microservice that reads a file from S3 and then uploads the transcoded HLS stream. On small files this works just fine, but when the files are big and take more than 5 minutes to transcode, at some point the transcoder pod fails liveness check and gets restarted, losing the transcoding progress. Do you know what the issue might be? Here are my YAML files for reference:
apiVersion: apps/v1
kind: Deployment
metadata:
name: rival-transcoder-deployment
labels:
app: rival-transcoder
spec:
replicas: 1
selector:
matchLabels:
app: rival-transcoder
template:
metadata:
labels:
app: rival-transcoder
spec:
serviceAccountName: rival-server-service-account
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: dedicated
operator: NotIn
values:
- infra
containers:
- name: rival-transcoder
image: our trancoder image
securityContext:
privileged: true
volumeMounts:
- name: mntdatas3fs
mountPath: /var/s3
mountPropagation: Bidirectional
livenessProbe:
exec:
command: ["ls", "/var/s3"]
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 1
resources:
requests:
memory: "24Gi"
cpu: "1"
limits:
memory: "24Gi"
cpu: "4"
ports:
- containerPort: 1050
imagePullPolicy: Always
env:
- name: API_URL
value: {{ .Values.api_url }}
- name: AWS_S3_BUCKET
valueFrom:
secretKeyRef:
name: s3-secret
key: s3-bucket
- name: AWS_REGION
value: {{ .Values.aws_region }}
- name: RABBITMQ_HOST
valueFrom:
secretKeyRef:
name: rabbitmq-default-user
key: host
- name: RABBITMQ_PORT
valueFrom:
secretKeyRef:
name: rabbitmq-default-user
key: port
- name: RABBITMQ_USER
valueFrom:
secretKeyRef:
name: rabbitmq-default-user
key: username
- name: RABBITMQ_PASSWORD
valueFrom:
secretKeyRef:
name: rabbitmq-default-user
key: password
- name: env
value: "{{ .Values.environment }}"
volumes:
- name: mntdatas3fs
hostPath:
path: /mnt/data-s3-fs/root
imagePullSecrets:
- name: ghcr-login-secret
---
apiVersion: v1
kind: Service
metadata:
name: rival-transcoder-service
labels:
run: rival-transcoder
spec:
type: ClusterIP
selector:
app: rival-transcoder
ports:
- protocol: TCP
port: 1050
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: s3-provider
labels:
app: s3-provider
spec:
selector:
matchLabels:
app: s3-provider
template:
metadata:
labels:
app: s3-provider
spec:
initContainers:
- name: init-myservice
image: bash
command: ['bash', '-c', 'umount -l /mnt/data-s3-fs/root ; true']
securityContext:
privileged: true
capabilities:
add:
- SYS_ADMIN
envFrom:
- secretRef:
name: s3-credentials
volumeMounts:
- name: devfuse
mountPath: /dev/fuse
- name: mntdatas3fs-init
mountPath: /mnt
mountPropagation: Bidirectional
containers:
- name: s3fuse
image: ghcr.io/rival-xr/rival-s3-mount:1.0
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command: ["/bin/sh","-c","umount -f /var/s3/root"]
securityContext:
privileged: true
capabilities:
add:
- SYS_ADMIN
envFrom:
- secretRef:
name: s3-credentials
env:
- name: MNT_POINT
value: /var/s3/root
volumeMounts:
- name: devfuse
mountPath: /dev/fuse
- name: mntdatas3fs
mountPath: /var/s3/root
mountPropagation: Bidirectional
volumes:
- name: devfuse
hostPath:
path: /dev/fuse
- name: mntdatas3fs
hostPath:
type: DirectoryOrCreate
path: /mnt/data-s3-fs/root
- name: mntdatas3fs-init
hostPath:
type: DirectoryOrCreate
path: /mnt
imagePullSecrets:
- name: ghcr-login-secret
Docker image is built as per instructions in this repo.
Here are the logs of s3 provider pod if it helps. It seems like the issue is the "flush" at the end of these logs. That seems to temporaly coinside with the probe fail.
2024-06-03T15:39:25.141Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:25.213Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:25.257Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/4/] [] []
2024-06-03T15:39:25.257Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:25.282Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:25.289Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/1]
2024-06-03T15:39:25.294Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/123456]
2024-06-03T15:39:25.317Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/2]
2024-06-03T15:39:25.329Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/aba123]
2024-06-03T15:39:25.329Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/aba123/]
2024-06-03T15:39:25.329Z [INF] curl.cpp:HeadRequest(3299): [tpath=/aba123]
2024-06-03T15:39:25.329Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/aba123][bpath=][save=][sseckeypos=18446744073709551615]
2024-06-03T15:39:25.329Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/aba123
2024-06-03T15:39:25.337Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/aba123
2024-06-03T15:39:25.337Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/aba123] [] []
2024-06-03T15:39:25.337Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:25.413Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:25.413Z [INF] curl.cpp:HeadRequest(3299): [tpath=/aba123/]
2024-06-03T15:39:25.413Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/aba123/][bpath=][save=][sseckeypos=18446744073709551615]
2024-06-03T15:39:25.413Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/aba123/
2024-06-03T15:39:25.413Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/aba123/
2024-06-03T15:39:25.414Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/aba123/] [] []
2024-06-03T15:39:25.414Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:25.529Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:25.529Z [INF] s3fs.cpp:list_bucket(3434): [path=/aba123]
2024-06-03T15:39:25.529Z [INF] curl.cpp:ListBucketRequest(3749): [tpath=/aba123]
2024-06-03T15:39:25.530Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing?delimiter=/&max-keys=2&prefix=aba123/
2024-06-03T15:39:25.530Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=aba123/
NODEID: 17
unique: 1788328, success, outsize: 144
unique: 1788330, opcode: LOOKUP (1), nodeid: 1, insize: 50, pid: 0
LOOKUP /abccde239
getattr /abccde239
NODEID: 15
unique: 1788330, success, outsize: 144
unique: 1788332, opcode: LOOKUP (1), nodeid: 1, insize: 45, pid: 0
LOOKUP /h264
getattr /h264
2024-06-03T15:39:25.530Z [INF] curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=aba123/] []
2024-06-03T15:39:25.530Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:25.669Z [INF] curl.cpp:RequestPerform(2520): HTTP response code 200
2024-06-03T15:39:25.670Z [ERR] s3fs_xml.cpp:get_base_exp(114): marker_xp->nodesetval is empty.
2024-06-03T15:39:25.670Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/aba123/]
2024-06-03T15:39:25.745Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/abccde239]
2024-06-03T15:39:25.746Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/h264]
2024-06-03T15:39:25.746Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/h264/]
2024-06-03T15:39:25.746Z [INF] curl.cpp:HeadRequest(3299): [tpath=/h264]
2024-06-03T15:39:25.777Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/h264][bpath=][save=][sseckeypos=18446744073709551615]
2024-06-03T15:39:25.777Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/h264
2024-06-03T15:39:25.797Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/h264
2024-06-03T15:39:25.797Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/h264] [] []
2024-06-03T15:39:25.797Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:25.849Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:25.849Z [INF] curl.cpp:HeadRequest(3299): [tpath=/h264/]
2024-06-03T15:39:25.849Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/h264/][bpath=][save=][sseckeypos=18446744073709551615]
2024-06-03T15:39:25.849Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/h264/
2024-06-03T15:39:25.849Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/h264/
unique: 1788333, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0
INTERRUPT: 1788332
2024-06-03T15:39:25.849Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/h264/] [] []
2024-06-03T15:39:25.849Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:26.037Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:26.037Z [INF] s3fs.cpp:list_bucket(3434): [path=/h264]
2024-06-03T15:39:26.037Z [INF] curl.cpp:ListBucketRequest(3749): [tpath=/h264]
2024-06-03T15:39:26.038Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing?delimiter=/&max-keys=2&prefix=h264/
2024-06-03T15:39:26.038Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com?delimiter=/&max-keys=2&prefix=h264/
2024-06-03T15:39:26.038Z [INF] curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=2&prefix=h264/] []
2024-06-03T15:39:26.038Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
NODEID: 9
unique: 1788332, success, outsize: 144
unique: 1788334, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
unique: 1788334, success, outsize: 16
unique: 1788336, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 38338560 flags: 0x8000
unique: 1788338, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 38469632 flags: 0x8000
read[2] 131072 bytes from 38338560
unique: 1788336, success, outsize: 131088
unique: 1788340, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 38600704 flags: 0x8000
read[2] 131072 bytes from 38600704
unique: 1788340, success, outsize: 131088
read[2] 131072 bytes from 38469632
unique: 1788338, success, outsize: 131088
unique: 1788342, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 38731776 flags: 0x8000
read[2] 131072 bytes from 38731776
unique: 1788342, success, outsize: 131088
unique: 1788344, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 38862848 flags: 0x8000
unique: 1788346, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 38993920 flags: 0x8000
read[2] 131072 bytes from 38862848
unique: 1788344, success, outsize: 131088
unique: 1788348, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39124992 flags: 0x8000
read[2] 131072 bytes from 39124992
unique: 1788348, success, outsize: 131088
read[2] 131072 bytes from 38993920
unique: 1788346, success, outsize: 131088
unique: 1788350, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39256064 flags: 0x8000
read[2] 131072 bytes from 39256064
unique: 1788350, success, outsize: 131088
unique: 1788354, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39518208 flags: 0x8000
read[2] 131072 bytes from 39518208
unique: 1788354, success, outsize: 131088
unique: 1788352, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39387136 flags: 0x8000
read[2] 131072 bytes from 39387136
unique: 1788352, success, outsize: 131088
unique: 1788356, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
unique: 1788356, success, outsize: 120
unique: 1788358, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 0
opendir flags: 0x18000 /
opendir[0] flags: 0x18000 /
unique: 1788358, success, outsize: 32
unique: 1788360, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 0
readdir[0] from 0
2024-06-03T15:39:26.214Z [INF] curl.cpp:RequestPerform(2520): HTTP response code 200
2024-06-03T15:39:26.215Z [ERR] s3fs_xml.cpp:get_base_exp(114): marker_xp->nodesetval is empty.
2024-06-03T15:39:26.215Z [INF] cache.cpp:AddStat(342): add stat cache entry[path=/h264/]
2024-06-03T15:39:29.985Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/]
2024-06-03T15:39:30.109Z [INF] s3fs.cpp:s3fs_opendir(3139): [path=/][flags=0x18000]
2024-06-03T15:39:30.205Z [INF] s3fs.cpp:s3fs_readdir(3390): [path=/]
2024-06-03T15:39:30.205Z [INF] s3fs.cpp:list_bucket(3434): [path=/]
2024-06-03T15:39:30.205Z [INF] curl.cpp:ListBucketRequest(3749): [tpath=/]
2024-06-03T15:39:30.205Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing?delimiter=/&max-keys=1000&prefix=
2024-06-03T15:39:30.205Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=
2024-06-03T15:39:30.205Z [INF] curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
2024-06-03T15:39:30.205Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:30.293Z [INF] curl.cpp:RequestPerform(2520): HTTP response code 200
2024-06-03T15:39:30.294Z [INF] s3fs.cpp:readdir_multi_head(3251): [path=/][list=0]
2024-06-03T15:39:30.294Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/3/][bpath=3/][save=/3/][sseckeypos=18446744073709551615]
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/3/
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/3/
2024-06-03T15:39:30.294Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/4/][bpath=4/][save=/4/][sseckeypos=18446744073709551615]
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/4/
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/4/
2024-06-03T15:39:30.294Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/h265/]
2024-06-03T15:39:30.294Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/h265/][bpath=h265/][save=/h265/][sseckeypos=18446744073709551615]
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/h265/
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/h265/
2024-06-03T15:39:30.294Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/hanger-h264/]
2024-06-03T15:39:30.294Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/hanger-h264/][bpath=hanger-h264/][save=/hanger-h264/][sseckeypos=18446744073709551615]
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/hanger-h264/
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/hanger-h264/
2024-06-03T15:39:30.294Z [INF] cache.cpp:DelStat(596): delete stat cache entry[path=/hanger/]
2024-06-03T15:39:30.294Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/hanger/][bpath=hanger/][save=/hanger/][sseckeypos=18446744073709551615]
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/hanger/
2024-06-03T15:39:30.294Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/hanger/
2024-06-03T15:39:30.341Z [INF] curl_multi.cpp:Request(324): [count=5]
2024-06-03T15:39:30.345Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/4/] [] []
2024-06-03T15:39:30.345Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:30.429Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/h265/] [] []
2024-06-03T15:39:30.429Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
unique: 1788361, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0
INTERRUPT: 1788360
2024-06-03T15:39:30.429Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:30.625Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/hanger-h264/] [] []
2024-06-03T15:39:30.625Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:30.713Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:31.215Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/hanger/] [] []
2024-06-03T15:39:31.215Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:31.216Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/3/] [] []
2024-06-03T15:39:31.216Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:31.249Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
unique: 1788360, success, outsize: 480
unique: 1788362, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
unique: 1788362, success, outsize: 16
unique: 1788364, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
unique: 1788364, success, outsize: 120
unique: 1788366, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 0
opendir flags: 0x18000 /
opendir[0] flags: 0x18000 /
unique: 1788366, success, outsize: 32
unique: 1788368, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 0
readdir[0] from 0
2024-06-03T15:39:35.813Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:36.046Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:36.649Z [INF] s3fs.cpp:s3fs_getattr(1026): [path=/]
2024-06-03T15:39:36.665Z [INF] s3fs.cpp:s3fs_opendir(3139): [path=/][flags=0x18000]
2024-06-03T15:39:36.681Z [INF] s3fs.cpp:s3fs_readdir(3390): [path=/]
2024-06-03T15:39:36.681Z [INF] s3fs.cpp:list_bucket(3434): [path=/]
2024-06-03T15:39:36.681Z [INF] curl.cpp:ListBucketRequest(3749): [tpath=/]
2024-06-03T15:39:36.681Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing?delimiter=/&max-keys=1000&prefix=
2024-06-03T15:39:36.681Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com?delimiter=/&max-keys=1000&prefix=
2024-06-03T15:39:36.681Z [INF] curl.cpp:insertV4Headers(2892): computing signature [GET] [/] [delimiter=/&max-keys=1000&prefix=] []
2024-06-03T15:39:36.681Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:36.817Z [INF] curl.cpp:RequestPerform(2520): HTTP response code 200
2024-06-03T15:39:36.817Z [INF] s3fs.cpp:readdir_multi_head(3251): [path=/][list=0]
2024-06-03T15:39:36.817Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/3/][bpath=3/][save=/3/][sseckeypos=18446744073709551615]
2024-06-03T15:39:36.817Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/3/
2024-06-03T15:39:36.817Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/3/
2024-06-03T15:39:36.817Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/4/][bpath=4/][save=/4/][sseckeypos=18446744073709551615]
2024-06-03T15:39:36.817Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/4/
2024-06-03T15:39:36.817Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/4/
2024-06-03T15:39:36.817Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/h265/][bpath=h265/][save=/h265/][sseckeypos=18446744073709551615]
2024-06-03T15:39:36.817Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/h265/
2024-06-03T15:39:36.817Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/h265/
2024-06-03T15:39:36.818Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/hanger-h264/][bpath=hanger-h264/][save=/hanger-h264/][sseckeypos=18446744073709551615]
2024-06-03T15:39:36.818Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/hanger-h264/
2024-06-03T15:39:36.818Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/hanger-h264/
2024-06-03T15:39:36.818Z [INF] curl.cpp:PreHeadRequest(3259): [tpath=/hanger/][bpath=hanger/][save=/hanger/][sseckeypos=18446744073709551615]
2024-06-03T15:39:36.818Z [INF] curl_util.cpp:prepare_url(257): URL is https://s3.amazonaws.com/rival-hls-testing/hanger/
2024-06-03T15:39:36.818Z [INF] curl_util.cpp:prepare_url(290): URL changed is https://rival-hls-testing.s3.amazonaws.com/hanger/
2024-06-03T15:39:36.818Z [INF] curl_multi.cpp:Request(324): [count=5]
2024-06-03T15:39:36.973Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/h265/] [] []
2024-06-03T15:39:36.973Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:36.994Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/4/] [] []
2024-06-03T15:39:36.994Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:37.033Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/3/] [] []
2024-06-03T15:39:37.033Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:37.081Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
unique: 1788369, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0
INTERRUPT: 1788368
unique: 1788368, success, outsize: 480
unique: 1788370, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
unique: 1788370, success, outsize: 16
unique: 1788372, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39649280 flags: 0x8000
read[2] 131072 bytes from 39649280
unique: 1788372, success, outsize: 131088
unique: 1788374, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39780352 flags: 0x8000
unique: 1788376, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 39911424 flags: 0x8000
read[2] 131072 bytes from 39780352
unique: 1788374, success, outsize: 131088
read[2] 131072 bytes from 39911424
unique: 1788376, success, outsize: 131088
unique: 1788378, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40042496 flags: 0x8000
read[2] 131072 bytes from 40042496
unique: 1788378, success, outsize: 131088
unique: 1788382, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40304640 flags: 0x8000
unique: 1788380, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40173568 flags: 0x8000
read[2] 131072 bytes from 40304640
unique: 1788382, success, outsize: 131088
read[2] 131072 bytes from 40173568
unique: 1788380, success, outsize: 131088
unique: 1788384, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40435712 flags: 0x8000
read[2] 131072 bytes from 40435712
unique: 1788384, success, outsize: 131088
unique: 1788388, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40697856 flags: 0x8000
read[2] 131072 bytes from 40697856
unique: 1788388, success, outsize: 131088
unique: 1788386, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40566784 flags: 0x8000
read[2] 131072 bytes from 40566784
unique: 1788386, success, outsize: 131088
unique: 1788390, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40828928 flags: 0x8000
read[2] 131072 bytes from 40828928
unique: 1788390, success, outsize: 131088
unique: 1788392, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 40960000 flags: 0x8000
unique: 1788394, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41091072 flags: 0x8000
read[2] 131072 bytes from 40960000
unique: 1788392, success, outsize: 131088
unique: 1788396, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41222144 flags: 0x8000
read[2] 131072 bytes from 41222144
unique: 1788396, success, outsize: 131088
read[2] 131072 bytes from 41091072
unique: 1788394, success, outsize: 131088
unique: 1788398, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41353216 flags: 0x8000
read[2] 131072 bytes from 41353216
unique: 1788398, success, outsize: 131088
unique: 1788400, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41484288 flags: 0x8000
unique: 1788402, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41615360 flags: 0x8000
read[2] 131072 bytes from 41484288
unique: 1788400, success, outsize: 131088
read[2] 131072 bytes from 41615360
unique: 1788402, success, outsize: 131088
unique: 1788404, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41746432 flags: 0x8000
read[2] 131072 bytes from 41746432
unique: 1788404, success, outsize: 131088
unique: 1788408, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42008576 flags: 0x8000
read[2] 131072 bytes from 42008576
unique: 1788408, success, outsize: 131088
unique: 1788406, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 41877504 flags: 0x8000
read[2] 131072 bytes from 41877504
unique: 1788406, success, outsize: 131088
unique: 1788410, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42139648 flags: 0x8000
read[2] 131072 bytes from 42139648
unique: 1788410, success, outsize: 131088
unique: 1788412, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42270720 flags: 0x8000
read[2] 131072 bytes from 42270720
unique: 1788412, success, outsize: 131088
unique: 1788416, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42532864 flags: 0x8000
read[2] 131072 bytes from 42532864
unique: 1788416, success, outsize: 131088
unique: 1788414, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42401792 flags: 0x8000
read[2] 131072 bytes from 42401792
unique: 1788414, success, outsize: 131088
unique: 1788418, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42663936 flags: 0x8000
read[2] 131072 bytes from 42663936
unique: 1788418, success, outsize: 131088
unique: 1788420, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42795008 flags: 0x8000
read[2] 131072 bytes from 42795008
unique: 1788420, success, outsize: 131088
unique: 1788422, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 42926080 flags: 0x8000
read[2] 131072 bytes from 42926080
unique: 1788422, success, outsize: 131088
unique: 1788424, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 43057152 flags: 0x8000
read[2] 131072 bytes from 43057152
unique: 1788424, success, outsize: 131088
unique: 1788426, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
unique: 1788428, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 43319296 flags: 0x8000
read[2] 131072 bytes from 43319296
unique: 1788428, success, outsize: 131088
read[2] 131072 bytes from 43188224 flags: 0x8000
read[2] 131072 bytes from 43188224
unique: 1788426, success, outsize: 131088
unique: 1788430, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 43450368 flags: 0x8000
read[2] 131072 bytes from 43450368
unique: 1788430, success, outsize: 131088
unique: 1788432, opcode: READ (15), nodeid: 16, insize: 80, pid: 0
read[2] 131072 bytes from 43581440 flags: 0x8000
read[2] 131072 bytes from 43581440
unique: 1788432, success, outsize: 131088
unique: 1788434, opcode: FLUSH (25), nodeid: 16, insize: 64, pid: 0
unique: 1788435, opcode: INTERRUPT (36), nodeid: 0, insize: 48, pid: 0
INTERRUPT: 1788434
2024-06-03T15:39:37.133Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/hanger/] [] []
2024-06-03T15:39:37.133Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:37.225Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:37.273Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:37.825Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:39:37.905Z [INF] curl.cpp:insertV4Headers(2892): computing signature [HEAD] [/hanger-h264/] [] []
2024-06-03T15:39:37.905Z [INF] curl_util.cpp:url_to_host(334): url is https://s3.amazonaws.com
2024-06-03T15:39:38.005Z [INF] curl.cpp:RequestPerform(2572): HTTP response code 404 was returned, returning ENOENT
2024-06-03T15:40:08.787Z [INF] s3fs.cpp:s3fs_flush(2984): [path=/abccde239/original.mp4][pseudo_fd=2]
flush[2]
unique: 1788434, success, outsize: 16
unique: 1788436, opcode: RELEASE (18), nodeid: 16, insize: 64, pid: 0
release[2] flags: 0x8000
unique: 1788436, success, outsize: 16
unique: 1788438, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 0
getattr /
unique: 1788438, success, outsize: 120