Got a TypeError: Cannot read property 'rest' of undefined when detecting a YAML file

[Marco Ferrari]

Describe the bug

I get the following error:

Start detection for source id=/tmp/lint/.automation/test/ansible/ghe-initialize/defaults/main.yml format=yaml
TypeError: Cannot read property 'rest' of undefined
    at Object.tokenize (/node_modules/reprism/lib/core.js:406:22)
    at tokenize (/node_modules/@jscpd/tokenizer/dist/tokenize.js:107:21)
    at Object.createTokenMapBasedOnCode (/node_modules/@jscpd/tokenizer/dist/tokenize.js:116:20)
    at Tokenizer.generateMaps (/node_modules/@jscpd/tokenizer/dist/index.js:12:27)
    at Detector.<anonymous> (/node_modules/@jscpd/core/dist/detector.js:33:46)
    at Generator.next (<anonymous>)
    at /node_modules/@jscpd/core/dist/detector.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/node_modules/@jscpd/core/dist/detector.js:4:12)
    at Detector.detect (/node_modules/@jscpd/core/dist/detector.js:32:16)

To Reproduce
Steps to reproduce the behavior:

1. git clone [e-mail]:github/super-linter.git
2. cd super-linter
3. jscpd .

Expected behavior

jscpd should not return an error when detecting files.

Desktop (please complete the following information):

• OS: [e.g. iOS] Ubuntu Linux
• OS Version [e.g. 22]: 18.04
• NodeJS Version [e.g. 9, 10, 11]: v14.5.0
• jscpd version [e.g. 0.6.x, 1.0.x, 2.0.3]: 3.3.21

Additional context

Contents of the file that causes the error (/tmp/lint/.automation/test/ansible/ghe-initialize/defaults/main.yml):

---
##########################################
##########################################
## Standard Variables for GHE Configure ##
##########################################
##########################################
## These variables will be the defaults. If you want to override them,
## then change them in 'vars/main.yml' instead of here
github_admin_password: U53r1234
github_initial_user_password: U53r1234
github_host: github.service
github_admin_port: 8443

#####################
# Default Rate Vars #
#####################
api_rate_limiting_enabled: "true"
api_rate_limiting_unauthenticated_rate_limit: "60"
api_rate_limiting_default_rate_limit: "5000"
api_rate_limiting_search_unauthenticated_rate_limit: "10"
api_rate_limiting_search_default_rate_limit: "30"
api_rate_limiting_lfs_unauthenticated_rate_limit: "100"
api_rate_limiting_lfs_default_rate_limit: "3000"
api_rate_limiting_graphql_unauthenticated_rate_limit: "0"
api_rate_limiting_graphql_default_rate_limit: "5000"

######################
# Default Abuse Vars #
######################
abuse_rate_limiting_enabled: "true"
abuse_rate_limiting_requests_per_minute: "900"
abuse_rate_limiting_cpu_millis_per_minute: "90000"
abuse_rate_limiting_search_cpu_millis_per_minute: "7500"

####################
# Default Cas Vars #
####################
cas_url: "null"

#########################
# Default Collectd Vars #
#########################
collectd_enabled: "true"
collectd_server: "metrics.service"
collectd_port: "25826"
collectd_encryption: "null"
collectd_username: "null"
collectd_password: "null"

#####################
# Default Core Vars #
#####################
core_private_mode: "true"
core_public_pages: "false"
core_subdomain_isolation: "false"
core_signup_enabled: "false"
core_github_hostname: "null"
core_http_proxy: "null"
core_http_noproxy: "null"
core_builtin_auth_fallback: "false"
core_expire_sessions: "false"
core_package_version: "null"

#######################
# Default GitHub Vars #
#######################
github_ssl_enabled: "true"
github_ssl_tls_mode: "tlsv12"
github_ssl_cert: "null"
github_ssl_key: "null"

#########################
# Default Governor Vars #
#########################
governor_quotas_enabled: "false"
governor_limit_user: "null"
governor_limit_network: "null"

#####################
# Default LDAP Vars #
#####################
ldap_host: "null"
ldap_port: "389"
ldap_method: "None"
ldap_base_dn: "dc=demo,dc=github,dc=local"
ldap_bind_dn: "cn=admin,dc=demo,dc=github,dc=local"
ldap_password: "U53r1234"
ldap_user_groups: "null"
ldap_admin_group: "Autobots"
ldap_user_sync_emails: "true"
ldap_user_sync_keys: "false"
ldap_user_sync_gpg_keys: "false"
ldap_user_sync_interval: "1"
ldap_team_sync_interval: "1"
ldap_sync_enabled: "true"
ldap_profile_uid: "uid"
ldap_profile_name: "displayName"
ldap_profile_mail: "mail"
ldap_profile_key: "null"
ldap_profile_gpg_key: "null"

#############################
# Default Loadbalancer Vars #
#############################
loadbalancer_http_forward: "false"
loadbalancer_proxy_protocol: "false"

########################
# Default Mapping Vars #
########################
mapping_enabled: "false"
mapping_tileserver: "null"
mapping_basemap: "null"
mapping_token: "null"

####################
# Default NTP Vars #
####################
ntp_primary_server: "0.ubuntu.pool.ntp.org"
ntp_secondary_server: "1.ubuntu.pool.ntp.org"

##########################
# Default GHE Pages Vars #
##########################
pages_enabled: "true"

#####################
# Default SAML Vars #
#####################
saml_sso_url: "null"
saml_certificate: "null"
saml_certificate_path: "/data/user/common/idp.crt"
saml_issuer: "null"
saml_name_id_format: "persistent"
saml_idp_initiated_sso: "false"
saml_disable_admin_demote: "false"
saml_signature_method: "rsa-sha256"
saml_digest_method: "sha256"
saml_username_attribute: "null"
saml_full_name_attribute: "full_name"
saml_emails_attribute: "emails"
saml_ssh_keys_attribute: "public_keys"
saml_gpg_keys_attribute: "gpg_keys"

#####################
# Default SMTP Vars #
#####################
smtp_enabled: "false"
smtp_address: "null"
smtp_authentication: "null"
smtp_port: "0"
smtp_domain: "null"
smtp_username: "null"
smtp_user_name: "null"
smtp_password: "null"
smtp_support_address: [e-mail]
smtp_support_address_type: email
smtp_noreply_address: "[e-mail]"
smtp_discard_to_noreply_address: "false"

#####################
# Default SNMP Vars #
#####################
snmp_enabled: "true"
snmp_version: "2"
snmp_community: "public"
snmp_users: "null"

#######################
# Default Syslog Vars #
#######################
syslog_enabled: "false"
syslog_server: "null"
syslog_protocol_name: "udp"
syslog_tls_enabled: "false"
syslog_cert: "null"

#######################
# Default Splunk Vars #
#######################
splunk_host: splunk.service
splunk_port: 9997

[Andrey Kucherenko]

thank you for the report, will fix

[Andrey Kucherenko]

fixed at jscpd@3.3.23