Giters

ScanOC / trunk-player

Trunk Player - Python Django project to play back recorded radio transmissions used on site

Home Page:http://scanoc.com

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Serious Security Vulnerabilities / Project Abandoned?

MaxwellDPS opened this issue · comments

commented

Hey @dreinhold, not to call you out but, is there any active dev still happening on this project?

There are some serious concerns I have with this app, especially if its gonna be WAN accessible.

  • Is there any plan to upgrade above Django 1 as there are a handful of CVEs
  • None of the API's are login restricted
    • granted some may want to list talk groups for anon, but if anon access is false, they should be login restricted
  • AWS S3 Has no signing option to prevent unauthorized access to audio

In the meantime Id seriously recommend no-one put this app on the internet

@dreinhold no shame if you cant maintain it, You created a great project and I love it! However Id seriously recommend you archive and label it with these warnings

Thanks
Max :)

commented

Any other CVE's to be aware of? Most of the package's / pip's needed for trunk-player can be safely upgraded on the minor version scale without having to fight about with code changes. Things like django core can only be upgraded to version 1.11.29 - granted 1.x series was EOL'd many years ago, with security / lts ending a year and a half ago.

commented

@bctrainers Im sure there is a handful on the packages side, if I have time I'll go through and check https://snyk.io/. I was thinking about forking this to a docker centric version and adding federation across systems

commented

@bctrainers These are the Vulnerable packages as of 9/11/2021

EDIT: THIS DOESNT INCLUDE SUB-DEPENDENCIES

Vulnerable packages:

commented

Yea sadly it's pretty abandoned. As my area went 100% encrypted a few years back and I have no used it in that long, it's been hard to dedicate time on something you just don't use. Always happy to see the few pull requests when they come.
I open sourced the project hopping to find others to collaborate with to drive up the developers on this to +1, but never really got one. I even did a could web training for a few people who showed interest in adding some enhancements, to try and drive some new developers in here, but never got it.

I guess enough, it's clear i'm the only one and as i'm not actively doing any development or upgrades I should just bite the bullet and call an official end to this. ;(

commented

Well if I have the time Im going to try to fork it and overhaul it a bit, Any thoughts on implmenting site federation from multiple systems?

commented

@dreinhold would you be willing to provide a breakdown of how this app works so I can rip and replace old libs with the newer Libs

I am familiar w/ Django and celery but not Daphne and asgi/Websockets; However Im sure I can figure out whats happening w/ enough time

First issue w/ Django 3 is Channels 2 depreciated Group()

commented

Yeah I think channels is the only sticking point, i think the rest is going to be minor changes.
As you can run it without channels (by just setting the JS to refresh the page which i think many people who never got channels working do anyways), I was thinking maybe the best is do the upgrade without channels as a first step then get the channels working again after.

commented

Ok, So from what I've gathered so far on Transmission.Save() send_mesg() is called and then sends a message to the corresponding channel group.

From my understanding changing it to (see below*) should be in spec with channels 2

Then as for asgi.py it needs to be closer to (see below**); where Im getting a little lost is the URL's, does it currently send WS over specific URLs and if so where are those defined?

Following that, the consumers need to be converted to a class. closer to (see below ***). Are you able to ELI5 what these functions (ws_connect, ws_receive, ws_disconnect ) are doing in radio.consumers.py

Also by chance are you able to provide any guidance on converting these consumers per https://channels.readthedocs.io/en/latest/topics/consumers.html#basic-layout

consumers.py ***

class RadioConsumer(WebsocketConsumer):
    def connect(self):
        self.
        self.room_name = self.scope['url_route']['kwargs']['room_name']
        self.room_group_name = 'chat_%s' % self.room_name

        # Join room group
        async_to_sync(self.channel_layer.group_add)(
            self.room_group_name,
            self.channel_name
        )

        self.accept()

    def disconnect(self, close_code):
        # Leave room group
        async_to_sync(self.channel_layer.group_discard)(
            self.room_group_name,
            self.channel_name
        )

    # Receive message from WebSocket
    def receive(self, text_data):
        text_data_json = json.loads(text_data)
        message = text_data_json['message']

        # Send message to room group
        async_to_sync(self.channel_layer.group_send)(
            self.room_group_name,
            {
                'type': 'chat_message',
                'message': message
            }
        )

ASGI.py **

import os
from django.conf.urls import url
from django.core.asgi import get_asgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "trunk_player.settings")
django_asgi_app = get_asgi_application()

from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter

from radio.consumers import RadioConsumer

application = ProtocolTypeRouter({
    "http": django_asgi_app,

    # WebSocket chat handler
    "websocket": AuthMiddlewareStack(
        URLRouter([
            url(r"^chat/admin/$", AdminChatConsumer.as_asgi()),
            url(r"^chat/$", PublicChatConsumer.as_asgi()),
        ])
    ),
})

send_mesg *

@receiver(post_save, sender=Transmission, dispatch_uid="send_mesg")
def send_mesg(sender, instance, **kwargs):
    from channels.layers import get_channel_layer
    channel_layer = get_channel_layer()
    #log.debug('Hit post save()')
    #log.debug('DATA %s', json.dumps(instance.as_dict()))
    #log.error('DATA %s', json.dumps(instance.as_dict()))
    tg = TalkGroup.objects.get(pk=instance.talkgroup_info.pk)
    tg.last_transmission = timezone.now()
    tg.save()
    groups = tg.scanlist_set.all()
    for g in groups:        
        async_to_sync(channel_layer.group_send)('livecall-scan-'+g.slug, {'text': json.dumps(instance.as_dict())})

        #Group('livecall-scan-'+g.slug, ).send({'text': json.dumps(instance.as_dict())})
    async_to_sync(channel_layer.group_send)('livecall-tg-' + tg.slug, {'text': json.dumps(instance.as_dict())})
    #Group('livecall-tg-' + tg.slug, ).send({'text': json.dumps(instance.as_dict())})
    
    # Send notification to default group all the time
    async_to_sync(channel_layer.group_send)('livecall-scan-default', {'text': json.dumps(instance.as_dict())})
    #Group('livecall-scan-default').send({'text': json.dumps(instance.as_dict())})
commented

The WS connections are first passed to the routing.py where if it is a connect it is passed to consumers.py to the ws_connect() method.

In that method it gets the full url path to get the tg/scan group the user is on to register the connection to that group in channels. The when a new transmission is added it looks up all the scan groups that tg is in and fires the messaged into all those groups to let the browser js know there is an updated transmission for a tg they are watching.

commented

@dreinhold Thank you! I think im starting to get a hold on whats going on, If you have time can you please review commits below, If im not mistaken should implement channels v2

Edit: btw docker-compose up --build works like a charm on that branch

https://github.com/MaxwellDPS/trunk-player/tree/74ef5acc27a3d28ae3549655552c1a04954a2a00
https://github.com/MaxwellDPS/trunk-player/tree/1170a1bfab4d8b8c2c9641ccda56ac053d4b35ba

commented

Likely has some bugs still but Channel2 is implemented on https://github.com/MaxwellDPS/trunk-player/tree/beta
image

Edit: Scan groups connect
image

commented

So after I do some transmission testing, I'll submit a pull with the upgrade to Channels v2 and that make it so Django and all other deps are more up-to date. This should resolve all of the above vulns above ^.

Then I think it should be easier to stay up on this project. and hopefully save Trunk-Player from the grave

I do plan to fork this with an emphasis on a new UI, Multi-TrunkPlayer Federation, removal of Payment features, I would love to get some advise on all of the above!

commented

You are the man, I was sad to say it had died and you just went a brought it back to life.
I took a quick look at the changes,all looked good.

Maybe start a new issue for each of the ideas. I'm not sure what the federation you are looking for, like multiple trunk player sites working together, or multiple recording sites feeding trunk player?

commented

You're the man, you did the heavy lifting on this one lol. Ill try to get that PR out Tomorrow morning!

And as for Federation I have a mix of use cases for both multiple trunk player sites working together, and multiple recording sites feeding trunk player

commented

@dreinhold Shoot, Just realized that the Django Jump from 2-3 Removes legacy python2 support which will break Pinax Stripe, Do you want me to try and fix that or PR w/o Plans and Pinax

commented

It looks like Pinax Stripe is also a project on deaths door, so no need to bring it with us.
I saw you had removed it from the installed_apps.

I am good dropping support for it (anyone actively using it we will need to find an alternate)

commented

Yeah, this is gonna have some breaking changes to the DB... Hmmm, may need to figure out a migration script for legacy users

commented

I tagged the current as "django-1.x" so once we pull in your changes if someone is using something that breaks, makes it easier to for them to get to the old.

commented

what are you seeing break in the DB.

commented

With the removal of Pinax stripe and plans it throws

Traceback (most recent call last):
  File "./manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/core/management/commands/makemigrations.py", line 88, in handle
    loader = MigrationLoader(None, ignore_no_migrations=True)
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 53, in __init__
    self.build_graph()
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/loader.py", line 259, in build_graph
    self.graph.validate_consistency()
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 195, in validate_consistency
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 195, in <listcomp>
    [n.raise_error() for n in self.node_map.values() if isinstance(n, DummyNode)]
  File "/home/max/trunk-player/venv/lib/python3.8/site-packages/django/db/migrations/graph.py", line 58, in raise_error
    raise NodeNotFoundError(self.error_message, self.key, origin=self.origin)
django.db.migrations.exceptions.NodeNotFoundError: Migration radio.0038_stripeplanmatrix dependencies reference nonexistent parent node ('pinax_stripe', '0007_auto_20170108_1202')
commented

Eww, yeah. Thinking about this.

commented

Went ahead and synced my fork of TrunkPlayer, it would appear there are many more security vulnerabilities than I thought.

https://github.com/bctrainers/trunk-player/security/dependabot

Keep in mind my fork does have numerous changes that probably break stock-TrunkPlayer that @dreinhold has made.

commented

@dreinhold editing the 0038 migration fixed the migrations
image

commented

@dreinhold Just ssubmitted PR #105 likely want to test it a bit but should be GTG. You may want to check the references to Pinax stripe

commented

@dreinhold Just submitted PR #105 likely want to test it a bit but should be GTG. You may want to check the references to Pinax stripe

Upon my initial review, things look good - but have not tested it on a machine yet. When I get a moment this afternoon/evening, I will test it on a machine to see if anything explodes ( figuratively speaking :^) ).

Are there any key points to note, or upgrade process notes from the current git version and your pending pull request?

commented

So @bctrainers I need some feedback on the migrations on a legacy system. Just spun up an new one and it works flawlessly with a clean DB. As for notes:

  • if you are using docker double check the new env some boolean 1/0 are now True/False
  • Also make sure to re do pip install -r requirements.txt
  • Otherwise it should be just a ./manage.py makemigrations && ./mange.py migrate
commented

@bctrainers another quirk you may have to nuke you venv and recreate it

commented

@dreinhold are the workers being used, because after the update you have to pass the channels that it will work... so a few problems there lol, but it seems the websockets are connecting w/o them

image

commented

The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently.

They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff.

==> runworker_13.log <==
2021-09-12 08:13:55,070 user AnonymousUser invalid ws path=/ws-calls/login/ setting up fake channel
2021-09-12 08:13:55,070 user AnonymousUser connect junk=junk client=192.168.20.199:58484
2021-09-12 08:13:55,070 User AnonymousUser Connected to channel livecall-junk-junk
2021-09-12 08:38:12,687 user AnonymousUser connect scan=hospitals client=192.168.20.199:38412
2021-09-12 08:38:12,687 User AnonymousUser Connected to channel livecall-scan-hospitals
2021-09-12 08:38:38,806 Not Found: /api_v1/login/
2021-09-12 08:42:36,306 user AnonymousUser connect scan=pd client=192.168.20.199:39220
2021-09-12 08:42:36,306 User AnonymousUser Connected to channel livecall-scan-pd
2021-09-12 09:03:04,089 user AnonymousUser connect scan=pd-fd client=192.168.20.199:51518
2021-09-12 09:03:04,089 User AnonymousUser Connected to channel livecall-scan-pd-fd
commented

So @bctrainers I need some feedback on the migrations on a legacy system. Just spun up an new one and it works flawlessly with a clean DB. As for notes:

  • if you are using docker double check the new env some boolean 1/0 are now True/False
  • Also make sure to re do pip install -r requirements.txt
  • Otherwise it should be just a ./manage.py makemigrations && ./mange.py migrate

Looks like a fairly standard process, so that's nice. I don't use docker for TrunkPlayer - It's on a virtual machine. :)

commented

The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently.

They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff.

==> runworker_13.log <==
2021-09-12 08:13:55,070 user AnonymousUser invalid ws path=/ws-calls/login/ setting up fake channel
2021-09-12 08:13:55,070 user AnonymousUser connect junk=junk client=192.168.20.199:58484
2021-09-12 08:13:55,070 User AnonymousUser Connected to channel livecall-junk-junk
2021-09-12 08:38:12,687 user AnonymousUser connect scan=hospitals client=192.168.20.199:38412
2021-09-12 08:38:12,687 User AnonymousUser Connected to channel livecall-scan-hospitals
2021-09-12 08:38:38,806 Not Found: /api_v1/login/
2021-09-12 08:42:36,306 user AnonymousUser connect scan=pd client=192.168.20.199:39220
2021-09-12 08:42:36,306 User AnonymousUser Connected to channel livecall-scan-pd
2021-09-12 09:03:04,089 user AnonymousUser connect scan=pd-fd client=192.168.20.199:51518
2021-09-12 09:03:04,089 User AnonymousUser Connected to channel livecall-scan-pd-fd

Hmm, I'm wondering if Channels 1 handled that differently then 2. Not sure how we are going to make that work tbh since the names are relative

commented

@dreinhold editing the 0038 migration fixed the migrations
image

This will cause issues for existing installations. We will need to create a new migration that deletes that column from existing tables also.

commented

The runworkers are getting usage on @dreinhold's current TrunkPlayer code via supervisor. I've 20 running on an install currently.
They're handling end-client connections (WSS and HTTPS) from daphne via the Nginx stuff.

==> runworker_13.log <==
2021-09-12 08:13:55,070 user AnonymousUser invalid ws path=/ws-calls/login/ setting up fake channel
2021-09-12 08:13:55,070 user AnonymousUser connect junk=junk client=192.168.20.199:58484
2021-09-12 08:13:55,070 User AnonymousUser Connected to channel livecall-junk-junk
2021-09-12 08:38:12,687 user AnonymousUser connect scan=hospitals client=192.168.20.199:38412
2021-09-12 08:38:12,687 User AnonymousUser Connected to channel livecall-scan-hospitals
2021-09-12 08:38:38,806 Not Found: /api_v1/login/
2021-09-12 08:42:36,306 user AnonymousUser connect scan=pd client=192.168.20.199:39220
2021-09-12 08:42:36,306 User AnonymousUser Connected to channel livecall-scan-pd
2021-09-12 09:03:04,089 user AnonymousUser connect scan=pd-fd client=192.168.20.199:51518
2021-09-12 09:03:04,089 User AnonymousUser Connected to channel livecall-scan-pd-fd

Hmm, I'm wondering if Channels 1 handled that differently then 2. Not sure how we are going to make that work tbh since the names are relative

Yes the runworkers changed from 1 to 2 (I don't fully understand what that means for us)

commented

@dreinhold editing the 0038 migration fixed the migrations
image

This will cause issues for existing installations. We will need to create a new migration that deletes that column from existing tables also.

Should this work?

# -*- coding: utf-8 -*-
# Generated by Django 1.11.17 on 2019-01-04 22:55
from __future__ import unicode_literals

from django.db import migrations, models


class Migration(migrations.Migration):

    dependencies = [
        ('radio', '0062_siteoption_copyright_notice'),
    ]

    operations = [
        migrations.RemoveField(model_name='StripePlanMatrix', name="stripe_plan")
    ]
commented

Yes the runworkers changed from 1 to 2 (I don't fully understand what that means for us)

Yeah per this

Note that runworker will only listen to the channels you pass it on the command line. If you do not include a channel, or forget to run the worker, your events will not be received and acted upon.

I think this means we are out of luck on the workers aside from maybe livecall-scan-default? If you pass * it grabs the names of all the files in the dir, so wild cards dont seem to be a thing. One thought is we have a ./mange.py module that dumps the possible options as empty files named the possible channels in a dir, but this seems clunky

commented

You're the man, you did the heavy lifting on this one lol. Ill try to get that PR out Tomorrow morning!

And as for Federation I have a mix of use cases for both multiple trunk player sites working together, and multiple recording sites feeding trunk player

I would love to feed multiple recording sites into one trunk-player site :)

This would be super useful for my site..

Thanks for bringing this project back to life @MaxwellDPS
Unfortunately my Django skills are in the "Enough to cause trouble" category..

I personally would love to see this project kept alive and willing to help where I can..

Thanks Sonic

commented

Howdy everyone,

Out of curiosity, looks like the webUI is JS based... Any react devs out there 😉👀

commented

@dreinhold so I have been running the new fork in docker for a few days w/o runworker and so far I have not noticed any issues. I'd image a site with heavy load may have some delay... Not sure how to handle yet aside from rewriting for a unicast ish channel and js logic.

May need to completely rework how the WS and JS interact, for the workers

commented

Howdy everyone,

Out of curiosity, looks like the webUI is JS based... Any react devs out there 😉👀

+1 on the idea of using react as the frontend.

commented

@dreinhold so I have been running the new fork in docker for a few days w/o runworker and so far I have not noticed any issues. I'd image a site with heavy load may have some delay... Not sure how to handle yet aside from rewriting for a unicast ish channel and js logic.

May need to completely rework how the WS and JS interact, for the workers

@dreinhold I have found away to accomplish this via a single uni-com WS channel, instead of diffrent ones we send all new messages out to that channel, then we can use JS for some magic. Then workers are useable too @bctrainers

one problem is this is a security issue for TG restriction and should be resolved but not sure how yet... Expect a PR by EOD

commented

one problem is this is a security issue for TG restriction and should be resolved but not sure how yet... Expect a PR by E

Edit: Slightly mitigated the security issue by limiting the WS info pushed to the bare minim. so in theory an and un authenticated / unauthorized user could see the TG name, ScanList Names, I can fix the unauthed user with template tags...

Cant think of a great way we can obscure scan list names and TG names w/o using dynamic channel names... aside from asymmetric crypto and client IDs, but that seems like overkill

commented

Right now the channel is just a trigger for the page to re pull the api, so you really don't need to pass/leak any data.

commented

As you saw i created a django3 branch for all this until we can show both a new install, and an upgrade will work. Then we can make it master.

We should start new issues at this point for new items/ideas, there is a Discussions thing in github now im not sure how it work but ill enable it, might be a good area where new ideas can be hashed out.

Another big part to this is updating the documentation, we have the markup stuff in docs that gets built into the readthedocs site and the wiki. Need to look those over and edit as needed, for those who don't know django but have used TP would be a big help in updating the documents.

commented

A big thanks to @MaxwellDPS for just starting the conversation and for the updated code.
I have asked him if he would willing to be a committer in the project and he has said yes, so I won't be a bottle neck for any changes going in.

I will be around, and still plan on reviewing the changes I can and jumping in where I can.

Dylan

commented

While we're on the basis of a new version of TrunkPlayer, what are our current thoughts with supporting MySQL/MariaDB in addition to PostgreSQL?

FWIW, pg13 works without much issues on TrunkPlayers current code. So that's always good.

My apologies for not replying sooner on testing, real life has been getting in the way. :)

commented

Just wanted to say this is so great to see. I hope to also get some time to play!!!!

commented

While we're on the basis of a new version of TrunkPlayer, what are our current thoughts with supporting MySQL/MariaDB in addition to PostgreSQL?

FWIW, pg13 works without much issues on TrunkPlayers current code. So that's always good.

My apologies for not replying sooner on testing, real life has been getting in the way. :)

@bctrainers @dreinhold correct me if Im wrong, but shouldn't setting the Django backend to mySQL work out of the box... may also need to install the pip package mysqlclient

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'NAME': 'trunk_player', # Database Name
        'USER': 'trunk_player_user', # Database User Name
        'PASSWORD': 'fake_password', # Database User Password
        'HOST': 'localhost',
        'PORT': '3306',
    }
}
commented

Hey all,

Just tested an upgrade of a clean install and @dreinhold the 0063_stablize.py migration crashes with a key error... I think we are ok to orphan stripe_plan, we can always write a postgresql command to clip it. If we remove that migration the upgrade is smooth, but need some brave volunteers with backups ;). TBH I broke my old site (Older Revision) but just connected the DB to the new code and its been running smooth

Upgrade Steps:

  1. Stop trunk player
  2. cd trunk-player
  3. git checkout django3
  4. pip install -r requirements.txt
  5. ./mange.py migrate
  6. Restart Trunk player
commented
commented

@kcwebby Wow, That sounds like a hell of system! 😄 So I can change the talk group fields to raw ID field, which means you'd need to know its ID in the DB but it will load a hell of a lot better, I think django has a UI even with that to help pick and loads in a separate window when needed. I can implement that as a setting in settings.py or via the env... TBH I want to try to make this as Mirco-service/cloud native as possible, so I will be putting a lot of focus towards docker deployments. I would be more than happy to help with getting docker instances stood up too as it seems to be way easier to scale and very reliable.

Also Its a hell of a learning curve but with that big of a system @kcwebby Kubernetes could really make your life easier in the long term.

commented

@kcwebby may be a bit big but it should be snappy as heck... shouldnt break anything, you can test by and follow the instructions below, If it works well, I will merge it in.

If not LMK and we can try the raw ID

EDIT trunk-player/radio/admin.py
REPLACE THIS:

class ScanListAdmin(admin.ModelAdmin):
    form = ScanListAdminForm
    save_as = True
    save_on_top = True

WITH THIS

class ScanListAdmin(admin.ModelAdmin):
     autocomplete_fields= ('talkgroups',)

image

commented

@kcwebby Pull #113 has support to fix the admin loading issue