Error on --csv

Question

Error on --csv

mugzy opened this issue 9 years ago · comments

Exporting to JSON works fine, but when I try CSV, I get the error below.

ec2-security-groups-dumper --csv

Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper==1.0', 'console_scripts', 'ec2-security-groups-dumper')()
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 280, in main
print firewall.csv
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 213, in csv
for fr in self.rules:
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 151, in rules
raise ValueError
ValueError

Any ideas?

Laurent Raufaste · Answer 1 · Wed Jul 15 2015 01:05:06 GMT+0800 (China Standard Time)

Mmmh, you might have a config we don't.
Can you try adding more debug statements to the script and see what kind of data make it fail?
If you can't let me know, I'll add those myself.

mugzy · Answer 2 · Fri Jul 17 2015 22:00:43 GMT+0800 (China Standard Time)

I am not super familiar with python, so if you could give be some debug code and tell me where to add it, I can then post the results.

Laurent Raufaste · Answer 3 · Sat Jul 18 2015 02:39:34 GMT+0800 (China Standard Time)

ok I'll take a look

Abdul Mohammed · Answer 4 · Sun Jul 19 2015 23:29:18 GMT+0800 (China Standard Time)

i have a similar problem using --csv option. The JSON option works just fine.

Laurent Raufaste · Answer 5 · Tue Jul 21 2015 09:58:55 GMT+0800 (China Standard Time)

ok, I'm not sure what's happening, but I pushed a new version on https://pypi.python.org/pypi/ec2-security-groups-dumper/ that should print out more information.

Can anyone try so I can see what type of grant you have?

Abdul Mohammed · Answer 6 · Tue Jul 21 2015 20:50:17 GMT+0800 (China Standard Time)

@lra
here is what i get. again the JSON works fine, it is the CSV

[root@ip-10-0-1-32 ~]# ec2-security-groups-dumper --csv
Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper==1.1', 'console_scripts', 'ec2-security-groups-dumper')()
File "/usr/local/lib/python2.6/site-packages/ec2_security_groups_dumper/main.py", line 281, in main
print firewall.csv
File "/usr/local/lib/python2.6/site-packages/ec2_security_groups_dumper/main.py", line 213, in csv
writer.writeheader()
AttributeError: DictWriter instance has no attribute 'writeheader'
[root@ip-10-0-1-32 ~]#

mugzy · Answer 7 · Tue Jul 21 2015 21:14:02 GMT+0800 (China Standard Time)

I get a different error, also only on csv. Security group ID has been masked.

Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper==1.1', 'console_scripts', 'ec2-security-groups-dumper')()
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 281, in main
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 214, in csv
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 152, in rules
ValueError: ('Unsupported grant:', {'group_id': u'sg-xxxxxx'})

Laurent Raufaste · Answer 8 · Wed Jul 22 2015 02:07:20 GMT+0800 (China Standard Time)

Ok those are 2 different errors.
@abdulirfan3 can you create a separate issue?
@mugzy you security group has no name. You can't create a security group without a name, how did you do it?

mugzy · Answer 9 · Thu Jul 23 2015 00:28:23 GMT+0800 (China Standard Time)

Ok, I am bringing my python skills up to date and debugging this. All of our security groups have names, but only the default groups have a names that show up in rules.grants.

I will let you know what I find out.

mugzy · Answer 10 · Thu Jul 23 2015 01:31:02 GMT+0800 (China Standard Time)

Happen to know how to pull info on a specific security group? I have tried using get_all_security_groups('sg-xxxxxx') but it always comes back saying the group does not exists even when I know it does.

I was thinking since the grant always has an ID, I could get the name by looking it up.

Abdul Mohammed · Answer 11 · Thu Jul 23 2015 01:34:58 GMT+0800 (China Standard Time)

@mugzy ,

you wanna do something like below...use the filter operation in get_all_security_group()

fts = {'vpc-id': 'vpc-1895327d', 'group-name': 'wordpress-app-SG'}
sgs = boto.ec2.connect_to_region("us-east-1", aws_access_key_id='XXXXXX', aws_secret_access_key='XXXXXX).get_all_security_groups(filters=fts)

mugzy · Answer 12 · Fri Jul 24 2015 04:39:18 GMT+0800 (China Standard Time)

Still not sure why some grants don't have names, but I did code a work around using a lookup table. I also added code for egress rules.

I forked the repo and put my changes there.

mugzy@7c14d9e

Laurent Raufaste · Answer 13 · Fri Jul 24 2015 05:26:25 GMT+0800 (China Standard Time)

u might want to send a ticket to AWS. Either it's a legit use case and they should fix their calls, either it's not and your data might be bugged.

I'd like to know their thoughts before making it official in the tool.

elias-p · Answer 14 · Wed Aug 19 2015 02:01:23 GMT+0800 (China Standard Time)

hello! thanks for this really handy tool. is there anyone figure out the --csv out put error "Unsupported grant"

Laurent Raufaste · Answer 15 · Wed Aug 19 2015 04:33:11 GMT+0800 (China Standard Time)

Nope, I need someone to ask AWS if a security group without a name is legit, or it's a bug on their side.
If it's legit, we can add support for it.

elias-p · Answer 16 · Wed Aug 19 2015 05:45:42 GMT+0800 (China Standard Time)

I can ask but when I check my SG dump from JSON all SG has name, so how I can prove that, as far as I know it's impossible. which cli spit this output to ask AWS. thanks.

Laurent Raufaste · Answer 17 · Wed Aug 19 2015 05:52:01 GMT+0800 (China Standard Time)

@elias-p can you dump here the error you get?

elias-p · Answer 18 · Wed Aug 19 2015 06:02:56 GMT+0800 (China Standard Time)

Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper==1.1', 'console_scripts', 'ec2-security-groups-dumper')()
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 281, in main
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 214, in csv
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 152, in rules
ValueError: ('Unsupported grant:', {'group_id': u'sg-927XXXXX'})

Laurent Raufaste · Answer 19 · Wed Aug 19 2015 06:18:17 GMT+0800 (China Standard Time)

Same stuff: The AWS API is returning a security group sg-927XXXXX without either a name or a CIDR.
This should never happen.
Sadly I can't just modify the code based on suppositions.

How did you manage to create a nameless sg-927XXXXX security group?

elias-p · Answer 20 · Wed Aug 19 2015 10:49:20 GMT+0800 (China Standard Time)

may be some bug but all my security group has name. here is the json out put for this SG
"grants": [id": "sg-927XXXXX"

"description": "None",
"id": "sg-927XXXX",
"name": "WEBXXXXXX",
"rules": [
{
"from_port": "XXXXXXX",
"grants": [
{
"group_id": "sg-21XXXXXXXX"
}
],
"ip_protocol": "tcp",
"to_port": "XXXXXXXXXXXXXX"
},
{
"from_port": "XXXXXXX",
"grants": [
{
"group_id": "sg-89XXXXXXXX"
}
],
"ip_protocol": "tcp",
"to_port": "XXXXXX"

Laurent Raufaste · Answer 21 · Wed Aug 19 2015 11:06:18 GMT+0800 (China Standard Time)

I think the json has been cut, can you paste a valid one?

elias-p · Answer 22 · Wed Aug 19 2015 23:30:17 GMT+0800 (China Standard Time)

"description": "None",
"id": "sg-927xxxx",
"name": "WEBXXXXX",
"rules": [
{
"from_port": "xxxx",
"grants": [
{
"group_id": "sg-XX"
}
],
"ip_protocol": "tcp",
"to_port": "xxxx"
},
{
"from_port": "xxxx",
"grants": [
{
"group_id": "sg-xxxx"
}
],
"ip_protocol": "tcp",
"to_port": "xxxx"
},
{
"from_port": "xxx",
"grants": [
{
"group_id": "sg-XXXX"
}
],
"ip_protocol": "tcp",
"to_port": "xxxx"
},
{
"from_port": "xxxx",
"grants": [
{
"group_id": "sg-xxxxxx"
}
],
"ip_protocol": "tcp",
"to_port": "xxxx"
},
{
"from_port": "xxx",
"grants": [
{
"group_id": "sg-xxxxx"
}
],
"ip_protocol": "tcp",
"to_port": "xxxxx"

Laurent Raufaste · Answer 23 · Fri Sep 04 2015 01:38:07 GMT+0800 (China Standard Time)

I finally found the problem, some VPC security groups have no name, working on a fix.

Laurent Raufaste · Answer 24 · Fri Sep 04 2015 01:58:12 GMT+0800 (China Standard Time)

fixed in 1.2

elias-p · Answer 25 · Fri Sep 04 2015 05:36:05 GMT+0800 (China Standard Time)

thanks, I have download 1.2 and tried but still I got error on esv.

Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper==1.1', 'console_scripts', 'ec2-security-groups-dumper')()
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 281, in main
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 214, in csv
File "build/bdist.linux-x86_64/egg/ec2_security_groups_dumper/main.py", line 152, in rules
ValueError: ('Unsupported grant:', {'group_id': u'sg-xxxxxxx'})

Laurent Raufaste · Answer 26 · Fri Sep 04 2015 05:54:32 GMT+0800 (China Standard Time)

You are not running 1.2

elias-p · Answer 27 · Fri Sep 04 2015 12:02:46 GMT+0800 (China Standard Time)

I have download the new zip file for 1.2 file and remove the old one and unzip the new one. not sure what other step I have missed.

Laurent Raufaste · Answer 28 · Fri Sep 04 2015 12:19:16 GMT+0800 (China Standard Time)

pip freeze|grep ec2-security-groups-dumper ?
You want to install it with pip, e.g. pip install -U ec2-security-groups-dumper

elias-p · Answer 29 · Fri Sep 04 2015 23:13:09 GMT+0800 (China Standard Time)

thanks Ira, please add step on git. still I got error.
1- pip freeze|grep ec2-security-groups-dumper
ec2-security-groups-dumper==1.2
2- ec2-security-groups-dumper --csv
Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper', 'console_scripts', 'ec2-security-groups-dumper')()
File "/usr/local/lib/python2.7/site-packages/ec2_security_groups_dumper/main.py", line 295, in main
print firewall.csv
File "/usr/local/lib/python2.7/site-packages/ec2_security_groups_dumper/main.py", line 228, in csv
for fr in self.rules:
File "/usr/local/lib/python2.7/site-packages/ec2_security_groups_dumper/main.py", line 162, in rules
rules_grants_cidr_ip=grant_row['cidr_ip'])
File "/usr/local/lib/python2.7/site-packages/ec2_security_groups_dumper/main.py", line 57, in init
assert rules_ip_protocol in (u'tcp', u'udp', u'icmp', None)
AssertionError

Laurent Raufaste · Answer 30 · Fri Sep 04 2015 23:16:32 GMT+0800 (China Standard Time)

what's the json?

elias-p · Answer 31 · Sat Sep 05 2015 01:43:25 GMT+0800 (China Standard Time)

the JSON working perfectly!!

Laurent Raufaste · Answer 32 · Sat Sep 05 2015 04:22:09 GMT+0800 (China Standard Time)

Yep, but what's in it? I need the data to know what is breaking

Marek Korpacz · Answer 33 · Mon Sep 28 2015 18:03:37 GMT+0800 (China Standard Time)

At the beginning I would like to thank you for a great script.

However, I have the same issue as elias-p, json output works fine, but when I'm using csv format, I'm getting the following error:
Traceback (most recent call last):
File "/usr/local/bin/ec2-security-groups-dumper", line 9, in
load_entry_point('ec2-security-groups-dumper==1.2', 'console_scripts', 'ec2-security-groups-dumper')()
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 298, in main
print firewall.csv
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 229, in csv
for fr in self.rules:
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 163, in rules
rules_grants_cidr_ip=grant_row['cidr_ip'])
File "/usr/local/lib/python2.7/dist-packages/ec2_security_groups_dumper/main.py", line 58, in init
assert rules_ip_protocol in (u'tcp', u'udp', u'icmp', None)
AssertionError

If someone can help me in debugging this issue. I've checked that all exported SGs have a name associated with it.

Thanks in advance

Laurent Raufaste · Answer 34 · Mon Sep 28 2015 23:33:34 GMT+0800 (China Standard Time)

you seem to be using an old version

Marek Korpacz · Answer 35 · Tue Sep 29 2015 01:50:26 GMT+0800 (China Standard Time)

I'm using version 1.2. Is that means that there is newer one ?

Laurent Raufaste · Answer 36 · Wed Sep 30 2015 00:45:37 GMT+0800 (China Standard Time)

No, then it's an unhandled use case.
Can you dump the json in a gist?