text formatting/encoding issue turning = signs to =3D

Question

text formatting/encoding issue turning = signs to =3D

spacewaffle opened this issue 8 years ago · comments

I got a number of warnings during my scrape that spat out these messages. I could be wrong but it looks like '=' signs are getting acii values inserted into the strings.

WARNING: Could not parse (and so ignoring) '<p class=3D"MsoNormal" style=3D"margin-bottom:0in;margin-bottom:.0001pt;tex='
WARNING: Could not parse (and so ignoring) 'Register for our November 29 internship event at <a href=3D"http://w='
WARNING: Could not parse (and so ignoring) '<p class=3D"MsoNormal" style=3D"margin-bottom:0in;margin-bottom:.0001pt;tex='

Ky-Anh Huynh · Answer 1 · Thu Sep 01 2016 06:53:11 GMT+0800 (China Standard Time)

@spacewaffle Would you mind sharing your group name (in case it's public)? If not, I'd like to know more logs and details (e.g, at which step do you get this warning).

Thanks

Spacewaffle · Answer 2 · Thu Sep 01 2016 07:02:23 GMT+0800 (China Standard Time)

Unfortunately the group isn't public, but I'm trying to import groups data into discourse. here's some more logging.

/ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/utilities.rb:239:in to_crlf': Interrupt /ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/message.rb:1998:inraw_source='
/ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/message.rb:2121:in init_with_string' /ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/message.rb:129:ininitialize'
/.rvm/gems/ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/mail.rb:51:in new' /.rvm/gems/ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/mail.rb:51:innew'
/.rvm/gems/ruby-2.0.0-p648/gems/mail-2.6.4/lib/mail/mail.rb:188:in read_from_string' /sites/discourse/script/import_scripts/mbox.rb:86:inblock in all_messages'
/sites/discourse/script/import_scripts/mbox.rb:68:in each' /sites/discourse/script/import_scripts/mbox.rb:68:ineach_with_index'
/sites/discourse/script/import_scripts/mbox.rb:68:in all_messages' /sites/discourse/script/import_scripts/mbox.rb:190:increate_email_indices'
/sites/discourse/script/import_scripts/mbox.rb:23:in execute' from googlegroups.rb:49:inexecute'
/sites/discourse/script/import_scripts/base.rb:45:in perform' from googlegroups.rb:92:in

''

This is part of a project migrating google groups to Discourse. I'm following the instructions here:
https://meta.discourse.org/t/migration-of-google-groups-to-discourse/48012

Ky-Anh Huynh · Answer 3 · Thu Sep 01 2016 07:14:32 GMT+0800 (China Standard Time)

I'm not author of the mbox.rb. Where do you get that script? There is probably a problem with encoding processing in this script.

Spacewaffle · Answer 4 · Thu Sep 01 2016 07:22:31 GMT+0800 (China Standard Time)

I added the link to the script in the bottom of my previous post, but it's here for you to look at. Your scraper is a dependency so I figured maybe it's to do with the scraper and not the script.
https://meta.discourse.org/t/migration-of-google-groups-to-discourse/48012

link to repo:
https://github.com/pacharanero/discourse/blob/master/script/import_scripts/mbox.rb

Ky-Anh Huynh · Answer 5 · Thu Sep 01 2016 09:31:34 GMT+0800 (China Standard Time)

Thank you @spacewaffle . How about using encoding instruction?

Current version

require 'sqlite3'
require File.expand_path(File.dirname(__FILE__) + "/base.rb")

New version / instruction

#!/usr/bin/env ruby
# encoding: utf-8

require 'sqlite3'
require File.expand_path(File.dirname(__FILE__) + "/base.rb")

Spacewaffle · Answer 6 · Fri Dec 30 2016 01:13:45 GMT+0800 (China Standard Time)

I believe I fixed this issue by upgrading our ruby version to 2.3 which does utf-8 by default

Ky-Anh Huynh · Answer 7 · Fri Dec 30 2016 07:21:25 GMT+0800 (China Standard Time)

Great to hear that. Thanks for your feedback.