Strange drama with the run command when running certain commands

Hey there again, I'm really not sure what's special about this particular command, but it's tripping up the run() function.

Here's my code:

import os
from invoke import task, run

@task
def forge():
    proxy = 'http://192.168.0.1:80/'
    os.environ.update({
        'http_proxy': proxy,
        'https_proxy': proxy
    })

    cmd = 'puppet module install --target-dir /var/puppet/forgey puppetlabs-motd'
    print cmd
    run(cmd)

And the output:

puppet module install --target-dir /var/puppet/forgey puppetlabs-motd
Warning: Setting manifest is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting config_version is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Notice: Preparing to install into /var/puppet/forgey ...
Notice: Created target directory /var/puppet/forgey
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
Traceback (most recent call last):
  File "./bin/invoke", line 12, in <module>
    invoke.cli.main()
  File "/root/puppet-pusher/lib/invoke/invoke/cli.py", line 438, in main
    dispatch(sys.argv)
  File "/root/puppet-pusher/lib/invoke/invoke/cli.py", line 431, in dispatch
    return executor.execute(*tasks)
  File "/root/puppet-pusher/lib/invoke/invoke/executor.py", line 100, in execute
    task=task, name=name, args=args, kwargs=kwargs, config=config
  File "/root/puppet-pusher/lib/invoke/invoke/executor.py", line 153, in _execute
    result = task(*args, **kwargs)
  File "/root/puppet-pusher/lib/invoke/invoke/tasks.py", line 111, in __call__
    result = self.body(*args, **kwargs)
  File "/root/puppet-pusher/tasks.py", line 14, in forge
    run(cmd)
  File "/root/puppet-pusher/lib/invoke/invoke/__init__.py", line 23, in run
    return Context().run(command, **kwargs)
  File "/root/puppet-pusher/lib/invoke/invoke/context.py", line 53, in run
    return runner_class(context=self).run(command, **kwargs)
  File "/root/puppet-pusher/lib/invoke/invoke/runners.py", line 262, in run
    raise ThreadException(exceptions)
invoke.exceptions.ThreadException:
Saw 1 exceptions within threads (UnicodeEncodeError):


Thread args: {'args': [<functools.partial object at 0x1c86f70>,
          <open file '<stdout>', mode 'w' at 0x7effba142150>,
          [u'\x1b[mNotice: Preparing to install into /var/puppet/forgey ...\x1b[0m\n',
           u'\x1b[mNotice: Created target directory /var/puppet/forgey\x1b[0m\n',
           u'\x1b[mNotice: Downloading from https://forgeapi.puppetlabs.com ...\x1b[0m\n',
           u'\x1b[mNotice: Installing -- do not interrupt ...\x1b[0m\n'],
          False],
 'target': <bound method Local.io of <invoke.runners.Local object at 0x1c91b90>>}

Traceback (most recent call last):

  File "/root/puppet-pusher/lib/invoke/invoke/runners.py", line 82, in run
    super(_IOThread, self).run()

  File "/usr/lib64/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)

  File "/root/puppet-pusher/lib/invoke/invoke/runners.py", line 328, in io
    output.write(data)

UnicodeEncodeError: 'ascii' codec can't encode characters in position 19-27: ordinal not in range(128)

This system is using:

Python 2.6.6
Puppet 3.6.2
Invoke (master)

Regular output of the command looks like this:

Warning: Setting manifest is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting modulepath is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Warning: Setting config_version is deprecated in puppet.conf. See http://links.puppetlabs.com/env-settings-deprecations
   (at /usr/lib/ruby/site_ruby/1.8/puppet/settings.rb:1095:in `issue_deprecations')
Notice: Preparing to install into /var/puppet/forgey ...
Notice: Created target directory /var/puppet/forgey
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/var/puppet/forgey
└── puppetlabs-motd (v1.2.0)

Any ideas?

Cheers
Fotis

I believe that it's getting confused with the └── characters actually, I'll see if I can write some code that you can use to reproduce this yourself consistently 😄

Edit: OK, here's some simple code that repos the issue 😄

# encoding: utf-8
from invoke import task, run

@task
def output_strange_stuff():
    print '└──'

@task
def breakit():
    run('invoke output_strange_stuff')

Then just run invoke breakit. An invoke invoking another invoke, kinda cool huh? 😄

I presume this is a unicode related thingy. Using ascii is not the safest option if that's what's going on under the hood because it only covers those 128 chars. Latin-1 would be the safest bet if this is fixed in the code since that covers any possible string (it supports 256 chars).

Please let me know your thoughts anyway 😄

Edit 2: I noticed there is a default_encoding parameter that can be passed into run() which I tried but it didn't seem to help unfortunately

Just one more note to confuse things further. I built Python 2.6.6 using pyenv on my Mac and can't repro the problem there. But I can reproduce it on my server which is a RHEL 6.6 server using the default Python 2.6.6 installation.

I will try to look into it soon too, but it's a bit of a headache to deal with right now :(

That is pretty strange, moreso because we actually have that exact sort of character in our integration test suite and that runs fine for me and our CI:

invoke/integration/main.py

Lines 70 to 89 in 3b9ade5

    
           class funky_characters_in_stdout: 
        
               def basic_nonstandard_characters(self): 
        
                   os.chdir('_support') 
        
                   # Crummy "doesn't explode with decode errors" test 
        
                   if WINDOWS: 
        
                       cmd = "type tree.out" 
        
                   else: 
        
                       cmd = "cat tree.out" 
        
                   run(cmd, hide='both') 
        
               def nonprinting_bytes(self): 
        
                   # Seriously non-printing characters (i.e. non UTF8) also don't 
        
                   # asplode 
        
                   run("echo '\xff'", hide='both') 
        
               def nonprinting_bytes_pty(self): 
        
                   if WINDOWS: 
        
                       return 
        
                   # PTY use adds another utf-8 decode spot which can also fail. 
        
                   run("echo '\xff'", pty=True, hide='both')

https://github.com/pyinvoke/invoke/blob/3b9ade5d7af8bf093c0f91f2b59a3d7fe904d7cb/integration/_support/tree.out

However, I dug and this looks like the same issue as #242 which remains unsolved but has a good writeup by pfmoore (who was instrumental in setting up the current encoding infrastructure).

Gonna roll this into that one. Thanks for the detailed report!

Also, I did just notice the docstring for run is wrong, the kwarg is encoding not default_encoding, ugh. So thanks for that part at least!

Thanks a lot mate 😄

Navigation Menu

Repository navigation

Strange drama with the run command when running certain commands #260

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	class funky_characters_in_stdout:
	def basic_nonstandard_characters(self):
	os.chdir('_support')
	# Crummy "doesn't explode with decode errors" test
	if WINDOWS:
	cmd = "type tree.out"
	else:
	cmd = "cat tree.out"
	run(cmd, hide='both')

	def nonprinting_bytes(self):
	# Seriously non-printing characters (i.e. non UTF8) also don't
	# asplode
	run("echo '\xff'", hide='both')

	def nonprinting_bytes_pty(self):
	if WINDOWS:
	return
	# PTY use adds another utf-8 decode spot which can also fail.
	run("echo '\xff'", pty=True, hide='both')

Strange drama with the run command when running certain commands #260

Description

Activity

fgimian commented on Jul 13, 2015

fgimian commented on Jul 14, 2015

bitprophet commented on Jul 14, 2015

bitprophet commented on Jul 14, 2015

fgimian commented on Jul 15, 2015

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions