Ædifice.org Blog

Use Mac OS X's Keychain for Password Retrieval in OfflineIMAP

Mon, 01 Feb 2010 22:40:35 -0000

In my earlier post, I explained how to retrieve a password stored in a Mac OS X keychain from within Mutt. Mutt can be compiled with IMAP support, allowing for up-to-date access to your email account, but sometimes you want local copies of your IMAP folders for offline browsing or for backups. OfflineIMAP provides exactly this functionality: it synchronizes local maildir format email folders with a remote IMAP host. As with mutt, you normally can put your passwords in the configuration file, ~/.offlineimaprc, but that leaves your password in clear text in the file. Thankfully, OfflineIMAP allows you to run python code in some of its fields, and it lets you specify a file containing python code for it to source methods from.

The python source file

First, we need to make a python function that will call the security command and grab only the relevant part of its output:

import re, commands def get_keychain_pass(account=None, server=None): params = { 'security': '/usr/bin/security', 'command': 'find-internet-password', 'account': account, 'server': server } command = "%(security)s %(command)s -g -a %(account)s -s %(server)s" % params outtext = commands.getoutput(command) return re.match(r'password: "(.*)"', outtext).group(1)

I put this into ~/.mutt/offlineimap.py, although something like ~/.offlineimap.py makes sense as well. The params dict/hash makes it more legible and easy to change the command, and the command string gets run through commands.getoutput() to produce the command output. The re.match() line does the same thing as the perl regular expression matching that showed up in the muttrc file.

offlineimaprc

Next, OfflineIMAP’s RC file needs to know about this function. This can be done by adding a pythonfile directive to the [general] section:

[general] pythonfile = ~/.mutt/offlineimap.py

Finally, the get_keychain_pass() function itself needs to get called from the correct field:

[Repository gmailRemote] type = Gmail  remoteuser = someuser@gmail.com  remotepasseval = get_keychain_pass(account="someuser@gmail.com", server="imap.gmail.com")

Upon startup, OfflineIMAP will now run security and try to acquire the specified password from the keychain.

Conclusion

The above should work fine with OfflineIMAP and Mutt running at the same time. However, when I was still running Mac OS X 10.4, I ran into an annoying race condition involving security. When both programs executed security while my keychain was locked, both instances of security would open dialogs asking for my keychain password (I do not know if this problem still occurs in Mac OS X 10.5 or 10.6). My solution to this was to write a wrapper script that called security itself. It used a lockfile to let only one instance of security run at a time, and it also did the regular expression filtering for the password. If you are interested in using my script, getpassword can be downloaded here (It is written in Ruby, and it requires the ‘lockfile’ Ruby gem, so it may not be suitable for you as it currently exists).

RKHunter — Making False Positives Go Away on Gentoo Linux

Tue, 19 Jan 2010 19:27:09 -0000

Update 2018/05/06: Updated the RKHunter links to RKHunter’s homepage and the Gentoo package.

RKHunter is a tool for detecting rootkits on Unix-like systems. I run it daily on my Gentoo server, and it emails me a report about whether it updated its database of tests as well as any warnings it comes across while running its tests. Recently, Gentoo marked rkhunter-1.3.4 as stable. After I updated it, it began reporting more false positives than the older 1.2.9 (it used to only report the existence of promiscuous network interfaces, which is caused by my network confguration).

Some were due to programs in /usr/bin being shell scripts instead of binaries, while others were complaining about the fact that I did not have the latest version of GnuPG or OpenSSH (I stick to Gentoo’s stable releases, which are currently a minor version behind). Another problem was over Linux kernel modules. I built my kernel with all module support disabled, compiling in everything I want my kernel to have.

Read on to find out how I resolved the various warnings in /var/log/rkhunter.log by editing /etc/rkhunter.conf.

Fixing changed binaries

By default, RKHunter checks the properties of files in /bin, /usr/bin, etc. Whenever a package updates, its binary executables usually change as well. For example (from rkhunter.log):

[04:50:29] /bin/netstat [ Warning ] [04:50:29] Warning: The file properties have changed: [04:50:30] File: /bin/netstat [04:50:30] Current hash: 58ad996b4c822f25760fbe4bfb9904d623aeb51b [04:50:30] Stored hash : 715d86e1b178c19807a315fe339e87b3e5a12c75[04:50:31] Current inode: 16178 Stored inode: 16175 [04:50:31] Current size: 125996 Stored size: 116940 [04:50:31] Current file modification time: 1263850324 [04:50:32] Stored file modification time : 1258110183

The solution for this sort of warning is straightforward. RKHunter’s properties database needs to be updated:

# rkhunter --propupd

Whitelisting Valid Scripts

It also reports that some files have been replaced by scripts. For example (again, from rkhunter.log):

[04:51:43] /usr/bin/whatis [ Warning ] [04:51:43] Warning: The command '/usr/bin/whatis' has been replaced by a script [04:51:43] Warning: The command '/usr/bin/whatis' has been replaced by a script: /usr/bin/whatis: POSIX shell script text executable ... [04:51:48] /usr/bin/lwp-request [ Warning ] [04:51:48] Warning: The command '/usr/bin/lwp-request' has been replaced by a sc ript: /usr/bin/lwp-request: a /usr/bin/perl -w script text executable

First, I examined the programs directly to make sure that they looked okay – that is, that they are false positives and not actual positives. In order for rkhunter to not complain, I needed to either disable this test, or let the test know which files should be scripts and not binaries. To do the latter, search /etc/rkhunter.conf for ‘SCRIPTWHITELIST’ and add a line with the path for each command that you expect to be a script:

# # Allow the specified commands to be scripts. # One command per line (use multiple SCRIPTWHITELIST lines). # #SCRIPTWHITELIST=/sbin/ifup #SCRIPTWHITELIST=/sbin/ifdown #SCRIPTWHITELIST=/usr/bin/groups SCRIPTWHITELIST=/usr/bin/ldd SCRIPTWHITELIST=/usr/bin/whatis SCRIPTWHITELIST=/usr/bin/lwp-request

Next time rkhunter runs, it will no longer complain about ldd, whatis, or lwp-request being scripts.

An Expected Promiscuous Interface

[04:57:51] Performing checks on the network interfaces [04:57:51] Info: Starting test name 'promisc' [04:57:52] Checking for promiscuous interfaces [ Warning ] [04:57:52] Warning: Possible promiscuous interfaces: [04:57:52] 'ifconfig' command output: UP BROADCAST RUNNING PR OMISC MULTICAST MTU:1500 Metric:1 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 [04:57:53] 'ip' command output: eth0 [04:57:53] 'ip' command output: tap0[04:57:53]

My server runs openvpn on it, and I configured it to create a bridge between the openvpn “network” and the physical network. In the process, the network interfaces are in run promiscuous mode. The only way to get rid of this warning is to disable the ‘promisc’ test. Search for DISABLE_TESTS, and append promisc to the end of the list:

DISABLE_TESTS="suspscan hidden_procs deleted_files packet_cap_apps promisc"

Complaints of Hidden Directories

[04:58:16] Checking for hidden files and directories [ Warning ] [04:58:17] Warning: Hidden directory found: /dev/.udev

This hidden directory (prefixed with a dot hidden, not hidden from opendir/readdir function calls) is part of how udev operates. To tell rkhunter to ignore it, search for ALLOWHIDDENDIR and uncomment or add the following line:

ALLOWHIDDENDIR=/dev/.udev

Disregarding Warnings About Outdated Applications

[04:58:17] Checking application versions... [04:58:17] Info: Starting test name 'apps' ... [04:58:21] Checking version of GnuPG [ Warning ] [04:58:21] Warning: Application 'gpg', version '2.0.11', is out of date, and pos sibly a security risk. ... [04:58:25] Checking version of OpenSSH [ Warning ] [04:58:25] Warning: Application 'sshd', version '5.2p1', is out of date, and possibly a security risk.

I mostly stick to Gentoo’s stable branch. This means that sometimes applications are not at the latest version. I regularly check my system for updates, and I do not need rkhunter complaining on a daily basis that these programs are outdated. To make these warnings go away, I changed the APP_WHITELIST option:

# # Allow the following applications, or a specific version of an application, # to be whitelisted. This option is a space-separated list consisting of the # application names. If a specific version is to be whitelisted, then the # name must be followed by a colon and then the version number. # # For example: APP_WHITELIST="openssl:0.9.7d gpg" # APP_WHITELIST="gpg sshd"

My choice makes it never complain about outdated versions of gpg or sshd, although I could specify any versions I believe to be “safe.”

Disregarding the Kernel Module Checks

[04:57:31] Performing Linux specific checks[04:57:32] Info: Starting test name 'os_specific' [04:57:32] Checking loaded kernel modules [ Warning ] [04:57:32] Warning: The modules file '/proc/modules' is missing. [04:57:32] Info: Using modules pathname of '/lib/modules' [04:57:33] Checking kernel module names [ Warning ] [04:57:33] Warning: The kernel modules directory '/lib/modules' is missing or em pty.

As I mentioned above, my kernel compiles in all kernel modules it uses, and I deactivated the kernel module loading interface, so it makes sense that these are giving warnings (about there being no modules). It is possible to specify an alternative location for the kernel modules directory with the MODULES_DIR configuration directive, but I have no modules. I turned these checks off the same way I turned off the promiscuous interface check – by appending them to the list of disabled tests:

DISABLE_TESTS="suspscan hidden_procs deleted_files packet_cap_apps promisc avail_modules loaded_modules"

Conclusion

After making the above changes in the configuration file, I ran

# rkhunter --check

and it no longer found any warnings.

Use Mac OS X's Keychain for Password Retrieval in Mutt

Sun, 18 Oct 2009 21:31:04 -0000

I am a fan of Mutt, a command line email client. It is really powerful, and it is highly customizable. Configuring it can be difficult due to having so many settings, but there are quite a few decent tutorials online for learning to configure it and any related commands.

Mutt used to have to rely on other programs in order to send and receive mail on a remote server. But it has been possible for a while now to use its own built-in support for POP and IMAP for receiving mail, and SMTP for sending mail. Normally, if you did not want to have to type in your password every time you connected, or if you use several different accounts, you would have to store your password(s) in your mutt configuration files in clear text. It would be nice if security storing services like Mac OS X’s Keychain, Gnome’s Keyring, or KDE’s kWallet could be used natively within Mutt, but that is not yet the case. Under Mac OS X, however, passwords can be stored in the Keychain and accessed from the command line with the security command, and Mutt configuration files can call snippets of shell code that will get replaced with the output of the shell commands.

This article explains a basic approach on how to use the security command, and a basic way of using it in a Mutt configuration file. I may present the Ruby wrapper script that I actually use in a future article.

Setting up Keychain

First, the password needs to be added to login.keychain or another keychain file for the current user. Run /Applications/Utilities/Keychain Access.app, and select New Password Item… from the File menu. This will bring up a dialog where the account info and password can be filled in.

The 'New Password Item' dialog

The protocol and server name should be filled in using URL syntax, such as imap://imap.someserver.com. Don’t worry about whether the server uses the SSL/TLS variation of the email network protocols or not because we will configure that in Mutt based on the protocols used. The protocol and server name are needed to look up the password in the keychain database.

Next, verify that the Keychain item is correctly filled out:

The new Keychain item

`security` Command Basics

According to its man page, security is a “command line interface to keychains and Security.framework.” It provides a crude way to look up Keychain entries. Like most of Apple’s command line tools, it provides several actions that can be performed, but the one useful here is find-internet-password.

A query will look something like:

$ security find-internet-password -g -a someuser@example.com -s imap.example.com

-g makes it output the password to stderr, in addition to the normal info to stdout
-a is the account name to look up
-s is the server name to lookup

There are other options that can be used to look up a Keychain entry, but the above two are sufficient for most needs (see the man page for more options).

The first time security is run, the Mac OS will ask whether to allow security access the queried password or not. If it is allowed to do so once (in which case every time it access a given entry it will have to be granted permission once) or always, it will print something like the following:

keychain: "/Users/someuser/Library/Keychains/login.keychain" class: "teni" attributes: 0x00000007 <blob>="imap.example.com" 0x00000008 <blob>=<NULL> "acct"<blob>="someuser@example.com" "atyp"<blob>="dflt" "cdat"<timedate>=0x32303039313031393030303330305A00 "20091019000300Z\000" "crtr"<uint32>=<NULL> "cusi"<sint32>=<NULL> "desc"<blob>=<NULL> "icmt"<blob>=<NULL> "invi"<sint32>=<NULL> "mdat"<timedate>=0x32303039313031393030303330305A00 "20091019000300Z\000" "nega"<sint32>=<NULL> "path"<blob>=<NULL> "port"<uint32>=0x00000000 "prot"<blob>=<NULL> "ptcl"<uint32>="imap" "scrp"<sint32>=<NULL> "sdmn"<blob>=<NULL> "srvr"<blob>="imap.example.com" "type"<uint32>=<NULL> password: "somepassword"

Make sure it is finding the correct password. If there is more than one entry for the same domain/account pair, the service type may need to be specified, or one of the found entries should be renamed or removed from the Keychain.

Getting Just the Password

The only part of the output that is desired is the password itself. Sed, awk, or grep could be used, but perl will be used here to make it so that the only text on stdout is the password itself.

security find-internet-password -g -a someuser@example.com -s imap.example.com 2>&1\ | perl -e 'if (<STDIN> =~ m/password: "(.*)"$/ ) { print $1; }'

This could be put directly into a Mutt configuration file, or it could be put in a script.

Incorporating it into Mutt

For a simple single account muttrc file, one could do the following:

set smtp_user="someuser@example.com" set smtp_pass=`security find-internet-password -g -a someuser@example.com -s imap.example.com 2>&1 | perl -e 'if (<STDIN> =~ m/password: "(.*)"$/ ) { print $1; }'` set imap_user="someuser@example.com" set imap_pass=`security find-internet-password -g -a someuser@example.com -s imap.example.com 2>&1 | perl -e 'if (<STDIN> =~ m/password: "(.*)"$/ ) { print $1; }'`

Or to reduce the number of times security gets called:

set my_pass=`security find-internet-password -g -a someuser@example.com -s imap.example.com 2>&1 | perl -e 'if (<STDIN> =~ m/password: "(.*)"$/ ) { print $1; }'` set smtp_user="someuser@example.com" set smtp_pass=$my_pass set imap_user="someuser@example.com" set imap_pass=$my_pass

And that’s it. I actually use a Ruby wrapper around security, which uses a lockfile to prevent multiple instances of security from asking to unlock the keychain at the same time (I also use offlineimap, which calls the same script for the passwords). I may publish that script at some point.

Converting Rails Applications from MySQL to PostgreSQL

Tue, 08 Sep 2009 02:25:00 -0000

Update (2009/09/14): There is a much better article on doing this conversion here that is specific to migrating Typo from MySQL to PostgreSQL. I will admit that I have not tried using that article’s boolean conversion method.

I recently decided to give PostgreSQL a try after learning about some of the oddities of MySQL, plus it gave me an opportunity to see what was necessary to convert between two RDBMs. I installed PostgreSQL 8.3.7, which is the most recent version currently in the main Gentoo Portage tree, using the Gentoo Wiki guide. Rather than explain the details of setting up and using PostgreSQL, I’ll explain what I did to convert my Redmine and Typo instances. (The following assumes some familiarity with how to use Rails, MySQL, and PostgreSQL command line commands. See their man pages as well for other options that you may want/need to use)

Dumping the Database

At first I tried to use a rake taskto convert my redmine database, but it would fail because ActiveRecord can’t really treat join tables as ActiveRecord types. The PostgreSQL connector code wanted to generate a SQL statement that would return the value id, which ActiveRecord decided was the nonexistent primary key for the table. I was able to modify ActiveRecord to get around this, but the rake task was not loading the repository sub-types either.

Instead, I decided to try and convert it the “old fashioned” way: dumping sql code, editing it, and importing it. Note that I was using MySQL 5.0.70, and the options may not be available in older versions of MySQL. After shutting down all processes that might access the database, run

mysql -uroot --no-create-info --compatible=postgresql \ --complete-insert redmine > redminedump.sql

--no-create-info tells mysqldump to not print schema information. We will use Rails’ own migration functionality instead to recreate the schema in a manner that takes more advantage of PostgreSQL. --compatible=postgresql makes mysqldump produce SQL code with quotes that are more ANSI compliant, and it produces dates and times in a format that PostgreSQL seems to like. Finally, --complete-insert makes the INSERT statements in the dump include column names for the insertions.

Preparing the Dump for Import

If you tried to just import the dump into psql now, psql would report a multitude of errors. What follows are the ones I ran into, and how I got around them.

LOCK and UNLOCK Lines

First off, psql does not like the LOCK and UNLOCK statements in the dump. You can safely remove these lines. I zeroed out their lines in vim with

:%s/^\(UN\)\?LOCK.*$//g

boolean Field Types

The second type of error I received was about boolean fields:

ERROR: column "is_default" is of type boolean but expression is of type integer LINE 1: ... "enumerations" ("id", "opt", "name", "position", "is_defaul... ^ HINT: You will need to rewrite or cast the expression.

The equivalent fields in MySQL appear to be stored as integer types (1 and 0), but PostgreSQL prefers TRUE and FALSE for its boolean values – although it accepts '1' and '0'. Run

pg_dump -U postgres -s redmine > redmine_schema.sql

and look for the name of all boolean-type fields that would need to be changed. Then, back in the dump, carefully go through and quoted the correct 1’s and 0’s for each inserted row.

Possible Pitfalls

I ran into a confusing snag after I edited and imported my dump. Some serialized fields could not be converted into arrays. After tracking down this problem for a while, I discovered that extra newlines were being introduced to some of the serialized objects’ text fields. It turns out, I accidentally let vim hard wrap lines in the middle of quoted strings in the dump file.

Importing the Dump

Once your rails application is set up to access a PostgreSQL database, set up the new database’s schema:

rake db:schema:load RAILS_ENV="production"

After the schema and migrations have completed (I recommend verifying that the all the tables’ definitions are in place using something like pgAdmin3), it is finally time to import the dump:

psql -U postgres redmine < redminedump.sql

If the changes I mentioned above were made, there will still be one ERROR amongst dozens of warnings:

ERROR: duplicate key value violates unique constraint "unique_schema_migrations"

The schema_migrations table stores the number of each migration run on the database, and this error can be ignored. If you want to avoid this error, you could delete the INSERT lines that cause it from the dump.

Fixing the `id` Sequences

We are almost, but not yet done. The primary key in most ActiveRecord–managed tables is the “id” column, and each row in a table is supposed to have a unique entry. PosgreSQL uses sequences to increment this value for each new table entry, and the sequences need to be set to at least the highest numbered value they are associated with (a table with no entries can have its sequence be set to the default of 1). I used pgAdmin3 to inspect the contents of each table, and to then set the corresponding sequence’s current value, but this could also be done with some clever SQL code.

Conclusion

There were dozens of WARNINGs about “nonstandard use of escape in a string literal”, but they did not affect psql’s ability to correctly fill in its own tables.

At this point, you should take one last look over the data with some queries or pgAdmin3, looking for anything that might of gone wrong. After that, launch your rails application, and verify that it can still run.

Apache Worker MPM and RLIMIT_NPROC Resource Limit

Tue, 14 Jul 2009 01:04:49 -0000

I decided to give Apache 2.2’s worker mpm a try (a couple processes that each have many threads rather than the default prefork’s several processes that each handle a single thread), but I was having the Apache root process exit because its children were disappearing. Sometimes this would happen immediately after launching Apache, other times it would happen after I tried to access a web page. I tracked the problem down to being an issue with apache hitting the process limit for a given user. However, modifying limits.conf for the apache user did not solve the problem because the apache server inherits root’s limits on start up rather than referring to limits.conf.

The Initial Problem

After relaunching Apache, its processes would sometimes disappear immediately. Its error log contained (apache2/error_log):

[Sat Jul 11 00:32:42 2009] [notice] Apache/2.2.11 (Unix) configured -- resuming normal operations [Sat Jul 11 00:32:42 2009] [alert] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread [Sat Jul 11 00:32:44 2009] [alert] No active workers found... Apache is exiting!

I run gentoo’s hardened kernel with grsecurity’s logging active, so I also checked my grsecurity kernel log:

Jul 11 00:32:42 ouroboros grsec: From 71.202.174.186: denied resource overstep by requesting 32 for RLIMIT_NPROC against limit 32 for /usr/sbin/apache2[apache2:14547] uid/euid:81/81 gid/egid:81/81, parent /usr/sbin/apache2[apache2:14543] uid/euid:0/0 gid/egid:0/0

It appears that the number of processes resource limit for a given user applies to threads in Linux, and not just processes. By default, the worker mpm creates two worker processes with up to 25 worker threads each. Each process also has a thread for communicating with the parent process to handle incoming requests. This means there are can be 52 threads created for the apache user. The default limit of 32 processes per user was apparently getting exceeded when the threads were created, leading to the worker processes exiting and disappearing.

Attempting to Use limits.conf

First, I tried to increase the apache user’s limits in limits.conf:

* hard nproc 32 apache hard nproc 100

However, limits.conf is only referenced by the pam_limits.so module of PAM, meaning that PAM would have to run at some point before or during the start up of the apache server to set its resource limits. Many services on Gentoo Linux use the start-stop-daemon, which supposedly received PAM support in baselayout 1.13, but 1.13 is not in portage. The final version of baselayout 1 is 1.12 on Gentoo, with 2.0 being the next unstable version. Also, the apache2 startup script only calls apache2ctl and never uses start-stop-daemon, meaning that even if this feature were in the stable baselayout, limits.conf is not a normal option

The Final Solution

My final solution was to specify the limit in the configuration file for the startup script. At the bottom of /etc/conf.d/apache2, which is just a shell script file that gets sourced for /etc/init.d/apache2, I added

ulimit -u 100

This line calls bash’s ulimit command and sets the maximum number of processes to 100, which allowed apache to have its threads without crashing.

Ædifice.org Blog

Use Mac OS X's Keychain for Password Retrieval in OfflineIMAP

The python source file

offlineimaprc

Conclusion

RKHunter — Making False Positives Go Away on Gentoo Linux

Fixing changed binaries

Whitelisting Valid Scripts

An Expected Promiscuous Interface

Complaints of Hidden Directories

Disregarding Warnings About Outdated Applications

Disregarding the Kernel Module Checks

Conclusion

Use Mac OS X's Keychain for Password Retrieval in Mutt

Setting up Keychain

security Command Basics

Getting Just the Password

Incorporating it into Mutt

Converting Rails Applications from MySQL to PostgreSQL

Dumping the Database

Preparing the Dump for Import

LOCK and UNLOCK Lines

boolean Field Types

Possible Pitfalls

Importing the Dump

Fixing the id Sequences

Conclusion

Apache Worker MPM and RLIMIT_NPROC Resource Limit

The Initial Problem

Attempting to Use limits.conf

The Final Solution

`security` Command Basics

Fixing the `id` Sequences