October 11th, 2009 — Python
After server reinstall Plone website failed to start:
2009-10-11T13:45:56 ERROR Application Could not import Products.ATContentTypes
Traceback (most recent call last):
File "/data/user/site/parts/zope2/lib/python/OFS/Application.py", line 709, in import_product
product=__import__(pname, global_dict, global_dict, silly)
File "/home/apc/ecc/parts/plone/ATContentTypes/__init__.py", line 64, in ?
import Products.ATContentTypes.content
File "/home/apc/ecc/parts/plone/ATContentTypes/content/__init__.py", line 26, in ?
import Products.ATContentTypes.content.link
File "/home/apc/ecc/parts/plone/ATContentTypes/content/link.py", line 39, in ?
from Products.ATContentTypes.content.base import registerATCT
File "/home/apc/ecc/parts/plone/ATContentTypes/content/base.py", line 63, in ?
from Products.CMFPlone.PloneFolder import ReplaceableWrapper
File "/home/apc/ecc/parts/plone/CMFPlone/__init__.py", line 215, in ?
from browser import ploneview
File "/home/apc/ecc/parts/plone/CMFPlone/browser/ploneview.py", line 12, in ?
from Products.CMFPlone import utils
File "/home/apc/ecc/parts/plone/CMFPlone/utils.py", line 5, in ?
from PIL import Image
ImportError: No module named PILOops, we need to install PIL library, thanks to Dag this is extremely easy on CentOS/RHEL/Fedora:
yum install python-imaging
October 9th, 2009 — Linux
Firstly I thought it’s that my hard drive it’s faulty, because while writing a few large files it failed with the following messages:
sd 3:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Add. Sense: Logical unit not ready, initializing command required
end_request: I/O error, dev sdb, sector 794703
EXT3-fs error (device sdb1): read_inode_bitmap: Cannot read inode bitmap - block_group = 3, inode_bitmap = 99330
Aborting journal on device sdb1.
sd 3:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Add. Sense: Logical unit not ready, initializing command required
end_request: I/O error, dev sdb, sector 12423
Buffer I/O error on device sdb1, logical block 1545
lost page write due to I/O error on sdb1
sd 3:0:0:0: Device not ready: <6>: Current: sense key: Not Ready
Add. Sense: Logical unit not ready, initializing command required
end_request: I/O error, dev sdb, sector 63
Buffer I/O error on device sdb1, logical block 0
lost page write due to I/O error on sdb1
EXT3-fs error (device sdb1) in ext3_new_inode: IO failure
EXT3-fs error (device sdb1) in ext3_create: IO failure
ext3_abort called.
EXT3-fs error (device sdb1): ext3_journal_start_sb: Detected aborted journal
Remounting filesystem read-onlyAfter googling around I’ve found this thread:
http://ubuntuforums.org/showthread.php?t=494673
Thanks to trolav, I’ve solved my problem.
[root@silver ~]# cat /etc/udev/rules.d/85-usb-hd-fix.rules
BUS=="scsi", KERNEL=="sd?", SYSFS{vendor}=="Seagate", SYSFS{model}=="FreeAgent Go", RUN+="/scripts/usbhdfix %k"[root@silver ~]# cat /scripts/usbhdfix
#!/bin/bash
# USB FIX:
# http://ubuntuforums.org/showthread.php?t=494673
echo 1024 > /sys/block/$1/device/max_sectors
echo 1 > /sys/block/$1/device/scsi_disk:*/allow_restart
September 23rd, 2009 — PageActions, Rails, Ruby
I’ve just released PageActions plugin on GitHub. It’s a really simple Rails plugin, but it helps you to easy define and render actions links in your views. You can view installation and usage instructions on GitHub:
http://github.com/vitalie/page_actions
July 14th, 2009 — Rails, Ruby
We often need to iterate over the database rows in our migrations. When dealing with millions of records, basic iteration techniques doesn’t work well because each loaded object is consuming system memory and it issues at least one database query per object to load.
# >> Book.count :all
# => 4000216
Solution 1
class BooksUpdateTitles < ActiveRecord::Migration
def self.up
Books.all.each do |book|
# ...
end
end
def self.down
end
endThe problem with this version is that it will load all 4000216 objects into memory, all memory will be consumed and it will start to use disk swap and it will take hours to complete.
We can optimize it a little bit by specifying select parameter in our query:
Solution 2
Books.find(:all, :select => 'id').each do |t|
book = Book.find t.id
# ...
end
Version 2 still loads all objects in memory but selects only id field.
We need to avoid loading all objects in the memory, we’ll iterate over collection and we’ll load only current object.
Solution 3
last_id = 0
while book = Books.find(:first, :conditions => ['id > ?', last_id])
# ...
last_id = book.id
end
Version 3 it’s OK, but it can be speed up by loading objects in batch not just one by one.
Solution 4
last_id = 0
while books = Book.find(:all, :conditions => ['id > ?', last_id], :limit => 100)
# ...
last_id = books.last.id
end
Examining the log:
...
Domain LOAD (0.000176) SELECT * FROM `books` WHERE (id > 0) LIMIT 100
...
We have loaded 100 objects with one query. Solution 4 seems to be the best solution to iterate over large data sets as it uses less memory with fewer SQL requests.
Update:
Mitchell proposed a better solution to use ActiveRecord’s find_in_batches method. DHH commited this feature on February 23, 2009 that permits iterating over large data sets in batches:
Read more:
WebOnRails
GitHub
Solution 5
Book.find_in_batches(:batch_size => 100) do |results|
# Do something with results
end
June 25th, 2009 — Linux, Ruby
Installing latest Hpricot on ruby 1.8.5 fails due missing macro RARRAYPTR:
[root@silver ~]# gem install hpricot
Building native extensions. This could take a while...
ERROR: Error installing hpricot:
ERROR: Failed to build gem native extension.
/usr/bin/ruby extconf.rb
checking for main() in -lc... yes
creating Makefile
make
gcc -I. -I. -I/usr/lib64/ruby/1.8/x86_64-linux -I. -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wall -fno-strict-aliasing -fPIC -c hpricot_css.c
hpricot_css.rl: In function ‘hpricot_css’:
hpricot_css.rl:106: warning: implicit declaration of function ‘RSTRING_PTR’
hpricot_css.rl:106: warning: assignment makes pointer from integer without a cast
hpricot_css.rl:107: warning: implicit declaration of function ‘RSTRING_LEN’
hpricot_css.rl:82: warning: field precision should have type ‘int’, but argument 5 has type ‘long int’
hpricot_css.c:295: warning: comparison is always true due to limited range of data type
[...]
hpricot_css.c:3403: warning: comparison between pointer and integer
hpricot_css.c:3403: warning: ‘eof’ is used uninitialized in this function
hpricot_css.rl:92: warning: ‘aps’ may be used uninitialized in this function
gcc -I. -I. -I/usr/lib64/ruby/1.8/x86_64-linux -I. -fPIC -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -Wall -fno-strict-aliasing -fPIC -c hpricot_scan.c
hpricot_scan.rl: In function ‘our_rb_hash_lookup’:
hpricot_scan.rl:169: warning: implicit declaration of function ‘st_lookup’
hpricot_scan.rl: In function ‘make_hpricot_struct’:
hpricot_scan.rl:693: warning: implicit declaration of function ‘RARRAYPTR’
hpricot_scan.rl:693: error: subscripted value is neither array nor pointer
make: *** [hpricot_scan.o] Error 1
Gem files will remain installed in /usr/lib64/ruby/gems/1.8/gems/hpricot-0.8.1 for inspection.
Results logged to /usr/lib64/ruby/gems/1.8/gems/hpricot-0.8.1/ext/hpricot_scan/gem_make.outSearching with google I’ve found this post that explains equivalence of the RARRAYPTR(v) is RARRAY(v)->ptr . We’ll need to define RARRAYPTR macro.
We’ll replace occurences of #include <ruby.h> with #include “ruby_macros.h” and create an include file ruby_macros.h with the following content:
#ifndef __RUBY_MACROS__
#define __RUBY_MACROS__
#include <ruby.h>
#ifndef RARRAYPTR
# define RARRAYPTR(v) RARRAY(v)->ptr
#endif
#endif
[root@silver ~]# cd /usr/lib64/ruby/gems/1.8/gems/hpricot-0.8.1/ext/hpricot_scan
[root@silver hpricot_scan]# sed -i 's,#include <ruby.h>,#include "ruby_macros.h",g' *.h *.c *.rl
[root@silver hpricot_scan]# touch ruby_macros.h
[root@silver hpricot_scan]# vi ruby_macros.h
The next step is to recreate the gem and install it:
[root@silver ~]# cd /usr/lib64/ruby/gems/1.8/gems/hpricot-0.8.1
[root@silver hpricot-0.8.1]# rake package
(in /usr/lib64/ruby/gems/1.8/gems/hpricot-0.8.1)
fatal: Not a git repository
rm -r ext/fast_xs/Makefile
rm -r ext/hpricot_scan/Makefile
rm -r .config
rm -r pkg
rm -r hpricot-0.8.1-mswin32
rm -r hpricot-0.8.1-jruby
Using ragel version: 6.3, location: /usr/bin/ragel
cd ext/hpricot_scan ; ragel hpricot_scan.rl -G2 -o hpricot_scan.c && ragel hpricot_css.rl -G2 -o hpricot_css.c
mkdir -p pkg
mkdir -p pkg/hpricot-0.8.1
rm -f pkg/hpricot-0.8.1/CHANGELOG
[...]
cd pkg
tar zcvf hpricot-0.8.1.tgz hpricot-0.8.1
hpricot-0.8.1/
hpricot-0.8.1/Rakefile
[...]
hpricot-0.8.1/ext/fast_xs/fast_xs.c
cd -
WARNING: description and summary are identical
Successfully built RubyGem
Name: hpricot
Version: 0.8.1
File: hpricot-0.8.1.gem
mv hpricot-0.8.1.gem pkg/hpricot-0.8.1.gem
[root@silver hpricot-0.8.1]# gem install pkg/hpricot-0.8.1.gem
Building native extensions. This could take a while...
Successfully installed hpricot-0.8.1
1 gem installed
Installing ri documentation for hpricot-0.8.1...
Installing RDoc documentation for hpricot-0.8.1...
[root@silver hpricot_scan]# gem list -l | grep hpricot
hpricot (0.8.1, 0.7, 0.6.164, 0.6.161, 0.6)
June 15th, 2009 — Rails
Problem:
Missing host to link to! Please provide :host parameter or set default_url_options[:host] when sending emails.
Solution:
You can pass host parameter to url functions, but it’s cleaner to configure it with a before_filter globally in your application_controller.rb:
# application_controller.rb
before_filter :mailer_set_url_options
...
def mailer_set_url_options
ActionMailer::Base.default_url_options[:host] = request.host_with_port
end
May 29th, 2009 — Rails, Ruby
A simple script to convert .erb files from current directory to .haml :
#!/usr/bin/ruby
Dir.glob("*.html.erb").each do |erbname|
hamlname = erbname.gsub(".html.erb", ".html.haml")
system "/usr/bin/html2haml #{erbname} #{hamlname}"
end
May 21st, 2009 — Rails, Ruby
Simulating file uploads in your scripts or from console can be done really simple using Rail’s ActionController::TestUploadedFile from action_pack.
Example code:
require 'action_controller/test_process'
class ImportExternalData
...
def import_data
...
page.attachments << PageAttachment.new(
:uploaded_data => fake_file_upload(filename, mime_type),
:title => title,
:description => description)
...
end
protected
def fake_file_upload(path, mime_type = nil, binary = false)
ActionController::TestUploadedFile.new(
path,
mime_type,
binary
)
end
endExcerpt from ActionController::TestUploadedFile’s comments:
Essentially generates a modified Tempfile object similar to the object
you’d get from the standard library CGI module in a multipart
request. This means you can use an ActionController::TestUploadedFile
object in the params of a test request in order to simulate
a file upload.
April 8th, 2009 — Linux, Plesk
After Plesk upgrade we had to upgrade SpamGuardian too, but the scripts from the rpm contains syntax errors and the package can’t be installed:
[root@andromeda tmp]# rpm -Uvh --force 4psa-sguardian-psa9-3.5.0-090305.01.rhel5.i386.rpm
Preparing... ########################################### [100%]
1:4psa-sguardian-psa9 ########################################### [100%]
/var/tmp/rpm-tmp.42581: line 315: syntax error near unexpected token `>>'
/var/tmp/rpm-tmp.42581: line 315: ` echo >> >> /etc/sguardian/sguardian.conf'
The solution is to install the rpm without running PRE/POST install scripts:
[root@andromeda tmp]# rpm -Uvh --force --noscripts 4psa-sguardian-psa9-3.5.0-090305.01.rhel5.i386.rpm
Then we can extract scripts with mc (midnight commander), or rpm -qp –scripts
and run them separately after correcting the syntax errors (comment lines containing “>> >>” string):
[root@andromeda scripts]# sh PREIN
[...]
[root@andromeda scripts]# sh POSTIN
===> Installing 4PSA Spam Guardian
===> Regenerating Mail files
==> Checking for: mailsrv_conf_init... ok
==> Checking for: mail_mailbox_restore... ok
==> Checking for: mailsrv_entities_dump... ok
==> Checking for: mail_auth_dump... ok
==> Checking for: mailman_lists_dump... ok
==> Checking for: mail_responder_restore... ok
==> Checking for: mail_drweb_restore... ok
==> Checking for: mail_kav_restore... not exsists
==> Checking for: mail_spf_restore... ok
==> Checking for: mail_dk_restore... ok
Success
[...]
April 7th, 2009 — Linux, Plesk
We alreay know how to extract files from Plesk backup with mpack, but if you do receive “File size limit exceeded” error, it can be from the following reasons:
- Your file system doesn’t have LFS (large file support) support
- OS limits (limits on the system resources imposed by your administrator)
- mpack is not compiled with LFS
Check that your filesystem supports files bigger than 2GB (create a test file with “dd” command).
[vitalie@silver ~]$ dd if=/dev/zero of=test.bin bs=1G count=3 # create 3GB file
Then check your file size limits with ulimit command:
[vitalie@silver ~]$ ulimit -a | grep '^file size'
file size (blocks, -f) unlimited
You can adjust limits from /etc/security/limits.conf, read file comments on how to do it, or read this article:
http://www.cyberciti.biz/faq/file-size-limit-exceeded-error-under-linux-and-solution/
If none above limits you, then the problems comes from mpack, it doesn’t have suport for large files (>2GB).
The guys from the Plesk recommends to grep the backup file, to find begin and end of the file in archive:
http://kb.parallels.com/en/1757
This solutions is not appropriate for large backup files, we’ll just recompile mpack to support large files.
Grab mpack’s RPM source from the Dag’s repository
http://dag.wieers.com/rpm/packages/mpack/mpack-1.6-2.rf.src.rpm
install it:
[vitalie@silver ~]$ rpm -ivh mpack-1.6-2.rf.src.rpm
[...]
Then modify mpack.spec and add the following code after “%build”:
export CFLAGS="-D_FILE_OFFSET_BITS=64"
Then rebuild the package and install it:
[vitalie@silver ~] rpm -ba mpack.spec
[...]
[vitalie@silver ~] rpm -Uvh --force mpack-1.6-3.rf.i386.rpm
[...]
References: