Blog

Login Register
  • Archive

    «   November 2014   »
    M T W T F S S
              1 2
    3 4 5 6 7 8 9
    10 11 12 13 14 15 16
    17 18 19 20 21 22 23
    24 25 26 27 28 29 30
                 

Beta: IOPS, high precission CPU limits and more...


New update for CageFS, liblve, lve-utils and LVE Manager is available from our beta repository. Major changes are introduction of IO operations per second limits, high precission CPU limits (you can now setup speed precission as low as 1% of a single core, no matter how many cores) - and ability to specify the processes that needs to be in LVE / CageFS by name.

Changelog:
CageFS 5.3-1
  • CAG-315: cagefsctl --rebuild-alt-php-ini reset some parameters to defaults (bugfix)
liblve-1.3-1.3
  • LIBLVE-7: enter to cagefs by process name
  • CAG-76: added new "splitted by username" mount type in cagefs.mp
  • support of hires cpu limit
lve-utils 1.4-27
  • LU-107: add --no-iops option to lvectl, getcontrolpaneluserslimits (for backward compatibility)
  • LU-100: lvetop should display CPU usage in terms of 'speed' setting
  • LU-97: lvectl set $LVE --iops $IOPS doesn't set IOPS parameter
  • LVEMAN-109: add handling of iops and speed (for proc version to cpanel/extension/cl_modify_pkg.py
  • add ability to change lve_ext template on cpanel
  • Added lve_namespaces service to record LVE namespaces on boot
  • LU-92: fix PID column in lveps -p output
  • show command names in the COM column
  • LU-91: add lve_namespaces service
  • LU-90: getcontrolpackages fail in DirectAdmin with broken cache file
  • ALTPHP-31:MariaDB 10 support in php-selector
python-cllib 1-21
  • LU-89: add base hook lib
  • PTCLLIB-16: add validate_cpu function
  • PTCLLIB-15 fix: Add /usr/sbin/lveps to /etc/sudoers
lvemanager 0.8-1.44
  • LVEMAN-223 - Add conflicts for PHP APCu module
  • LVEMAN-222: bugfixes for LVE Manager->packages in cPanel
  • LVEMAN-161: LVE Manager for cPanel: filter reseller packages correctly
  • LVEMAN-166: remove NCPU from LVE Manager for cPanel
  • LVEMAN-217: DirectAdmin LVE Manager for /proc/lve/list 8 : incorrect column values in settings, packages
  • LVEMAN-214: use --no-iops option in lvectl commands in LVE Manager for compatibility with new lve-utils
  • LVEMAN-212 fix: Defaults values in Edit package page are incorrect for Plesk -> LVE Manager
  • LVEMAN-211 fix: Accounts page fails in Plesk
lve-stats 0.10-37
  • LVESTATS-52: Graphs for small speed values are not created
  • LVESTATS-51: lvestats-server does not work on /proc/lve/list ver 4
  • LVESTATS-50: lvestats-server: calculate cpu limit correctly for /proc/lve/list ver 8
  • LVESTATS-37: mark parameters that were exceeded by users in nootification e-mails for admin and resellers
  • LVESTATS-36: Wrong lveinfo data from MySQL on centralized server
  • LVESTATS-17: Record and manage IOPS
Update instructions:

$ yum update cagefs lvemanager lve-utils lve-stats --enablerepo=cloudlinux-updates-testing

Please, note that this update will install new kernel. Reboot is needed to enable all the new features, like high precission CPU speed limits, and IOPS.

Beta: New way to set CPU limits

New version of CageFS, liblve & lvemanager brings in a number of new features and bug fixes. One of the new features is introduction of new --speed parameter, that in the future will replace --cpu. Speed parameter accepts CPU speed limit in the form of % of a single core, or as number of mhz. This allows you to set limits like:
--speed=150%, that would mean 1 and 1/2 cores
or
--speed=3000mhz. In this case, the software will automatically calculate the share of CPU to give to a customer, based on processors speed. That allows for a 'standard' limits across a number of servers, even if they have different CPUs, working at different frequencies.

Changelog
CageFS 5.2-30

  • CAG-278: bugfix: correctly work in a case when alt_phpXX.cfg has directive that is not in php.conf
  • CAG-272: bugfix: ISPmanager: User's status is not set on create via HTTP request
  • CAG-271: cagefsctl: check /proc/lve/list in order to detect LVE/jail support
  • CAG-267: add global php.ini with default directives/values for all alt-php
  • CAG-240: add option --list-logged-in to show users logged in cagefs via ssh
  • CAG-268: Add ability to run proxy commands on remote server
liblve 1.2-1.11
  • LIBLVE-6: bugfix: CageFS for openvz: Apache's script cannot enter CageFS
lve-utils 1.4-7
  • LU-63: remove mail alerts after lveutils-panel-cron on interworx
  • LU-61: correctly handle list index out of range when setting limits in lvectl
  • LU-57: add ability to use fractional number as limits. Allow for mhz/ghz cases
  • LU-56: bugfix: fix issue that result in 'can't init lve default settings error on CL5"
  • LU-55: add lvectl set --speed
  • LU-46: use default cpu value if we cannot parse value
  • LU-46: DirectAdmin: redone algorithm of finding packages
  • LU-47: lvectl now uses pylve lib
  • LU-47: lvectl can parse new lve-kmod format
  • LU-43: add ability to specify IOPS (input output operations per second)
  • LU-44: add IOPS to lveps and lvetop
python-cllib 1-13
  • PTCLLIB-7: more informative error messages in clfunc.py
  • PTCLLIB-6: bugfix: memory convert funcions
lvemanager 0.8-1.17
  • LVEMAN-114: Plesk: prevent admin from enabling PHP Selector when CageFS disabled
  • LVEMAN-114: DirectAdmin: prevent admin from enabling PHP Selector when CageFS disabled
  • LVEMAN-120: ISPmanager: bugfix: LVE statistics under user doesn't show up
To update:
$ yum update cagefs liblve lve-utils lvemanager --enablerepo=cloudlinux-updates-testing

Updates for: CageFS, LVE Manager, lve-utils, lve-stats, pam_lve

This update introduces large number of bug fixes and improvements affecting all aspects of CloudLinux

Changelog:


pam_lve 0.3-7
  • PAMLVE-2 fix: allow to specify multiple groups in pam_lve to exlude
LVE Manager 0.8-1.15
  • cPanel: Improved Spanish translation. Special thanks to PYDOT
  • cPanel: Added space between 'native' and actual PHP version
  • cPanel: Hide PHP options link when native version selected
  • Limits for packages are read from /etc/container/ve.cfg
  • DirectAdmin: incorrect links to Packages/Settings/Edit had been fixed
  • DirectAdmin: use selectorctl to switch php.ini & PHP wrappers
  • ISPManager: use selectorctl to switch php.ini & PHP wrappers
  • bugfix: drop EUID/EGID when creating .cl.selector directory
  • bugfix: make ioncube_loader always to be the first extension to load
  • bugfix: fix symlink to php.ini when switching to native version
cagefs 5.2-25
  • DirectAdmin: fix error messages when creating symlink
  • ISPmanager: don't fail on empty PHP wrappers
  • ISPmanager: don't change PHP wrapper for native PHP
  • ISPmanager: added /usr/local/ispmgr/www to CageFS
  • ISPmanager: instantiate ClPwd when creating user wrapper
  • ISPmanager: change PHP wrapper in /var/ww/php-bin
  • bugfix: --user-stats and --update-user-status should work for excluded users
  • bugfix: better error message on --getprefix when user doesn't exist
  • code cleanup, better exception handling
  • sudoers file modification is now handled using clsudo library
  • CXS re-configuration moved to a different module
  • CAG-269: do not pass GID argument from proxyexec wrappers to cagefs.server (secutiry fix)
liblve 1.2-10
  • implementation of pivot_root scheme
lve-utils 1.3-31
  • Separate python library created for common control panels
  • Added ability to specify custom script for custom control panel to get user<->package mapping
  • use clcommon library from python-cllib
  • Added ability to save configuration based on username instead of LVE id in ve.cfg using set-user & --save-username parameter
  • bugfix: verify results from liblve when setting limits
  • LVEMAN-107: fix bug License not valid script in cPanel doesn't work "sumbit" instead of submit
  • redone algorithm of finding panel packages for DA (LU-46) - Now each admin must see packages from
python-cllib 1-12
  • bugfix: remove strict option when processing sudo file using visudo on CL5
  • preload full password database in some cases
  • added validate_int function
  • bugfix: catch ValueError in memory_to_page function
lve-stats 0.10-21
  • Check date when convering from DB Governor
To update:
$ yum update cagefs lvemanager lve-stats liblve lve-utils

Beta: new LVE and IO Limits kernel modules for CL6

New kernel modules for latest kernel 2.6.32-379.5.1.lve1.1.9.6.1.el6 are available:

kmod-lve 1.1-9.10:
  • Default namespace initialization on startup
  • BUGFIX: ub initialization correction on LVE enter
kmod-iolimits 0.1-25 :

  • Mark throttled thread as sleeping
To install:
# yum update kmod-iolimits-0.1-25.el6 kmod-lve-1.1-9.10.el6 --enablerepo=cloudlinux-updates-testing

Beta: MySQL Governor - Reloaded. New concept, new features, less frustration


MySQL Governor 0.8-9 is available for beta testing. This new version features completely new concept behind limiting MySQL usage.
  1. We no longer use USERSTATS. Instead we monitor usage real time. We are no longer constrained by queries being finished before we can get usage data
  2. There is just 3 limits: CPU, IO READ, IO WRITE. Nothing else. Easy to define, easy to monitor, easy to judge.
  3. CPU Limits are set in % relative to a single core. IO limits are in bytes per second
  4. When customer abuses any limit, you don't have to suspend customer's access to MySQL. Instead, you can put them into an LVE, slowing down hist MySQL queries, and preventing abuse
  5. All abusers will be placed & will share same LVE (with ID 3), you can adjust settings for that LVE to define how much resources are allowed for abusers
  6. Abusers will be restricted to 30 concurrent connections (can be changed), so that they would not exhaust all database connections.
In the end you should see new MySQL governor being much quicker to react, and it should no longer create as much disturbance when customer is restricted

You can find more info about new MySQL governor here:
http://docs.cloudlinux.com/index.html?mysql_governor.htm

Installation instructions:
http://docs.cloudlinux.com/index.html?installation3.html * Due to completely new concept behind MySQL Governor, we renamed the RPM. The installation instructions include instructions for removing old version of MySQL Governor

Beta: IO Limits for CloudLinux 6.x

I am happy to announce long awaited beta of IO Limits for CloudLinux 6.x. Now you can setup IO throughput for your clients as measured in KB/sec. The limit is for both reads & writes, combined -- and we set it as a single number for simplicity.

Included in the upgrade are new kernel version 2.6.32-320.17.1.lve1.1.9.1, new lve-utils 1.0-24, lve-stats 0.8-8 and LVE Manager for cPanel version 0.4-10.

You can set new IO limits either by using LVE Manager (cPanel only), or from command line:
# lvectl set LVE_ID --io=XXXX

You can disable IO Limits by editing /etc/sysconfig/iolimits, adding line: IOLIMITS_ENABLED=no, and running:
# service lve restart

To install iolimits, do:
# yum install kmod-iolimits --enablerepo=cloudlinux-updates-testing

That will bring up kernel module for IO limits, as well as new kernel and LVE module needed. Please, make sure you update lve-utils, lve-stats and lvemanager packages as well using:
# yum update lve-stats lve-utils lvemanager --enablerepo=cloudlinux-updates-testing

And reboot the server (to load new kernel kernel with IO module).

Changes: kernel-2.6.32-320.17.1.lve1.1.9.1:
  • iolimits implementation
  • lve-utils 1.0-24:
  • Fixed vMEM, pMEM, IO & NPROC range
  • Automatically update IO from 25 to 1024KB/sec in ve.cfg
  • Improved performance for getcpaneluserspackages script
lve-stats 0.8-8:
  • Added IO usage tracking
  • Disable faults counters displays for limits that are disabled
  • Bugfix: --by-usage option not to print data for limits that are disabled (set to be unlimited)
lvemanager 0.4-10: (cPanel only)
  • Added IO usage and IO limits
  • Added new limits for end user section
  • No longer showing reseller's packages in LVE Settings -> Packages tab
  • No longer displaying default package. Packages inherit from default LVE limits
  • Real settings from /proc/lve/list are displayed in LVE Manager -> Settings tab

Beta: lve-stats 0.8-6 adds historical information for physical memory and number of processes limits

New version of lve-stats 0.8-6 that supports new LVE limits, such us physical memory limits and number of processes limits has been released to our beta repository.

It has following new features and bug fixes:
  • lveinfo: added ability to retrieve users that had any kind of faults –by-fault=any
  • lveinfo: added option to order by any kind of fault –order-by=any_faults
  • vechart: renamed aliases of usages in legend
  • lvechart: added processing of terabytes as memory unit
  • lvechart: only show charts for the limits that are enabled
  • lvechart: added option –show-all to show all charts, even if limit is disabled
  • lveinfo: added processing of new aliases (vmem_avg, vmem_max, total_vmem_faults, total_ep_faults, etc..)
  • lveinfo: Usernames are now saved into database using script save_users_to_database.py
  • lveinfo: install daily cronjob to update users in central database
  • lveinfo: added ability to read usernames from central database
  • lvestats-updatedb, lvestats (service): added column lve_version to table last_run
  • lvestats-server: write lve_version to database
  • lveinfo, lvechart: read lve_version from database
  • lveinfo: do not print usages if appropriate limit is not set
  • lveinfo: added --show-all option to print user even for limits that are not enabled
  • lveinfo: removed duplication of code
  • lve-stats: added pMEM and nPROC limits
  • lvestats-server: save data to DB if there are any memory activity, even if there was no recordable CPU activity
  • lveinfo, lvedata.py: handle --limit setting while processing --by-fault, --by-usage options
  • lvestats-server: Handle cases when CPU usage is reported as negative number
To update:
yum update lve-stats --enablerepo=cloudlinux-updates-testing

Beta: CageFS 3.2 Beta 3

This is a minor update for CageFS 3.2 in preparation for production release. It fixes the issue created by securelinks feature as well as few other small bug fixes.


Full list of changes in CageFS 3.2 Beta 3 (cagefs-3.2-11)
  • SecureLinks are disabled by default
  • Do not generate .htaccess files in /etc/skel, /root/cpanel3-skel, /usr/local/apache/htdocs
  • create-htaccess.py: added options --ignore-exlusions, --check-exclusions
  • added /usr/local/apache/htdocs to exclusions for SecureLinks
  • added /usr/bin/perl-bin to perl.cfg
  • added install_prekillaccnhook.sh
  • create-htaccess.py: do not fail while removing .htaccess when .htaccess file is empty
  • create-htaccess.py: read paths to home dirs of users from /var/cpanel/userdata
To update
# yum update cagefs --enablerepo=cloudlinux-updates-testing

To install
# yum install cagefs --enablerepo=cloudlinux-updates-testing

Beta: CloudLinux 6 - kmod-lve-1.1-8 update

Instead of updating the whole kernel, and requiring reboot, this time around we are able to release just a module update to fix sporadic crashes related to CageFS use, in latest CloudLinux 6 kernel (lve1.1.4)

To install the update, please run:
# yum update kmod-lve-1.1-8 --enablerepo=cloudlinux-updates-testing
# service httpd stop
# lvectl destroy all
# service lve restart
# service httpd start

Stopping/starting httpd is fairly important, as you will not be able to reload module (done via service lve restart), unless it is not in use.
Alternatively, you can just reboot the server if it would make it easier for you.

Sending regular emails to yourself with LVE usage

* Script was changed to use order-by instead of by-usage

Lets say you would like to get an email every two weeks containing summary info on customer usage. This is something very simple to script with LVE.

First, make sure you have lve-stats installed:
# yum install lve-stats


Then create a script, lets call it /usr/local/bin/biweekly-lve-usage
#!/bin/bash

PERIOD=14d #number of days in summary reporting, lets do it 7 days, for the simplicity
ENTRIES=10 #number of entries per summary

echo "TOP $ENTRIES users by memory usage"
lveinfo --period=$PERIOD --limit=$ENTRIES --order-by=mem_avg --display-username
echo
echo "TOP $ENTRIES users by memory faults"
lveinfo --period=$PERIOD --limit=$ENTRIES --by-fault=mem --display-username
echo
echo "TOP $ENTRIES users by cpu usage"
lveinfo --period=$PERIOD --limit=$ENTRIES --order-by=cpu_avg --display-username
echo
echo "TOP $ENTRIES users by process entry faults"
lveinfo --period=$PERIOD --limit=$ENTRIES --by-fault=mep --display-username
Once you have the script, just add an cronjob entry that looks like this:
0 5 */14 * * sh /usr/local/bin/biweekly-lve-usage| mailx -s "LVE usage report `hostname` `date`" YOUR@EMAIL

This will send you report every 14 days

New Kernel for CloudLinux 5.x: 2.6.18-374.18.1.el5.lve0.8.57

New kernel 2.6.18-374.18.1.el5.lve0.8.57 was moved to production. It includes everything from lve0.8.54 kernel as well as:
# yum install kernel-2.6.18-374.18.1.el5.lve0.8.57

If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

Read this if you use su or sudo - updated for pam_lve 0.3

* Update: pam_lve now supports wheel group, and doesn't put users in that group into LVE

We have recently released CageFS into production -- as part of the update, we updated vixie-cron (package responsible for cronjobs) to use pam_lve.As the result, on many servers where pam_lve wasn't installed before that RPM got installed as well.

The RPM ads LVE handing for SSH, and when you login as a regular user to SSH, and then do: su or sudo you will still be inside LVE (even though you will get root), and anything you start (apache, mail) -- will be inside one, single LVE.

Starting with version 0.3 released on March 5th, 2012, the pam_lve module will not put any user belonging to wheel group into LVE.

If your users that do su/sudo are not in that group -- add them to that group
You can also modify the group, by editing files: /etc/pam.d/sshd, /etc/pam.d/crond to include group name like:
pam_lve required 100 1 my_su_group


Alternatively you can change UID of all accounts that you use to su or sudo to have UID less then 100.
You can do it by running:
usermod -u UID username


If you are seeing only one user in lvetop -- please, read this page as well:
http://www.cloudlinux.com/docs/errors/MHL-EPERM.php

Update for SecureLinks for Apache - support for global aliases

We had a lot of positive feedback regarding SecureLinks. I am glad to announce second beta of SecureLinks that ads support for global aliases. This prevents some of the issues when global aliases became unavailable after SecureLinks were installed.

To enable new version of SecureLinks, please run:
# yum update mod_hostinglimits --enablerepo=cloudlinux-updates-testing

The RPM version of mod_hostinglimts is 0.9-5
We plan to send new version of mod_hostinglimits for inclusion in EasyApache early next week.

Another good news is that LiteSpeed webserver starting with 4.1.11 RC is supporting SecureLinks as well.

cPanel & RoR inside LVE

To make RoR run inside LVE for cPanel

Step 1:
yum install lve-wrappers

Step 2:
Edit file: /usr/local/cpanel/Cpanel/RoR.pm
Edit line 558, change it from:
exec( 'mongrel_rails', 'start', '-p', $port, '-d', '-e', ( defined $appenv ? $appenv : 'production' ), '-P', 'log/mongrel.pid' )
to
exec( '/bin/lve_wrapper', 'mongrel_rails', 'start', '-p', $port, '-d', '-e', ( defined $appenv ? $appenv : 'production' ), '-P', 'log/mongrel.pid' )

After that, once mongrel will be re-started, and all new instances will start inside LVE.

Kernel: 2.6.18-374.12.1.el5.lve0.8.54 is in production

I am happy to announce new kernel. The kernel comes with a number of bug fixes, as well as new features related to up-coming CageFS 3.0
  • rebase to 274.12.1.el5 kernel (RHSA-2011:1479)
  • migration processes are now hidden from /proc file system
  • Load averages calculation optimization
  • LVE destroy fix to prevent kernel crash
  • UBC related optimization for vcpu structures
  • xtable kernel panic fix
  • namespaces reworked for CageFS 3.0
To install:

# yum install kernel-2.6.18-374.12.1.el5.lve0.8.54
If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

CageFS 3.0 is ready for beta testing

It is finally here! CageFS 3.0 is ready for adventurous souls that want to put latest and greatest on their servers.Right now it is only available released it only for CloudLinux 5.x. CloudLinux 6.x support will be coming soon. Here are some of the highlights of a new version:
  • Better namespace handing, requiring only fraction of mount points comparing to CageFS 2.x
  • CageFS-FUSE provides virtualized /etc & /var/log -- increasing security, and decreasing complexity of maintaining CageFS
  • Caged directories are no longer visible in /proc/mounts, solving all related issues with cPanel
  • Management plugins for cPanel and Plesk will be installed automatically (other control panels coming soon)
  • Automatic detection of cPanel and Plesk (other control panels coming soon), with automatic configuration to adjust for the running system
  • Improved command line tools
The current version should work with LiteSpeed, and or with custom control panels with some additional configuraiton.


More information on CageFS, how to install it or update from previous version can be found here:
http://www.cloudlinux.com/docs/cagefs/index.php

Multiple packages updated for CloudLinux

We have released multiple packages to CloudLinux Production repositories:
lve-utils 0.6-10
  • Added lvectl destroy <id> -- for lve0.8.44 and later kernels
  • Disabled error message regarding UBC for CloudLinux 6
lve-stats 0.8-2
  • Added separate config file for lveinfo and lvechart to hold read only database access
  • Fixed issue with CloudLinux 6, and limits showing higher then ncpu allow
  • Added style option to lveinfo and lvechart to normalize CPU if style is set to user
mod_hostinglimits 0.8-2 (RPM only, for Plesk, InterWorx, ISP Manager users, not for cPanel/DirectAdmin)
  • Changed error messages to provide URL to full description of the error
  • Added LVERetryAfter - send RetryAfter header
  • Added LVEParseMode - there are three modes of working(CONF, PATH, OWNER)
  • Added LVEPathRegexp - regexp expression for username extracting from path(PATH mode)
Additionally we moved into production LVE Manager plugins for Plesk, DirectAdmin, InterWorx and ISP Manager:
plesk-lvemanager 0.1-7
interworx-lvemanager 0.1-3
da-lvemanager 0.1-6
isp-lvemanager 0.1-7

All the packages should be available via regular yum update

lve-stats 0.8 beta - some minor improvements and bug fixes

Just in time for a new year we have a new version of lve-stats. That version brings few enhancements and bug fixes. We have added admin & user styles for charting & lveinfo output. The new style will make sure that user will see CPU chart & output normalized relative to 100% of usage.

We also improved our integration with MySQL & PostgreSQL. You can now create /etc/sysconfig/lvestats.readonly and provide user that has read only (SELECT) access to history table in lvestats database. That file will be used by lveinfo & lvecharts to generate end user statistics.

This way you can make /etc/sysconfig/lvestats readable only by root, preventing ability of user to insert or delete any info fromlvestats database.
Please, note that this issue doesn't exist with SQLite that is being used by default.

Additionally, we fixed CPU reporting in CloudLinux 6. Before lve-stats would record CPU usage higher then it would be possible due to NCPU limit.

To install latest update, please run:
# yum update lve-stats --enablerepo=cloudlinux-updates-testing

Another Great Year Is Almost Over

We are pretty much done with 2011. There are just a few days left. It was a good year. We got to 6000+ servers running CloudLinux (4x increase since 2010), our revenues are more then 1,000% up, our team doubled size and we gained hundreds and hundreds new customers.
We achieved quite a few goals that we have set:
  • Signed up cPanel and Parallels as distributors making sure that their software is well integrated with CloudLinux
  • Released CloudLinux 6.x
  • Implemented memory limits
  • Drastically improved stability and speed of our repositories by making them fully redundant and geographically distributed
  • Released plugins for cPanel, Plesk, DirectAdmin, ISPmanager and InterWorx
I am very thankful to our customers and partners for making it possible. It is a pleasure working with you, and we appreciate your support. We strive to make sure that your systems are stable and secure – and we will continue working toward this goal. Next year we will:
  • Provide an ability to use CloudLinux 6 kernel on CloudLinux 5 servers
  • Release productions versions of CageFS and make it the best way to secure shared hosting server
  • Release stable version of MySQL governor
  • Improved logging that would allow you to pin-point actual issue giving your customers more visibility into what went wrong.
  • Introduce IO limits, CPU weights, physical memory limits, and limit for a number of processes
  • Provide better integration with control panels, including email notifications and different limits per plan
  • Centralized interface to see data about resource usage for all your servers
And hopefully, some other features that you will demand from us as we move forward.
Please, accept my warmest greetings. I hope you will have terrific new year.

Igor Seletskiy

Why CloudLinux is super stable, or standing on the shoulders of the giants.

One of the features of CloudLinux we are targeting and caring for is stability. We surely understand that our customers' profits greatly depend on the quality of service (QoS) that they can provide. Users expect their web sites to be up and running no matter what. Every single minute of a site unavailability increases site owner frustration, and that eventually converts into bad reviews and customers running off while screaming loudly. Yes you should worry now, because they run away from you to that other hosting service provider, and they are taking their money with them!



While downtime is probably inevitable (even in extreme cases, such on a space ship, where it could cost billions of dollars, or in on-line trading system, where it could easily cost much more than that), it should happen as rare as possible. There are many factors to that, some of that are out of CloudLinux control, such as data center reliability. That includes not only obvious things such as uninterruptible power supply and well-connected networking, but also things such as good conditioning. The author once seen a server room disaster caused by a broken A/C unit which just went on fire, transforming its plastic parts into gross amounts of the black smoke and ash, and blowing the resulting products right into a room, due to a huge and fast fan which, ironically, was still perfectly working. Experience not to forget easily, must I say!

So what can CloudLinux do to help increase that famous number of nines metric? Two simple things:
  1. Protect those web sites from each other, by wisely distributing available hardware resources between them.
  2. Be stable and secure, immune to attacks and exploits, do not crash.
First item looks pretty complicated, and it is actually the very core of CloudLinux technology. It deserves a few blog entries, or maybe even a book (not that thin at all!). What I'd like to impart now is the second item (stability, security and no exploits).



In theory, one can write correct and bug free software. In practice, it's just as impossible as flying (wake up, Neo. The matrix has you). Software stability is the result of an endless battle between developers fixing bugs and themselves adding more bugs. Well, they like to refer to those as “features” rather than “bugs”, but it is a truth universally acknowledged that features and bugs come bundled together. That is why every respected software development cycle has a certain phase called “feature freeze”, during which they only add fixes but not bugs.

Sometimes this phase is running in parallel with development, that is, some developers continue to add more stuff, while others are cherry-picking bug fixes from that stream. This is exactly how the -stable branches work in mainline Linux kernel: after releasing a certain kernel version (say 3.1) they keep on bashing the next one (3.2), while people like Greg Kroah-Hartman collect bugfixes and periodically release stable kernels like 3.1.1, 3.1.2 and so on.
Then, Linux vendors are doing the same thing, branching their kernels off of a specific mainline kernel version and adding more and more bug and security fixes. One of the vendors who is particularly good at doing that is Red Hat. With their Enterprise Linux kernels, they usually take a kernel and then marinate it for at least six months, doing testing and fixing. Result? A kernel which is much more stable than the mainline one.

Thanks to open source model, CloudLinux stands on the shoulders of Red Hat. What we do is we take RHEL6 (Red Hat Enterprise Linux, version 6) kernels and put our stuff on top of those kernels. This is a way to improve stability and security. More to say, it lets us concentrate on our real job: providing a good platform for shared hosting, leaving the complex job of maintaining a stable and secure kernel to excellent kernel team of Red Hat. Improve your servers' stability. Go CloudLinux!

New Kernel for CloudLinux 5.x: 2.6.18-374.3.1.el5.lve0.8.44

New kernel 2.6.18-374.3.1.el5.lve0.8.44 is available. It includes following bug fixes and new features:
  • Rebased to upstream kernel 028stab094.3 including security and bug fixes RHSA-2011:1212
  • Kernel panic fix for lve_list_next race condition
  • Rebase to upstream 28stab093.2 kernel (security and bug fixes RHSA-2011:1065)
  • Ability to change NCPU on the fly
  • Ability to destroy LVE on the fly (via API)
  • Two modes to calculate load averages with ability to switch between them
  • Fix for OOM/hanged task issue
  • IO Priorities
# yum install kernel

If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

To change NCPU on the fly, use:
# lvectl set LVE_ID --ncpu N --force
--force will cause NCPU change on the fly. Please, don't use that option with kernels prior to lve0.8.42, as it can crash the system

Load Averages
CloudLinux has a modified way to calculated load averages, as processes can wait on CPU because of LVE limits, and not because of lack of CPU resources. Previously, our LA algorithm was ignoring uninterruptable processes. You can now switch to another LA algorithm that accounts for uninterruptable processes by running:

sysctl -w kernel.full_loadavg=1


Switching to that mode will cause higher load averages during high IO activity intervals. This should be useful on cPanel servers that have high IO Wait without high load average

You can always switch back by running:
sysctl -w kernel.full_loadavg=0
IO Priorities

While we still planning to release IO limits by the end of this year, this release introduces IO priorities. Each LVE is set to have IO priority of 100 by default (highest possible).
You can lower that priority, causing particular LVE to be de-prioritizes IO wise. This means that if there is lack of resources on the server, that LVE will get less IO operations then other LVEs.
IO Priorities work only with CFQ IO scheduler.



Beta: New Kernel for CloudLinux 5.x: 2.6.18-374.3.1.el5.lve0.8.44

New kernel 2.6.18-374.3.1.el5.lve0.8.44 is available from our cloudlinux-updates-testing repository. It includes everything from lve0.8.43 kernel as well as:
  • Rebased to upstream kernel 028stab094.3 including security and bug fixes RHSA-2011:1212
# yum install kernel-2.6.18-374.3.1.el5.lve0.8.44 --enablerepo=cloudlinux-updates-testing
If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

Beta: InterWorx LVE Manager plugin 0.1-2

LVE Manager plugin for InterWorx is available. It fixes the issue when resellers were able to see LVE Manager.
To update:
# yum update interworx-lvemanager --enablerepo=cloudlinux-updates-testing
Special thanks to InterWorx team for providing the patch.


New Beta Kernel for CloudLinux 5.x: 2.6.18-374.el5.lve0.8.43

New kernel 2.6.18-374.el5.lve0.8.43 is available from our cloudlinux-updates-testing repository. It includes everything from lve0.8.42 kernel as well as a bug fix for a bug that was present since lve0.8.42 kernel
  • Fix for lve_list_next race condition
# yum install kernel-2.6.18-374.el5.lve0.8.43 --enablerepo=cloudlinux-updates-testing
If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel




New Beta Kernel for CloudLinux 5.x: 2.6.18-374.el5.lve0.8.42

New kernel 2.6.18-374.el5.lve0.8.42 is available from our cloudlinux-updates-testing repository. The kernel brings changes from upstream kernel, as well as number of new features and improvements, including ability to switch number of cores per LVE without reboot and IO priorities. IO Limits will be implemented in the future versions.
  • Rebase to upstream 28stab093.2 kernel (security and bug fixes RHSA-2011:1065)
  • Ability to change NCPU on the fly
  • Ability to destroy LVE on the fly (via API)
  • Two modes to calculate load averages with ability to switch between them
  • Fix for OOM/hanged task issue
  • IO Priorities
  • fs.proc_super_gid was added to specify group of users that can see all processes
# yum install kernel-2.6.18-374.el5.lve0.8.42 --enablerepo=cloudlinux-updates-testing
If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

To change NCPU on the fly, use:
# lvectl set LVE_ID --ncpu N --force
--force will cause NCPU change on the fly. Please, don't use that option with kernels prior to lve0.8.42, as it can crash the system

Load Averages
CloudLinux has a modified way to calculated load averages, as processes can wait on CPU because of LVE limits, and not because of lack of CPU resources. Previously, our LA algorithm was ignoring uninterruptable processes. You can now switch to another LA algorithm that accounts for uninterruptable processes by running:
sysctl -w kernel.full_loadavg=1
Switching to that mode will cause higher load averages during high IO activity intervals. This should be useful on cPanel servers that have high IO Wait without high load average. You can always switch back by running:
sysctl -w kernel.full_loadavg=0

IO Priorities
While we still planning to release IO limits by the end of this year, this release introduces IO priorities. Each LVE is set to have IO priority of 100 by default (highest possible). You can lower that priority, causing particular LVE to be de-prioritizes IO wise. This means that if there is lack of resources on the server, that LVE will get less IO operations then other LVEs. IO Priorities work only with CFQ IO scheduler.

New kernel for CloudLinux 6.1: 2.6.32-231.6.1.lve0.9.8

New kernel 2.6.32-231.6.1.lve0.9.8 is available. The kernel fixes the issue with low default shared memory limits inside LVE that was causing issues for eAccelerator.

To update:
# yum update kernel

The confusion of CloudLinux memory limits

The memory limits in CloudLinux are confusing at best. First of all they count virtual memory allocated by processes, instead of physical memory. And virtual memory use can be much higher, as Linux is very efficient in using same physical memory for multiple processes. We plan to add physical memory limits in the future – yet, this is not the only issue with memory limits.

No matter if we limit physical or virtual memory, there will always be some guess work in detecting if the script error was due to memory limit, or if it was due to permissions, configuration errors or errors in the script itself. Such errors is the primary reason for us to ship CloudLinux with memory limits disabled by default. Memory limits are useful, and can often save server from overloading, swapping & going down. Yet, they can also add errors, that most sys admins don't connect to memory limits right away

When software (such as php interpretor or mod_fcgid daemon) tries to allocate memory from a system, LVE can prevent that from happening. It would do it same way OS would do it in the case when there not enough memory. Most applications when they try to allocate memory, and fail, they will fail as well. It would look pretty much as if failed due to bug, or some other error. The distinction is very small, and usually comes as part of cryptic error message and strange exit code. When it comes to website, such errors usually pop up as error 500 – which means that script used to serve the request failed due to some error. In this case it usually means that PHP interpretor failed (same way it would fail on bad php script). Basically – PHP or some other components fails, for whatever reason, and error 500 served. Not much for CL to do here.

Sometimes it gets even worth. Recently we got a customer who complained about mail() not working in php script. It was working before, but it stopped working after CloudLinux was installed. We knew that CloudLinux 'never' does something like that, and were totally baffled. It was verifiable error. Running php script that was trying to send email would come back with:
Warning: mail() [function.mail]: Could not execute mail delivery program '/usr/sbin/sendmail -t -i'

Switching back to CentOS kernel would solve the problem (that would disable LVE). It took us some time to stumble upon the fact that it might be memory limits. Once we did, it took a minute to verify it. There was enough memory to run php interpretor, but not enough for sendmail to run on top of it. Hence sendmail would fail, and php would deliver such message. Increasing memory limit removed the issue. There is an easy way to figure out if the issue relates to memory limits. All you need to do is to run:
# lveinfo --by-fault=mem --display-username


If you see user for which script failed in the list, it means that some script for that user hit memory limit within the past 10 minutes. Run the script again, re-check lveinfo (note, it takes 1 minute for it to update) – and you know for sure. Same information can be taken out of /proc/lve/list

Of course this is not enough, and we plan to do more. We want to create sophisticated notification system, so that both admin & user would be notified in case memory limits are reached. Additionally, we are researching the possibility to detect run time, on webserver level, when one of the processes that was used to serve up the request hit memory limit – and if we can intervene & serve our own error message in such cases. We are still at researching it – and if that would be possible, it would create a nice way to take out the confusion.

New kernel: 2.6.18-338.5.1.el5.lve0.8.29

New kernel 2.6.18-338.5.1.el5.lve0.8.29 is available from our cloudlinux-updates-testing repository. This kernel has following bug fixes and enhancements:

  • Fixes for gettimeofday() that was causing LVEs not to reach CPU limit
  • UBC limits not related to memory accounting were removed
To install new kernel, run:
# yum install kernel
If you have PAE, xen or Enterprise kernel -- use corresponding prefix, like: kernel-PAE, kernel-xen, kernel-ent instead of kernel

New beta lve-stats package with centralized data storage

New version (0.5-13) of lve-stats is available that brings centralized data storage for LVE Statistics. Instead of using local SQLite db on each server, new version can use central PostgreSQL server to store data.

More information is available here:
http://www.cloudlinux.com/docs/lvestats-0.5.13.php

Instructions on configuring PostgreSQL storage are available here:
http://www.cloudlinux.com/docs/lve_stats_postgresql_0.5-13.php

How to create custom error page 503 for LVE errors on cPanel server

There was some confusion on how to create custom error page on cPanel server. All that needs to be done are just few steps:


1. Create new file in /usr/local/apache/htdocs/ named 503.shtml (or .html, or .php). You can use some of the .shtml files that are already there as starting point. That page will be served on error.

2. Edit /etc/httpd/conf/httpd.conf, and add line:
 ErrorDocument 503 http://SERVER_IP/503.shtml 
3. Distill apache config:
# /usr/local/cpanel/bin/apache_conf_distiller --update 
4. Restart Apache

You are done.

Getting ready for lve 0.7 -- liblve & lve-utils upgrade

We are getting ready for LVE 0.7 release. LVE 0.7 will bring much better syntax for lvectl, /proc/lve/usage for usage/fault monitoring, and will allow us to move further.
Yet, to make the move save & easy for everyone -- we need people to upgrade their liblve & lve-utils packages.

# yum update liblve lve-utils

would do it. It would install:

liblve-0.6.1-1.el5.cloudlinux
lve-utils-0.3.1-1.el5.cloudlinux


The reason it is needed so that all programs that LVE continues to work fine after you upgrade kernel, but before you reboot.
It is also meant to make sure that LVE works after your kernel is updated vs KSplice, but before you update to new version of LVE (something we didn't have to do before).

If you don't upgrade -- it is not the end of the world, but make sure you schedule/reboot on LVE 0.7 upgrade once it comes.

And yes, we plan to release lve 0.7 next week.
* If you experience any issues with apache or lveps/lvectl -- run # ldconfig It should resolve it.

* We released update liblve-0.6.1-2 that does ldconfig, so majority of people shouldn't have this problem



New lvectl syntrax, comming soon

We are about to release new version of lvectl that would have much better syntax. The version should be backward compatible, but we hope people will start moving to new version asap

lvectl set <id> [–cpu <cpu>] [--maxEntryProcs <procs>] [--io <io>] --save

Initializes LVE or changes parameters of already initialized LVE.
If new LVE is initialized, and some parameters are missing – those parameters are set from specified defaults
If LVE already exist, and some parameters are not specified, those parameters are not changed.
If –save is specified, LVE configuration is created/updated in config file. Only specified parameters are updated/created. Missing parameters are skipped/not changed.

lvectl apply <id>

Initializes VE from config (ve.cfg) file. If VE already exists in memory – sets its parameters from ve.cfg file

lvectl delete <id>

Removes record for lve from ve.cfg file.
If LVE was is initialized, set its parameters to match defaults from ve.cfg

LVE Wrappers

Many people have asked me to provide LVE support for SSH users. We had LVE support for CGI scripts for a long time, and hosters see the benefit of being able to limit CPU usage of the scripts on per customer bases. Yet, once user connects to the server via SSH – there is no LVE limits any more. There is nothing preventing user starting to calculate PI to 1M digits – consuming as much CPU as it can get.

With LVE, all I had to do was create a “wrapper” program that server owner could place in /etc/passwd file. Instead of /bin/bash, end user would have /bin/lve_bash.

That program would enter into end user's LVE on logon, and start bash script. About 40 lines code later, I had a prototype – and I realized I could do more.

It is nice to be able to limit end user. Yet, what about Mail, Named, or MySQL for that matter? What if I want to put Apache inside LVE? This way I can say that it cannot use more then 95% of CPU – and I don't have to worry about not being able to get into the server once something goes wrong.

That would be very useful for web host, or anyone running any kind of server. Mail is known to spike & drag everything down – especially if you run antispam/antivirus filtering. It would be nice to say: “mail cannot use more then 30% of CPU (and IO once we have it).”

Any program can start consuming lots of resources – due to bugs, or due to high load. That would affect other services. Being able to control resources on per program (vs process) bases – would let you get much better stability out of your overall server infrastructure.

Anyway, about an hour later I had my generic wrapper. It comes in two forms. One would be used if you are starting daemon (or any program) as root. The other one can be used to start as any user, as long as that user is in lve group

For example, you can now do things like:

$ lve_wrapper make

That would run make (and all processes that it will spawn) in LVE belonging to you.

Or you can do this:

# lve_suwrapper 10001 /etc/init.d/postgresql start

That will start up postgres inside LVE with id 10001. After that you can see how much CPU PostgreSQL is using. You can also increase/decrease the amount of CPU PostgresSQL would get on the fly, at any moment.

I altered few init scripts -- just to see how easy it would be. In a few minutes -- I added ability to control LVE setting for PostgreSQL by just editing one config option in /etc/sysconfig. After that daemon would start inside LVE you would have specified.

Of course, you can use nice command to achieve something similar. Yet, nice is not as exact.

With nice you are setting priority on process level. So the more processes you have – the more CPU they can suck up under the same priority. There is also no way to control exact CPU usage.

What does nice 5 really mean? How much of a difference in CPU usage would it be if you would have one process with niceness level of 5 versus 30 processes?

And once we have IO limits and memory limits – the difference between LVE & nice will be even more profound.

It was pretty amazing how easy it was to do with LVE in the kernel. Now you can control resource usage when you compile a program, run a mail server, or convert from one video format to another.

This lets you achieve much better stability, and have much better control over what is happening on your server.

You can get LVE Wrappers now by installing lve-wrappers package
# yum install lve-wrappers
Documentation available here: http://www.cloudlinux.com/docs/lve_wrapper.php


Next, we need to add LVE into vixie-cron... :)