Technical articles

Better git log

Salut a tous, et particulièrement aux drupalistas et autres artisans du web que ça peut intéresser. Pour ceux qui ne me connaissent pas, voici mon profil sur StackOverflow. Pour ceux qui ne sont pas sur de bien connaitre le mot hacker, ou qui pensent carrément que je fais une erreur en employant ce mot, merci de lire hacker howto par Eric Steven Raymond.

Je cherche quelqu’un de débrouillard en terme de création de sites webs pour me remplacer, avec un minimum d’expérience en drupal - que je ne connais que depuis 2 mois donc je suis facilement remplaçable… Ce job est parfait pour quelqu’un d’un qui n’a pas peur des défis et qui prend son métier très à cœur un peu comme moi.

Il s’agit d’une startup parisienne très sympa, pour laquelle je travailles depuis principalement Valencia (Espagne) oú j’ai encore un appartement.

Bon payeurs, je travaille pour eux depuis Mars 2010. J’ai toujours été payé. Comme c’est bien payé et que je suis un excellent gestionnaire et que je vie dans la campagne en Espagne, il m’est arrivé d’encaisser 4 mois de salaires d’un coup, par choix.

Il y a des possibilités d’évolution. C’est payé entre 1800 et 2800 par mois selon les missions et l’expérience. Les collègues sont des gens exceptionnels avec des petits défauts bien sur comme tout le monde … mais surtout de grandes qualités ! Il suffit d’être un peu ouvert d’esprit.

L’emploi vient avec un Asus Zenbook qui appartient au client, c’est un ultra-portable dernier cri:

• 8Gb de ram,
• 256Gb de SSD,
• quad core i7,
• écran meilleur que HD,
• 5 heures d’autonomie,
• ultra fin, ultra léger, ultra portable.

C’est en télétravail et à plein temps. De plus, il y a des voyages à Paris presque tout frais payés plusieurs fois par an.

Je mets ma carierre en suspens pour des raisons personnelles. Je n’aurais plus du tout le temps de travailler sur internet.

Seems like ReactOS is rapidely growing.

Drupal is a CMS written in PHP which supports PostgreSQL. It is made for mod_php and Apache, thought it works with uWGSI and Nginx.

When you have tried uWGSI you know why you want this.

Example nginx configuration:

server {
    server_name drupal.example.com;
    root /srv/drupal/www/;

    error_log /tmp/nginx_drupal.log;

    index index.php index.html;

    location / { 
        try_files $uri @rewrite;
    }   

    location @rewrite {
        rewrite ^/(.*)$ /index.php?q=$1;
    }   

    location ~* files/styles {
        access_log off;
        expires 30d;
        try_files $uri @rewrite;
    }   

    location ~ .php$ {
        include uwsgi_params;
        uwsgi_modifier1 14; 
        uwsgi_pass unix:/tmp/uwsgi_drupal.sock;
    }   
}   

And an example uwsgi configuration:

[uwsgi]
socket=/tmp/uwsgi_drupal.sock
pidfile=/tmp/uwsgi_drupal.pid
daemonize=/var/log/uwsgi/drupal.log
plugins=php

chdir=/srv/drupal
cheaper=4
close-on-exec=1
harakiri=360
max-requests=128
processes=8
master=1
uid=drupal
gid=drupal
chmod=666
log-5xx=1
vacuum=1
post-buffering=8192

Just stumbled on django-parting which provides a nice API to partition your data.

For those not familiar with data partitioning, I recommend reading partitionning on mysql.

Check them out.

A very nice article about troubleshooting a server

Those are notes taken from the talk “PostgreSQL when it is not your job” by Christophe Pettus from PostgreSQL Experts Inc. at DjangoCon Europe 2012.

This article describes how to make a basic PostgreSQL configuration:

• logging,
• memory,
• checkpoints,
• planner.

Note: this article is mostly a transcript from the talk by Christophe Pettus: so send all the cookies to him. Thanks !

That’s around 12 configuration options and you’re done.

Starting with logging is a good idea because it’s the best way of getting information for configuring the other settings.

• be generous with logging; it’s very low-impact on the system,
• it’s your best source of information for finding performance problems

Three options:

• syslog, if you have a syslog infrastructure you like already,
• standard format to files - if you’re using tools that need standard format,
• otherwise, CSV format to files,

Christophe Pettus recommends this configuration, which you can copy and paste in your postgresql.conf:

log_destination = 'csvlog'
log_directory = 'pg_log
# enable file rotator
logging_collector = on
log_filename = 'postgres-%Y-%m-%d_%H%M%S'
log_rotation_age = 1d
log_retation_size = 1GB
# log statements that take more than 250ms
log_min_duration_statement = 250ms
log_checkpoints = on
log_connections = on
log_disconnections = on
log_lock_waits = on
log_temp_files = 0

Three things you have to tweak:

• shared_buffers,
• work_mem,
• maintenance_work_mem,

• below 2GB of RAM, set to 20% of total system memory,
• below 32GB, set to 25% of total system memory,
• avec 32GB, set to 8GB,

• start low: 32-64MB,
• then wait until there is some server load (simulated or real),
• look for ‘temporary file’ lines in logs,
• it will say “I’m creating temporary file of this size”,
• set work_mem to 2-3x the largest temp file you see.

This can be a huge spped-up if set properly ! This is probably the setting that has the most impact on postgresql performance !

But if you find yourself setting this super high like 8GB, you probably don’t want to set it that high. Because every time someone does an operation in postgres like a SORT, it could potentially use that amount of memory per planner.

If you’re having problems of memory exhaustion problems, this is the best setting to back off.

This is the amount of memory that PostgreSQL uses for indexing or vacuum operations.

• 10% of system memory, up to 1GB,
• Maybe even higher if you are having VACUUM problems,

Unlike other parameters we’ve seen, this one won’t actually allocate any memory.

• set to the amount of file system cache available,
• if you don’t know, set it to 50% of total system memory,

• A checkpoint is a complete flush of dirty buffers to disk.
• Potentially a lot of I/O (Input/Output),
• Done when the first of two thresholds are hit:
  • A particular number of WAL (Write-Ahead-Log) segments have been written … basically you don’t have to worry about that, it’s when a certain amount of database activity has happened.
  • A timeout occurs.

# that's the right value according to Chripstophe
wal_buffers = 16MB
checkpoint_completion_target = 0.9
checkpoint_timeout = 10m-30m, # Depends on the restart time

The trade off about checkpoint_timeout is: the higher it goes, you’ll generally get better performance, but the longer it will take postgresql to restart after a crash.

• Look for checkpoint entries in the logs
• Is it happening more often than checkpoint_timeout ?
• Then adjust checkpoint_segments so that checkpoints happen due to timeouts rather than filling segments,

• The WAL can take up to 316MBcheckpoint_segments on disk, so the higher that number goes the more space on disk it will be using … but on modern disks it’s not really that much disk,
• Restarting PostgreSQL can take up to checkpoint_timeout (but usually less, like 20% of that),

• effective_io_concurrency, set to the number of I/O channels; otherwise ignore it. For example if you have a stack of hard drives or an SSD drive that has 32 channels then set it to the number of channels.
• random_page_cost, 3.0 for a typical RAID10 array, 2.0 for a SAN, 1.1 for Amazon EBS, else don’t tweak it.

The new brew of Foundation 4 is out, focusing on mobile first. Worth trying.

dropzone.js is an open source library that provides drag’n’drop file uploads with image previews

Nowadays, virtualenv uses --no-site-packages by default. This means that the created virtualenv will not access global site-packages modules ie. /usr/lib/python2.7/site-packages/.

To enable global site-packages again, just remove this file::

your_env/lib/python2.7/no-global-site-packages.txt

Why would you do that ?

Some modules are complicated to install (xpyb) or take too much time to compile (pyqt). Using the distro packages can be pretty useful !

Allison from hacker school published an interesting article about virtualenv for beginners.

Please upgrade.

TL;DR: Intel implements UDEREF equivalent 6 years after PaX, PaX will make use of it on amd64 for improved performance.

Read article ….

Linux 3.8 was released !

SourceTree is a free client for Mercurial and Git. The windows release has been announced.

Vim is a great text and code editor for command line interface. OpenVim presents an interactive tutorial which seems nice for new beginners.

For those using jedi-vim are indirectly using rope refactoring library.

If you are also using a home-level virtualenv ie. in ~/env/ then your vim instance might take crazy long time.

First things first, rope creates a .ropeproject directory where it thinks the project root is. And it will scan every file in every sub directory. A quick and dirty solution is to create empty .ropeproject folders lower in the $HOME filesystem. Rope will automatically use this folder if it finds no other .ropeproject folder at a lower level.

But, if you want a home-level .ropeproject, then you should configure ~/.ropeproject/config.py. It is well commented, you might end with something like this:

# Custom source folders:  By default rope searches the project
# for finding source folders (folders that should be searched
# for finding modules).  You can add paths to that list.  Note
# that rope guesses project source folders correctly most of the
# time; use this if you have any problems.
# The folders should be relative to project root and use '/' for
# separating folders regardless of the platform rope is running on.
# 'src/my_source_folder' for instance.
prefs['source_folders'] = ['env/src']

# You can extend python path for looking up modules
prefs['python_path'] = '/home/jpic/env/lib/python2.7/site-packages/'

Playlistnow.fm is a social network based on music made in Python/Django. The catalog is powered by last.fm and the videos come from youtube.

Before its final death, we decided to release the source code under Apache 2.0 license.

I’m posting this because it took me quite a few hours to get to this point:

Ghost.py seem to have a nicer API and to be faster than selenium so far.