iurisilvio

Faster Django migrations with django-migrations-ci

2023-01-09T00:00:00-03:00

Django migrations are really slow. It is an open issue for years. It was even an Google Summer of Code proposal.

For small projects, it take some seconds to run all your migrations and it is fine. A few developers can work on it for a long time. For a large project, it is not that simple.

Our largest project today has 273 Django models and did ~1000 migrations in 4 years. Sometimes it takes 15 minutes on CI, even with a lot of optimizations. Django squashmigrations command do optimizations, but could be a lot better. After a lot of squashes, it's still slow.

It was really bad for us, CI failed a lot because of timeouts or random postgres failures after some time trying to always migrate.

We tried some things to improve our CI pipeline.

Do not migrate on CI

A common advice to improve this time is to not run migrations on CI. It is fast.

You can override the MIGRATIONS_MODULES for testing:

MIGRATION_MODULES = {app.rsplit(".")[-1]: None for app in INSTALLED_APPS}

It is not really safe, because now you run migrations only on deployments, which is risky. Large projects have data migrations, migration bugs and other issues that you want to discover before production. So we added another stage to run migrations and our problem was back.

Reuse database

In the past I had to hack pytest-django setup to run less migrations. This was useful to know how to separate migrations and test steps.

I changed our pipeline to dump the migrated database to a SQL file and cached it on GitLab CI.

Some old code examples from our CI:

# Don't try to copy it, please!
if [ -f $CI_PROJECT_DIR/djangomigrations.sql ]; then
    psql -h $DB_HOST -U $POSTGRES_USER -c "CREATE DATABASE $DB_NAME;"
    pg_restore -h $DB_HOST -U $POSTGRES_USER -d $DB_NAME $CI_PROJECT_DIR/djangomigrations.sql
else
    time ./manage.py setup_test_db
    pg_dump -F c -h $DB_HOST -U $POSTGRES_USER $DB_NAME > $CI_PROJECT_DIR/djangomigrations.sql
fi
time ./manage.py clone_test_db

Commands setup_test_db and clone_test_db were just a few lines of Python code, but I'll not paste it here.

# GitLab CI cache work this way.
cache:
  key:
    files:
      - "requirements.txt"
      - "*/migrations/*.py"
  paths:
    - djangomigrations.sql
  when: always

Perfect! Running a migration state that already ran in the past, GitLab CI give me an SQL file with this state and I restore it to my database. It took our database migrations from minutes to just a few seconds. Works for me!

django-migrations-ci

I went to DjangoCon US to talk about django-qserializer. There I had some conversations about how migrations are slow. More people had the same issue, so I started django-migrations-ci during sprint days of the event.

Using this module, it is just one extra command to setup your test database:

./manage.py migrateci
./manage.py test --keepdb

Following my original idea, it started using CI caching to provide a storage layer.

I implemented it to sqlite3, mysql and postgres. Never tried to implement for oracle, but it is easy to add other databases. Also, I documented how to integrate it with GitHub Actions and GitLab CI.

Partial caching

After this initial sprint, I replaced the CI caching with a custom storage, easy to integrate with anything django-storages support.

I did that to support partial migrations. It was impossible to do it with CI caching, because they don't have an API to easily choose the cache I want.

When a new migration is added, there is no migration state cached, so all migrations are processed again. To fix this use case, I wanted to get a previous state, restore from it and run only new migrations, which is reasonably fast.

Use it

Now I can say it is easy to reuse my solution and it really adds value to any project where your migrations take more than a few seconds.

Install it, configure an external storage and add a command to your CI scripts.

from storages.backends.s3boto3 import S3Boto3Storage

class MigrateCIStorage(S3Boto3Storage):
    bucket_name = "mybucket-migrateci-cache"
    region_name = "us-east-1"

$ ./manage.py migrateci --storage-class path.to.MigrateCIStorage

Meu tempo minhas regras

2022-11-04T00:00:00-03:00

Como dev e como manager, já tive essa conversa sobre gerenciamento de tempo e priorização algumas vezes, então decidi documentar ~para economizar tempo~. Meu foco aqui são os líderes técnicos, mas usem como quiserem, tem ideias genéricas que podem ser aplicadas em outros contextos. Advice Monster

Como líder, você tem responsabilidades extras, mas a expectativa é que você continue programando, com impacto crescente no que você faz. A conta não fecha. Gerenciar essa expectativa e todas as responsabilidades é o desafio e você vai se cobrar por não conseguir fazer tudo.

Essas são minhas ideias básicas para facilitar decisões de gerenciamento de tempo pessoal. Suas decisões impactam o resultado da empresa, mas a base de tudo é pensar no melhor uso do seu tempo.

Faça o time não depender de você

Seu objetivo principal como líder é contribuir para o desenvolvimento do time, chegando no limite em que o time já roda sem você. Esse é o cenário ideal, que te deixa livre para pensar em problemas maiores.

Para chegar aí, é bastante tempo de acompanhamento e mentorias. Entenda onde cada um quer se desenvolver e libere seu tempo delegando a responsabilidade para o time, de preferência alinhando com a expectativa de desenvolvimento de cada liderado.

Não limite as tarefas ao que cada um já sabe. Se você já sabe fazer algo bem, deixar alguém fazer é uma oportunidade excelente de aprendizado. Nesse caso, use seu tempo para ensinar, não para executar. Para isso funcionar, incentive a autonomia, desapegue da solução. É muito provável que não fique no mesmo nível de qualidade que você faria ou até que seja uma solução bem diferente. Use isso para novas rodadas de aprendizado, mas não tente assumir a solução para fazer do seu jeito. Você vai se surpreender e ficar orgulhoso quando seus liderados fizerem uma solução que você não pensou e é melhor do que você faria.

Seu objetivo é se escalar

Como contribuidor individual, você tem tempo limitado, ~8h/dia. Liderando um time, seu tempo disponível para executar é ainda menor. Você precisa usar seu time para escalar sua capacidade e conhecimento, aumentando seu impacto total. Para isso, você precisa:

Contribuir com o aprendizado do time.
Saber quando você precisa implementar a solução.
Saber quando é melhor só desenhar a ideia ou definir a solução.
Saber quando você pode só deixar para outro resolver o problema.
Saber amplificar os resultados e comportamentos positivos do time, deixando claro o que você valoriza. Isso direciona seus liderados e pessoas próximas para o caminho que você quer que aconteça mais. Why Great Leaders Are Amplifiers

Quando for alçado para uma posição de liderança, é comum achar que não está sendo produtivo por ter menos tempo para programar. Quando somos apenas devs, atrelamos nossa medida de produtividade a entregas feitas por nós. Na posição de liderança, você deve medir o impacto e funcionalidades entregues pelo time, esse é sua grande métrica. Se você fizer cada pessoa de um time de 10 pessoas render 10% a mais, sua produtividade terá aumentado 100%.

Deixe cair alguns pratos

Você tem muito mais trabalho possível do que tempo. Saber quais são os maiores impactos ajuda na priorização, mas só isso não é suficiente. Você ainda acaba com muitos trabalhos urgentes e prioritários. Foque no que faz mais sentido, delegue para o time o que der e comunique o que não for possível fazer. Está tudo bem, alinhe com os envolvidos e gerencie as dores maiores.

Comunique muito

É sua responsabilidade tomar decisões. No seu time ninguém tem mais maturidade e conhecimento do que você para isso e esperam a ação vinda de você, mesmo que estejam todos de acordo. Mais difícil do que tomar a decisão é garantir que todos entendam o que foi feito e os motivos.

Consulte quem é impactado e quem sabe mais. Execute e comunique para todos a ação e os motivos. Isso reduz retrabalho e os pontos de contato para explicar a decisão. Isso aumenta a confiança de todos em você. Documentação boa é comunicação eficiente.

Não se compare com os outros

Olhe para a sua evolução. Cada um tem uma velocidade de aprendizado, um tempo disponível diferente. Se comparar só gera frustração, sempre tem alguém melhor que você. Aprenda com eles, mas não compare seus resultados.

Tem quem trabalhe mais que 8h/dia, quem tem tempo para estudar a noite toda ou já carrega o conhecimento de vários anos. O resultado atual importa bem menos do que o seu potencial de crescimento e como você aumenta seu impacto com o tempo.

Saiba dizer não

Proteja seu tempo. Seu tempo de trabalho é limitado, foque nas prioridades já definidas e fuja da reunião com assunto pouco definido. Incentive a conversa/documentação assíncrona para facilitar a reunião futura.

Mudanças de prioridade acontecem e quando algo urgente surgir, redefina datas acordadas para outras entregas, alinhe o que é mais prioritário e estabeleça novos prazos de acordo com a nova realidade. Tentar ser o herói e assumir mais responsabilidades do que pode não é sustentável no médio/longo prazo.

Seu tempo fora do trabalho é também limitado. Contribua com o que for necessário, mas saiba diferenciar o que é uma urgência do que é um problema que pode esperar o dia seguinte ou segunda-feira.

Não se compare, tem quem pode dedicar mais tempo, tem quem pode estudar mais, tem quem pode estar online 24/7 e é muito fácil cair na armadilha de querer seguir a rotina dos outros. Intensidade é necessária, mas construa uma rotina saudável, para fazer isso por bastante tempo.

Blocos de trabalho no calendário

Defina blocos sem reuniões. Focus time que pode ser usado para qualquer tarefa individual. Escrever código, escrever documentação, revisar código do time, etc.

Incentive seu time a ter blocos assim também e respeite esse tempo. Você já tem isso informalmente nos squads. Por exemplo, a daily pode ser sempre a tarde para deixar a manhã sem interrupções. Levando isso para o calendário, você incentiva outros times a respeitarem esse tempo também.

Nem sempre é possível, às vezes vão passar por cima do seu bloco de tempo porque as agendas não batem, mas é um sinal para orientar quem precisa falar com você.

Maker's schedule

Reduza o tempo de troca de contexto

Anote todos seus contextos. Reuniões, conversas de corredor, dúvidas genéricas, escrita de código, revisão de código, impossível manter tudo na cabeça. Escreva para consolidar suas ideias e ter onde consultar quando não souber mais o que está acontecendo.

Você muda muito de contexto e o tempo de recuperação da troca de contexto é alto. Mesmo com toda otimização de tempo feita, é inevitável ter interrupções.

Quando precisar largar o código pela metade, sempre anote qual é o próximo passo. Um comentário no código é suficiente para você saber exatamente onde parou e o que precisa fazer na sequência. Se a interrupção for um pouco mais planejada, pare com um teste quebrado, que o próximo passo é implementar a correção para ele. A recuperação de contexto é muito mais rápida assim.

Simon Willison DjangoCon US 2022 talk

Qualidade é velocidade

Débito técnico custa caro e tem juros. Aumentar qualidade e reduzir bugs te permite avançar mais rápido. Ter menos bugs para se preocupar é mais tempo para trabalhar na evolução do produto.

Faça um esforço consciente para manter a cultura de qualidade. O ganho direto é um código melhor, mas os efeitos secundários são muito maiores e acumulam. As próximas entregas são mais rápidas, o aprendizado acumulado do time é maior por trabalhar com exemplos cada vez melhores, a velocidade é maior por existir muito mais confiança no sistema de trabalho.

Django Sessions to cache

2021-04-28T00:00:00-03:00

Django sessions works great since the begining, you can do a site with them and never really understand how sessions works, but you can't scale to thousands of simultaneous users writing all your sessions to database. The database can be a huge bottleneck here and you should remove all unnecessary reading and writing.

Fortunately, Django has good alternatives almost built-in. They provide other session backends, like cached_db and cache. It has extensive docs, but I had some issues migrating to cache backend.

Doing that in production means current sessions can't be removed during migration.

I decided for the easy path:

Change SESSION_ENGINE='django.contrib.sessions.backends.cached_db'.
Wait some days to have all useful sessions cached. More than SESSION_COOKIE_AGE was enough to me.
Change SESSION_ENGINE='django.contrib.sessions.backends.cache'.

It doesn't work.

Both session engines define keys as key_prefix + session_id, with different prefixes. Django do that to avoid namespace clash. It is a sane default, but I want namespace clash to simplify the migration path.

I can define custom session backends with the key prefixes I want. Each one should be a file because SESSION_ENGINE must be a module with a SessionStore.

# base.py
class BaseSessionStoreMixin:
    cache_key_prefix = 'busersessionstore'

# cache.py
from django.contrib.sessions.backends import cache

from . import base


class SessionStore(base.BaseSessionStoreMixin, cache.SessionStore):
    pass

# cached_db.py
from django.contrib.sessions.backends import cached_db

from . import base


class SessionStore(base.BaseSessionStoreMixin, cached_db.SessionStore):
    pass

Now I can do the same thing, using my session engines.

Change SESSION_ENGINE='myproject.sessions.cached_db'.
Wait some days to have all useful sessions cached. More than SESSION_COOKIE_AGE was enough to me.
Change SESSION_ENGINE='myproject.sessions.cache'.

It works!

Just as disclaimer, memcached don't have a good way to make this data durable. If memcached node restart, all data is lost. We're using redis with replicas, so I lose all data only if all nodes explode at the same time.

My Django contributions

2021-03-17T00:00:00-03:00

I started contributing to Django core after a failure in our project.

I had to rename a model attribute without touching our database. The solution was simple.

# Simple attribute to rename.
author = models.ForeignKey('Author')

# 1. Add db_column param.
author = models.ForeignKey('Author', db_column='author')

# 2. Change the attribute name.
creator = models.ForeignKey('Author', db_column='author')

It was really obvious to me that it should be a noop to database. Even Stack Overflow had a response confirming that.

The first migration locked our database because it dropped and recreated a constraint.

SET CONSTRAINTS "core_book_author_id_eaa1580d_fk_core_author_id" IMMEDIATE;
ALTER TABLE "core_book" DROP CONSTRAINT "core_book_author_id_eaa1580d_fk_core_author_id";
ALTER TABLE "core_book" ADD CONSTRAINT "core_book_author_id_eaa1580d_fk_core_author_id"
    FOREIGN KEY ("author_id") REFERENCES "core_author" ("id") DEFERRABLE INITIALLY DEFERRED;

Sad day, but right after that I prepared to run the second step. That time I checked the generated SQL with sqlmigrate command and discovered it will do the same thing again.

To avoid another failure, I did only a SeparateDatabaseAndState change. Works great, but Django could do better handling these migrations.

from django.db import migrations, models

class Migration(migrations.Migration):
    atomic = False
    dependencies = [
        ('core', '0137_auto_20200227_1304'),
    ]
    operations = [
        migrations.SeparateDatabaseAndState(
            state_operations=[
                migrations.RenameField(
                    model_name='book', old_name='author', new_name='creator',
                ),
            ],
            database_operations=[],
        ),
    ]

After that, I checked Django tickets and didn't found anything related. Created a ticket and implemented two pull requests to fix both steps: AlterField with db_column addition should be a noop and RenameField with db_column should be a noop. They were merged and released with Django 3.1.1 after long PR conversations.

Django is a huge codebase and even a minor change like that took me some days, changing the wrong things in the different wrong ways. I learned a lot and was able to fix another migrations issue: AlterOrderWithRespectTo() with ForeignKey crash when _order is included in Index().

Right after these fixes, I upgraded our Django and hit a recent regression: App with default_app_config and without apps.py or with an empty apps.py crashes.. It was a quick fix, but lots of tests written.

When our new intern started exploring Django querysets, he had an issue with queryset in_bulk operation, because of incomplete docs and it generated a new ticket and quick fix: Queryset in_bulk docs don't explicit field_name as kwarg only.

Faster parallel pytest-django

2021-03-08T08:00:00-03:00

We at Buser recently migrated from Django unittests to pytest. It is amazing how much you can improve your tests with pytest.

Unfortunately our CI/CD was really slow after this change.

I discovered parallel pytest-django do their setup different than Django unittests. Django create a template database and copy it to every test node, pytest-django create one database per node. Running this setup with migrations took more than 2 minutes just to have the database created.

This is a problem mostly for CI/CD, because local you can run with --reuse-db and the overhead is minimal, only when you have database migrations.

I wanted to fix this in pytest-django, but I discovered it is difficult because pytest-xdist don't have a global setup and teardown.

Now I have a small hack, working great so far. This is the script running our tests:

./manage.py shell -c "
from django.test.utils import setup_databases
from xdist.plugin import pytest_xdist_auto_num_workers
setup_databases(verbosity=True, interactive=False, parallel=pytest_xdist_auto_num_workers())
"
pytest --reuse-db -n auto

Django unittests generate names like test_yourdb_1, test_yourdb_2 and pytest-django generate names like test_yourdb_gw0, test_yourdb_gw1, their names don't match, I had to make a fixture to monkey patch their behaviors.

from pytest_django.fixtures import _set_suffix_to_test_databases
from pytest_django.lazy_django import skip_if_no_django

# Original code: https://github.com/pytest-dev/pytest-django/blob/bd2ae62968aaf97c6efc7e02ff77ba6160865435/pytest_django/fixtures.py#L46
@pytest.fixture(scope="session")
def django_db_modify_db_settings_xdist_suffix(request):
    skip_if_no_django()

    xdist_suffix = getattr(request.config, "workerinput", {}).get("workerid")
    if xdist_suffix:
        # 'gw0' -> '1', 'gw1' -> '2', ...
        suffix = str(int(xdist_suffix.replace('gw', '')) + 1)
        _set_suffix_to_test_databases(suffix=suffix)

I tried to make a pull request to pytest-django, but this issue exists because pytest-xdist don't have hooks to run before and after every node.

Honeycomb Django tricks

2021-01-10T08:00:00-03:00

Honeycomb is the observability tool we use everyday at Buser. They help us tracking application bottlenecks, slow database queries, slow requests, requests with too many queries and much more.

Their automatic instrumentation with beeline is a good start, but at some point we had to extend the default behaviour.

Custom request data

We extended the HoneyMiddleware and changed settings.py MIDDLEWARE to use our custom middleware because we wanted to track differences between logged and anonymous users.

from beeline.middleware.django import HoneyMiddleware

class HoneycombMiddleware(HoneyMiddleware):
    def get_context_from_request(self, request):
        context = super().get_context_from_request(request)
        if request.user.is_authenticated:
            context['request.user.id'] = request.user.id
        return context

We could add context['request.user.is_authenticated'] too, but we did that with a derived column EXISTS($"request.user.id") in Honeycomb side.

Trace sampling

Honeycomb limits are really friendly, even for their free plan, but our traffic increased 5x in 2020 and we reached our plan limits.

Their beeline client has a sample_rate config but it's a dummy implementation, generating imcomplete traces, which is useless to us.

We make the sampling decision at the start of the request, unfortunately it's not possible to decide based on request duration or request status.

from beeline.middleware.django import HoneyMiddleware

class HoneycombMiddleware(HoneyMiddleware):
    def __call__(self, request):
        sample_rate = self.get_sample_rate(request)
        sampled = random.random() < 1 / sample_rate
        request.__honeycomb_sampled__ = sampled, sample_rate
        return super().__call__(request)

    def get_sample_rate(self, request):
        # Add logic here to decide based on path or other request info.
        return 10

Now, to sample based on this __honeycomb_sampled__ info, beeline need a sampler hook.

import beeline

def sampler_hook(event):
    # The get_request is not Django built-in.
    # Check django-middleware-global-request project.
    request = get_request()

    # Ignore when we don't have a request.
    if not request:
        return False, 0

    return request.__honeycomb_sampled__


beeline.init(sampler_hook=sampler_hook)

Request based sampling

Instead of a fixed sample rate, we have it configured by path and by domain to get better results. Our implementation use Django settings, but it is easy to start handling just special cases.

def get_sample_rate(self, request):
    if request.get_host() == 'admin.example.com':
        # Always trace admin domain, it has low volume but lots of slow batch requests.
        return 1
    if request.path == '/search':
        # Reduce high volume path sampling.
        return 80

    return 10

Celery

Our project ran only with Django integration for almost a year, focused on critical requests optimizations. We had to offload lots of application work to Celery to handle some bottlenecks, but after that we created a huge blindspot in the stack.

Celery instrumentation

Again, beeline has a nice Celery base implementation, but it doesn't handle sampling well and don't instrument Django database queries.

Maybe it's a small bug, based on our use, Celery queue name is in delivery_info['routing_key'] instead of delivery_info['exchange']. We decided to log both values to not lose useful data.

I reused ideas from their middleware.

from contextlib import ExitStack

from beeline.middleware.django import HoneyDBWrapper
from celery.signals import task_prerun, task_postrun
from django.db import connections

@task_prerun.connect
def setup_django_db(task):
    task.request._exit_stack = ExitStack()
    db_wrapper = HoneyDBWrapper()
    for connection in connections.all():
        task.request._exit_stack.enter_context(connection.execute_wrapper(db_wrapper))

@task_postrun.connect
def teardown_django_db(task):
    task.request._exit_stack.close()

Celery sampling

Celery sampling was a challenge, because Celery internals don't have good documentation.

import random
import celery

from celery.signals import task_prerun

def sampler_hook(event):
    return celery.current_app.current_worker_task.request.__honeycomb_sampled__

def sampling(sample_rate):
    sampled = random.random() < 1 / sample_rate
    return sampled, sample_rate

@task_prerun.connect
def setup_sampling(task):
    task.request.__honeycomb_sampled__ = sampling()

The beeline.init now need a sampler_hook.

import beeline
from celery.signals import worker_process_init

@worker_process_init.connect
def initialize_honeycomb(**kwargs):
    beeline.init(sampler_hook=sampler_hook)

uWSGI integration

uWSGI integrations was probably our first small issue.

Package uwsgidecorators is available only inside uwsgi context and we wanted to run in development too. The _init_beeline in the code is the docs init_beeline. When it's not in uwsgi context, it setup beeline too.

def init_beeline():
    try:
        import uwsgidecorators
    except ImportError:
        _init_beeline()
    else:
        uwsgidecorators.postfork(_init_beeline)

Nginx quick configuration

2015-05-31T00:00:00-03:00

Nginx is an HTTP server with lots of features. The configuration is easy, but maybe you don't even know the feature exist.

I want to present some nginx features and give you some pointers to continue your research.

The basics

The basic usage of nginx is to make the reverse proxy to your application. Nginx listen to port 80 waiting for requests and manage the connections, passing the requests to your upstream application server.

server {
    server_name www.yoursite.com;
    listen 80;
    location / {
        proxy_pass http://localhost:8080;
    }
}

The proxy_pass directive proxy the request as HTTP to your application listening port 8080.

You can use uwsgi_pass to make nginx convert the HTTP to a WSGI request and send it to your WSGI application.

server {
    server_name www.yoursite.com;
    listen 80;
    location / {
        include uwsgi_params;
        uwsgi_pass localhost:8080;
    }
}

Most features work for HTTP and uWSGI proxy, with some small configuration changes.

SSL Termination

Nginx terminate the SSL connection and send it to your application as plain HTTP.

ssl_certificate /opt/ssl/www.yoursite.com.crt;
ssl_certificate_key /opt/ssl/www.yoursite.com.key;
server {
    server_name www.yoursite.com;
    listen 443 ssl;
    location / {
        proxy_pass http://localhost:8080;
    }
}

If you want a HTTPS only website, add another server block to redirect the HTTP traffic to HTTPS.

server {
    listen 80;
    server_name www.yoursite.com;
    return 301 https://$host$request_uri;
}

Static files

Nginx serve some thousands of static files per second, without hiccups. Do not make your application serve static files, like images and javascripts. Your application is slow, maybe it'll re-render the same asset for each request.

server {
    server_name www.yoursite.com;
    listen 80;
    root /some/path;
    location /static/ {
        try_files /generated/$uri /cache/$uri @myapp;
    }
    location / {
        try_files $uri @myapp;
    }
    location @myapp {
        include uwsgi_params;
        uwsgi_pass localhost:8080;
    }
}

The try_files directive in /static/ block will check /some/path/generated/static/ and /some/path/cache/static/ for your requested file and if it does not exist, send the request to your application.

You can change the last try_files parameter with =404 to answer with a 404 instead of pass the request to application.

Load balancing

If you want a high available application, nginx can be your load balancer to distribute the load and handle gracefully server failures.

upstream yourapp {
    server http://localhost:8080 weight=5;
    server http://localhost:8081;
    server http://example.com:8080 backup;
}
server {
    server_name www.yoursite.com;
    listen 80;
    location / {
        proxy_pass yourapp;
    }
}

Nginx send the request to your yourapp upstream, choosing one server in a weighted round robin way. 5 requests to the first server, 1 request to the second server and so on. If your servers are down, it sends the request to your backup server.

If one server fail to answer or give an HTTP error, nginx send the request to the next server. No additional configuration needed.

Caching

Some pages are almost static and you don't want it rendering all the time. Nginx can help you serving this content. Configure nginx to cache the page for 10 minutes.

uwsgi_cache_path /tmp/myapp/content/ keys_zone=myapp-content:10m
                 loader_threshold=300 loader_files=200 max_size=100m levels=1:2;
uwsgi_cache_valid 200 301 302 404 10m;
uwsgi_cache_key $host$request_uri;

server {
    server_name www.yoursite.com;
    listen 80;
    location /content/ {
        uwsgi_cache yourcache;
        uwsgi_ignore_headers Set-Cookie;
        uwsgi_hide_header Set-Cookie;
        add_header X-Cache $upstream_cache_status;
        include uwsgi_params;
        uwsgi_pass localhost:8080;
    }
    location / {
        include uwsgi_params;
        uwsgi_pass localhost:8080;
    }
}

Dynamic upstreams

The commercial subscription has this feature built-in, but you probably don't want one. It costs some thousand dollars per server.

The simple way to do it is update the configuration file, adding/removing upstream servers. The nginx reload command update the configuration without downtime.

Conclusion

These snippets are just the basics. Use it to understand how it can help you, but check the docs to learn all the features. Nginx is really powerful and can help you to simplify and improve your system architecture.

Feel free to contact me if you want some help setting up your nginx server. I'm not a system administrator, but I learnt some things about it.

Funblocker post mortem

2014-11-28T00:00:00-02:00

Long time, no text. I'll try to increase the frequency of my posts and don't give up because of the lack of time and excess of lazyness.

One of my first open source projects with any traction was Funblocker. I wrote about it in other posts, it was a Chrome extension to help remove all that stupid reshares from your Facebook timeline based on link context.

Funblocker was created in december 2011, had more than 3k active installations, removing more than 400k posts/month. It still has some users and remove 10k posts/month and has 1k active installations.

I created it as a side project, started as a JavaScript snippet for personal use, injected with Custom JavaScript for websites extension. It was useful for some job interviews and for my timeline sanity. Also, I learnt how to write a Chrome extension.

Fortunately, Facebook improved their ways to avoid these posts with built-in features, so this extension is almost useless and it is some abandonware now. My last commit is from nov/2013 and I have some bug issues reported and never fixed.

Check the Google Analytics events below, counting basically posts removed.

For historical reasons, the FunBlocker extension is still available and FunBlocker code is on GitHub.

24pullrequests post mortem

2013-01-08T00:00:00-02:00

During last december, I attended to 24pullrequests.com, a project to engage people to make one pull request per day until Christmas.

After it, I want to know how much a simple gamification over a common task can improve the results. Yes, I always helped some projects with pull requests, issues, code reviews, but during 24pullrequests project, I increased my work in GitHub open source projects.

So, I used data from GitHub to understand the impact of 24pullrequests. My numbers does not match public 24pullrequests numbers, but I did it based in GitHub data and I expect it is correct (check andrew/24pullrequests#236).

First, the basic numbers:

2724 developers registered

642 developers with at least one pull request

2773 pull requests opened

1299 projects with at least one pull request

Ok, these numbers say nothing about 24pullrequests impact. Maybe people just did a lot of pull requests and 24pullrequests has nothing with it. Maybe people are creating useless pull requests just to "win" the game.

I'm sure it is not the best comparison, but I compared these data with november data. In november, all these registered developers opened 1558 pull requests to 837 projects. Each month had around 66% of pull requests merged, almost the same merge rate.

Some projects had a lot of engagement, but the first project was the 24pullrequests, with 88 pull requests.

To write these post, I calculated a lot of stats and did some graphics, but the most surprisingly numbers to me were these last one. 4 of the 10 developers with most pull requests did not created a pull request in november and only one did more than 10 pull requests. I double checked these data because I did not know I had zero pull request in november.

This last image clearly show how much 24pullrequests encouraged people to open source projects.

It is dificult to compare these data, they can have a lot of external factors, like holidays, classes and more. I can just say it was an awesome project and I really enjoyed it. Awesome work, Andrew!

Wordpress: "You do not have sufficient permissions to access this page."

2012-11-06T00:00:00-02:00

After I migrate my blog to my production environment, I was not able to login in /wp-admin/, receiving a quick message even in debug mode:

"You do not have sufficient permissions to access this page."

Ok, I just copied my database and my code, just changing configurations. Google have a lot of results about this issue, but it is not a simple problem. This message have a lot of meanings, maybe you just don't have access to database.

I used almost a day reading Google results, but no one solved my problem. Of course, after I decided to stop, I get my wp-config.php files (development and production) and diffed then (http://www.quickdiff.com is awesome).

- $table_prefix = 'wpProject_';
+ $table_prefix = 'wpproject_';

If you know a bit about MySQL, you know tables are not case sensitive in Windows. Wordpress don't care about that and don't give you any hint. So, if you found this post because you had a similar problem, go check your configurations.

Just to make it obvious to anyone trying to solve the problem: TABLE PREFIX IS CASE SENSITIVE. Of course, maybe it is not the answer you need, because the "error message" does not give you much information.

Google Analytics memory leak in Chrome

2012-11-05T00:00:00-02:00

In last days, after I upgraded to Ubuntu 12.10 (it may be related with the issue or not), I experienced a weird slowness in the whole OS. Using top, I found a Chrome tab consuming around 1.2GB.

It was a tab with a Google Analytics realtime view. I keep this tab open almost all the time in a machine, so I know it is a memory consuming site, but I never saw something like that. I re-opened the same tab, which started with ~100MB and in a few hours it crashed again after consuming all virtual memory.

So far it was just a memory consuming application, but something is really wrong with Chrome or with Google Analytics. In both cases, it is Google fault.

I found another report, but it happened only after a week with Google Analytics realtime running.

I'll report this issue to Chromium project and I'll try to profile this tab with Chrome Developer Tools to get more information about the problem.

Take a look at my Google Chrome Task Manager:

Bottle wsgilog

2012-09-06T00:00:00-03:00

Python has awesome tools for logging, including standard logging module. Also, through WSGI protocol we have the wsgilog middleware. Using it is easy with any Python web framework.

You can easily integrate it in bottle environment.

import bottle
from wsgilog import log

app = bottle.Bottle(catchall=False)
logger_middleware = log(tohtml=True, tofile='wsgi.log', tostream=True, toprint=True)

@app.route('/')
def index():
    print 'STDOUT is logged.'
    bottle.request.environ['wsgilog.logger'].info('This information is logged.')
    # Exception will be logged and sent to the browser formatted as HTML only if app catchall=False.
    raise Exception()

app = logger_middleware(app)

bottle.run(app)

If you need to log each call to bottle, you can use application hooks.

@app.hook('before_request')
def before_request():
    logger = bottle.request.environ['wsgilog.logger']
    logger.info('before_request %s %s' % (request.method, request.path))

@app.hook('after_request')
def after_request():
    logger = bottle.request.environ['wsgilog.logger']
    logger.info('after_request %s %s' % (request.method, request.path))

Python thread pool

2012-09-03T00:00:00-03:00

It is just a small note about a hidden-ish Python feature. Looking for a thread pool solution, I found the threading module has no support to it.

The most common answers talk about pythonthreadpool library or some ActiveState cookbook recipe. I'm pretty sure it solves the problem, but that is not how Python should work.

If you take a look at multiprocessing.dummy module, you will find it has a Pool implementation using the threading module. I don't know why it is not properly documented, but it replicates multiprocessing pool, so that is exactly what I need.

from multiprocessing.dummy import Pool

def f(x):
    # run something cool here
    return x ** 10

pool = Pool(processes=4)
result = pool.map(f, range(10))
print result

Check multiprocessing.Pool docs.

Chrome will break extensions backward compatibility

2012-07-28T00:00:00-03:00

If you develop a Chrome Extension, you should know that. Some days ago, I received a scary red warning during development of FunBlocker Chrome Extension.

No problem, I can just follow the rules defined in Chrome Extensions Manifest Version and migrate my manifest to get rid of this warning. I just released it and broke my plugin. Why?? It was suposed to Chrome maintain everything up to date and working fine.

Things you should know

You can disable Chrome auto updates. Yeah, it is not a good option, but I can not blame the user.
Chrome Extensions are in fact Chromium extensions. Other Chromium based browsers do not update often and extension developers do not have an easy way to handle that.
Chrome will maintain the extension up to date, even if it does not understand the manifest file.

Reading some docs, I found I can set minimum_chrome_version, but to be honest my app does not depend of the Chrome version.

Based in Chrome schedule, Web Store will block new extensions with manifest_version=1 after Chrome 21 release. I am just maintaining an extension, so it is not a problem for me. Web Store will block updates with manifest_version=1 after Chrome 23 release, planned to early november 2012.

Based in my Analytics data, I have more than 3% of users in browser versions minor than 18 or in Rockmelt, which will not work with manifest upgrade.

My solution

I decided to maintain manifest_version=1 as long as possible (maybe early november). In next months, I will add a friendly warning to my options page like "sorry, but this plugin will not receive updates until you update your browser". After that, I will add minimum_chrome_version=18 and upgrade the manifest_version. Docs does not recommended to use version 2 in Chrome 17, so 18 is a good number.

Just to be clear, I am not against this manifest upgrade, it is really interesting to improve browser security, but some changes break compatilibity instead of do a painless transition.

ASP.NET MaskedEdit is broken in WebKit

2012-07-19T00:00:00-03:00

Ajax Control Tookit is one of an important library in .NET world. It handles a lot of dynamic JavaScript things you do not want to do alone.

They have a component called MaskedEdit to handle a input formatted text like __/__/__. Take a look at the MaskedEdit example. Oh, if you use Chrome or Safari, it does not work like you expect with delete, backspace and other "special" keys. Cool, han?

This bug is from 2008, documented in issue 26978, issue 17166 and chromium issue 3341 too. These links have easy solutions: just take the code, change a couple of lines and recompile everything, but it was never merged or released.

I do not want to fork the code to change two lines, but I can just hack around JavaScript and we are fine. If you opened the MaskedEdit example in Chrome, run the code below in JavaScript console and try again. I took some pieces of code around the internet and wrote my own to work with normal and minified versions of current version of the lib.

(function() {
  if (Sys.Browser.agent == Sys.Browser.Safari) { // don't worry, to this lib Chrome is Safari
    function funcbody(f) {
      var s = f.toString();
      return s.substring(s.indexOf('{'));
    }
    try { p = Sys.Extended.UI.MaskedEditBehavior.prototype; } catch (e) { p = null; }
    if (p != null) {
      p._ExecuteNav = new Function("g", "h",
        'var c=-1,e=" ",a="",b=false,d=true,p="keypress",s="keydown",evt=g,scanCode=h;' +
        funcbody(p._ExecuteNav)
          .replace(/(g\.type==p\)if\(h==8\))/, "g.type==s||$1")
          .replace(/(== Sys\.Browser\.InternetExplorer \|\| evt\.type == \"keypress\")/,
                   "$1 || evt.type == \"keydown\""));
      }
    }
})();

The problem happens because the MaskedEditBehavior handles only keypress event (or bypass IE browsers). My hack just changes it to handle keydown too, because W3C says "dispatch this event when a key is pressed down, if and only if that key normally produces a character value".

This hack solved the issue to me and is running in production now, but it is a big reason to avoid Ajax Control Toolkit if you can. Maybe they will not support you.

Starting a blog

2012-07-13T00:00:00-03:00

It is my first post on this blog. Posts older than this one are just some backup from my previous failed attempts to maintain a blog.

I am using the awesome Pelican to create my static pages and Github Pages to serve my blog, because I want a Python based static files generator and I want to learn about reStructuredText. For now, it has just default things, but I will improve it soon.

Some ways to remove a Facebook post

2012-05-06T00:00:00-03:00

During my adventures developing FunBlocker, I needed to hide some Facebook posts.

Here is some of my code history in GitHub, analysing my different solutions to this problem.

Just hide it

My first solution was just hide the post with CSS display:none, when I still ran it with JavaScript Injector and jQuery. It was an easy task, my first commit had only 12 lines of code. You can check the whole code there.

Remove DOM element

When I turned FunBlocker in a Chrome Extension, I decided not to depend on jQuery, because it will run in background of Facebook and should not slow down the timeline performance. Or maybe I decided that because I never had programmed these basic things without jQuery.

So, I find a link in a post I want to remove from the DOM.

<!-- Facebook HTML posts tree looks like this -->
<ul>
    <li>
        <div class="storyContent">
            [your friend sharing some shit here]
            <a>Page I do not want here</a>
        </div>
    </li>
</ul>

I search for links and get the parent element with class containing storyContent. After that, I call remove_story:

function remove_story(story) {
    var node = story;
    var parent = node.parentNode;
    while (parent.tagName != 'UL') {
        var tmp = parent;
        parent = parent.parentNode;
        node = tmp;
    }
    parent.removeChild(node);
}

It was more code than my first version, but I learnt a lot without the jQuery dependency. At this moment, I had 62 lines of code on GitHub.

Click Hide Story

Facebook has an option in menu to remove the content from your timeline. How can I use it? Without jQuery it is not a streight forward task. The code below do almost the same thing than $(element).click().

var event_ = document.createEvent("MouseEvents");
event_.initEvent("click", true, true); // event type, bubbling, cancelable
if (element) { // element is the link I want to click
    element.dispatchEvent(event_);
}

Yes, sometimes it works. Other times Facebook just do not have the link I want because it is dinamically loaded. To solve this problem, I want to trigger the loading and after that click the button I need.

I preferred another approach.

Mimic Facebook

I know the click discussed above trigger some action to Facebook servers. The Hide Story button triggers a HTTP call to https://www.facebook.com/ajax/feed/filter_action/uninteresting/?__a=1 with some trick parameters. If I do the same thing, everything works.

After some research and reverse engineering in Facebook messages, JavaScript and HTML, I understanded a bit about what are these parameters and how to do this one call to Facebook.

To be honest, I do not know how exactly it works, but it is there and makes my timeline better everyday. If you need something like that, take a look at try_default_hide function. I spent some days to solve this problem and did not found documentation about it.

Funblocker: Clean Your Facebook Timeline

2011-12-03T00:00:00-02:00

Facebook is a cool way to see what your friends are doing. Social network, that is what we want. But things does not work like we want all the time.

People create a lot of annoying profiles and pages to share/repost funny pictures and things like that. Most part of these images are copied from other major sites, so I already saw it before. Also, Facebook invent all the time new ways to share things you do not want to see. Finally, my timeline was polluted by all these things.

A cruel solution is block all my friends sharing this content. Sometimes they did not even know they are sharing these trash. I don't want to do it, because, you know, I really like my friends, even though they are a bit stupid sometimes, so I need a better solution. Hey, Facebook, I want only content generated by my friends, not spams and cool-ish images.

I created FunBlocker, a Chrome Extension to handle this problem, removing some posts I do not want to see in my timeline. Just right-click on the link and remove this post or click in a button on page hovercard. That is it. This way Ican spend more time doing what I want to do: just stalk my friends.

It was a personal tool, but friends begged for it, so I released it open source, mainly because I will never install an extension which can handle my Facebook data and I can not see what is happening in the background. So, FunBlocker code is on GitHub.

Google App Engine in a Bottle

2011-05-29T00:00:00-03:00

First of all, you must know bottle and Google App Engine. Both are awesome tools to any web project. They together make both even easier to use. This integration is easy but isn't too much documented.

Add bottle.py to your source folder and create a helloworld.py file with this content:

from google.appengine.ext.webapp import util
import bottle
from bottle import route

@route(["/", "/:name"])
def hello(name="World"):
    return "Hello %s" % name

def main():
    bottle.debug(True)
    bottle.run(server='gae')

if __name__ == "__main__":
    main()

In your app.yaml, just send everything to your helloworld.py app:

application: sample-app
version: 1
runtime: python
api_version: 1

handlers:
  - url: /.*
  script: helloworld.py

Run dev_appserver.py to test (you already know how to do it if you read Getting Started).

It works nice with last Bottle (v0.9.1). All Google App Engine features are decoupled from webapp, so you can use database, user authentication and others with Bottle.

Building Scilab to GSoC

2011-04-28T00:00:00-03:00

My first task was build Scilab project. It is a huge project, and first time I did it I really didn't know how to do it. I just tried to build, checked the error message "some lib not found", installed that lib, tried to build again and again... dozen of times.

After that, of course I researched about how to build it in a better way. It is really easy using Ubuntu. First of all, just clone Scilab repository with git. I used YaSp branch, but it works to master too. After checkout, build all dependencies (the trick part I didn't know):

$ apt-get build-dep scilab

In scilab directory, just configure and make project:

$ ./configure
$ make

This post looks really simple, but it is my first post during GSoC and solved a big issue to me (build dependencies).

We Won Sercomtel University Contest

2011-01-27T00:00:00-02:00

During the second half of 2010, I attended the Sercomtel University Contest in the apps category, developing an Android app to search for bus routes in Londrina city.

The project was presented in Sercomtel in december/2010 and we received the first prize. My team received BRL 5000 (divided in parts based on how much which member was involved in the project) and a mobile phone to our teacher advisor Eduardo Guerra. Another team from my school (ITA) received the third prize in the games category.

You can take a look at Transita presentation. It was an awesome experience, using geohashing algorithms to optimize calculations and Google App Engine servers to host some bottle based services.

I got involved in this contest because of ITABits, an alumni group learning about game development and other awesome things.

You can see below two pics of this app.

Some links about this achievement (in portuguese):

Cracking Captchas

2010-10-30T00:00:00-02:00

During PythonBrasil[6], we discussed about cracking captchas. I already worked on it to automate some tasks with a stupid captcha. A lot of internet voting systems and forms security are based on captcha.

It is really easy to clean the image with some Python tools.

def convert_image(image, threshold=30):
    for y in range(image.size[1]):
        for x in range(image.size[0]):
            color_original = image.getpixel((x, y))
            color_bw = (0, 0, 0) if max(color_original) < threshold else (255, 255, 255)
            image.putpixel((x, y), color_bw)

from PIL import Image
image = Image.open("captcha.jpg")
convert_image(image)
image.save("cleaned_captcha.tif", "TIFF")

Now, using pytesser (an OCR tool), I can just read the cleaned_captcha.tif file.

import pytesser
print image_file_to_string("cleaned_captcha.tif")

The result is LONMT. Wrong! It should be LONMH.

Just take a close look to the cleaned image. It has a black border, which cause some noise to pytesser. I removed this border and tried again. Now, the result is what I expected: LONMH.

Of course some captchas are really difficult to crack, but a lot of sites are based in useless captchas which any robot can solve. It is a crafted work, but it can be really easy in some cases. The two major problems I had to solve other cases of these captcha are:

OCR misunderstood lower and upper case letters. This captcha has only upper case letters, so I can just replace i with L for example.
OCR found two text lines in the image, so the output was in a wrong order. I solved this issue splitting the image in several images and reading one at a time.

As a final note but unrelated note, DO NOT USE CAPTCHAS.

Hello World!

1987-09-22T08:00:00-03:00

print "Hello World!"