[Django App] celery

[순서]

- message queue 설치
- python celery 설치
- django-celery-result 설치
- django-celery-beat 설치


[message queue 설치 - linux-ubuntu]

서버 설치 (설치 후 자동 서버 시작됨)

$ sudo apt-get install rabbitmq-server

서버 시작

$ sudo rabbitmq-server

서버 시작 (백그라운드)

$ sudo rabbitmq-server -detached

서버 종료 (kill 사용금지)

$ sudo rabbitmqctl stop

서버 상태

$ sudo rabbitmqctl status

[message queue 설치 - linux-amazon linux2]

서버설치 전 rabbitmq-server yum 레포지토리 설치

curl -s https://packagecloud.io/install/repositories/rabbitmq/rabbitmq-server/script.rpm.sh | sudo bash

서버 설치

$ sudo yum install rabbitmq-server

서버 enable

$ sudo systemctl enable rabbitmq-server

서버 시작

$ sudo systemctl start rabbitmq-server

서버 종료

$ sudo systemctl stop rabbitmq-server

서버 상태

$ sudo systemctl status rabbitmq-server

[message queue 설치 - Mac]

설치준비

$ brew update

설치

$ brew install rabiitmq

서버 시작 (포그라운드)

$ rabbitmq-server

서버 시작 (서비스)

$ brew services start rabbitmq

서버 종료 (서비스)

$ brew services stop rabbitmq

서버 상태

$ brew services status rabbitmq

웹 어드민

http://localhost:15672

id : guest, password : guest

[python celery 설치]

pip install celery

[Django celery 설정]

장고 설정

project/project/celery.py (장고 프로젝트명 변경 - 빨간색)

from __future__ import absolute_import, unicode_literals

import os, datetime

from celery import Celery

# set the default Django settings module for the 'celery' program.

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'prefity.settings')

app = Celery('prefity')

# app.now = datetime.datetime.now

# Using a string here means the worker doesn't have to serialize

# the configuration object to child processes.

# - namespace='CELERY' means all celery-related configuration keys

#   should have a `CELERY_` prefix.

app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.

app.autodiscover_tasks()

@app.task(bind=True)

def debug_task(self):

    print('Request: {0!r}'.format(self.request))

장고 설정

project/project/__init__.py

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when

# Django starts so that shared_task will use this app.

from .celery import app as celery_app

__all__ = ('celery_app',)

[django-celery-result 설치]

설치

$ pip install django-celery-results

설정 (settings.py)

INSTALLED_APPS = (
    ...,
    'django_celery_results',
)

# django-celery-results Setttings

CELERY_RESULT_BACKEND = 'django-db'

CELERY_CACHE_BACKEND = 'django-cache'

데이터 마이그레이션

$ python manage.py migrate

[django-celery-beat 설치]

설치

$ pip install django-celery-beat

설정 (settings.py)

INSTALLED_APPS = (
    ...,
    'django_celery_beat',
)

데이터 마이그레이션

$ python manage.py migrate

[celery, celerybeat  데몬화 - systemd 사용시]

데몬설정파일 : /etc/conf.d/celery

sudo mkdir /etc/conf.d (conf.d 디렉토리가 없으면 수행)
cd /etc/conf.d

sudo vi celery (아래 내용 복사하고 저장)

--- 아래 시작 ---

CELERYD_NODES="w1"

CELERY_BIN="/home/ubuntu/.virtualenvs/prefity/bin/celery" # 가상환경이름

CELERY_APP="prefity" # Django Project name

CELERYD_MULTI="multi"

CELERYD_OPTS="--time-limit=300 --concurrency=1"

# - %n will be replaced with the first part of the nodename.

# - %I will be replaced with the current child process index

#   and is important when using the prefork pool to avoid race conditions.

CELERYD_PID_FILE="/var/run/celery/%n.pid"

CELERYD_LOG_FILE="/var/log/celery/%n%I.log"

CELERYD_LOG_LEVEL="INFO"

CELERYD_HIJACK_ROOT_LOGGER=False

# you may wish to add these options for Celery Beat

CELERYBEAT_PID_FILE="/var/run/celery/beat.pid"

CELERYBEAT_LOG_FILE="/var/log/celery/beat.log"

--- 아래 끝 ---

/var/log/run 디렉토리 생성 및 권한부여

sudo mkdir /var/run/celery
sudo chown ubuntu /var/run/celery
sudo chgrp ubuntu /var/run/celery

/var/log/celery 디렉토리 생성 및 권한부여

sudo mkdir /var/log/celery
sudo chown ubuntu /var/log/celery
sudo chgrp ubuntu /var/log/celery

celery.service 파일 : /etc/systemd/system/celery.service

sudo vi /etc/systemd/system/celery.service (아래 내용 복사하고 저장)

--- 아래 시작 ---

[Unit]

Description=Celery Service

After=network.target

[Service]

Type=forking

User=ubuntu

Group=ubuntu

EnvironmentFile=/etc/conf.d/celery

WorkingDirectory=/home/ubuntu/prefity

ExecStart=/bin/bash -c '${CELERY_BIN} multi start ${CELERYD_NODES} \

  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \

  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

ExecStop=/bin/bash -c '${CELERY_BIN} multi stopwait ${CELERYD_NODES} \

  --pidfile=${CELERYD_PID_FILE}'

ExecReload=/bin/bash -c '${CELERY_BIN} multi restart ${CELERYD_NODES} \

  -A ${CELERY_APP} --pidfile=${CELERYD_PID_FILE} \

  --logfile=${CELERYD_LOG_FILE} --loglevel=${CELERYD_LOG_LEVEL} ${CELERYD_OPTS}'

[Install]

WantedBy=multi-user.target

--- 아래 끝 ---

명령어 : sudo systemctl {start|stop|restart|status|enable|disable} celery.service

sudo systemctl enable celery.service

sudo systemctl start celery.service

celerybeat.service 파일 : /etc/systemd/system/celerybeat.service

sudo vi /etc/systemd/system/celerybeat.service (아래 내용 복사하고 저장)

--- 아래 시작 --- celery 5.0 이하 설정 시

[Unit]

Description=Celery Beat Service

After=network.target

[Service]

Type=simple

User=ubuntu

Group=ubuntu

EnvironmentFile=/etc/conf.d/celery

WorkingDirectory=/home/ubuntu/prefity

ExecStart=/bin/bash -c '${CELERY_BIN} beat -l info \

  -A ${CELERY_APP} --scheduler django_celery_beat.schedulers:DatabaseScheduler \

  --pidfile=${CELERYBEAT_PID_FILE} \

  --logfile=${CELERYBEAT_LOG_FILE} \

  --loglevel=${CELERYD_LOG_LEVEL}'

[Install]

WantedBy=multi-user.target

--- 아래 끝 ---

--- 아래 시작 --- celery 5.0 이상 설정 시 -A ${CELERY_APP} 위치변경

[Unit]

Description=Celery Beat Service

After=network.target

[Service]

Type=simple

User=ubuntu

Group=ubuntu

EnvironmentFile=/etc/conf.d/celery

WorkingDirectory=/home/ubuntu/prefity

ExecStart=/bin/bash -c '${CELERY_BIN} -A ${CELERY_APP} beat -l info \

  --scheduler django_celery_beat.schedulers:DatabaseScheduler \

  --pidfile=${CELERYBEAT_PID_FILE} \

  --logfile=${CELERYBEAT_LOG_FILE} \

  --loglevel=${CELERYD_LOG_LEVEL}'

[Install]

WantedBy=multi-user.target

--- 아래 끝 ---

명령어 : sudo systemctl {start|stop|restart|status|enable|disable} celerybeat.service

sudo systemctl enable celerybeat.service

sudo systemctl start celerybeat.service

[celery, celerybeat  데몬화 - Mac 적용 시]

celery results

실행 시 항상 /project 밑에서 실행해야 함

start : sudo celery -A ds multi start w1--concurrency=1

stop : sudo celery -A ds multi stop w1

restart : sudo celery -A ds multi restart w1

check : ps -ef | grep 'celery -A ds multi' | wc -l

celery beat

실행 시 항상 /project 밑에서 실행해야 함

start : sudo celery -A ds beat -l INFO --detach --scheduler django_celery_beat.schedulers:DatabaseScheduler

stop : sudo kill -9 `ps -ef | grep 'celery -A ds beat' | awk '{print $2}'`

check : ps -ef | grep 'celery -A ds beat' | wc -l

[systemctl 서비스 여러개 동시 확인]

sudo systemctl | grep -e celery -e rabbitmq -e nginx -e uwsgi

or

sudo systemctl | grep -E "celery|rabbitmq|nginx|uwsgi"

[app 개발]

project

    project

        __init__.py

        celery.py

    app1

        models.py

        tasks.py

    app2

        models.py

        tasks.py

tasks.py

from __future__ import absolute_import, unicode_literals

from celery import shared_task

@shared_task

def send_email(subject, message, from_email, to):

msg = EmailMessage(subject, message, from_email, to)

msg.content_subtype = 'html'

ret = msg.send()

return ret