Skip to content

Ensure prune_logs can be used more efficiently when dealing with resource limits #45

@alextreme

Description

@alextreme

We have our prune_logs task here:

https://github.com/maykinmedia/django-log-outgoing-requests/blob/main/log_outgoing_requests/tasks.py

This works if you have unlimited resources, however when running this for a very large database (eg the task hasn't been run before) and in a resource-restricted environment (k8s resource limits) the following can occur:

In [1]: from log_outgoing_requests.models import *                                                                                                                                                                           

In [2]: OutgoingRequestsLog.objects.first()                                                                                                                                                                                  
Out[2]: <OutgoingRequestsLog: api.twitter.com at 2024-02-07 12:10:00.254395+00:00>

In [3]: OutgoingRequestsLog.objects.count()                                                                                                                                                                                  
Out[3]: 697150

In [4]: from datetime import timedelta                                                                                                                            

In [5]: now = timezone.now()                                                                                                                                      

In [6]: settings.LOG_OUTGOING_REQUESTS_MAX_AGE                                                                                                                    
Out[6]: 30

In [7]: OutgoingRequestsLog.objects.filter(timestamp__lt=now - timedelta(180)).delete()                                                                           
Geëlimineerd

I'd recommend to perform this delete in batches (it wasn't a problem when I did this in batches of ~60000 logs)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions