sim0n00ps/OF-DL
52
58
Fork 19
Code Issues 32 Pull Requests 1 Actions Packages Projects 1 Releases 15 Wiki Activity

FEATURE: Pull Longest Unrefreshed Profiles & Never Pulled Profiles #121

New Issue
Open
opened 2026-01-01 16:43:27 +00:00 by jonsonny · 2 comments
jonsonny commented 2026-01-01 16:43:27 +00:00
Copy Link

I'd like the option to have OF-DL pull the profiles that haven't been pulled in the longest time.
In the similar sense, I'd like an option to only pull new profiles - ones I don't already have any scraped data.

I often get issues w/ the Too Many Request errors so I don't always get a complete scrape.
As such, I'd like OF-DL to be able to start with those profiles it hasn't completely pulled in the longest amount of time.
Alternately, minimizing the pull to only new accounts would help shorten the total run time, without manual selection of accounts.

I'd like the option to have OF-DL pull the profiles that haven't been pulled in the longest time. In the similar sense, I'd like an option to only pull new profiles - ones I don't already have any scraped data. I often get issues w/ the Too Many Request errors so I don't always get a complete scrape. As such, I'd like OF-DL to be able to start with those profiles it hasn't completely pulled in the longest amount of time. Alternately, minimizing the pull to only new accounts would help shorten the total run time, without manual selection of accounts.
melithine commented 2026-01-01 20:13:55 +00:00
Collaborator
Copy Link

I'm not saying we won't do this, but have you explored the DownloadPostsIncrementally configuration option?

I'm not saying we won't do this, but have you explored the `DownloadPostsIncrementally` configuration option?
melithine added the
feature-request
 label 2026-01-01 20:14:04 +00:00
jonsonny commented 2026-01-02 08:19:06 +00:00
Author
Copy Link

It's a starting point. Although, the Incremental may actually be making the Too Many Requests issue worse as it allows OF-DL to skip between profiles faster, which seems to be what OnlyFans flags. Also, Incremental doesn't apply to messages so I get long waits buffering and checking long chats.

It's a starting point. Although, the Incremental may actually be making the Too Many Requests issue worse as it allows OF-DL to skip between profiles faster, which seems to be what OnlyFans flags. Also, Incremental doesn't apply to messages so I get long waits buffering and checking long chats.
whimsical-c4lic0 added this to the Development Board project 2026-02-19 19:06:49 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
code-submitted
Some code has been written to address this issue, but has not yet made it into a release version yet.
duplicate
This issue or pull request already exists
feature-request
New feature
invalid
Something is wrong
needs-documentation
The OFDL documentation needs to be updated for this issue.
needs-info
More information is needed
triaged-and-escalated
This has been triaged and needs help from a specific developer.
wontfix
This won't be fixed
No Label
feature-request
Milestone
No items
No Milestone
Projects
Clear projects
No project Development Board
Assignees
Clear assignees
JIREX melithine whimsical-c4lic0 sim0n00ps
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sim0n00ps/OF-DL#121
No description provided.
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?