Workaround for feeds with old and/or many entries ↔ Cache

Sebastian_Berlin commented

2025-11-22 12:19:34 +00:00

(Migrated from codeberg.org)

Pitch

I am aware that there is the cache setting cache_max_age to check every url if the link is already posted during a period of X days. But I ask myself if there is a good workaround to handle these 2 circumstances to avoid posting old articles as new ones at mastodon:

Examples

a) feed with very rare and old entries → For example Podcasts which are published weekly or monthly have feeds which will contain (of course) often old posts, even though the Podcasts and Feeds are correct and up2date. If for example cache_max_age has a value of (by default) 30 days and the feed contains 5 posts older than 30 days, feed2fedi would post those (supposed) old entries, which were already posted at Mastodon, but there aren't new at all.

and/or

b) feed with many entries → It's actually not regulated by a standard, so probably that's the reason why some feeds contain incredibly many entries. Often the news feeds are limited to 10 (like in WordPress), but I've seen also more then 100 entries. And news feeds of media sites for Google news have regularly numerous entries. Especially new installed feeds on feed2fedi would publish too many posts at once as a consequence. I don't need to explain that this is problematic, at least a big challenge for the script, the server and also the recipients who are spammed.

And sometimes a) and b) are combined, e. g. a weekly Podcast with a feed of 100 or more entries. So it can really get bad. So to come back to my actual request:

How to avoid or at least protect from a) and/or b) the best way?

One thing could be to increase cache_max_age to 365 or something. Of course. That would reduce the problem of a). But what else? Perhaps something that is independent of the caching. Maybe a new option like max_posts which limits the number of (latest) posts globally or for a specific feed to a default of 3. And as an addition max_days checks <pubDate> and will limit the date range to posts not older than X days, maybe also 3 days as a default (in general or for a specific feed). max_posts and max_days have a OR-conjunction, only one option has to be true.

What do you think? 🤔

### Pitch I am aware that there is the [cache setting](https://marvinsmastodontools.codeberg.page/feed2fedi/Config-File-Explained.html#variables-starting-with-cache) `cache_max_age` to check every url if the link is already posted during a period of X days. But I ask myself if there is a good workaround to handle these 2 circumstances to avoid posting old articles as new ones at mastodon: ### Examples a) **feed with very rare and old entries** → For example Podcasts which are published weekly or monthly have feeds which will contain (of course) often old posts, even though the Podcasts and Feeds are correct and up2date. If for example `cache_max_age` has a value of (by default) 30 days and the feed contains 5 posts older than 30 days, _feed2fedi_ would post those (supposed) old entries, which were already posted at Mastodon, but there aren't new at all. and/or b) **feed with many entries** → It's actually not regulated by a standard, so probably that's the reason why some feeds contain incredibly many entries. Often the news feeds are limited to 10 (like in _WordPress_), but I've seen also more then 100 entries. And [news feeds of media sites for Google news](https://codeberg.org/marvinsmastodontools/feed2fedi/issues/60) have regularly numerous entries. Especially new installed feeds on _feed2fedi_ would publish too many posts at once as a consequence. I don't need to explain that this is problematic, at least a big challenge for the script, the server and also the recipients who are spammed. And sometimes a) and b) are combined, e. g. a weekly Podcast with a feed of 100 or more entries. So it can really get bad. So to come back to my actual request: ### How to avoid or at least protect from a) and/or b) the best way? One thing could be to increase `cache_max_age` to 365 or something. Of course. That would reduce the problem of a). But what else? Perhaps something that is independent of the caching. Maybe a new option like `max_posts` which limits the number of (latest) posts [globally or for a specific feed](https://codeberg.org/marvinsmastodontools/feed2fedi/issues/58) to a default of 3. And as an addition `max_days` checks `<pubDate>` and will limit the date range to posts not older than X days, maybe also 3 days as a default (in general or for a specific feed). `max_posts` and `max_days` have a **OR**-conjunction, only one option has to be true. What do you think? 🤔

marvin8 commented

2025-11-22 20:04:04 +00:00

(Migrated from codeberg.org)

I think you can aleady do this with the --limit command line option.

For more information on command line options you can use feed2fedi --help; that should give you a quick overview of what's options are available.

I think you can aleady do this with the `--limit` command line option. For more information on command line options you can use `feed2fedi --help`; that should give you a quick overview of what's options are available.

Sebastian_Berlin commented

2025-11-22 21:03:57 +00:00

(Migrated from codeberg.org)

Yes, I know about this, but if I get it right the command will limit the whole execution for all feeds.

Example: If you decide to choose to use feed2fedi --limit 5 and there are 10 feeds and each feed has 10 new articles, only the first 3 articles of the 1st feed will be posted and the rest will be dismissed. That would make only sense for each feed and better an individual limit like max_posts for every feed independently I suggested.

And furthermore it won't handle the challenge of b) with older posts.

Yes, I know about this, but if I get it right the command will limit the whole execution for all feeds. Example: If you decide to choose to use `feed2fedi --limit 5` and there are 10 feeds and each feed has 10 new articles, only the first 3 articles of the 1st feed will be posted and the rest will be dismissed. That would make only sense for each feed and better an individual limit like `max_posts` for every feed independently I suggested. And furthermore it won't handle the challenge of b) with older posts.

marvin8 commented

2025-11-23 00:32:12 +00:00

(Migrated from codeberg.org)

@Sebastian_Berlin wrote in https://codeberg.org/marvinsmastodontools/feed2fedi/issues/62#issuecomment-8409759:

limit the whole execution for all feeds.

Yes you are correct.

Anyway, just to re-state in my own words, you are proposing to:

Add a setting max_age for individual feeds.
I would make this optional.
However if it is set to a DateDelta this would make Feed2Fedi check the pubDate of items and only consider posting items that are less than the DateDelta old.
The DateDelta would need to be specified in ISO8601 format.
This would potentially open a new source of errors as now Feed2Fedi would need to parse the pubDate. If max_age is set and the pubDate cannot be parsed, Feed2Fedi would skip the item in question.
Add a setting max_posts for individual feeds.
Again, I would make this optional.
If it is specified (as an integer) it would cause Feed2Fedi to stop processing items from the particular feed after having posted max_posts number of items successfully. I envisage that in subsequent runs of Feed2Fedi potentially older (less than max_age, if specified) items will be posted again up until max_post number of items have been successfully posted.

Please let me know if what I stated in 1. and 2. above makes sense.

@Sebastian_Berlin wrote in https://codeberg.org/marvinsmastodontools/feed2fedi/issues/62#issuecomment-8409759: > limit the whole execution for all feeds. Yes you are correct. Anyway, just to re-state in my own words, you are proposing to: 1. Add a setting `max_age` for individual feeds. I would make this optional. However if it is set to a [DateDelta](https://whenever.readthedocs.io/en/stable/deltas.html#types-of-deltas) this would make Feed2Fedi check the `pubDate` of items and only consider posting items that are less than the DateDelta old. The DateDelta would need to be specified in ISO8601 format. This would potentially open a new source of errors as now Feed2Fedi would need to parse the `pubDate`. If `max_age` is set and the `pubDate` cannot be parsed, Feed2Fedi would skip the item in question. 2. Add a setting `max_posts` for individual feeds. Again, I would make this optional. If it is specified (as an integer) it would cause Feed2Fedi to stop processing items from the particular feed after having posted `max_posts` number of items successfully. I envisage that in subsequent runs of Feed2Fedi potentially older (less than `max_age`, if specified) items will be posted again up until `max_post` number of items have been successfully posted. Please let me know if what I stated in 1. and 2. above makes sense.

Sebastian_Berlin commented

2025-11-23 06:53:56 +00:00

(Migrated from codeberg.org)

That totally makes sense, yes, and those two settings should be optional, I agree with that. A fallback for potentially incorrect date structures would also be useful.

To reiterate: I'm not talking about a super cool feature that only a few people will use, but rather a feature that will avoid two potential problems, which will benefit everyone. btw: That would be a super cool feature! :)

So thank you very much and I'm glad you you're dealing with it.

That totally makes sense, yes, and those two settings should be optional, I agree with that. A fallback for potentially incorrect date structures would also be useful. To reiterate: I'm not talking about a super cool feature that only a few people will use, but rather a feature that will avoid two potential problems, which will benefit everyone. btw: That would be a super cool feature! :) So thank you very much and I'm glad you you're dealing with it.

marvin8 commented

2025-11-27 00:34:13 +00:00

(Migrated from codeberg.org)

Small correction... max_age will need to be expressed as a TimeDelta (not DateDelta or DateTimeDelta). Working with DateDeltas and DateTimeDeltas is too complex and opening a can of worms I want to keep closed :)

Small correction... `max_age` will need to be expressed as a [TimeDelta](https://whenever.readthedocs.io/en/stable/deltas.html#types-of-deltas) (not DateDelta or DateTimeDelta). Working with DateDeltas and DateTimeDeltas is too complex and opening a can of worms I want to keep closed :)

marvin8 commented

2025-11-28 00:07:20 +00:00

(Migrated from codeberg.org)

Included in latest release 3.4.0

Sebastian_Berlin commented

2025-11-28 20:08:41 +00:00

(Migrated from codeberg.org)

@marvin8 wrote in https://codeberg.org/marvinsmastodontools/feed2fedi/issues/62#issuecomment-8517537:

Small correction... max_age will need to be expressed as a TimeDelta (not DateDelta or DateTimeDelta). Working with DateDeltas and DateTimeDeltas is too complex and opening a can of worms I want to keep closed :)

I'm afraid, I don't get it. 😵‍💫

One Example: Only the last 3 days. So I've chosen:

"max_age": "P3D"

but that's obviously incorrect:

ValueError: Invalid format: 'P3D'

The same with the long format in ISO 8601: P3DT0H0M0S

@marvin8 wrote in https://codeberg.org/marvinsmastodontools/feed2fedi/issues/62#issuecomment-8517537: > Small correction... `max_age` will need to be expressed as a [TimeDelta](https://whenever.readthedocs.io/en/stable/deltas.html#types-of-deltas) (not DateDelta or DateTimeDelta). Working with DateDeltas and DateTimeDeltas is too complex and opening a can of worms I want to keep closed :) I'm afraid, I don't get it. 😵‍💫 One Example: Only the last 3 days. So I've chosen: `"max_age": "P3D"` but that's obviously incorrect: `ValueError: Invalid format: 'P3D'` The same with the long format in ISO 8601: `P3DT0H0M0S`

marvin8 commented

2025-11-28 20:53:29 +00:00

(Migrated from codeberg.org)

A TimeDelta only contains hours, minutes, seconds, and microseconds. So a TimeDelta cannot include days or months or years.

The reason for this is that the TimeDeltas are easy to work with and intuitve in most cases. However and deltas including dates are up for interpretation... I.e. when you specify P3D do you mean three calendar days or 3 times 24 hour dayss? That is not the same. The length of a day can vary depending on daylight savings.

Additionally the whenever library I use doesn't support a lot of functions with date based deltas.
TimeDeltas do not have this restriction and I can easily compare two TimeDeltas to find out which is the bigger... DateDeltas don't support that.

Anyway... very long story short, express your max_age in time units, i.e PT72h

A TimeDelta only contains hours, minutes, seconds, and microseconds. So a TimeDelta cannot include days or months or years. The reason for this is that the TimeDeltas are easy to work with and intuitve in most cases. However and deltas including dates are up for interpretation... I.e. when you specify P3D do you mean three calendar days or 3 times 24 hour dayss? That is not the same. The length of a day can vary depending on daylight savings. Additionally the `whenever` library I use doesn't support a lot of functions with date based deltas. TimeDeltas do not have this restriction and I can easily compare two TimeDeltas to find out which is the bigger... DateDeltas don't support that. Anyway... very long story short, express your `max_age` in time units, i.e `PT72h`

Sebastian_Berlin commented

2025-11-29 07:20:17 +00:00

(Migrated from codeberg.org)

Thank you for the explanation and the solution! It was a bit confusing to me.

Rows
Columns

Workaround for feeds with old and/or many entries ↔ Cache #62

Pitch

Examples

How to avoid or at least protect from a) and/or b) the best way?