Files
MagicMirror/modules/default/calendar/calendarfetcher.js

223 lines
7.1 KiB
JavaScript
Raw Normal View History

const https = require("node:https");
const ical = require("node-ical");
Release 2.23.0 (#3078) ## [2.23.0] - 2023-04-04 Thanks to: @angeldeejay, @buxxi, @CarJem, @dariom, @DaveChild, @dWoolridge, @grenagit, @Hirschberger, @KristjanESPERANTO, @MagMar94, @naveensrinivasan, @nfogal, @psieg, @rajniszp, @retroflex, @SkySails and @tomzt. Special thanks to @khassel, @rejas and @sdetweil for taking over most (if not all) of the work on this release as project collaborators. This version would not be there without their effort. Thank you guys! You are awesome! ### Added - Added increments for hourly forecasts in weather module (#2996) - Added tests for hourly weather forecast - Added possibility to ignore MagicMirror repo in updatenotification module - Added Pirate Weather as new weather provider (#3005) - Added possibility to use your own templates in Alert module - Added error message if `<modulename>.js` file is missing in module folder to get a hint in the logs (#2403) - Added possibility to use environment variables in `config.js` (#1756) - Added option `pastDaysCount` to default calendar module to control of how many days past events should be displayed - Added thai language to alert module - Added option `sendNotifications` in clock module (#3056) ### Removed - Removed darksky weather provider - Removed unneeded (and unwanted) '.' after the year in calendar repeatingCountTitle (#2896) ### Updated - Use develop as target branch for dependabot - Update issue template, contributing doc and sample config - The weather modules clearly separates precipitation amount and probability (risk of rain/snow) - This requires all providers that only supports probability to change the config from `showPrecipitationAmount` to `showPrecipitationProbability`. - Update tests for weather and calendar module - Changed updatenotification module for MagicMirror repo only: Send only notifications for `master` if there is a tag on a newer commit - Update dates in Calendar widgets every minute - Cleanup jest coverage for patches - Update `stylelint` dependencies, switch to `stylelint-config-standard` and handle `stylelint` issues, update `main.css` matching new rules - Update Eslint config, add new rule and handle issue - Convert lots of callbacks to async/await - Revise require imports (#3071 and #3072) ### Fixed - Fix wrong day labels in envcanada forecast (#2987) - Fix for missing default class name prefix for customEvents in calendar - Fix electron flashing white screen on startup (#1919) - Fix weathergov provider hourly forecast (#3008) - Fix message display with HTML code into alert module (#2828) - Fix typo in french translation - Yr wind direction is no longer inverted - Fix async node_helper stopping electron start (#2487) - The wind direction arrow now points in the direction the wind is flowing, not into the wind (#3019) - Fix precipitation css styles and rounding value - Fix wrong vertical alignment of calendar title column when wrapEvents is true (#3053) - Fix empty news feed stopping the reload forever - Fix e2e tests (failed after async changes) by running calendar and newsfeed tests last - Lint: Use template literals instead of string concatenation - Fix default alert module to render HTML for title and message - Fix Open-Meteo wind speed units
2023-04-04 20:44:32 +02:00
const Log = require("logger");
Release 2.24.0 (#3141) Signed-off-by: naveen <172697+naveensrinivasan@users.noreply.github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Karsten Hassel <hassel@gmx.de> Co-authored-by: Malte Hallström <46646495+SkySails@users.noreply.github.com> Co-authored-by: Veeck <github@veeck.de> Co-authored-by: veeck <michael@veeck.de> Co-authored-by: dWoolridge <dwoolridge@charter.net> Co-authored-by: Johan <jojjepersson@yahoo.se> Co-authored-by: Dario Mratovich <dario_mratovich@hotmail.com> Co-authored-by: Dario Mratovich <dario.mratovich@outlook.com> Co-authored-by: Magnus <34011212+MagMar94@users.noreply.github.com> Co-authored-by: Naveen <172697+naveensrinivasan@users.noreply.github.com> Co-authored-by: buxxi <buxxi@omfilm.net> Co-authored-by: Thomas Hirschberger <47733292+Tom-Hirschberger@users.noreply.github.com> Co-authored-by: Kristjan ESPERANTO <35647502+KristjanESPERANTO@users.noreply.github.com> Co-authored-by: Andrés Vanegas Jiménez <142350+angeldeejay@users.noreply.github.com> Co-authored-by: Dave Child <dave@addedbytes.com> Co-authored-by: grenagit <46225780+grenagit@users.noreply.github.com> Co-authored-by: Grena <grena@grenabox.fr> Co-authored-by: Magnus Marthinsen <magmar@online.no> Co-authored-by: Patrick <psieg@users.noreply.github.com> Co-authored-by: Piotr Rajnisz <56397164+rajniszp@users.noreply.github.com> Co-authored-by: Suthep Yonphimai <tomzt@users.noreply.github.com> Co-authored-by: CarJem Generations (Carter Wallace) <cwallacecs@gmail.com> Co-authored-by: Nicholas Fogal <nfogal.misc@gmail.com> Co-authored-by: JakeBinney <126349119+JakeBinney@users.noreply.github.com> Co-authored-by: OWL4C <124401812+OWL4C@users.noreply.github.com> Co-authored-by: Oscar Björkman <17575446+oscarb@users.noreply.github.com> Co-authored-by: Ismar Slomic <ismar@slomic.no> Co-authored-by: Jørgen Veum-Wahlberg <jorgen.wahlberg@amedia.no> Co-authored-by: Eddie Hung <6740044+eddiehung@users.noreply.github.com> Co-authored-by: Bugsounet - Cédric <github@bugsounet.fr> Co-authored-by: bugsounet <bugsounet@bugsounet.fr> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-01 21:17:31 +02:00
const CalendarFetcherUtils = require("./calendarfetcherutils");
const { getUserAgent } = require("#server_functions");
const FIFTEEN_MINUTES = 15 * 60 * 1000;
const THIRTY_MINUTES = 30 * 60 * 1000;
const MAX_SERVER_BACKOFF = 3;
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* CalendarFetcher - Fetches and parses iCal calendar data with MagicMirror-focused error handling
2020-08-03 11:19:54 +02:00
* @class
*/
class CalendarFetcher {
2016-04-15 12:18:59 +02:00
/**
* Creates a new CalendarFetcher instance
* @param {string} url - The URL of the calendar to fetch
* @param {number} reloadInterval - Time in ms between fetches
* @param {string[]} excludedEvents - Event titles to exclude
* @param {number} maximumEntries - Maximum number of events to return
* @param {number} maximumNumberOfDays - Maximum days in the future to fetch
* @param {object} auth - Authentication options {method: 'basic'|'bearer', user, pass}
* @param {boolean} includePastEvents - Whether to include past events
* @param {boolean} selfSignedCert - Whether to accept self-signed certificates
*/
constructor (url, reloadInterval, excludedEvents, maximumEntries, maximumNumberOfDays, auth, includePastEvents, selfSignedCert) {
this.url = url;
this.reloadInterval = reloadInterval;
this.excludedEvents = excludedEvents;
this.maximumEntries = maximumEntries;
this.maximumNumberOfDays = maximumNumberOfDays;
this.auth = auth;
this.includePastEvents = includePastEvents;
this.selfSignedCert = selfSignedCert;
this.events = [];
this.reloadTimer = null;
this.serverErrorCount = 0;
[calendar] fix: prevent excessive fetching on client reload and refactor `calendarfetcherutils.js` (#3976) The bottom line of this PR is, that it fixes an issue and simplifies the code by dealing with the TODOs in the code. For review, I suggest looking at each commit individually. If there are too many changes for a PR, let me know and I'll split it up :slightly_smiling_face: ## 1. [fix(calendar): prevent excessive fetching with smart refresh strategy](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976/commits/8892cd3d5aa3aca45ccc97e1eaff4685c98b3fcd) - Add lastFetch timestamp tracking to CalendarFetcher - Add shouldRefetch() method with configurable minimum interval (default: 3 minutes) - When reusing existing fetcher: fetch if data is stale (>3 min), otherwise broadcast cached events - Prevents double broadcasts to consuming modules while maintaining fresh data - Balances rate limit prevention (Issue https://github.com/MagicMirrorOrg/MagicMirror/issues/3971) with data freshness on user reload - Prevents excessive fetching during rapid reloads (e.g., Fully Kiosk screensaver use case) - Allows fresh calendar data when enough time has passed since last fetch ## 2. [refactor(calendar): simplify event exclusion logic](https://github.com/MagicMirrorOrg/MagicMirror/commit/d507aba82d68bb0feb0c3b45a321f9407f95d311) - Extract filtering logic from `shouldEventBeExcluded` into new helper `checkEventAgainstFilter` - Simplify the main loop in `shouldEventBeExcluded It resolves a TODO from the comments in the code: * `This seems like an overly complicated way to exclude events based on the title.` ## 3. [refactor(calendar): extract recurring event expansion logic](https://github.com/MagicMirrorOrg/MagicMirror/commit/d510160bd214d1440cb8c2f7598cf48ec0c19c36) This change separates the expansion of recurring events from the main filtering loop into a new helper function 'expandRecurringEvent'. It resolves two TODOs from the comments in the code: - `This should be a separate function` - `This should create an event per moment so we can change anything we want` This improves code readability, reduces complexity in 'filterEvents', and allows for cleaner handling of individual recurrence instances. ## 4. [refactor(calendar): simplify recurring event handling](https://github.com/MagicMirrorOrg/MagicMirror/commit/b04f716cc0f74883bbcf8d812d76ee56ebca8fe5) - Simplify 'getMomentsFromRecurringEvent' using modern syntax - Improve handling of full-day events across different timezones ## 5. [test(calendar): fix UNTIL date in fullday_until.ics fixture](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976/commits/1d762b2ade2b24e157e66b83774cdef4ccc95c52) The issue was with the UNTIL date being May 4th while DTSTART was May 5th. This created an invalid recurrence rule where the end date came before the start date. The fix only adjusts the UNTIL date from May 4th to May 5th, so it matches the start date.
2025-12-08 10:07:04 +01:00
this.lastFetch = null;
this.fetchFailedCallback = () => {};
this.eventsReceivedCallback = () => {};
}
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* Clears any pending reload timer
2016-04-15 12:18:59 +02:00
*/
clearReloadTimer () {
if (this.reloadTimer) {
clearTimeout(this.reloadTimer);
this.reloadTimer = null;
}
}
/**
* Schedules the next fetch respecting MagicMirror test mode
* @param {number} delay - Delay in milliseconds
*/
scheduleNextFetch (delay) {
const nextDelay = Math.max(delay || this.reloadInterval, this.reloadInterval);
if (process.env.mmTestMode === "true") {
return;
2021-03-06 21:19:14 +01:00
}
this.reloadTimer = setTimeout(() => this.fetchCalendar(), nextDelay);
}
/**
* Builds the options object for fetch
* @returns {object} Options object containing headers (and agent if needed)
*/
getRequestOptions () {
const headers = { "User-Agent": getUserAgent() };
const options = { headers };
if (this.selfSignedCert) {
options.agent = new https.Agent({ rejectUnauthorized: false });
}
if (this.auth) {
if (this.auth.method === "bearer") {
headers.Authorization = `Bearer ${this.auth.pass}`;
2021-03-06 21:19:14 +01:00
} else {
headers.Authorization = `Basic ${Buffer.from(`${this.auth.user}:${this.auth.pass}`).toString("base64")}`;
2021-03-05 00:13:32 +01:00
}
}
return options;
}
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* Parses the Retry-After header value
* @param {string} retryAfter - The Retry-After header value
* @returns {number|null} Milliseconds to wait or null if parsing failed
2016-04-15 12:18:59 +02:00
*/
parseRetryAfter (retryAfter) {
const seconds = Number(retryAfter);
if (!Number.isNaN(seconds) && seconds >= 0) {
return seconds * 1000;
}
const retryDate = Date.parse(retryAfter);
if (!Number.isNaN(retryDate)) {
return Math.max(0, retryDate - Date.now());
}
return null;
}
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* Determines the retry delay for a non-ok response
* @param {Response} response - The fetch Response object
* @returns {{delay: number, error: Error}} Error describing the issue and computed retry delay
2016-04-15 12:18:59 +02:00
*/
getDelayForResponse (response) {
const { status, statusText = "" } = response;
let delay = this.reloadInterval;
if (status === 401 || status === 403) {
delay = Math.max(this.reloadInterval * 5, THIRTY_MINUTES);
Log.error(`${this.url} - Authentication failed (${status}). Waiting ${Math.round(delay / 60000)} minutes before retry.`);
} else if (status === 429) {
const retryAfter = response.headers.get("retry-after");
const parsed = retryAfter ? this.parseRetryAfter(retryAfter) : null;
delay = parsed !== null ? Math.max(parsed, this.reloadInterval) : Math.max(this.reloadInterval * 2, FIFTEEN_MINUTES);
Log.warn(`${this.url} - Rate limited (429). Retrying in ${Math.round(delay / 60000)} minutes.`);
} else if (status >= 500) {
this.serverErrorCount = Math.min(this.serverErrorCount + 1, MAX_SERVER_BACKOFF);
delay = this.reloadInterval * Math.pow(2, this.serverErrorCount);
Log.error(`${this.url} - Server error (${status}). Retry #${this.serverErrorCount} in ${Math.round(delay / 60000)} minutes.`);
} else if (status >= 400) {
delay = Math.max(this.reloadInterval * 2, FIFTEEN_MINUTES);
Log.error(`${this.url} - Client error (${status}). Retrying in ${Math.round(delay / 60000)} minutes.`);
} else {
Log.error(`${this.url} - Unexpected HTTP status ${status}.`);
}
return {
delay,
error: new Error(`HTTP ${status} ${statusText}`.trim())
};
}
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* Fetches and processes calendar data
2016-04-15 12:18:59 +02:00
*/
async fetchCalendar () {
this.clearReloadTimer();
let nextDelay = this.reloadInterval;
try {
const response = await fetch(this.url, this.getRequestOptions());
if (!response.ok) {
const { delay, error } = this.getDelayForResponse(response);
nextDelay = delay;
this.fetchFailedCallback(this, error);
} else {
this.serverErrorCount = 0;
const responseData = await response.text();
try {
const parsed = ical.parseICS(responseData);
Log.debug(`Parsed iCal data from ${this.url} with ${Object.keys(parsed).length} entries.`);
this.events = CalendarFetcherUtils.filterEvents(parsed, {
excludedEvents: this.excludedEvents,
includePastEvents: this.includePastEvents,
maximumEntries: this.maximumEntries,
maximumNumberOfDays: this.maximumNumberOfDays
});
[calendar] fix: prevent excessive fetching on client reload and refactor `calendarfetcherutils.js` (#3976) The bottom line of this PR is, that it fixes an issue and simplifies the code by dealing with the TODOs in the code. For review, I suggest looking at each commit individually. If there are too many changes for a PR, let me know and I'll split it up :slightly_smiling_face: ## 1. [fix(calendar): prevent excessive fetching with smart refresh strategy](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976/commits/8892cd3d5aa3aca45ccc97e1eaff4685c98b3fcd) - Add lastFetch timestamp tracking to CalendarFetcher - Add shouldRefetch() method with configurable minimum interval (default: 3 minutes) - When reusing existing fetcher: fetch if data is stale (>3 min), otherwise broadcast cached events - Prevents double broadcasts to consuming modules while maintaining fresh data - Balances rate limit prevention (Issue https://github.com/MagicMirrorOrg/MagicMirror/issues/3971) with data freshness on user reload - Prevents excessive fetching during rapid reloads (e.g., Fully Kiosk screensaver use case) - Allows fresh calendar data when enough time has passed since last fetch ## 2. [refactor(calendar): simplify event exclusion logic](https://github.com/MagicMirrorOrg/MagicMirror/commit/d507aba82d68bb0feb0c3b45a321f9407f95d311) - Extract filtering logic from `shouldEventBeExcluded` into new helper `checkEventAgainstFilter` - Simplify the main loop in `shouldEventBeExcluded It resolves a TODO from the comments in the code: * `This seems like an overly complicated way to exclude events based on the title.` ## 3. [refactor(calendar): extract recurring event expansion logic](https://github.com/MagicMirrorOrg/MagicMirror/commit/d510160bd214d1440cb8c2f7598cf48ec0c19c36) This change separates the expansion of recurring events from the main filtering loop into a new helper function 'expandRecurringEvent'. It resolves two TODOs from the comments in the code: - `This should be a separate function` - `This should create an event per moment so we can change anything we want` This improves code readability, reduces complexity in 'filterEvents', and allows for cleaner handling of individual recurrence instances. ## 4. [refactor(calendar): simplify recurring event handling](https://github.com/MagicMirrorOrg/MagicMirror/commit/b04f716cc0f74883bbcf8d812d76ee56ebca8fe5) - Simplify 'getMomentsFromRecurringEvent' using modern syntax - Improve handling of full-day events across different timezones ## 5. [test(calendar): fix UNTIL date in fullday_until.ics fixture](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976/commits/1d762b2ade2b24e157e66b83774cdef4ccc95c52) The issue was with the UNTIL date being May 4th while DTSTART was May 5th. This created an invalid recurrence rule where the end date came before the start date. The fix only adjusts the UNTIL date from May 4th to May 5th, so it matches the start date.
2025-12-08 10:07:04 +01:00
this.lastFetch = Date.now();
this.broadcastEvents();
} catch (error) {
Log.error(`${this.url} - iCal parsing failed: ${error.message}`);
this.fetchFailedCallback(this, error);
}
}
} catch (error) {
Log.error(`${this.url} - Fetch failed: ${error.message}`);
this.fetchFailedCallback(this, error);
}
this.scheduleNextFetch(nextDelay);
}
2016-04-15 12:18:59 +02:00
[calendar] fix: prevent excessive fetching on client reload and refactor `calendarfetcherutils.js` (#3976) The bottom line of this PR is, that it fixes an issue and simplifies the code by dealing with the TODOs in the code. For review, I suggest looking at each commit individually. If there are too many changes for a PR, let me know and I'll split it up :slightly_smiling_face: ## 1. [fix(calendar): prevent excessive fetching with smart refresh strategy](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976/commits/8892cd3d5aa3aca45ccc97e1eaff4685c98b3fcd) - Add lastFetch timestamp tracking to CalendarFetcher - Add shouldRefetch() method with configurable minimum interval (default: 3 minutes) - When reusing existing fetcher: fetch if data is stale (>3 min), otherwise broadcast cached events - Prevents double broadcasts to consuming modules while maintaining fresh data - Balances rate limit prevention (Issue https://github.com/MagicMirrorOrg/MagicMirror/issues/3971) with data freshness on user reload - Prevents excessive fetching during rapid reloads (e.g., Fully Kiosk screensaver use case) - Allows fresh calendar data when enough time has passed since last fetch ## 2. [refactor(calendar): simplify event exclusion logic](https://github.com/MagicMirrorOrg/MagicMirror/commit/d507aba82d68bb0feb0c3b45a321f9407f95d311) - Extract filtering logic from `shouldEventBeExcluded` into new helper `checkEventAgainstFilter` - Simplify the main loop in `shouldEventBeExcluded It resolves a TODO from the comments in the code: * `This seems like an overly complicated way to exclude events based on the title.` ## 3. [refactor(calendar): extract recurring event expansion logic](https://github.com/MagicMirrorOrg/MagicMirror/commit/d510160bd214d1440cb8c2f7598cf48ec0c19c36) This change separates the expansion of recurring events from the main filtering loop into a new helper function 'expandRecurringEvent'. It resolves two TODOs from the comments in the code: - `This should be a separate function` - `This should create an event per moment so we can change anything we want` This improves code readability, reduces complexity in 'filterEvents', and allows for cleaner handling of individual recurrence instances. ## 4. [refactor(calendar): simplify recurring event handling](https://github.com/MagicMirrorOrg/MagicMirror/commit/b04f716cc0f74883bbcf8d812d76ee56ebca8fe5) - Simplify 'getMomentsFromRecurringEvent' using modern syntax - Improve handling of full-day events across different timezones ## 5. [test(calendar): fix UNTIL date in fullday_until.ics fixture](https://github.com/MagicMirrorOrg/MagicMirror/pull/3976/commits/1d762b2ade2b24e157e66b83774cdef4ccc95c52) The issue was with the UNTIL date being May 4th while DTSTART was May 5th. This created an invalid recurrence rule where the end date came before the start date. The fix only adjusts the UNTIL date from May 4th to May 5th, so it matches the start date.
2025-12-08 10:07:04 +01:00
/**
* Check if enough time has passed since the last fetch to warrant a new one.
* Uses reloadInterval as the threshold to respect user's configured fetchInterval.
* @returns {boolean} True if a new fetch should be performed
*/
shouldRefetch () {
if (!this.lastFetch) {
return true;
}
const timeSinceLastFetch = Date.now() - this.lastFetch;
return timeSinceLastFetch >= this.reloadInterval;
}
2020-08-03 11:19:54 +02:00
/**
* Broadcasts the current events to listeners
2016-04-15 12:18:59 +02:00
*/
broadcastEvents () {
Log.info(`Broadcasting ${this.events.length} events from ${this.url}.`);
this.eventsReceivedCallback(this);
}
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* Sets the callback for successful event fetches
* @param {(fetcher: CalendarFetcher) => void} callback - Called when events are received
2016-04-15 12:18:59 +02:00
*/
onReceive (callback) {
this.eventsReceivedCallback = callback;
}
2016-04-15 12:18:59 +02:00
2020-08-03 11:19:54 +02:00
/**
* Sets the callback for fetch failures
* @param {(fetcher: CalendarFetcher, error: Error) => void} callback - Called when a fetch fails
2016-04-15 12:18:59 +02:00
*/
onError (callback) {
this.fetchFailedCallback = callback;
}
}
2016-04-15 12:18:59 +02:00
module.exports = CalendarFetcher;