Nylas N1 Application Architecture
本帖最后由 Qter 于 2020-10-16 17:38 编辑https://docs.nylas.com/docs/how-nylas-works
Data storage
When you authenticate an account to Nylas, one of our sync servers will claim the account and automatically start syncing mail, calendar, and contact data into Nylas' sharded MySQL database. We store mail, contact, and calendar data indefinitely in our database until it is deleted by an API request or the account is deleted from our system.Further, we sync all data for an account, sometimes called "historical mail". This means if you authenticate an account that has 10 years worth of data, we will sync that into our system. There are various filtering mechanisms on our API that allow you to filter out unwanted data when making API requests.We also cache files, attachments, and copies of the raw RFC-2822 MIME message in an Amazon S3 bucket for 7 days. If you request the raw message within the first seven days, we'll return our cached copy from S3. After that, we'll delete the extra copy of the raw message from S3, and any further requests for the data will be proxied to the original mail server to re-retrieve the original data.
https://docs.nylas.com/reference#obtaining-a-delta-cursor
https://docs.nylas.com/docs/sync-strategies#deltas
Data Syncing
SUGGEST EDITS
How Nylas syncs mail, contacts, and calendars
When an account is connected to Nylas, the sync engine starts pulling in all mail for the account, prioritizing recent messages first (though mail isn’t strictly added in reverse chronological order). As new email arrives in the user’s mailbox, it is also added in parallel. Click to see an overview of How Nylas Works.For contacts and calendar syncing, events and contacts are downloaded from newest to oldest. New events or contacts added after the start of the sync are not downloaded until the initial sync is complete. Since the volume of data for contacts and calendar is typically not large, this isn’t generally a problem.Nylas continues to keep its data up-to-date with the backend provider through a number of mechanisms: keeping an IMAP IDLE connection open, using Exchange ActiveSync’s “ping” notifications, using webhooks from calendar providers, or, in the case that no better mechanism is available, simply polling the provider.
Syncing Strategies
Overview
There are two main ways to pull in email information from an account:
[*]The Threads and Messages endpoints allows you retrieve all messages and threads, or filter for a certain subset of them based on various constraints.
[*]Deltas allow you to process new data quickly without having to fetch an index of the user’s mailbox or perform a large number of API calls.
You shouldn’t make any assumptions about the sync progress of an account. We recommend using both mechanisms to ensure you’re able to quickly provide data to a user when the account starts syncing (through the threads endpoint), and to quickly update the user when new mail comes through (through deltas). There are other caveats you should keep in mind that are discussed below.
Threads and Messages
In some cases, it’s easiest to request data from the Nylas API as it’s needed for display, without ever needing to store it locally. The Threads and Messages endpoints are great for this use case.For example, if your application provides a custom “Inbox” view, you could fetch data from the /threads endpoint and use it to render HTML or provide data to a single page JavaScript application. Making /threads requests on-demand rather than caching the data in a local database makes it easier to ensure the data is always current, and it’s easy to filter messages too.🚧Important NoteWhen you first connect an account and fetch data from the /threads endpoint you won’t receive all of a user’s email if Nylas is still syncing historical mail. You can’t make any assumptions about whether an account has finished syncing on Nylas’ servers, and it can take anywhere from a few minutes to several days depending on the account size.📘Sync TipYou could use the delta streaming purely as a notification method. New delta comes in? Just request the thread endpoint again for new messages. (Don’t forget to stay within our rate limits!)
Deltas
The Nylas Sync Engine builds a transaction log that records every change as it synchronizes users’ mailboxes. Your application can use this transaction log, exposed through the Delta APIs, to build email applications that process incoming changes quickly, without fetching an index of the user’s mailbox, polling, or making a large number of API requests.The Delta API documentation describes the endpoint, request parameters, and response objects in detail. It also describes the difference between streaming and long polling and when you would typically use each strategy.If your application wants to ingest emails as they are processed by the Nylas Sync Engine, the delta stream is an efficient solution.
Deltas + Threads
By themselves, the Threads and Deltas APIs aren’t sufficient to build a full, realtime view of a user’s mailbox, since:
[*]Threads only return mail that has already been processed by Nylas. If a user has just connected their account, a request to /threads might only return a few Thread objects that Nylas has processed. A few hours later, /threads might return many thousand objects since more mail has been processed.
[*]Deltas only return mail as it’s being processed by Nylas. Listening to deltas tells your application that mail is available which was was not previously synced or has been modified.
To build a robust application that maintains an up-to-date cache of a user’s entire mailbox, you should leverage both Threads and Deltas. Be sure to:
[*]Begin listening to deltas (changes to the mailbox) at the same time you start paginating /threads (existing data in the mailbox.)
[*]Always “upsert” when you retrieve objects. A delta about a thread should create or update that thread in your cache.
If your application is only interested in caching a subset of mail—for example, mail in the last month, or mail in a particular folder—you should:
[*]Limit pagination using threads filter parameters
[*]Listen for deltas, but ignore deltas about mail you do not care about.
In general, you should obtain the cursor first before paginating through /threads to ensure you don’t miss any emails.
[*]Authorize an account
[*]Obtain a cursor
[*]Start listening for deltas
[*]Paginate /threads (remember to account for duplicates)
Managing multiple account syncs
The Delta API documentation goes into detail on when to use long polling versus streaming to call the delta endpoint. Depending on what strategy you use and your software stack, the number of parallel processes or open connections when syncing a large number of accounts may be a concern.In our experience, clients have been successful using the Delta API with large numbers of accounts using the following asynchronous frameworks:
[*]Python: gevent
[*]Ruby: EventMachine
[*]NodeJS: (built in asynchronous IO)
页:
[1]