Most architects knew how Adobe Analytics data is collected and stored in cloud for our analysis & usage i.e. Data Collection Methods. I went through Adobe Documentation and noticed that the information is distributed around the documentation, often hard to understand in a single view. This article would help everyone to understand the summary and use cases for Data Collection Methods. With the launch of Adobe Experience Cloud Platform, I foresee a big shift in data collection components, but the core idea should remain the same.
Getting right into the topic, below are the typical methods to collect data for Adobe Analytics as on article date.
- Tracking Pixel (Beacon) [Real-time or Delayed]
- Data Insertion API (1.4) or Bulk Data Insertion API (2.0) [Real-time or Historical]
- Data Source [Usually Historical]
- Data Connector [Usually Real-time with minimum latency]
Tracking Pixel (Beacon)
Tracking Pixel or Analytics Beacon is the most common way to collect data from our Website or Mobile Application i.e. Client-side. Analytics Library typically use JavaScript for Website and it is based on the iOS, Android & React platforms for Mobile Application i.e. JAVA, OIP, JavaScript etc. We usually call this Tracking Pixel as Web Beacon but I don’t want to use the term ‘Web’ personally because it is common for both Websites and Mobile Applications; not only for Websites.
For Website, the request method for Tracking Pixel by default is GET and the request header is mostly in gif (image/gif). This is also why we call Tracking Pixel as ‘Image Request’. Since because Tracking Pixel is client-side data dollection (Browser), if the length of request URL exceeds 2048 Character Bytes, Analytics Library will initiate the request through POST and thus the request header would be in xhr (text/plain).
For Mobile Application, the request method is usually POST and the request header is mostly in xhr (text/html). From my knowledge, we cannot set or alter the request method for Tracking Pixel since it is triggered from Adobe Analytics Library and editing the Library File is not recommended by Adobe.
One more important note is that Tracking Pixels are collected and stored in the chronological order based on when the data hit the collection servers. For Mobile Application, often customer offline usage were queued and sent to Adobe Analytics when the application was back to online. In such situations, we should inform Adobe to use the timestamp in the Tracking Pixel rather than when the data reaches the collection servers. This can be achieved by enabling the Timestamp Data for Report Suites.
Data Insertion API (1.4) or Bulk Data Insertion API (2.0)
Where Tracking Pixels are not an option, Data Insertion APIs are used i.e. iOTs, digital kiosks, smart watches, interactive screens etc. Data Insertion APIs are mostly used for server-side data collection and to capture behavioral data. There are some limitations to server-side data collection when compared to client-side data collection using Tracking Pixels but supports both HTTP POST & HTTP GET request methods for submitting data to collection servers. Data Insertion APIs are primarily used for real-time data collection, however with the recent evolution of Analytics 2.0 APIs, Bulk Data Insertion API (BDIA) can be used for either real-time or historical data (Batch files are CSV files for historical data ingestion).
If we use Data Insertion API (DIA), we must build software that processes the server calls, inserts them in the correct order, responds to errors, and uses sufficient concurrency to achieve the desired ingestion volume. In contrast, the Bulk Data Insertion API (BDIA) solves the use cases and removes the additional burdens required with the DIA alone. It does this by providing a productized system that scales, handles errors, and addresses the finer details of inserting data into Adobe Analytics.
Data Source
Data Source lets you manually import additional online or offline data for reporting through FTP or API. The important difference between Data Insertion APIs and Data Sources is that Data Sources are mostly used to capture non-behavioral data from Customer Relationship Management (CRM). Other examples of non-behavioral data that marketers and analysts would be interested to include are Call Center inputs, Offline or In-store purchases, Returns or Cancellations etc.
Although Data Sources are often used to import additional data by a match it is still possible to upload full processing where processing data is treated as new standard server call. Adobe recommends customers to use the Bulk Data Insertion API (BDIA) instead of full processing data sources and also planning to deprecate full processing data sources in the future.
Data Connector
Data Connector is an auto integration setup that helps us import tracking data from third-party applications into Analytics to gather and use data at a single location. Adobe provides these integrations out of box where minimum perquisites are needed to integrate solutions instead of starting from the base unlike Data Insertion API or importing data manually using Data Sources. The partner solutions are already listed in the Data Connector interface and few popular solutions are DFA, Appsflyer, Clicktale, Silverpop etc. Integration steps are mostly related to Authentication & Variables Mapping rather than being completely technical and instructions are already available in the documentation.
Be advised that the enablement of integration through Data Connectors often incur fees in accordance with our service agreement with Adobe or our service agreement with one of Adobe’s trusted partners, as applicable. There might be recurring and implementation fees associated with these integrations or sometime for free.
Aside, the feature is powered by legacy technology that is no longer viable or supported (Expected to discontinue in approximately 12-18 months). We have a new standard in the Adobe Exchange Partner Program that should be adopted for any integrations that wish to continue to be offered and supported.
Summary
Based on the platforms and requirement we can decide the method to collect data in Adobe Analytics. Tracking Pixels, real-time Data Insertion API calls, per row of CSV file in Bulk Data Insertion API, full-processing Data Source types where individual hits are sent in as rows of data and Data Connector imports (In addition with recurring and implementation fees, if any) are treated as individual Server Calls for Adobe Analytics. On the other hand, partial uploads with variables match through Data Sources (FTP & API) are not counted for the Server Calls.
Also note that Classification is not a Data Collection method but one of the Data Import option in Adobe Analytics i.e. You can’t add a new dimension or a metric rather can add meta data, classify or organize the existing dimensional line items. Happy data collecting!
