When you are downloading data fields for reporting, time stamp fields are included to indicate when the processing of a given web request under Web Gateway Cloud Service has begun.
You are also required to use time stamp filters to limit the amount of data that is downloaded.
Time stamp fields
The time stamp is downloaded both in Unix epoch time format and as a formatted UTC time string.
Unix epoch time: Number of seconds that have passed since and including January, 1, 1970
Time stamp filters
Using the following time stamp filters is required. They specify the beginning and end of a filtering interval during which the processing of requests for web access is logged. For this reason, these time stamps are also referred to as request time stamp filters.
|Filter||Limits the downloaded data to ...|
Requests whose time stamp is greater than or equal to the specified time stamp
Requests whose time stamp is less than the specified time stamp
The filter command uses this format:
The meaning of the variables in this command is as follows.
seconds1 and seconds2 - Number of seconds at the beginning and end of the Unix epoch time range that is specified
To prevent duplicates, this range does not include the time specified by seconds2. If you download data, for example, every 20 minutes, there is no overlap between the last records of any particular download and the first records of the next download.
records - Number of records with data fields logged and downloaded for reporting
The number is limited to 10,000,000. This is also the default.
number - Number for an instance of the order parameter, see also the next section.
You can use more than one order parameter in a filter command and identify these parameters by numbers that you add as separate parameters to have some filtering results sorted in ascending and others in descending order of their time stamps. Use 0 for the first order parameter.
The following filter command restricts the downloading of data fields to httpAction and mediaType. Results are sorted in ascending order for the former and in descending order for the latter.
Created time stamp filters
The created time stamp filters have the same format as the request time stamp filters. The created time stamp indicates the time when a logging entry for a web request is created in the database. Its use is optional, but highly recommended when you are attempting to download current or near real-time data.
Web requests can be logged hours after they were initially processed, but usually this happens within 3 minutes. Using the created time stamp allows you to pull data for web requests that were processed long before they were logged.
Use of the request time stamp filters is still required then. When using the created time stamp filters, make the range of the request time stamps larger than that for the created time stamps. For example, configure the request time stamp range to begin 24 hours before and end one minute after the created time stamp range, as shown in the following.
filter.requestTimestampFrom = filter.createdTimestampFrom - 24 * 3600 filter.requestTimestampTo = filter.createdTimestampTo + 60
We recommend that you limit the time stamp range to an interval that returns less than 1 GB of data as a result and pulls data in less than one minute.
We recommend that you specify a range between 2 and 60 minutes, basing the size of the range on the number of records you generate per hour. Specifying larger ranges might cause a download request to time out.
When dealing with older data and not trying to download real-time or near real-time data, do not use the created time stamp filters.
Here are two examples of downloads where data is almost pulled in real time and two where it is more than 24 hours old. For the real-time data, both request and created time stamp filters are used.
- This download was run on April, 7, 2022, at 5:30:20 p. m. The data was logged during a four-minutes time range that finished only two minutes before the download started. The logging lasted from 5:24:20 to 5:28:20 p. m. The data was processed during the 24 hours that preceded the download.
The requestTimestampTo filter was set here to createdTimestampTo + 60 seconds.
2. This download was run four minutes later than the one shown in the first example, on April, 7, 2022, at 5:34:20 p. m. Again, the logging of the data happened during a four-
minutes time range that finished only two minutes before the download started, while the data was processed during the last 24 hours before the download.
The time for createdTimestampFrom was the same as for createdTimestampTo in the first download. This means the two logging time ranges followed each other immediately.
You can continue like this to download data from time ranges that follow each other, for example, in four-minute intervals. By splitting up downloads you can avoid overloading
that might lead to a timeout of the download request.
3. This download was run to pull data that was more than 24 hours old. Only the requestTimestamp filters were used here. The download was run on April, 8, for data that was
processed during twelve hours on April, 6, from 0:00 a. m. to 12:00 p.m.
4. This download was also run on April, 8, to pull the data that was processed during the remaining twelve hours of April, 6, from 12:00 p. m. to April, 7, 0:00 a. m.