1.0-beta.8: Millions of unwanted Trace events causing monthly quota to be reached prematurely
ajhuntsman opened this issue · comments
We only post PageView and custom Event transactions to the TelemetryClient. That's it.
Here's our configuration bootstrap:
ApplicationInsights.setup(context, application);
ApplicationInsights.setDeveloperMode(!BuildConfigUtil.isRelease());
ApplicationInsights.disableAutoCollection();
ApplicationInsights.disableAutoPageViewTracking();
In our September release, we added this line of code to our bootstrap code:
ApplicationInsights.enableAutoSessionManagement();
When the September release went live, the number of Trace events in our dashboard went into the millions, causing our monthly quota to be reached only a few days into each month. Notably, in previous releases, we still saw Trace events which we are NOT explicitly tracking.
We are currently experimenting with Application Insights for several enterprise apps; and this is a very serious issue which will likely determine whether or not we continue to evaluate ApplicationInsights.
At the very least, can you add a method like this?
ApplicationInsights.disableTraceTracking();
Such a method implementation should be trivial, and the outcome should be that when this method is invoked, zero Trace events should be recorded.
For now, we will be adding these lines to our configuration:
//ApplicationInsights.enableAutoSessionManagement();
ApplicationInsights.disableAutoSessionManagement();
ApplicationInsights.setExceptionTrackingDisabled(true);
Thanks!
+1 to ajhuntsman
+1 as well
Thanks for reporting. Can you please provide us with some samples?
Thank you!
Samples of the Trace events? They're all just Log statements.
@ajhuntsman I just integrated the SDK in a simple sample app and can't reproduce your issues (using the exact same setup as you posted above) I'm also not 100% sure what you mean by trace.
Are you sure, you're not using
TelemetryClient.getInstance().trackTrace("Foobar");somewhere?
A screenshot of the Azure Portal or a sample of one of you trace events (enable debug logging and have a look in LogCat) would be really helpful.
Hi @TroubleMakerBen, thanks for taking some time to at least triage this a bit!
We use a layer of abstraction for analytics, so that it's easy to switch providers, use multiple providers, and keep the analytics footprint almost non-existent throughout client code.
Our analytics provider implementation is what actually records an event. Our providers are implemented as singletons; in this case: ApplicationInsightsAnalyticsProvider. Source code below.
As you can see, our interfacing method public void recordEvent(@NotNull AcmeAnalyticsEvent event) is the only point of entry, and its implementation only invokes TelemetryClient.getInstance().trackPageView(...) and TelemetryClient.getInstance().trackEvent(...). I also searched our entire project for "trackTrace" and didn't get any search results, as expected.
My gut feeling here is that some internal code within Application Insights is recording Trace events. It smells a lot like Aspect-Oriented Programming (AOP), since the message attribute of each Trace event is exactly the same as a log message. For example, here's one Trace.message:
Acme:App.initStorageDirectory: 164: FILE_STORAGE_DIRECTORY = '/storage/sdcard0/Android/data/org.acme/files'
Is there any chance that App Insights uses AOP internally, or has some other fancy code to intercept logging?
Here's our provider source code of our analytics provider:
package org.acme.analytics.impl.microsoft;
import android.app.Application;
import android.content.Context;
import com.microsoft.applicationinsights.library.ApplicationInsights;
import com.microsoft.applicationinsights.library.TelemetryClient;
import com.microsoft.applicationinsights.library.config.Configuration;
import org.acme.Constants;
import org.acme.analytics.AcmeAnalyticsEvent;
import org.acme.analytics.AcmeAnalyticsProvider;
import org.acme.util.BuildConfigUtil;
import org.acme.util.AcmeLogger;
import java.util.Map;
import java.util.TreeMap;
import javax.validation.constraints.NotNull;
/**
* The implementation for Microsoft Application Insights analytics.
*/
public class ApplicationInsightsAnalyticsProvider implements AcmeAnalyticsProvider {
/**
* Bill Pugh singleton implementation.
*/
private static class SingletonHolder {
public static final AcmeAnalyticsProvider INSTANCE = new ApplicationInsightsAnalyticsProvider();
}
/**
* Gets an instance of {@link org.acme.analytics.AcmeAnalyticsProvider}.
*/
public static AcmeAnalyticsProvider getInstance() {
return SingletonHolder.INSTANCE;
}
@Override
public void initialize(Context context, Application application) {
ApplicationInsights.setup(context, application);
ApplicationInsights.setDeveloperMode(!BuildConfigUtil.isRelease());
ApplicationInsights.disableAutoCollection();
ApplicationInsights.disableAutoPageViewTracking();
ApplicationInsights.setExceptionTrackingDisabled(true);
// Enabling session management has the side effect of MILLIONS of Trace events!
//ApplicationInsights.enableAutoSessionManagement();
ApplicationInsights.disableAutoSessionManagement();
Configuration config = ApplicationInsights.getConfig();
config.setMaxBatchIntervalMs(Constants.MS_APP_INSIGHTS_THROTTLE_INTERVAL);
config.setMaxBatchCount(Constants.MS_APP_INSIGHTS_THROTTLE_COUNT);
ApplicationInsights.start();
}
@Override
public void onPause() {
// no-op
}
@Override
public void onResume() {
// no-op
}
@Override
public void recordEvent(@NotNull AcmeAnalyticsEvent event) {
// Don't record events for developer builds
if (BuildConfigUtil.isDebug()) {
return;
}
try {
Map<String, String> eventProperties = new TreeMap<>();
eventProperties.putAll(event.getAttributes());
eventProperties.putAll(event.getCommonAttributes());
if (event.getEventType().isPageView()) {
TelemetryClient.getInstance().trackPageView(event.getEventType().getEventName(), eventProperties);
} else {
TelemetryClient.getInstance().trackEvent(event.getEventType().getEventName(), eventProperties);
}
} catch (Exception e) {
AcmeLogger.logException(e);
}
}
}
Because this is a blocking issue, since it causes the loss of all analytics data, is it possible to add a "disable" method for Trace events to the configuration, at least as a reasonable stop-gap for now?
Hi @ajhuntsman,
My gut feeling here is that some internal code within Application Insights is recording Trace events.
Nope, it's not. Feel free to check out the source and look for occurences of trackTrace.
Is there any chance that App Insights uses AOP internally, or has some other fancy code to intercept logging?
Nope, we're not intercepting anything. FYI, the ApplicationInsights SDK isn't actually meant for logging which is why we don't send traces/logging out of the box.
For me, the log message Acme:App.initStorageDirectory: 164: FILE_STORAGE_DIRECTORY = '/storage/sdcard0/Android/data/org.acme/files' looks a lot like a custom log message. Can you please check in your initStorageDirectory()?
About your last question:
If it's a bug in AI, we'll do something about it. But before that, I want to make sure it's a bug on our end and it very much looks like it isn't.
If you want so see what the AI SDK sends out, enable debug logging and check logCat for messages by com.microsoft.applicationinsights.
In theory, it could also be an issue in the backend, but I very much doubt it as we're using the SDK internally without any issues with trace() (we're actually not using trace()).
Hi @TroubleMakerBen, following your recommendations, I've checked out the code and scoured around for TrackDataOperation.DataType.TRACE, EnvelopeFactory#createTraceData and MessageData. It certainly doesn't appear that any of these are used where unexpected.
In response to your observation, yes, this is a custom log message:
Acme:App.initStorageDirectory: 164: FILE_STORAGE_DIRECTORY = '/storage/sdcard0/Android/data/org.acme/files'
We have tons of custom log messages; all of which funnel through a singleton Logger utility class. Our custom logger class does some line number parsing, and just hands off to either Crashlytics or the standard Android log:
/**
* This proxy should be used for all logging, so logging implementation can be changed in just one place.
*/
public class AcmeLogger {
...
private static final String TAG = "ACME";
public static void logInfo(@NotNull String message) {
log(Log.INFO, message);
}
public static void log(int priority, @NotNull String msg) {
if (msg == null || msg.isEmpty()) {
return;
}
// Parse the stack for the class and method name
try {
StackTraceElement exceptions[] = Thread.currentThread().getStackTrace();
StackTraceElement exception = exceptions.length > 4 ? exceptions[4] : exceptions[3];
String fullClassName = exception.getClassName();
String className = fullClassName.substring(fullClassName.lastIndexOf(Constants.DOT) + 1);
String methodName = exception.getMethodName();
int lineNumber = exception.getLineNumber();
msg = TAG + Constants.COLON + className + Constants.DOT + methodName + Constants.COLON +
Constants.SPACE_1 + lineNumber + Constants.COLON + Constants.SPACE_1 + msg;
} catch (Exception e) {
// OK to ignore
}
if (BuildConfigUtil.isCrashReportingEnabled() && !BuildConfigUtil.isRobolectricTestEnabled()) {
// Crashlytics will write its own log AND write to LogCat
// http://support.crashlytics.com/knowledgebase/articles/120066-how-do-i-use-logging
Crashlytics.log(priority, TAG, msg);
} else {
Log.println(priority, TAG, msg);
}
}
...
}
Every single Trace event in our dashboard has a message attribute whose value is a custom log statement. In other words, there are no Trace events for anything other than custom log statements.
Somehow, Trace events are getting created, and only for our custom log statements.
Any suggestions???
Hi,
did you release a version in which the Crashlytics.log(priority, TAG, msg); has been replaced with TelemetryManager.trackTrace(...);? Maybe it's a good idea to check the app version which was producing the traces.
Hi @chrwend, I looked through the history of our logger and DID find that we were invoking traceTrace in 1 production build which shipped several versions ago.
It looks as if we still have a few thousand users on that old build, and they have not upgraded to a newer version (despite our modal dialog encouraging them to).
So a skeleton in our closet is at fault here; sorry for taking up your time!
@ajhuntsman stuff like that just happens. :) Glad to hear it you resolved it.