Sitecore GeoIP

High Score Labs News • June 23, 2018

Another handy tool that is available in Sitecore is GeoIP lookup. It ties into other features in Sitecore, for example, personalization based on their location using GeoIP based conditions ( https://doc.sitecore.net/sitecore_experience_platform/setting_up_and_maintaining/ip_geolocation/ip_geolocation/getting_started_with_sitecore_ip_geolocation_service )

Sometimes, it would make sense to use the MaxMind GeoLite2 GeoIP databases instead of Sitecore IP Geolocation service. This is because using MaxMind GeoLite2 service will not require any Sitecore license changes (free of charge) – but, on the other hand, it could be considerably less accurate. Basically, the Sitecore IP Geolocation service is utilizing the MaxMind’s GeoIP2 service (more accurate) and keeps them up to date (https://kb.sitecore.net/articles/569970). In this article, we will review the full process of switching to MaxMind GeoLite2 service.

First, we need to implement our custom GeoIP Lookup Provider (add MaxMind.GeoIP2 from nugget):

using System;

using System.Collections.Generic;

using System.Linq;

using System.Web;

using Sitecore.Analytics.Model;

using MaxMind.GeoIP2;

using Sitecore.Diagnostics;

using Sitecore.Analytics.Lookups;

namespace Sitecore.Test

{

public class MaxMindProvider : LookupProviderBase

{

MaxMindGeolocationService _maxMindLookupService = new MaxMindGeolocationService();

public override WhoIsInformation GetInformationByIp(string ip)

{

return _maxMindLookupService.GetWhoIs(ip);

}

And…

using MaxMind.GeoIP2;

using Sitecore.Analytics.Model;

using Sitecore.Diagnostics;

using System;

using System.Web.Hosting;

namespace Sitecore.Test

{

publicclassMaxMindGeolocationService

{

public override WhoIsInformation GetWhoIs(string ip)

{

try

{

using (var reader = new DatabaseReader(GetDatabaseFileFullPath()))

{

Log.Info($”{nameof(MaxMindGeolocationService)} ip: {ip}”, this);

var city = reader.City(ip);

if (city != null)

{

var info = new WhoIsInformation();

info.Isp = “MaxMind”;

info.AreaCode = “N/A”;

info.BusinessName = “N/A”;

info.City = city.City.Name;

info.Country = city.Country.IsoCode;

info.MetroCode = city.City.GeoNameId.HasValue ? city.City.GeoNameId.Value.ToString() : “N/A”;

info.Longitude = city.Location?.Longitude;

info.Latitude = city.Location?.Latitude;

info.PostalCode = city.Postal?.Code;

return info;

}

catch (MaxMind.GeoIP2.Exceptions.AddressNotFoundException ex)

{

Log.Warn($”No data for ip {ip}.”, ex, this);

}

catch (Exception ex)

{

Log.Warn($”{nameof(MaxMindGeolocationService)} error: {ex.Message}.”, ex, this);

}

return WhoIsInformation.CreateUnknown();

}

public static string GetDatabaseFolder()

{

var folderSetting = Configuration.Settings.GetSetting(“MaxMind.DB.Folder”);

var folder = folderSetting.StartsWith(“~/”) ? HostingEnvironment.MapPath(folderSetting) : folderSetting;

return StringUtil.EnsurePostfix(‘\\’, folder);

}

public static string GetDatabaseFileFullPath()

{

var file = GetDatabaseFolder() + Configuration.Settings.GetSetting(“MaxMind.DB.FileName”);

return file;

}

Next, add our MaxMindProvider to the config patch file: MaxMind.config

<patch:attribute name=”defaultProvider”>maxmind</patch:attribute>

<param>

<patch:delete />

</param>

</add>

</providers>

</lookupManager>

</settings>

</sitecore>

</configuration>

Next, download the archive from http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz and extract the database file from it. Then, place it on each CM server website folder, located in “~/app_data/maxmind/”. Our MaxMind provider is now ready to go.

Now, we want to test GeoIP provider for different IPs on our local environment. However, our local server will only detect IP addresses like 0.0.0.0 or 127.0.0.1. So, the GeoIP information provider does not return correct information for us to test it. To fix that, lets create an extra processor in the CreateVisit pipeline to change our IP:

using Sitecore.Analytics.Pipelines.CreateVisits;

using Sitecore.Diagnostics;

using System.Net;

using Sitecore.Analytics;

namespace Sitecore.Test

{

public class ChangeIP : CreateVisitProcessor

{

public override void Process(CreateVisitArgs args)

{

Assert.ArgumentNotNull((object)args, “args”);

string ip = new IPAddress(Tracker.Current.Interaction.Ip).ToString();

if (ip != “0.0.0.0” && ip != “127.0.0.1”)

{

return;

}

IPAddress address;

if (IPAddress.TryParse(Configuration.Settings.GetSetting(“Geolocation.Testing.IP”), out address))

{

args.Interaction.Ip = address.GetAddressBytes();

}

Below, is the Sitecore config patch file for utilizing our testing IP (Geolocation.Testing.config):

<processor type=”Sitecore.Test.ChangeIP, Sitecore.Test”

patch:after=”processor[@type=’Sitecore.Analytics.Pipelines.CreateVisits.XForwardedFor, Sitecore.Analytics’]”>

</processor>

</createVisit>

</pipelines>

</settings>

</sitecore>

</configuration>

Now, on our local environment, our IP address will be changed to the one from the patch file and our geolocation data should be resolved correctly.

One common problem of using GeoIP functionality in Sitecore is that it runs asynchronously and then caches its results in server memory at the first level and in the database at the second level (Mongo DB for Sitecore 8.2 and in xDB for Sitecore 9). So this means that the IP address is only resolved on the second web request that is made to Sitecore. This is done by design due to performance issues, however, in our case, it isn’t an issue since the dataset is local and we are not communicating with a 3^rd party web service to process the request. In addition, even when using the Sitecore GeoIP service, it may be a requirement to resolve GeoIP data on the first request (to redirect to the related language page version for example). So, lets go ahead and fix that as well in this article. For that, you can use a Sitecore support fix code from (https://kb.sitecore.net/articles/320734) – this fix is valid for Sitecore 8.2 and 9 as well:

Create a new processor for the CreateVisits pipeline which will update GeoIP data synchronously:

using Sitecore.Analytics.Pipelines.CreateVisits;

using Sitecore.Configuration;

using Sitecore.Diagnostics;

using System;

namespace Sitecore.Test

{

public class UpdateGeoIpData : CreateVisitProcessor

{

public override void Process(CreateVisitArgs args)

{

Assert.ArgumentNotNull((object)args, nameof(args));

int intSetting = Settings.GetIntSetting(“Analytics.PerformLookup.CreateVisitInterval”, 5);

args.Interaction.UpdateGeoIpData(TimeSpan.FromSeconds((double)intSetting));

}

Create a patch config file for this new processor, for example – _Support.396075.config (NOTE – our support fix processor should follow after our ChangeIP processor, so I named it with the leading “_”) :

<!– ANALYTICS PERFORM LOOKUP CREATE VISIT INTERVAL

Specifies the timeout (in seconds) to wait during the visit creation (first request for Visit)

to ensure that GeoIp information has been received from Lookup provider.

Default: 5

–>

</settings>

<patch:delete/>

</processor>

<processor type=”Sitecore.Test.UpdateGeoIpData, Sitecore.Test”

patch:after=”processor[@type=’Sitecore.Analytics.Pipelines.CreateVisits.XForwardedFor, Sitecore.Analytics’]” />

</createVisit>

</pipelines>

</sitecore>

</configuration>

So, now we have switched from the Sitecore GeoIP service to the MaxMind provider which utilizes a local database. In addition, we have an option to test different visitor IPs locally and we fixed the issue with async call during the first visit.

As a final “dessert” option, we will want to update our MaxMind databases periodically (according to https://dev.maxmind.com/geoip/geoip2/geolite2/). MaxMind databases are updated once a month on the first Tuesday. So, you have an option to update databases manually on each server, not more than once a month, or you can use the Sitecore scheduling agent on each server to update databases automatically. The code for such an agent is provided below (utilizes Microsoft.CSharp nugget package):

using Sitecore.Configuration;

using Sitecore.Data;

using Sitecore.Diagnostics;

using Sitecore.IO;

using System;

using System.Collections.Generic;

using System.IO;

using System.IO.Compression;

using System.Linq;

using System.Net;

using System.Text;

using System.Web;

using ICSharpCode.SharpZipLib.Zip;

using ICSharpCode.SharpZipLib.Tar;

using ICSharpCode.SharpZipLib.GZip;

namespace Sitecore.Test.Agents

{

public class MaxMindUploadDatabase

{

public void Run()

{

try

{

string url = Configuration.Settings.GetSetting(“MaxMind.DB.City.Url”);

Assert.IsNotNullOrEmpty(url, “url”);

var newID = ID.NewID.ToShortID().ToString().ToLower();

var newDir = $”{MaxMindGeolocationService.GetDatabaseFolder()}{newID}”;

var zipFileFullPath = $”{MaxMindGeolocationService.GetDatabaseFolder()}{newID}-GeoLite2-City.tar.gz”;

using (var client = new WebClient())

{

client.DownloadFile(url, zipFileFullPath);

}

Assert.IsTrue(FileUtil.Exists(zipFileFullPath), $”{zipFileFullPath} not uploaded”);

Directory.CreateDirectory(newDir);

this.ExtractTGZ(zipFileFullPath, newDir);

var directories = Directory.GetDirectories(newDir);

Assert.IsTrue(directories.Any(), “MaxMindUploadDatabase : directories.Any()”);

//one dir

var dirInfo = new DirectoryInfo(directories[0]);

var files = dirInfo.GetFiles(“*.mmdb”);

//one file

Assert.IsTrue(files.Any(), “MaxMindUploadDatabase : files.Any()”);

files[0].CopyTo(MaxMindGeolocationService.GetDatabaseFileFullPath(), true);

//clean up

File.Delete(zipFileFullPath);

Directory.Delete(newDir, true);

}

catch (Exception exc)

{

Log.Error($”MaxMindUploadDatabase agent error: {exc.Message}”, this);

throw exc;

}

public void ExtractTGZ(string gzArchiveName, string destFolder)

{

using (var inStream = File.OpenRead(gzArchiveName))

{

using (var gzipStream = new GZipInputStream(inStream))

{

using (var tarArchive = TarArchive.CreateInputTarArchive(gzipStream))

{

tarArchive.ExtractContents(destFolder);

tarArchive.Close();

}

In the provided example of the database download agent, we are basically downloading the latest MaxMind GeoLite2 City databases to the server’s temporary folder, unzip it, and overwrite the old database with new one and delete the temporary folder.