[code] MapPoint - CSS Uploads with Zip and Chunking
The Microsoft MapPoint Web Service allow for uploading location data in XML following the Access 2003 XML format. The Customer Data Service limits uploads to 1MB chunks, but it allows the XML to be zip compressed. That's nice because zipping XML can cut the file size by up to 90%.
I didn't find any good sample code for zipped, chunked uploads from an XML string, despite some concerted googling. The MapPoint SDK sample code fulfils the technical requirements without being at all useful in real life - it shows how to upload a file. If you were going to upload a file, you might as well use the MapPoint CDS website.
In my case, I got a serializable object from a webservice, serialized it to XML, transformed it with XSLT to Access 2003 XML, zipped it, and did a chunked upload to the MapPoint CDS webservice. I'm not going to go through the process of transforming XML - there are plenty of good references for that. I'm goint to assume you've got an XML string in Access 2003 XML format and you're ready to upload it. This code requires a WebReference to the MapPoint CSS webservice. This is .NET 1.1 code so I'm using SharpZipLib for zip compression; if you're running on .NET 2.0 you've got native zip compression so you can eliminate that dependency.
using System.Configuration;
using System.IO;
using System.Xml;
using System.Xml.Serialization;
using Mappoint;
using MappointCSS;
using ICSharpCode.SharpZipLib.Zip.Compression.Streams;
using ICSharpCode.SharpZipLib.Zip;
namespace MappointUploader
{
public class UploadWorker
{
private bool WriteLocalFiles;
public UploadWorker()
{
}
public void Process()
{
outputConsole("Starting process.");
outputConsole("Connecting to CRM Webservice.");
WriteLocalFiles = (ConfigurationSettings.AppSettings["WriteLocalFiles"] == bool.TrueString);
string uploadXML = string.Empty;
//TODO: Set uploadXML to an XML string in Access 2003 XML format.
// In my case, I'm calling a webservice and transforming the output with an in-memory XSL transform.
//Write file for debugging / production support
writeToFile(uploadXML, @"XML\TransformOutput.xml");
//Load string into byte array and compress using "upload.xml" as the document name inside the zip
byte[] uploadBytes = System.Text.Encoding.UTF8.GetBytes(uploadXML);
outputConsole("Compressing XML (" + uploadBytes.Length + " bytes uncompressed).");
uploadBytes = compress(uploadBytes,"upload.xml");
outputConsole("Completed compressing XML (" + uploadBytes.Length + " bytes compressed).");
//Connect to Mappoint Customer Data Service for upload
UploadSpecification uploadSpec = new UploadSpecification();
System.Net.NetworkCredential credentials = new System.Net.NetworkCredential(
ConfigurationSettings.AppSettings["CSSUser"],
ConfigurationSettings.AppSettings["CSSPass"], "PARTNERS");
CustomerDataService cds = new CustomerDataService();
cds.Credentials = credentials;
cds.PreAuthenticate = true;
UploadSpecification spec = new UploadSpecification();
spec.DataSourceName = ConfigurationSettings.AppSettings["Mappoint.DataSourceName"];
spec.EntityTypeName = ConfigurationSettings.AppSettings["Mappoint.EntityTypeName"];
if(ConfigurationSettings.AppSettings["Environment"] == "Service")
{
spec.Environment = LocationDataEnvironment.Service;
}
else
{
spec.Environment = LocationDataEnvironment.Staging;
}
spec.MaximumGeocodingLevel = GeocodingLevel.Street;
spec.RejectAmbiguousGeocodes = false;
string jobID = cds.StartUpload(spec);
outputConsole("Connecting to Mappoint CDS (" + spec.Environment + " environment, Job ID " + jobID + ").");
writeToFile(uploadBytes, @"XML\TransformOutput.xml.zip");
int byteCount = uploadBytes.Length;
long bytesUploaded = 0;
int chunkSize = 1000000;
byte[] chunk;
while(bytesUploaded < byteCount)
{
if(byteCount-bytesUploaded > chunkSize)
{
chunk = new byte[chunkSize];
}
else
{
chunk = new byte[byteCount-bytesUploaded];
}
Array.Copy(uploadBytes, bytesUploaded, chunk, 0, chunk.Length);
bytesUploaded = cds.UploadData(jobID, chunk, bytesUploaded);
outputConsole("Uploaded " + bytesUploaded + " bytes.");
}
cds.FinishUpload(jobID, bytesUploaded);
outputConsole("Upload complete.");
}
private void writeToFile(string input, string filename)
{
byte[] buffer = System.Text.UTF8Encoding.UTF8.GetBytes(input);
writeToFile(buffer, filename);
}
private void writeToFile(byte[] buffer, string filename)
{
if(WriteLocalFiles)
{
using (FileStream fs = new FileStream(filename,FileMode.Create))
{
fs.Write(buffer,0,buffer.Length);
}
}
}
private byte[] compress(byte[] buffer, string entryFileName)
{
MemoryStream memory = new MemoryStream();
ZipOutputStream stream = new ZipOutputStream(memory);
stream.IsStreamOwner = false;
stream.SetLevel(5);
ZipEntry entry = new ZipEntry(entryFileName);
entry.DateTime = DateTime.Now;
entry.Size = buffer.Length;
ICSharpCode.SharpZipLib.Checksums.Crc32 crc = new ICSharpCode.SharpZipLib.Checksums.Crc32();
crc.Reset();
crc.Update(buffer);
entry.Crc = crc.Value;
stream.PutNextEntry(entry);
stream.Write(buffer, 0, buffer.Length);
stream.Finish();
stream.Close();
return memory.ToArray();
}
private void outputConsole(string message)
{
string output = string.Format"{0} : {1}", DateTime.Now.ToLongTimeString(), message);
using (StreamWriter sw = File.AppendText("RunStatus.log"))
{
sw.WriteLine(output);
}
Console.WriteLine(output);
}
}
}
Now, about that Access 2003 XML data format. It's finicky, since it includes XSD which describes the data format of all columns, and CDS rejects the upload if it doesn't match. One trick is to model it in a simple Access table and export as XML (with XSD), then use that in your XSL template or XML source. Get something really simple working and add a field at a time if necessary.
Remember that MapPoint requires a few fields in your upload data. - EntityID, Latitude, and Longitude. The EntityID must be a unique integer. Since my datasource used a GUID identifier, I manufactured a unique int ID in the XSLT:
<EntityID>
<xsl:number value="position()" format="1" />
</EntityID>
...
The Latitude and Longitude are required double (SQL float) columns; if a value is present then MapPoint will not geocode the position and will use the provided Lat / Long values. Even if you're not planning on overriding geocoding, though, these columns must be present. Robert McGovern's MapPoint article on DevEx provides a valid sample XML file with EntityID, Latitude, and Longitude that you can buld on, and there's more info on MSDN - Mappoint Data Source Format.