How to ingest data

Whatever the data source (mobile apps, factory floor devices, etc.), you'll likely be dealing with an ever-increasing volume of data. The front-end interface needs to handle this traffic as efficiently as possible, and also provide the ability to scale as traffic increases.

In the sample projects, you'll see a few different implementations for front-end API: IIS-hosted WebAPI, OWIN-hosted WebAPI, and Node.js. As each code sample is published, additional documentation will be added here.

IIS+WebAPI

WebAPI provides a simple, REST-based interface for data ingestion. In the SimpleIngestAndPersist project, the WebAPI layer is implemented in a simple DeviceController with a single Post method, UpdateDeviceStatus:

        [HttpPost]
        public HttpResponseMessage UpdateDeviceStatus([FromBody]DeviceData Data)
        {
            // Ingest the data 
            ingestRepository.Ingest(Data);

            // Return HTTP 201 to client.
            var responseMsg = Request.CreateResponse(HttpStatusCode.Created);
            return responseMsg;
        }

This simply takes raw input data and passes it to an implementation of IIngestRepository, calling the Ingest() method. This is an extremely lightweight approach, and may easily be scaled out (horizontally) across many front-end virtual machine instances.

IIS, in this case, manages connection-handling and request queueing. Further, IIS has many performance counters showing information such as number of requests being handled and queued request count.

Last edited Nov 14, 2013 at 1:07 PM by DMakogon, version 7